Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Authors: Marvin F. da Silva
Felix Dangel
Sageev Oore
Venue: ICML 2025 spotlightposter (2025)
Keywords: generalization
symmetry
sharpness
flatness
riemannian geometry
loss landscape
Primary area: deep_learning->theory

Abstract: The concept of sharpness has been successfully applied to traditional architectures like MLPs and CNNs to predict their generalization. For transformers, however, recent work reported weak correlation between flatness and generalization. We argue that existing sharpness measures fail for transformers, because they have much richer symmetries in their attention mechanism that induce directions in parameter space along which the network or its loss remain identical. We posit that sharpness must account fully for these symmetries, and thus we redefine it on a quotient manifold that results from quotienting out the transformer symmetries, thereby removing their ambiguities. Leveraging tools from Riemannian geometry, we propose a fully general notion of sharpness, in terms of a geodesic ball on the symmetry-corrected quotient manifold. In practice, we need to resort to approximating the geodesics. Doing so up to first order yields existing adaptive sharpness measures, and we demonstrate that including higher-order terms is crucial to recover correlation with generalization. We present results on diagonal networks with synthetic data, and show that our geodesic sharpness reveals strong correlation for real-world transformers on both text and image classification tasks.

Hide & Seek: Transformer Symmetries Obscure Sharpness & &iiemannian Geometry Finds It
Marvin F. da Silva 12 Felix Dangel
Sageev Oore!
Abstract
broader theoretical insights into generalization itself.
The concept of sharpness has been successfully applied to traditional architectures like MI and CNNs to predict their generalization. For trans- formers, however, recent work reported weak cor- relation between flatness and generalization. We argue that existing sharpness measures fail for transformers, because they have much richer sym- metries in their attention mechanism that induce directions in parameter space along which the net- work or its loss remain identical. We posit that sharpness must account fully for these symme- tries, and thus we redefine on a quotient man- ifold that results from quotienting out the trans- former symmetries, thereby removing their ambi- guities. Leveraging tools from Riemannian geom- etry, we propose a fully general notion of sharp- ness in terms of geodesic ball on the symmetry- corrected quotient manifold. In practice, we need to approximate the geodesics. Doing so up to first order yields existing adaptive sharpness measures, and demonstrate that including higher-order terms is crucial to recover correlation with gen- cralization. We present results on diagonal nets with synthetic data and show that our geodesic sharpness reveals strong correlation with general- ization for real-world transformers on both text and image classification tasks.
There is a long history of hypotheses linking sharpness to generalization, but evidence has been conflicting (Hochre- iter & Schmidhuber, 1994; Andriushchenko et al., 2023). Generalization has been speculated as correlating with flat- ness, but recent evidence has indicated that, in the case of transformers, it has little to correlation whatsoever. Mea- sures of sharpness have varied widely, ranging from trace of the Hessian to worst-case loss within a local neighborhood, with adaptive and relative variations proposed to address specific challenges (Kwon et al., 2021; Petzka et al., 2021).
We suspect that some of the confusion stems from the speci- ficity of the problem these measures have attempted to ad- dress: the issue of parameter rescaling. In contrast, we argue that rescaling (Dinh et al., 2017) is merely a special case of a broader, more fundamental obstacle to measuring sharpness accurately: the presence of full and continuous parameter symmetries. Addressing this challenge is crucial to ensure that we are studying the right quantity when investigating the relationship between sharpness and generalization.
Beyond discrete permutation symmetries, neural nets nat- urally exhibit continuous symmetries in their parameter space. These symmetries are intrinsic, data-indeppndent properties that emerge from standard architectural compo- nents. For example: normalization layers (Ioffe & Szegedy, 2015; Ba al., 2016; Wu & He, 2018) induce scale invari- ance on the pre-normalization weights (Salimans & Kingma, 2016); homogencous activation functions like ReLU intro- duce re-scaling symmetries between pre- and post-activation weights (Dinh et al., 2017); some normalization layers and softmax impose translation symmetries in the preceding layer's s biases (Kunin et al., 2021). As a result, arguably almost any NN, along with its corresponding loss, exhibit symmetries and can therefore represent the same function using different parameter values (Figure la).
1. Introduction
Predicting generalization of neural nets (Nes)-the discrep- ancy between training and test set performance-renee remains an open challenge. Generalization-predel prediccive metrics are valu- able though: they enable explicit regularization of training to enhance generalization (Foret et al., 2021), 1), and provide
Adaptive flatness (Kwon et al., 2021) accounts for some symmetries, both element- and filter-wise re-scaling, but fails to capture the attention mechanism's 's full symmetry, represented by GL(h) (re-scaling by invertible h x matri- ces, where h the hidden dimension), as we will discuss later. Aiming break the cycle between discovery of a
Facully of Computer Science, Dalhousie University. Halifax, Canada À Veector Insitute Artificial Intelligence, Toronto, Canada. Correspondence to: Marvin F. da Silva <marvinf silva dal.ca
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
(a) Loss
(b) Euclidean gradient norm
(c) Riemannian gradient norm
1.00
1,0
0.75
0.8
0.50
25
0.6
0.00
85
0.4
50
0.3
75
0000
0.0
01
Figure 1: Quantities from the Riemannian quotient manifold respect the loss landscape's symmetry; Euclidean quantities do not. We illustrate this here for synthetic least squares regression task with two-layer NN, where ++ 020 1 with scalar parameters O E R²² and input v E R (i.e. each layer is linear function). The NN is re-scale invariant, i.e. has (1) symmetry: For any a E IR \ {0}, the parameters (01, 0%2) = (a 10, represent the same function. (a) The loss function inherits this symmetry and has hyperbolic level sets. (b) The Euclidean gradient norm does not share the loss function's geometry and changes throughout an orbit where the NN function remains constant. (c) The Riemannian gradient norm follows the loss function's 's symmetry and remains constant throughout an orbit, i.e., it does not suffer from ambiguities for two points in parameter space that represent the same NN function.
specific symmetry and techniques to deal with it, we ask:
follow geodesic paths in the quotient manifold rather than straight lines in the ambient space.
Can we provide one-size-fits-man recipe for developing symmetry-invariant quantities for a wider range of continu- ous symmetries?
(c)
For diagonal nets, we analytically solve geodesic sharp- ness and find strong correlation with generalization. Then, we apply our approach to the unstudied and higher-dimensional GL(h) symmetry in the attention mechanism. On both large vision transformers and language models, we empirically find stronger corre- lation than any previously seen (that we are aware of) between our geodesic sharpness and generalization.
Here, we positively answer this question by proposing a prin- cipled approach to eliminate ambiguities stemming from symmetry. Essentially, this boils down to using the geom- etry that correctly captures symmetry-imposed parameter equivalences. We apply concepts from Riemannian geome- try to work on the Riemannian quotient manifold implied by a symmetry group (Boumal, 2023, $9). We thus identify objects on the quotient manifold like the Riemannian met- ric and gradient-and and show how to translate them back to the Euclidean space. Our contributions are the following:
2. Related Work
Symmetry versus reparameterization: Kristiadi et al. (2023) pointed out how to fix ambiguities stemming from reparameterization, i.e. change of variables to a new pa- rameter space: Invariance under reparameterization follows by correctly transforming the (often implicitly treated) Rie- mannian metric into the new coordinates. Our work focuses on invariance of the parameter space M under a symmetry group 9 with action &b G x M + M, (g. 0) + (6gg 0) that operates on a single parameter space.
(a) We introduce the application of Riemannian geome- try (Boumal, 2023) to the study of NN parameter space symmetries by using geometry from the quotient man- ifold induced by a symmetry as a general recipe to remove symmetry-induee -nnuucd ambiguities in parameter space. We do so by translating concepts like gradients from the quotient manifold back to the original space through horizontal lifts.
Symmetry teleportation: Another ways to use symmetry- implied ambiguity is to view it as a degree of freedom and develop adaptation heuristics improve algorithms which are not symmetry-agnosst (Zhao et al., 2022a).
(b) Based on (a), we propose and analyze geodesic sharp- ness, a novel adaptive sharpness measure: By Taylor- expanding our refined geometry, we show that (i) sym- metries introduce curvature into the parameter space, which (ii) results in previous adaptive sharpness mea- sures when ignored. Geodesic sharpness differs from traditional sharpness measures in two key aspects: (i) the norm of the perturbation parameter is redefined to reflect the underlying geometry; (ii) perturbations
Geometric constraints & NN dynamics: Previous stud- ies analyze how parameter space symmetries impose geo- metric constraints on derivatives and introduce conserved quantities during training (Kunin et al., 2021). Our approach
2

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
differs by systematically removing symmetry-induced am- biguity through quotienting out the the symmetry group.
3. Preliminary Definitions, Notation & Math
Generalization measures: We consider a neural net fo with parameters 0 E F that trained on data set D using loss function é by minimizing the empirical risk
We generalize earlier post-hoe solutions for simpler sym- metries (e.g., GL())) to more complex, higher-dimensione symmetries such as Gh,, common in neural network attention mechanisms. Unlike Kunin et al. (2021), who con- sider geometry in augmented spaces for simpler symmetries, we directly use the quotient space geometry. Objects are then 'lifted' back into the original space, yielding symmetry- corrected quantities. This method provides a principled framework capable of handling high-dimensional symme- tries, leading to a more effective dimensionality reduction.
Di (0)
E( {(fo(a),))
D
Our goal is to compute a quantity on the training data that predictive of the network's generalization, i.c. performance held-out data set.
Sharpness: A popular way to predict generalization is via sharpness-ias -ic., how much the loss changes when per- turbing the weights in a small neighbourhood- like average (Sww) or worst-case sharpness (Smax) (Keskar et al., 2017)
Quotient manifolds in deep neural networks: Ranga- mani et al. (2019) introduce quotient manifold construc- tion for re-scaling symmetries and then use the Riemannian spectral norm as measure of worst-case flatness. This differs from our approach in several ways:
Sae Es (f 5) Ls(0)), 5 ~ 1~O00.pp1
Smax Es max 1A12Sp
(Ls(0 + -
(a) Our approach is more general and contains both the (h( symmetry of transformers, and the original re- scaling/scaling symmetry of CNNs/MLPs, rendering it applicable to a wider range of modern architectures.
with batches S ~ Drain of size [3] m, neighbourhood size p. and perturbation 8. Near critical points, they closely re- late to the Hessian H (and thus parameter space curvature): Swg x Tr ()) and Smax X Amax H))
(b) Our experimental setup is more challenging: we test on large-scale models (large transformers vs CNNs) and large-scale datasets (ImageNet vs CIFAR-10), Sharp- measures that account for re-scaling/scaling sym- metries (c.g. adaptive sharpness) work quite well CIFAR-10 with CNNs, and tends to break down on datasets like ImageNet with transformers.
daattive sharpness: Hessian-based sharpness measures can be made to assume arbitrary values by rescaling param- eters, even though the NN function stays the same. To fix this inconsistency, Kwon et al. (2021) proposed adaptive sharpness (invariant under special symmetries), and An- driushchenko et al. (2023) use adaptive notions of sharpness that invariant element- tiise scaling.
(c) Conceptually, Rangamani et al. (2019) defines worst- case sharpness on the usual norm-ball, appropriately generalized to the Riemannian setting. We propose instead that the ball should be the one traced out by geodesics, to better respect the underlying geometry.
Ts(0 + 8) - (1) [180ellaSp
S max (wn, c) Egg
max
with scaling vector e (usually set to 10), Kwon et al., 2021).
(d) Performance-wise, our approach is cheaper as it does not use the Hessian and only uses symmetry-corrected gradients (see Dagréou et al. (2024) for an in-depth cost comparison of computing Hessians vs gradients).
The problem: Adaptive sharpness only considers the sym- metry induced by element wise re-scaling. But symmetries of transformers go beyond the invariance that adaptive sharp- ness captures. Maybe unsurprisingly, Andriushchenko et al. (2023) find inconsistent trends for adaptive sharpness in transformers, with sharpness failing to correlate with gen- eralisation, versus other architectures. We hypothesize this is related to adaptive sharpness not accounting for the full symmetry in transformers. In this paper, we address this. The central question is: If adaptive sharpness is the fix for a special symmetry, can we provide a more general solu- tion for the symmetries of transformers, to fix the above inconsistency?
Relative sharpness: Another promising approach to sharpness was proposed by Petzka et al. (2021), where the generalization gap is shown admit a decomposition into representativeness term and feature robustness term. Fo- cusing on the feature robustness term, they introduce relative sharpness, which is invariant to a layer- and neuron- weiee re-scaling, and performs better than traditional sharpness measures (Adilova al., 2023; Walter et al., 2025).
3

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
3.1. Symmetries in Neural Networks
canonical form in Definition 3.1. In contrast to the symme- tries from Section 3.1, they lead to more drastic dimension- ality reduction. Consider for example a single self-attention layer where h = hv hgg. The number of trainable pa- rameters is 4h2² and the two GL.(d) symmetries reduce the effective dimension to 4h²² - 2 dimm(dlaaaa (h) = 2hh, i.c. i.e. they render half the parameter space redundant. We hypothesize that the impact of -iw-deeniioonll symmetry on objects like the Euclidean Hessian's 's trace (Dinh et al., 2017) may be amplified for such higher-dimensional symmetries.
Here, we give brief overview and make the notion of NN symmetries more concrete, focusing those studied by Kunin et al. (2021). Those symmetries lead to rather small effective dimensionality reduction as they are often of G1( or GL (1), , but they can still impact the network behaviour. Let denote the parameters of neural net, 1 a binary mask, and 1-A its complement such that their sum is a vector of ones, 1 +1A 1. Let A == 0 @ 1A with O the element-wise product. Further, let A1, be two disjoint subsets, A1 A2 = 0 with masks 1A, 1A2 Then we have the following common symmetries, characterized by their symmetry group G, such that for any E G the parameters (g, f) and f represent the same function:
3.3. Mathematical Concepts for Riemannian Geometry
We now outline required properties of manifolds for the full development of our approach. We list essential concepts here, and provide definitions and a brief review Appendix B. For further information, see for instance Lee (2003). Fig- ure 2 illustrates the main concepts we will require.
Translation: aa 0) = 14 O a a with a E A
Scaling: (a 0) - - + A with a E R>o
A2 + +0 (A(VA)) Re-scaling: vb 0) A + with a E R>0
Ambient embedding space: We assume that the manifold of possible parameters is embedded in a linear Euclidean space & = Idd with d the number of parameters. We can think of E as the ambient space. For instance, for loss function 7: E + R, 0 + Z(0). we can use ML libraries to evaluate its value, as well as its Euclidean gradient
Their associated groups are - R², (L)() In practice, there may be multiple symmetries acting onto disjoint parameter sub-spaces. Note that scaling is essen- tially the symmetry that adaptive sharpness corrects for.
O
grad 8=
E R
3.2. Rescale Symmetry of Transformers
08:
i=1,....d
Transformers exhibit a nigher-dimension symmetry than the previous examples; we formalize the treatment of this symmetry in the following canonical form.
Because the geometry of & is flat i.e. uses the standard metric 02) 0, Oz, this object consists of partial derivatives. However, the Riemannian generalization will add correction terms. In what follows we consider only the restriction of objects like to the parameter manifold.
Definition 3.1 (Functional GL-symmetric building block). Consider a function F(G. H) on m x R that con- sumes two matrices G E R H E Rm but only uses the product GH T i.e. [(G, H)= = () for some over mxn, J is symmetric under the general linear group
Definition 3.2. We take M to be the manifold of network parameters, and consider it sub-manifold embedded into ε, the computational space of matrices on which all our numerical calculations are done. We call M the total space. On the total space we have a loss function ?: M + R.
Gh = {A E h A invertible]
with (GL))L()) = 112 and action
Our goal is calculate derivatives/geometricc quantities af- removing the NN's symmetries. The symmetry relation induces natural equivalence classes, which we write [0), and explain in Appendix B.I. We let M = M/ ~ represent the quotient of the original parameter space manifold by the equivalence relation ~ associated with the symmetry (Appendix B.2). We also require tangent vectors; these are straightforward on the total space M, but the tangent space of the quotient manifold, M, requires more machinery: ver- tical and horizontal spaces, and corresponding lifts. These concepts are all defined in Appendix B.3.
((A, (G, H)) = (GA-1 )HAA.
In other words, we can insert and then absorb the iden- tity A into G, H to obtain equivalent parameters GA-1 HAT T that represent the same function.
Example A.2 illustrates GL symmetry for a shallow lin- car net. Indeed, many popular NN building blocks feature this form, most prominently the attention mechanism in transformers (Vaswani et al., 2017). We give the attention symmetry in Example 1, and we provide the symmetry for low-rank adapters (Hu et al., 2022) in Example A.3. These examples are NN building blocks that introduce GL symme- tries into loss function and can all be treated through the
Once we endow M with a smooth inner product over its tangent vectors, we obtain a Riemannian manifold (defined in Appendix B.4). This construction lets us analyze differ-
4

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
M = M/9
E
Ambient embedding space
M
Total space
M
M
Quotient space
g
Symmetry group
F.
Points the total space
2, y
Points on the quotient space
&
Tangent vector in the tangent space at point F, M
Er
Tangent vector the langent space at point x, ..M
EV
Vertical component of & in the vertical space V.M
CH
Horizontal component of Ça in the horizontal space
M.MM M,M, horizontal lift of &.
Figure 2: Illustrative sketch relating total and quotient space and their tangent spaces. tangent vector at a point in total space, εa E T can be decomposed into a horizontal component _ and a vertical component C The vertical component points along the direction where the quotient space x = [+}] remains unaffected. The horizontal component points along the direction that changes the equivalence class. We can use E as a representation of the tangent vector &.r T2M on the quotient space. The component Exd represents the horizontal lift of &.-
ential objects that live on quotient manifolds, in the ambient space in a natural way. Furthermore, this allows us to de- fine the horizontal space as the orthogonal complement of the vertical space (Appendix B.4), and to define a Rieman- nian gradient (Appendix B.5). Most properties from the Euclidean case still hold for the Riemannian gradient, but of particular interest us is the fact that the direction grad is still the steepest-ascent direction of ʃ at point x.
is just the usual definition of a ball, since the geodesics are straight lines. If & E M is horizontal vector, and 7tt) is a geodesic starting at A and with initial velocity E:
max ( L LL(TE(O)) (4) 1E
S (w) Es
If the initial velocity, E, is horizontal vector, then the veloc- ity of the geodesic, 5g. will stay horizontal. The choice of 1 in (1) is not as arbitrary as it first seems (do Carmo, 1992): since for a positive a, 7 %e(at) 7 (f), positions reached with arbitrary t can be reached by instead fixing
We additionally make use of geodesic curves. Intuitively, geodesic curves can either be seen as curves of minimal distance between two points on a manifold M, or equiv- alently, as curves through a given point with some initial velocity, and whose acceleration zero-a a generalization of Euclidean straight lines. See Appendix B.6 for details.
= 1 and manipulating the initial velocity's 's norm via p.
When we do not have an analytical solution for the geodesic, we can use the approximation:
Putting it all together, this gives us a recipe for computing quantities invariant to a given symmetry relation: (i) find a Riemannian metric compatible with this symmetry; (ii) determine the vertical space for the symmetry relation; (iii) use the metric find the orthogonal complement of this vertical space, i.e. the projector into the horizontal space; (iv) find the horizontal geodesics. Combined, these steps allow us to do calculations in the quotient manifold along the proper paths (given by geodesics).
=t(1) (0))=+(())++
+ -
(5)
where E = (E') is the initial (horizontal) velocity, and ri are the Christoffel symbols. We show that geodesic sharp- reduces to adaptive sharpness measures in Appendix F, under appropriate metric choices and by taking first-order approximation to the geodesics. that is, ignoring the terms corresponding to the curvature, r
4. Geodesic Sharpness
5. Geodesic Sharpness in Practice
We posit that adaptive sharpness measures should take into account the geometry of the quotient parameter manifold that arises after removing symmetries from the parameter space. We base our sharpness measure on the notion of a geodesic ball: the set of points that can be reached by geodesics, starting at a point p and whose initial velocity has norm smaller than p. after one time unit. In Rd this
We now apply geodesic sharpness concrete examples. A fully worked out scalar toy model is in Appendix D.
Following previous works by Dziugaite et al. (2020); Kwon et al. (2021); Andriushchenko et al. (2023), we use the Kendall rank correlation coefficient (Kendall, 1938) to as- sess the correlation between generalization and sharpness
5

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
in the empirical validations of our approach:
=00699
=00866
T =00.83
Generalization Gap
2 81, M(M - 1)
Σ
sign(4: 11) sign(s; - -8))
1.5
1.0
0.5
where f. and are the vectors of observed variables between which we are measuring correlation.
2.2
3.0
3.5
Adaptive
Geodesic (inv)
Geodesic (mix)
Although the criterion of symmetry compatibility restricts the class of suitable metrics, these are not necessarily unique. As long as is prefer one metric over another, except for practical aspects like numerical cost and stability. We will present results two commetry-compaticn
Figure 3: Adaptive vs. geodessi sharpness on diagonal nets. The generalization gap is the test loss (remember all models are trained to 10-5 training loss). The correlation coefficient's magnitude is larger for geodesic sharpness.
we have no reason to
metrics that are simple, yet non- trivial, and often used in the related literature on Riemannian optimization on fixed-rank matrix spaces (Luo et al., 2023).
The minimum norm least squares predictor is B. (XT (XX) xx) y = X y. Using Equation (4) (details in Appendix E), we get (to first and to second order)
5.1. Diagonal Networks
We start by studying diagonal linear nets, one of the sim- plest non-trivial neural networks (Pesme et al. (2021), Wood- worth et al. (2020)). These have two parameters, u, v, and predict a label, y. given an input, æ, via y = x (u 3 v)- We consider linear regression with labels y data matrix X Bnxd and take as our loss L(u, v) [ v) v1l2. Our parameter manifold M is RR x IRd.
S mux: inv (u, v) =4p1.30 (go B.)
(8)
+ 4pp max [(86)²]
which depends on and the difference between the learned, and the optimal minimum norm, predictor. Eq. 8 is the square of adaptive sharpness (when the residual |/5b C ll2 is small) if very carefully chosen hyperparameters were used (by contrast, this result naturally appears using our geodesic approach). For the second metric, we have
R="
The nets are symmetric under element-wise rescaling: (u, v) + (ou,a vv), leaves f= = u ④ v and I. invariant.
S max: mix (u, v) = p1lB0 -3.112-
Metric: At point (u, v) E M, for two tangent vectors "7 (Mu: Mo), v (Va, U,) E M, we use the following two symmetry-compatibe metries:
5.1.1. EMPIRICAL VALIDATION
Experimental setup: We follow Andriushchenko et al. (2023), generate a randomly distributed data matrix X, a random ground-truth vector 3 that 90% sparse, and train 50 diagonal networks to 10-5 training loss regression task.
d pu 12)
nave (z))2)>
un in
+
(n, v) mix
We focus on the more practically relevant case of over- parametrization (d > n). One downside of this is that the theoretical expressions derived in the previous section, while useful sanity check, no longer hold (since overparameteri- zation breaks the assumption X T X Id=200). To obtain our geometric sharpness, we directly solve Equation (4).
Horizontal space:
Both have the same horizontal space
The ni
MM = (o) m1o) E Ti
Vi
2²
vi
Results: All three notions of sharpness able to predict generalization (Figure 3). Geodesic sharpness, although closely related for diagonal nets to adaptive worst-case sharpness, does slightly better. This applies to both metrics studied, and they perform roughly the same. See Section 7 for comments about the sign of the correlation.
Geodesics:
= 2e the geodesics are With b, == Th w
Yim (t)² (u) exp(bbt), of exp(b:t))
Ymix (t)² (u) / 2b,t, vi / 2b,t
with starting points un and vg. i.c. the trained parameters.
5.2. Attention Layers
Geodesic sharpness: Assume that X = Ia (An- driushchenko al., 2023), and denote Go u4g @ co-
Next, we look at the symmetric functional block from Defi- nition 3.1. Our computation space E Runh BR mx
6

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
((++m)) and restrict weights to have full column rank:
r
=0044
=-0.71
=00.0
Generalization Gap
0.20
Assumption 5.1. The rank of G, H.H. corresponds to their number of columns, rank G) = rank H) h.
0.15
0.10
This implies h S n, m, which is usually satisfied (multi- head) attention layers (Example A. for the default choices of dy. While the weights of multi-head attention tend to have high column rank (Yu & Wu, 2023), they are not guaranteed to be full column rank. To account for this, we introduce a small relaxation parameter, e, to the Gram ma- trices s.t. G G + G G + ed,. Empirically, we observe that as long as is sufficiently small, it does not affect our results (Appendix H.2). Therefore, we restrict both G, H to the set of fixed-rank matrices, M R2 x IR 1 where h
0.05
Adaptive
Geodesic (inv)
Geodesic (mix)
Figure Adaptive vs. geodesic sharpness on ImageNet ViTs. We 72 trained models from Wortsman al. (2022), and measure their generalization gap the dif- ference between test and train error. The correlation coeffi- cient's magnitude is larger for geodesic sharpness.
R R = {B E Rx Irank( [) = k.. We can represent point E E M by matrix tuple (G, H) B x RR Its tangent space T,Mix is
them with Eq. For horizontal tangent vectors (EG,EH) we have for (.m
T&M {1- (NG, 7H) E R x Rmxk]
(T tn Eo = - Eo (c(GTG)) + EG (11)
+ GG(G (G(GTG)-LEGS
Metric:
We endow M with the two metries (.,-) inv.mix T,M x T,M + R (proof they are valid in Appendix I.)::
(similar for the H components). For (: >ii the geodesic equations are coupled and the components are
in. =-=====1r =
(G G) - + (H) H) "NRSH)
Hy'nn
[(T)) mix 5" EG H + H EHl (H" H)-1 GG) H)-1
(12)
() = II ((H H)naca + (G' Gmkct)
(10)
(the H components are similar, proof in Appendix 1.2).
They differ from the Euclidean metric that simply flattens and concatenates the matrix tuples into vectors and takes their dot product, (n Tr (n&Sa + {H) Impor- tantly, they are invariant under symmetries of the attention mechanism, and thus define valid metrics on the quotient manifold (Absil et al., 2008).
5.3. Transformers
Transformers have mix of attention layers and layers with more restricted symmetries for which adaptive sharpness is more appropriate. We present in Appendix C. how we treat each layer of transformers. We introduce relaxations In Appendix C.2 we present Algorithm 1, which we use to solve for geodesic sharpness.
Horizontal space: For ( mix and ÊG. Rxr we have (for proof, see for example Luo et al. (2023))
5.3.1. 1. EMPIRICAL VALIDATION: VISION TRANSFORMERS
H M EH) LGGH H = GT Ex),
Experimental setup: We follow Andriushchenko et al. (2023), and look at models obtained from fine-tuning CLIP on ImageNet-1k 1k (Radford et al., 2021). Specifically, we use the trained classifiers after fine-tuning CLIP ViT-B/32 on ImageNet with randomly selected hyperparameters from (Wortsman et al., 2022). We compute adaptive worst-case, and our geodesic, sharpness on the same 2048 data points from the ImageNett training set, divided into batches of 256, by calculating sharpness on each batch separately, then av- eraging the results. The generalization gap is the difference between test and training error.
mmix M E(E, G EG H Gf GF H)
Projection onto horizontal space: Given & E T in the total tangent space, the horizontal space is
Hin. mix M {(Ec GA EH H(A T)}
where A jiv solves the Sylvester equation AA - AAT =
B. with A = G T H, B = GIGHTS EH
GH H, whereas A mix has an explicit form: A mix 1/2 HH H) - EG).
Results: Figure 4 shows our results. We find a strong correlation between geodesic sharpness and the generaliza- tion gap on ImageNet. This correlation is stronger than
We are unaware of analytical solutions for the geodesics of either (Eq. 9 and Eq. 10), so we approximate
Geodesics:

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
=0.06
=0.28
=0.38
=00.99
Generalization Gap
Figure 6: Extension of Figure to relative sharpness. We find that rela- tive flatness (Petzka al., 2021) fails to find significant correlation, com- pared to our geodesic sharpness.
0.108
0.101
0.102
7.5 Geodesic (inv)
Adaptive
Geodesic (mix)
Relative
Figure 5: Adaptive vs. vs. geodesic sharpness on MNLI lan- guage models. We use 35 trained models from McCoy al. (2020), and show the generalization gap on the MNLI dev matched set (Williams et al., 2018). Geodesic sharpness shows the largest correlation.
for various points on an orbit that represent the same func- tion. Specifically, we reparametrize using A = aG, where G is random standard Gaussian matrix (which is almost always invertible and sampled once in each run), and control the scale a. We sample one G for each attention head. We compare this with adaptive sharpness.
that observed with adaptive sharpness and is consistently negative, implying that the geodesically sharpest models studied on ImageNet are those that generalize best-contrary to what might have been expected, but consistent with the correlation from the diagonal networks.
Results: Figure 7 visualizes the sharpness ratio before and after reparameterization. The colors represent different values of the scale factor, which goes from 10-2 2 to 10?. Our numerically computed geodesic sharpness remains constant.
5.3.2. EMPIRICAL VALIDATION: LANGUAGE MODELS
1.6
Adkepative
Experimental Setup: We also consider BERT models that were fine-tuned on MNLI (Williams et al., 2018) by Mc- Coy et al. (2020). We compute adaptive worst-case, and our geodesic, sharpness on the same 1024 data points from the MNLI training set, with batches of 128 points, by calcu- lating then averaging sharpness each batch.
Sharpness ratio
Geodesic
1.4
10-2
10-1
109
10a
10a
Scale factor (o)
Results: Figure shows our results. We find a consistent correlation between geodesic sharpness and the generaliza- tion gap on MNLI for both metrics, while adaptive sharpness (r 0.06) cannot find any correlation. The correlation is positive, i.e. geodesically flatter models generalize better.
Figure 7: Variation of adaptive geodesic sharpness within an orbit where the neural net function remains unchanged. We show the ratios between the original sharp- ness and the sharpness obtained after applying a symmetry transformation. Geodesic sharpness stays constant, whereas adaptive sharpness assumes several different values.
6. Additional Experiments
6.1. Comparison With Relative Sharpness
7. Remarks, Limitations & Future Work
Relative sharpness (Petzka et al., 2021) is a promising sharp- ness measure that has proven useful in regularizing trans- former training, outperforming other approaches (Adilova et al., 2023). We compare it with our geodesic sharpness in the language model setting from Section 5.3.2; see Figure 6.
Discovering correlation: Adaptive sharpness, as dis- cussed thoroughly by Andriushchenko et al. (2023), is un- able to reveal a correlation between sharpness and gener- alization for transformers. Our geodesic sharpness con- sistently recovers strong correlation on transformers, and strengthens the correlation the case of diagonal networks.
6.2. Verification of Reparametrization Invariance
Mathematically, geodesic sharpness is invariant to symmetry taasformations of the form of Equation (3). Here, we verify empirically that our practical version that can be computed efficiently numerically close to invariant.
Metric choice:
Our results are robust w.r.t. the choice of metric, as long as it captures the parameter symmetry. The mixed metric yields slightly better results on BERT, perhaps owing to its more stable numerics (e.g. possible inversion of nearly singular matrices is side-stepped). Additionally, the mixed metric avoids calling expensive Sylvester equation solvers and has a simple horizontal space projection.
Experimental setup: We take single batch and language model from Section 5.3.2, and compute geodesic sharpness
8

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Sign of the correlation: One of our surprising results is that the sign of the correlation between geodesic sharpness and generalization varies depending on the setting and is at times negative, somewhat at odds with the common view that sharpness always positively correlates with generaliza- tion (i.e., flatter models generalize better). This artifact is not inherent to our proposed metrics. E.g., adaptive sharpness anti-correlates with generalization our diagonal network setting, but was previously found to positively correlate with generalization on other tasks (K won et al., 2021).
formers (ImageNet), and language models (MNLI), we demonstrated that properly accounting for the transformer symmetries restores the correlation between sharpness and generalization. Interestingly, our findings indicate that the sign of the correlation between sharpness and generalization can vary across tasks, suggesting deeper underlying relation- ships involving data distribution and model structure. This work lays the groundwork for further exploration of these interactions and motivates future research into geometry- informed optimization strategies tailored to transformers.
Our geodesic sharpness improves over adaptive sharpness in the following sense: Where adaptive sharpness finds no correlation, our metrics do find a signed correlation, and where adaptive sharpness finds signed correlation, our metries find a stronger similarly-signed correlation. That is, we for the first time observe consistent correlations within- task for transformers, opening questions for further study.
Impact Statement
This paper presents work whose goal is to advance the study of deep learning. There are potential indirect societal conse- quences of our work, none which we feel must be specifi- cally highlighted here.
Acknowledgements
Limitations: While our geodesic sharpness more gen- eral than previous measures, there remain symmetries for which taking the quotient may be computationally expen- sive or intractable. Still, we think that accounting for some symmetry is better than none, and even under computational constraints it could be useful as diagnostic "probe".
We would like to express our sincere gratitude to Agustinus Kristiadi and Rob Brekelmans for early feedback on the manuscript. Resources used in preparing this research were provided, in part, by NSERC, the Province of Ontario, the Govemment of Canada through CIFAR, and companies sponsoring the Vector Institute.
Our new measures detect previously undetected correlation with generalization. In the process, however, we also discov- ered that the sign of the correlation, while consistent across metrics and models, can vary across tasks. Until this new variability is understood, this limits the utility of geodesic sharpness, e.g. for regularizing transformer training.
References
Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization algo- rithms on matrix manifolds. 2008.
Adilova, L., Abourayya, A., Li, J., Dada, A., Petzka, H., Egger, J.. Kleesick, J., and Kamp, M. Fam: Relative flatness aware minimization, 2023.
Future work: Our work is concerned with accounting for parameter space symmetries that are data-independent. This opens up the question: what is the role of data and how can it be integrated into our framework? A more complete understanding of the interplay between data and parameter symmetries might help explain when geodesic sharpness correlates or anti-correlates with generalization.
Andriushchenko, M., Croce, F., Müller, M., Hein, M., and Flam- marion, N. A modern look at the relationship between sharpness and generalization. 2023.
Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. 2016.
Boumal, N. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
8. Conclusion
In this paper, we revisited the limitations of traditional sharpness measures attempting to predict generalization for transformers, highlighting how traditional sharpness measures fail to properly account for the rich Gh))) sym- metries present in transformers. Addressing this, we intro- duced geodesic sharpness, a measure defined on the Rieman- nian quotient manifold obtained by quotienting out trans- former symmetries. This framework provides a principled, symmetry-aware measure of sharpness and contains prior adaptive sharpness metrics as first-order approximations.
Croce, F. and Hein, M. Reliable evaluation of adversarial robust- ness with an ensemble of diverse parameter-fres freec attacks, 2020.
Dagréou, M., Ablin, P., Vaiter, S., and Moreau, T. How to compute hessian- products? In ICLR Blogposts 2024, 2024.
Dinh, I.., Pascanu, R., Bengio, S., and Bengio, Sharp minima can generalize for deep nets, 2017.
do Carmo, M. Riemannian Geometry. Mathematics (Boston, Mass.). Birkhäuser, 1992 ISBN 9783764334901.
Dziugaite, G. K., Drouin, A., Neal, B., Rajkumar, N., Caballero, E.,, Wang. L., Mitliagkas, I,, and Roy, D. M. search robust
Through experiments on diagonal networks, vision trans-

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
measures of generalization. In Advances in Neural Informa- tion Processing Systems, volume 33, pp. 11723-11733. Curran Associates, Inc., 2020.
Rangamani, A., Nguyen, N. H., Kumar, A., Phan, D., Chin, H., and Tran, T. D. Scale Invariant Flatness Mcasure for Deep Network Minima, February 2019.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness- aware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021.
Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural net- works. Advances in neural information processing systems (NeurIPS), 29, 2016.
Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems (NIPS), 1994.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L.. u., and Polosukhin, I. Attention all you need, In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Walter, N. P., Adilova, L., Vreeken, J., and Kamp, M. The un- canny valley: Exploring adversarial robustness from a flatness perspective, 2025.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Williams, A.. Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112-1 1122, 2018.
Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Gener- alization gap and sharp minima, 2017.
Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I,, Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models, 2020.
Kirrinnis, P. Fast algorithms for the sylvester equation. Theoretical Computer Science, 259(1):623 638, 2001. ISSN 0304-3975.
Wortsman, M. Ilharco, G., Yitzhak Gadre, S., Roclofs, R., Gontijo- Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L.. Model soups: averaging weights of multiple fine-tuned models improves accuracy with- out increasing inference time. arXiv e-prints, 2022.
Kristiadi, A., Dangel, F., and Hennig, P. The geometry of neural nets' parameter spaces under reparametrization. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Kunin, D., Sogastuy-Brena J., Ganguli, S., Yamins, D. L.,, and Tanaka, H. Symmetry, conservation laws, and learning dynam- ics neural networks. In International Conference on Learning Representations (ICLR), 2021.
Wu, and He, K. Group normalization, 2018
Yen, J.-N., Si, S., Meng, 7., Yu, F., Surya Duvvuri, S., Dhillon, I. S., Hsich, C.-J., and Kumar, S. LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. arXiv e-prints, 2024.
Kwon, J., Kim, J., Park, H., and Choi, K. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning (ICML), 2021.
Yu, H. and Wu, J. Compressing transformers: Features are low- rank, but weights are not! Proceedings of the AAAI Conference Artificial Intelligence, 37/9):11007-11113 Jun. 2023.
Lee. J. Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer, 2003. ISBN 9780387954486.
Zhao, B., Dehmamy, N., Walters, R., and Yu, R. Symmetry tele-
portation for accelerated optimization. In Advances Neural Information Processing Systems (NeurIPS), 2022a.
Luo, Y., Li, X., and Zhang. A. R. On geometric connections of embedded and quotient geometries in riemannian fixed-rank matrix optimization, 2023.
Zhao, Y., Zhang. H. and Hu, X. Penalizing gradient norm for efficiently improving generalization in deep learning. ICML, 2022b.
McCoy, R. T., Min, J., and Linzen, T. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance, 2020.
Pesme, S., Pillaud- Vivien, L., and Flammarion, N. Implicit bias of sgd for diagonal linear networks: provable benefit of stochas- ticity, 2021.
Petzka, H., Kamp, M., Adilova, I.., Sminchisescu, C., and Boley,
M. Relative flatness and generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. CoRR, wttps//11..000000 2021.
10

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It (Supplemental Material)
We provide in Table a summary of correlation coefficients between sharpness and generalization for our experiments.
Rank correlation coefficient
Setting
Adaptive sharpness (,,)) geodesic sharpness (,,))-ix,, geodesic sharpness O
Diagonal networks
-0.68
-0.83
-0.86
ImageNet -0.41
-0.71
-0.7
MNLI
0.06
0.28
0.38
Table 1: Summary of the correlation between sharpness measures and generalization. We boldface the best performing metric
In the sections that follow, we provide additional details to supplement the main text.
A. Additional Examples of GL symmetries Symmetries in Neural Networks
Example A.1 (Self-attention (Vaswani al., 2017)). Given a sequence X IRtxd with & tokens and model dimension self-attention (SA) uses four matrices Wq. W1 E Rd44, W. W. Rdxd, (usually, d d, dy) produce new t x d sequence
SA(W WK. W. W.)
XW W. X
(13)
= softmax
XWWW.
ddz
This block contains two GL symmetries: one dimension dg between the key and query projection weights, G, +- Wyp W. and one of dimension d, between the value and out projection weights, G, H < W., W.T. Similar to Eq. 14, we can account for biases in the key, query, and value projections by appending them their weight,
W1 6
W b
+ G.H W. b
G.
W.
4: Commonly, H attention heads { (Wi. W W v.s: W: independently process X and concatenate their results into the final output (usually d, = d, = 4/H). This introduces 2H1 GL symmetries. Everything also applies to general attention where, instead of X, independent data is fed as keys, queries, and values to Eq. 13.
Example A.2 (Shallow linear net). Consider two-layer linear net NN(W2, Wi) W.Waa with weight matrices W1 Rxxds W2 & and some input dd. This net has GL symmetry with correspondence G, H. +- W2, W, T to Definition 3.1. With first-layer bias, we have
x (1)
W bi) W2 (Wr bi)
(14)
corresponding to G, H Wz. (W, b1)T.
Example A.3 (Low-rank adapters (LoRA, Hu et al. (2022))). Fine-tuning tasks with large language models add trainable low-rank perturbation L E [Rdixt R Idda to the pre-trained weight W Xd2
LoRA(W) W + LR
(15)

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
introducing a GL(h) symmetry where G, H. L, R. Yen al. (2024) propose an invariant way train the parameters L, R and show that doing so improves the result obtained via LoRA.
B. Concepts and Review for Riemannian Geometry
Recall that M is the total space: the manifold parameters of our network. Also, on the total space we have a loss function (: M + R. Useful resources are Lee (2003), Absil et al. (2008), and Boumal (2023).
B.1. Orbit of
A symmetry relation naturally defines an equivalence relation: two points x, y E M are equivalent under the symmetry, they can be mapped onto each other by the action,
+
By Eg 2))
(16)
In other words, we let orbit(x:) :- {v x) g E g) be all points on the total space that are reachable from x through the action of G, all points in an orbit are equivalent. Instead of orbit (1)). we will write
[c]] == (y MMEMIN MMMiy~a x~
(17)
for the symmetry-induced equivalence class [x] of E M.
Let's further assume that i is symmetric under G. i.e. for any x E M and all g G. [(x) [()(g.)))
B.2. Quotient M and Natural Projection
we take the quotient of the original parameter space manifold M, by the equivalence relation, ~, induced by the symmetries of our neural architecture, we get a quotient M = M/ ~. Under certain conditions, MM is a quotient manifold. The mapping between point in total space its equivalence class is called the natural projection:
Definition B.1. Let :: : M + M/ ~. be defined by T + x. π is called the natural, or canonical projection. We use x()) to denote æ viewed as a point of M :- M/ m.
B.3. Tangent Space, Vertical and Horizontal Spaces
Tangent vectors on the total space M, embedded in a vector space È can be viewed as tangent vectors to E, but the tangent space of the quotient manifold, M is not as straightforward. First, note that any element & E T&M that satisfies (+{{)[E] = & (where is the differential) is a candidate for representation of & = T.M. These aren't 't unique, and as we wish work without any numerical ambiguity introduce the notions of the vertical and horizontal spaces:
Definition B.2. For a quotient manifold M = MM/ ~-, the vertical space at 7 E .M is the subspace V, = TTF = ker DT(x))
where {ī E M y -~ = is the fiber of ž. The complement V is the horizontal space F: T&M V @ H
Definition B.3. There is only one element & that belongs to H₃ and satisfies Da (7) [Ex] E. This unique vector is called the horizontal lift of of & at Z. We denote the operator that affects the procedure by lift (-) When the ambient space, & is a subset of R=x,, the horizontal space can also be seen as such a subset, providing convenient matrix representation of priori abstract tangent vectors of M.
B.4. Riemannian Manifold
We give our total space M smooth inner product over its tangent vectors to give a Riemannian manifold.
Definition B.4. A Riemannian manifold is pair (M, g), where M is a smooth manifold and g is Riemannian metric, defined as the inner product on the tangent space T for each point x E M, M,(.)) T.M x T.M + R. We also use the notation (, >> to denote the inner product.
Note that this definition is not as arcane as it may appear since any smooth manifold admits a Riemannian metric, and we can consider the space of parameters of most neural architectures as constituting a smooth manifold, admitting at least a simple, Euclidean, metric.
12

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
The horizontal space can now be defined the orthogonal complement of the vertical space: H3 (V2)+ {u T&M-
(u, u) for all V2). Additionally, letting Ir denote the metric on M, if for every M and every Ex, &a in T.M, In (Ez, (x) does not depend on ž (xx) then, Is (En Cx) In Gn(En, Ca) defines a valid metric the quotient manifold M.
B.5. Riemannian Gradient
Definition B.5. If f is smooth scalar field on Riemannian manifold M, then the gradient of f at T. grad is the unique element of T&M such that
(g)ad
(f())) ( T.M
If f is a function on M, that induces a function f on quotient manifold M of M, then we can express the horizontal lift of grad at œ as
lift lirta f) g())
B.6. Geodesic Curves Definition B.6.
Geodesic curves, 7, the curves of minimal distance between two points manifold M. The distance along the geodesic is called the geodesic distance. If M is a Riemannian quotient manifold of M, with canonical projection and 7 is geodesic on M, then 7 = π 7 is geodesic curve on M.
(b) Alternatively, geodesics, 7(() can be defined eurves from given point p E M, (i.e., 7(0 p), with initial velocity, (0) = & E TpM, such that their acceleration is zero (a generalization of Euclidean straight lines). This characterizatton provides with the following equation in local coordinates for the geodesic:
dea d12
L drtt dr jer
dl dl
where a are the Christoffel symbols, P derived the curves that minima of the energy functional
(ggan Ox
Ogun Oze
Ogan Ox
Additionally, the geodesics can also be
S(>) = = s10)(1(t),)0
This second perspective will prove useful for the geodesics of the attention layers.
If the initial velocity tangent vector, ç, is horizontal then, Vt, (t) E Ha(t), that is, if the velocity vector starts out as horizontal, then it will stay horizontal. We call these geodesics, horizontal geodesics. The curve 7 = π f is geodesic of the quotient manifold M, with the same length 7. This also holds the other way, i.e., geodesic in the quotient manifold can be lifted to horizontal geodesic in the total space.
C. Geodesic sharpness: practical concerns
C.1. Transformers
Transformers, introduced by Vaswani et al. (2017), consist of multiheaded self-attention and feedforward layers, both wrapped in residual connections and layer normalizations. Visual transformers, in addition, tend to have convolutional layers. Mathematically, focusing for the moment on the multi-headed attention blocks,
MultiHead K, V)
head, W
where head, Attention(Qw. KW. VW.)
13

---

Hide & Seek:
Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
[K²
where Attention(Q)) K, V) = softmax
V. From this we can ascertain the following symmetries: vdz
1) (W.. w/k) (W w YG e GL (chead)
2) (WY, WT) -
YG E GL (cheaa)
where W are the columns of Wo that are relevant for the matrix multiplication with each WVY taking into consideration the head concatenation procedure.
In the full transformer model when solving for geodesic sharpness, for each layer, apply Eq. 5 to each (W W and (W.Y, WP), using Eq. 11. This results in horizontal vectors
and (EY .."). For the non-attention parameters, w, (belonging fully connected layers, convolutional layers and layer norm), keep to the recipe of adaptive sharpness, so that || (Ew ? w ")
112- The norm of the full update vector, & concat(S, S5 &. -.. (.)..w). where over all parameters of the network is implicit, is || Σ (IIC? & ((C)","))) + 11
C.2. Algorithm
Following the lead of Andriushchenko al. (2023), we Auto-PIGG, proposed in Croce & Hein (2020), but now optimizing the horizontal vector instead of the input. In Algorithm 1, é is the loss over the batch are optimizing over, S is the feasible set of horizontal vectors, 5, with norm smaller than and Ps is the projection onto this set. I are the Christoffel symbols for the parameters. y) and W are fixed hyperparameters, which we keep as in Andriushchenko et al. (2023), and the two conditions in Line 20 can be found in Croce & Hein (2020). The only differences to the algorithm employed to calculate adaptive sharpness are in lines 3, 8, 10, and 12. For the metric (,, Juix the only differences are the Christoffel symbols and in the Riemannian gradient (V + Vel (H (H²H)* }
Algorithm Auto-PGD
Input: objective function f, perturbation set S, g(o), initial weights w()) 17 Nuter W {wos .... wn
2: Output: &mux. Emax
3: ((1) &- ()( [D) (T (())
b Perturb weights according to Eq. 5
4: {(1) Ps (E(0) + [z((((((1)
5: lmx &- (ax {(D(1))
6: Smax f- ((0) if max ((w (0) else Emax + g(1)
7: for
Nuer do
8: (k+11 +- (1)) (())_ g())))
Perturb weights according to Eq. 5
9: if (0) an attention weight then
10:
- V Te((v((+1) (0) (0)
Make attention gradients Riemannian
else
12:
g Vel(b(k+1)) (0)2
Make the other gradients Riemannian
13:
end
if
14:
((++1) + Ps (5()) + n1))
15:
E(K+1) Ps (R) + a(z((+1) ( =(k)) (1 a)(((H) -(--1)))
if ((((++))))))) > l max then
Emux &- ((k-1) and lynx {(v((+1))
18:
end if
19:
if K W then
20:
if Condition 1 or Condition 2 then
+ n/2 and (t-1) +- Wmxx
21:
22:
end
23:
end
end for
14

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
C.3. Complexity
Geodesic sharpness is slightly more expensive than adaptive sharpness in the following sense: Our approach consists of three steps: 1) perturbing the weights according to Eq. 5, 2) optimizing the perturbations with gradient descent, and 3) projecting them onto the feasible set, i.c. horizontal vectors within the geodesic ball with small enough norm.
Steps 1) and 2) are also present in adaptive sharpness. Step 1) in our approach is slightly more expensive because we need to evaluate the quadratic form that involves the Christoffel symbols (given by Eq. 11 and Eq. 12); this step introduces plprrmms weight matrix multiplications, but these are quite efficient. Making the gradients Riemannian, costs another Rparras weight matrix multiplications. Neither these bottleneck our approach. For ( Step 3) requires solving Sylvester equation project the direction of the updated geodesic back onto the horizontal space. This solve is cubic in h (Kirrinnis, 2001), but is usually small (e.g. 64 in the ImageNet and BERT experiments). For (r,,)mmx, only efficient matrix multiplications required.
On practical transformers, we expect the bottleneck to be the forward and backward propagations, just like in adaptive sharpness.
D. Geodesic sharpness: Scalar Toy model
To make our approach explicit, we illustrate it on NN with two scalar parameters and H, square loss, and a single (scalar) training point (z; y). We use ( throughout. For this example, everything analytically tractable. We also contrast our sharpness measure with previously proposed ones to highlight its invariance.
Since require full column-ranke
parameter space is M R. x R. with R, IR \ (0).
Metric:
At point HG E M, for two tangent vectors 17 (ng, MH )
(va, (HG T(a M, we have
(n. v.
nGVG
THVH
6
v
(18)
G
2
H
We denote the inverse metric by ghd (G² H²
Horizontal space: H(G.H) = {(na, nH) E Tia.
Rig G
9g H
Geodesics:
To compute the geodesics on the quotient space, we need the Christoffel symbols Tim-
Using coordinate system (p², .²) = (G, H), we have the following equation for geodesic 7(1) = (1 (t),, with initial conditions 7()) (Go, Ho) and +(0)= (1G., mH.)
PoG d12
d7G 2
11
= 0
dt
and similarly for H with 13 instead of r]
The Christoffel symbols can be found using the metric, and its inverse. Using the Einstein notation and denoting the inverse of g by the use of upper indices:
( (ggmk Oxr
0gml Dx
Ogkl mx
1 k1 50
Then
(8gml
Ogml Op
Ogkt 0pm
rl
= 29"
Op
=
G
1
T2 22
H
15

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
All other Christoffel symbols are 0. Our geodesic equations then become (we omit the derivation for H, which identical but with G ++ H)
d'aa
1
d
2
di2
YG
This ODE has the (unique) solution (t) = Ag exp(bat). Taking into account the initial conditions, Ag = =G: Aµ Ho and due the definition of the horizontal space, by
no and би Ge
nu Ho"
this becomes
((t) Go exp
MG t Go
Ho exp
MH
Ho
One important detail to note is that these geodesics are not complete, that is, not all two points can be connected by geodesic. Points with different signs cannot be connected, which makes sense since we excluded the origin from the acceptable parameters and in ID we need to cross it to connect points with differing signs. All points that lie in the same quadrant as (Go, Ho) can be connected through geodesic.
Putting all together
max 2² (exlleexp(()) -) 2yxGaHH(lexp(2)) 11] -1)
Laa ((Go, Ho))
(19)
Letting V/ Go Hor, this becomes:
Shax ((Go; Ho))
max (exp(44) 1) 2yyo (eyo(exp(2b) -1) [lbilS
(20)
Since ли is completely determined by nG can ignore the maximization over it.
Since in practice we'll take p < 1, we Taylor expand to get
S 4ppyoll2 yol
This presents an issue when the residual, ly yo ) is zero, so we can also expand to second order, to get, when Jy yol 2 0
Shn x lolllly 2yol 2p²y0 yo
This is, up to constants, just ||G O H| This is also invariant to GL, transformations, as expected.
Very close the minimum we only capture (second-order in p) properties of the network, bit further away from it we capture (first-order in p) mix of data and network properties.
Comparison with more traditional measures:
The local average and worst case Euclidean sharpness (at a minimum) are
San = Tr LLL = G + H²
Smax Amax LL = G2 H²
Adaptive sharpness is defined as
S (w, c) = ESnPn
[Ls( + - Ls(w)), ~~ ~~(00, (diag(())
S (w, c) EsnPm
max |800 |600 Sp
Ls(w + 5) - Ls(w)
By picking very carefully one can get
Sh (w, = |GH|
Sp S (w,c) |GH]
By contrast, in our approach there is no need for careful hyperparameter choices
16

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Geodesic flatness with more data points:
How does the geodesic flatness look like with more data points?
Ls(G, H) = Σ (GHz - ye)²
which leads to (defining uf GHzz):
b 2v2p) )
-1) - -200? (oxp( |6| v2p -1)]
S = max Σ
(0/)²
exp
(21)
b i=1
[b]
Taylor expanding (in p) once more, we see that
S
R max
1
2v2p
Vi u.(u! - -) 22)
(22)
b
n 1=1
Which maximizes Eq. 22, depends on the sign of Σm= (v: - w)] b < if the sum negative, the reverse the opposite is true.
D.1. Traditional flatness
In Figure 8 we extend Figure to include the trace of the Hessian, both Euclidean and Riemannian. The trace of the network Hessian is quantity that can be used to quantify flatness. We plot, for the scalar toy model, the level sets of: a) the loss function; b) the Euclidean and Riemannian gradient; c) the traces of the Euclidean and Riemannian network Hessian. Several features of the plots are important note: a) the Riemannian version of the gradient and Hessian have the same level set geometry the loss function; b) both the Riemannian gradient norm and the trace of the Riemannian Hessian have smaller values throughout than their Euclidean equivalents; c) the trace of the Riemannian Hessian actually reaches O when the local minimum, whereas the Euclidean Hessian actually attains its highest value there; d) the Euclidean trace of the Hessian cannot distinguish between a minimum and a maximum whereas the Riemannian trace can actually do so. Even for simple flatness measures, correcting for the quotient geometry can provide a much clearer picture.
E. Geodesic sharpness: Diagonal networks in full generality
E.1. Metric (6)
Metric: At point (u,v) v) M, for two tangent vectors 77
(Nu 7w), (vu T(u.u),, we have
nu
m) inv Σ
(23)
(ui)2
(2i)²
Horizontal space: H(n,)) = (1/, (o) Timnj
n
Vi E (1....,d}}
Geodesics: We define b' n4 u
Tiyi E "l.....d.... so that
r(t) (u(t), v(t)) (uy exp(b,t), oi exp(b,)t) Vi {l.....d}
(24)
where un and vó are the initial positions for our parameters, i.e., the parameters that the network actually learned.
Geodesic sharpness: We assume that in what follows x = Ida, and we denote So vA O Vo. 7r (exp(2b't). t). exp ex)(2B4t)) ,B = (u, @ v.)) = Po O nr = x y. Note that 3. just the optimal least squares
predictor when x Id. With this notation
Smax
Σ [bb]
max
((3)) (066%/77 3% 1)] 2(3 Y B)
(25)
17

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Euclidean
Riemannian
(a) Loss
(b) Gradient norm
(d) Gradient norm
1.00
1.0
0.75
0.8
0.50
0.25
0.00
0.4
(c) Hessian trace
(e) Hessian trace
0.50
0.2
0.75
1.000
-1.0 -0.5
0-0
0.5
1.0
Figure 8: Quantities from the Riemannian quotient manifold respect the loss landscape's symmetry; Euclidean quantities do not. We use a synthetic least squares regression task with two-layer NN x + 0201 æ with scalar parameters 0; E R and input x E R. The NN is re-scale invariant, i.e. has GL(1) symmetry: For any E IR \ {0), the parameters (8) = (a a002 represent the same function. (a) The loss function inherits this symmetry and has hyperbolic level sets. (b,c) The Euclidean gradient norm does not share the loss function's geometry and changes throughout an orbit where the NN function remains constant. (d,e) The Riemannian gradient norm and Hessian trace follow the loss function's symmetry and remain constant throughout an orbit, i.c. they do not suffer from ambiguities for two points in parameter space that represent the same NN function. All quantities were normalized to [O; and we fixed six points in parameter space and computed the level sets running through them to illustrate the geometry.
At a first glance, this expression does not seem to have a simple interpretation, but we Taylor expand to second order B (since p supposed be small):
Smax 2 max 4b ||b|Sp
4b" D3o.9.
(26)
where (B) i -.....dd, r' (B) + (r) We separate the analysis Eq.26 into three cases:
{(B B)). I.....d} d} and Doo.s. diag(3 (.)
case a): * and first order suffices
26 becomes
Smix
max 4b
|bb|S
with solution Smax = 4p||r|]. This is essentially the gradient norm- a useful quantity for understanding generalization (Zhao al., 2022b).
case b): 0 Here we necessarily have consider the second order terms, that Eq.26 becomes
Smax max 46d D |6115P
This has the well known solution of Smax p²> Amax (Dpo.a.)= p² max((2))²). This is just || & which is the square of what would get by using adaptive sharpness, Eq,39, with very carefully chosen hyper-parameter c. This is quantity that is useful when our ground-truth, 3 is dense.
case c): r * O and we need both first and second order terms In this case, 2q.26 needs to be considered in full, and we solve the maximization problem using Lagrange multipliers. The Lagrangian will be
= -4bb's r 4b" De.. + A(b - p²)
The KKT conditions then are
18

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
ac
Bo. b + 2X6 0 -8D
(27)
0b
X(b B p²) =
(28)
A 0
(29)
If the constraint is not active, then À 0 and
1
b.
2 B.
Da,
In practice, unless p is large the constraint will always be active, in which case 27 becomes
4r a... b 2X(B)
(b b - 0²) =
A
this then becomes
2 - AD (0..33)-11
d
(ri)²
(A 1-(33((3) + +r'))² = p²
A 0
E.2. Metric (7)
We follow the same approach as in the previous section. The main difference will be in the form of the geodesies: (l) @ v() = (uo @ vo) @ (1 + 2b(), where b' nh zh neural network as it were single layer, with predictor u @ v. that it then perturbs linearly determine sharpness. For (,, mixx and denoting by D3 diag(B)):
er as in the previous section. This essentially treats the two-layer
Smar max 4 (B) B.) + 6 D3bl
(30)
||l|| Sp
We also have that
mixr
... (mt) (z)))((11t) + +...]
(31)
(n
(nt)
+ (w)))
+
(32)
(w')² (v+)²
+ 2(0)) +... = 12D bl12
(33)
Substituting 2D 8, Eq. 30 becomes
Smaz
max 18 ( - B.) + 5
(34)
with the solution (up to constants)
Smaz = -|1B0 - -Ba112
(35)
19

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
F. Geodesic Sharpness: GL1 symmetry and Adaptive Sharpness
What happens instead of a general GLn symmetry, we factor out a GL1 re-scaling symmetry? That is, we identify, element-wise, g) g') if 3a E R. \ {0} s.t. ã a'' and y a a-'g. y.
This is the symmetry present in diagonal networks, and utilize the metric given by Eq. 6, reproduced below for convenience of the reader:
Nu u (u))
nove (otj²
[Mas no), (Vus vo)]
(36)
Note that this metric is equivalent to the following metrie:
(Ta. mo), (Vu: vn]] [(/a//al ne/lol) (Mu/lu), M// Po/lmDee
(37)
where Sen is the usual Euclidean metric and the division is taken to element-wise. Denoting the concatenation of all tangent vectors by ç. the concatenation of all parameters by w, we have ||]|| = ||///w|ll..
In this situation Eq. 4 becomes (y denotes geodesics as usual)
Sina (w. c)
Eg
max |IE//w/llaSp
(s()g(()) ( l=((
(38)
If we then ignore the corrections induced by the geometry of the metric on the geodesics, i.e,, take 5g(1) = -5((0)+ w+ then get
S (w, c) = ES
max [|E//w 1os
Ls(w + - Ls (w)
(39)
which is exactly the formula for adaptive sharpness.
G. Geodesic Sharpness: Ablations
In this appendix we conduct ablation studies on geodesic sharpness (Equation (4)). There are two main components to recipe that differ from adaptive sharpness: the norm [|č]| b) the weight update formula, which instead of the usual
w = w + & takes into account the curvature induced by the parameter space symmetries w' w + 5 - f c Below we turn off these components one by one and re-compute the resulting sharpness on MNLI using the BERT models described in Section 5.3.2.
Metric (9): In Figure 9 show the results for our ablation studies using metric (9). The norm component is much more impactful than the recond-order weight corrections. Turning off the second-order weight corrections results in small performance drop only.
Metric (10): In Figure we show the results for our ablation studies using metric (10). The norm component still much more impactful than the second-order weight corrections for this metric, but now the second-order weight corrections essential, and without them sharpness loses considerable amount of predictive power.
H. Geodesic Sharpness: Ranks and Relaxation
H.1. Ranks: how natural is Assumption 5.1?
In general, in non- networks there is a tendency towards low-rank representations, which might make Assumption 5.1 seem excessive and counter to realistic situations. However, while the learned WoWT tend to be low-rank, Wq and WK (on which Assumption 5.1 ought to apply) themselves are usually high/full (column) rank (Yu Wu, 2023).
20

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Gendedic sherpnes vs. gensealiation pap emd.35
Geodesic sherphess vs penseaiaation gep em0.22
sharpness gereralicatio gap.
0.100
0.168
0.100
0.307
Bowend 8000
0.30
0.000
0.306
0.165
0.39
0.184
0.00
0.300
0.123
0.10
0352
6.5
70
1.2
7.4
6.8
5.0
6.5 0.6
7.0
3.5
Ozedak,
Figure 9: The generalization gap the MNLI dev matched set (Williams al., 2018) worst-case adaptive sharpness with metric (9) is shown for 35 models from (McCoy et al., 2020). On the left we plot the results when we turn off the corrected norm, and the middle when we turn off the second-order weight corrections. Right are the results with no ablations.
Geoderir shergmess va peneralication p4p-r-A01
geteralication gap.
sharpness
0.160
100
0.307
0.307
0.30
0.306
0.300
of
0.305
0.105
0.305
0.300
0.30
0.323
0.32
5.0
5-2
4.5 5.5
3.1
5.2
3.5
5.2
4.5
3.6 3.5
arprico
Gcodak,,
Figure 10: The generalization gap the MNLI dev matched set (Williams et al., 2018) vs. worst-case adaptive sharpness with metric (10) is shown for 35 models from (McCoy et al., 2020). On the left we plot the results when we turn off the corrected norm, and on the middle when we turn off the second-order weight corrections. On the right are the results with no ablations.
H.2. Relaxation
Due to the definition of metric 9, we need to invert matrices of the type of W WQ When these are not full-rank, numerical stability can suffer. Due to floating-point precision rounding errors, practice W Wq is always invertible, but sometimes the inverted matrices have huge singular values. To combat this, we introduce relaxation parameter, that W3 > W2 + eTh, which dampens the resulting singular values. Although we cannot take it to be exactly zero, as long as it is small enough, numerical stability is improved and the results remain roughly the same. We study the effects of varying this parameter on our results empirically below (Figure 11), using the same setup as Section 5.3.2. The results are not significantly affected by the variation this parameter.
Additional Derivations and Proofs
L.. Proof that Eq. 9 defines valid Riemannian metric
Eq. 9 defines valid metric the total space M if it is smooth, and for each point (G, H) M it defines valid inner product on the tangent space M. That it is smooth is obvious, so we show that (ū.S) =
Tr (IG - (H H) T defines valid inner product:
ii Symmetry (),C) (2,1): omitting the H term it is identical, (7. 2) -T (GG -)-a
Tr ((G (TT ( 7)
21

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
Geodesic sharpness generalization gap.r=0.24
Geodesic sharpness generalization gap.r=0.25
0.108
0.108
0.107
0.107
Genere Denerrlization gap
Generaiitalion
0.106
0.105
0.105
0.104
0.164
0.102
0.102
7.4
Geodesic sharpness
Geodesic sharpness
Relaxation 10-2
Relaxation = 10
sharpness vs. gener alization gap.r=0.21
Geodesic sharpness vs. genera lization g=p.1=0.28
0.108
0.108
0.107
Generalization gap
Gemerelizationgaa gap
0.106
0.106
0.105
0.104
0.104
0.103
0.102
0.102
Geodesic sharpness
6.6
7.2 Geodesic sharpness
7.0
7.6
7.0
Relaxation 10
Relaxation
Figure 11:
The generalization gap on the MNLI dev matched set (Williams al., 2018) vs. worst-case adaptive sharpness (left) and geodesic sharpness ((r,))")), is shown for 35 models from (McCoy et al., 2020). Only the relaxation parameter differs between plots. The results stay broadly the same.
(ii) Bilinearity (an + (Č, a(n, X) + D(C,A) (7, an + bc): follows by linearity of the trace;
(ii) Positive Definiteness (17, n7) 2 using assumption 5.1, GT G is invertible and sesitive-definite this means that (G G)- is also dositive-definite and (ū, 1) 2 0, with equality only when 7j
The proof that Equation (10) defines a valid metric is analogous.
1.2. Derivation of the geodesic corrections for attention
We apply the Euler-Lagrange formalism the energy functional to derive the geodesic equation the attention quotient
manifold, and hence D) Ek remembering that geodesics, in local coordinates, obey the equation
d²a²
+
ri ds d=
de di 0.
dt
Starting from
22

---

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
E(7) = J: Ltt dt==
J// (1((),
(40)
L/¹
(tt))tt (x((t)))))))))))) dt
(41)
The Euler-I.agrange equation, for the G part only, reads
d ( 0C
0C
0
(42)
di 0G
0G
We have
0C 2G 2G (''') ( G)"
(43)
0G
d 0L
2
(44)
dt 0G
So that Eq. 42 becomes:
G --G (C'G+ (c=G)= =0
(45)
From which we read
r 2 MSGSG
(C0 (5A)
(46)
The same reasoning is used to deduce Eq. 12.
L.3. Metrics related by scaling and constants
If is metric and Ixnled = Cg + D, then from Eq. 40 and Eq. 42 we see that the geodesics induced by Isaated are the same as those induced by The geodesic sharpness induced by Iscaled is
S (w) = EgND
max || Psestes
Ls (5(T)xccated(1)) Ls ()
EBnD
max DEp
Ls((g(1)) Lss
Ls(5g(1)) - Ls( [1Ell-SP
= BgsD
max
So they are the same up to some re-definition of the hyperparameter p.
23


Reviews:
[{"review_id": "L0wDcR4CaX", "reviewer": "Reviewer_jPWt", "summary": "This paper investigates the connection between sharpness and generalization for models with self attention layers by properly accounting for symmetries present in the models. The authors consider the quotient manifold of parameters and measure sharpness within a geodesic ball on the quotient manifold. The paper claims to introduce the application of Riemannian geometry to deep network parameter symmetry, the notion of geodesic sharpness, and solve for geodesic sharpness in diagonal networks and measure it empirically in transformers.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and helpful suggestions for improvement.\nWe will make sure to mention the reference [1] that you brought to our attention, and we will contrast it to our paper which goes beyond the re-scaling symmetry. \nPlease find our answers to your remaining concerns below.\n\n**Q1**\n\n> Can you reconcile your empirical and theoretical findings [...]\n\nThank you for pointing this out! There is indeed some context missing for it to fully make sense, but these findings are not contradictory.\n\n**TL;DR: The theoretical derivation (Equation (13)) assumes the underparameterized regime (less parameters than data), while the empirical results (Figure (3)) are for the practically more relevant overparameterized regime (more parameters than data), where deriving a closed-form is intractable.**\n\nHere are the details:\n\n- **Theoretical assumptions break down.**\n  The theoretical expression (Equation (13)) assumes that $X^\\top X = I_{d\\times d}$ with data matrix $X \\in \\mathbb{R}^{n \\times d}$ (number of data $n$ and parameters $d$).\n  For this to be feasible, $n\\geq d$, i.e. we are in the underparameterized regime.\n  This assumption allows us to derive closed-form expressions for the unique optimal predictor and its sharpness (see, e.g. [2]).\n  Unfortunately, analyzing the practically more relevant overparameterized regime exactly is intractable.\n\n- **We can reconcile empirical results with the theoretical prediction.**\n  If we re-run the experiment from Figure (3) (overparameterized regime) in the **underparameterized regime**, we indeed obtain the positive correlation between sharpness and generalization, **as predicted by the theory in Equation (13)**.\n  We believe it is practically less interesting because large models typically operate in the overparameterized regime.\n  \n- **Our presented empirical findings are consistent with other works.**\n  The findings we report for overparameterized diagonal networks in the original submission agree with findings from other works.\n  The anti-correlation of worst-case sharpness with generalization was also found in [3].\n---\n\n**Q2**\n> What is the relationship between sharpness and generalization you can report?\n\nThank you for this important question!\nWhat we can report is that contrary to [3], once we account for symmetry, **there is a relationship between sharpness and generalization, and the Kendall-tau significantly differs from zero**. This is what we show in our experiments and we will make sure to make this more explicit in the text.\nWe believe that answering how the different correlation signs come into being is beyond the scope of our paper.\nThe full story is more complicated and rightfully deserves future investigation:\n\nBased on very recent insights from [4] which reports that sharpness minimization differs significantly between vision and language tasks, we hypothesize that the data distribution could play a significant role.\nAlso, Reviewer hcv3 pointed out one interesting avenue to understanding this change in correlation sign via stability of the labels that we hope to follow up on in the very near future.\n\n**Other strengths and weaknesses**\n\n>While the paper is easy to follow, it is unclear what the implications of its findings are [...]?\n\nOur findings provide a foundation for future practical applications. Understanding the sign of the correlation between sharpness and generalization is the limiting factor, but once that is done, we expect geodesic sharpness to be useful for regularizing training.\n\n> What is the time complexity [...]?\n\nThis is a good question!\n- **Computational cost:**\n   Geodesic sharpness does not significantly differ in time complexity from adaptive sharpness.\n  **In our language model experiments, adaptive sharpness takes 1.5s per step, while our $S_{\\text{inv}}$ takes 3s, and our $S_{\\text{mix}}$ takes around 2s**.\n\n- **Why we do not consider the Riemannian Hessian:** \nMainly for reasons of computational efficiency. We deal with models with upwards of $100 M$ parameters, and even accessing quantities such as the Hessian trace through multiple Hessian-vector products is computationally expensive (this was already the case in [3]).\n\n---\n\n**References:**\n\n[1] Rangamani, A., Nguyen, N. H., Kumar, A., Phan, D., Chin, S. P., & Tran, T. D. (2021, June). A scale invariant measure of flatness for deep network minima. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1680-1684). IEEE.\n\n[2] Roger Grosse (2022). \"A Toy Model: Linear Regression\". University of Toronto, Topics in Machine Learning:  Neural Net Training Dynamics.\n\n[3] Andriushchenko, M., Croce, F., Müller, M., Hein, M., and Flammarion, N. A modern look at the relationship between sharpness and generalization. 2023.\n\n[4] Sidak Pal Singh, Hossein Mobahi, Atish Agarwala, and Yann Dauphin. Avoiding spurious sharpness minimization broadens applicability of SAM. arXiv preprint arXiv:2502.02407, 2025.", "claims_and_evidence": "The theoretical and empirical results for diagonal networks seem contradictory. The analytical derivation suggests that when the estimated predictor is close to the optimal predictor, the sharpness should be small (Since $S \\propto \\|\\| \\beta_0 - \\beta_* \\|\\|_2$). The empirical results show instead that larger sharpness is correlated with smaller test error, which means the estimated predictor is not close to the optimal predictor. Something does not seem right here.\n\nThe empirical results in image transformers and language models seem contradictory. Sharpness is correlated with better performance for image transformers and anti-correlated for LMs. Is this actually a meaningful correlate of generalization?", "methods_and_evaluation_criteria": "The authors discuss LoRA adapters but do not conduct experiments for this scenario. The geodesic sharpness measure proposed by the authors could also be adapted to Residual networks that contain GL(h) symmetries within a residual block. Including these experiments would make the paper stronger.", "theoretical_claims": "Skimmed through the Diagonal networks sharpness derivation, seems correct to me though I did not check details.", "experimental_designs_or_analyses": "No apparent issues.", "relation_to_broader_scientific_literature": "This paper proposes a quotient manifold and metric for GL(h) symmetries in deep network models that go beyond rescaling symmetries. This claim of theirs is accurate.", "essential_references_not_discussed": "The authors claim that they are the first to apply Riemannian quotient manifolds to study deep network parameter symmetry. This was already done in a prior paper [1] that proposes a quotient manifold (along with a metric) for rescaling symmetries in MLPs/CNNs. The authors do not cite or discuss the relationship of their paper to this one.\n\n[1] Rangamani, A., Nguyen, N. H., Kumar, A., Phan, D., Chin, S. P., & Tran, T. D. (2021, June). A scale invariant measure of flatness for deep network minima. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1680-1684). IEEE.", "other_strengths_and_weaknesses": "While the paper is easy to follow, it is unclear what the implications of its findings are. Are we able to leverage geodesic sharpness during optimization to find better minima? Can we prove tighter generalization bounds?\n\nWhat is the time complexity of finding the geodesic sharpness? Why do they choose the algorithm in appendix C instead of a Riemannian Hessian based measure? How do the two compare?", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Can you reconcile your empirical and theoretical findings for the case of diagonal networks? I am confused how larger sharpness means better test loss\n\n2. Please consider citing [1] since it already introduces a Riemannian quotient manifold of NN parameters albeit for rescaling symmetries.\n\n3. What is the relationship between sharpness and generalization you can report? Your results seem to contradict the conventional wisdom that flat minima generalize better.\n\n\n[1] Rangamani, A., Nguyen, N. H., Kumar, A., Phan, D., Chin, S. P., & Tran, T. D. (2021, June). A scale invariant measure of flatness for deep network minima. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1680-1684). IEEE.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oLie0CNkps", "reviewer": "Reviewer_4oTA", "summary": "The paper introduces geodesic sharpness, a novel adaptive sharpness measure defined on a quotient manifold that factors out the rich symmetries in transformer parameter spaces (notably the high-dimensional GL(h) symmetry in attention). By leveraging Riemannian geometry, the authors redefine perturbation norms and paths (using geodesics) so that the measure is invariant to symmetry-induced redundancies. They show that when geodesic sharpness is approximated beyond first order, it recovers strong correlations with generalization, in contrast to traditional adaptive sharpness measures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and their suggestions for improving the paper. We'll endeavour to include as much as possible in any final version.\n\n**Theoretical Claims**\n\n> While the derivations are largely convincing, the reliance on approximations and assumptions about the quotient manifold’s structure in complex networks are points that could benefit from further clarification.\n\nThanks for pointing this out! We'll include a more detailed discussion on this in future versions.\n\n**Other Comments Or Suggestions**\n\n> A more detailed ablation study on the effect of metric choice (invariant vs. mixed) could help clarify when one might be preferred over the other.\n\nThis is indeed a philosophically interesting question.\nFrom a theoretical perspective, we do not have any reason to prefer one metric over another as long as both of them correctly reflect symmetries.\n\nIn practice, the metrics perform very similarly, with the mixed metric tending to perform slightly better and being faster to run. The mixed metric's numerics are also advantageous as we do not need to solve a Sylvester equation to project into the horizontal space.\n\n>Expanding the discussion on the conditions under which geodesic sharpness may flip its correlation sign would benefit practitioners.\n\nThank you for this important question! \nThis is something we're keen on investigating in future work, as it is one of the main factors limiting our approach's utility during training.\nWe believe this will require significant research and taking into account more aspects, e.g. the data distribution (e.g. hypothesized by Reviewer hcv3), which are currently not considered in our framework that purely focuses on parameter space symmetries.\n\n---\nWe hope we have addressed all of your concerns and look forward to discussing any outstanding concerns in the discussion period.", "claims_and_evidence": "The main claims are that (a) existing sharpness measures fail in transformers due to unaddressed parameter symmetries, and (b) by reinterpreting sharpness on the quotient manifold, one can obtain a measure (geodesic sharpness) that correlates strongly with generalization. The evidence includes rigorous derivations for simple diagonal networks and empirical evaluations on vision transformers (fine-tuned CLIP) and language models (BERT fine-tuned on MNLI) where Kendall’s tau correlations are consistently stronger for geodesic sharpness. Overall, the claims are well supported, though the variability in the sign of the correlation across tasks suggests further investigation is warranted.", "methods_and_evaluation_criteria": "The method reformulates the sharpness measure by defining the quotient manifold induced by network symmetries, lifting Euclidean objects to their Riemannian counterparts, and approximating geodesic paths to measure worst-case loss variation within a geodesic ball.\nEvaluation is based on the correlation between the sharpness measure and generalization gap across controlled toy experiments and real-world transformer settings. The approach is conceptually sound, though practical geodesic approximations may require careful tuning.", "theoretical_claims": "While the derivations are largely convincing, the reliance on approximations and assumptions about the quotient manifold’s structure in complex networks are points that could benefit from further clarification.", "experimental_designs_or_analyses": "The experimental setup is well thought out.", "relation_to_broader_scientific_literature": "The work extends the literature on generalization by connecting sharpness measures with the geometry of parameter space. By addressing higher-dimensional symmetries inherent in transformer architectures, it bridges a gap between geometric approaches in optimization and practical issues in modern deep learning.", "essential_references_not_discussed": "There are no missing references to my knowledge.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "A more detailed ablation study on the effect of metric choice (invariant vs. mixed) could help clarify when one might be preferred over the other.\nExpanding the discussion on the conditions under which geodesic sharpness may flip its correlation sign would benefit practitioners.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KP7E45Pkc", "reviewer": "Reviewer_hcv3", "summary": "The paper proposes to define the sharpness of the loss curve of neural networks via Riemannian geometry in order to account for symmetries in network parameters. While some reparameterization-invariant sharpness measures exist, they do not account for all possible symmetries in parameters, in particular not for the symmetries of attention layers in transformers. The paper provides an instantiation of the proposed geodesic sharpness for this type of symmetry in transformer architectures. The paper finds that geodesic sharpness correlates stronger with generalization than the previously proposed adaptive sharpness measure.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for their thoughtful review and their really interesting suggestions.\nWe had not fully considered the possible connections with relative flatness, but find these to potentially be a really fruitful avenue of research.\n\n**Claims And Evidence:**\n>Although the reparameterization-invariance follows from the derivation of geodesic sharpness, it would have been nice to empirically verify this, as well.\n\nThis is a good suggestion, and we will add this to any final versions in an appendix (similar to Figure 1 in Andriushchenko et al.).\n\n**Experimental Designs Or Analyses:**\n> The result that sharpness is negatively correlated with generalization for vision transformers is curious. [...]\n\nThank you for this great observation! This could quite possibly help explain this curious phenomenon.\nOne possible way to test for this (at least on synthetic datasets) would be to use a similar experimental setup to that used in [1], modified for classification with diagonal networks, where we can artificially control local label constancy through class separation.\nMore broadly, and with an eye to possible further experiments, if the reviewer happens to be aware of any possibly useful approximations to the local constancy of labels, we would welcome any such suggestions.\nIn principle, we need access to the data-generating process details to estimate the labels' local constancy, something we don't have for ImageNet.\n\nWe also suspect that looking into the data distribution is the most promising future direction for understanding the sign of the correlation flip.\nFor instance, in our synthetic regression task, we observe differing behaviours for the correlation between sharpness and generalization in the under or overparametrized regimes.\nThe data itself can introduce additional symmetries in the overparametrized regime since in this regime $n<d$, where $n$ is the number of data points and $d$ is the dimension, and so the data matrix $X$, which is $n \\times d$,  always has a non-trivial null space by the rank-nullity theorem.\nThis implies that two predictors $\\beta = u \\odot v, \\beta' = u' \\odot v'$ should be equivalent if there is $z \\in Null(X)$ s.t. $\\beta=\\beta'+z$.\nIn the underparametrized regime, where $n>d$, this is no longer necessarily the case, and we expect the null-space to be trivial, thus making these symmetries disappear.\nWe are unsure at this moment of what concrete impact these additional symmetries have but we intend to investigate them further.\nMore broadly, data-dependent symmetries are already known in the literature (e.g. [2]), but remain, in our opinion, underexplored.\n\n\n**Relation To Broader Scientific Literature:**\n> That is, regularizing with relative flatness improves generalization also for transformers [2] and its behaviour wrt. adversarial examples is similar-ish for CNNs and transformers [3]. Since computing it for the penultimate layer and the CE loss is very efficient [3], it would be interesting to discuss geodesic sharpness wrt. relative sharpness.\n\nThank you for the suggestion. We agree it would be interesting to add this to the paper, and we'll endeavour to do so in any final version.\n\n---\n**Questions** \n> Q1: Have you considered measuring average geodesic sharpness, as well? \n\nThis is a good question! We have considered it, especially since [3] reports that, at least for diagonal networks, average sharpness can have different correlation signs with generalization (positively correlated vs anti-correlated and vice-versa).\nWe did not present results on it in our paper due to its mathematical complexity: we would need to find a computationally feasible algorithm to properly integrate over the high-dimensional geodesic ball.\nIt was unclear to us whether this was entirely possible, although a Monte Carlo approach might yield results that are \"good enough\".\n\n**double-check**\n\n>Q2: Are the symmetries in [4] only instances of scaling and re-scaling?\n\nExactly!\n\n---\nWe hope we have addressed all of your questions and look forward to any further questions and insights that might arise during the discussion.\n\n[1] Petzka, Henning, et al. \"Relative flatness and generalization.\" Advances in neural information processing systems 34 (2021): 18420-18432.\n\n[2] Zhao, Bo, et al. \"Symmetries, flat minima, and the conserved quantities of gradient flow.\" International Conference on Learning Representations 2023.\n\n[3] Andriushchenko, M., Croce, F., Müller, M., Hein, M., and Flammarion, N. A modern look at the relationship between sharpness and generalization. 2023.", "claims_and_evidence": "The paper's derivations of geodesic sharpness are sound. The experiments show a superior correlation of geodesic sharpness with generalization, compared to adaptive sharpness. Although the reparameterization-invariance follows from the derivation of geodesic sharpness, it would have been nice to empirically verify this, as well.", "methods_and_evaluation_criteria": "The evaluation criteria are sound, geodesic sharpness is compared to adaptive sharpness in terms of its correlation with generalization for diagonal networks and transformers. The empirical evaluation could be a bit more comprehensive in terms of architectures, benchmark datasets and baseline generalization measures, but since the focus of the paper is on the theoretical contribution, I find the amount of experiments adequate.", "theoretical_claims": "I have checked the theoretical claims and proofs. The proofs are presented clearly, up to my limited understanding of Riemannian geometry. Both claims and proofs are sound.", "experimental_designs_or_analyses": "The experimental design is sound. The result that sharpness is negatively correlated with generalization for vision transformers is curious. A potential explanation might be the requirement of locally constant labels suggested by the analysis in [1]: their analysis suggests that sharpness only correlates with generalization if labels in representation space can be assumed to be locally constant, i.e., small perturbations of the representation do not change the true label ($P(y|x)\\approx P(y|x+\\xi)$). If for the vision transformer small perturbations in the representation should lead to strong changes in the true label, then one would expect a negative correlation between sharpness and generalization. Of course, this argument holds for relative sharpness, and thus might not be true for geodesic sharpness.\n\n[1] Petzka, Henning, et al. \"Relative flatness and generalization.\" Advances in neural information processing systems 34 (2021): 18420-18432.", "relation_to_broader_scientific_literature": "While Maksym Andriushchenko and his co-authors have shown that many sharpness measures do not correlate well with generalization (in particular the SAM-based ones), relative flatness [1] appears to work better, also with transformers, although no direct comparison has been made - to the best of my knowledge. That is, regularizing with relative flatness improves generalization also for transformers [2] and its behavior wrt. adversarial examples is similar-ish for CNNs and transformers [3]. Since computing it for the penultimate layer and the CE loss is very efficient [3], it would be interesting to discuss geodesic sharpness wrt. relative sharpness.\n\n[2] Adilova, Linara, et al. \"FAM: Relative Flatness Aware Minimization.\" Topological, Algebraic and Geometric Learning Workshops 2023. PMLR, 2023.\n[3] Walter, Nils Philipp, et al. \"The uncanny valley: Exploring adversarial robustness from a flatness perspective.\" arXiv preprint arXiv:2405.16918 (2024).", "essential_references_not_discussed": "The paper discusses essential references, to the best of my knowledge.", "other_strengths_and_weaknesses": "The paper tackles an important issue in research on the relationship between sharpness of the loss surface and generalization, namely that of symmetries in parameter space. The contributions are sound and original. While more empirical evaluation is required to show the practical significance of the proposed geodesic sharpness, its significance from a theoretical side is solid.", "other_comments_or_suggestions": "I have no further comments.\n\n######### After Rebuttal ############\n\nI maintain my positive assessment and recommend acceptance.", "questions_for_authors": "Q1: Have you considered measuring average geodesic sharpness, as well?\nQ2: Are the symmetries in [4] only instances of scaling and re-scaling?\n\n[4] Petzka, Henning, Martin Trimmel, and Cristian Sminchisescu. \"Notes on the symmetries of 2-layer relu-networks.\" Proceedings of the northern lights deep learning workshop. Vol. 1. 2020.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FAVpMUofc", "reviewer": "Reviewer_EGqd", "summary": "The paper demonstrates that sharpness metrics on transformers are not a reliable proxy for generalization due to the symmetry properties of the attention mechanism. The author proposes using a Riemannian space, specifically a quotient manifold derived from the symmetry group. Within this space, they introduce a geometric sharpness metric and show that, for diagonal networks, an analytical solution exists. Experimental results on diagonal networks and Vision Transformers (ViTs) reveal a strong correlation between generalization and the sharpness metric. However, the correlation is less significant for language models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for their comprehensive review and for their helpful suggestions. \n\n**Other Comments Or Suggestions:**\n> The Table 1 should be in the paper\n\nThank you for pointing this out. We will move it to the main body of the paper.\n\n---\n\n**Questions**\n\n> Q1: What is the complexity and the computation cost of estimating geodesic sharpness? How does it compare to classical and adaptive sharpness? \n\nThis is a good question!\nWe will add the following to the paper:\n\nGeodesic sharpness does not significantly differ in time complexity from adaptive sharpness (or classical sharpness with a nearly identical time complexity).\nIn our language model experiments, **adaptive sharpness takes 1.5s per step, while our $S_{\\text{inv}}$ takes 3s, and our $S_{\\text{mix}}$ around 2s per step**.\nFor $S_\\text{inv}$, the main additional overhead is inverting a $d_{\\text{head}} \\times d_{\\text{head}}$ matrix and performing a Sylvester solve in SciPy on CPU, as there is no such solve in PyTorch.\n\n>Q2: How sensitive is the approach to different architectures? Only one architecture is considered for each experimental task, which could introduce bias in the results.\n\nThis is indeed a limitation of the experimental setup we inherit from [1].\nTo facilitate the comparison with [1], we decided to focus on the same architectures that were present there, but this could have a hidden bias.\nThe reason for which this is done in [1] is to always compare models within the same loss surface (albeit at different points).\nSince the existence of correlation with generalization exists for all tasks we study, we suspect that this is not architecture-dependent, but are conducting further experiments with a broader set of ViT models to determine whether this indeed is the case. \n\n---\n\nWe hope we have addressed all of your questions and look forward to any further questions and insights that might come up during the discussion period.\n\n[1] Andriushchenko, M., Croce, F., Müller, M., Hein, M., and Flammarion, N. A modern look at the relationship between sharpness and generalization. 2023.", "claims_and_evidence": "The main claim is that classical sharpness is not well-suited for transformers due to specific symmetries in the parameter space. The second claim is that adapting the sharpness calculation to the quotient space, with respect to the symmetry group, improves its correlation with generalization. Both claims are supported theoretically and experimentally. The only reservation is that adaptive sharpness still performs reasonably well for vision models in the experimental results, although the proposed method is significantly better.", "methods_and_evaluation_criteria": "The evaluation follows protocols described in various papers. These methods are applied not only to the limited case of diagonal networks but also to pre-trained ViTs and language models. While deeper experimentation would be beneficial, the proposed method and evaluation criteria are well-founded.", "theoretical_claims": "read the proofs but did not verify their correctness.", "experimental_designs_or_analyses": "I checked the validity of the three experiments. The protocols are straightforward and do not exhibit any issues.", "relation_to_broader_scientific_literature": "The paper is well-situated within the current literature, and the authors demonstrate a strong understanding of the state of the art in the field.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:**  \n- The paper is very well written despite the complexity of the domain.  \n- The proposed approach is elegant and general enough to open important perspectives for generalization estimation using sharpness.  \n\n**Weaknesses:**  \n- The paper frequently refers to the appendix, making it more challenging to read.  \n- The experimental section is convincing but limited in terms of architectures.", "other_comments_or_suggestions": "- The Table 1 should be in the paper", "questions_for_authors": "- What is the complexity and the computation cost of estimating geodesic sharpness? How does it compare to classical and adaptive sharpness? These questions are not sufficiently addressed in the paper and are not explored in the experimental section.  \n\n- How sensitive is the approach to different architectures? Only one architecture is considered for each experimental task, which could introduce bias in the results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper argues that understanding the link between flat minima and generalization is hindered by continuous parameter symmetries inherent in common architectures. To address this ambiguity, the authors propose using Riemannian geometry on the quotient manifold induced by these symmetries. Based on this framework, they introduce geodesic sharpness, a new adaptive sharpness measure where perturbations follow symmetry-respecting geodesic paths using the quotient manifold's metric. This method theoretically accounts for symmetry-induced curvature ignored by prior adaptive measures. Empirically, particularly when applied to the complex GL(h) symmetry within attention mechanisms, geodesic sharpness exhibits significantly stronger correlations with generalization performance on large transformers and language models than previously reported measures.

Reviewers generally acknowledged the importance of addressing parameter symmetries and found the proposed geodesic sharpness approach using Riemannian geometry to be theoretically sound, principled, and elegant. Several reviewers highlighted the potential significance of the work and noted that the empirical results, particularly on vision transformers, showed a stronger correlation between geodesic sharpness and generalization compared to adaptive sharpness. Two reviewers explicitly recommended acceptance, with another leaning towards acceptance.

Despite the positive reception, reviewers raised several concerns. A key point of discussion involved contradictory empirical results: the correlation between sharpness and generalization varied in sign between diagonal networks (theory vs. empirical overparameterized regime) and across tasks (vision vs. language models). Reviewer jPWt initially questioned the novelty claim regarding Riemannian quotient manifolds, pointing out a prior work (Rangamani et al., 2021) focused on rescaling symmetries, which the authors agreed to cite and differentiate from their more general approach. Other questions focused on the computational cost (which authors clarified is comparable to adaptive sharpness), the need for more extensive experiments across different architectures and datasets, the lack of empirical verification for reparameterization-invariance, and the need for further investigation into why the correlation sign flips between tasks. The authors provided detailed rebuttals, clarifying theoretical assumptions versus practical regimes, discussing potential reasons for correlation sign flips (hypothesizing data distribution roles and label stability), comparing their method's generality and efficiency to the cited prior work.

Overall, the assessment leans towards acceptance and in agreement with the reviewers, I believe addressing parameter symmetries is an important problem, and the proposed geodesic sharpness provides an interesting and principled approach. Therefore, I recommend acceptance. Please incorporate the suggested additions and clarifications into the final version.
Decision: Accept (spotlight poster)