A simple connection from loss flatness to compressed neural representations
Authors: Shirui Chen
Stefano Recanatesi
Eric Todd SheaBrown
Venue: Accepted by TMLR ()

Abstract: Despite extensive study, the fundamental significance of sharpness---the trace of the loss Hessian at local minima---remains unclear. While often associated with generalization, recent work reveals inconsistencies in this relationship. We explore an alternative perspective by investigating how sharpness relates to the geometric structure of neural representations in feature space. Specifically, we build from earlier work by Ma and Ying to broadly study compression of representations, defined as the degree to which neural activations concentrate when inputs are locally perturbed. We introduce three quantitative measures: the Local Volumetric Ratio (LVR), which captures volume contraction through the network; the Maximum Local Sensitivity (MLS), which measures maximum output change normalized by the magnitude of input perturbations; and Local Dimensionality, which captures uniformity of compression across directions.

We derive upper bounds showing that LVR and MLS are mathematically constrained by sharpness: flatter minima necessarily limit these compression metrics. These bounds extend to reparametrization-invariant sharpness (measures unchanged under layer rescaling), addressing a key limitation of standard sharpness. We introduce network-wide variants (NMLS, NVR) that account for all layer weights, providing tighter and more stable bounds than prior single-layer analyses. Empirically, we validate these predictions across feedforward, convolutional, and transformer architectures, demonstrating consistent positive correlation between sharpness and compression metrics. Our results suggest that sharpness fundamentally quantifies representation compression rather than generalization directly, offering a resolution to contradictory findings on the sharpness-generalization relationship and establishing a principled mathematical link between parameter-space geometry and feature-space structure. Code is available at \url{https://github.com/chinsengi/sharpness-compression}.

Published in Transactions on Machine Learning Research (01/2026)
A simple connection from loss flatness to compressed neural representations
Shirui Chen
sc2560uw.cd
Deportment of Applied Mathematics Computational Neuroscience Center University of Washington
Stefano Recanatesi
Aesjamo.reannaleii@mmaa com
Technion, Israel Institute of Technology
Eric Shea-Brown
elsb@uw. edu
Department of Applied Mathematics
Computational Neuroscience Center
University of Washington
Reviewed on OpenReview: https: // openreview. net/ forum? id- GgpQbU9bFR
Abstract
Despile extensive study, the fundamental significance of sharpness the trace of the loss Hessian at local minima remains unclear. While often associated with generalization, recent work reveals inconsistencies in this relationship. We explore an alternative perspective by investigating how sharpness relates to the geometric structure of neural representations in feature space. Specifically, we build from earlier work by Ma and Ying to broadly study compression of representations, defined the degree to which neural activations concentrate when inputs are locally perturbed. We introduce three quantitative measures: the Local Volumetric Ratio (LVR), which captures volume contraction through the network; the Maximum Local Sensitivity (MLS), which mcasures maximum output change normalized by the magnitude of input perturbations; and Local Dimensionality, which captures uniformity of compression across directions.
We derive upper bounds showing that LVR and MLS are mathematically constrained by sharpness: flatter minima necessarily limit these compression metrics. These bounds extend to reparametrization-nnvaaaa sharpness (measures unchanged under layer rescaling), addressing key limitation of standard sharpness. We introduce net work variants (NMLS, NVR) that account for all layer weights, providing tighter and more stable bounds than prior single-layer analyses. Empirically, validate these predictions across feedforward, convolutional, and transformer architectures, demonstrating consistent positive correlation between sharpness and compression metrics. Our results suggest that sharpness fundamentally quantifies representation compression rather than generalization directly, offering a resolution lo contradictory findings on the sharpness-gennrralizaiio relationship and establishing principled mathematical link between parameter-space geometry and feature-space structure. www. Code is available at https: http:////....oongogaaaaaaassssse
1 Introduction
There has been long-standing interest in sharpness, a geometric metric in the porameter space that measures the flatness of the loss landscape local minima. Flat minima refer to regions in the loss landscape where the loss function has a relatively large basin, and the loss does not change much in different directions around the minimum. Empirical studies and theoretical analyses have shown that training deep neural

---

Published in Transactions on Machine Learning Research (01/2026)
b)
Training function
Test function
flat
Sharp
flat
Sharp
Loss
Smaller gap
Larger gap
Larger gap?
Figure 1: A schematic illustration of why flatness alone does not reliably predict generalization. (a) Under the common but unjustified assumption that the test loss is a horizontal shift of the training loss, flat minima trivially exhibit a smaller generalization gap than sharp minima. (b) In a hypothetical but possible scenario, the test loss differs from the training loss by an additive perturbation rather than a simple shift. Here the flat minimum can exhibit larger generalization gap than the sharp minimum, demonstrating that curvature alone is insufficient to prediet generalization.
networks using stochastic gradient descent (SGD) with small batch size and a high learning rate often converges to flat and wide minima (Ma & Ying 2021 Blanc et al. 2020b Geiger et al. 2021 Li et al.
2022: Wu et al. 2018 Jastrzebski et al. 2018 Xie et al. 2021 Zhu et al. 2019). Many works conjecture that flat minima lead to a simpler model (shorter description length), and thus are less likely to overfit and
more likely to generalize well (Hochreiter Schmidhuber 1997 Keskar et al.) 2016 Wu al., 2018 Yang et al. 2023). Based on this rationale, sharpness-await minimization (SAM) has been popular method for improving modcl's generalization ability. However, recent work has shown that SAM does not only minimizo
sharpness achieve superior generalization performance (Andriushchenko & Flammarion, 2022 Wen et al. 2023). More confusingly, it remains unclear whether flatness correlates positively with the generalization capacity of the network (Dinh et al. 2017 Andriushchenko et al. 2023 Yang et al. 2021). and even when it does, the correlation is not perfect (Neyshabur et al.] 20177: Jiang et al. 2019). In particular, Dinh et al, (2017) argues that one can construct very sharp networks that generalize well through reparametrization, while Andriushchenko al. (2023) show that even reparamctrization-ineee innaaiaaaaiinnnnnianntt sharpnoss cannot capture the relationship between sharpness and generalization.
Figure D illustrates the crux of this issue. The common intuition for why flat minima should generalize better Hochreiter & Schmidhuber 1997 Keskar et al. 2016) implicitly assumes that the test loss landscape is a horizontally shifted copy of the training loss (Figure Ib): under such a shift, the loss al a flat minimum changes little while the loss at a sharp minimum increases substantially, yielding a smaller generalization gap at the flat minimum. However, there is no principled justification for this horizontal-shift assumption. In a hypothetical setting where the test loss differs from the training loss by a random perturbation (Figure [p), the generalization gap at flat minimum could be larger than sharp minimum, contradicting the standard
2

---

Published in Transactions on Machine Learning Research (01/2026)
narrative. This suggests that sharpness, while clearly capturing meaningful geometric structure, may not be directly measuring generalization ability.
In light of this paradox, we take a different perspective: rather than asking what sharpness tells us about generalization, we ask what sharpness tells us about the representations learned by the network. We show that there exists a more consistent relationship by investigating how sharpness near interpolation solutions the porameter space influences local geometric features of neural representations in the feature space. By building a relationship between sharpness and the local compression of neural representations, we argue that sharpness, in its essence, measures the compression of neural representations. Specifically, we show that sharpness decreases and the minimum flattens, sharpness-rclated quantities upper-bound certain compression measures, meaning that the neural represenlation must also undergo some degree of compression. We also note how local dimensionality is a compression metric of a distinct nature and therefore does not necessarily correlate with sharpness.
More specifically, our work makes the following novel contributions:
1. Refined and new compression metrics with sharpness bounds. We refine and identify feature space quantities that quantify compression and are bounded by sharpness: local volumetric ratio (LVR) and maximum local sensitivity (MI.S). The former is a new metric this context and the latter refines related quantity in carlier work. D We derive explicit upper bounds showing these metrics are constrained by sharpness-relates related quantities. This establishes direct mathematical link between parameter-space geometry (sharpness) and feature-space geometry (compression), applicable to feodforward networks with quadratic loss.
2. Improved bounds via network-wide metrics. We improve the bound on MLS from Ma & Ying (2021) and propose Network MLS (NMLS), which considers sensitivity at all intermediate layers rather than only the input. This yields bounds that consistently predict positive correlation with sharpness across diverse experimental settings, overcoming limitations of single-layer analyses where prefactors can dominate and obscure the sarrpnes--omppeestie relationship.
3. Extension to ieparametrizatiooninnaaaar invariant sharpness (i.e., unchanged under layer rescaling) tightens our bounds, addressing a key criticism of standard sharpness measures and providing novel interpretation: reparamctrization- invariant sharpness quantifies robustness of outputs to internal representations.
sharpness. We prove that using reparametrization-
Empirical validation across architectures. We validate our theoretical predictions through experiments on feedforward (MLP), convolutional (VGG-11, LeNet), and transformer (ViT) architec- tures, demonstrating that LVR and MLS/NMLS consistently correlate with their sharpness-related bounds during training and across pretrained models.
Connection to neural collapse and intermediate representations. We exlend our analysis to penultimate and intermediate layer representations, relating our framework to neural collapse phenomena. Our bounds apply most naturally when the number of classes exceeds feature dimension, relevant for settings like language modeling and large-scale retrieval.
On bound tightness. We emphasize that our theoretical results provide upper bounds rather than equalities. Moreover, these bounds depend not only on sharpness but also on weight norms and input magnitudes. Consequently, for two networks with identical sharpness but different weight scales, the bounds may differ. However, our extensive empirical validation demonstrates that despite this apparent looseness, the bounds are sufficient to predict consistent positive correlation between sharpness and compression metrics across training dynamics and diverse architectures. This empirical validation is essential to our claims and demonstrates the practical utility of our theoretical framework.
With these results, we help reveal the nature of sharpness through the interplay between key properties of trained neural networks in parameter space and feature space.
We collectively term ML., NMLS 'compression metrics', because these quantities measure how compressed//onccetrraae nerse-pe input/Internal neural representations is after going through the network.
3

---

Published in Transactions on Machine Learning Research (01/2026)
Paper organization. Section 2 reviews the pioncering arguments Mf & Ying (2021) that flatter minima can constrain the gradient of network output with respect to network input, and extends the formulation to the multidimensional input case, Section 3 proves that lower sharpness implies a lower upper bound on two metrics of the compression of the representation manifold in feature space: the local volumetric ratio and empirically verifies our theory by calculating various compression metrics, their theoretical bounds, and sharpness for models during training as well as pretrained ones. Finally, Section discusses how these conditions help explain why there are mixed results on the relationship between sharpness and generalization in the literature, by looking through the alternative lens of compressed neural representations.
the maximum local sensitivity (MLS) (Section 3.1 Section 3.2 Section |
Key terminology. Before proceeding, we clarify key terms used throughout this work. Reporametrization refers to equivalent network parameterizations that produce identical input-output mappings; for example, scaling the weights of layer I by factor a and layer 1 + 1 by 1/a leaves the network function with ReLU nonlinearity unchanged as demonstrated in Dinh et al. (2017). Reparametrivationinnaal
sharpness measures remain constant under such transformations, addressing the concern that standard sharpness can be arbitrarily changod without affecting network behavior.
The compression metries we study are: Local Volumetric Rotio (LVR), which quantifies how much small input volume contracts as it propagates through the network; Maximum Local Sensitivity (MLS), which measures the maximum rate of change of network output with respect to impul perturbations; and Network MLS (NMLS), which extends MLS by considering sensitivity at all intermediate layer representations rather than only the input layer.
2 Background and setup
Consider leedforward neural network f with input data x RM and parameters 0. The output of the network is:
y=
(1)
where y E R (N < M). We consider a quadratic loss L(y. Vtrue) = Ily Viruell,, function of the outputs and ground truth Ytrue- In the following, we will simply write I(y), L(J(x, , 0)) or simply L(O) to highlight. the dependence of the loss on the output, the network, or its parameters.
Sharpness measures how much the loss gradient changes when the network parameters are perturbed, and is defined by the sum of the eigenvalues of the Hessian:
S(0) = TT(H),
(2)
with H = V() being the Hessian. The trace of the Hessian, Tr (V²L(0)), is not the only definition of sharpness, but many sharpness minimization methods have been theoretically shown to reduce this quantity in interpolating models. Specifically, assuming that the training loss minimizers lie on a smooth manifold
Coopprr 2018 Fehrman et al., 2020), methods like Sharpness- Aware Minimization (SAM) (Foret et al. 2020
when used with batch size and sufficiently small learning rate and perturbation radius (Wen et al. 2022
Bartlett al. 2023), or Label Noise SGD with a small enough learning rate (Blane et al. [2020a] Damian et al. 2021), tend to favor interpolating solutions with low Hessian trace. Therefore, focus our analysis on the trace of the Hessian.
Scope and loss function considerations. Our theoretical analysis is conducted using the quadratic (MSE) loss, which allows to derive the key relationship in Equation 3 connecting sharpness to network gradients. However, we note important considerations regarding other loss functions. For cross-cntropy (CE) loss, Granziol (2020) showed that Hessian-based flatness measures become unreliable: solutions with large weights and low CE lass can exhibit artificially small Hessian traces, appearing deceptively 'flat' despite poor generalization properties. Moreover, the Hessian rank of CE loss boundod by the number of neurons times the number of classes, causing many cigenvalues to be zero or near-zero regardless of the solution quality. Consequently, networks with L2 regularization (which have smaller weights) can be sharper yet generalize
4

---

Published in Transactions on Machine Learning Research (01/2026)
better than unregularized networks, directly contradicting the datnness-ennerlliaatu hypothesis for CE loss. This fundamenially breaks the connection between CF loss sharpness and generalization.
Despite this limitation of CE loss sharpness, our framework can still provide insights for networks trained with CE loss through two pathways. First, our theoretical results extend to logistic loss with label smoothing because using the logistic loss with label smoothing yields the same set of minimizers and flattest minimizers as corresponding problem using mean squared error (see Lemma A.13 in Wen al. (2023)). Second, and more practically relevant, our compression metrics (LVR, MIS, NMLS) depend on the network function f itself rather than the training loss. Once network is trained (with any loss function), its compression properties by examining how it transforms inputs to outputs. The empirical success of this approach is demonstrated in Figure 5 where we observe meaningful correlations between compression and sharpness (computed assuming MSE loss) for 181 pretrained ViT models that were trained with CE loss. For these networks, we compute sharpness using MSE loss applied to the trained network, which allows us differentiate between solutions even though their original CE training loss was near zero.
can analyze
Following (Ma & Ying 2021 Ratzon et al. 2023), define 0* to be "exact interpolation solution" the zero training loss manifold in the parameter space (the zero loss manifold in what follows), where fx, 0*) = for all i's (with i E {1..n} indexing the training set) and L(Ø") = O. On the zero loss manifold, in particular, we have
- Σ Vol(x;
(3)
where ||-||æ is the Frobenius norm. We state a proof of this equality, which appears in Ma & Ying 2021
and [en et al. (2023). in Appendix C In practice, the parameter will never reach an exact interpolation solution due to the gradient noise of SGD; however, Equation (3) is a good approximation of the sharpness
when the training loss is sufficiently small (see error bounds Lemma C.1
To why minimizing the sharpness of the solution leads to more compressod representations, we need to move from the parameter space to the input space. To do we clarify the proof of Equation (4) in Ma & Ying (2021) that relates adversarial robustness to sharpness in the following. The improvements we made are summarized at the end of this section. Let W be the input weights (the parameters of the first lincar layer) of the network, and 0 be the rest of the parameters. Following Ma & Ying 2021), as the weights W multiply the inputs x, we have the following identities:
/w/ 0)|r= V
F3a =
||]|||| 2 |
(4)
O)) =
where J = @xx0)
is a complex expression computed with backpropagation. From Equation (4h and the
x(Wx)
sub-multiplicative property of the Frobenius norm and the matrix n-norm we have:
W 0|
f < (V/
/w/((x 0)
(5)
We call Equation
5 the linear stability trick. As result, we have
Vx (x, 001
v x (x, 1²
V V fxx, 0')
(6)
min;
X, 7
Vel(x; 0')
min:
x 11
A| 18|2, ||AB|22
< ||||
5

---

Published in Transactions on Machine Learning Research (01/2026)
This reveals the impact of flatness on the input sensitivity when & 2. Equation 6 holds for any positive k. Thus, the effect of input perturbations is upper-bounded by the sharpness of the loss function (cf. Equation
33 Note that Equation (6 corresponds to Equation (4) in Ma & Ying 2021) with multivariable
output.
While the experiments of Ma & Ying (0021
empirically show a high correlation between the left-hand side of Equation 6 and the sharpness, Equation (6) does not explain such a correlation by itself because of the scaling factor / leading to mixed positive and/or negative correlations with sharpness under different experimental settings. In the next section, we will improve this bound to relate sharpness to various metrics measuring robustness and compression of representations. More specifically, compared to Equation (4) of Ma & Ying (021 we make the following improvements:
This factor makes the right-hand side of Equation (6) highly variable,
We replace the reciprocal of the minimum with the quadratic mean achieve a more stable bound (Proposition 3.8 This term remains relevant common practice in deep learning does not normalize the input by ils 2-norm, as this would erase information about the modulus of the input.
2. While Ma & Ying (021) only considers scalar output, we extend the result to consider networks with multivariable output throughout the paper.
3. We introduce new metrics such Network Volumetric Ratio and Network MLS (Definition 3.5 and
Definition 3.9 and their sharpness-related bounds (Proposition 3.6 and Proposition 3.10 w which
have two advantages compared to prior results (cf. the hand side of Equation
(a) our metries consider all linear weights that bounds remain stable to weight changes during training.
(b) they avoid the gap between derivative w.r.t. the first layer weights and the derivative w...t. all weights, i.c. the sccond incquality in Eq. 6, thus tightening the bound.
Moreover, we show that the underlying theory readily extends to networks with residual connections in Appendix
3 From robustness to inputs to compression of representations
Building upon the linear stability bound (Equation (5 established in Section 2 this section derives our main theoretical results connecting sharpness to compression of neural representations.
Key assumptions. Our analysis assumes: the network has reached an approximate interpolation solution 0" where training loss is near zero, allowing us to the sharpness characterization from Equation (3) (ii) the network architecture consists of linear layers (including convolutional layers, which are lincar operations) with nonlinear act activations, though we do not assume specific activation functions for the main bounds (the theory extends to networks with residual/skip connections as detailed in Appendix B and (iii) all bounds are derived using the quadratic (MSE) loss function. While our theoretical derivations require MSE loss, the compression metrics themselves (LVR, MLS, NMLS) depend only on the network function f and can be computed for networks trained with any loss function, as discussed in Section 2 Throughout, work in the local regime where first-order Taylor approximations are valid.
3.1 Sharpness bounds local volumetric transformation in the feature space
To understand how networks compress their representations, we first study how local volumes in input space transform as they pass through the network. a network compresses its inputs, small volumes around input points should contract as they are mapped to feature space. We quantify this via the local volumetric ratio, the ratio between the volume of hypercube of side length h x, H(x), and its image under transformation f, [xH()) 0"). By the change of variables formula from multivariate calculus, the infinitesimal volume

---

Published in Transactions on Machine Learning Research (01/2026)
transformation is characterized by the Jacobian matrix. Consider the mapping f RR R with N M and Jacobian J = VxJ E RNNd The volume scaling factor is:
(x)()(((x) (*),
d [oll = lim (x)(H(x))
h-0
dot((JJT) (product singular values of J)
= det (Vx((())))))))
where the equality holds because for a rectangular matrix J E RRNX with N < M, the N-dimensional volume scaling equals the product of its N singular values oI....,N which equals /de((JJ") since JJ" E RNNN is a square matrix with eigenvalues oN.....oN.. We formalize this as a definition:
The Local Volumetric Ratio at input x of network f with parameters 0 is defined as d ((x,) = det (Vx((Vx))'))
Definition 3.1.
Exploiting the bound on the gradients derived earlier in Equation 5 we derive a similar bound for the volumetric ratio:
Lemma 3.2.
Tr TV N/2
d doll S
(8)
Proof. The inequality follows from the arithnetic-gcometric mean incquality applied to the squared singular values of Vxl. The equality uses the trace identity (Vx") = =I[V//1 See Appendix D for the complete derivation.
Next, introduce measure of the volumetric ratio averaged across input samples.
Definition 3.3. The Local Volumetric Ratio (LVR) of network with parameters 0 is defined as the sample mean of Local Volumetric Ratio at different input somples: dV Ex=1 d sol
We now establish our first main result: an upper bound LVR in terms of sharpness. This bound will show that flatter minima (lower sharpness) necessarily constrain the volumetric ratio, establishing mathematical link between parameter-space flatness and feature-space compression.
Proposition 3.4. The local volumetric ratio is upper bounded by a sharpness related quantity:
N-N/2 n
n
W
1N
"nS((")
N/2
dV I
n Σ [ 0")
0'
n
x;
2N
for all N 1.
The proof of the above inequalities is given in Appendix
The bound in Proposition 3.4 only considers sensitivity at the input layer. However, analysis by examining compression at all intermediate layers, which allows to incorporate all network weights rather than just the first-layer weights. This yields tighter bounds that are more stable during training. We denote the input to the /-th linear layer as x for / - input. Similarly, W, the weight matrix of the L-th linear linear/convan conoolutooll layer. With a slight abuse of notation, we use fi to denote the mapping from the input of the 1-th layer to the final output.
strengthen
1,2, 1,2,.... ,L, where x] = is the network
Definition 3.5. The Network Volumetric Ratio (NVR) is defined as the sum of the local volumetric ralios dV,, for all Jt, that is, dVnet Et=,00 dV,
By applying Equation ( to every intermediate layer, we obtain network-wide bound:

---

Published in Transactions on Machine Learning Research (01/2026)
Proposition 3.6.
The network volumetric ratio is upper bounded by sharpness related quantity:
b N N-N
22 W 2
(nS(6")
dV Σ 1=1
(10)
n Hi
2N
N
Again, a detailed derivation of the above inequalities is given in Appendix D Proposition 3.4 and Proposi- tion 3.6 imply that flatter minima of the loss function in parameter space contribute to local compression of the data's representation manifold.
3.2 Maximum Local Sensitivity as an allied metric to track neural representation geometry
We observe that the equality condition in the first line of Equation rarely holds in practice. To achieve equality, we would need all singular values of the Jacobian matrix Vxf to be identical. However, our experiments in Section f show that the local dimensionality decreases rapidly with training onset; this implies that Vx/" Vxf has a non-uniform eigenspectrum (i.e., some directions being particularly elongated, corresponding a lower overall dimension). Morcover, the volume will decrease rapidly as the smallest. eigenvalue vanishes. Thus, although sharpness upper bounds the volumetric ratio and often correlates reasonably with it (see Appendix H.2 the correlation is far from perfect.
Fortunately, considcring only the maximum cigenvalue instead of the product of all cigenvalues alleviates this discrepancy (recall that det Vvx/"Vx in Definition 3.1 is the product of all eigenvalues). This motivates an alternative compression metric based on the largest singular value.
Definition 3.7. The Maximum Local Sensitivity (MLS) of network f is defined to be MTS, = - (xx)(xx)l22 which is the sample mean of the largest singular value of xxf.
Intuitively, MLS is the largest possible average local change of f(x) when the norm of the perturbation to x is regularized. MLS is also referred to as adversorial robustness or Lipschitz constant of the model function
in Ma & Ying (021)- We now derive analogous bound for ML. that mirrors Proposition 3.4 but avoids the product-of-eigenvalle issue.
Proposition 3.8. The marimum local sensitivity is upper bounded by sharpness-related quantity:
1 Σ
MLS,
V
0') Will
1/0
(11)
The derivation of the above bound is included in Appendix D representations, we empirically show in Appendix H..2 that MLS has a higher correlation with sharpness than the local volumetric ratio. We include more analysis of the tightness of this bound in Appendix and discuss its connection to other works therein.
As alternative measure of compressed
As with volumetric ratio, considering only the first layer can lead to unstable bounds dominated by prefactor variations. We therefore introduce network-wide version that examines sensitivity all intermediate layers, yiclding bounds that more consistently predict positive correlation with sharpness across experimental settings.
Definition 3.9. The Network Marimum Local Sensitivity (NMLS) of network f is defoned as the sum of MLS,, for all I, i.e. EL_ El_, ML-
Recall that x/ is the input the (-th chnaarconnoluutinal layer for sample x; and fi is the mapping from the input of 1-th layer to the final output. Extending the analysis from Proposition 3.8 to all layers yields:
Proposition 3.10. The network marimum local sensitivily is upper bounded by sharpness related quantily:
NMLS 1 ΣΣΣNN
0-)122
1/00)112
(12)
8

---

Published in Transactions on Machine Learning Research (01/2026)
The derivation is in Appendix E The advantage of NMLS is that instead of only considering the robustness of the final output w.r.t. the input, NMLS considers the robustness of the output w.r.t. all hidden-layer representations. This allows us to derive a bound that not only considers the weights in the first linear layer but also all other linear weights. We observe in Section 4.2 that while MLS could be negatively correlated with the right-hand side of Equation (T7), NMLS has a positive correlation with right-hand side of Equation (12) consistently. We do observe that MLS is still positively correlated with sharpness (Figure H.8). Therefore, the only possible reason for this negative correlation is the factor before sharpness in the MLS bound involving the first-layer weights and the quadratic mean (see Equation
11
Why LVR and MLS quantify compression. Having introduced our main compression metrics, we clarify why these quantities measure compression of neural representations. In our framework, compression refers to the concentration or contraction of representations when inputs are locally perturbed. LVR. quantifies this via volume contraction: if small volume in inpul space is mapped to smaller volume in representation space, the network has compressed that region of its representation manifold. Mathematically, LVR < 1 indicates volume contraction (comprossion), LVR = 1 indicates volume prescrvation, and LVR > 1 indicates volune expansion. Similarly, MLS measures compression through sensitivity: lower MLS means the network output changes less in response to input perturbations, indicating that nearby inputs are mapped to nearby (compressed) outputs. Networks with high compression have both low LVR (contracted volumes) and low MIS (reduced sensitivity).
Importantly, our theoretical bounds do not guarantee that networks will compress (i.e., achieve LVR < 1 or small MLS). The bounds only establish that flatter minima constrain these metries to be smaller. However, our extensive empirical results in Section ! demonstrate that trained networks do indeed exhibit compression: we observe decreasing LVR and MLS values during training, particularly in the late stages when networks find flatter minima. The correlation between sharpness and these compression metrics validates that flatter minima are associated with more compressed representations in practice, even though the bounds themselves are upper bounds rather than guarantees of compression.
On the role of weight norms in the bounds. A critical aspect of our bounds is their dependence on both sharpness and weight-related prefactors (e.g.. ||||//||>|| ratios). This raises a natural question: does lower sharpness always imply more comprossion? Thcorctically, this is not guaranteed. Consider two networks at different interpolating solutions: if one has lower sharpness but substantially larger weight norms, the bound could be looser despite the decrease in sharpness. In principle, a decrease in sharpness might not fully counteract inereases in the prefactor terms. We provide detailed empirical analysis of where tightness is lost in the chain of inequalities comprising our bounds in Appendix H
However, despite this ambiguity in theory, the bounds may still apply usefully in practice. One important setting where this is the case is for fired network architecture during training, when the prefactors remain relatively stable the network evolves toward flatter minima. Then weight norms (and input statistics) do not fluctuate dramatically within training trajectory, so the sharpness term dominates the variation in the bound. This explains the strong positive correlations we observe between sharpness and compression metrics during training (e.g., Figure 2] Figure 3). In contrast, when comparing across different network configurations such as our analysis of 181 pretrained ViT models with varying architectures, datasets, and training procodures-the prefactors can vary substantially. This is preciscly why we employ adaptive sharpness (which normalizes by weight norms) and normalized MLS in Figure 5 these normalized metrics account for the prefactor variations and reveal that the sharpness compression relationship holds even across heterogeneous model families. The empirical robustness of this correlation, both within training trajectories and across diverse architectures when properly normalized, demonstrates that the bounds capture meaningful relationships even despite their theoretical limitations.
3.3 Local dimensionality is tied to, but not bounded by, sharpness
While LVR and MI.S capture compression via volume contraction and maximum sensitivity, they do not capture the uniformity of compression across different directions. A network might compress representations uniformly (reducing all directions proportionally) anisotropically (compressing certain directions more
9

---

Published in Transactions on Machine Learning Research (01/2026)
than others). This distinction is captured by local dimensionality, which measures whether the cigenvalues of the Jacobian are uniform or concentrated in a few directions. As will show, unlike LVR and ML.S, local dimensionality cannot be directly bounded by sharpness alone, as it depends on eigenvalue ratios rather than their magnitudes.
Consider an input data point drawn from the training set: x= x; for a specific i E fl....,,, Let the set of all possible perturbations around in the input space be samples from an isotropic normal distribution, B(x),, (Xx aT), where CB()). aI, with I as the identity matrix, is the covariance matrix. We first propagate x through the network transforming each point into its corresponding image f(x). Following Taylor expansion for points within ((x)a in the limit a + O, we have:
f(x) (xx + V 100 x) + O[Ixx x||²).
(13)
We can express the limit of the covariance matrix C)(B(x)) of the output f(x) as C)(((x))
ct lim a-+
Vxf Ox
(14)
Our covariance expressions caplure the distribution of the samples in B(x), as they go through the net netw f(x, 0*). The local Participation Ratio based on this covariance is given by:
(pR(f(x)) = lim Tr(C)(B(x).))) a-30 Tr To ((C(((()))))
TrCCJim
(15)
|Recanatesi et al. 2022 cf. nonlocal measures in Gao et al. 2017 2016).
Litwin-Kumar al. 2017 Mazzucato et al.
Definition 3.11. The Local Dimensionalityy of network f is defined the sample mean of local participation rulio al diffferent inpul sumples: = Ei= (x:(J(x:))
-
This quantity in some sense represents the sparseness of the eigenvalues of Clim: If we let A be all the eigenvalues of Clim, then the local dimensionality can be written as DPR maximum value when all eigenvalues are equal to each other, and its minimum when all eigenvalues except for the leading one are zero. Note that the quantity retains the same value when À is arbitrarily scaled. As consequence, it is hard to find relationship between the local dimensionality and the fundamental quantity on which our bounds are based: [/Vx/(x, 0' which is ||||
which attains its
3.4 Relation to reparaamttrizaiion-invaaaaa sharpness
Dinh et al. 2017) argues that robust sharpness metric should have the ingarammtrizationnnnvaaitt property, meaning that scaling the neighboring linear layer weights should not change the metric. While the bounds in Proposition 3.8 and Proposition 3.10 are not strictly opprranetizzationnnaaaii
sharpness metrics that are redesigned (Tsuzuku el. al.] 2019) achieve invariance can be proved to lighten our sharpness is proposed in (003); Kwon et al. (2021), and again show that it upper-bounds input-invariant
bounds (see Appendix F.I). Another more aggressive reparametrizationninvaaat
Andriushchenko et al.
MLS in Appendix F.2 We also empirically evaluate the relative flatness (Petzka el al., 2021 which is also
rpparanotriaaiionnnnaaaaall in Appendix
H.2 but no significant correlation is observed. Overall, we provide a novel perspective: reparaaettizationnnnaaaatt sharpness is characterized by the robustness of outputs to
internal neural representations.
3.5 Connection to neural collapse and compressed neural representations
The neural collapse phenomenon (Papyan et al. 2020 Zhu et. al. 2021 indicates that the within-class variance the features in the penultimate layer vanishes at the terminal phase of training; allied studies have also found compressed neural representations at this and other points internal to trained neural networks Farrell et al. 2202 (see also Farrell et al. 2023) Shwartz- Ziv & Tishby 2017). We next show that the present sharpness-based approach can describe related properties at penultimate and intermediate network layers.
10

---

Published in Transactions on Machine Learning Research (01/2026)
While neural collapse is often described as global phenomenon (within-class variance across all samples of class), our results are local, quantifying robustness of layer responses for nearby points in the input space. Nevertheless, a connection emerges when viewing within-class samples as local perturbations around their class mean, a perspective we develop below.
We first apply our method to study the penultimate-layer features. To accomplish this we can adapt the linear stability trick in Equation (5) to establish a relationship between their robustness and sharpness. More concretely, can show that
W
[Vx min ( |
(16)
Here again, W denotes the first-layer weights, W. denotes the last-layer linear classifier weights, and g is the penultimate- layer feature. miin (W)) is defined as the square root of the smallest eigenvalue of W WL. The proof is given in Appendix
To extend this analysis to representations before the penultimate layer, note that inequality 16 extends to any middle-layer representation, as detailed in Appendix A that very similar conclusions apply directly.
Let the feature dimension be d and the number of classes be K. Then, WL kxd, and (min(Wu))) if d > K; otherwise, it is the smallest singular value of W.. An effective bound on the robustness of penultinate-layer (or other internal-layer) features is obtained when d S K, i.e., when the number of classes is at least as large as the feature dimension. This condition naturally satisfied in settings such as language modeling, retrieval systems, and face recognition applications, where the number classes (vocabulary size, number of items, number of identities) is typically much larger than the feature dimension.
Interestingly, these are also settings where generalized neural collapse phenomena have been observed (iang et al. 2023). While our bounds are local (quantifying robustness to small input perturbations), potential connection to neural collapse emerges when we consider samples from the same class as local perturbations around the class mean. In neural collapse, the within-class variance of features vanishes, meaning all samples from a class produce nearly identical features. This can be viewed through our framework as extreme robustness: if the network maps all within-class samples (which differ from the class mean by small to moderate perturbations to the same feature representation, the features exhibit strong compression and robustness properties that our bounds capture locally. This said, the relationship between our local robustness bounds and the global geometric structure of neural collapse, including other aspects not considered here (e.g., the simplex equiangular tight frame geometry) remains an open question for future investigation.
4 Experiments
All networks are trained with MSE (quadratic) loss except for the pretroined ViTs in Section 4.3
4.1 Sharpness and compression metrics during training: verifying the theory
The theoretical results derived above show that when the training loss is low, measures of compression of the network's representation are upper-bounded by a function of the sharpness of the loss function in parameter space. This links sharpness and representation compression: the flatter the loss landscape, the lower the upper bound on the representation's compression metrics.
To empirically verify whether these bounds are sufficiently tight to show clear relationship between sharpness and representation compression, we trained a VGG-11 network (Simonyan & Zisserman 2015) to classify images from the CIFAR-10 dataset (Krizhevsky 2009 and calculated the (approximate) sharpness
(Equation 3 the log volumetric ratio (Equation (7) MLS (Definition 3.7 and NMLS (Definition 3.9
during the training phase (Figure) 2 and Figure 3).
We trained the network using SGD on CIFAR-10 images and explored the influence of two specific parameters that previous work has shown to affect the network's 's sharpness: learning rate and batch size (Jastrzebski
11

---

Published in Transactions on Machine Learning Research (01/2026)
Test accuracy
NMLS
Sharpness
12.5
Pa pi tn de
woulle www.kt foe dol
value
95
10.0
15
M www.eeeooooneeeeeeeeeeeee
7.5
10
93
5.0
0.05
Log volume
MLS
Log local dimension
0.1
-6
1.0
0.2
value
0.4
-8
-10
0.5
0.2
-12
50
100
150
0
50
100
150
0
50
100
150
iteration (x5000)
iteration (x5000)
iteration (x5000)
Figure 2: Trends in key variables across SGD training of the VGG-11 network with fixed batch size (equal to 20) and varying learning rates (0.05, 0.1 and 0.2). Higher learning rates lead to lower sharpness and hence stronger compression. From left to right: Test accuracy, NMLS, sharpness (square root of Equation (3) ), log
volumetric ratio (Fquation MTSS, and local dimensionality of the network output (Equation 115)
et al.
2018 For each combination of learning rate and batch size parameters, computed all quantities
across 100 input samples and averaged across five different random initializations for network weights.
In Figure 2 we study the link between sharpness and representation compression with a fixed batch size (of 20). We observe that when the network reaches the interpolation regime, that is, when training loss is extremely low, the sharpness decreases with compression metrics, including MT.S, NMLS, and volume. The trend consistent across multiple learning rates for fixed batch size, and MLS and NMLS match better with the trend of sharpness.
In Figure 3 repeated the experiments while keeping the learning rate fixed at 0.1 and varying the batch size. The same broadly consistent trends emerged, linking a decrease in the sharpness to a compression in the neural representation. However, we also found that while sharpness stops decreasing after about 5 10a iterations for a batch size 32, the volume continues to decrease as learning proceeds. This suggests that other mechanisms, beyond sharpness, may be at play in driving the compression of volumes.
We repeated the experiments with an MLP trained the Fashion dataset (Xiao el. al. 2017 (Figure 1.10 and Figure I.1Ip. The sharpness again follows the same trend MLS and NMLS, consistent with our bound. The volume continues to decrease after the sharpness platcaus, albcit at much slower rate, again matching our theory, while suggesting that an additional factor may be involved in its decrease.
4.2 Correlation between compression metrics and their sharpness-relatees bounds
We next test the correlation between both sides of the bounds that we derive more generally. In Figure show pairwise scatter plots between MLS (resp. NMLS) and the sharpness-related bound on MLS (resp. NMLS) (we include an exhaustive set of correlation matrices in Appendix H.2 Interestingly, we find that quantities that only consider a single layer of weights, such as the relative flatness (Petzka el. al. 2021 | and the bound on the MLS (Proposition 3.8 can exhibit negative correlation between both sides the bound in some cases (Figures H.7 and H.8). This demonstrates the nccessity introducing NMLS to explain the relationship between sharpness and compression. Nevertheless, we find that all compression metrics, such MLS, NMLS, and LVR, introduced in Section g correlate well with sharpness in all of our experiments
12

---

Published in Transactions on Machine Learning Research (01/2026)
Test accuracy
NMLS
Sharpness
12
15
value
95
8
10
6
batch size
93
Log volume
MLS
Log local dimension
0.6
20
-6
1.00
value
-8
0.75
0.4
0.50
-10
0.2
0.25
50
100
150
0
50
100
150
0
50
100
150
iteration (x5000)
iteration (x5000)
iteration (x5000)
Figure 3: Trends in key variables across SGD training of the VGG-11 network with fixed learning rate size (equal to 0.1) and varying batch size (8, 20, and 32). Smaller batch sizes lead to lower sharpness and hence stronger compression. From left to right in row-wise order: test accuracy, NMLS, sharpness (square root of Equation 3 ) log volumetric ratio (Equation 7 MLS, and local dimensionality of the network oulpui.
(Equation
15
(Appendix H.2 Although the bound in Proposition 3.4 loose, the log LVR still correlates positively with sharpness and MLS.
4.3 Empirical evidence in Vision Transformers (ViTs)
Since our theory applics to linear and convolutional layers as well as residual layers (Appendix) B relationships among sharpness and compression, as demonstrated above for VGG-11 and MLP networks, it should hold more generally in modern architectures such as the Vision Transformer (VIT) and its variants. However, naive ways of evaluating the quantities discussed in previous sections are computationally prohibitive. Instead, we look at the MLS normalized by the norm of the input and the clementwise adaptive sharpness defined in Andriushchenko et al. (2023); Kwon et al. (2021). Both of the metrics can be estimated efficiently for large networks. Specifically, in Figure
plot the normalized MLS against the deeenntwss--ddaptee sharpness. The analytical relationship between the normalized MLS and the clemontwiso-adaatine sharpnoss and the details of the numerical approximation we used are given in Appendix F.2 and Appendix G respectively, For all the models, we attach a sigmoid layer to the output logits and use MSE loss to calculate the adaptive sharpness. F the results for 181 pretrained VIT models provided by the timm package (Wightman 2019). We observe that there is a general trend that lower sharpness indeed implics lower MLS. However, there are also outlier clusters, with data corresponding to the same model class; an interesting futurr direction would be to understand the mochanisms driving this outlier behavior. Interestingly, did not obscrve this correlation between unnormalized metrics, indicating that weight scales should be taken into account when comparing between different models.
4.4 Sharpness and local dimensionality
A priori, it is unclear whether the local dimensionality of the neural representations should increase or decrease as the volume is compressod. For example, the volume could docrcase while maintaining its overall form and symmetry, thus preserving its dimensionality. Alternatively, one or more of the directions in the relevant tangent space could be selectively compressed, leading to overall reduction in dimensionality.
13

---

Published in Transactions on Machine Learning Research (01/2026)
A. CIFAR-10 VGGG11
B. FashionMNIST MLP
C. CIFAR-TO LeNet
MLS vs
0.92
8.0
P
50.60
160
p 0.69
300
7.5
140
MLS bound
250
7.0
6.5
120
200
6.0
MLS
100
150
5.5
5.0
80
100
100
0.0
NMLS vs NMLS bound
90
0.92
p
2000
NMLS bound
NMLS bound
350
300
1500
NMLS
250
1250
750
150
5.0
7.5
10.0
12.5
9.5
10.0
10.5 NMLS
11.0
10
12 NMLS
14
16
NMLS
We trained 100 different models for each combination of datasots and networks by varying learning rates, batch size, and random initializations. Pairwise scatter plots between MLS (resp. NMLS) and the relatee bound MLS (resp. NMLS) are shown here. For MLS (resp. NMLS) bound see Proposition 3.8 (resp. Proposition 3.10 The Pearson correlation coefficient is shown in the top-left corner for each scatter plot. See Appendix for the full pairwisc scatter matrix.
Figure 4:
Figure 2 and Figure 3 show our experiments computing the local dimensionality over the course training. Here, we find that the local dimensionality of the representation decreases as the loss decreases to near 0, which is consistent with the viewpoint that the network compresses representations in feature space as much possible, retaining only the directions that code for task-relevant features (Berner et al.) 2020 Cohen et al. 2020). However, the local dimensionality exhibits unpredictable behavior that cannot be explained by the sharpness once the network finds an approximate interpolation solution and training continues. Further experiments also demonstrate a weaker correlation of sharpness and local dimensionality compared to other metrics such as MLS and volume (Figures H.7 to H.9h. This discrepancy is consistent with the bounds established by our theory, which only bound the numerator Equation (15). It also consistent with the property of local dimensionality that we described in Section 3.3 overall: it encodes the sparseness of the eigenvalues but it does not encode the magnitude of them. This shows how local dimensionality is distinct. quality of network representations compared with volume, and is driven by mechanisms that differ from sharpness alone. We emphasize that the dimensionality around point on the "global" manifold of unit activities; dimension on larger scales (i.e., across categories or large sets of task inputs (Farrell et al. 2022
study here is a local measure, the finest scale
Kothapalli et al. 2022 Zhu et al. 2021 Ansuini et al. 2019 Papyan et al. 2020 may show different trends.
Recanatesi et al. 2019
5 Discussion: connection to generalization
So far we have avoided remarking on implications of our results for the relationship between generalization and sharpness or compression because generalization is not the main focus of our work. However, our work may have implications for future research on this relationship, and we briefly discuss this here. Our theoretical
14

---

Published in Transactions on Machine Learning Research (01/2026)
x
104
MobileViT
x
x
Adaptive Sharpness
103
X
X
FastViT
10?
20000
40000
60000
80000
Normalized MLS
Figure 5: Adaptive sharpness vs Normalized MLS 181 ViT models and variants. Different colors represent. different model classes. For most models, there is a positive correlation between Sharpness and MLS. However, outlier clusters also exist, for MobileViT (Mehta & Rastegari] 2022) models in the upper left corner, and two ViS (Vasu et al. 2023
models in the lower right corner.
results establish that sharpness directly controls compression neural cesseentttioos-spcifiiiii how robust intermediate representations are to input perturbations. By "robustness representations, we mean that small changes to the input produce proportionally small changes in the network's internal feature representations (quantified by low LVR and MLS). This compression property desirable in adversarial settings where want representations to remain stable under input perturbations.
However, representation robustness alone does not guarantee better generalization. Ma & Ying 2021 gave generalization bound based on adversarial robustness, but the theorem requires that most test data lie close to the training data, which demands extensive training coverage. In contrast, our correlation analysis in Appendix H.2 shows that sharpness consistently has a higher correlation with compression metrics than with the gencralization gap. Additional experiments in Appendix I show that lower sharpness does not always imply better lest accuracy.
This phenomenon is also observed in applied machine learning literature. For example, it is well-established that large learning rates lead to lower sharpness (flatter minima), relationship we also observe in our Figure 2 and consistent with Cohen et. al. (2020) and Wu et al. (2022). low sharpness directly implied better generalization, we would expect large learning rates to improve out-of fiitof--ditributtonn (OOD) performance. However, Wortsman et al. (2022) demonstrates the opposite: large learning rates can severely hurt OOD generalization. This shows that low sharpness alone-and the compressed representations it produces-dae does not guarantee superior generalization.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Thus, our work offers possible resolution to the contradictory results the relationship between (reporaanttriaaiioninvvaaanll sharpness and generalization (Wen et al., 2023 Andriushchenko et al. 2023 sharpness directly determines representation compression, but whether this compression translates to superior gencralization may depend on other latent factors beyond just sharpness.
One possible factor that compression metrics (LVR, MTS, NMT.) measure robustness to small random perturbations networks with low sharpness have representations that are insensitive noise. However, good generalization sometimes requires networks to remain sensitive semantically meaningful differences in the imput. To illustrate this distinction, consider a large language model (LLM) performing 'needle in a haystack" test, where it must extract specific information from long context. When the query changes meaningfully (e.g,, asking for a different fact), the output should change substantially to provide the correct answer- the network must be sensitive to the semantic content the query. Yet simultaneously, the network should be robust small random perturbations in the text, such as typos or minor paraphrasing. Sharpness controls the latter (robustness to random noise), but whether this robustness aids generalization depends on the nature of the distribution shift at lest time: if test inputs primarily differ from training via noise-like perturbations, compression helps; if they differ in task-relevant semantic features that require discrimination, excessive compression may reduce the network's s ability to distinguish these meaningful differences.
6 Summary and Conclusion
This work presents dual perspective, uniting views in both parameter and feature space, of several properties of trained neural networks. We identify two representation compression metrics that are bounded by sharpness local volumetric ratio and maximum local sensitivity and give new explicit formulas for these bounds. We conducted extensive experiments with feedforward, convolutional, and attention-based networks and found that the predictions of these bounds are borne out for these networks, illustrating how MT. particular is strongly correlated with sharpness. Overall, we establish explicit links between sharpness properties in parameter spaces and compression and robustness properties in the feature space.
By demonstrating both how these links can be tight, and how and when they may also become loose, we propose that taking this dual perspective can bring more clarity to the often confusing question of what sharpness actually quantifies in practice. Indeed, many works, as reviewed in the introduction, have demonstrated how sharpness can lead to generalization, but recent studies have established contradictory results. Therefore, our work contributes the further exploration of sharpness- neural network performance. Future directions include tightening the derived bounds given knowledge of the data distribution, and investigating the conditions under which representation compression translates to improved generalization.
methods for improving
16

---

Published in Transactions on Machine Learning Research (01/2026)
References
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-awarr minimization. In International Conference on Machine Learning, pp. 639 668. PMLR, 2022.
Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. arXiv preprint orXiv:2302.07011 2023.
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Peter Bartlett, Philip M Long, and Olivier Bousquct. The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. Journal of Machine Learning Research, (316):1-36, 2023.
Julius Berner, Philipp Grohs, and Armulf Jentzen. Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. SIAM Journal on Mathematics of Data
Science, 2(3):631-66) 2020. Publisher: STAM.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory, pp. 483 513. PMLR, 2020a.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an orsteenn-uhlenbeck like process, 2020b. URL
09080
Uri Cohen, YuoYeon Chung, Daniel D. Lee, and Haim Sompolinsky. Separability and geometry of object manifolds deep neural networks. Nature commumications, 11(1):746, 2020. Publisher: Nature Publishing Group UK London.
Yaim Cooper. The loss landscape of overparameterizzed neural networks. ar preprint arXiv:1804.10200 2018.
Alex Damian, Tengyn Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances Neural Information Processing Systems, 34:27449-27448 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference Machine Learning, pp. 1019 1028. PMLR, 2017.
Yuguang Fang, K.A. Loparo, and Xiangbo Feng. Inequalities for the trace of matrix product. IEEE Transactions on Automatic Control, (9(1)) 2490, 1994. doi: 19.19419..6241...
Matthew Farrell, Stefano Recanatesi, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brown. Gradient- based learning drives robust. representations in recurrent neural networks by balancing compression and expansion. Nature Machine Intelligence, 4(6):564 573, 2022. Publisher: Nature Publishing Group UK London.
Matthew Farrell, Stefano Recanatesi, and Eric Shea-Brown. From lazy to rich to exclusive task representations neural networks and neural codes. Current opinion neurobiology, 83:102780, 2023.
Benjamin Fehrman, Benjamin Gess, and Armulf Jentzen. Convergence rates for the stochastic gradient descent. method for non-convex objective functions. Journal of Machine Learning Research, 21(136):1 48, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01. 01441, 2020.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRwie, pp. 214262, 2017.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Mario Geiger, Leonardo Petrini, and Matthieu Wyart. Landscape and training regimes in deep learning.
Physics Reports, 924:1-18, 2021. ISSN 0370-1573. doi: 11100///.plysrrp.222...... URL https
www.sciencedirect.oo cww
Diego Granziol. Flatness is a false friend, 2020. URL
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral computation, 9(1):1-42, 1997.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey Three Factors Influencing Minima in SGD, September 2018. URL http: http://axxvvv.o orgo abs/
1711.04623 arXiv:1711.04623 (cs, stat].
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number classes. arXiv preprint wrXi::3300003311, 2023.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Pantastic generalization measures and where to find them. erXiv preprint arXiv:1912.021 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04888 2016.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206. 0400:2206.0041,,, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://wwy cs. toronto. edu/ http///kii/leamnng---ie feartr-s-eeatutr22000000 2ot.urrrs-----RRRpPP Td.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-awee aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905 5914. PMLR, 2021.
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? - mathematical framework, 2022. URL http:://aaxxv..r/abb///////s
06914
Ashok Litwin-Kumar. Kameron Decker Harris, Richard Axel, Haim Sompolinsky, and LF Abbott. Optimal degrees of synaptic connectivity. Neuron, 93(3):1153-11)) 1164, 2017.
Chao Ma and Lexing Ying. On linear stability of SGD and input-smoothness of neural networks, 2021. URL http://arxiv org/ absp//05./22.....//// 13462
Luca Mazzucato, Alfredo Fontanini, and Giancarlo La Camera. Stimuli Reduce the Dimensionality of Cortical Activity, Frontiers in Systems Neuroscience, 10, February 2016. ISSN 1662-5137, doi: 1..3889ffnsys.2011 00011. URL https::/www. ncbi nlm. mih httpah/////oa/rrrrlcc//////1111/
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer, 2022. URL http:://aaiii..ogg///////1..01.0 02178
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
Vardan Papyan, XY Han, and David I, Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652 24663, 2020.
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. Advances in neural information processing systems, 34:18420-18432, 2021.
Aviv Ratzon, Dori Derdikman, and Omri Barak. Representational drift as a result of implicit reg- ularization, 2023. URL https:://mv http::///wwbiooiivvooooooo wwtp/o//t//////////oooooo/o/tt/t 10t.////n///..111//22222.. Section: New Results.
1111/11211.1111 05. 04. 539512v3
Pages:
18

---

Published in Transactions on Machine Learning Research (01/2026)
Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brow Brown. Dimensionality compression and expansion in deep neural networks. arXiv preprint trxivv:9000 00443, 2019.
Stefano Recanatesi, Serena Bradde, Vijay Balasubramanian, Nicholas A Steinmetz, and Eric Shea-Brown. A scale-dependent measure of system dimensionality. Patterns, 3(8), 2022.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint Xrv::11733.00 1000000000000,, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largo-scale image recognition, 2015.
Yusuke Tsuzuku, Issci Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis, 2019.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using struct reparamederization, 2023. URL
14189
Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-aware minimization minimize sharpness? arXiv preprint ar vrxXi:2211.05772 2022.
Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness achieve better generalization, 2023.
Ross Wightman. Pytorch image models. httpps//ggthabb..cc/rrrrrgaatttt
2019.
Mitchell Wortsman, Gabriel Ihharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 7959-7971, June 2022.
Lei Wu, Chao Ma, and Weinan E. How SGD Selects the Global Minima in Over-parameterrieed Learn- ing: A Dynamical Stability Perspective. In Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL httpss:/ http://ppprr..pp........ nips. cc/paerr files/ htles//apeer/011/heass 1881228668879300004666444000 -bwtffff-bttttt...oo.o html
Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXie: 708. 008.07747, 2017. URL
Zeke Xie, Issei Sato, and Masashi Sugiyama. Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima, January 2021. URL http:://arrdv.orr/aab////0 03495 arXiv:2002.03495 [es, stat].
Ning Yang, Chao Tang, and Yuhai Tu. Stochastic gradient descent introduces an effective landscape- dependent regularization favoring flat solutions. Physical Review Letters, 130(23):237101, 2023. doi:
10.1103/PhysRevLeet 130.237101. httLhhtts:///iie httppp/r...aaa/...//////a/aeetta 13..:/etttt......... 237101 Publisher: American Physical Society.
Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems, 34:18722-18733, 2021.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The Anisotropic Noise Stochastic Gradient. Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects, June 2019. URL http::/arrv..rrg/bba///0//// wwtp/ba/////..000.....1 arXiv:1803.00199 [es, stat).
19

---

Published in Transactions on Machine Learning Research (01/2026)
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820 29834, 2021.
20

---

Published in Transactions on Machine Learning Research (01/2026)
A Linear stability trick on penultimate-layer and middle-layer features
We define g(x) to be the penullimate-layer features such that f(x) Wr.g(WW (Wr) With slight abuse of notation, we define J = (g(Wx) Similar to Equation ( we have
oWx
[W/((Wx 0)(( |
(17)
Vxg Wx; 0) JW,
Lemma A.1. ||AB| 2 Amin Cain(AAA ()]A)|| where Amin is the smallest eigenvalue.
Proof. By the definition of Frobenius norm;
|AB| TY(BB A²) =
(18)
From Fang ct al. 19994. we have that for positive semidefinite matrices P and Q,
Am (P) Tr(Q) & Tr(PQ)
(19)
Therefore,
TH(A" 2 Amin (A Tr(BB") = Amin A A)]
(20)
As result, [WLJ]le 2 V Amiu of classes, and Wz E Rxd Then, Amin W W/) vanishes when d > K, otherwise /Amin(WT (WTWL) "min(W,) the smallest singular value of Wr Therefore,
Amia(W (WE WE)llll.. Let d be the feature dimension, and K be the number
V Vx (xx |JW|LF
S |
[WL
[
(21)
Amin (WTWD)
IV 0)
|||]
||x|2 Amin (WWW,)
More generally, if consider arbitrary intermediate-layen representations, write (00[<heg[))) where is the transformation from linear transformed input to the intermediate-layer representations and h is the mapping from the representations to the output of the network. Then Equation 17 becomes
Vw//(Wx 0) ||
(22)
V Vx9( Wx 0 JW,
where J @g(Wx) and Jn
[xx 0y Wx)
Therefore, similar to Equation 21 we have
0) |W|2
x Wxx 0)
(23)
||x||2 / Amin (J Jn.
21

---

Published in Transactions on Machine Learning Research (01/2026)
B Adaptation of Inequality 6 to Residual Layers
We need slightly adapt the proof in Equations 44 and (5) Consider network whose first layer has a residual connection: y= g f(Wz)), where is the nonlinearity with bias (e.g. f(x) taah( + 6)), and is the rest of the mappings in the network. Then we have
Vwwg( + [(W))) |J |
(24)
Veg(z + f(Wx)) JKW
where J 2g(x+f(Wa) and K af(Wx) ol))
a(Wa)
Now, we get the bound for the difference between MLS of input and the MIS of imput the next layer:
Therefore, 1V f(W S [1l + [[J |2/l wil & |||l22 +
[[Wll.. /(
|
[V + f(Wx))ll || s Ve f(Wz))ll
[Wll
(25)
||1||
Notice that if we apply this inequality to every residual layer in the network, and sum the left-hand side, we will get a telescoping sum on the left-hand side. Assuming the last layer is linear with weights Wz, we get Vwg((z= + f(Wx))) The right- hand side is bounded by
[|Vzg(x f(Wxx))ll2 l| s E|=
[Willa
|
sharpness due to Cauchy, see also Equation
42
C Proof of Equation (3)
Lemma C.1. 1 is an approzimate interpolation solution, i.e. (x; 0) y-ll for i E (1,2, ,,], and second derivatives of the network function V fxx; 0)1 M is bounded, then
S(0") 1 Σ
Vol (x) 0(s)
(26)
Proof. Using basic calculus we get
S(0) Tr(Vz
Tr(V - y,l13)
2n
Tr Vo(([(((()) 0) y.)² (x](x; 0))
2n
0
((f(x; 0) yz)² T Vol 0)),
00,
m
0
(f 0) -y)) T fo, 0)
n
00,
Vo, (xi, 0| ( 0) yz)² " Vo, (x:,
Vof(x;, 1)
(f(x;, 0) - y.) T (x;;
n
22

---

Published in Transactions on Machine Learning Research (01/2026)
Therefore
1 m S(0)
m ΣΣ Σ ==
Vo (x, 0)
V. (f(x,, 0) y.) Vo, f(x,, 0) mMe O(s).
(27)
In other words, when the network reaches zero training error and enters the interpolation phase (i.e., fits all training data correctly), Equation (3) will be a good enough approximation of the sharpness because the quadratic training loss sufficiently small.
Proof of Equation (8). Proposition 3.4 and Proposition 3.6
D.1 Proof of Equation 8
We prove the bound on the local volumetric ratio using the arithmetic- geommerrc mean inequality, Let J= Vxf NxM with singular values ON, O. By the AM-GM incquality applied to the squared singular values:
Vdet(JJT)
II
9
N
N/2
2
(AM-GM inequality)
N
1=1
Tr(JJT)
N/2
(28)
(eigenvalues of JJ are of)
Tr(JJJ) J)
N/2
(trace identity: Tr(AB) Tr(BA))
[
N/2
(Frobenius norm: || Tr(J" J)).
N
This completes the proof of Equation
D.2 Proof of Proposition 3.4 and Proposition 3.6
For notational simplicity, we write fi := fx, 0*) in what follows. Because of Equation following incquality due to Cauchy-Swartz incquality,
we have the
V w f
Vxfillé < W
Xx
(29)
w
Wwflll
n
xx
112
Since the input weights W is just a part of all the weights (0) of the network, we have Vwli < Velll
We next show the correctness of Proposition 3.4 with standard lemma.
Lemma D.I. For veclor x, ||x||> p 2 ||x[|, for S p 5 x..
Proof. First show that for < 1, we have (Ial + |b))* < |a| [b] It's trivial when either or b is 0. So W.L.O.G, assume that |a] < [b], and divide both sides by [8/k. Therefore it suffices to show that
23

---

Published in Transactions on Machine Learning Research (01/2026)
for t 1 1,+ < t +1. Let f(t) t1 - t - 1, then f(0) 0, and ((() k(1 [) - ktt-l,
Because k - < 0, 1+ and I. < 1, 1k-1 > > (1 + 1)k-1 Therefore f'(1) < 0 and f(() < 0 for
<<t<l. 1. Combining all cases, we have (/a) + 161} - la|* + 161z for 0 < k < 1. By induction, we have
& En lanl.
Now we can prove the lemma using the conclusion above,
p/9-1/p
(1 E(En 18)/// )
1/0 =
-
1/8
Σ ænl 8
Σ a0 1
1
|zn|?
Now we can prove Proposition 3.4 Proposition. The local volumetric ratio is upper bounded by sharpness related quantity:
N-N/2
n,5(0")
N/2
dV 1
[ 0-) )1> S
(30)
n
x
2N
N
for all N 2 1.
Proof. Take the in Lemma D.1 to be |/Vof( 1I? and let 1.g k, then we get
1/k
Σ [(Vollll)))))
Vofil?..
(31)
Therefore,
k/2
Vw 1 W
V
Vo
xx
X
1=1
(32)
n k/2-1
W
S(0r)k/2
X
Next, we show that the first inequality in Equation weights.
32
be tightened by considering all lincar layer
Proposition. The network volumetric ratio is upper bounded by a sharpness related quantity:
t N-N/2 L
I
W 1N
(nS((0))
N/2
Σ dV, I
n ΣV S
(33)
N
Proof. Recall that the input to the L-th lincar layer is 2a for 1 =,1,, 1,2.... L. In particular, x/ the input of the entire network. Similarly, Wt is the weight matrix of 1-th linear/cone conaauutoollli layer. With slight abuse of notation,
f to denote the mapping from the activity of 1-th layer to the final output, and f : (x, 0*). We can apply Cauchy-Swarzz inequality again to get
2
V
V w. 11
(34)
W
2
Vw.
x
24
1=
24

---

Published in Transactions on Machine Learning Research (01/2026)
Using Lemma D.1 again we have
1/K
Σ
Vw.
IVofili
1=1
(35)
1/K
=
The second equality holds because both sides represent the same gradients in the computation graph. Therefore from Equation 34
we have
W 2k 1-
V Vn
5(0")*
(36)
7
n
2b
E Proof of Proposition 3.8 and Proposition 3.10
Below we give the proof of Proposition 3.8
Proposition. The marimum local sensitivity is upper bounded by sharpness related quantity:
MLS-
(x; < |
1
S(0))1/2
(37)
n
x
Proof. From Equation
we get
1
fwlll
MLS
V Vxlill
|
(38)
|x, ||
Now the Sauchy-Schwarz inequality tells us that
n Vw
(39)
11
|
Therefore
MLS < [ |||2,
(40)
< W 12
1/2
X
Now we can prove Proposition 3.10
Proposition. The network marimum local sensitivity is upper bounded by a sharpness related quantity:
NMIS 1
W 1(00)112
(X} 00)12 S
(41)
25

---

Published in Transactions on Machine Learning Research (01/2026)
Proof. We can apply Equation obtain
40
to every linear layer and again apply the Cauchy-Schwarz inequality to
L n
NMLS
x.
(x),
12)
W 1
Vw.
X
2
Y2
1=1
(42)
W
V
| 2
1(00)
Note that the gap in the last inequality is significantly smaller than that of Equation consider all linear weights.
40 since now we
F Reparametrizziion-inwararr sharpness and input-invariant MLS
In this section, we demonstrate that various epparametrizatioon-nvaat literature can be understood as measuring robustness of outputs inputs and internal reprosentations through ML.S-like quantities. As discussed in Section 3.4, this provides a novel perspective: reparamnetriialloo-invvaannn sharpness is fundamentally characterized by the robustness of outputs to internal neural representations. We examine two difforent approaches from the litcrature Tsuzuku et al. (2019) 's matrix-normalized and normalized sharpness (Section F F.1). and Kwon et al. (2021); Andriushchenko et al. (2023) elementwise-
sharpness metrics proposed in the
adaptive sharpness (Section F.2 showing how each relates to different variants of MLS.
F.1 Reparametiizaioo--vaaaian sharpness in Tsuzuku et al. (2019)
In this appendix, we show that the reparameetiiaatoonnivviiinn sharpness metrics introdee Tsuzuku et al. (2019) can be seen an effort tighten the bound that we derived above. We formalize metrics from their work and establish their connections to MLS.
Definition F.1
sharpness
(suuuuuu et al. 2019 ) The matriz-normalized sharpness
of neural network is defined as
Smatrie (0)
((W.))
(3)
1=1
where S(Wi) is the trace of Hessian of the loss w.r.t. the weights of the 1-th layer, and [ is the spectral norm of the weight matria at layer Proposition F.2. The matriz- norriiz-ormmlized sharpness upper bounds weighted MLS across layers:
x MLS' < Smatriz(
(44)
-
where x' layer I.
n En
is the inverse of the average input norm at layer 1, and MLS' is the MLS at
1x
Proof. From Equation
40
we have
MLSS
<
Will
Willa ((W.)) = Smatrix (0).
(45)
1
1=1
26

---

Published in Transactions on Machine Learning Research (01/2026)
Note that similar inequality holds if we use Frobenius norm instead of 2-norm of the weights.
Definition F.3 (Normalized sharpness (Tsuzuku et al. 2019 ) The normalized sharpness of neural
network is defined the solution to the optimization problem:
min Σ 0.d 4.
8L aW (0,0,) 422(0109)3
W
Snormakizdd (0)
(46)
where o, a' are scaling vectors and X hyperporemeter.
Proposition F.4. The normalized sharpness optimization problem is equivalent to minimizing an upper bound on intarinn MLS-like quantity:
S normakized (0) 2 Idiag(o""")V<<<lle (x)]2)
(47)
where the quantity the right is invariant under the transformation Wx W diag(o )xd
Proof. Note that by Lemma C.1
8LL BW oW
Moreover, we have
W
W
Σ [IW.,, +
Vw... Vw.,, ()
4X((or0')²
(o,
Idiag( or J
|| | /aliiag W diag(o'
(''=)
(48)
( (
where J = af(wx.0)
OWW (sec some of the caleulations in Equation (4)). Therefore, the optimization problem
Equation 46p is equivalent to choosing o, a' minimize the upper bound on a scale-invariant MLLS-liee quan- tity (the quantity is invariant under the transformation of the first layer from Wx W diag(o (iiag(o->)diian (ddiag( )x), where diag(or) x becomes the new input).
For simplicity, we do not scale the original dataset in our work and only compare MLS within the same dataset. As result, can characterize those waw.ammtrizationnnaaaatet sharpness metrics by the robustness of output to the input. If we consider all linear weights in the network, then those metrics indicate the robustness of output to internal network representations.
In summary, we have shown that Tsuzuku et al. 2019 's reparametrizaatin-innaan
sharpness metrics can be interpreted through our MLS framework: their matrix normmaized sharpness corresponds exactly to the upper bound on weighted MLS across layers, while their normalized sharpness minimizes scalc-invariant MLS-like quantity. This establishes that these metrics, while designed for reparametrization invariance, are fundamentally measuring the robustness of outputs to inputs and internal representations.
F.2
eeparaametizzation-nvaiiaa sharpness upper-bounds input-invariant MLS
Having shown that Tsuzuku et al. (2019 's metrics tighten our standard MLS/NMIL bounds, we now examine sharpness. In this subsection, we prove a formal result relating Meeeentwis-adapptiv sharpness from Kwon et al. (2021): [Andriushchenko et al. (2023) to a different variant of MI.S: input-invariant MI.S. This demonstrates that different reparamctrization- invvrrian sharpness metrics from the literature can be understood as measuring robustness of outputs inputs through different. MT.S variants.
a different approach lo wwwprramteriialiinnnnaiiia
27

---

Published in Transactions on Machine Learning Research (01/2026)
In this appendix, consider the adaptive average-case n-sharpness considered in Kwon et al. (2021) Andriushchenko et al. 2023
S (w, w() - 2 BS 6. 50 [Ls(w + +6 Ls())]
(49)
which is shown to be elementwise adaplive sharpness in Andriushchenko el al. 2023 They also show that
for thrice differentiable loss, L(w), the average-case elementwise adaptive sharpness can be written as
Su (w, (w, (w) E.... Tr (s(V o ///w [/]l//T]] O(p).
(50)
Definition F.5. We define the Elementuisse-Adaptar Sharpness Sudaptive to be
Sadaptive (w) - lim Saag (w, /w]) = Es-p. [Tr Lg2 & [//ll///]]]
(51)
p-0
In this appendix, we focus on the property of Sadeptive instead of the approximation Equation the proof of Lemma C.I we have the following lemma.
50
Adapting
Lemma F.6. O is an approcimate interpolation solution, i.e. []](x; 0) yill < e for i E (1,2, ,,,,, V fxx, 0)ll < M for all j, and is MSE loss, then
1 n m
S Sedaptice (0")
Σ
0, Vo, 0)12 O(E),
(52)
n2
where m is the number of parameters.
Proof. Using basic calculus we get
Sudaptive (f)
Tr(V [f(x; 0) - o
2n
Tr Ve 0) v.) Vof(x, o T
2n iml
m
0 00; ((f(x; 0) y.) " 0));
(J(x,, 0) Vo, S(x,
00 )
V (x, 0) 10,,(((x, 0) - y.)" v
0
Vo, f(x. 0)1
0 0)12((xx,,0) 0) Vo 0)
Therefore
Sadaptive (0)
ΣΣ
Σ
0 Vo, f(x, 0]|l3
Σ
(f 0) 10, 10, f(x,
fx,, 0)1 < mMMe = =(EE)
n
== j=1
(53)
28

---

Published in Transactions on Machine Learning Research (01/2026)
Definition F.7. We define the Input-invariant MLS of network f :RM IRN to be
2 1 (1)0)2
(54)
where up is the p-th entry of i-th training sample.
It turns out that again the adaptive sharpness upper bounds the input-invariant MLS. Proposition F.8. Assuming that the condition of Lemma F.6 holds, then dlementwise-adappti sharpness upper-bounds invuttinvariatt MLS:
m
E Vo, 0xx; nd ΣΣ 10 of (o)))) n
(55)
Proof. Now we adapt the linear stability trick. For 0 = W, the first-layer weights, we have
0,1² vo, = Σ
W
i,ik
Σ
w
x
(56)
jk
ki
1
k=1
Σ
V up 11|2
d
where same as in Equation 44. J = af(wx:0) o(Wx) Vx/ 0) JW, and Fp is the p-th entry of x. Taking the
sample mean of both sides proves the proposition.
G Numerical approximation of normalized MLS and elementwise-adaptive sharpness
In this appendix, we detail how we approximate the normalized MLS and adaptive sharpness in Section 4.3 Note that for all network f the last layer the sigmoid function, so the out is bounded (0, 1), and we use MSE loss to be consistent with the rest of the paper.
For the adaptive sharpness, adopt the definition in Andriushchenko et al. (2033 and use the sample mean to approxinate the expectation in Equation [49 Therefore, for network ((w),
Sadaptive (f)
L(x; w + 6,) L(x;:w,
(57)
num
1=1
where 8 ~ (O 0.01 diag(Iw/3)).
For normalized ML.S, we first reiterate the definition from the main text. We use normalized MLS below as an approximation to the input-invariant MLS (Definition F.7), because the latter is computationally prohibitive for modern large ViTs, On the other hand, there is an efficient way to estimate normalized MLS as detailed below.
Definition G.I. We define the normalized MLS Σ'L ||x IV. 11
Therefore, to approximate normalized MLS, we need to approximate || fllz. By definition of matrix 2-norm,
|Vz 8/12 max [lf(x + 8) f
V>,ll, sup ||5]|
(58)
||811
To solve this optimization problem, we start from a randomly sampled vector 8 that has the same shape the network input, and update 8 using gradient descent.
29

---

Published in Transactions on Machine Learning Research (01/2026)
H Empirical analysis of the bound
H.1 Tightness of the bound
In this section, we mainly explore the tightnoss of the bound in Equation
11
for reasons discussed in
Section 3.2 First we rewrite Equation as
1 x x1, 0*)
MLS
12)
n
VwJ(xx (x, 1'Il
= B
|x:12 l22
(59)
12
V Vwf(x; 0') C
n
X
1
W
12
1(0
D
2
Thus Equation
11 consists of 3 different steps of relaxations. We analyze them one by one:
(A B) The equality holds when | J112 = 12llll/J and || ||/]|2 where af(Wx:0) 8(Wx)
The former equality requires that W and J have the same left singular vectors. The latter requires J to have zero singular values except for the largest singular value. Since J depends on the specific neural network architecture and training process, we test the tightness of this bound empirically (Figure H.
2. (B S. C) The equality requires
[W, to be the same for all i. In other words, the bound is xxlla tight when [[w,(x; does not vary too much from sample sample.
[xll
3. D) The equality holds the model is linear, i... O = W.
We empirically verify the tightness of the above bounds in Figure H..6
H.2 Correlation analysis
We empirically show how different metries correlate with each other, and how these correlations can be predicted from our bounds. We train 100 VGG-11 networks with different batch sizes, learning rates, and random initialization to classify images from the CIFAR-10 dataset, and plot pairwise scatter plots between different quantities at the end of the training: local dimensionality, sharpness (square root of Equation (3), )),,
log volune (Equation (7),, MLS (Equation (11b,, NMLS (Equation (12),, generalization gap (gen gap), D
(Equation (59)) bound (right- (right-hand side of Equation (12) and relative sharpness (Petzka et al.) 2021 (see
Figure H.7 We only include CIFAR-10 data with 2 labels ensure that the final training accuracy close to 100%.
We repeat the analysis MLPs and LeNets trained on the FashionMNIST dataset and the CIFAR-10 dataset (Figure H.8 and Figure H.9 We find that
The bound over NMLS, MLS, and NMLS introduced in Equation 12 and Equation correlates positively with the generalization gap.
consistently
Although the bound in Equation is loose, log volume correlates well with sharpness and MLS.
Sharpness is positively correlated with the generalization gap, indicating that little reparamctrization effect (Dinh al., 2017) happening during training, i... the network weights do not change too much during training. This is consistent. with observations in Ma & Ying (202))
30

---

Published in Transactions on Machine Learning Research (01/2026)
8
6
Bounds
Value
A
4
B
C
2
D
0
O 25
50 75
100
125
150
iteration (x5000)
Figure H.6: Empirical nisst of the bounds. We empirically verify that the inequalities in Equation 59 hold and test their tightness. The results are shown for fully connected feedforward network trained on the FashionMNIST dataset. The quantities A, B, C, and D are defined in Equation 59). We see that the gap between C and D is large compared to the gap between A and B or B and C. This indicates that partial sharpness V/ (x;; 0") 0")[|r (sensitivity of the loss w.r.t. only the input weights) is more indicative of the change in the maximum local sensitivity (A). Indeed, correlation analysis shows that bound C is positively correlated with MLS while bound D, perhaps surprisingly, is negatively correlated with MLS (Figure H.8
4. The bound derived in Equation 12 correlates positively with NMLS in all experiments.
MLS that only consider the first layer weights can sometimes negatively correlate with the bound derived in Equation 11] (Figure H.8
6. Relative flatness that only consider the last layer weights introduced in (Petzka et al. 2021 shows weak (even negative) correlation with the generalization gap. Note that "relative flatness' is a misnomer that easier understood as "relative sharpness', and is supposed to be positively correlated with the generalization gap.
31

---

Published in Transactions on Machine Learning Research (01/2026)
Local
-0.38
Sharpness
15
-0.69
Log volume
-12
-0.52
0.92
p= 0.83 0.83
1.0
MLS
-0.51
0.9%
0.84
0.97
NMLLS
1.13-013
0.48
0.29
0.40
0.44
gem gap
30
-0.37
p 0.99
p 0.78
0.92
p 0.98
D47
MLS bound
300
100
NMLS bound
-044
0,9%
6.76
p
2000
relative fathess
0.20
p-40,4
p.
0-
PF 0.10
8.= 0.17 0.17
-12.5-10.077.5 Log volume
5 10
100 200 300 MLS bound
1000
2000 NMLS bound
Local dim
Sharpness
MLS
NMLS
gen gap
relative flatness
Figure H.7: Pairwise correlation among different metrics. We trained 100 different VGG-11 networks on the CIFAR-10 dataset using vanilla SGD with different learning rates, batch sizes, and random initializations and plot pairwise scatter plots between different quantitics: local dimensionality, sharpness (square root of
Equation (3)), log volume (Equation (7), MLS (Equation (1I), NMLS (Equation (12),, generalization gap
(gen gap). MLS bound (Proposition 3.8). NMLS bound (Proposition 3.10) and relative sharpness ((Petzka
et al. 2021 The Pearson correlation coefficient p is shown in the top-left corner for each pair of quantities.
See Appendix H.2 for a summary of the findings in this figure.
32

---

Published in Transactions on Machine Learning Research (01/2026)
2.5
Local dim
2.0
1.5
0.65
Sharpness
26
9,52
Log volume
1.25
p
p
MLS
0.75
:0.62
20
-0.57
10.0
gen gap
9.5
0,72
0.62
0.60
0.60
066
00770
0.95
0.66
0.89
0.68
0.51
0.63
80
NMLS
flatnees
50
0:64
0.76
-0.55
-0.78
-028
20.18
0.80
00.1
20
2.5
15
20 Sharpness
-6
0.5
1.0
10
10
20 relative flatness
Local dim
Log volume
MLS NMLS
gen gap
MLS bound
NMLS bound
Figure H.8: Pairwise correlation among different metrics. We trained 100 different 4-layer MLPs on the FashionMNIST dataset using vanilla SGD with different learning rates, batch size, and random initializations and plot pairwise scatter plots between different quantitics: local dimensionality, sharpness (square root of Equation (3), log volume (Equation (7), MLS (Equation (II), NMLS (Equation (12),, generalization gap
(gen gap). MLS bound (Proposition 3.8). NMLS bound (Proposition 3.10) and relative sharpness ((Petzka
et al. 2021 The Pearson correlation coefficient p is shown in the top-left corner for each pair of quantities.
See Appendix H.2 for a summary of the findings in this figure.
33

---

Published in Transactions on Machine Learning Research (01/2026)
p 0.32
15
Log volume
-047
p 0.79
1.5
MLS
1.0
10
p
NALLS
12.5
p
p=
0.41
p
p
MLS hound
100
NMLS bound
400
p
300
200
relative flatnees
0.07
0.29
0.
0.28
p
20
10
10
-20
-10
10
15
100 MLS bound
200
400
20 relative flatness
Local dim
Sharpness
volume
MLS
NMLS
gen
NMLS
bound
Figure H.9: Pairwise correlation among different metrics. We trained 100 different LeNets on the CIFAR-10 dataset using vanilla SGD with different learning rates, batch size, and random initializations and plot pairwise scatter plots between different quantitics: local dimensionality, sharpness (square root of Equation (3)), log volume (Equation (7), MLS (Equation (1),, NMLS (Equation (12),, generalization gap
(gen gap). MLS bound (Proposition 3.8). NMLS bound (Proposition 3.10) and relative sharpness ((Petzka
et al. 2021 The Pearson correlation coefficient p is shown in the top-left corner for each pair of quantities.
See Appendix H.2 for a summary of the findings in this figure.
34

---

Published in Transactions on Machine Learning Research (01/2026)
Additional experiments
Test accuracy
NMLS
Sharpness
89
5
25
value
88
4
87
20
batch size
86
Log volume
MLS
Log local dimension
1.25
2.0
20
32
value
-3
1.00
Aw Mig yhn / Mo
1.5
-4
0.75
-5
0.50
1.0
50
100
150
100 iteration (x5000)
150
100 150
iteration (x5000)
iteration (x5000)
Figure L.10: Trends in key variables across SGD training of a 4-layer MLP with fixed learning rate (equal to 0.1) and varying batch size (8, 20, and 32). MLS/NMLS closely follows the trend of sharpness during the training. From left to right: test accuracy, NMLS, sharpness (square root of Equation (3)), log volumetric ratio (Equation MLS (Equation 11 and local dimensionality of the network output (Equation (15)).
Test accuracy
NMLS
Sharpness
89
30
Mil
value
88
ww Mmm
25
87
3
20
86
0.05
Log volume
MLS
Log local dimension
-2
2.0
0.1
0.2
-3
1.00
value
1.5
0.75
0.50
1.0
50
100
150
50
100
150
100
150
iteration (x5000)
iteration (x5000)
iteration (x5000)
Figure 1.11: Trends in key variables across SGD training of a 4-layer MLP with fixed batch size (equal to 20) and varying learning rates (0.05, 0.1 and 0.2). MLS/NMLS closely follows the trend of sharpness during the training. From left to right: test accuracy, NMLS, sharpness (square root of Equation (3p), log volumetric ratio (Equation 7 MLS (Equation 11 and local dinensionality of the network output (Equation (15)).
35

---

Published in Transactions on Machine Learning Research (01/2026)
J Sharpness and compression on test set data
Even though Equation 3 is exact for interpolation solutions only (i.c., those with zero loss), we found that the lest loss is small enough (Figure J.12) so that it should be a good approximation for test data as well. Therefore we analyzed our simulations to study trends in sharpness and volume for these held-out, test data as well (Figure J.12p. We observed that this sharpness increased rather than diminished as a result training. We hypothesized that sharpness could correlate with the difficulty of classifying testing points. This was supported by the fact. that the sharpness of misclassified test data was even greater than that of all test. data. Again
that MLS has the same trend the sharpness. Despite this increase in sharpness, the volume followed the same pattern as the training set. This suggests that compression in representation space is a robust phenomenon that can be driven by additional phenomena beyond sharpness. Nevertheless, the compression still is weaker for misclassified test samples that have higher sharpness than other test samples. Overall, these results emphasize an interesting distinction between how sharpness evolves for training vs. test. data.
Test loss
Sharpness
Log volume
MLS
Log local dimension
0.075
J0
0.050
samples test samples
0.2
Misclassified samples
0.0
200
iteration (x1000)
iteration (x1000)
Iteration (x1000)
iteration (x1000)
iteration (x1000)
Figure J.12: Trends in key variables across SGD training of the VGG-11 network with fixed learning rate (equal 0.1) and batch size (equal 20) for samples of the test set. After the loss minimized, we compute sharpness and volume the test set. Moreover, the same quantities test set or only samples that misclassified. In order from left to right in row-wise order: test loss, sharpness (Equation (2),, log volumetric ratio (Equation 77),, MLS, and local dimensionality of the network output (Equation 15
computed separately over the entire
K
Computational resources and code availability
All experiments can be run on one NVIDIA Quadro RTX 6000 GPU. The code is available at https:
whtpgg//ssapppeascaeeeeeee //gi //github. wtt.ccoo/ccccccc http://ccccccccaaaaaaaaaaaeeeeeo
36


Reviews:
[{"review_id": "hwYJsd9jUp", "reviewer": "Reviewer_8k2n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The current paper aims to build a connection between the sharpness quantity and quantities related to compression, both theoretically and \nempirically. \n\nIt is a bit surprising that all the theoretical results are not that technical, and only some basic calculus and norm inequalities are used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The connection between loss of flatness to compressed neural representations is explained using the bounds established in Section 3. I look over the experiments in Section 5, and I think the correlation is pretty clear for the specific model and data set. \n\nHowever, I am not sure whether the connection is actually \"SIMPLE.\" I will explain this in more detail later.", "audience_yn": "Yes", "audience_explanation": "The author considers LVR and MLR, which quantify compression, and shows they are correlated with sharpness, both theoretically and empirically. \n\nThough the bounds may not be that satisfactory, I would say the connection itself is interesting. \n\nThe author defers the discussion of the role of sharpness to generalization to the end; I do not think I really understand what the author is trying to say there. I will explain my confusion in more detail below. \n\nIn summary, I do not think the paper gives a theoretically founded and clear explanation of the relationship between sharpness and generalization. But raising the connection between sharpness and the two compression quantification measures is interesting.", "requested_changes": "**\"Simple\" Connection:** \n\nI am not sure whether the bounds given in Propositions 3.4, 3.6, 3.8, and 3.10 actually show a \"simple \" connection. At best, they are just simple bounds. For two networks with the same architecture and the same inputs, but with different weights, it is possible, based solely on the bounds, that the RHS is larger for one network, while the LHS is smaller for the same network. \n\nMoreover, even ignoring the previous issue, those prefactors in those bounds are not controlled in any way, particularly for those inputs to the intermediate layers, for which the author needs some lower bounds. Isn't it very possible that those prefactors are very large, making the bound totally useless? For two networks with the same architecture and the same inputs, but with different weights, even if their sharpnesses are the same, the bounds can still be different because of the prefactors. \n\nI think Reviewer sf7P has a similar concern. \n\nSo, I feel that the simple connection mentioned in the title is not well-grounded in terms of those theoretical bounds. At the very least, the author should provide more information about those bounds in their summary of contributions on page 2. And they shall explain a bit about the insufficiency of the bounds in a separate section, or a remark environment, or a paragraph starting with bolded font words indicating the insufficiency.\n\n**Compression**:\n\nA major contribution of the paper is the introduction of two compression measures: LVR and MLS. I am actually not entirely sure why these two quantities measure the compression. Can the author say a bit more? Also, should these quantities be less than 1 so that we know there is an actual \"compression\" going on? Do the bounds with sharpness (in Section 3) by any means show that those quantities can be less than 1, so some compression is going on? \n\n**Claims and explanation I do not understand** \n\nThere are a few places where I do not understand what the author means. I hope the author can spend some time explaining them to me: \n\n- On page 3, I do not understand the implication in the \"therefore\" in \"We note that for the cross-entropy loss function, the Hessian vanishes as the cross-entropy (CE) loss approaches 0 (Granziol, 2020; Wu et al., 2018a). Therefore, the sharpness of CE loss cannot differentiate between local minima with different traces of the Hessian.\" \n\n     To be more specific, I thought sharpness is the traces of the Hessian, so why can it not distinguish the local minima with different traces of the Hessian?\n\n- Continuing the same paragraph, in the end, it says \"1) MSE loss and CE loss share the same minimum, so Equation (3) will hold for networks trained with CE loss;\"  \n\n    I honestly don't understand this. First, why can MSE and CE loss have the same minimum? Is there a mathematical proof of this? Second, why can Equation (3) still hold for the CE loss? I thought (3) is derived using the particular form of MSE, and such a form is not present for CE. In particular, I thought the loss cannot be 0 for the CE loss.  \n\n    At the very least, what are \"our results\" referred to here? The bounds in Section 3? \n\n- In Section 5, the author says, \"Sharpness alone is only directly responsible for the robustness of representations, and only together with other conditions or implicit biases does sharpness lead to superior generalization.\" \n\n    I am not sure how compression relates to the robustness of representations. And I thought implicit biases seem to be too general. In the context of sharpness, I thought implicit biases meant biasing towards flat solutions. \n\n- In Section 5, the second paragraph, the author says, \"..., but Wortsman et al. (2022) shows that a large learning rate can severely hurt OOD generalization performance. More intuitively, consider a scenario where one provides a large language model (LLM) with a long text sequence and instructs it to find a specific piece of information, often called the \"needle in a haystack\" test. Even a slight alteration in the instructions (a tiny portion of the input) given to the model should lead to a notable difference in its output, depending on the desired information. Therefore, compression of network output is not a desirable\nproperty in this case.\" \n\n    Two things:\n    - First, what is OOD? \n\n    - Second, I do not understand the example here. I thought the network was compressed, but the output is not compressed, at least not directly. What do you mean by the compression of network output? It sounds like the example is basically saying the input-output relationship is very delicate, and we cannot use a compressed or simple network to model such a delicate relationship. \n\n- At the beginning of Section 3.5, the author said, \"(though extensions to network versions, that treat robustness to features that develop earlier in the network may be possible and are a focus of ongoing study).\" This sentence does not look correct. Please consider rewriting it. \n\n- I do not understand why the author says, \"with high probability\" before (13). I thought (13) is a deterministic result. Since you only need (14), can you give a more rigorous derivation of the last equality of (14)?\n\n**Other comments** \n\n- It is probably important to note that everything here has no RELU. \n\n- Please mention where the proof of Lemma 3.2 is. \n\n- In end of Page 6, the $C_{B ( \\bar x)}$ seems to be weird", "broader_impact_concerns": "I do not see issuses concerning ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "idoMq9BGYk", "reviewer": "Reviewer_KMtE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines the relationship between sharpness (loss of flatness around minima) and the compression of neural representations. The authors define/measure three local representation metrics—Local Volumetric Ratio (LVR), Maximum Local Sensitivity (MLS), and Network MLS (NMLS)—and prove upper bounds that tie these to sharpness (e.g., via Tr(H), Eq. (3), and Props. 3.4/3.8/3.10). Empirically, they report positive correlations between sharpness and compression metrics across MLP/CNN/ViT models, with some caveats and outliers, and discuss extensions to reparameterization-invariant sharpness.\n\n## Strengths:\nClear technical target: Connects parameter-space geometry (sharpness) to feature-space geometry (compression), which is interesting and timely.\nConcrete metrics: LVR/MLS/NMLS are well-defined and computable; NMLS is a sensible layer-aware refinement.\nTheoretical bounds: Inequalities that upper-bound compression by sharpness are simple and readable.\n\n## Weaknesses:\nClarity/flow is below bar. The abstract introduces “compression” and “reparameterization-invariant sharpness” with no setup or definitions, and the contribution bullets are stated as math without intuition or implications. The narrative arc is weak—there’s no clear thread connecting the problem, the proposed metrics, and the results. Core terms (“reparameterization,” “short description length models”) aren’t defined up front. Section 3—the main technical section—is poorly organized: definitions/lemmas/propositions arrive without motivation; Eq. (7) appears abruptly, later tied to Eq. (14), and its link to Def. 3.1 is unclear. The paper lacks an orienting figure to explain sharpness vs. flatness and how the metrics behave. In short, the exposition (almost the entire paper, especially the abstract and contribution list) needs a substantial rewrite to foreground definitions and implications.", "claims_and_evidence_yn": "No", "claims_explanation": "Primarily because the paper is not presented in a proper, comprehensive way, almost the entire paper requires a rewrite, specifically the abstract and introduction. The contribution list is math-only with no intuition, scope, or implications. Section 3 interleaves definitions/lemmas/propositions without motivation or stated assumptions. I can’t verify the claims even if they were true.", "audience_yn": "Yes", "audience_explanation": "The sharpness–compression connection is timely and relevant to TMLR readers (optimization, generalization, representation learning).\nNew metrics (LVR/MLS/NMLS) and a reparameterization-invariant angle could spark discussion, follow-up work, and tooling—even if the current exposition is below bar.", "requested_changes": "**A bstract rewrite:** Clearly state the problem first; define “compression” and “reparameterization-invariant sharpness” before using them; summarize contributions with intuition and scope (not just math).\n\n**Up-front definitions:** Define “reparameterization,” “short description length models,” and LVR/MLS/NMLS with a one-paragraph intuition each and where they are used.\n\n**Section 3 reorganization:** Provide a roadmap; motivate each definition/lemma/proposition; introduce Eq. (7) with derivation/context, make its link to Def. 3.1 and Eq. (14) explicit; state assumptions near each result.\n\n**Contributions list rewrite:** Replace math-only bullets with implications (“so what?”), scope, and limitations; make it accessible to a wider audience.\nFigures/exposition: Add an explanatory figure contrasting sharp vs. flat minima and showing how LVR/MLS/NMLS behave on a toy landscape; captions must state takeaways.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Min2TF44Ch", "reviewer": "Reviewer_sf7P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The role of sharpness in trained deep neural networks remains unclear in the existing literature. The authors of this work offer a different perspective on sharpness, arguing sharpness measures the degree of compression in the network's representation space. Specifically, this work argues if a deep network's parameters lie in a flatter minima, the representations are compressed to a higher degree. The authors study two compression metrics - local volumetric ratio (LVR) and (network) maximum local sensitivity ( (N)MLS ) - and derive upper bounds on the metrics that depend on sharpness. They include empirical results on a variety of architectures on image classification datasets. \n\n\n\n**Strengths:**\n1. Connecting the geometry of the loss landscape in the parameter space to the geometry of the features in the representation space is interesting, and can potentially provide deeper insights into the interpretability of trained networks.\n\n2. The empirical results on image classification datasets are convincing in showing a clear correlation between LVR / (N)MLS and sharpness.\n\n\n\n**Weaknesses / questions:**\n1. The derived bounds are architecture-independent, and are quite loose as a result, as seen in Figures 3 and H.5. \n\n2. The claim that the derived bounds are loss-agnostic does not seem entirely accurate (third paragraph in Section 2). The derivation of Eq. 3 depends on the use of the quadratic loss, and the proofs of Propositions 3.4, 3.6, 3.8, and 3.10 all use Eq. 3. Therefore, it seems to me the theoretical results depend on the quadratic loss. \n\n3. Propositions 3.4, 3.6, 3.8, and 3.10 respectively provide upper bound LVR, MLS, and NMLS that depend on sharpness. However, the bounds also depend on the norms of the weight matrices themselves, which clearly depends on the solution $\\boldsymbol{\\theta}^\\star$. Due to this dependence, it's not clear to me that a lower sharpness *always* implies more compression, as it's not obvious a decrease in sharpness from one interpolating solution to another should always \"counteract\" any potential increases in $||\\mathbf{W}||_2$ and/or $||\\mathbf{W}_l||_2$ due to the change in $\\boldsymbol{\\theta}^\\star$. Although this seems to be the case from the empirical results, could the authors comment on this point? \n\n4. In Section 3.5 after Eq. 16, the authors state “On the other hand, our theory then broadly applies to cases where the number of classes is much larger than the feature dimension, …, where generalized neural collapse can occur (Jiang et al., 2023).” I understand due to the dependence on $\\sigma_{min}(\\mathbf{W}_L)$, Eq. 16 provides a non-vacuous bound only when the number of classes is large, which is the setting where generalized neural collapse arises. However, I don’t quite understand how Eq. 16 relates to the actual phenomena of generalized neural collapse (i.e., the geometry of the last-layer classifier and the penultimate-layer features), so it is unclear to me why the authors discuss generalized neural collapse here. I would be grateful for clarification on this point of discussion.\n\n5. A common feature compression metric used in the neural collapse literature is the trace of the per-class and global covariances of the hidden-layer representations (see Eqs. 5-7 in [1], and also [2, 3, 4]), sometimes called *within-class variability*. I’d be curious to see the relationship between within-class variability and either LVR / (N)MLS, or sharpness. For example, is there any correlation between the within-class variabilities, and the corresponding “per-class LVR / (N)MLS,\" i.e., LVR / (N)MLS but only calculated for samples in each class (if this is a meaningful metric to begin with)?\n\n6. The authors focus on the trace of the Hessian at interpolating solutions as their sharpness metric, but also mention there are other measures of sharpness. Do you empirically observe the same correlation between LVR / (N)MLS and other notions of sharpness? For instance, are the predictions / logits of trained networks with lower LVR / (N)MLS more robust to noise injected into the parameters? \n\n\n\n\n[1] Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, & Qing Qu. “Understanding deep representation learning via layerwise feature compression and discrimination.” arXiv preprint arXiv:2311.02960, 2023.\n\n[2] Vignesh Kothapalli, Tom Tirer, and Joan Bruna. “A neural collapse perspective on feature evolution in graph neural networks.” arXiv preprint arXiv:2307.01951, 2023.\n\n[3] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. “Feature learning in deep classifiers through intermediate neural collapse.” In International Conference on Machine Learning, 2023.\n\n[4] Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. “Perturbation analysis of neural collapse.” In International Conference on Machine Learning, 2023", "claims_and_evidence_yn": "Yes", "claims_explanation": "One minor claim that does not appear accurate is that their theoretical results are loss-independent (third paragraph in Section 2); it seems they depend on the quadratic loss (see point 2 in weaknesses / questions above). However, all other claims appear accurate and are supported with clear evidence. The authors provide proofs in the appendix, which are fairly easy to follow and appear correct. The empirical results on different architectures and datasets show a clear correlation between LVR / (N)MLS and sharpness, which support the authors’ main claims.", "audience_yn": "Yes", "audience_explanation": "Drawing a connection between the sharpness of deep neural network parameters and the network’s resulting representation space is an interesting viewpoint, which can enhance network interpretability and inspire follow-up works to advance this perspective.", "requested_changes": "I recommend the following changes to strengthen the work, but they are not critical in my decision for recommending acceptance. \n \n1. Clarifying the discussion between the result in Eq. 16 and neural collapse. \n\n2. Additional experimental results on the relationship between feature compression metrics commonly studied in neural collapse literature, and either LVR / (N)MLS or sharpness. I believe this would also strengthen the connection between the authors' work and neural collapse as discussed in Section 3.5. \n\n3. Additional experimental results showing a correlation between (N)MLS / LVR and other sharpness metrics. \n\nSee points 4 - 6 in weaknesses / questions above for further elaboration.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)