Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods
Authors: Andres Fernandez
Frank Schneider
Maren Mahsereci
Philipp Hennig
Venue: Accepted by TMLR ()

Abstract: Recently, it has been observed that when training a deep neural net with SGD, the majority of the loss landscape's curvature quickly concentrates in a tiny *top* eigenspace of the loss Hessian, which remains largely stable thereafter.
Independently, it has been shown that successful magnitude pruning masks for deep neural nets emerge early in training and remain stable thereafter.
In this work, we study these two phenomena jointly and show that they are connected:
We develop a methodology to measure the similarity between arbitrary parameter masks and Hessian eigenspaces via Grassmannian metrics. We identify *overlap* as the most useful such metric due to its interpretability and stability.
To compute *overlap*, we develop a matrix-free algorithm based on sketched SVDs that allows us to compute over 1000 Hessian eigenpairs for nets with over 10M parameters --an unprecedented scale by several orders of magnitude.
Our experiments reveal an *overlap* between magnitude parameter masks and top Hessian eigenspaces consistently higher than chance-level, and that this effect gets accentuated for larger network sizes.
This result indicates that *top Hessian eigenvectors tend to be concentrated around larger parameters*, or equivalently, that *larger parameters tend to align with directions of larger loss curvature*.
Our work provides a methodology to approximate and analyze deep learning Hessians at scale, as well as a novel insight on the structure of their eigenspace

Published in Transactions on Machine Learning Research (04/2025)
Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods
Andres Fernandez Tübingen AT Center University of Tübingen
a.femandessumni-uubbnnnn..dd
Frank Schneider
Tübiingen AI Center University of Tübingen
Maren Mahsereci Yahoo Research
Philipp Hennig
Tübingen Al Center University of Tübingen
Reviewed OpenReview: https: openreview. net/ per//frmm? id=
Abstract
Recently, it has been observed that when training deep neural net with SGD, the majority of the loss landscape's curvature quickly concentrates in tiny top cigenspace of the loss Hessian, which remains largely stable thereafter. Independently, it has been shown that successful magnitude pruning masks for deep neural nets emerge early in training and remain stable thereafter. this work, we study these two phenomena jointly and show that they are connected: We develop a methodology to measure the similarity between arbitrary parameter masks and Hessian eigenspaces via Grassmannian metrics. We identify overlap the most useful such metric due to its interpretability and stability. To compute overlap, develop matrix-free algorithm based on sketched SVDs that allows us to compute over 1000 Hessian eigenpairs for nets with over 10M parameters an umprecedented scale by several orders of magnitudc. Our experiments reveal an overlap between magnitude parameter masks and top Hessian eigenspaces consistently higher than chance-level, and that this effect gets accentuated for larger network sizes. This result indicates that top Hessian eigenwectors tend to be concentrated around larger parameters, or equivalently, that larger parameters tend to align with directions of larger loss curcature. Our work provides methodology to approximate and analyze deep learning Hessians at scale, well as novel insight on the structure of their eigenspace.
1 Introduction
Deep learning (DL) benefits from overparametrization; but not all parameters are equally important. Often, substantial portion of parameters can be pruned, i.c. removed, without compromising the model's performance (see Blalock et al., 2020; Hoeller el al., 2021). One efficient and popular method to identify these subnetworks is via parameter magnitude (Han et al., 2015). Interestingly, these subnetworks materialize very early in training (Frankle Carbin, 2019), and once they emerge, their topology stops changing significantly (Achille al., 2019; You et al., 2020). In other words, competitive subnetworks crystallize early in training and remain slable thereafter (Section 2.2, Figures 15 and 16).

---

Published in Transactions on Machine Learning Research (04/2025)
MLP 16x16 MNIST
ResNet18 ImageNet
Exact Overlup
103
Step 2000 (Test)
0.15
Sketched Overlap
overlap baseelie
4000 (Test)
Step 8000 (Test)
overlap
102
0.10
10a
0.05
0.00
100
400 600
800
1000
100
200
300
400
500
Training step
Figure 1: Overlap between top-k parameter magnitude masks and top-k Hessian eigenspaces is consistently and substantially above random chance. (Left) Measurements for a 7030- 0000-parmmtter network trained on 16x 16 downsampled MNTST until convergence (see Section 6.1 and Figure 15): Exact overlap between top-k parameters and eigenvectors with k = 350 (-), approximate overlap via sketched eigendecomppositon (-), and chance-level baseline (--) (see Section 4.2). Note how overlap is larger than chance, and sketched overlap is a good approximation. (Right) Lines show the ratio sketched overlap chance-level baseline for model with > 11M parameters trained on IMAGENET (see Section 6.2), at three different points during training, and as a function of k. Note how overlap is always higher than baseline, up to a factor of 1000.
Concurrent research focuses on the loss Hcssian, charactorizing the loss landscape's curvature. Multiple studies found that empirically the Hessian spectrum separates into two parts: The bulk subspace, with many, near-zero eigenvalues, and the top subspace with a few eigenvalues of significantly larger magnitude (e.g. Dauphin et al., 2014; Sagun et al., 2018). Crucially, Gur-Ari et al. (2018) observed that, after initial training iterations, the gradient predominantly lies in this top subspace where it remains relatively stable throughout training. Analogous to the work on parameter masks, this indicates that the top Hessian eigenspace crystallizes early in training and tends to remain stable (Section 2.1, Figures 15 and 16).
In this work, we explore the connection between these largely independent lines of research- both reporting early crystallization of substructures. Our contributions include:
1. We establish a common theoretical ground for comparing binary parameter masks my (which select k parameters and discard the rest) and top-k eigenspaces of Hessian. Both can be cast as rank-k orthogonal matrices belonging to the same Stiefel manifold (Section 3). This allows a direct comparison of their spans using Grasmannian metrics' (see Section 4.1). We review popular Grassmannian metrics and identify overlop (Eq. (2)) as the most meaningful metric (Section 4.2).
2. To elficiently compule overlap, we develop SEIGH (Section 5 and Alg. 2), a matrix-free eigendecomposit based on sketched SVDs (Tropp et al., 2019). Our open source implementation' allows to compute top-k Hessian eigendecompositions for >> 103 on neural networks with over 10M parameters, an unprecedented scale by orders of magnitude.
3. We provide empirical evidence that the similarities between the spaces induced by parameter magnitude masks and top-k Hessian eigenspaces are consistently and substantially higher than random chance (Figure 1). This suggests that, in DL, top Hessian eigenvectors tend to be concentrated around larger parameters throughout the training process (Section 6).
Seminal work "Optimal Brain Damage (LeCun et al., 1989) established theoretical link between individual parameter walues and loss curvature, but only at convergence. Recent work often focuses on loss curvature for laser-wise parameter groups (e.g. Sankar et al., 2021), or involves just a few (A 10) Hessian
*Grassmonniaas are manifolds of low-dimensional subspaces of given vector space.
Phttps://githut com/ hamm/andres--ae fr/hessian, str/heasian_overlaap
2

---

Published in Transactions on Machine Learning Research (04/2025)
eigenpairs without emphasis parameters (e.g. Gur-Ari et al., 2018; Papyan, 2020; Dangel et al., 2022). In contrast, we connect Hessian eigenspaces to the spans of arbitrary parameter subsets, e.g. magnitude masks, throughout the entire training including the relevant early stage in an interpretable and scalable manner. We can thus jointly study the connection between these two phenomena and provide novel insights into a neural net's Hessian structure. This connection implies being able to approximate expensive Hessian quantities via cheap parameter inspection, which bears potential for downstream tasks where the Hessian plays & prominent role, such as optimization (Martens, 2016), pruning (LeCun et al., 1989), or uncertainty estimation (Kristiadi et al., 2021).
2 Hessians, Parameters and Early Crystallization
We consider a supervised classifcatton/reeressinn setup with train set Dusin = [(.,, ] of labeled data (In, e XxY, from an unknown data-generating distribution P. The neural net fo( : Y maps inputs x to predictions D via parameters OER.. A loss function E:YYY " - R20 penalizes differences between prediction ÿ and true label y. The goal is to minimize the inaccessible risk Lp(0) = {((ff( y)dP via the proxy empirical rish D (0) N Σn ((f((zn), Yn). For large N, we approximate .D...... (0) using mini-batches B Drrain of B < N samples. For J, l twice differentialle, we use the gradienl g(0)= == (oo( and the Hessian H())= = (OL(O)ERDXD) With g, H we refer to any loss gradient or Hessian. To emphasize the data domain, we use a subindex, c.g. Hg for the Hessian of the mini-batch loss.
2.1 The Hessian in Deep Learning
The Hessian plays a prominent role in DL applications. A useful characterization of H its cigendccomposition H -UUAU Ei=1 Au Here, U is orthogonal, with eigenveclors ui and A is diagonal and real-valued with (ordered) eigenvalues |i|| ... |Ap]- We call (Tk): :- {u) the lop-k eigenbasis of H, and span((7(0) the lop-k eigenspace. The top-k eigendecompooitionn (k) = Ei=1 Auuuu minimizes || H H for all
unitarily invariant norms (Golub & Van Loan, 2013, Th. 2.4.8).
Recent litcrature has extensively investigated the Hessian spectrum of neural nets, revealing that the eigenvalues are typically clustered into two parts: (1) The bulk of eigenvalues with near-zero magnitude and (2) few top eigenvalues with significantly larger magnitude (e.g. Sagun et al., 2018; Papyan, 2019). Thus, H can be well-approximated by its top eigenpairs. For the Hessian eigenspace, Li et al. (2018a) showed that projecting the whole space onto a few random, fixed dimensions still allows Stochastic Gradient Descent (SGD) to perform competitively-proviiii
enough dimensions are given-leadiing to the idea of an intrinsic dimensionality of problems. Gur-Ari et al. (2018) observed that this restriction fixed subspace seems happen spontancously anyway: After brief initial training period, the gradient predominately lies within small subspace spanned by the few top Hessian eigenvectors and this space changes little over the remaining training process. However, these phenomena might rely on optimizer and model choices (Li et al., 2018a; Ghorbani et al., 2019).
lower-dimensional,
One fundamental issue greatly limiting the use of H for DL is its prohibitively large size, with D2 entries. Consequently, most scalable methods are matria-free, relying on Hessian- Products (HVPs) to compute linear maps == He in just O(D) memory and time (Pearlmutter, 1994). Examples are the computation of spectral densities (Yao et al., 2020; Papyan, 2018) or top-k eigendecompositions (Gur-Ari et al., 2018; Dangel et al., 2022). To make Hessian properties more accessible, specialized DL libraries have been developed recently (e.g. Dangel et al., 2020; Yao et al., 2020; Elsayed & Mahmood, 2022; Dangel et al., 2025), but efficiently accessing large portions of the Hessian remains a major challenge (see Section 5).
2.2 Parameter Masks and Early Crystallization
Binary parameter masks my E BD, with B (0, 1}, consisting of k ones and the rest zeros, can be used to define subsets of parameters from a given model. A mask is k-sparse if it has k non-zero elements. We measure mask sparsity using the ratio p:= For non-binary vectors ve RD,, we instead mcasure whether a small subset of indices i contains a large proportion of the total norm of the vector. When i known, this can be directly expressed as the ratio: x(D)= := ||v.16/11=113 (see Hurley & Rickard, 2009). A popular choice
3

---

Published in Transactions on Machine Learning Research (04/2025)
for my is to take the k-largest parameters by magnitude: it is computationally cheap, and has been shown to be effective for neural net pruning (Han et al., 2015), leading to smaller models that can often achieve competitive performances with pE[1%, 10%] (e.g. Gale et al., 2019; Blalock et al., 2020; Hoefler et al., 2021).
In this work, we do not perform any pruning, but rather focus on a key feature of parameter magnitude masks. Not only can they lead to competitive performance when used for pruning and be found very early training (Frankle & Carbin, 2019; Frankle et al., 2020) but they also stabilize soon afterwards: You et al. (2020) compared Hamming distances between periodically extracted pruning masks and found they stop changing early in training, aligning with the loss of information plasticity reported in Achille et al. (2019). This early crystallization bears a striking parallel with observations made for the top Hessian subspace, and motivates our investigation: Are these two phenomena connected? How would one measure this connection? What would such a connection tell us about the parameters and the Hessian?
3 Parameter Masks as Orthogonal Projections
Quantifying the connection between parameter magnitude masks and top-k Hessian eigenspaces both exhibiting early crystallization that both (*k) (the top-k eigenbasis) and any k-sparse mask my can be characterized as elements of the same compact Stiefel manifold gDx*= (Q:QeRDxA, QTQ== R (Absil et al., 2004), where I E RR denotes the identity. Specifically for masks, we further consider the subset Da c Oxk, with columns having exactly one 1, each row at most one 1, and the rest is zcros.
way to relate mask to a subspace. In this section, observe
Reordering parameters: Recall from Section 2.1 that the Hessian eigenvalues are sorted by descending magnitude, exposing single cutting point between top and bulk eigenspace at dimension k. To simplify notation, impose a similar cutting point to the paramcters, by defining permutation matrix PeM for any given mask mg, such that the mask entries are grouped in selected (i.e. k entries with my m,=1) 1) and discarded (i.e. m; 0): i.c. m PT m= = (1,....1,, 1, 1,0, ...,).. 0). For a parameter magnitude mask, for example, we can define P to order the parameters by nonincreasing magnitude such that وj = |Pm] > Pm]lj We can now permute the parameters 0 PT0, as well as the Hessian rows and columns : == P HP= This has no loss of generality, since (m. 0, H) (m, 0, H) is an isomorphism, H and H are similar, and the loss curvature remains unaltered (0 H0 0 H0). Then, any permuted k-sparsc masking operation can be expressed as:
Ix p (m= e 0) = mg Ō (o 0) 0= I.
where ID - (1)) E ER which is clearly an element of the (binary) Stiefel manifold MD Also note that this holds for any P. not only for those defined for paramcter magnitude masks.
Partitioning H: Consider now the following partition of the reordered Hessian with V, DeRkkk W, Ee (DD-)) and V, WT E
V
W
D
V W
VT
H
(1)
W
E
W T
U-PTU
A
-T=UTP
With this partition, (+))= (V) is the top-k Hessian cigenbasis, and (W) the bulk cigenbasis. Conversely, the rows (viw) correspond to the selected parameters, and (V)W) to the discorded ones. Since Ü is orthogonal, we have (7()) e Dxk i.c. an element of the same Stiefel manifold.
4 Measuring Subspace Similarity via Grassmannian Metrics
We set out now to quantify the similarity between k-sparse parameter mask and the top-k Hessian eigenspace. Specifically, we are only interested in the similarity of their spans, since the spaces are the ones
3This can he generalized to masks and eigenspaces with different & using Schubert varieties (Ye & Lim, 2016).
4

---

Published in Transactions on Machine Learning Research (04/2025)
reported to undergo early crystallization. To connect parameter spaces to loss curvature, one may consider randomly perturbing the parameters, and empirically measuring the impact on the loss. This turns out to be problematic due to the non-PSD nature of H (see Appendix A.1). Various workarounds to obtain nonnegative measurements can lead to promising results, but they also entail tradcoffs (see Appendix A.2). Instead, Grassmannian metrics provide a natural and theoretically grounded way of achieving our goal, given that k-sparse parameter masks and the top-k Hessian eigenspace can both be cast as elements of the same Stiefel manifold (Section 3). We review popular Grassmannian metries in Section 4.1, and analyze them in more depth in Section 4.2, finding that the overlap metric, highlighted here, is both interpretable and stable:
1 1 1 cos (o) 1
overlap(Q =
Intuitively, we can see how a zero overlap between Q, and Qj indicates that the spans of both matrices are orthogonal to cach other, and an overlap of indicates that both spaces are identical. We can also interpret overlap as the oolationinvvriian cosine similarity between Q1 and Q2, as induced by the standard Euclidean metric matrices.
4.1 Grassmann Manifolds and their Metrics
Grassmann manifolds are extensively studied (e.g. Witten, 1988; Absil et al., 2004; Bendokat et al., 2020) and have recently been applied to DL (c.g. Gur-Ari et al., 2018; Zhang et al., 2018; Dangel et al., 2022). A Grassmann manifold GR.D is the set of all k-dimensional subspaces of given D-dimensional Euclidean space. Two orthogonal matrices Q1,Q ExD map to the same element ge g if and only if their column span is identical. Thus, the subset of all matrices in D that map to g. - span (apan(Q)) Q;) forms an equinalence class 5.. = (Q;: 2, Qu 2 - I&} (e.g. Edelman et al., 1998).
The geodesics (i.e. shortest paths) between two elements in GE,D are available in closed-forn form and can be characterized in terms of the principal angles oe [0, $J i.e. the "amount of rotation" required to transition from one space to another. The principal angles between two matrices in Pxk can be obtained via an SVD of their product, satisfying the invariance to changes within S.e (see Appendix A.3). Based on this, there are several Grassmannian metrics capturing different notions of distance between subspaces (e.g. Edelman et al., 1998; Qiu et al., 2005). In the following, we highlight a few. We abbreviate dist.(gi, 0z) = m(F:-)) as dist. - f(o), where+ here parametrizes any unitarily invariant norm:
a) Geodesic distance: This is the are length of the geodesic between the respective spaces in G, defined as dist. = ||a||2= [0, JVR].
b) Chordal norm: dist.... obtained by minimizing ||Q. -zzyl. over orthogonal matrices (Z1, Zz) (for that reason it is also called Hausdorff distance). The la and E norms admit a closed-form solution in terms of principal angles: diste.= sin (3or) blee e [o, v2) and dist ||2 |in (2a)lla= /22].
c) Projection norm: Also called the gap metric, it uses the unique orthogonal projector representation , Here, we also have closed-form expressions for the l and Ey norms: distp = [(sin(a)]laa E [o. 1] and distp.r = ||sin( 102lo)lll2200 [0, vR.
of a given subspace, i.e. V = Q.Q, as follows: distp. || 1e, -
d) Fubini-Study: This quantity is measure of the acule angle between both spaces, generalized to higher dimensions: dist, = arceos ( ([det(Q) Q,)I) arccos (II, cos( [ 3].
e) Overlap:
The overlop= = & L Q, 11:3 I0, 1] quantity was used in Gur-Ari et al. (2018) to measure subspace similarity. Tt. is not metric per se, since it is highest for equivalent subspaces and decreases with their distance, but it is bijection of dist,.r: as follows: Q11/= IIQT 1 dist?.
[|(cos(a)] 1- (c))s(r) 1-
While the above metrics apply to any pair of matrices from Dxt there are also relevant metrics specific to Dxc (boolcan subset of Dxt defined Scction 3), that can be characterized in a similar manner. Consider an arbitrary pair of k-sparse masks (m, m,), and their corresponding orthogonal projectors $ diag(m). Then we have:
5

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
1.0
Figure 2: Behaviour of different Grassmannian metrics for random pairs of matrices and masks. (Left) As a function of D, for different sparsity ratios p:= (Right) As a function of p, for different ambient dimensions D. Lines show the median metric for 50 random pairs, the (almost imperceptible) shaded regions span the 5-95 percentiles (see Fig- ure 11 for broader distributions). Each row shows selected Grassmannian metric (see Section 4.1 for more metrics); dist,zz is representative example of a collapsing metric, being & 0 almost everywhere. The overlop metric is non non-collapsing, and its expectation equals p.
0.5
0.5
0.0
0.0
1.0
1.0
dista
0.5
0.5
0.0
0.0
1.0
1.0
overlap
0.5
0.5
0.0
0.0
1000
2000
0.0
0.5
1.0
D
P
p= 0.01
p=
D
128
D
0.05
p= 0.4
1 256
1) m 1024
1) Intersection over Union (IoU): Typically used as an evaluation metric, it is defined as the relative
number of entries present in both masks, i.e. loU miomi m.vm E [0, 1J. Then we have m; = +1 1,
k- t-overlap, and if both masks are k-sparse, we also have m, U m = 2k - (m,)m,) = k(2 - overlap), yielding the bijection overlap= 21ot' 1410U"
ii) Hamming distance:
This quantity, defined as the minimum number of bit-flips needed to pass from one mask to another, was used in You et al. (2020) to measure distances between pruning masks. It is in fact a Grassmannian metric: dist, == ||vm - m = 1| -0- 22llž = dist [0, k], which means that the bijection overlap- 1- 1
g dist, also holds.
4.2 Comparing Grassmannian Metrics
Our aim is now to identify the most useful Grassmannian metric for comparing neural net parameter masks and Hessian eigenspaces. To empirically compare how different Grassmannian metrics change as function of ambient dimension D (for fixed p) and sparsity ratio p= (for fixed D), we conduct a synthetic experiment: We randomly draw matrices from Dak and masks from M and compute various Grassmannian metrics between them, normalized to be [], with higher metric indicating higher similarity. Results are highlighted in Figure 2 (experimental procodure and further experiments in Appendix A.4). This empirical analysis is complemented with theoretical result showing that the expected overlap equals exactly p, as seen in Figure 2 (proof in Appendix A.5). The main insights from this analysis (Appendix A.6 extends the discussion) are:
T. For a fixed sparsity ratio p. the expectation of all metrics becomes predictable as D increases: Already in the a 10x regime (much smaller than modern DL scenarios), expectations converge tightly to values that seem to only depend on p (Figure 2 left). We gathered empirical bascline values in Table 2.
11. Not all metrics are equally informative: Some metrics quickly collapse to 0 as D increases and p decreases (sec e.g. Figure 2 (top) or Figurcs 12 and 13). Looking at the metric definitions in Scction 4.1, we can see the reason: Collapsing metrics (e.g. distap, dist, and dista) are strongly influenced by individual angles, whereas non-collapsing metrics (e.g. disty, dist dist and overlap) average over all angles.
With these insights, we identify overlop as the most suitable metric among the reviewed ones, since: (1) It provides stable and efficient baseline: Among the non-collapsing metrics, it features the smoothest expeclation, equaling exactly p= k/D. This provides us with stable, simple and theoretically-grounde baseline, establishing "chance-level overlap" to compare against'. This is particularly beneficial for deep
II Appendix A.7 we also compare this analytical baseline the empirical alternative of sampling random masks, showing yields similar results.
6

---

Published in Transactions on Machine Learning Research (04/2025)
learning, bypassing the need to compute empirical baselines for large D over many p. (2) It is related to other melvics: overlop can be mapped to other popular metrics such as the Hamming distance, ToU, or dist via bijections (see Section 4.1). (3) It has precedence in the literature: The overlap metric has been used to mcasure eigenspace similarity in recent deep learning works (o.g. Gur-Ari et al., 2018; Dangel et al., 2022).
5 Sketched Hessian Eigendecompositions to Measure Overlap
To compute the overlap between parameter magnitude masks and top-k Hessian eigenspaces, we can now simply plug in the permutation from Section into Eq. (2): Qi becomes the permuted mask matrix ID and Q2 the correspondingly permuted top eigenspace (1)),, yielding:
(eerlap(ID,k. )=0)==
12 - k
= IVVE
(3)
i.e. our problem reduces to computing the Frobenius norm of V, a A k submatrix of (+) (Eq. (1)).
A case for eigendecompositions: We aim to estimate | where V is part of Ü a D x k orthogonal matrix spanning the top eigenspace. Since all standard algorithms to find such matrices (Gram-Schmine Householder transformation, Givens rotation) require to keep track of least k full vectors (Golub & Van Loan, 2013), memory requirement of O(kD) seems inescupable in general. Still, one may recognize that any orthogonal matrix Q with the same span (1) yields the same Frobenius norm (due to unitary invariance). However, we need to ensure that the span of said Q docs not overlap with the bulk cigenspace: It needs to be exclusively associated with the lop-k eigenvalues. As a consequence, knowledge of the eigenvalues also seems inescapable in general. In conclusion, to measure the top-k overlap, we argue that a rank-k k eigendecomposition is generally needed.
A case for sketched decompositions: Among the methods that satisfy the memory requirement of (kD), orthogonal iterations (Golub & Van Loan, 2013), an extension of Rayleigh's power method, is popular and effective one, requiring ((k) measurements per iteration and iterations, to converge at a rate proportional to / (Golub & Van Loan, 2013, Th. 7.3.1). This leads to large T for smoothly decaying spectra, likely scenario for H, thus O(Tk) Hessian-Vector Products (HVPs) can become infeasible. Ritz accelerations provide better convergence but still suffer from this issue (Golub & Van Loan, 2013, 10.1). Alternatively, Lanczos iterations convergence faster (Saad, 1980), (Golub & Van Loan, 2013, 10.1.6), and can be used to approximate eigenpairs via Ritz pairs (Golub & Van Loan, 2013, 8.1.4) as well as spectral densitics of c.g. largo-scale deep learning Hessians via. Stochastic Lanczos Quadrature (Papyan, 2018). The main issue here is that measurements must be done in sequential form, which can get very slow for H, and they involve matrix powers, numerically unstable for rank-defficient matrices like H. Sketched methods (e.g. Halko et al., 2011), based on random measurements, not only satisfy O(kD) memory, they also exhibit good convergence for (lk) measurements and are parallelizable and numerically stable (Halko et al., 2011, 1.4.2, 4.2, and 6.2). This, combined with the ability to perform matrix-free random measurements (Tropp et al., 2019, 3.2), makes it possible to compute Hessian eigendecompositioon at unprecedented scales, since the main bottleneck now the O(kD) memory requirement (Tropp et al., 2019, 7.1) (for example, storing 1000 eigenpairs for a Re (He et al., 2016) with 13M float parameters requires roughly 50GB of memory, same as 100 eigenpairs for a 10x larger model). Last but not least, affordable a posteriori methods allow to measure approximation error and rank of the recovered decomposition (Halko et al., 2011; Tropp et al., 2019).
The core idea behind the sketched SVD is that, given a linear operator Ae CPxDD of numerical rank k, any orghogn matrices Pe Dixoo Qe Daxno that approximately capture the column and row space of A, respectively, satisfy
AQQ and can be efficiently obtained from n, n,= O(k) random measurements followed by QR orthogonalization (Halko et al., 2011). Typically, n, min( DR), which results in core matrix C= PPPP*QQ Cuo that is much smaller than A and can be decomposed via classical methods, In summary, potentially large and matric-free linear operator A is decomposed into two thin matrices (P, P) and a small matrir C. A further development features an oversampled inner matrix My: Y AΩ2 Choo for random measurements L, of columns each (Boutsidis et al., 2016), where n, is typically slightly larger
We use lowcase n, indicate smaller dimension than D.
7

---

Published in Transactions on Machine Learning Research (04/2025)
than no- The SSVD from (Tropp et al., 2019, Sec. 2), gathered in Alg. 1, follows this idea:
A R AQ)Q* & P A0 Q (PU)
(4)
C-UEV* (svd)
This yields an SVD, since PU and QV are orthogonal. It requires n, + 2n, measurements see lines 1-3 in Alg. 1: outer measurements are used to produce P and Q, and inner measurements to produce C-, followed by thin matrix operations only. The memory cost is dominated by storing P and Q, i.e. O(no(D2+DR)) Arithmetic is dominated by the QR orthogonalizations needed to obtain P and Q, as well as the two least- squares problems needed to solve the pseudoinverses (see (Tropp et al., 2019)). Crucially, this approximation only requires a single pass over A, yiclds tight bounds (Tropp et al., 2019, Th. 5.1) and leads to superior performanice (Tropp et al., 2019, Sec. 7), due to its numerical stability, oversampled Mr, as well as uncorrelated measurements between P, Q and Mj (Halko et al., 2011, 5.5) (Tropp et al., 2019, 2.8.1).
Algorithm 1: SSVD (from Tropp al. (2019))
Algorithm 2: SEIGH
Input: n,EN
|/ No.
Input: n,E
of neasurements
Input: n, n
// No. outer measurenenta
Input: & n,
1/ Ko. of
Input: AEDDi
Input: AECDaD
// Linear operator
1/ Hernitian Linear operator
Input: TyE
// Left inner randon natrix
Input: TeC
// Left inner randon matrix
Input: YOE DD.oO
Input: S2 REECDM(
// Left outar randon natrix
1/ Right inner randon natrix
Input: 52 Daxmi
// Right inner randon natrix
Input: So Dxmo
1/ Right outer randon natrix
Input: NOE Dammo
// Right outer randon natrix
C UEC
H/ PeC QeC.xmi U
Output: (Q.U,A) with QUAU* ~ A
Output: (P,U,I, Q*) with PUEV*Q** A
Pertorn
[X, Y] natrix concatenation.
1/ Perforn outer neasurements
11 Bote the recycled No neasuremente.
M - A
11 C
1 Mo - AΩo
11 Dxx,
MRO - Allo
11 CPxme
2 M1- - T= ['A Mol // M, A(O, Rol 0
1/ Parform inner neasurenents
Orthogenalize outer ressurenents
M1 AtA
Q has orthonornal colunna least squares EIGH (Ū, A, V*) svd(Mr)
(Q, ) MM)
1/ Orthogoualize ize outer neseurements
Solve core
(Q. - -
(o)
// P has orthonormal colurns
// Q has orthonermal colurns 1/ Solve core natrix via least. squarea and SVD
5 C2 (T+Q)tÚ
6 CR - (S*
C - P
7 C - CLACE
11 C C is Hermitian
C [
(U, cigh(C)
// A the eigenvalues
(,, - SV)(C) return (P.U. (P. Q*)
// singular values
return A)
Leveraging Symmetry: While previous works studied Hermitian extensions (Halko et al., 2011; Clarkson & Woodruff, 2017; Tropp et al., 2017), none of them proposes single-pass sketched eigendecomposition via oversampled and uncorrelated inner matrix. In this work, we propose SEIGH (Alg. 2), a variant of SSVD that retains core oversampling while leveraging conjugate symmetry drastically reduce the required measurements and memory. Without loss of generality (Halko et al., 2011), assume Q P, only Q needs to be computed via AΩ outer measurements. Then, C is also Hermitian and can be eigendecomposed:
A ~ Q(Q* AQ)Q* Q( AJA [( -Q - (TTT
M,-UAV=
C-C+LUAU= (eigh)
Setting = would further simplify the structure, saving one pseudoinverse, but there is one caveat: We also wish to recycle part of the AΩ2 inner measurements to obtain Q, since this is a major bottleneck for the Hessian. But recycling mcasurements and sctting Y 22 would lead correlated measurements between Q and M, which, as mentioned, has negative impact in the quality of this procedure (Tropp et al., 2019, 2.8.1). We opt to recycle the measurements, while keeping the uncorrelated Y, Ω. and the two pseudoinverses.
8

---

Published in Transactions on Machine Learning Research (04/2025)
This way, only n, measurements are needed, memory is roughly halved, and arithmetic is reduced by one QR decomposition and increased by one inner SVD. With SEIGH, we are now able to approximate [| for large Hessians, leading to the sketched overlap approximation.
6 Magnitude Masks and Hessian Eigenspaces Overlap Substantially
We now investigate the similarity between the space spanned by the parameter magnitude masks and the top Hessian eigenspaces over the course of neural network training. We first study a small-scale toy problem (Section 6.1), where all involved quantities can still be computed exactly. This allows us to verify the existence of early crystallization, as well as the quality of our sketched overlap approximation. We observe an overlap consistently above random chance. We then move onto larger problems (up to D> 11M and k= 1500), where we also confirm that the similarity between parameter magnitude masks and top Hessian eigenspaces is orders of magnitude above random chance (Section 6.2), and that this effect is accentuated with network size. We report overlap results up to step 8000: since larger problems take longer to train, this covers different stages of training, but all of them reach well past initialization (see Table 1 for detailed accuracy at terminal steps). This provides us with a sufficient and convenient range to compare the observed high overlap across all problems.
6.1 Exact Computations on 16x 16 MNIST
Table 1: Overview of experimental settings, detailing number of model parameters (D), learning rate (n), batch size (B), steps per epoch (T), test accuracy (acc) at step I, number of train/test stest samples used to compute Htrain/ Heeet (Nerain/ Ntest respectively), and number of SEIGH outer measurements (no, see
Alg. 2)
Problem
Model
D
B
T
Nerwin/Nee /Nest
The
16x 16 MNIST
tanh MLP Martens & Grosse (2015)
7030
0.3
500
100
95.78%
(+= 1000)
500/500
355
CIFAR-10
3c3D-CNN Schneider et al. (2019)
895,210
0.0226
128
312
74.52%
(+- 8000)
500/500
1000
CIFAR-100
AL-CNN-C Springeuberg et al. (2015)
1,387,108
0,1658
256
156
40,50%
(1 8000)
n.a./1000
1000
ImageNet
RESNET-18 He et al. (2016)
11,689,512
0.1
150
8207
17.33% (1- 8000)
n.a./5000
1500
For our toy problem, we seek a setup that is overparametrized enough to exhibit mask crystallization, but small enough that full Hessian eigendecompositions can be computed. This is achieved by the setup from Martens & Grosse (2015), which is able to achieve zero training loss downsampled MNIST digit classification with only 7030 parameters. We train using SGD as detailed in Table 1, reaching a test accuracy of 95.78% after 10 epochs (Figure 15). To compute the training and Lest Hessians, we use a fixed set of 500 random samples (50 per class) from each respective data split. At every fifth training step t;, we compute the sparsity ratio k (defined in Scction 2.2) for the paramcters 0 and the Hossian spectrum A, observing that both subspaces experience early collapse (Figure 15). Then, for pairs of steps (tz.t,), we compute the IoU between successive parameter magnitude masks, well as the overlap between successive Hessian eigenspaces. We observe that both subspaces also experience early crystallization (Figure 16). We reiterate that we merely inspect the parameter magnitude masks, but we never apply them, i.e., we never prune the network. Once we established our desired scenario, we compute several Grassmannian metrics between the spans of parameter magnitude masks and top Hessian eigenspaces (Figure 17). All non-collapsing metrics report that both subspaces show a subslantial and consistent similarity. In particular, our sketched overlop approximation closely tracks the exact overlop (Figure 1, left), which is largest shortly after initialization and then decays, but still stabilizes well above random chance.
6.2 Sketched overlap on larger problems (CIFAR- 10, CIFAR-100 and ImageNet)
We aim now verify our discovery of substantial overlap in larger sctups. Using the DEEPOBS framework (Schneider et al., 2019) and SGD, we train larger models on CIFAR-10/100 Krizhevsky & Hinton (2009) and on IMAGENET Deng et. al. (2009) (see Table 1 for details). At steps (2000, 4000, 8000), we gather the
9

---

parameters and compute the top Hcssian cigenspaces using SEIGH, which allows us to compute the sketchod Published in Transactions on Machine Learning Research (04/2025)
CIEAR-10
CIFAR-100
ImageNet
10:2
10-3
10
Step: 2000
10
Sketched Overlop (Tes))
Skatched Overlp (Tmin)
10-5
10
10
10-3
Step: 4000
10
10-6
10-10
10-2
10"
10
Step: 8000
10"44
10-5
10-6
Figure 3: Overlap between parameter magnitude masks and top Hessian eigenspaces function of & at three different points during the training process. Shown is the sketched overlap between the mask of the largest k parameters by magnitude and the top-k Hessian eigenspace. Consistently across the training process and for all problems and values of k, we observe an overlap substantially larger than the random chance bascline (--) introduced in Section 4.2.
overlay for every & [l,...,no) mo). To compute the Hessians, we choose Ntrain/ / to have several (balanced) samples per class, and n, to be substantially larger than the number classes, which has been shown be linked to the top eigenspace dimensionality (e.g. Gur-Ari et al., 2018; Papyan, 2020). We elaborate on these choices and report runtimes in Appendix B.2. Note that training CIFAR-100 and IMAGENET relies noisy data augmentations, which lead to nondeterministic behaviour in Htrain. For this reason, compute their overlap for Hestt only. This choice is supported by the CIFAR-10 results, showing minimal difference in overlap for either Hirain or Hiest-
We observe that the measured overlap is signéficant for all problems, steps and dimensions (Figure and Figure left), confirming the findings from our 16 16 MNIST toy problem. Furthermore, this effect is IMAGENET
accentwated by model size, surpassing faclor of 10 limes the baseline for the RESNET-18 (Figure 4, right). The observed sketched overlap also raises in the very early steps, perhaps linked to the crystallization covered in Section 2. We also note that, while the precision of sketched decompositions is reported to decay sharply for eigenpairs close to n, (Tropp et al., 2019, 7.9), this does not affect our observation, since the obtained overlap is particularly high for the lower regimes of k, far from ng and where the method is most reliable.
7 Conclusion
We started with the observation that, at the early stages neural network training, both parameter magnitude masks and loss Hessian eigenspaces collapse and crystallize. To connect these two phenomena throughout the training process, we proposed a principled methodology that compares these two fundamental deep learning quantities using Grassmannian metrics, and identified overlap as a particularly advantageous one. We further developed SEIGH, matrix-free sketched eigendecomposition that works for Hessians of over 107 parameters, allowing us to approximate overlap. Our experiments reveal an overlap between parameter magnitude masks and top Hessian eigenspaces well above chance level for all observed problems, training steps and dimensionalities, indicating that, in DL, top Hessian eigenvectors lend to be concentrated around larger parameters, or equivalently, that large parameters tend to align with directions of large loss curvature.
While the obtained overlap may be considered small in absolute terms, the fact that it is orders of magnitude above chance could play crucial role at large scales, analogously to how mini-batch gradients produce
10

---

Published in Transactions on Machine Learning Research (04/2025)
Sketched Overlap
overlap baseline
ImageNet CIFAR- CIEAR-10
10-3
10a
10-4
10
10-5
10a
10a
10-4
10"5
10
10a
10"
10a
10a
10
0
2000
4000
6000
8000
0
200
400
800
800
1000
Step
k
Baseline)
Step
(Trin) (Trin)
50 Train)
& 50 (Tst)
50 (Random Baseline)
Step 2000 (Tes))
Step 4000 (Test)
Step 5000 (Test)
100 (Tmin)
100 (Tes)
100 (Random Baseline)
Figure 4: (Left) Overlap between parameter magnitude masks and top Hessian eigenspaces as a function training step. Shown is the sketched overlap between masks of k-largest parameters by magnitude and top-k Hessian cigenspaces, for different values of l different training steps. The observed overlap raises early on, and is consistently and substantially above random chance baseline (--, introduced in Section 4.2). (Right) Factor by which the measured sketched overlap is greater than the random baseline. For most of the measured training process and choices of k, the observed overlap is at least 10x larger than random chance baseline. This multiple factor over the random baseline is largest for small k and increases with network size, surpassing 103.
very noisy fluctuations in the short term, but lead to qualitative performance and generalization over larger training spans. The relevance of this connection could also bear potential for downstream tasks where the Hessian plays a prominent role (e.g. by approximating expensive Hessian quantities via cheap parameter inspection).
One main limitation of this method is that it still does not scale up to contemporary network sizes (sce Appendix B.2). In this sense, we emphasize that our methodology is primarily meant as an analysis technique to understand and leverage the connection between parameters and loss curvature, and not necessarily part of the downstream applications themselves. In Appendix A.2, we have also explored alternative approaches that show potential as more scalable alternatives to computing overlap.
Acknowledgments
The authors thank Joel A. Tropp for background on the distance between random subspaces. AF was supported by the DFG SPP 2298 (Project HE 7114/5-1). FS was supported by funds from the Cyber Valley Research Fund of the State of Baden- Württemberg,, Further, the authors gratefully acknowledge financial support by the European Rescarch Council through ERC CoG Action 101123955 ANUBIS; the DFG Cluster of Excellence "Machine Learning New Perspectives for Science', EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039A); and the Carl Zciss Foundation, (project "Certification and Foundations of Safe Machine Learning Systems in Healthcare") ) as well as funds from the Ministry of Science, Research and Arts of the State of Baden- Württemberg.
11

---

Published in Transactions on Machine Learning Research (04/2025)
References
P. A. Absil, Robert Mahony, and Rodolphe Sepulchre. Riemannian Geometry of Grassmann Manifolds with a View on Algorithmic Computation. Acta Applicandae Mathematicae, 2004.
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical Learning Periods in Deep Networks. In International Conference on Learning Representalions (ICLR), 2019.
Thomas Bendokat, Ralf Zümmermann, and Picrro-Antoine Absil. A Grassmann Manifold Handbook: Basic Geometry and Computational Aspects, 2020.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the State Neural Network Pruning? In Proceedings of Machine Learning and Systems (MLSys), 2020.
Christos Boutsidis, David P. Woodruff, and Peilin Zhong. Optimal Principal Component Analysis in Annual ACM Symposium on Theory
Distributed and Streaming Models. Proceedings of the of Computing, 2016.
Kenneth L. Clarkson and David P. Woodruff. Low-rank psd approximation in input-sparsity time, In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, 2017.
Felix Dangel, Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into Backprop. In
International Conference on Learning Representations (ICLR), 2020.
Felix Dangel, Lukas Tatzel, and Philipp Hennig. ViVIT: Curvature Access Through The Generalized Gauss-Newton's Low-Rank Structure. Transactions Machine Learning Research (TMLR), 2022.
Felix Dangel, Runa Eschenhagen, Weronika Ormanice, Andres Fernandez, Lukas Tatzel, and Agustimus Kristiadi. Position: Curvature matrices should be democratized via linear operators, 2025.
Yann N. Dauphin, Razvan Pascanu, Caglar Gulcchre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identilying and Attacking the Saddle Point Problem High--Dimensionl Non-Convex Optimization. Advances Neural Information Processing Systems (NeurIPS), 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Persi Diaconis and Mehrdad Shahshahani. The Subgroup Algorithm for Generating Uniform Random Variables. Probability in the Engineering and Informational Sciences, 1987.
Alan Edelman, Tomás A. Arias, and Steven T. Smith. The Geometry of Algorithms with Orthogonality Constraints. SIAM Journal on Matrix Analysis and Applications, 1998.
Mohamed Elsayed and A. Rupam Mahmood. HesScale: Scalable Computation of Hessian Diagonals, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Machine Learning (ICML), 2021.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations (ICLR), 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. In International Conference on Machine Learning (ICML), 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks, 2019.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. International Conference on Machine Leurning (ICML), 2019.
Gene Н. Golub and Charles F. Van Loan. Matris Computations. The Johns Hopkins University Press, 2013.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient Descent. Happens in Tiny Subspace, 2018.
12

---

Published in Transactions on Machine Learning Research (04/2025)
N. Halko, P. G. Martinsson, and J. A. Tropp. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions. STAM M Review, 2011.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both Weights and Connections for Efficient Neural Network. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 2021.
Niall Hurley and Scott Rickard. Comparing Mcasures of Sparsity. IEEE Transactions on Information Theory, 2009.
Agustimus Kristiadi, Matthias Hein, and Philipp Hennig. Learnable uncertainty under Laplace approximations. In Conference Uncerlainly in Arlificial Intelligence (UAT), 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage. In Advances in Neural Information Processing Systems (NeurIPS), 1989.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the Intrinsic Dimension of Objective Landscapes. In International Conference on Learning Representations (ICLR), 2018a.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018b.
James Martens. Deep learning via Hessian-free optimization. In International Conference on Machine Learning (ICML), 2010.
James Martens. Second-order Optimization for Neural Networks. PhD dissertation, University of Toronto, 2016.
James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-Factored Approximate Curvature. In International Conference on Machine Learning (ICML), 2015.
Yuri Neretin. On Jordan Angles and the Triangle Inequality in Grassmann Manifolds. Geometriae Dedicala, 2001.
Vardan Papyan. The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. arXiv: Learning, 2018.
Vardan Papyan. Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians. In International Conference on Machine Learning (ICML), 2019.
Vardan Papyan. Traces of Class/Cross-Clas Structure Pervade Deep Learning Spectra. Journal of Machine Learning Research, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Ginelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, peg-hperformnce deep learning library, 2019.
Barak A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Compulation, 1994.
Li Qiu, Y. X. Zhang, and Chi- Chi-Kwong Li. Unitarily Invariant Metrics on the Grassmann Space. SIAM Journal on Matric Analysis and Applications, 2005.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Yousef Saad. On the Rates of Convergence of the Lanczos and the Block-Lanczos Methods. SIAM Journal on Numerioal Analysis, 1980.
Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical Analysis of the Hessian of Aver-Parametrizet Neural Networks, 2018. ICLR Workshop.
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A Deeper Look at the Hessian Eigenspectrum Deep Neural Networks and its Applications to Regularization. Proceedings of the AAAI Conference on Artificial Intelligence 2021.
Frank Schneider, Lukas Balles, and Philipp Hennig, DeepOBS: A Deep Learning Optimizer Benchmark Suite. In International Conference on Learning Representations (ICLR), 2019.
Nicol N Schraudolph. Fast curvature malrix-vector products for second-order gradient descent. Newral Computation, 2002.
Jost T. Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLR Workshop, 2015.
Joel A. Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher. Fixed-rank approximation of a positive- semidefinite matrix from streaming data. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Joel A. Tropp, Alp Yurtsover, Madeleine Udell, and Volkan Cevher. Streaming low-rank matrix approximation with an application to scientific simulation. STAM Journal on Scienlific Compuling, 2019.
Edward Witten. Quantum field theory, Grassmannians, and algebraic curves. Communiicttions in Mathemalioal Physics, 1988.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. PyHessian: Neural Networks Through the Lens the Hessian. In IEEE International Conference on Big Data (Big Data), 2020.
Ke Ye and Lek-Heng Lim. Schubert Varieties and Distances between Subspaces of Different Dimensions. SIAM Journal on Matriz Analysis and Applications, 2016.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks. In International Conference on Learning Represendations (ICLR), 2020.
Jiayao Zhang, Guangxu Zhu, Robert W. Heath, and Kaibin Huang. Grassmannian Learning: Embedding Geometry Awareness in Shallow and Deep Learning, 2018.
A Supplementary Material for Grassmannian Metrics
A.1 Perturbing the Parameters to Measure their Importance
In order to prove that a certain subset parameters "sensitive" without using the thcoretical framework we propose, one may consider a perturbation study where large parameters are perturbed, and the resulting loss variation is measured. More variation would then relate to more sensitivity. Here we show that, in general, such an experiment does not yield an informative quantity towards this goal (unlike our proposed Grassmannians), due to the non-PSD nature of H, (i.c. negative and positive directions of curvature may cancel out for arbitrary sets of parameters) and the lack of details about the loss landscape regularity.
At any parameter vector OER and under additive perturbations d the loss can be expressed via its Taylor expansion (for gradient g, Hessian H and Lagrange remainder Rs):
L(O 8) L(O) g 8 + =6 H8 + Ra(
(5)
2
14

---

Published in Transactions on Machine Learning Research (04/2025)
Then, given parameter mask meB?, the proposed perturbation study would consist in drawing random perturbations & from any i.i.d. distribution N with zero mean / and unit variance, and then mask said perturbations via dm = (m e 8) and add them to 0 in order to estimate the expectation:
1 (H, EN [6m O + EN [Ra 66 2
Ev [ + ml = L( g Ex
This way, we are perturbing only the selected parameters in m, and mcasuring their impact on the (known) loss L(O): If [e] is larger, this can be interpreted as the subset & being more "sensitive".
Since we assumed um == En [ōm]] = 0 (otherwise reparametrize (0 + dm)= =: (0 + jmm + (8m pm) such that the reparamctrized mcan of the perturbations will be 0), and we also assumed (maskod) unit covariance Cov [6m] m: we have:
Cov [ov[6m] = Ev [[ōm (m) Mmm = EN 16 Im
Then, the expectation of the perturbed loss simplifies:
Ey [Z(0 5m]] I(0) + g E' [6m]]
H. H.N f6 mm EN (5))
0
Im
-()) + Ttm (H) + EN (R((mm)
We see now how, in general, this technique cannot be relied upon: since H is non-PSD in general, any of its subtraces could cancel Ry, leading to O. Therefore, this procedure is not guaranteed to be informative in measuring the impact of parameter subsets on the loss curvature. In contrast, our proposed method is guaranteed yield larger similarities whenever the selected parameters in m tend align with directions of larger loss curvature, as shown in the paper.
A Alternatively, one could consider to sample perturbations for parameters one by one, i.e. by setting m to equal 1 at exactly mi, and 0 otherwise. Then, the expectation of the perturbed loss becomes:
EN [T(O 5m)] I(O)) + Ha + EN [Ra (5m))
(6)
The idea here is that the aggregated perturbation e' = Eeled is now more likely to be informative, since positive and negative diagonal entries don't cancel anymore. This still entails a problematic trade-off that favours our Grassmannian characterization, for the following reasons: (1) In DL, the loss landscape may be far from convex or rogular (c.g. Li et al., 2018b, Fig. 1), and to interpret e' mcaningfully we would still require some characlerization of Rg similar in spirit to the theoretically founded Grasmannian that we use. (2) This method requires to estimate one expectation per scalar parameter. For a set of k parameters and n samples per expectation, this means performing O(n k) forward propagations over given dataset. In contrast, method is theoretically grounded and proposed sketched method requires just O(k) HVPs (see Sention 5 and Halko et al. (2011, 4.2) for more discussion).
A.2 Beyond Perturbing the Parameters Measure their Importance
Despite the various sources of error and computational issues related to parameter perturbations (see Appendix A.1), one may still feel compelled to explore related strategies, seeking more efficient and conceptually simple alternative to our proposed Grassmannian methodology. In this section, we introduce three such alternatives which, despite of some issues, constitute a promising line of investigation. We discuss here some of their main aspects in terms of model error, computation and empirical behaviour.
Nonnegative Causs-Newton Subtraces: A main issue with H is its non-PSD nature. Instead, the idea here is to target the Gauss-Newton matrix G, which is PSD approximation H (Schraudolph, 2002).
15

---

Published in Transactions on Machine Learning Research (04/2025)
Since PSD matrices have nonnegative diagonal entries, Gj could serve proxy measurement, denoting parameter sensitivity for 0z. In that case, larger trace for given subset of indices could mean that the corresponding parameter space is associated with larger loss curvature.
Instead casting this as a parameter perturbation study, which would be exposed to the R3 approximation error as well as the impossibility of disentangling G from H (see Appendix A.1), we propose here measure the diagonal entries of G one by one using Gauss-Neuton Vector Products (GNVPs). Since we are just the k largest parameters by magnitude, this just requires O(D) memory and (kk) GNVPs, thus being fully parallelizable and substantially faster than computing overlap. Furthermore, GNVPss are roughly twice as fast as HVPs, and require half the memory (Martens, 2010, 4.2).
interested in the entries corresponding
Once we have obtained the top-k diagonal entries (i.e. the diagonal entries of G corresponding to the k-largest parameters), we need a way to compare them with overlap. For this, we propose to compute the following normalized frace summation:
EK TT(G)
Σ (E((L..dE)
G
[0,1]
Where the i1,..,, indices run the paramcters by descending magnitude. We can see that & increases monotonically with k, reaching fD= 1. Furthermore, define the baseline corresponding to uniformly distributed trace as */D. This baseline behaves analogously to our "chance-level" overlap baseline from Section 4.2: if & > Ex, the top-k parameters are associated with an above-chance trace (which, as already mentioned, we treat as a proxy for curvature in the case G). See Figurcs 6 and 7 for an illustration.
Nonnegative Squared- -essiinn Subtraces: The rationale and computations involved here are identical as for the previously described G, but using H instead. This is viable, since 2² is also PSD and [H²] ] can be efficiently and exactly computed with single HVP as follows:
[H - H²e H
This technique is also substantially faster than overlap, requiring only O(D) memory and O(k) parallelizable HVPs. Although we do not assume H ~ H, this experiment aims to explore whether subtraces of H² can also be treated as proxy for curvature, in the same fashion as previously discussed for G.
Adversarial Perturbations: Recall from Appendix A.1 and Eq. (6) that a random perturbation study would require O(n k) forward propoagations, for potentially large Instead of random sampling, the idea here is to use single "optimal" perturbation, such that curvature is marimized in that direction. This is precisely the idea behind Sharpness-Awar with constrained radius À E R, such that the loss L(O + Xc*) is maximized, has the approximate closed-form solution c* = 9/llel1z. In our sctting from Eq. (6), we want to evaluate the following masked loss delta:
Minimization (SAM) (Foret et al., 2021), where the perturbation
Ax,= L(0) - L(0 + X(e, o et))
where the i index covers each of the top-k parameters by magnitude. This doesn't rely on PSD-ness anymore, since A comprises absolute values for individual parameters. It is also very fast to compute, since we just need O(D) memory, one forward- and backpropagation to obtain L(0) and plus one additional forward propagation per i. Once we have the deltas, we can define a normalized summation, analogous to Eq. (7), that can be used to compare against ε (see Figures 6 and 7):
SA.6 =:=
Σ Ari E [0,11]
(8)
x
(e(
16

---

Published in Transactions on Machine Learning Research (04/2025)
Experimental results: So far, we introduced three mcasurement techniques that are more efficient than overlap. We now ask: Can they be used as reliable prozy for parometer sensitivily? Do they yield measurements above baseline? How do they relate to overlap? To answer this, we gathered the above metries for steps 0, 100 and 1000 of the MNTST setup described in Section 6.1. The steps were chosen to cover initialization, mid-training and convergence, as illustrated in Figure 5. Results can be found in Fig- ures and 7, together with the corresponding overlap and uniform baselinc for comparison. We chose different values of to cover different perturbation scales.
Test
Accuracy
0.6
0.4
200
400
600
900
1000
Step
Figure 5: Training evolution of the model used
to report Figures 6, 7 and 14. It corresponds to the 7030-parameter model trained on 16 x 16 downsampled MNIST, described in Section 6.1.
Comparing both figures, we first observe that training and test data look quite similar throughout the whole training, which consistent with the small gap between training and test accuracies from Figure Crucially, we observe that, like overlap, all 3 prozy measurements tend to be well above chance-level baseline in most of their domain, supporting their potential connection to the Hessian eigenspace. This is promising, but it comes with the caveat of more unstable behaviour, both across k all proxies tend to be below baseline for small k- and across training slep they can go well above overlap early on, but then they steeply collapse towards baseline. The instability of these overlap proxics, together with their approximate nature, makes them a challenging but potentially rewarding target for future research.
scalable replacements for overlap, and suggesting a possible
Step:
Step: 100
Step: 1000
10-1
10-2
10
11
10-2
10
10
R:
Uniform Baseline
Overlap
Grrain SAM Pertur, (train, A=100)
112
train
SAM Pertur. (train, 0.1I)
SAM Pertur, (train, =0033)
Figure 6: Proxy overlap measurements, training split: (lop row) The normalized trace summation feature from Eq. (7), as well as the corresponding H² normalized trace summation. (bottom row) The SAM perturbation feature from Eq. (8), obtained for three different values of X, to cover different perturbation scales. (both rows) All features are compared to the uniform baseline k/D and the overlop associated to the k-largest parameters by magnitude. See Figure 5 for respective accuracies at given steps.
A.3 Main Properties of Grassmannian Metrics
A desirable property of Grassmann manifolds the availability closod-form expressions for their geodesics (i.c. the shortest paths between any two elements gi gy E 9) and metric functions. We can thus measure distances between subspaces in interpretable and computationally amenable manner: geodesies from gi to
17

---

Published in Transactions on Machine Learning Research (04/2025)
Step:
Step: 100
Step: 1000
10
10-2
10
10
10
10
10-3
10
50
150
250
300
100
200
250
335
50
150
250
300
A:
A:
Uniform Baseline
Overlap
Gtest
H
SAM Perturb, (test, À =0.1)
SAM Perturb, (test, =033)
SAM Perturb, (test, 1=1.0)
Figure 7: Proxy overlap measurements, test split: (top row) The G normalized trace summation feature from Eq. (7), well the corresponding H hormalized trace summation. (bottom row) The SAM perturbation feature from Eq. (8), obtained for three different values of X, to cover different perturbation scales. (both rous) All features are compared to the uniform baseline k/D and the overlop associated to the k-largest parameters by magnitude. See Figure 5 for respective accuracies at given steps.
9j follow circular trajectories, and therefore their distance can be interpreted as the "amount of rotation" needed to go from one space to another. Such rotations can be succinctly expressed in terms of principal angles E [0, 31 and they can be efficiently obtained from Q E O matrices via their Singular Value Decomposition (SVD) (e.g. Edelman et al., 1998, s. 4.3):
Q Qj =: Li- diag ( (cos(a (an(o))))) RT L, R orthogonal.
(9)
Since Q, have orthonormal columns, (o)( [0.1]* (e.g. Neretin, 2001), and more similar spans will yield larger singular values, which translate to smaller rotations. Importantly, singular values invariant. under similarity:
diag (cos(o,-)) L Q, = -T, Q (Q,
(10)
(11)
In other words, they are invariant under the action of any orthogonal Z, which means that ar does not change if we replace an input matrix with any other matrix from the same equivalence class (see definition for S Section 4.1). The family functions that satisfy this invariance, plus the axioms of metric spaces, form the family of Grassmannian metrics (e.g. Qiu et al., 2005), each capturing different notion of distance between subspaces (largest principal angle, sum of principal angles...). See Section 4.1 for more details on specific metrics.
A.4 Synthetic Experiment Grassmannian Metrics
Here provide details about the synthetic experiment discussed in Section 4.2. We compute the reviewed Grassmannian metrics a) to e) from Section 4.1 between randomly drawn matrices from Dxt and masks from Dx We use the subgroup algorithm (Diaconis & Shahshahani, 1987) to sample matrices uniformly from Duk and column permutations of ID to sample uniformly from M To delermine if the metrics behave differently for masks than for general orthogonal matrices, we inspect three different modalities; pairs of matrices (O-to-O, Figures and 9), masks (M-to-M, Figures 10 and 11), and matrix-mask pairs (O-to-M, Figures 12 and 13). We normalize all metrics, denoted by disty, to be [o, 1], with indicating highest similarity (i.e smallest distance). The overall procedure is gathered in Alg. 3.
18

---

Published in Transactions on Machine Learning Research (04/2025)
A Algorithm 3: Synthetic experiment on Grassmannian metrics (see Section 4.2 for details).
Input: [Di, D2....)
beight
Input: (2,,72....)
11 Kidth ratio (ry E [0, 1)
Input: ((0, 0), (O, (M)
Xodality
Input: (distg, distas, dist distps, distp, dista, overlap)
// Natric (normalizen ized)
Input: T
Nunber of randon capples I/ Besult dictionary)
Ø
for de (D1, D2. do
for r (ri, do k - max round(rrd)
for dist (distz, distez, distar, dist distp.7. dista, overlap) do
for (M1 ((0, 0), (0, M), (M) do
H -0 Ø
1/ Collection of camples
8
for (1....,TH T do
(2) "nd (M (M M2x H. H. u dist span(ppa)))
10
end
11
dd
// Gathor saxples into resuls
end
end
14
end
end
16 return R
We want to inspect how the distribution of the metrics changes function of sparsity and sparsity ratio R P= D. First, given fixed values of study the distribution of all Grassmannian metrics function of D (Figures 8, 10 and 12). Additionally, given fixed values of D we study how the Grassmannian metrics change as a function of (Figures 9, 11 and 13). We used the following values:
Number of random (matrix or mask) samples: T 50
For Figures 10 and 12 we investigate four different (fixed) ratios e (0.4, 0.2, 0.05, 0.01) at several incrcasing dimensions d E {16,32 32, 64, 128, 256, 512, 1024, 2048).
For Figures 9, 11 and 13 we investigated four (fixed) dimensions d {128, 256, 512, 1024) at several increasing ratios p e (0.005, 0.01, 0.05, 0.1, 0.33, 0.66, 0.9, 0.95, 0.99, 1).
Note how, in the second case, we concentrate the sparsity ratios around the extremes. This is to better capture the behaviour of collapsing metrics, as discussed in Section 4.2.
A.5 Expectation of overlap for Uniformly Random Matrices
In Section 4.2 we empirically observed that the expectation under uniformly distributed random matrices becomes predictable for all reviewed metrics, converging to a baseline value. Here we show that, for overlop, such expectation equals exactly p & This lemma depends on standard calculation that was communicated to by Joel A. Tropp.
Lemma A.1
Let Q1, Q2 be random matrices drown uniformly from the Stiefel monifold Dxk (Q:QER:: QQQ=++) Then,
k D
E (pa
(12)
Proof. We start by rewriting the definition of overlap, presented in Section 4.1, in terms of the trace of orthogonal projectors v =QQ E DxD We make use of the idempotence of V and the unitary invariance
19

---

Published in Transactions on Machine Learning Research (04/2025)
of the Frobenius norm:
1 overlap(span((u) (Qinssaaan(Qu,,ia span( Q2))
101 Q2lli +
22
Tr 1 v
(13)
1 Tr 1 k
(14)
We further observe that, if Q is drawn uniformly from OD*k, the marginal distribution of every column q is the uniform distribution over the Euclidean unit sphere, hence it is isotropic:
E [qq
D
(15)
Where In is the rank-D identity matrix. Then, leveraging linearity expectation, the orthogonal projector v QQ can be expressed as follows:
k E q:q. ] = D
E
V
Id
(16)
Now, given two independent realizations Q1, Q2, we form the associated orthogonal projectors 1 2. Write E1, l2 for the expectations of the respective distributions of Q1 and Q2. Then, leveraging independence of Qi and Q2, idempotence of v lincarity of expectation, and Tr( (TT) = k, we have:
k E1 Tr (T IDI])
22 D Th
E [T E, Th v
12
=
(17)
D
D
Replacing in the definition of overlop concludes the proof.
Table 2: Empirical baselines for Grassmannian metrics. Shown are the measured expectations (averaged over 50 samples at D=2048 2048) of different Grassmannian metrics between two uniformly random matrices in O for different values of p .. Note how overlap converges to exactly p (see Appendix A.5).
Metric
0.005
0.01
0.05
0.2
0.4
dista
0.03711
0.05191
0.11909
0.24534
0.36536
distay
0.00197
0.00161
0.00048
0.00046
0.00026
dist,,
0.02983
0.04202
0.09984
0.21754
0.33783
distpa
10-5
10-5
distpy
0.00248
0.00479
0.02513
0.17
0.22527
dista
0
pverlap
0.00495
0,00955
0.04962
U. 0.20022
0.39980
A.6 Extended Discussion on Grassmannian Metrics
We extend here the main insights presented in Section 4.2:
Mask-vs-maskk metrics have larger variance, and in some cases lower expectations: All distributions for the mask-vs-mask modality have higher variance compared to the other modalities (see Appendix A.4). Furthermore, the distributions for the disty and distar metrics seem to follow lower trajectories for the mask-vs-nn mask modality, compared to the other modalities. This is not the case for dist and overlop, whose expectation does not seem to be affected by the modality.
Extremal values of p lead to saturation, except for overlap: Most metrics exhibit nonlinear behaviour near the extremes (Figure 9). This particularly so for collapsing metrics, but saturation can also be observed in the non-collapsing ones. The only exception is overlap, whose expectation is linear as shown in Appendix A.5.
20

---

Published in Transactions on Machine Learning Research (04/2025)
dist
dist p2
dista
p 0.01
P 0.05
= 0.2
0.4
1.0
disty
dist o,
dist,. p.F
overlap
0.5
0.0
1000
2000
1000 2000
1000 2000
1000
2000
D
D
D
D
Figure 8: Grassmannian metrics for random pairs of matrices (Q1, Q2) "Ri (O 0D as a function of D. Each subplot shows different metric, and each color corresponds to different ratio For each (D, k), we sample 50 random pairs and report the median of the resulting distribution (line plot) well as the 5-95 percentiles (shaded regions).
D
distaz
dist
dista
D 128
D 256
D 512
D 1024
1.0
disty
dist
dist p.P
overlap
0.5
0.0
0.0
0.5
1.0 0.0
0.5 1.0 0.0
0.5 1.0 0.0
0.5
1.0
p
P
p
p
Figure Grassmannian metrics for random pairs of matrices (Q1, Q2) unff. (Q Dxk) as function of p 5. Each subplot shows different metric, and each color corresponds to different dimension D. For each (D, (D, sample 50 random pairs and report the median of the resulting distribution (line plot) as well as the 5-95 percentiles (shaded regions).
dist
distp
dista
0.01
P 0.05
P 0.2
0.4
1.0
disty
dist
dist FisF
overlap
0.5
0.0
0
1000
2000 0
1000 2000
1000 D
2000
1000
2000
D
D
D
Figure 10: Grassmannian metrics for random pairs of masks (Q), Q2) wgif. PM Dx as a function of D and for fixed p. Each subplot shows different Grassmannian metric with the different lines indicating four different ratios p. For each value of D, we report the median metric over 50 random pairs with the shaded regions showing the 5-95 percentiles.
21

---

Published in Transactions on Machine Learning Research (04/2025)
dist.p
dist
dist
D 128
D 256
D 512
D 1024
1.0
disty
dist o,
dist p.F
overlap
0.5
0.0
0.5
1.0 0.0
0.5 1.0 0.0
0.5 1.0
0.0
0.5
1.0
P
P
P
P
Grassmannian metrics for random pairs of masks (Q1, Q2) ugit (M MDkk as a function of p and for fixed D. Each subplot shows different Grassmannian metric with the different. lines indicating four different dimensions D. For each value of p, report the median metric over 50 random pairs with the shaded regions showing the 5-95 percentiles.
Figure 11:
dist
dist p2
dista
0.01
P 0.05
0.4
1.0
disty
dist
dist
overlap
0.5
0.0
1000
2000
1000
2000
1000
2000
1000
2000
D
D
D
D
Grassmannian metrics for random pairs of matrices and masks (Q1, Q2) ugi Dx MDxx) as function of D and for fixed p. Each subplot shows different Grassmannian metric with the different lines indicating four different ratios p. For each value of D, report the median metric over 50 random pairs with the shaded regions showing the 5-95 percentiles.
Figure 12:
dist
distp.2
dista
D 128
D 256
D 512
D 1024
1.0
disty
dist
dist pF
overlap
0.5
0.0
0.0
0.5
1.0 0.0
0.5 1.0 0.0
0.5
1.0 0.0
0.5
1.0
P
P
P
P
Grassmannian metrics for random pairs of matrices and masks (Q1, Q2) ugir (O MDxx) as a function of and for fixed D. Each subplot shows a different. Grassmannian metric with the different lines indicating four different dimensions D. For each value of p, we report the median metric over 50 random pairs with the shaded regions showing the 5-95 percentiles.
Figure 13:
22

---

Published in Transactions on Machine Learning Research (04/2025)
A.7 Analytical overlap Baseline vs. Sampling Random DL Masks
In Section 4.2 we justify the use of pt=
pt= R/D Section 6 to observe that the overlap between spaces spanned by magnitude pruning masks and top Hessian cigenspaces in DL is substantially large (see c.g. Figures 3 and 4).
analytical, "chance-level" baseline for overlop, that we use in
In this context, one natural question one may ask is: Do random DL masks also lead to higher-thane than-baselieeeeee overlaps? To answer this question, we trained the 16x 16 MNIST setup following Appendix A.2, and also for steps e {0, 100, 1000), we computed the overlap between the top-k Hessian eigenspace and 10 randomly chosen masks of & nonzeros (the steps were also chosen to cover initialization, mid-training and convergence, as illustrated in Figure 5). Results are shown in Figure 14: We observe that, indeed, random DL masks tend to behave like our analytical baseline, which supports the use of the analytical baseline in our experiments.
Step:
Step: 100
Step: 1000
overlap
Heain
10
10 3
10
10-1
overlap Hrest
10
10-3
10
100
200
100
200
300
100
200
300
k
k
Best magnitude mask
Param, magnitude mask
Sampled random masks
Analytical baseline
Figure 14: overlap measurements for different masking criteria against H (top row) and Hiest (bottom row). Best magnitude mask represents the overlap obtained we pick the k paramoters with largest. overlap with the lop-k eigenspace. Param. magnilude mask corresponds the k-largest parameters by magnitude. The random mask grey area covers one standard deviation for the overlap between H and 10 randomly sampled masks. See Figure for respoctive accuracics at given steps.
Another natural follow-up question is: How special are the parameter magnitude masks? Would some other mon-random mask also achieve high overlap? To tackle this question, also plot in Figure 14 the overlop for the top-k parameter magnitude masks, and the maximal overlap achievable by the best possible k-mask, serving as upper bound. We observe that, while the magnitude pruning mask is substantially larger thon baseline, it is still far from optimal.
Still, there limited number of masks that can yield high overlop: Due to the orthogonality of the Hessian eigenbasis, each column of (k()) must have l2 norm of 1. This means that if given set of parameters has overlap of a, the overlap for any non-intersecting set of parameters is at most - Intuitively, the overlap is limited resource, and therefore, masks with an overlap significantly above chance-level are bound to be scarce.
B Supplementary Material for the Experiments in Section 6
B.1 Results for 16x x 16 MNIST
This section includes supplementary material as part of the exhaustive experiments done for the MNIST toy problem. Figure 15 and Figure 16 verify the existence of early collapse and stabilization in both Hessian top space and parameter masks. They also show that SGD able to successfully train the model under this
23

---

Published in Transactions on Machine Learning Research (04/2025)
setting. Figure 17 corroborates that the behaviour of the Grassmannian metrics analyzed in Section 4 is also present for DI. problems.
1.0
Accuracy
Train
0.5
Validation
Test
1.0
Ratio=0.2
x(
Ratio=0.05
0.5
Ratio=0.01
Ratio=0.005
0.0
1.0
K 6(Atrann
Ratio=0.2
Ratio=0.05
0.5
Ratio=0.01
Ra00o=0.005
0.0
1.0
K (est))
Ratio=0.2
0.9
Ratio=0.05
Ratio=0.01
0.8
Ratio=
250
500
750
1000
1250
1500
1750
2000
Training step
Figure 15: Both the parameters of the neural network as well as their Hessian spectrum collapse early during training. (top) The top two subplots show the mini-batch loss LB as well as the train/vall tetn/valiaaiiontteee accuracy the 70r0-parrmeter model trained on 16x downsampled MNIST (see Section 6.1). (middle) Looking at the top 20%, 5%, 1%, 0.5% parameters by magnitude, we can see that very carly during training, most of the energy concentrated on a small subset of the paramcters (scc Section 2.2 for definition of k). For example, shortly after initialization, the top 0.5% largest, parameters by magnitude have roughly 1/4 of the total & norm of all parameters. (bottom) We behaviour for the Hessian spectrum on both the training set (fourth subplot) and the test set (fifth subplot). Only few steps after training, most of the energy is concentrated in only 0.5% of the cigenvalues.
observe similar
24

---

Published in Transactions on Machine Learning Research (04/2025)
Sparsity Ratio 0.005
Sparsity Ratio 0.01
Sparsity Ratio = 0.05
Sparsity Ratio 0.2
Parameter Mask IoU
0.8
1.0
overlap (roii)
50
0.8
100
0.6
1.50
0.4
200
0.2
250
0.0
1.0
overlap Overlap(Heaal (Hreat
30
100
0.8
150
0.4
200
0.2
250
0.0
Figure 16: Both parameter pruning masks (top row) and Hessians (bottom two rows) remain relatively stable after an initial training phase. Following You et al. (2020), the depicted matrices represent all pairwise similarities (i.e. higher is better) from the beginning of training (top left corners) until step 250 (bottom right corners), for the 7030-parameter model trained (see Section 6.1). For this reason, all matrices are symmetric and have unit diagonals. (top) Even when selecting only 0.5% of the parameters (left column), masks collected at different training steps show a remarkable similarity after an initial phase of training, (middle and bottom) The overlap metric for top Hessian eigenspaces on the train (middle) and test set (bottom) extracted at different training steps and for different subspace sizes (columns). Starting from initialization, the Hessian eigenspaces do not change significantly over the course of training. No substantial differences in behaviour between Hiest and train be observed.
16 16 downsampled MNIST
25

---

Published in Transactions on Machine Learning Research (04/2025)
Batch Loss
distg
0.50
0.25
0.00
1.00
dist.
0.75
0.50
dist..
0.50
0.25
0.00
1.00
0.75
dist
0.50
0.25
dist
0.50
0.25
0.00
1.00
0.75
dista
0.05
0.50
0.005
0.25
1.00
overlap
0.25
0.00
600
Training Iteration
Figure 17: Different Grassmannian metrics between pruning masks and top Hessian subspaces. Each line shows particular sparsity level i.e.. the ratio of unpruned parameters or the size of the top Hessian subspace relative to the full Hessian, as a function of training progress for the 7030-parameter model trained on 16 x 16 downsampled MNIST (see Section 6.1). All non-collapsing metrics reveal a consistent and substantial similarity between spaces spanned by pruning masks and top Hessian subspaces well above random chance (random baselines gathered from our synthetic experiments
shown in dashed lines for p 0.2), while collapsing metrics effectively due to the large D. For better visibility, limit the plot to the first 1000 steps but all metrics remain stable afterwards.
26

---

Published in Transactions on Machine Learning Research (04/2025)
B.2 Discussion Computational Resources
We now discuss the resources involved in the computation of the Hessian sigendeecomposiiions, well as the experimental design choices surrounding them. The goal is to reflect the scale of computations involved in our experiments, and to convey an idea of the overall resources needed to reproduce them. For more rigorous asymptotics involving memory, arithmetic and measurement costs, readers are invited to review the references exhaustively provided in Section 5, particularly (Tropp al., 2019, Sec. 4).
10a
8760
9586
Time (min)
10a
10a
88
100
97
126
174
24
13
10a
3
2.5
1.5
10a
layout
HVPs
morge
lstsq
TOTAL
overhead
CIFAR-I0
CIFAR-100
ImageNet
Figure 18: Runtimes of main SEIGH operations to compute a single Hessian eigendecomposition, assuming a single computer with 400GB RAM equipped with an NVIDIA A100 (40GB) graphics card. Note that the three involved eigendecompositions feature different networks, datasets, and rank, so a blind one-to- one comparison of runtimes is not possible: refer to Appendix B.2 for an interpretation of this figure.
Figure 18 gathers the average times it took to compute a single sketched Eigendecomposition, for cach one of the Hessians covered in Section 6.2. Specifically:
Resources needed to tune or train the model are here ignored. We used PyTorch Paszke et al. (2019).
The layoul step allocates the O(Dk) storage memory the form of virtual HDF5 datasets This allows efficiently write numerical data in parallel, which is instrumental for the scalability of the procodure.
The HVPs, computed with the help of CurvLinOps', correspond to the measurements in lines 1-2 of Alg. 2. As we can see, they take the bulk of the runtime: each HVP requires two forward and backward passes over the Hessian dataset (consisting of 500, 1000 and 5000 samples for the three respective problems, see Ntrain/Naa
Table 1). While this can be done batched fashion, the HVP runtime is linearly affected by dataset size. Another linear factor is the number of measurements (see n, in Table 1). Crucially, those can be fully parallelized, column can be divided by the number of available machines. This property of sketched methods, combined with the HDF5 technology, is the main enabler for scalabiltiy here.
the runtimes reported under the HVPs
The merge step stems from a technicality: most operative systems don't allow a process to open thousands files at the same time. Hence, we need to merge the virtual HDF5 dataset into a monolithic file. This can be done with only O(D) memory overhead.
Although (Tropp et al., 2019, Sec. 4) points out that the QR decomposition (line 3 of Alg, 2) dominates the (asymptotic) arithmetic cost for the whole procedure, interestingly this a relatively fast step. We also note that the main reason why we needed so much RAM memory was the need to load O(Dk) entries in in-memory to perform this step. Out-of-core QR orthogonalizations would greatly help reducing RAM requirements.
The remaining operations (lines 4-8 in Alg. 2) are here gathered under the lstsg label, which takes considerable portion of the overall runtime. To improve this, lines 5 and 6 could be parallelized, roughly halving the runtime, but still faster solutions would greatly help towards more scalable Hessian eigendeccomosiiions..
phttp:://gttabb..cm////rrrrr
27

---

Published in Transactions on Machine Learning Research (04/2025)
Figure 18 also reports the TOTAL runtime of the steps purely involved with the cigendecomposition. Further overheads, related e.g. to the use of computational cluster, are reported separately. In practice, computing e.g. single RESNET-18 Hessian eigendecomposition with up to fifteen A100s took a bit over a day. While this may seem like a hcavyweight computation, it is still a substantial improvement, enabling previously intraciable computations. In this section we have also outlined few ways which this could be further improved.
Due to the scale of the experiments, had to be conservative with the choice of hyperparameters. We chose dataset sizes Ntrain/NNest such that all classes the chosen number of measurements n, is much smaller than the ambient dimension D, the tremendously rank-defficient structure of H (see Section 2.1) allows for such a disparity. Still, it has been noted that the mumerical rank of H may be linked to the number of classes (eg Gur-Ari et al., 2018; Papyan, 2020; Dangel et al., 2022). For this reason we chose no values well above this limit.
balanced, and each class has least 5 examples. While
28


Reviews:
[{"review_id": "PwaZvfrX5t", "reviewer": "Reviewer_iShm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is very well written\n- Experiments are clean and scale well up to Image-net \n- Overall, I think it is a nice contribution supporting the alignment between the surviving parameters of the deep net with the top eigenspace of the Hessian, not only at convergence but early in training \n- The alignment is much more significant for larger networks, suggesting that this effect could be even provable for some infinite-width/depth architectures \n\n**Weaknesses**\n\n*Not really weaknesses two questions:*\n\n- Fig 13 shows both the largest magnitude parameters and the top eigenspace stabilizes early in training. I wonder how much stronger (if it is stronger) this self-stabilization effect is compared to the alignment between the largest magnitude param. space and the top eigenspace (?) A future investigation could be if the stabilization of the largest param. subspaces imply the stabilization of the top Hessian eigenspace.\n\n- I wonder if the authors tried this to measure the alignment $\\frac{1}{k} \\max_P \\text{Tr} (I^T P S )$ where $P$ is a $k \\times k$ permutation matrix. This would first match $k$ (orthonormal) vectors to other $k$ orthonormal vectors and then sum up over $k$ angles between the matched pairs. It is smaller than the overlap selected in the paper as it removed the angles between the vectors that are not closest to each other. I'm not sure how computationally expensive finding $P$ would be but if it is not expensive this could be a more stable measure (?)\n\n\n-------------------------------------------------------------------------------------------------------------\nPS. I apogize for the delay in review. I got stuck outside of my residence country due to visa issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on the observed connection between the sparsification of the deep network early in training and the stabilization of the top eigenspace of the Hessian. The methodology is to measure the alignment of the top Hessian subspace to the coordinates of the biggest magnitude parameters using $ \\frac{1}{k} || I^T S ||_F^2$  where $S$ is the top subspace, that is the so-called (k) overlap. \nA sketched SVD method is proposed and validated to compute $S$ which scales well even for Image-net scale architectures. The measured overlap is above the chance level supporting the observed connection between sparsification and eigenspace stabilization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oFN3FrOURC", "reviewer": "Reviewer_bbvJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is the clarity of the question and the approach used to answer it. Linking together sparsification procedures with the geometry of the loss landscape is a highly relevant topic. The sketching approach is a great fit here, and the descriptions of the overlap metric and its connection to sketching are also highly informative.\n\nMain weaknesses are due to presentation and diversity of model/dataset pairs; see **requested changes** below. An additional experiment using Gauss-Newton or squared Hessian metrics is necessary to secure my acceptance; I believe this is a simple experiment to run given the author's setup (requires a modest number of HVPs at checkpoints studied by the authors). Once this experiment is added I will likely switch my claims and evidence to \"yes\".\n\nEdit: Claims and evidence switched to \"yes\" after authors made changes to the text, including adding the requested experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work the authors explore the connection between large parameter magnitudes, which are used for sparsity masking, and the structure of the Hessian. They use a sketching method to show that indeed there is high overlap between parameters with large magnitudes and the large eigenspaces of the Hessian in a variety of models. This suggests a strong link between popular sparsification methods and loss landscape geometry.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is argued in Appendix A.1 that a perturbation approach would fail due to the non-PSD nature of the Hessian. There are a couple of paths around this: first is to use the Gauss-Newton component of the Hessian. This is PSD, contains most of the information about large eigenspaces, and GN-VP are generally easy to compute. Another alternative are metrics related to the squared Hessian, which are also obtainable by HVPs. Note that there are also perturbation-based approaches to obtaining this information without HVPs; see the relationship between Equations 2, 3, and 7 in [1]. By replacing $\\epsilon$ in this paper with the desired perturbation (e.g. a onehot vector representing perturbation of a single large parameter), I believe interesting statistics can be recovered.\n\n**I would like to see some experimental results on at least one of the above simpler metrics before acceptance; I do believe that the authors' methodology is sound, but this would be a good tie-in to simpler approaches.**\n\nFigure 2, top row is a bit hard to parse - maybe log y axis there?\n\nThe authors invoke the geometry of Grassmanians to define overlap. Overlap can also be motivated more simply via cosine similarity induced by the standard Euclidean metric on matrices, and the observation that the metric is rotationally invariant. Perhaps this can be used as the initial intuition building motivation, and then the Grassmanian formulation presented next in order to solidify the foundation believe this will improve digestibility of the paper by the ML community, as a broad swath of researchers will find this work interesting.\n\nFor section 4.2: might be better to start with the statement that overlap is the focus of the rest of the paper, and then go into the explanations for why? I also think that the expressions for the other metrics should be in the main text (or, at the very least, the specific equation numbers from the appendix should be referenced).\n\nIn Section 5 the authors criticize Lanczos methods for not generating eigenpairs; the approximate eigenvectors generated by the lanczos process can indeed be used to estimate eigenvectors for large eigenvalues in particular. In my opinion this is an important point.\n\nIn the presentation of the sketching algorithms, for Algorithm 2, what are the subscripts on $\\Omega$ in steps 1 and 2? It would also be good to indicate the dimensions of the output matrices as well. Additionally, indicating which dimensions are ``big'' vs small will be helpful. I also wonder if in the main text description there is a way to make more clear which quantities are obtained using HVPs strung together. May also be worth mentioning that a $V$-vp is implemented by composing vector-mask product with HVP (if I understand correctly!).\n\nCould be nice to add something like ViT to the experiments -  I think a S/16 or Ti/16 sized model should be pretty easy to run under these methods even with modest compute resources. **This experiment is not necessary to secure publication recommendation**.\n\n[1] https://arxiv.org/abs/2502.02407", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wgjE1PmABM", "reviewer": "Reviewer_MZ4v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Establishing a direct link between parameter magnitudes and Hessian eigenspaces provides a novel perspective on neural network structure and optimization.\n- The development of a sketched eigendecomposition algorithm addresses the computational challenges of analyzing large-scale Hessians, enabling experiments with models containing over 10 million parameters.\n- The consistent and substantial overlap observed across various models and datasets reinforces the proposed connection, with effects becoming more pronounced for larger networks.\n- The findings are applicable to several areas, including pruning, optimization, and understanding loss landscapes, making the contributions broadly impactful.\n\n\nWeaknesses:\n\n- The analysis primarily emphasizes early crystallization and does not explore whether these overlaps persist or evolve in later training stages.\n- Despite scalability improvements, the resource requirements for large-scale experiments seem to be substantial, potentially limiting adoption.\n- While the theoretical insights are significant, the paper does not directly demonstrate how these findings could be applied to improve training efficiency or model performance in real-world settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the relationship between parameter magnitude masks and top Hessian eigenspaces in deep neural networks, both of which exhibit early crystallization during training. The authors introduce a methodology to quantify the similarity between these two structures using Grassmannian metrics, identifying overlap as the most interpretable metric. They further propose a scalable, matrix-free eigendecomposition algorithm, enabling analysis of Hessians at unprecedented scales. Empirical results demonstrate a significant overlap between large parameter magnitudes and top Hessian eigenspaces, highlighting their structural connection. This work has implications for optimization, pruning, and uncertainty estimation in deep learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To improve practical relevance, it would be helpful to provide examples or experiments showing how the discovered connection can be leveraged to optimize training, enhance pruning techniques, or improve uncertainty estimation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is well-written, clear, and provides interesting insights that are of interest to the TMLR community. There were some questions raised by reviewers on, for example, using simpler measures of overlap via cosine similarity and additional experiments like perturbation study, but the authors did a good job in responding to these questions, and at the end of the rebuttal period, all reviewers unanimously recommended acceptance of the paper and AE concurs with this judgment and recommends accept as-is. However, the authors are still recommended to think more deeply about some open questions raised by the reviewers, such as the practical implications of their work and some additional, qualitative discussions outlining concrete use cases will be beneficial to further increase the impact of their work.
Decision: Accept (poster)