Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds
Authors: Ke Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Fisher information
information geometry
stochastic trace estimation
deep learning theory
deep classifiers
Primary area: learning theory
TL;DR: An unbiased, efficient Fisher information matrix estimator for deep classifiers with bounded variance.

Abstract: The high-dimensional parameter space of deep neural networks --- the neuromanifold --- is endowed with a unique metric tensor defined by the Fisher information. Reliable and scalable computation of this metric tensor is valuable for theorists and practitioners. Focusing on neural classifiers, we return to a low-dimensional space of probability distributions, which we call the core space, and examine the spectrum and envelopes of its Fisher information matrix. We extend our discoveries there to deterministic bounds for the metric tensor on the neuromanifold. We introduce an unbiased random estimator based on Hutchinson's trace method and derive related bounds. It can be evaluated efficiently with a single backward pass per batch, with a standard deviation bounded by the true value up to scaling.

DETERMINISTIC BOUNDS AND RANDOM ESTIMATES OF METRIC TENSORS ON NEUROMANIFOLDS
Ke Sun
CSIRO's Data61, Australia Ke. Sum@data6. wtw.e//daaall.....dd.... ru.u
ABSTRACT
The nigh-dimensional parameter space of deep neural networks the neuromani- fold - is endowed with a unique metric tensor defined by the Fisher information. Reliable and scalable computation of this metric tensor is valuable for theorists and practitioners. Focusing on neural classifiers, we return to a low-dimensional space of probability distributions, which call the space, and examine the spectrum and envelopes of its Fisher information matrix. We extend our discoveries there to deterministic bounds for the metric tensor on the neuromanifold. We introduce an unbiased random estimator based on Hutchinson's 's trace method and derive related bounds. It can be evaluated efficiently with single backward pass per batch, with standard deviation bounded by the true value up to scaling.
INTRODUCTION
Deep learning can be considered trajectory through the space of neural networks (neuromani- fold; Amari 2016), where each point is a neural network instance with prescribed architecture but different parameters. This work investigates classifier models in the form pyy x, 2),0). where æ the input features. y E {1,.... CC} is the class labels (C' 2 2), and 0 E θ is the network weights and biases. Given an unlabeled dataset DD2 = {.1, F2, ... }. the intrinsic structure is specified by the Fisher Information Matrix (FIM), defined as:
① log ply x, 0) log ply x, 22.0]
Olmy Dlay 0e
F(I) Σ
E plu (.,0)
Σ ZED_ x.0)
(1)
00
00
D8
DED
where (0) = log ply x, 0) denotes the log-likelihood. This based on a supervised model + y. For unsupervised models, one can treat constant and apply the same formula. Under regularity conditions, F(0) a dim x dim(f) positive semidefinite (psd) matrix varying smoothly with 0 O. Following Hotelling (1929), and independently Rao (1945), F(0) is used metric tensor on θ, representing a local, degenerate inner product!. For example, one can measure the intrinsic squared distance between 0 and 0 d0, where d0 is small dynamic θ, d0 F(O) d0.
The FIM is the unique metric tensor (Čencov, 1982) which underpins the information geometry of the neuromanifold θ (Amari, 2016). The most widely used application of the FIM is perhaps geometry-inspired optimizers such natural gradient (Amari, 1998), Adam (Kingma & Ba, 2015), and their variants (Martens & Grosse, 2015; Pascanu & Bengio, 2014; Yao et al., 2021; Lin et al., 2021). F is also applied to regularized fine-tum fine-tuning (Lodha et al., 2023), pruning (Heskes, 2000; Tu et al., 2016), transfer learning (Chen et al., 2018), and overcoming catastrophic forgetting (Kirkpatrick al., 2017). Theoretically, the FIM provides insights due its connection with the Hessian of the loss landscape and generalization (Hochreiter & Schmidhuber, 1997), and that any f-divergence is locally characterized by the FIM (Blyth, 1994).
Given its deep and broad background, estimating F(0) with theoretical guarantees estimation quality is important even in the absence a specific application pipeline. Inaccurate estimates can lead
In the literature, F(8) is sometimes referred to as a curvature matrix (Martens, 2020). In rigorous terms, F(0) is a degenerate Riemannian metric tensor (psd with null directions). which belongs to the broad family of singular semi-R Remi-Riemannian metrics (degenerate with an arbitrary signature; see Kupeli 1996; Sun & Nielsen 2025).

---

to overly aggressive overly conservative learning steps (Amari, 1998), or miscalculated saliency scores and suboptimal pruning decisions (Tu et al., 2016). In learning theory, a loosely estimated FIM undermines the validity of geodesic distances and the applicability of Cramér-Rao lower bounds, and may distort curvature-based sharpness, which is closely linked to generalization (Hochreiter & Schmidhuber, 1997). A widely used deterministic approximation is the empirical FIM (eFIM;
a.k.a. empirical Fisher; see e.g. Le Roux et al. 2007) F(0) == ((E,))DD
0eau Dexp
where D
09
30
{(11, 31), (I2, U2). ) is a labeled dataset. Alternatively, the Monte Carlo (MC) estimator F(0)
0 Sey L -> k=1 00 30 a0 where each Ty. is drawn uniformly from D2 and DR p(y (x), gives an unbiased stochastic estimate of F()).
We advance the state-of- th- in both deterministic and stochastic computations of the FIM. improv- ing accuracy in terms of bound gap and variance. We make the following contributions: 1 Envelopes of the FIM in the statistical simplex (space of output probabilities); @ Deterministic bounds of the FIM for classifier networks and their tightness analysis; @ A novel family of random FIM estimators based on Hutchinson's 's trick (Hutchinson, 1990; Skorski, 2021), which can be computed efficiently with bounded variance; @ An empirical study estimating the FIM of modern deep classifiers (e.g. DistilBERT; Sanh et al. 2019) to showcase the advantages of Hutchinson's ' estimate in real-world settings.
The remainder of this section introduces our notation. Section 2 develops fundamental bounds of the FIM in low dimensional spaces of probability distributions. Section extends the deterministic bounds into the high dimensional neuromanifold. Section 4 introduces Hutchinson's FIM estimator and discusses its theoretical properties validated by numerical simulations. Section 5 positions our work into the literature. Section 6 concludes.
NOTATIONS AND CONVENTIONS
We use lowercase letters such as À or a for both vectors and scalars, which should be distinguished based context, and capital letters such as A for matrices. All vectors are column vectors. A scalar-vector or vector-scalar derivative such as 00/00 yields a gradient vector of the same shape as the vector. A vector-vector derivative such as 0z/a0 denotes the dim(=) dim(() Jacobian matrix of the mapping 0 + z. || denotes the Euclidean norm for vectors and the Frobenius norm for matrices. || lle denotes the spectral norm (maximum singular value) of matrices. The metric tensors (variants of FIM) are listed in table 1.
Table 1: Metric tensors. We use I/I/I/1 /I for oow-dimensional statistical manifolds and use FIFI F for neuromanifolds. We optionally use superseripts to indicate the associated parameter space. For example, I and Ja denote the metric tensors on the statistical simplex and the space of neural networks with simplex-valuee outputs, respectively.
FIM
empirical FIM (eFIM) I(z) I(z)/F(0)) F(0)
Monte Carlo FIM (MC FIM) 1(x) (())
Hutchinson FIM I[(z) / F(O)
I(x) F(0)
2 GEOMETRY OF LOW-DIMENSIONAL CORE SPACES
Consider a classifier network p(y xx 0) = p(y
2(x. 0)), where 2(2,, 0) is the last layer's linear De.
output. Due to the chain rule, we plug
06zy 84
(1z)
into Eq. (1). Then, we can easily arrive at ds
02 00
0z 00
F(O)
Σ ZED=
I(=(0, 0))
which is in the form of a 2020)² from a liw-dimensional statistical manifold with metric I(z), to the much
matrix (Martens et al., 2010), a pullback metric tensor (Sun,
Strictly speaking. the pullback tensor requires the Jacobian of 0 + to have full column rank everywhere, which is not satisfied in typical settings of deep neural networks. This leads to degenerate (singular) metric tensors.
2

---

neuromanifold with metric F(0). In this section, we rediscover the geometrical structure of the low-dimensional statistical manifold, which we refer to as the core space, or simply the core.
In multi-class classification, y (given a feature vector x) follows categorical distribution ply i x| 0) = (z 0), = 1, 1, ,C. All possible categorical distributions over { , form a closed statistical simplex A == { (DI, ... .PC) E Pi = 1; vi. pr 2 0 The superscript
C' - 1 denotes the dimensionality of A and can be omitted. If E int (-) (interior of AC-1), can reparameterize p Soft Max(z), where E R is the logits. The core C is curved space, where p or & serves as a coordinate system, in the sense that different choices of p or 2 yield different distributions. By Eq. (1), the FIM is:
7 (2)
[(ey - D) - p] diag (p)-pp"
where diag (.) means the diagonal matrix constructed with given diagonal vector. Below, depending on context, diag (-) also denotes a diagonal vector extracted from a square matrix. (without subscripts) denotes a vector of all ones, ey denotes the one-hot vector with only the y bit activated, and Gij denotes the binary matrix with only the ij'th entry set to 1. Note that is redundant coordinate system as dim(z) = C C - 1. If 2 E int(A (2) has a one-dimensional kernel: for all E R. one can easily verify I (z)
Noting that (2) is a rank-1 I perturbation of the diagonal matrix diag (p), we can apply Cauchy's interlacing theorem and study the spectral properties of (s).
Theorem 1 (Spectrum of Simplex FIM). Assume the spectral decomposition [²) -' Nonaa
where A S ... S.A.A Xc- Then A = 0: 01 = e/llell: Ee= d = - l|pll: and
max {p:(1 (1(1-pil) pi)p (
C- -1 PIC-
1 ||
S Ao min (pic). 2 m - p.)), 1 - ||p113]
where P(C-1) and P(c) denote the second-larged and the largest elements of p. respectively.
The largest eigenvalue of (2),, denoted as Ac, and its associated eigenvector correspond to the "most informative" direction at any E A C-1, By Theorem λe can be bounded from above and below. The bound gap is at most min(P(c) P(C-1), max, (p.(1 - p.)) We have found through numerical simulations that, practice, the bounds in Theorem are tight and can provide an estimate of λo within narrow range. The lemma below gives lower and upper bounds of (z), both with a simpler structure than I (z), in the space of psd matrices under the Löwner (Loewner) partial order.
Lemma 2. Vz E in)(A assume the spectral decomposition I=(2) = E Nvvey where A1 S ... S AC-1 Ac. Then, AcuCuy 3 (z) 3 diag (p). Moreover, Acvory is the best rank-l representation of (2) in the sense that no rank-1 matrix B + AcuCuy satisfies AcuCvy = B 3 (z). Meanwhile, diag (p) is the best diagonal representation I in the sense that no diagonal matrix D A diag (p) satisfies (z) = D = diag (p).
The simplex FIM is upper-bounded by a diagonal matrix and lower-bounder by a rank-1 matrix. By Lemma 2, Acvovy is the lower envelope (greatest lower bound) of I in rank-1 matrices, and diag (p) is the upper envelope (least upper bound) of IA(2) in diagonal matrices. If the bounds in Lemma are used as deterministic estimate of - (z), the error can be controlled, as shown below.
Lemma 3. We have Vz E A, - (2) diag (p) || = ||||l 2 t. meanwhile,
s
min Ilell - P(C-1): V E p where P(s) denotes the entries of p sorted in ascending
order.
v EC- Pi, is the Fuclidean norm of trimmed p, i.e. the vector obtained by removing p's smallest and largest elements. By Lemma 3, the upper bound diag (p) always incurs an error of at least L/C.. Depending on p. the lower bound Accouy can more accurately estimate I as the error can go to zero.
Note
Alternatively, one can use random matrices to estimate (z). By Eq. (3), the rank-1 matrix R(y) (e, (ee TT) is unbiased estimator of I (z). The MC FIM of A is i m Σm= R(gx).
=1
3

---

where each Us ~ --pyy2)) The associated eFIM II' R(y), where y is a given empirical
sample. The lemma below shows the worst error of I (z).
Lemma 4. Vz E AC By E (1,-.-,, ,C,, such that [|((y) - (2)] 2 1 |p|| Xc -2p(1) 2 2||p|| - 11)
The first. "2" is tighter but the second "2" is easier interpret. The term || can be as large 1 (when p is close to one-hot), In such cases, using R(y) to estimate (z) may incur significant error y is adversarially chosen.
In classification tasks with multiple binary labels, we assume p(y: = 1 11 pr (i and that all dimensions of y are conditionally independent given z. All such distributions form C-ddimessinaal hypercube C (p) = {P11. PC) Vi, 0 S S 1}. which is the product space of 1 I-dimensional simplices. Consider p; = a(2) = 1/ + 2))) for i 1,..., C. In this case,
the FIM is diagonal matrix, given by
(2) diag ((P1 (1 P1), PC(1 pc))) diag (z1).... o' (zc)).
(4)
In what follows, unless stated otherwise, our results pertain to the core A as it is more commonly used and has more complex FIM as compared to C.
FIM FOR CLASSIFIER NETWORKS: DETERMINISTIC ANALYSIS
In this section, we give lower and upper bounds of (+) (Proposition 5) and analyze each bound gap (Propositions and 8). Our bounds result from simple matrix analysis and are more operational than related thcoretical bounds such as monotonicity of the FIM under marginalization or coarse- graining (Amari, 2016). Our bounds are novel in that ! they are built on envelopes (tightest bound) in the core, and @ they depend on the order statistics the output probability vector.
3.1 DETERMINISTIC LOWER AND UPPER BOUNDS
By Eq. (2), the neuromanifold FIM F(0) is determined by both the core space and the parameter- output Jacobian Ø= Similar to Lemma 2, we can have lower and upper bounds of _ (0) in the space of psd matrices (although these bounds are not envelopes as in Lemma 2).
Proposition 5. p(y xr, 0) E C-1 is categorical, then v0 E O, we have
ΣΣ A
C
O2
T
T Oz
2 == (0) = Σ Σ ply x. 0)
Ozy 00
02y
U2U
00
D0
00
ZED. C-C-++1
DED=
for all ls E {l...., ...,,, where >;- == A₂(I, 9) and U, == (z, f) denote the i-th eigenvalue and eigenvector of [I(2(x, ((xx0,))) 0)), ordered such that a < 5.. Xc
Remark. The LHS is sum of [Da) (number of sumples in Dz)) mutrices, each of rank k. Its rank is al most k.(Dz). The RHS a sum of C(Dz) matrices of rank-1 and potentially has a larger rank.
Remark. By Theorem A1= Therefore, the first "z" turns into "=" when k = =C=
If ppy|| is in C, then (2(2x, 0)) is diagonal as in Eq. (4). By Eq. (2), we have FF( Oz,, 30
(O2g T
EXED, Eg (1 - pb) summary. F(8) can be bounded or computed using the Jacobian 0z
which is similar to the upper bound in Proposition 5. In
de well as the output probabilities
p(y xr, 0). The following analysis depends the spectral properties of dz Across our statements, ds
00 we denote the singular values of 56 sorted in ascending order, as a (z, 0) .... < aC (z, 0). In Proposition 5, by taking the trace on all sides, the trace of the FIM can be bounded from above and below.
Corollary 6. Ifllu x, 0) E C-1 is categorical, then it holds for all 0 E θ that
Ozy 12
Ac(z,0)o (2)
Σ Σ
l12 0)0 (n)) < Σ
Σ
ply x 0)
a0
RED
RED
ZED 1
4

---

These bounds useful to get the overall scale of (0) without computing its exact value. The proposition below gives the error of the upper bound in Proposition in terms of the Frobenius norm.
Proposition 7 (Tightness of the Upper Bound), We have v0 E θ that
Σ =ED_
Oz 80
pla,0) 0/
-
Σ FED
Σ
ply 00
O2y
dzy
(0)
S
Σ
llp(a, 0)| (x, 0),
00
where p(z, 0) Soft 0)) denotes the output probability vector.
We use the Frobenius norm for matrices but it is not difficult to bound the spectral norm using similar techniques. By Proposition the error of the upper bound scales with the operator 2-norm (maximum singular value) of the parameter-outple Jacobian 0= As in the core space, the FIM upper bound 20
remains loose. For example, let p tend to be one-hot, the LHS Proposition 7 does not vanish, but scales with certain rows of de corresponding to the predicted y. Naturally, we also want examine 30 the error of the lower bound in Proposition 5, as detailed below.
Proposition 8 (Tightness of the Lower Bound). We have for all 0 E O and all k E ,,, - that
Σ IED, C++++1
X
O2
Oz 0A
FO(
S
Σ DED=
Σ 1=2
o itx (2, 0)
Clearly, as approaches a one-hot vector, all elements in the trimmed vector P(a), for i = 2, C-1, tend to zero, and the error approaches zero since its upper bound on the RHS goes to zero. From this view, the lower bound in Proposition 5 is better estimate as compared to the upper bound.
Remark. By noting that 0 S (x 0) S oc (2,0), we relax the bound in Proposition 8 and get
C-k Σ
Σ A
O= 20
- -
> RED
S
()
00
RED, C-D,i=C-
The estimation error of the low-rank lower bound in Proposition is controlled by the norms of the Jacobian and the trimmed probabilities (P(2).... PP(Ck))- The latter is upper bounded by P(c-k) (x, 0), the (k + 1)-th largest probability of each sample л. By comparing with the second "<" Proposition 7, one can easily observe that Proposition is tighter general.
3.2 EMPIRICAL FIM (EFIM)
Recall from the introduction that the eFIM F(0) gives biased, deterministic estimate of F(0). Intuitively, when the network is trained, computations based on the given labels are close to the expectation w.r.t. ply xx, 0), and F(O) is expected approximate F(8) well. However, the bias of F(O) can be enlarged if y is set adversarially. By simple derivations, F(0) = ΣxED, ( R(y)-
82 Observe that is similar Eq. (2), except I(=(æ, 0)) is replaced by its empirical counterpart
R y.) If the neural network output is in A, the eтror of cFIM can be bounded, as stated below.
Proposition 9. V0 O, Vy, we have | (0) プー (0)llo < ExED, (1+ ll (01
Here we need to switch to the spectral norm || ||o to get a simple expression of the upper bound. The approximation error of A (f) is controlled by the spectral norm of the parameter-outpi Jacobian. The error in the Frobenius norm is even larger. The bound is loose as compared to Propositions 7 and 8.
We have found in Lemma 4 that using R(y) to approximate I>(2) suffers from a large error if is chosen in tricky way. The same principle applies to using F(A) to approximate F(B). Proposition 10. V0 E O, Va, Fy, such that
(0z) 09
T I (=(x 0))
0= (
(0z) T
0z
R(v)
> oz
|l 0)11 Ac( 0) 2p (2,
30
89
a8
5

---

In the above inequality, the LHS is the enror of F(O) for I Dr. Therefore, when y is set unfavorably, the eFIM suffers from an approximation error that scales with the smallest singular ds value of 30 Among the investigated deterministic approximations of FA, the lower bound in Proposition provides the smallest guaranteed error but is relatively expensive to compute. We solve the computational issues in the next section.
HUTCHINSON'S ESTIMATE OF THE FIM
4.1 LIMITATIONS OF MONTE CARLO ESTIMATES
The quality of the MC estimate F()) can be arbitrarily bad. Consider the single-neurol model
2 = &r for binary classification, where 2, 8 x are all scalars, and @ close to zero. Then p ~ is
a fair Berouuli distribution. T(z) = p(1 - p) R 3 The Jacobian is simply åz 30 = J. and F(O) Ep(x) [=) d=] ] (Epcs) [x2]] A basic MC estimator takes the form F(0)
where 24 ss are independently and identically distributed according to p(æ). Its variance is Var (F) Jn (pp(s) (x*) E (e) [. We let p(x) be heavy-tailed distribution, e.g. Student's t-distribution
with degrees of freedom, so that Var(F) ) is large while F(0) is small. Then Ep(ss) (1²) -²²
and (x(=) (r-4) The ratio (xt=)("") 3(w-2) can be arbitrarily large when v + 4+ (B)(() x²)² w-4
3u²
Therefore the coefficient of variation (CV) (td(F))F(() is unbounded. Throughout our analysis, the CV is a key indicator of the quality of a FIM estimator, as a bounded CV for a non-negative random variable X ensures the random estimator's probability mass within [0, oy]. where and μ 0 K, then by Cantelli inequality, have
is the mean of X. If CV
StdX I
P(X ap) = P X 2X (a- SP SP
x X u
a K
Stdx
I
1
a
K
The general case more complicated, but follows similar idea. The variance of MC estimators depends on the 4th moment of the Jacobian 0= 86 w.r.t. p(x) while the mean value F(O) only depends on the 2nd moment of 39 is unbounded without further assumptions on p(r)). One can increase the number of samples to
0= The ratio of the MC estimator's variance F²)),, or the CV Std(F)/F(O).
reduce variance. However, this is computationally expensive especially in online settings.
4.2 HUTCHINSON'S ' ESTIMATE
In light of the challenges of MC estimates, we introduce new way get unbiased estimate of the FIM. First, compute the scaler-valued function
(D 0) = ΣΣ Viyl
plu x. (,,0)e,
(5)
DED y=1
where Ezy is a standard multivariate Gaussian vector of size C[D| or a Rademacher vector, and p(y .r, 0) has the same value as p④y x, 0) but is used with a stop-gradient operator, mean- ing its gradient is treated as zero, preventing error from backpropagating through ply x, 0). This p can be implemented by Tensor. detach () in PyTorch (Paszke et al., 2019) or simi- lar functions in other auto- differentiation (AD) frameworks. Second, the gradient vector 80 =
0e Eay can be evaluated via AD, e.g. by h backward () in PyTorch. Third, the random psd matrix F(0)= =
ELED, E Vp(y x. 0) 0f.u
ah ab ae T. which we refer to as the "Hutchinson's 's estimate" (of the FIM). can be used to estimate F(8). By straightforward derivations,
E (F(O)) = Σ ΣE E m(
Volul2,0) 0) x', Daa Ge,'s R ( = F(0). (6)
a0 ((
ZED, ae
The last "=" is because Ep(e) (EayEa'r)) = 1 if = x' and = y', and (Esesr = 0 2b otherwise. Considering 33 as implicit representation of the FIM, its computational cost consists

---

of ① evaluating the function; @ the backward pass to compute the gradient of h. The cost is the same evaluating the gradient of the loss EnED Ey= (0), noting that is the log- likelihood randomly flipped by Grussian/Rademacher vector. Moreover, h can reuse the logits already computed during the forward pass. Therefore @h requires merely one additional backward pass, making it practical for large-scale networks. In summary, F(O) is a model-agnostic, unbiased estimator of F() for general statistical models, independent of neural network architectures and applicable to non-neural network models as well. Hutchinson's 's estimate has provable unbiasedness and variance bounds, as formally established below.
Theorem 11. For all 0 E O we have Ep(e) (I(()) = F(O). lf((E) is standard multivariate Gaussian, then (Fr(Fu(()) = (0)2 if P(E) is standard multivariate Rademacher, then Var(E,((0)) = (0) - 2 E.ED, Ey= ('(y) x. ()( (0l )
It is known that the Rademacher distribution yields smaller variance for Hutchinson's method com- pared to the Gaussian distribution. In what follows, p(E) Rademacher by default. By Theorem 11, (O)) S v (0). Thus the CV (td(FF.(0)/FFia is bounded by v2. We only investigate the diagonal of Hutchinson's 's estimate because the diagonal FIM is widely used, but our results are readily extended to off-diagonal entries.
Remark. For a dataset with J minibatches, each with a diagonal FIM (O)( computed with an independent probe, we have Fi)(0) E Fl)(())) (0). By Theorem 11, Var(Fu((O))
S 2 (F) Moreover, we roughly approximate Ra(()= (O). Then, Var(F.(0)) S 2 (ii((0)
E/=1 Var(F! is (0)) S 2 (F (
3 (F)(0))² At the dataset level, the variance is inversely proportional to J. while the computation cost grows linearly with J, presenting typical accurayy-computatio trade-off.
Remark. Taking trace on both sides of Eptt) (F(#)) = F(0). we get Epter 0b 1?2) tr(F())). The 36
squared Euclidean norm of 80 is an unbiased estimate of the trace of the FIM. This is useful for efficient computations of related regularizers (Peebles et al., 2020).
An alternative Hutchinson's estimate based on the equivalent FIM expression F(0) =
4 4E Ey=
@ x.0) @yrlyla.0) (see e.g. the first unnumbered equation Sun & Nielsen
58
89
2017) is detailed in section B. We find that in practice its performance is similar to the above F.
Note that sample of the random matrix (()) always rank- 1: rank (0)) 1 S rank F(0). Ideally, one can compute the numerical average of more than one F(0) samples to reduce variance and recover the rank, each requiring a separate backward pass. Due to computational constraints in deep learning practice, far fewer (e.g., 1) samples are used. Instead, accumulated statistics along the learning path 0, + 02 - ... can be used to maintain an (exponential) moving average of F(O.). The underlying assumption is that 01, O..... connected by small learning steps lie close to one another in the parameter space. Therefore, averaging F(0;) provides a reasonable approximation of the local FIM with sufficient rank.
4.3 DIAGONAL CORE
For multi-label classification, and for computing the upper bound in Proposition 5, the core matrix is diagonal, in the form IDG (2(2, 0)) = diag (x, (2, 0)), and the associated FIM
is FDG(() (0) = ΣeED_
(0)) DDG (2(z, ((z.,)) 0)) 0g
àz In the former case, (2, 0) = p(v x. (1
ply Jr, 0)); in the latter case, (,, 0) = p(y xr, 0). Here, superscripts e.g., "DG" for diagonal
and "I.R(K)""TTR" for low-rank - indicate the parametric form of the core FIM, in contrast to denoting the core space as in IA, We define the scalar-valued function
h DG)
Σ Σ
v ((,,))) 0))
(7)
ZED, y=1
7

---

where Sxy are standard Rademacher samples that are independent across all and y. Similar to the derivation steps in section 1, we first compute the random vector 2b through AD, and then "00
compute FDG (0) = 8e
ag an (or its diagonal blocks) to estimate (D)
00
For computing the upper bound in Proposition 5, (g(2z 0) = Py (ar, 0), then we find that Eq. (5) and Eq. (7) are similar. The only difference is that, the "raw" logits 2y in Eq. (7) are replaced by Exy (0) =
By - log E, exp(zy) in Eq. (5). Compared Bzy 88 the gradient 30
d6 day
8d E p(y 2, 0) day is
80
centered. Due to their computational similarity, in practice, one should use Eq. (5) instead of Eq. (7)
and get an unbiased estimate of (0). Eq. (7) is useful when the dimensions of y are conditionally independent given 2, e.g. for computing (0).
4.4 LOW-RANK CORE
By Proposition 5, (O (0)= We define
0, 01 O), (zr
(L)(K) = 3
Vi(2,0) (a, 0)z(2, Enis zED. CC
(8)
where Eas are independent standard Rademacher samples, and & E (l..... ,.......--. CC- 13. For computing g..(k) (0), we only need k(D21 Rademacher samples, as compared to C\D22 samples for computing h(0) and (0). Correspondingly, FLR(k)
(g-a))) an.-R((k)
used to estimate FLR(k) (0).
20
When = I, we simply denote FLR == FLR(1), LLR == h LLR)), and FLR == kLR(1),
It remains to compute Ax( 0) and v3 0). which requires spectral decomposition of C. x C matrix for each x E Dz- The cost is only acceptable when C' is small to moderate. In our CIFAR-100 experiments (C 100), the computational speed of FLR(k) drops to roughly half that of F. k 1, however, (x 0) and (x 0) can be computed more efficiently using the power iteration. By Eq. (3), starting from a random unit vector vgg, we compute
po rec - TTCCP [Z)(2) lp r6c 2p "ppl'
(A())
[+1 c
for 1,2,-... until convergence or fixed number of iterations is reached. Then, Ac = pp= (c) (p ) For computing Ae and ve for all E DD,, the per-iteration computational cost (C)D,,,), so the total cost is ((TC)Dal) for T iterations. The number of iterations required increases as the spectral gap 7 == Ac Ac-1 decreases. Convergence can be slow when ry is small (e.g., for near ucar-uniform output distributions). In our implementation, we simply use a fixed iteration budget (e.g.. = 30). All our estimators: h, h DG and (LR(() can be computed solely based on the neural network output logits 2(x, 0) for each D,,-
(vc
4.5 NUMERICAL SIMULATIONS
We compute the diagonal FIM of the following models: @ DistilBERT (Sanh et al., 2019; Wolf al., 2020) fine-tuned on the Stanford Sentiment Treebank v2 (SST-2) (Socher et al., 2013) (with C' = 2 classes); 2 DistilBERT (pretrained) with a randomly initialized classification head for DBpedia ontology classification (Lehmann al., 2015) (C 14); @ RoBERTa-base (Liu al., 2019) fine-tuned on Multi Genti Natural Language Inference (MNLI) corpus (Williams et al., 2018) (C 8); & SmageNet-pretrainee ResNet-50 (He al., 2016) with a random classification head for C1F image classification (Krizhevsky, 2009) (C 100); @ Same as @ but with an ImageNet- pretrained EfficientNet-BO (Tan Le, 2019) backbone; @ Wav2Vec2-ban (Baevski et al., 2020) (pretrained) with a random classification head on SpeechCommands audio classification (Warden, 2018) (C 12).
For all datasets, the diagonal FIM Fi is computed on a fixed random subset of 128 batches with batch size of B- = 64. We evaluate the ground truth Fss using its closed-fore form expression in Eq. (1), which requires 8192C' backward passes and is impractical to the histograms of F& on RoBERTa-bee base, ResNet-50, and Wav2Vec2-basse including the zero atom (probability mass zero). Other datasets and models are omitted due to space constraints. The
the full dataset. Figure/ I shows
8

---

Embedding (39.0M)
RB1 (21.3M)
RB2 (21.3M)
RB3 (21.3M)
RB4 (21.3M)
Classifier (592.9k)
0eb:3000
bs
Fa
10-2
(a) RoBERTa-base MNLI
(9.5k)
Block1
(1.2M)
(7.1M)
Block4 (15.0M) Classifier
2.0%
C-00 Gendd
2-18
1-1% Aerds
10
10"1
10
16
16"
10"
10
18"9
10:4
10"
16
1219
(b) ResNet-50 on CIFAR-100
FeatureExtractor (4.2M) FeatureProjection (395.0k)
Encoder (89.8M)
Classifier (199.9k)
Fi
10t4
(c) Wav2 base on SpeechCommands
Figure 1: Histograms of the ground-trute touun diagonal FIM entries Fu on a logarithmic x-axis. The zero atom is displayed as vertical bar at the left edge of each plot. From top to bottom, three different models (associated with three different ncuromanifolds) are shown. From left to right, successive components from input output and their parameter counts displayed. Ç denotes the zero probability. denotes the average value of Fu in the component. The solid and dashed vertical lines indicate the median and the pas quantile of strictly positive values, respectively.
distribution of Fa differs substantially across tasks. For example, in RoBERTa-base, the embedding layers exhibit large atom at corresponding unobserved vocabulary, whereas intermediate transformer layers show the largest Fisher information. Similar patterns are observed in other NLP tasks.
We only compare FIM estimators that can be computed using single backward pass per batch, including the empirical FIM F (0), (0) (Hutchinson's unbiased estimate), IFDG (0) (upper-biased
estimate of Fsa), FLR (0), and FLR(2)(0) (0) (lower-biased estimate of Fsi). The MC estimate f is excluded, because it requires B backward passes per batch (B: batch size) and is less applicable production settings. Table 2 shows the relative mean absolute error (ReIMAE), defined as the average ratio of the absolute error to the ground-truth value, with 10-12 added for numerical stability. For
example, the ReIMAF of empirical FIM is dim((] (8) Fa Pite
Because Fa is typically small
in magnitude, ReIMAE is more interpretable than the mean absolute error (MAE). In general, Fa is the most accurate, with a ReIMAE of approximately 0.2, corresponding to = 20% relative deviation from the ground truth. This improvement arises because FF is unbiased, whereas other baselines
biased. Nevertheless, FLR and FLR(2) are the most accurate on SST-2 and MNLI. This is because, on these two tasks, the model fine-tuned and the core FIM exhibits an approximately low-rank structure. The empirical FIM and FDG are the least accurate.
The computational speeds of all methods broadly similar. Hutchinson's estimate is fast as the empirical FIM. contrast, FRRR and F _l.((2) are more expensive, because they rely on power iterations spectral decompositions of the core FIM. The bottom line is: to compute the diagonal FIM, we recommend Hutchinson's unbiased estimate F over the empirical FIM F. For fine-tuned models, one may alternatively use ELR or pLR(k) for potentially improved accuracy.
RELATED WORK
prominent application of Fisher information in deep learning is the natural gradient (Amari, 1998) and its variants. The Adam optimizer (Kingma & Ba, 2015) uses the empirical diagonal FIM.

---

Table 2: ReIMAE w.r.. the ground-truth diagonal FIM entries Fu for different FIM estimators (columns) across tasks (rows). Numbers in parentheses mean speedup factors relative to the empirical FIM (larger is faster). CIFAR-100 is used for both ResNet-50 (R) and EfficientNet-Bo (E).
F
FDG
FLR
F FLR(2)
SST-2
(x1)
0.18 (x1.07)
341 (x1.07)
0.05 (x0.96)
0.05 (x1000)
DBpedia
0.59 (x1)
0.22 (x1.00)
(x1.00)
0.8 (x0.933
(x0.93))
MNLI
(x19
0.16()
(x0.97)
0.11 (x0.96)
(x12
CIFAR-100 (R)
(x17
0.11 (x0.99)
(x1.01
0.97 (x0.97)
0.95
CIFAR-100 (E)
0.11 (x17
(.120x
0.98 (x0.98)
0.96
SpeechCommands
0.17 (x.8(x1)
(x0.97 (x4.7))
0.39 (x0.89)
0.22 (x0.91)
Efforts have been made to obtain more accurate approximations of F(8) at the expense of higher computational cost, such as modeling the diagonal blocks of F(O) with Kronecker products (Martens, 2020) or component-wise FIMs (Ollivier, 2015; Sun & Nielsen, 2017), or computing F(0) through low-rank approximations (Le Roux et al., 2007; Botev et al., 2017). The FIM can be alternatively defined on a sub-model (Sun & Niclsen, 2017) instead of the global mapping z + y or based on a-embeddings of parametric family (Nielsen, 2017). AdaHessian (Yao et al., 2021) uses Hutchinson probes to approximate the diagonal Hessian.
From theoretical perspectives, the quality of the Kronecker approximation is discussed (Martens & Grosse, 2015) with its error bounded. It is well known that the eFIM differs from F(Ø) (Pascanu & Bengio, 2014; Martens, 2020; Kunstner al., 2020) and leads distinct optimization paths. The accuracy of two different MC approximations of F(0) is analyzed (Guo & Spall, 2019; Soen & Sun, 2021; 2024; Sun & Spall, 2021), which lie in the framework of MC information geometry (Nielsen & Hadjeres, 2019). In our analysis, the Hutchinson estimate F (0) has unique advantages over both MC and the eFIM. Notably, the MC estimate requires computing ofegs for multiple samples of (#x: Úts). while (0) typically needs to evaluate one gradient vector 0h
30
70 Our bounds improve over existing bounds, egg. those of F(0) (Soen & Sun, 2024), through carefully analyzing the core space.
Hutchinson's 's stochastic trace estimator is used to estimate the trace of the FIM (Jastrzebski et al., 2021), or the FIM for Gaussian processes (Stein al.. 2013; Geoga et al., 2020) where the FIM entries are in the form of trace. Closely related to this are computations involving the Hessian, where Hutchinson's ' trick is applied to compute the Hessian trace (Hu et al., 2024), the principal curvature (Böttcher & Wheeler, 2024), or related regularizers (Peebles et al., 2020). The Hessian trace estimator implemented in deep learning libraries (Dangel et al., 2020; Yao et al., 2020) and usually relies on the bessian-vector product. Our estimator F differs from prior efforts in three key aspects: ① unbiased for the true FIM; @ coming with explicit variance bounds; 3 scalable to modern architectures. Crucially, no previous method simultaneously satisfies all three properties. By leveraging both Hutchinson's trick and AD interfaces, F avoids the need for expensive Hessian computations approximations, and is well-suited in scalable settings. As demonstrated in the experiments, FF can be applied to different classifier networks regardless of the architecture.
CONCLUSION
We explore the FIM F(O) of classifier networks, focusing on the case of multi-class classifica- tion. We provide deterministic lower and upper bounds of the FIM based on related bounds in the low-dimensional core space. We discover a new family of random estimators F(0) based on Hutchinson's 's method. They offer theoretical guarantees on estimation quality (bounded variance) and can be computed efficiently via auto-differentiation The proposed F(0) is readily integrated into deep learning libraries (Dangel et al., 2020; Yao et al., 2020) for efficiently evaluating the FIM or approximating the Hessian. Our analysis in the core space gives insights and useful tools for information geometry where the probability simplex is widely used. As limitation, the results here concern static computations of F(t) at a fixed 9 but are not directly integrated into downstream applications that employ F(#) during learning. For example, new deep learning optimizers not developed here and left as future work. Advanced variance- reduction techniques (Meyer et al., 2021) that could improve F(0) remain be investigated.
10

---

ACKNOWLEDGMENTS
The author thanks Frank Nielsen and the anonymous reviewers for their insightful feedback.
ETHICS STATEMENT
The author has read the ICLR Code of Ethics and confirms that this research fully complies with the Code of Ethics.
REPRODUCIBILITY STATEMENT
All assumptions and proofs of the formal statements are provided the main text and the appendix. A Pa PyToch library to compute Hutchinson's 's FIM for general deep classifiers is available at https: / / thub. com/ sunk wwun/ffimmeeeeeeeeeeeeeeeeeeeeee fim-estimtttrrrr
THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs are used as side reference for language editing purpose (grammar, wording, and translation).
LLMs are not used in developing the technical contents or the core results.
REFERENCES
Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Comput., 10(2):251-276
1998.
Shun-ichi Amari. Information Geometry and Its Applications, volume 194 of Applied Mathematical Sciences. Springer- Verlag, Berlin, 2016.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 12449-12460. Curran Associates, Inc., 2020.
Stephen Blyth. Local divergence and association. Biometrika, 81(3):579-584, 1994.
Alcksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton optimisation for deep learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp- 557-565. PMLR, 2017.
Lucas Böttcher and Gregory Wheeler. Visualizing high-dimensional loss landscapes with Hessian directions. Journal of Statistical Mechanics: Theory and Experiment, 2024(2):023001, 2024.
N. N. Čencov. Statistical Decision Rules and Optimal Inference. Translations of mathematical monographs. American Mathematical Society, 1982.
Shixing Chen, Caojin Zhang, and Ming Dong. Coupled end-to-end transfer learning with generalized Fisher information. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4329 4338, 2018.
Felix Dangel, Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into backprop. In International Conference Learning Representations (ICLR), 2020.
Christopher J. Geoga, Mihai Anitescu, and Michael L. Stein. Scalable Gaussian process computations using hierarchical matrices. Journal of Computational and Graphical Statistics, 29(2):227-237, 2020.
Shenghan Guo and James C. Spall. Relative accuracy of two methods for approximating observed Fisher information. In Data-Driven Modeling, Filtering and Control: Methods and applications, pp. 189-211. IET Press, London, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR '16, pp. 770-778. IEFE, 2016.
11

---

Tom Heskes. On "natural" learning and pruning in multilayered perceptrons. Neural Computation, 12(4):881-901, 2000.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Comput., 9(1):1-42, January 1997.
Harold Hotelling. Spaces of statistical parameters. American Mathematical Society Meeting, 1929.
(unpublished. Presented orally by O. Ore during the meeting).
Zheyuan Hu, Zekun Shi, George Em Karniadakis, and Kenji Kawaguchi. Hutchinson estimation for high-dimensional and high-order physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering. 424: 16883, 2024. ISSN 0045-7825.
M. F. Hutchinson. stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics Simulation and Computation, 19(2):433 450, 1990.
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg. Huan Wang, Caiming Xiong. Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic Fisher explosion: Early phase Fisher matrix impacts generalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4772-4784. PMLR, 18-24 Jul 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Traaska-Barwinskee Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 14((1))3521-3522 2017. doi: 10. 10.1073/pnas. 1835114.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical Fisher approxima- tion for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4133 444.. Curran Associates, Inc., 2020.
D.N. Kupeli. Singular Semi-Riemunnian Geometry, volume 366 of Mathematics and Its Applications. Springer, Netherlands, 1996.
Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gradient algorithm. In Advances in neural information processing systems, volume 20, pp. 849- 856. Curran Associates, Inc., 2007.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. DBpedia A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web, (2):167-195, 2015.
Wu Lin, Frank Nielsen, Khan Mohammad Emtiyaz, and Mark Schmidt. Tractable structured natural- gradient descent using local parameterizzatons.. In Marina Meila and Tong Zhang (eds.), Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6680 6680-6691. PMLR, 18 24 Jul 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 1692, 2019. URL http:://axxiv.orrtte abs /000s//9900 11692.
Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitry Petrov, and Soundararajan Srinivasan. On surgical fine-tuning for language encoder. In EMNLP 2023, pp. 1-9. EMNLP, 2023.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76 2020.
12

---

James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factore approximate curvature. In International Conference on Machine Learning (ICML). pp. PMLR, 2015.
James Martens et al. Deep learning via Hessian-free optimization. In International Conference Machine Learning (ICML), volume 27, pp. 735-742, 2010.
Raphael A. Meyer, Cameron Musco, Christopher Musco, and David P. Woodruff. Hutch++: Optimal stochastic trace estimation. In Proceedings of the 4th Symposium on Simplicity in Algorithms (SOSA), pp. 142-155, 2021.
Frank Nielsen. The o-representations of the Fisher information matrix, 2017. https:// Franknielsen github. io/bl og/ a a paall inee.html..
Frank Nielsen and Gaëtan Hadjeres. Monte Carlo information-geomelte structures. In Frank Nielsen (ed.), Geometric Structures of Information, pp. 193..3. Springer International Publishing, Cham, 2019.
Yann Ollivier. Riemannian metrics for neural networks I: feedforward networks. Information and Inference: Journal of the IMA, 4(2):108-155) 2015.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In International Conference on Learning Representations, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035. Curran Associates, Inc., 2019. https:
https://pytorch: /tttto/ct..cc.ccco/ooooooooo.cht org/.
William Peebles, John Peebles, Jun Yan Zhu, Alexei A. Efros, and Antonio Torralba. The Hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of (ECCV) European Conference on Computer Vision, pp. 581 597, August 2020.
C. Rao. Information and the accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37(3):81-91, 1945.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108 108, 2019.
Maciej Skorski. Modern analysis of Hutchinson's 's trace estimator. In 2021 55th Annual Conference on Information Sciences and Systems (CISS), pp. 1-5, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, 2013.
Alexander Soen and Ke Sun. On the variance of the Fisher information for deep learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 5708-5716 Curran Associates, Inc., 2021.
Alexander Soen and Ke Sun. Trade-Offs of diagonal Fisher information matrix estimators. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 5870-5912, Curran Associates, Inc., 2024.
Michael L. Stein, Jie Chen, and Mihai Anitescu. Stochastic approximation of score functions for Gaussian processes. The Annals of Applied Statistics, 7(2): 162 1191, 2013.
Ke Sun. Information geometry for data geometry through pullbacks. In Deep Learning through Information Geometry (Workshop at NeurIPS 2020), 2020.
Ke Sun and Frank Nielsen. Relative Fisher information and natural gradient for learning large modular models. In International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 3289-3298. PMLR, 2017.
13

---

Ke Sun and Frank Nielsen. A geometric modeling of Occam's 's razor in deep learning. Information Geometry, 2025. Special Issue: Half a Century of Information Geometry, Part 2. Formerly titled "Lightlike neuromanifolds, Occam's 's razor and deep learning".
Shiqing Sun and James C. Spall. Connection of diagonal Hessian estimates to natural gradients in stochastic optimization. In Proceedings of the 55th Annual Conference Information Sciences and Systems (CISS), 2021.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 2019.
Ming Tu, Visar Berisha, Yu Cao, and Jae-sun Seo. Reducing the model order deep neural networks using information theory. In IEEE Computer Society Annual Symposium on VISI, ISVLSI, pp- 93-98, 2016.
Pete Warden. Speech Commands: A dataset for limited- speech recognition. CoRR, Abs//104.03329, 2018. URL http: / wrwaa.avvv.......a...a. owwg/ org/ abs 000p//899400209
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1, pp. 1 1112-1122. Association for Computational Linguistics, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the the-he art natural language processing. In Qun Liu and David Schlangen (eds.), Empirical Methods in Natural Lan- guage Processing (EMNLP): System Demonstrations, pp. 38 38-45. Association for Computational Linguisties, 2020. https: //e::///uuuue
co.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. PyHessian: Neural networks through the lens of the Hessian. In IEEE international conference on big data (Big Data), pp- 581-590. IEEE, IEEE Computer Society, 2020.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. AdaHessian: An adaptive second order optimizer for machine learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10665-10673, 2021.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutioaa networks for text classification. CoRR, abs/1509.01626, 2015.
A FURTHER ANALYSIS IN THE CORE SPACE
The lemma below gives the average error (variance) of using R(y) to estimate I (z), where y is a random variable distributed according to p(y 2).
Lemma 12. The element-wise variance of the random matrix Rly), denoted by Var(Rey). is given by
Var(Rey) = P:(1 (1 4p;(1- p:)) pp))
ii==;;
(PP P5 4p:p;)
otherwise.
Vi (ay( < 1/16. For both diagonal and off-diagonal entries, the coefficient of variation (CV) Std(Re,)/IIg 2)] can be arbitrarily large, where Std(-) means standard deviation.
Proof. We first look at the diagonal entries of R(y). We have
(1
p.)² yf-=;
R2 = ([y = pp))
pz
otherwise.
14

---

Therefore
[((,,)) = p.(1 -p:) + - p1 - p.(1 - p.) = (2). This shows that R4 an unbiased estimator of the diagonal entries of (z). We have
B(R pi(1 pp)² p4)p pi - p.) [[1 p.)² + płl
= p. p.) (1 (1-pp)= pi(l - p.) + +
Therefore
Var(R;;) E(R (B(R.)))
Pi(1 pi) [(1 - -p;)² -0pp-p1(11 p.) - p3 pi)²
= Pr((1 - pc) [11 (p)p;)² - 2pp - pc) + +
- =Pi(1 - pt)(1 41p - P.))
= =Z² 4Z
1
2
1
(x)
+
S
8
16
16"
The coefficient of variation (CV)
v //ar( Ra))
(
4IA (2)) (x)
4
A (z)
unbounded. As I + 0, the CV can take arbitrarily large values. Next, we consider the off-diagona entries of R. For i j. we have
R₂ = (lyy
(ly = -] Pý) = P:P; [v = =lp, - [V = =jllp.
Hence,
B(R;;) = P.Pj PPP; - PiP; = P:P; = Iz3 (=).
At the same time,
BRRR)) (pip, ly ilp; [v jlpp)²
piv) +E (Iy -lp² + [y = jllp 21 = 1] 11 2 = jlpŽr;)
= .r.2 + +p.T + P.P 2p?r3 2r
=PID) + +pPP; 3p
=PiPS (pi + Pj 3p.p;).
Therefore
Var(B.,) E(R?)) - (E(R;y))²
P.P; (p: + Pj 3p,p,) p..
PPPp (p: +p 4p.P,)
S PiPp (1 - -4p:P,)
1 2
1
- pips 8
16
16'
The coefficient of variation
(Va(R..))
PiP; (pi + P2 4p.P;)
[2)
pip}
Ps
Pj
is unbounded. As either p. + 0 Pj + O, the CV can take arbitrarily large values.
By Lemma 12, when using the rank-1 matrix R(y) as an estimator of (2),,, the absolute error is bounded, but the relative error given by the CV unbounded. One may alternatively use the rank-2 random matrix (v) Cyp pp" to estimate (2). Obviously
have TB ((R((y)) diag (p) pp" (2) and thus (v) is unbiased. The variance appears only on the diagonal, while all off- diagoiaao entries are deterministic with zero variance. This R' (y)) is not used in our
development but is of theoretical interest.
15

---

B AN ALTERNATIVE HUTCHINSON ESTIMATOR
We can rewrite the FIM in Eq. (1) as
dy p(y
I, 0) δ pLu I, 0)
F(0) - Σ
00
00
BED y=1
We define
h aqrt (Dx, =2 Σ
Σ
ply x, kays
(9)
ZED, y=1
where {Ery} are i.i.d, standard normal (or Rademacher) random variables. Then, we can use AD to compute
Σ
p(y x.
2
Exyx
00
00
RED
Then,
Sharrt Bh.qrt
Feqrt (0)
(10)
08
00T
gives an unbiased estimate of the FIM F(0), with bounded variance (details are straightforward and omitted for brevity).
This peqrt differs from two aspects:
It requires no detach () operation;
The square root can be avoided by noting
Vply 1oly11,0) exp 2
(x(x): log Expp(z (z,, (1,
where (x, 0) log E, exp(zy 0)) can be computed via PyTorch's log tog_axf method.
Esqrt is numerically more stable because it does not require clipping the operand inside the square root to be above zero. In experiments, however, notice little difference compared to F. All presented experimental results are produced using introduced in the main text.
C SUPPLEMENTARY EXPERIMENTS
Figure shows the distribution of the ground-truth diagonal FIMs of DistilBERT on SST-2, Distil- BERT on DBpedia, and EfficientNet-BO
CIFAR-I00. The classification head exhibits the largest Fisher information among all components at random initialization, whereas its Fisher information is comparatively small in fine-tuned models. In an early draft, we included experiments on DistilBERT for AG News (Zhang et al., 2015) topic classification (C = 4 classes), which has been streamlined to make space for other types of datasets and to present a more representative range of class counts C. All numerical results presented in this paper were obtained NVIDIA H100 SXM5 GPUs.
D PROOF OF THEOREM
Proof. Recall the closed-form expression for the simplex FIM:
I> = diag (p) - pp
Therefore
C
[=) = (diag (p) - -pp )e = p-
Σ
Pi
p = p p- p
16

---

Embedding (23.8M) TL01 (7-1M)
TL02 (7:1M)
TL03 (7.1M)
TL04 (7.1M)
TL0S (7.1M)
TL06 (7.1M) Classifier (592.1k)
c-0%
1-0%
Kw0%
Fu
10
10
16
10-9
(a) DistiBBERT SST-2
Embeddiig TL01
TLO2
TL05 (7.1M)
TL06 (7.1M) Classifier (601.4k)
pm3.6
Se
10
10
10tH
10"
10th
10th
20t
10:4
10th
10
10-4
20th
10
10
10th
(b) DistilBERT on DBpedia
EF1 (2.4k)
EF2 (63.4k)
EF3 (786.1k)
EF4 (3.2M)
Classifier (128.1k)
bwo%
Cec%
p=3.4e-03
Fi
1012t
10-1
(c) EfficientNet-BOO on CIFAR-100
Figure 2: Histograms of the ground-truth diagonal FIM entries Fx on a logarithmic x-axis. The zero atom is displayed as vertical bar the left edge of each plot. From left to right, successive components from input to output and their parameter counts are displayed. Ç denotes the zero probability. l denotes the average value of F& in the component. The solid and dashed vertical lines indicate the median and the pas quantile of strictly positive values, respectively.
Therefore {te E R) is the one-dimensional kernel of IZ). Since (2) t O, must have A1 = 0, and 11 = e/ e/|le]|
To analyze the of the eigenvalues of I have
Σ A (r( tr(diag (p)) tr(pp ) 1 tr(p pT p 1 ||||
Below, we consider the maximum eigenvalue Ac: We know that
Ac = sup u I (z)u.
|| 11
Therefore
Vi, de 2 TT = p:(1 - -p:)-
Therefore dc 2 max, P:(1 p.). At the same time, because A = O. we have
Σ x= A2 + + + S -C-
Therefore
أーー A
1
pp 2
A >
C
C
Because
diag (p) - 1 + +pp T
By Cauchy's interlacing theorem, we have
Ac- S SPC-1 A P(C)
17

---

It remains prove the upper bounds of Ac. First, we have
c
A = sup u (=)u. = sup |||||=1
Σ paut - u0T
I| |=1
sup 1| |=
p:u maxp; P(C),
which has just been proved using Cauchy's interlacing theorem.
By Gershgorin circle theorem, X must lie in one of the Gershgorin discs, given by the closed intervals
p1(1 p:)
Σ j#
Pips: Pa(1 p4)
Σ
PiPj
i= ,,. C.
Therefore
Ac I max pi(1 - -pi) + Σ P:Pj
J
max (pi(1 pi) + Pc pt)) maxpi(1 ps).
Because (1) 2 0.
Ac S >> 11
The statement follows immediately by combining the above lower and upper bounds of Aç-
E
PROOF OF LEMMA 2
Proof. Because A (z) = 0, all its eigenvalues are non-negative. We have
1
(x) Acaete Σ
aaaa 2 0.
i=1
We show that Acvent is the best rank- representation of I (z). Assume there exists u o such that (²) = uu 2 ACUCU Then
I (2)0C dc 2 (uou)² 2 Ac.
Therefore
viuu = =V/C. Assume that u = E a, then ac = vou = =VAc. Moreover, we have
u
Ac 2 | |ul
uu
||ul|2 = = a
|ul|
|||||
Therefore, vi + C, a; In summary, AvAue. Hence, uu ActoVe
We have
diag (p) (=) diag (p) (diag (p) pp pp > 0.
Therefore diag (p) I (z). Assume that diag (q) satisfies
(x) z diag (g) I diag (p).
Then
diag (p) I (z) = pp = diag (g) - - (2) 0
Therefore
diag (q) [2) = 3pp (B < 1).
Consequently,
diag (g) 7 (z) + 8pp diag (p) pp" T 3pp diag (p) + (9 1)pp T.
Therefore all off-diagonal entries of (3 1)pp" are zero. We must have 8 = and thus diag (q) = diag (p).
18

---

F PROOF OF LEMMA 3
Proof. We have
C-1
C-1
= Σ inaia
i=1
C-1
Σ x = ir(A(( Ac = 1 111p|1 Xc.
By Theorem 1, we have dc 2 PC- Therefore
|Aoeeet - (=)1 S - |lell - P(C-1)-
-
By Cauchy's interlacing theorem (see our proof of Theorem 1), we have
Vi {1, ,I,..., C -C-1.. 13. A P())
Hence
G-1
Mevvouy - (z)1l
Σ
X2
Σ Pla
The statement follows immediately by combining the above upper bounds.
G
PROOF OF LEMMA 4
Proof. The spectrum of R(y)
505 fey pl12.
The spectrum (2),, by assumption, is
S.... S. Ac-1 Ao.
By Hoffman -Wieland inequality, we have V2 E C-11 y E f1...,, CC:
[1((v) - -
Σ X3 + (Ac - 1le, 2|
2 |Ac - lle 11
|Ac e Ey p) T p + 2e T pl
T
IAc ||p|1 2p
max(Ac ||pl|2 2p
1 ||p|12 d 22,y
By Theorem 1, we have do I 1 - || One choose y so that Py P(1)- then
[[(y) - (2)1 2 1+ ||||22 do 2p(1)
こ 1+ ||p||2 - (1- - ||p)13) 2P(1)
= =2|1p||2 - -2p(1)-
19

---

H
PROOF OF PROPOSITION 5
Proof. Similar to Lemma 2, we have
C
Σ
xx) 3 diag (p).
C=C
Therefore
C
02
T
Oz
(2 T
O2
(0z
T
dz
Vx, 9
1
I (z(2,
0))
1
diag (p)
00
00
08
00
00
00
C-
Therefore
0z
T
Oz
02
T
8z
C
Oz Oz:
vO Σ ZED,
Σ 1-
Ai
0/44
Σ zED;
I (= 0))
Σ
Ps
08
00
06
00
00 06
xED, t=1
I PROOF OF COROLLARY 6
Proof. We first prove the upper bound. By Proposition 5, we have
0zg dz, Pt 00 00T
(0) =
JED
Taking the trace on both sides, we get
Oz, Oz, 00 0gT
tr(F S Σ Σ Pitr
ZED,
0z. Ozi a0 00
Pstr
ZED, i=1
Ozi 2
Σ ZED
Pi
00
The lower bound is not straightforward from Proposition
By Eq. (2), we have
(
0z 80
dz
dz
Oz
tr( tr(FA(()) = Σ tr
Σ tr zED
IA
00
00
00
ZED
Note that 20 80
is C' C matrix with sorted eigenvalues of 0) < S oz 0). By
Theorem 1, I (z) is another x C' matrix with sorted eigenvalues >1 (z, 0) S ... S \c(x, 0).
Applying the Neumann trace inequality, we get
tr (r) > Σ
Σ (c(2, RED=
-4
wED= 1=2
The last ">" " holds because all terms (2 0 2
(2, 0) are neg-negative.
J PROOF OF PROPOSITION 7
Proof. Denote the singular values of 0= 38 as 0 < ... Scc Then the eigenvalues of the C x C
Hermitian matrix 0= @zz) are of oz.
20

---

To prove the upper bound, we have
Σ
C
Ozi 00
T
dz;
F²)
Ps
00
RED, ==1
02
d2
Σ FED,
(diag (p) diag (p) + pp") )
00
a0
0x
0=
Σ RED
Pp
a0
00
Oz
T
Oz
Oz
T
Oz
Σ
pp
T
pp
T
00
00
00
00
2ED,
Oz
0z
0z
02
Σ ZED,
P
T
pp
T
p
00
00
00
00
Oz
82
p
P
30
00
ZED
Oz
02
Σ D
P
00
a0
Σ
||I|3
T
Oz
(0:
p
||e|| 00
08
|P
ZED,
Σ
zED.
Now ready to prove the lower bound. From the above, have
Σ
dzi
dz:
FA(D)
Σ ZED:
0z
Oz 00
pp
00
00
ae
>ED 1=1
Denote wx:)
(z))
p. Then
Oz,
T
0z:
Σ
f (8)
= Σ (x))
Pj
0
RED= i=1
DED=
Σ )(
tr
LED
(w(x)Tw(x))
RED
|k(())|
xED=
The last ">" follows from
tr (w(x)((z) w(x')w(z)))) = tr (w(x))Twr))(( (x())(())))))))))))))))))))))) Tw(')) = (o(z))((z)) 20 0.
21

---

K
PROOF OF PROPOSITION 8
Proof. First, we can have a loose bound:
Σ
Σ
0z
Oz
=²
00
00
rEDD C-++
02
Oz
0z
T
02
Σ
Σ
A
ii
Σ
I
(z)
00
00
00
00
RED C-
Dz
02
T
C- k
0z
Σ
Σ
00
00
DED2
C-K
O2z
8z
P(C-R)
(Due to that
Σ
Aerey 3P(C-))
08
BED2
0z
12))
Σ
PCC
00
00
DED
@a= The eigenvalues of 00 (2a) 7) are of s... oz. We have
(0=)
02
CC-
02
Σ AeaEy
00
08
02)
T
C- k
Oz
02
T
C-
0z
Σ Aaare,
Σ
00
00
00
00
0=
02
C-
27
Σ
T
=tr
00
00
2
0z
dz
<tr
(Due to tr(A)) < (tr(A213)))
00
00
2
0z
0z
Σ
X² X2
(Note λ 0)
00
00
k
A7
1=2
The last "<" is due to von Neumann's 's trace inequality. We also have the Cauchy interlacing
d S P(2) Aa S P(3) .... S SA-11 S P(C-1)-
22

---

To sum up,
0z
O=
Σ
(0)
30
00
RED,
k+1
02
Oz
iiaaay
00
30
XED
C-k
Σ
xED,
i=
C-k
Σ
Σ
S+kP
=
2=
If one relaxes vi E .., C - k), P(t) S P(C-N), then we get the loose bound proved earlier.
L PROOF OF PROPOSITION 9 Proof.
Oz
dz
02
0z
|F(0)) - -0]l==
(0)lo=
Σ RED,
I(=(,,0))
Σ
(ey -))ey p)
00
00
RED,
Σ
0z a0
[diag (p) -pp - (ey () - T
0z
00
IE D.
(0x)
(02)
0z
Σ
[diag (p) -pp T - -ey p)(e T
(8)
00
RED_
82
02
< Σ
||diag (p) -pp - -(ey (e p) 11.
00
00
zED,
Σ
diag (p) ppp (ey (ee - p) Tl.
zED.
Now we examine the matrix diag (p) ppd T (ey - (e - T) By Theorem 1, the spectrum of
diag (p)
pp is
\ = 0 O X2 S Ac.
By the Cauchy interlacing theorem, the spectrum of diag (p) - -pp - -ey - -p)(ey - T,) ,givnn by
X1, ... Xz, must satisfy
X S = 0 X/2 S X2 ... ..... S dc
with at least one non-ivsitive eigenvalue: X₁ < 0. Therefore
diag (p) pp (e, (), p) TIl maxt - X2, Ac).
We also have
Xi inf T [diag (p) -pp (ey p)(c p) - w:
> inf -uT [(ey - (e - 1
w:||u|||
= -(ey - p) p))((ep (ey p)
= =-(1 + pp T
p 2p,)
2py 2!pl?2
23

---

Therefore
diag (p) pp" -Cey (p)c- - 1) max (1 + ||p|2 2py, Ac}
< max 1 + llal12 2py. - |11|1
$ 1 + ||Pl.
In summary,
|F(00) |=(0)-(0))
(0)llo S Σ o8 + +lpl)
DED
M PROOF OF PROPOSITION 10
Proof.
(
(02)
dz (z(x,
10z)
d2
I
I
(z(0,
01)
00
00
30
30
(02
(I(xx(1)) I (z(x, 0)]
02
00
00
15
(
02
T
0z
[diag (v) - ppp' (e, - p)e - p) []
00
00
(02 00 u
T
02
||diag (p) - pp T - [e,, - (p) -p]
sup
a0
a:||a -1
2 sup
(1)U [diag (p) - pp T (eg - -p) - p) [] ocjool
o: |v|[==
20(1)
|ddaa (p) - pp T - (eg p)(ey - -p) -) l|a
(lle, c (fee
20
P p
-pll (ley - p)ley =) - >cc)
Ey -p |ee, pll
=o |1ly - 2||² Acl
=0()
1+ ||p|1 Ac - 2pyl
We choose Py P(1), therefore Fy such that
02
I (z(zz 0))
02
02 T
I
(z(x,
az
00
00
00
00
2011 [1 + 1|p12 Ac - -2p(1/-
N PROOF OF THEOREM 11
Proof. From the derivations in the main text, already know that Epte) F(0) F(0). To show the estimator's variance, we first consider the case when p(E) is a standard multivariate Gaussian distribution. Note that both hDD., 0) and 0h/00, are the form of sum of independent Gaussian random variables. Hence,
ab 00;
Elay Exy G
(01,
(Olay) 2
Σ ZED,
Σ V=1
ply x, 0)
0,
Σ DED,
Σ
ply x,
00
a0,
24

---

Therefore
IE P(C)
Oh
Σ
Σ
p(u x. 0)
(
Olay
F:((0);
30;
00:
wED=
I P(E)
(F).
a0,
Therefore
0b
Var(F(O)))
E p(C)
F((0)= - 2F
a0;
We now consider that p(E) is Rademacher.
Ø6
Oh 00,
2
Var(F(O.))
E P(C)
IE
00,
06
E p(E)
(0)
00,
1
Olzv 00,
E m(E)
Σ
v p(y x. B)
Eas
- (((
LED x=1
[Plzy
Σ
a0,-
ZED, v=1
[bly
2
(ale's' 2 08;
+3
Σ
plu x.
p(y |x'
F2).
00:
Note that
Hlay a0,
Fi(0)
Σ Σ
p(u xx 10y1x,00
BED yml
(bay
(Dlay) 2
(Dla'g'
= Σ Σ
p2 x, 0)
Σ (x.))
p(yl x, 0)
p(y' (2',
(000
00,
00;
00;
D. 1
Hence,
(8ly 00;
Var(F(0,))= 3F -2 Σ Σ
p (y x, 0)
F2((0)
SED, y=1
(bly
4
2 - Σ
p (y) xx, 0)
00,
DED 8=1
O ACCURACY OF HUTCHINSON'S ESTIMATE ON DIAGONAL AND LOW-RANK CORES
In this section, we show that Hutchinson's 's estimates (D) and [Lll(() are both unbiased with bounded variance.
Proposition 13. The random matrix FDG(() is an unbiased estimator of (D) The variance of
its diagonal elements is Var (RDG (0)) 2(JDG (0))2 2 2x E 27 0){ 25
y1
25

---

Proof.
Ohh [0bDc
DG
ah DG
E (LDG(0))
= E P(C)
00
00
p(C)
azy
Dāy
E P({)
Sylaz 0)
Say
Σ
Sy (x', ,0
2000 Exr
D0
RED, y=1
x'ED y'
C
Ozy Dzg'
Σ E
V O) (5' (x',0)
E (EnyEry)
00 00
plg)
RED, F'EDD y'=1
Σ
(I 0)
Ozy Ozy 30 00
RED, y=1
022
I (z(zz, 0)) 8z
00
00
EED (D)(())
Therefore,
DG
C ΣV
(8zy)
2
E
(FDG(()
E P(E)
Σ
Ca 0)
F (0).
00;
08;
P(E)
RED,
36 DG
Dzy Sum
E P(E)
E P(E)
Σ
V (x, 0)
00;
30;
DED y=1
(iiy
(Oāy) 2
(aay 2
)
Σ
(xx)
+3
Σ
Cx(x,))
So (x', 0)
00;
00:
00;-
FED,
O2y 00,
3(F - Σ
(x,, 0)
RED= y=1
Hence,
DG 1
Var (ar(FDG(()) = E
(FDG(0))3
00,
p(C)
C
(0zg) 4
=2(FRG (0))2 -2 Σ Σ (5, 0)
zED.
v=1
Proposition 14. (L) is an unbiased estimate of FLL((O): the variance of its diagonal elements is Var (FLR (0)) = (0))² - 2 2.EED, 1z(x, 0) ( 0)
Az
89.
26

---

Proof.
The proof is similar to Proposition 13 and is also based Hutchinson's 's trick.
E (FLB(()
P(E)
Ohhh LR
06 TR
= E
p(S)
00
00
002)
/Acls'0)ecll ()T (0z
Σ (xcr1,0)
vc(2, Er
Σ
Exr
p())
(8
(88)
x'ED=
(0z)
(02)
02
(x)
vo(z, (2, 0)
00
TED = =FUR (0).
Therefore
((03)
0z
T
vo 0) 01)
2
E
(FLR (0)) = E Ac(r, 0)
(88,)
ve
F(L((() (0):
P(()
ZED,
Oh LR 4
Oz
E P(E)
E'
Σ LEDD2
Vocl,, ( 0)
Eo
00;-
P(E)
00:
0z
Σ \c(2, (v0(2,0) 0)
00:-
ZED,
3 A2( 0) (10(1z,)) 0) Oz
Ao(x',0) (volae 0)
0z
)²
00,
00,
xFx
Oz
=()) - Σ 0)(x, (
0)
00:
ZED.
Hence,
( @hh Lit
Var (PIR = E
- (O))
p(E)
00:
0z
((FFR((0)) - Σ 1x, 0) (vo(z, 0)
00;
2ED.
< 22 by Proposition 13, and at the same time, we have Std(F)RR (0) S v by Proposition 14.
We have Std(FDG i (0)) Std(FDC(0))/=1)
Thus both estimators have theoretical guarantees on their estimation quality (CV bounded by 2
27


Reviews:
[{"review_id": "Ay9oxTdine", "reviewer": "Reviewer_YTgp", "summary": "The paper analyzes the Fisher Information Matrix (FIM) for neural classifiers through deterministic bounds based on spectral analysis and proposes a novel Hutchinson-based estimator. The theoretical development is rigorous with complete proofs in the appendix, though experimental validation remains limited.", "questions": "Technical Concerns\n\nComputational complexity underspecified: Power iteration convergence for computing λ_C, v_C could be slow near uniform distributions (small spectral gap). The O(MC|𝒟_x|) cost needs quantification.\nImplementation gaps: No guidance on choosing probe count for target accuracy, mini-batch handling, or numerical stability considerations.\n\nMinor Issues\n\nNotation overload (multiple uses of F̂ for both empirical and MC Fisher)\nSome bounds are loose (e.g., diagonal upper bound always has error ≥ 1/C). Correct me if im wrong/misinterpret\n\nQuestions for Authors\n\nWhat is the wall-clock time comparison with K-FAC for equivalent accuracy?\nCan you demonstrate improved optimization performance in a practical setting?\nHow does mini-batching affect the variance bounds?\n\nVerdict\n\nThe paper makes solid theoretical contributions with rigorous mathematical development. However, it lacks the experimental validation needed to demonstrate practical value. The theory is great and extensive, but the work feels a little missing in terms of comprehensive experiments showing real optimization improvements. But I am also open to understanding if this mainly geared to a paper of more theoretical build up value (or step in that direction) than a fully practical ready usage.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Solid mathematical foundation: All major theorems have complete proofs in the appendix. The spectral analysis of the simplex FIM (Theorem 1) and envelope characterizations (Lemma 2) are rigorously established.\n\nNovel computational approach: The Hutchinson estimator with “detach-and-mix” construction provides unbiased FIM estimates with provably bounded diagonal CV ≤ √2, addressing variance issues in Monte Carlo methods.\n\nComprehensive theoretical analysis: The paper provides both deterministic bounds and stochastic estimates with detailed variance analysis for Gaussian and Rademacher distributions.", "weaknesses": "Experiments: Only DistilBERT tested on AG News and SST-2. Missing: (i) comparison with K-FAC, diagonal empirical Fisher (Adam), exact Gauss-Newton; (ii) optimization performance metrics; (iii) other architectures (CNNs, ResNets). Just I guess a general expansion in this area would be nice, though you note each formulation may require a from-scratch derivation for each. Please comment on this\n\nUnclear practical advantage: No demonstration of how the estimator improves optimization or learning dynamics compared to existing methods.\n\nTerminology issue: “Singular semi-Riemannian metric” is incorrect I believe - the FIM is degenerate PSD, not indefinite.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtncScMyUK", "reviewer": "Reviewer_u7Fy", "summary": "The authors propose an unbiased estimator for the Fisher metric in the parameter space of a neural network, in the setting of multiclass classification, where the underlying geometry is determined by the simplex. The main estimator, along with two additional approximations, is straightforward to compute and leverages automatic differentiation. The authors provide a theoretical analysis to analyze the accuracy and some properties of the proposed estimators. The theoretical claims are further supported by empirical demonstrations.", "questions": "Q1. It would be helpful to include a comparison with a Monte Carlo approach in terms of both computational complexity and accuracy.\n\nQ2. Is it correct that the LB estimator typically performs better when the predicted distribution approaches one of the corners of the simplex (see Lemma 3)?\n\nQ3. I suggest including the true Fisher information in the experimental comparison, at least in settings where it is feasible.\n\nQ4. A simple synthetic example where the full Fisher information matrix can be computed and visualized would help demonstrate the behavior of the estimator.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of approximating the Fisher metric in neural network settings is timely and very important. The proposed estimators are simple and can be easily computed in practice.\n- The technical part of the paper appears to be sound and the theoretical claims correct, but I have not followed every proof in detail. The experiments support the theoretical analysis.\n- The paper is generally well written and accessible.", "weaknesses": "- Since the paper proposes an estimator for the Fisher metric, I would expect additional empirical demonstrations regarding the estimator’s accuracy and efficiency. I acknowledge that the theoretical results constitute the primary contribution, but I believe that further benchmark experiments would help demonstrate the properties of the estimator and also consider comparisons to related methods. The current experimental section provides some insight, but additional comparisons would strengthen the paper.\n- As the authors mention in Section 5, Hutchinson-type estimators have been used in related contexts. While the extension to neural networks is appreciated and well motivated, as a non-expert in Fisher information analysis, I cannot argue which of the theoretical contributions are novel and which may already be known.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PatpJvipfo", "reviewer": "Reviewer_rVQk", "summary": "This paper proposes deterministic bounds and Hutchinson-based stochastic estimators for the Fisher Information Matrix (FIM) on neural manifolds. The method provides unbiased, bounded-variance FIM estimates requiring only one backward pass, making it scalable to large networks. Experiments on DistilBERT show feasible and stable approximations compared with empirical and Monte Carlo estimators.", "questions": "see wk", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Provides a unified and theoretically grounded framework for deterministic and stochastic FIM estimation with clear variance guarantees.  \n- The Hutchinson estimator is efficient and easy to integrate into deep learning pipelines, enabling scalable information-geometric analysis.", "weaknesses": "- The paper does not validate the estimator on analytically known probability distributions, especially in high-dimensional settings. This makes it difficult to verify the claimed accuracy of the proposed bounds and stochastic estimates.\n- The practical impact of improved FIM accuracy on downstream tasks (e.g., optimization, generalization) is not demonstrated. In some real applications [a], approximation may even be preferable to precision, since neural networks themselves are inherently approximate. The authors should strengthen the discussion and empirical evidence on how their estimator benefits real learning tasks.\n\n[a] Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LKcWFRnY23", "reviewer": "Reviewer_bqwM", "summary": "This work presents a new approach along with theoretical results for estimating the Fisher Information Matrix (FIM) in deep neural networks. The deterministic upper and lower bounds were examined via the geometric properties in low-dimensional probability simplex spaces, followed by extending to high-dimensional parameter spaces using pullback metrics. This estimator can be computed with just a single backward pass, providing significant computational benefits over existing Monte Carlo and empirical FIM methods for large networks like DistilBERT.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical results are sufficient and rigorous.\n\n2. The ease of implementation in standard deep learning frameworks, which is evident by the empirical validation with better MAE while keeping computational costs similar. The framework's independence from architecture and its ability to apply to both diagonal and low-rank approximations offer flexible trade-offs between accuracy and computational cost.", "weaknesses": "1. Empirical validation relies solely on DistilBERT and the results from two text classification datasets. How computational overhead scales with network depth and width, memory needs for large models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)