Trade-Offs of Diagonal Fisher Information Matrix Estimators
Authors: Alexander Soen
Ke Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Fisher Information
Deep Learning
Information Geometry
Neuromanifold
Primary area: learning_theory
TL;DR: We bound the variances of random estimators of the diagonal Fisher information matrix.

Abstract: The Fisher information matrix can be used to characterize the local geometry of
the parameter space of neural networks. It elucidates insightful theories and
useful tools to understand and optimize neural networks. Given its high
computational cost, practitioners often use random estimators and evaluate only
the diagonal entries. We examine two popular estimators whose accuracy and sample
complexity depend on their associated variances. We derive bounds of the
variances and instantiate them in neural networks for regression and
classification. We navigate trade-offs for both estimators based on analytical
and numerical studies. We find that the variance quantities depend on the
non-linearity w.r.t. different parameter groups and should not be neglected when
estimating the Fisher information.

Trade-Offs of Diagonal Fisher Information Matrix Estimators
Alexander Soen
Ke Sun
The Australian National University
CSIRO's Data61
RIKEN AIP
The Australian National University Sun@dat dal.6l..a......... csiro.ar
a eeexnder.soeneang
Abstract
The Fisher information matrix can be used to characterize the local geometry of the parameter space of neural networks. It elucidates insightful theories and useful tools to understand and optimize neural networks. Given its high computational cost, practitioners often use random estimators and evaluate only the diagonal entries. We examine two popular estimators whose accuracy and sample complexity depend on their associated variances. We derive bounds of the variances and instantiate them in neural networks for regression and classification. We navigate trade-offs for both estimators based on analytical and numerical studies. We find that the variance quantities depend on the non-lineane non-line w.r.t. different parameter groups and should not be neglected when estimating the Fisher information.
1 Settings
In the parameter space of neural networks (NNs), i.e. the neuromanifold [1], the network weights and biases play the role of coordinate system and the local metric tensor can be described by the Fisher Information Matrix (FIM). As result, empirical estimation of the FIM helps reveal the geometry of the loss landscape and the intrinsic structure of the neuromanifold. Utilizing these insights has lead to efficient optimization algorithms, e.g., the natural gradient [1] and Adam [16].
A NN with inputs and stochastic outputs y can be specified by a conditional p.d.f. p(y a;; 0), where f is the NN's weights and biases. This paper considers the general parametric form
ply x: 0) = π (y) exp (t ha))) - (xh)
where hg: R - R maps I-dimensional inputs @ to 7'-dimensional exponential family param- eters, t(y) is a vector of sufficient statisties, Tyy is base measure, and F()) is the log-partition function (normalizing the exponential). For example, if y denotes class labels and ((y) maps to its corresponding one-hot vectors, then Eq. (1) is associated with multi-class classification network.
Assuming that the marginal distribution (zx) is varameter-free, we define parametric joint distribu- tions p(r, y: 0) q(x:) (w x: 0). The (joint) FIM is defined as I(0) = EEg(()) [I(@] =)), where
i log ply x: 0) log p(y x;; 0)
0 logp(y x;; 0)
I(0 x:)
E p(al
E
00
00
p(u a:
0000
is the 'conditional FIM'. The second equality (*) holds if ho activation functions are in (2R) (i.e., he sufficiently smooth NN). I(O x:) does not have this equivalent expression (*) for NNs with ReLU activation functions [37]. Both I(0) and I(O x) define dim (0) x dim(0) positive semi-definite (PSD) matrices. The distinction in notation is to emphasize that the joint FIM (0) (depending only on 0) is simply the average over individual conditional FIMs I(O x)) (depending on both 0 and x:)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Text (NLL)
Regression Gaursian)
Classification (Categorical)
Traedoo
NG with Z.
0.70
NG with 2n
Initial Paramn d, Params
0.65
64
198
100
01
37
125
255
Training Epoch
Training Epoch
Figure 1: Natural gradient (NG) descent using / on a 2D toy dataset for regression (linear regression) and classification (logistic regression) (details in Appendix A). Inset plot shows the parameter updates throughout training. Here, the variance of (0) is generally lower than I, (0).
In practice, the FIM typically computationally expensive and needs be estimated. Given g(x)) and a NN with weights and biases 0 parameterizing p(y x; 0), as per Eq. (1), we consider two commonly used estimators of the FIM [11, 37] given by
(0) Σ
N
e Olg yv mx)) i logopyy (Fa)
and [0)
N Σ
8 log plye ax))
(3)
00
00
0000
where ply (x)) = p æx 0) and (x1, yi).... (ØN, yN) are i..dd sampled from p(æ, y; 0). A
conditional variant the estimators, denoted Ż, (0) a)) and (2 æ)) can defined by fixing
xN and sampling y)..... yN independently from p(y a:) in Eq. (3) details omitted for brevity.
= 11
Both estimators, (0) and (0), are random matrices with the same shape as I(0). By Eq. (2), they are unbiased - for (0), this only holds if activations functions are in C2 (8). Following Eq. (1)'s setting, the estimation variances of I, (0) and Σ2 (0) can be expressed in closed form and upper bounded [37]. This provides an important, yet not widely discussed, tool for quantifying the estimators' accuracy [11] and hence insights for where when different estimators should be used. Despite this, for deep NNs, neither these variances nor their bounds can be computed efficiently due to the huge dimensionality of 0.
This work focuses on estimating the diagonal entries of the FIM and their associated variances. Our results - including estimators of the FIM, their variances, and their variance bounds - can be implemented through automatic differentiation. These computational tools empower to practically explore the trade-offs between the two estimators. For example, Fig. shows natural gradient descent [1] for generalized linear models on a toy dataset, where (0) is preferable (especially for regression) and Ż, (0) suffers from high variance and an unstable learning curve. Our analytical results reveal how moments of the output exponential family and gradients of the NN in Eq. (1) affects the FIM estimators. We discover general decomposition of the estimators' variances corresponding to the samples of æ and y. We investigate different scenarios where each FIM estimator is the preferred one and then connect our analysis to the empirical FIM.
2 Related Work
Prior efforts aim analyze the structure of the FIM of NNs with random weights [34, 14, 15, 3, 31]. This body of work hinges on utilizing tools from random matrix theory and spectral analysis, characterizing the behavior and statistics of the FIM. One insight is that randomly weighted NNs have FIMs with majority of eigenvalues close to zero; with the other eigenvalues taking large values [14, 15]. In our work, the randomness stems from sampling from data distributions p(æ, y) which follows the principle of Monte Carlo (MC) information geometry [29] that approximates information geometric quantities via MC estimation. We examine a different subject on how the distribution of the FIM matrix manifold is affected by finite sampling of the data distribution.
In the literature of NN optimization, main focus on deriving a computationally friendly proxy for the FIM. One can consider the unit-wise FIM [30. 20, 39, 3] (also known quasi-diagonal FIM [30]), where a block-diagonal approximation of the FIM is taken to capture intra-neuron curvature information. Or one can consider the block-diagonal layer-wise FIM where each block corresponds to parameters within layer [19, 27, 26, 12, 35, 13). NN optimizers can approximate the inverse FIM [36] or approximate the product of the inverse FIM and the gradient vector [35].

---

Much less attention is paid to how related approximations deviate from the true FIM [11, 37] or how optimization is affected by such deviation [41]. For the univariate case, one can study the asymptotic variance of the Fisher information [11] with the central limit theorem. In deep NNs, the estimation variance of the FIM can be derived in closed form and bounded [37]. However, our former analysis [37] has two limitations: (1) the variance tensors are 4D and can not be easily computed; (2) only the norm of these tensors are bounded, and it is not clear how the variance is distributed among individual parameters. The current work tackles these limitations by focusing
the diagonal elements of the FIM. Our results can be computed numerically at a reasonable cost in typical learning settings. We provide novel bounds that one quantify the accuracy of the FIM computation w.r.t. individual parameters or subgroup of parameters.
Issues of utilizing the empirical FIM to approximate the FIM have been highlighted [32. 25]. For example, estimators of the FIM do not in general capture any second-order information about the og-likelihood [18]. The empirical FIM is a biased estimator and can be connected with our unbiased estimators via a generalized definition of the Fisher matrix in Section
Alternative to the FIM, the Generalized Gauss-Newton (GGN) matrix - a Hessian approximator
was originally motivated through the squared loss for non-linear models [25]. The GGN is equivalent to the FIM when a loss function is taken to be the empirical expectation of the negative log-likelihood of Eq. (1) [12, 32, 25].
3 Variance of Diagonal FIM Estimators
In our notations, all vectors such as æ, y. and are column vectors. We use i to index random samples æ and y and use and index the NN weights and biases O. We shorthand h - he. p(y æ)) = pyy æ;; 0), and p(y, x)) = p(y, x; 0) whenever the parameters O is clear from context. distinguish between jointly calculated values versus conditioned values with fixed a. By default, the derivatives are w.r.t. O. For example, 0,h = ah and h = /P// We adopt Einstein notation to express tensor summations, so that an index appearing as both subscript and a superscript in the same term indicates a summation. For example, x= ya denotes Σ. x= za For clarity, we mix standard Susum and Einstein notation. We denote the variance and covariance of random variables by Var(-) and Cov(-), respectively.
To be consistent, we use 'I x conditioning'
Based on the parametric form of the model in Eq. (1), the diagonal entries of the FIM estimators in Eq. (3) can be written as'
(0.) = (Z)(0))
1
N
OF(h(xx)
2 @h*(xx) ta ((vs))
N
00:
00:
;
(2,))) (Ia(0)
1
Σ
N
(PFF()())))
8 (h)(ax)
to (a())))
i2 N
8PA
020.
Correspondingly, the i'th diagonal entry of the FIM I(0), which is the expected value of (0,, and (0.), is denoted as (0,)). Notation is abused in Z(O.). I, (0,). and (2)),) as they depend on the whole O vector rather than solely on 0. Clearly (0.) 2 O, while there is no guarantee for (0.) which can be negative. Our results will be expressed in terms of the (central) moments of t(y):
na (z)
=
E [v));
I(h x)) IE
[(((g) - ()())) - (7(x)) "]:
p(u
x)
p(y x)
(C x))
E
[(t(v) - n(x)) @ (t(y) - (=x)) @ (t(y) - (=))) @ (t(y) n(x)))
p(y x))
where "s" denotes the tensor product. We denote the covariance of w.r.t. to p(y ax) as Cov a)) noting that I(h| x) = Cov"(t æ). The 4D tensor KCt(t) x)) denotes the 4th central moment of
t(y) w.r.t. p(y x). These central moments correspond to the cumulants of t(y), i.e. the derivatives of F w.r.t the natural parameters h(xx) of the exponential family. Therefore, the derivatives of F in i, (0.) and (0,) can further be written in terms of ()))) and I(h a)) following the chain rule. Practically, ī, and Iz involves computing the Jacobian (z)) 000 and the Hessian d2 h( respectively.
'This and subsequent derivations be found in the appendix.
3

---

Table I: Exponential family statistics with eigenvalue upper bounds for moments. For classification, a (z) denotes the softmax of logit h(z). f denotes exact eigenvalues rather than upper bounds.
Setting
Exp. Family
Ouiput y
Sufficient Statistic t(v)
UB Xmax (1 x))
UB Amax (AC(t (x)
Regression
(Iso.) Gaussian
y
3 2 min max (ar)
min ( (a). - (=)
Classfication
Calegorical
[C]
C
([y = oJ......U bV c
1
[ 113)
In practice, both estimators can be computed via automatic differentiation [33, 6]. In terms of complexity, by restricting to just the diagonal elements I(0.), we need to calculate (0) elements (originally (dimm(0) x dim(()) for the full FIM). Although the log-partition function for general exponential family distributions can be complicated, for the ones used in NNs (determined by the loss functions used in optimization) [37] the log-partition function is usually in closed-form: and thus the cumulants (zxx) and [(h æ) can be calculated efficiently.
Indeed, the primary cost of the estimators comes from evaluating the gradient information of the NN, given by (h(z)) and h0)(()))))))))))))))))))) //)/0/ The former can be calculated easily. The latter is costly even when restricted to the diagonal elements of the FIM. With the Hessian's quadratic complexity, in practice approximations are used to reduce the computational overhead [4, 45, 46, 8]. In this case, additional error and (potentially) variance may be introduced as a result of the Hessian approximation. Note, the computational cost of the Hessian can still be manageable for the last few layers close to the output. By the chain rule, we only require a sub-computational graph from the output layer to a certain layer to compute the Hessian of that layer. Despite this, there is still memory cost that scales quadratically with the number of parameters for non- non-linear activation functions [6].
The high cost of Hessian computation does not justify refraining from using 1z. Depending on the setting (chosen loss function), an estimator's 's variance can outweigh the benefits of lower compu- tational costs [37]. This is especially true when the FIM is used in an offline setting - where the Hessian's 's cost can be tolerated - to study, e.g., the singular structure of the neuromanifold [2, 40], the curvature of the loss [7], to quantify model sensitivity [28]. and to evaluate the quality of the local optimum [14, 15), elc.
To study the quality of (0) and (0), it is natural to examine the variance of the estimators [37]: (, æ²) = (ar(I, æ)), where (0: x) = (I, m)) Ú E (1,2)) is the i'th diagonal element of (0) æ). Similar to 2, (8,) and (2,).. (0, xx) and (0, æ)) depend on the vector and are abuses of notation. An estimator with smaller variance indicates that it is more accurate and more likely to be close to the true FIM. Based on the variance. one can derive sample complexity bounds of the diagonal FIM via Chebyshev's 's inequality, see for instance [37, Section 3.4].
By its definition, V, (0: a)) has simple closed form, which was proved in [37) and is restated below. Lemma 3.1. Vo E Re Vi = 1,....,dd. dim(O).
I(O,| x) = o, (x: 0 1 (x) (h x)) 1
(4)
(1 m) N
0,h 0 (z) O0) 0, (x)).h"')) [Ka Inb(h x)) Iea(h æ)], (5)
1
(2(0, (:) = N (=)0 Ias (x)
Given fixed E Rt both (0, (x) and (2 x)) have order of OLL/N), with N denoting the number of samples of yv. They further depend on two factors: 1 the derivatives of the parameter- output mapping 0 + h stored in a T' d0m(O) matrix, either 0, or h where the latter can be expensive to calculate; and @ the central moments of t whose computation only scales with T (the number of output units) and is independent to di).
From an information geometry [1] perspective, I(0). (0), and (2) are all pullback tensors of different orders. For example, I(0) is the pullback tensor of I(h) and the singular Riemannia metric [40]. They induce the geometric structures of the neuromanifold (parameterized by 0) based on the corresponding low dimensional structures of the exponential family (parameterized by h).

---

Practical Variance Estimation
To further understand the dependencies of the derivative and central moment terms, the FIM I00 |
and variances of estimators (0; ax) can bounded to strengthen intuition and to provide computationally convenient proxy of the interested quantities.
Theorem 4.1. Vx E R²,
[a,h(m)) Amin (I(h) x)) S I(0, 2:) S [0,h(x))13 Amax (I(h) 2)))
(7)
1 N |(a(())1 Ami (M) S V1 (0,12)
S N 11a,(())) Amax (M),
(8)
N [00h(2)]l Amin (I x)) S V2(0, (x) S N 100h(x) Amax ( x)), =
(9)
where == (t x) I(h x & I(h x); Amin / Amax denotes the minimum maximum matrix eigenvalue; and Amin, Amax: : R RTxTxTXT + R are defined as
Amiu (T) = inf u u un Tabed; and Amax (T) = sup u:||u||a=1
u un "uuuurr u w:||u||2=1
Tabed-
(10)
To help ground Theorem 4.1, summarize different sufficient statistics quantities for common learning settings in Table - with further learning setting implications presented in Section 5. Note that Eqs. (8) and (9) (and many subsequent results) can be further generalized for off-diagonal elements. See Appendix C for details. Compared to prior work [37], Theorem 4.1 provides bounds for individual elements of the variance tensors, where the NN weights (the derivatives) and sufficient statistics (the eigenvalues) are neatly disentangled into a product. From a technical point of view, this comes from difference in proof technique: utilize variational definitions and computations of cigenvalues to establish bounds whereas [37] primarily applies Hölder's 's inequality.
We stress that Amin ( (T) and À (T) in Eq. (10) correspond to tensor eigenvalues iff T is a super- symmetric tensor [23] (a.k.a. totally symmetric tensor), i.e., indices permutation invariant. In this case, Eq. (10) is exactly the maximum and minimum Z-cigenvalues. These variational forms mirror the Courant-Fischer min-mat max theorem for symmetric matrices [42]. In the case of Eq. (8), with M = (t æ)) I(h x) & I(h æ:), the tensor is not a supersymmetric tensor in general. Despite this, we note that the lower bound of Eq. (8) is non-trivial. A weaker bound than Eq. (8) can be established based on the Z-eigenvalue of the supersymmetrie tensor (t)
Corollary 4.2, Va E RA
Amin (IP x)) I(h| x) @ I(h x)) 2 max f0, Amin (KC((| > (I(h[z))
(11)
Kmax (K x) I(h x)) & I(h| x)) < Amax (KP x)) - 2mn (I(h|x:))
(12)
The tensor eigenvalue is typically expensive to calculate. However in case, the eigenvalues x)) (t and (XCP(t æ)) on the RHS of Eqs. (11) and (12) can be calculated via [17]'s method with ((T4/4!) complexity. In this paper, we assume T is reasonably bounded and are mainly concerned with the complexity w.r.t. df) From this perspective, all our bounds scale linearly w.r.t. dim ()), and thus can be computed efficiently.
When t(y) - (ax is bounded (e.g. in classification), we can upper bound Amax (JCP(t x)) with Amax (I(h) x)), which is casier to calculate.
Proposition 4.3. Suppose [t(yy) (xx) (3xx) | < B. Then, Smax (KP(t| S BAmax (I(h) x)) S B²
As long as the sufficient statistics t(y) has bounded norm || we have that |ty) ( <
4|lt||2 < Do. A similar lower bound can be established for the minimum tensor eigenvalue Amin (IC* (t x)) 2 Xa (I(h) x)), but this ends up being trivial when applying Corollary 4.2's lower bound, Eq. (11).
Examining Theorem 4.1 reveals several trade-offs. An immediate observation that the first order gradients of fzx)) correspond to the robustness of h to parameter misspecification (w.r.t an input
5

---

x) and (1(0; | æ)) will be large when small shifts in parameter space yield large changes in the output h (x).. Another observation is how the spectrum of [[h a)) affects the scale of I(O; a:) and the estimator variances. In particular, when Amin (I(h) æ)) increases, the scale of Vi (0. æ)) decreases but the scale of I(0; 2)) and V22 [0, æ:) increases. When Xmax (I(h æ)) decreases, then the opposite scaling occurs. With these two observations, there is a tension in how the scale of [0, x) follows the different variances (0 æ) and V22 [0, æ)). The element-wiss FIM [(, æ)) follows V (0; (x) in terms of the scale of
2). As such, from the bounds in Eqs. (7) and (8), the scale of [(0;
NN derivatives (0, (2))a at the same time, I(0; x)) follows (2200; x) in terms of the spectrum of sufficient statistics moment I(h x))
Remark 4.4. Typically, h is the linear output units: h(x)) W-1h (a:), where W-1 is the weights
of the last layer, and h-1 1h_(()) (a) is the second last layer's output. We have V2(0; x)) = 0 < Vi (0,, x) for any 0, in W-1- A smaller variance (2 ax) is guaranteed for the last layer regardless of the choice of the exponential family in Eq. (1).
Remark 4.5. h(x)- o + C-2)+ + c-1 defines the NN mapping w.r.t. the j'th neuron in the second last layer, where w²2 and w",, are incoming and outgoing links of the interested neuron, respectively; hz2((x) is the output of the third last layer; and ø is the activation function. The "constants' C-2 and c-1 denote an aggregation of all terms which are independent
of w" and w²_, in their respective layers. The Hessian of ha (a) w.r.t. w² is 82 (2))
(w-,)x- (2]=(="==+++00
(h- (xx) By Theorem 4.1, V2(0; x>) can be arbitrarily small depending on æ² (h ( + C-2). For example, if (t) 1/(1 + exp(-t)), then (² = (1) ((t))( 20())). In this case, for neuron in the second last layer, sufficient condition for V2 æ) = 0 (and having Σg favored against Σ, is ( + C-2 = 0 for the neuron's s pre- activation... When the pre-activation value is saturated (-00 or oc), we also have that (0 z) (0, x) = 0. Alternatively, suppose that o(t) = SoftPlus(t) = log(1 + exp(t)), a continuous relaxation of ReLU, then (t) (1()) (() where (t() = 1/(1 + exp(-t)).
Then a sufficient condition for V2(0,: x) = 0 with Vi (6, x)) * for a neuron in the second last
layer is h I (z) + C-2 + +00.
These observations are further clarified by looking at related quantities over multiple parameters. So far we have only examined the variance of the FIM element -wie w.r.t. parameters 0, To study all parameters jointly, we consider the trace variances of the FIM estimators: for any j E {1,2), (0 æ) denotes the trace of the covariance matrix of diag(I, (0 x)), where diag(-) extracts a matrix's 's diagonal elements into column vector. We present upper bounds of these joint quantities.
Corollary 4.6. For any x E Ri
tr (I(0) m)) < [[a((=)]l> min {tr [I(h|=))) [[h(())) Amax (I(h) m))};
(13)
Ct (t| - [[ x)) (0h()) Amas (mn)
1 [[0h())]3 min
(e x) < N
(14)
1
V2(0 () < N
|[dHes(h| æ)]l - min (tr[I(h [))), [|dHes(h [P Amax (I(h x))}
(15)
where dHes((h x) = (Heaa(Hes(h1) ()) ..... dlaag(Hes(hs [x))) and || ||||| is the Frobenius norm.
This upper bound comes from integrating the parameter-wise variances in Theorem 4.1 and incorpo- rating trace variance bound which utilizes the full spectrum of the NN derivatives and sufficient statistics quantities. This is fully depicted in Theorem D. Lower bounds can also be derived in terms of singular values (deferred to the Appendix). Note the upper bounds in Corollary 4.6 can be improved by expressing the min function's 's first term with singular value quantitics.
Having the min function in Corollary 4.6 is helpful as it shows a trade-off between two upper bounds: the scale of NN derivatives Öh(x) and 8 hh h(x)) versus the spectrum of the sufficient statistic terms. In the case of Eqs. (13) and (15), the trace of [hh x)) is exactly the sum of all eigenvalues, including Amax (I(h) | ))) This can be helpful when the scale of the NN derivatives are not bounded by a small value. It should be noted that, by the chain rule, these NN derivatives scale with the overall sharpness / flatness [22] of the landscape of the loss, i.e., the log-likelihood of Eq. (1). For NNs with large derivatives, the first term of the min could yield tight bounds of the variance, and one can therefore avoid dealing with the quadratic scaling of [ja((a) (a: in the second term. On the other hand, if the sharpness of the NN h can be controlled, e.g. via sharpness aware minimization [10], then one can

---

benefit from the second term of the min and avoid computing the full spectrum of [(h x)) in the first term.
Joint FIM Estimators In the above, we considered the variance of conditional FIMs, which can scale differently depending on the input æ. Prior work's analysis was limited that of conditional FIMs (and their estimators) [37]. Nevertheless, the 'joint FIM' estimators (8,) depend on sampling of x w.r.t. the data distribution q(x). The bounds in Eq. (7) can be extended to the joint FIM I(0.) = (x) [(; ar) by simply taking an expectation Eg(x)) over the bounds. To analyze the variances of the joint FIM estimators (0.), we present the following theorem which connects the prior results established for (0; x),, e.g. Theorem 4.1, via the law of total variance.
Theorem 4.7. For any j E (1, 2), given N₂ samples of x ~ ((x:) and samples of y p(y a))
for each x sampled,
1 v2(0,) = N₂
Var (I(O, + No
E [2(0; [+]], x)], g(x)
(16)
where Var (I(0; x)) is the variance of I(0, x) w.r.t. q(x).
The dependence on N, the number of samples of y for each fixed a, is hidden in (0 |2). When N 1, the hierarchical sampling described in the Theorem corresponds to an i.i.d. sampling of the joint distribution p(æ., y).
The variance incurred when estimating the FIM has two components. The first term on the RHS of Eq. (16) characterizes the randomness of the FIM w.r.t. g(x)), i.e., the input randomness. It vanishes when the FIM is estimated by taking the expectation w.r.t. g(x),, or the number of samples æ is large enough. The second term (although also depending on N₂) comes from the sampling of y i according to p(y x)), Le., the output randomness, which scales with the central moments of fty). If the NN is trained that p(y æ²) tends to be deterministic, this term will disappear leaving the first term to dominate. Eq. (16) can be further generalized using the law of total covariance to extend prior work considering conditional FIM covariances [37] to joint FIM covariances. Theorem 4.7 connects the variance of assuming a fixed input æ with multiple samples y/s with the variance of pairs of samples (0),, yk.) The (0, x)) bounds in this section can thus be applied to the corresponding joint variance (6,) by using this theorem. This is straightforward and omitted.
The first variance term in Theorem 4.7 is difficult to compute in practice: it relies how the closed-form FIM varies w.r.t. ((x). As such, it is useful to bound the first term into computable quantities.
Lemma 4.8. Var (I(O; [x)) < Eq(x)) [1), X (I(h [[h|)))]]
This upper bound is very similar to the 4th central moment term Åmax (t x)) when considering the variance upper bound (2 x)) in Theorem 4.1 and Corollary 4.2. In general, the eigenvalue terms of Lemma 4.8 and Theorem 4.1 are distinct, i.e., X2 (I(h x)) / (ACPt x)). This is especially true for the classification and regression problems explored in this paper (see Table I). However, the maximum eigenvalues can be related for exponential families with bounded sufficient statistic via Proposition 4.3, making both bounds depend only on Amax (I(h az)).
The total number of samples of (æx, ys) is N₂ N. In terms of sample complexity of the joint variance, using Theorem 4.7 and Lemma 4.8, the bound's 's rate is given by O(1/N, + 1/(N, NN).
5
Case Studies
To make our theoretic results more concrete, we consider regression and classification settings, which correspond to specifyyng the exponential family in Eq. (1) to isotropic Gaussian distribution and categorical distribution, respectively. We include an empirical analysis of NNs trained on MNIST. Notably, our analysis considers general multi-dimensionall NN output. This extends the case studies of [37] which was limited to 1D distributions due to the limitations of their bounds (and their associated computational costs of dealing with a 4D tensor of the full covariance).
Regression: Isotropic Gaussian Distribution To characterize regression, we consider Gaussian distributions. As per Eq. (I), we have t(y) =y= अ2. and base measure TTy) exp(- - y).
7

---

This corresponds to the case where ((x) is trained via the squared loss. In this case, [(h x) I and K2 ababa(( (t x) Iat x) (ha x)) + [oc(h] æ) [na(h x) + (n Φ) Ine (h x). The derivatives of the log-partition function F(h) yields these central moments [37, Lemma 5]. To apply Theorem 4.1, we examine the extreme eigenvalues of I(h] x) and (* x) I(h œ)) & I(h m))
Proposition 5.1. Suppose that Eg. (1) is an isotropic Gaussian distribution. Then:
Amiu (I z)) Amax (I( x)) = 1;
Amin (t -[hhhhhh110 m)) 0 I(h æ)) = Xmaxx (t x)) I(h a)) 00 I(h) [[]=))) = 2.
Hence, for regression the eigenvalues of sufficient statistics quantities in our bounds are equal. As such, in this case, the bound for I(;; æx), (1( I)), and (0, 2) in Theorem 4.1 are all equalities.
As I(;; (>) can be written exactly in terms of the gradients of h. in practice one does not need to utilize a random estimator when computing the conditional FIM, which simplifies to a Gauss-Newton matrix for the squared loss [25]. When computing the FIM over a random sample æ. variance still appears due to Theorem 4.7. By Lemma 4.8 and Proposition 5.1, the variance over a joint distribution is bounded by function of the derivative (a)())) over the marginal input distributions: V(O,) < [ll h(x) li < maxxxeSupp()) ((0,((z) I In other words, the overall variance to approximate the joint FIM bounded by the gradients of the NN outputs.
Classification: Categorical Distribution For multi-class classification, we instantiate our ex- ponential family with a categorical distribution (with aly) 1). This corresponds to training classifier NN with the log-loss. Let y = [C] for T = C classes defining the possible labels. Let t(y) = ([u = .... [v = C]), where [r] = when the predicate is true and [r] 0 otherwise. Noting our results do not depend on minimal sufficiency, this t(y) is sufficient but not minimal suffi- cient. In this setting, the NN outputs h correspond the logits of the label probabilities. The resulting probabilities p(y) x) are the softmax values of h denoted by (x) = SoftMax(h(z)) E [0, 1J".
T (where Diag(a(z)) is the diagonal matrix with its diagonal entries set to (I)), whose eigenvalues do not follow a convenient pattern as C' increases [44]. Likewise, the maximum eigenvalue of KCt(t x)) is not available in simple closed form. As such, we provide upper bounds for the maximum eigenvalues of I(h ar) and (P x) I(h| x)) @ Ihh 1) using Corollary 4.2 and Proposition 4.3.
Under this setting, we have I(h x)) =
Theorem 5.2. Suppose that Eq. (1) is categorical distribution. With mmax (x) = max; (x)):
Amax (I(h x)) S m(x); and Amax (t x)) - I(h Φ) 2 I(h x)) < 2 12m(()) m(x),
where m(x)= min (amax (x), 1- 1o)
This upper bounds provides a tension. When the first term of m(x) is maximized, the second is minimized, and vere-versa. In particular, the dominating term depends on the uncertainty of the NN's output. When the NN's output is near random, e.g. at initialization, the first term will dominate with mmax(()) & 1 1/C. However, as the NN becomes more certain with its prediction, the second term will start dominating: a more deterministic output ply ar) - 1 implies that Xmax Xm x)) O.
Empirical Verification: Classification We examine the MNIST classification task [21] (CC BY-SA 3.0) using multilayer perceptrons (MLP) with four densely connected layers, sigmoid activations, and a dropout layer. For classification, we consider a categorical distribution with C = 10 class labels. For a random æ from the test set, we compute both estimators 2, (0, a:) and (0, a:) using N 5, 000 samples. We record the variances of each estimator and compute their bounds based on Theorem 4.1. For all 20 training epochs, the Fisher information (FI) and their variances of individual parameters are aggregated via arithmetic averages over four parameter groups (corresponding the four layers).
In Fig. 2, we present the variance scale of the estimators [9: x) and i2, x) in log-space: and UB the tightness of the bounds in Theorem 4.1 by consider the log-ratio log V,ū = where UB the upper bounds in Theorem 4.1. In this experiment, the UB is much tighter than the lower bound (LB), which is omitted in the figures for elarity. More experimental results are given in Appendix F.
We varied the NN's architecture and activation function. Across different settings, the proposed UB and LB are always valid. In Fig. 2, one can observe that the diagonal FIM and the associated
8

---

Group 784 2561
Group (Linear
Param Group
[Lnner
Group
10)
Avg (og(uB)))) 10g 18
-10
5.0
2.5
0.0
012345
012345
012345
012345
Training Epoch
UB of
UB
Figure 2: MNIST for layer MLP with sigmoid activations. Top: The estimated Fisher information (FI) variances, and variance bounds across parameter groups and 20 training epochs. The FI (green line) is estimated using I (I2 almost identical and not shown for clarity). The s.t.d. (square root of variance) is shown for variances and their bounds. Bottom: the log-ratio of Theorem 4.13 's upper bounds (UBs) and the true variances. The closer to O, the better the UB. In the right most column, the variance In vanishes: V2(0; æ) = 0 < (1 æ)). Thus related curves of I2 are not shown.
variances have a small magnitude. For example, in the first layer, V (0 x>) and V2 x>) are roughly 1BB
110 Rv 5 10-5, The log-ratio log (0; m) (0; 2x). Comparatively, V2(0, æ)) has a tighter UB which is approximately 10 times larger than itself. The UB serves useful hint on the order of magnitude of the variances. In Appendix D, we present tighter bounds which are more expensive to compute.
means that the UB roughly 50 times larger than
In the first three layers of the MLP, Vi (0, a)) presents smaller value than V2(0: w)) meaning that 2 can more accurately estimate the diagonal FIM. Interestingly, this not true for the last layer: (0, a>) becomes zero while Σ, presents the largest variance across parameter groups. Due to this, one should always prefer Σn over Ên for the last layer. In the last two layers, Ig is in simple closed form and, hence, does not need automatic differentiation to calculate (see Remarks 4.4 and 4.5). The shape of the variance curves are sensitive to the choices of activation functions á and inputs æ. In general, the variance in the first few epochs presents more dynamics than the rest of the training process. If one uses log-sigmoid activations ((t) = - log(1 (1 + exp(-()) (which is equivalent to ((t) -SoftPlus-tt) as per Remark 4.4), the variances of I, and Iz only appear in the randomly initialized NN and quickly vanish once training starts, as shown in Appendix In this case, the learner more easily approaches a nearly linear region of the loss landscape where local optima lie. practice, one should estimate and examine the scale of variances - which should not be neglected as per Fig. - before choosing preferred diagonal FIM estimator.
Relationship with the "Empirical Fisher"
In some scenarios, even the estimators of the diagonal FIM i, (0) and (0) can be prohibitively expensive. Part of the cost comes from requiring label samples 3/k for each Tk. as per Eq. (3). For example, when the FIM is used in an iterative optimization procedure, yı: 's need to be re-sampled at each learning step w.r.t. the current h alongside their backpropagation (accounting for sampling).
As such, alternative 'FIM-like' objects have been explored which replace the samples from p(y x)) with samples from an underlying true (but unknown) data distribution q(y) x:) [20, 27]. We define the data's joint distribution as ( y) = (()){(B) z). Analogous to the FIM, the data Fisher
information matrix (DFIM) can defined as the PSD tensor I(00 Eg(x)) [I(0 ))), with
a loggps
x)) a log p(y xx)
(äh)
(hh
I x)) = E
I(h xx)
(17)
4(9
m)
00
00
00
00
where 1(h x)- = Egis m)) (t(}) - n)x))) - n T] denotes the 2nd (non-central) moment of (t()) - mx))) w.r.t. ( x). and 0h is the Jacobian of the map - h. In the special case that (lyl m) p(y| w; 0). then I(O a) becomes exactly I(O (z).
The DFIM T(O m²) in Eq. (17) is a more general definition. Compared to the FIM I(0 .)) yields a different PSD tensor on the 0 parameter space (the neuromanifold) depending dis-

---

tribution ( y), which is neither necessarily on the same neuromanifold nor necessarily para- metric at all. The asymmetry in the true data distribution and the empirical one results in differ- ent geometric structures [5]. By definition, we have I(@ x) > (ØKL/00) (ØKI/00) " where o( m) KT,(0)= = f { a:) log p(y =;0) dÿ is the Kullback-Leibler (KI.) divergence, or the loss in a pa- rameter learning scenario. The DFIM can be regarded surrogate function of the squared gradient of the KL divergence. It is a symmetric covariant tensor and satisfies the same rule w.r.t. reparameterriiznioonnn the FIM. Consider the reparameterization 0 S. the DFIM becomes [(<| = (00/ac) I(O) x) (00/ac).
Notice that f)(=) = Eois æ) ((t) # v7(x)) in general. As such, there will be a miss-match when utilizing I(h| z) as a substitute for [hh æ)) However, as learning progresses and p(ý x)) becomes
more similar to the data's true labeling posterior q(ś æ), the DFIM will become closer to the FIM.
If qaz y) .N Ex=1d 6(æ xxx 5ly yk) is defined by the observed samples, DFIM gives the widely used "Empirical Fisher" [25], whose diagonal entries are
I(0,)) Σ (z)) (ta(9r) no (o(xx)))
where (1).S))..... Næ NN) are i.i.d. sampled from q( Similar to (0, x)) an estimator with a fixed input x can be considered, denoted as i(0, x)).
Given the computational benefits of using the data directly bypassing separate sampling routine many popular optimization methods employ the empirical Fisher or its approximation. For instance, the Adam optimizer [16] uses the empirical Fisher to approximate the diagonal FIM. However, switching from sampling yı yr is anything but superficial [25, Chapter 11] i(0) is not an unbiased estimator of I()) as I(h ) is different from I(h æz).
The biased nature of the empirical Fisher affects the other moments as well. In particular, we do not have the same equivalence of covariance and the metric being pulled back by - [38].
Lemma 6.1. Given the conditional data distribution q( (x), the covariance of given x is given by
Cov" (t x)) I(h æ>) AH(z),
(18)
where AH(x)) = (17(2)) (1(()) T.
As result, although the variance of the estimator i(e, x>) takes a similar form to V (0; x>) (i.e., Eq. (8)), its sufficient statistic terms do not exclusively consist of central moments. Noting the miss-match in ()) j (z)), Lemma 6.1 reveals an additional term which shifts I(h x) away from
the 2nd central moment of ((9) (w.r.t. g x)). Instead, these sufficient statistic terms correspond to non-central moments of t($) (7(x). Some corresponding empirical Fisher / DFIM bounds are characterized in Appendix G.
Conclusion
We have analyzed two different estimators 2, (0) and (0) for the diagonal entries of the FIM. The variances of these estimators are determined by both the non-linearly of the neural network and the moments of the exponential family. We have identified distinct scenarios which estimator is preferable. For example, ReLU networks can only apply ī, (0) due lack of smoothness. As another example, (0) has zero variance in the last layer and thus always preferable than Ž, (0). Similarly, the second last layer, I2) has simple closed form and potentially preferable for neurons in their linear regions (see Remark 4.5). In general, one has to apply Theorem 4.1 based on their specific neural network and settings and choose the estimator with the smaller variance. Our results suggest that, from variance perspective, uniformly utilizing one of the FIM estimators (0) is often suboptimal in NNs. Our work has further extended from analyzing the conditional FIM estimators (0) x) to the joint FIM estimators 2, and we have examined the relationship between the investigated estimators and the empirical Fisher. Future directions include extending the analysis of the variance FIM estimators to block diagonals (e.g. [26, 35]) and adapting current NN optimizers (e.g. [16]) to incorporate the variance of FIM estimators.
10

---

Acknowledgments and Disclosure of Funding
The authors thank Frank Nielsen, James C. Spall, and the anonymous reviewers for their insightful feedback and many constructive comments.
References
[1] Shun-ichi Amari. Information Geometry and Its Applications, volume 194 of Applied Mathe- matical Sciences. Springer- Verlag, Berlin, 2016.
[2] Shun-ichi Amari, Tomoko Ozeki, Ryo Karakida, Yuki Yoshida, and Masato Okada. Dynamics of learning in MLP: Natural gradient and singularity revisited. Neural Computation, 30(1):
1-33, 2018.
[3] Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher information and natural gradient learning in random deep networks. In International Conference on Artificial Intelligence and Statistics, pages 694-702, PMLR, 2019.
[4] Sue Becker, Yann Le Cun, et al. Improving the convergence of back pack-propagation learning with second order methods. In Proceedings of the 1988 connectionist models summer school, pages 29-37, 1988.
[5] Frank Critchley, Paul Marriott, and Mark Salmon. Preferred point geometry and statistical manifolds. The Annals of Statistics, pages 1197-1224, 1993.
[6] Felix Dangel, Frederik Kunstner, and Philipp Hennig. Backpack: Packing more into backprop. In International Conference on Learning Representations, 2020.
[7] Bradley Efron. Curvature and inference for maximum likelihood estimates. The Annals of Statistics, 46(4):1664-116 1692, 2018.
[8] Mohamed Elsayed and A Rupam Mahmood. Hesscale: Scalable computation of Hessian diagonals. arXiv preprint arXIv:2210.11639, 2022.
[9] Reuben Feinman. Pytorch-minimize: a library for numerical optimization with autograd, 2021. ie.
URL //tttts::/gitiie wtw.tttt...ooo...eeeeoommeeett weeeiiannpytochhheeeeeee
[10] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awart min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[11) Shenghan Guo and James C. Spall. Relative accuracy of two methods for approximating observed Fisher information. Data-Driven Modeling, Filtering and Control: Methods and applications, pages 189-211. IET Press, London, 2019.
[12] Tom Heskes. On "natural" learning and pruning in multilayered perceptrons. Neural Computa- tion, (((4):88-9011 2000.
[13] Ryo Karakida and Kazuki Osawa. Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks. Advances in neural information processing systems, 13:1099111000 2020.
[14] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics Fisher information in deep neural networks: Mean field approach. In International Conference Artificial Intelligence and Statistics, pages 1032-1041. PMLR, 2019.
[15] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Pathological spectra of the Fisher information metric and its variants in deep neural networks. Neural Computation, 33(8): 2274-2307, 2021.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.

---

[17] Tamara G Kolda and Jackson R Mayo. An adaptive shifted power method for computing generalized tensor eigenpairs. SIAM Journal on Matrix Analysis and Applications. 35(4): 1563-1581, 2014.
[18] Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems, pages 4133-4144 Curran Associates, Inc., 2020.
[19] Takio Kurita. Iterative weighted least squares algorithms for neural networks classifiers, New generation computing, 12:375-394, 1994.
[20] Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gradient algorithm. Advances in neural information processing systems, 20, 2007.
[21] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online], 2, 2010.
[22] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[23] Lek-Heng Lim. Singular values and eigenvalues of tensors: variational approach. In Ist IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, pages 129-132. IEEE, 2005.
[24] Albert W. Marshall, Ingram Olkin, and Barry C. Arnold. Inequalities: Theory of Majorization and its Applications, volume 143 of Springer Series in Statistics (SSS). Springer, second edition, 2011.
[25] James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146): 1-76, 2020.
[26] James Martens and Roger Grosse. Optimizing neural networks with Stonecker-factorgs approximate curvature. In International Conference on Machine Learning, pages 2408-2417. PMLR, 2015.
[27] James Martens et al. Deep learning via Hessian free optimization. In ICML, volume 27, pages 735-742, 2010.
[28] Peter Nickl, Lu Xu, Dharmesh Tailor, Thomas Möllenhoff, and Mohammad Emtiyaz E Khan. The memory -entury-perturbatioon equation: Understanding model's sensitivity to data. In Advances in Neural Information Processing Systems, volume 36, pages 26923-26949, Curran Associates, Inc., 2023.
[29] Frank Nielsen and Gaëtan Hadjeres. Monte Carlo Information- Structures, pages 69-103. Springer International Publishing, 2019.
[30] Yann Ollivier. Riemannian metries for neural networks I: feedforward networks. Information and Inference: A Journal of the IMA, 102):108-13,, 2015.
[31] Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1664, 2020.
[32] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In International Conference on Learning Representations, 2014.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035. Curran Associates, Inc., 2019.
[34] Jeffrey Pennington and Pratik Worah. The spectrum of the Fisher information matrix of single-hidden-layer neural network. In Advances in Neural Information Processing Systems, pages 5415-5424, 2018.
12

---

[35] Yi Ren and Donald Goldfarb. Tensor normal training for deep learning models. In Advances in Neural Information Processing Systems, volume 34, pages 26040-26052. Curran Associates, Inc., 2021.
[36] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems, volume 33, pages 18098-188 18109. Curran Associates, Inc., 2020.
[37] Alexander Soen and Ke Sun. On the variance of the Fisher information for deep learning. In Advances in Neural Information Processing Systems, volume 34, pages 5708-5719. Curran Associates, Inc., 2021.
[38] Ke Sun. Information geometry for data geometry through pullbacks. Deep Learning through Information Geometry (Workshop at NeurIPS 2020), 2020.
[39] Ke Sun and Frank Nielsen. Relative Fisher information and natural gradient for learning large modular models. International Conference on Machine Learning, pages 3289-3298, 2017.
[40] Ke Sun and Frank Nielsen. A geometric modeling of Occam's razor in deep learning. arXiv preprint arXiv: 1 2019.
[41] Shiqing Sun and James C. Spall. Connection of diagonal Hessian estimates natural gradients in stochastic optimization. In Proceedings of the 55th Annual Conference on Information Sciences and Systems (CISS), 2021.
[42] Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.
[43] Sheng-De Wang. Te-Son Kuo, and Chen-Fa Hsu. Trace bounds on the solution of the algebraic matrix riccati and łyаpunov equation. IEEE Transactions on Automatic Control, (1()):654 656, 1986.
[44] Christopher S Withers and Saralees Nadarajah. The spectral decomposition and inverse of multinomial and negative multinomial covariances. Brazilian Journal of Probability and Statistics, pages 376-380, 2014.
[45] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the Hessian. In 2020 IEEE international conference big data (Big data), pages 581-590. IEEE, 2020.
[46] Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney, Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference artificial intelligence, volume 35, pages 10665-10673, 2021.
13

---

Supplementary Material
Abstract
This is the Supplementary Material to Paper "Trade-Off's of Diagonal Fisher Infor- mation Matrix Estimators". To differentiate with the numberings in the main file, the numbering of Theorems is letter-based (A, B.....
Table of Contents
Additional Results
+ Appendix A: Natural Gradient Toy Data Example
Pg 15
+
Appendix B: The Conditional Variances in Closed Form
Pg 15
Appendix C: Off-DDagonal Variance
Pg 19
Appendix D: Bounding the Trace Variance by Full Spectrum
Pg 20
+ Appendix E: Second Central Moment of Categorical Distribution
Pg 22
Appendix F: Empirical Results Continued
Pg 23
4 Appendix G: "Empirical Fisher" Continued
Pg 23
Proof
Appendix H: Derivation of Eq. (3) Using Log-Partition Function Derivatives
Pg 25
Appendix I: Proof of Eq. (7)
Pg 27
Appendix J: Proof of Eq. (8)
Pg 27
> Appendix K: Proof of Eq. (9)
Pg 28
Appendix L: Proof of Corollary 4.2
Pg 28
Appendix M: Proof of Proposition 4.3
Pg 29
Appendix N: Proof of Corollary 4.6
Pg 29
+ Appendix O: Proof of Theorem 4.7
Pg 31
Appendix P: Proof of Lemma 4.8
Pg 34
+ Appendix Q: Proof of Proposition 5.1
Pg 35
Appendix R: Proof of Theorem 5.2
Pg 35
Appendix S: Proof of Lemma 6.1
Pg 36
Appendix T: Proof of Corollary G.1
Pg 36
Appendix U: Proof of Corollary G.2
Pg 37
14

---

A Natural Gradient Toy Data Example
The following section describes the data and models of Fig. 1. In general, the toy data and models constructed consists of taking ID output setting presented by Section 5, where the NN he (ar) is linear function.
A.I Data
The 2D input data a E R² sampled from simple isotropic centered Gaussian (o I). A linear response variable E R is defined by the following:
a
0 Orrue
where Perue = (1,
The outputs of y for the cases of regression and classification are differentiated by how is used in sampling:
Vregression ~ 1,
yelasilication
Bern(p (a)),
where (2) (1 + exp(-2)) is the logistic function.
A.II Model
The model (x) = 0 x consists of a linear function; and the exponential family Eq. (1) chosen to be 1D isotropic Gaussian and binary multinomial distribution (Bernoulli) for regression and classification, respectively. This corresponds to Section 5 for 1D outputs. Notice that the model exactly matches the data generating function.
A.III Training
Natural gradient descent (NGD) is taken using both Zi (0) and (0). The estimated FIM utilize only a single y x sample for each input æ. We use learning rate of f 0.01 over 256 epochs. training set of 256 data points are sampled. At each iteration of NGD, we sample 4 random points from the training set for the update. The test loss is evaluated on a test set 4096 data points sampled.
A.IV Variance Plot of Example
Larger version of Fig. with additional variance sum plotted over time is given by Fig. I. Note that variance sum is including off diagonals. Further note that the variance is calculated over joint sample in (x. y).
A.V
Other Seeds
We further present other random seed of the teaser plot in Figs. II to IV.
The Conditional Variances in Closed Form
We consider the diagonal entries of the conditional FIM I(0; (x) and the conditional variances (0, a)) of its estimators closed form.
Proof of Lemma 3.1. The proof directly follows from [37, Equation 6), [37, Theorem 4], and [37, Theorem 6]. In what follows, we provide proof of the Lemma utilizing the notation of this paper for completeness. We prove the statement one equation at time.
15

---

Regression (Ganesian)
0.74
Classification (Categorica)
Text Los (NLL)
2.0-
Trajette
Trijettrre ae
Intiall Paramaa
MG with Ra
Trie Paramn
1.5
0.68
1.0
0.40
0.5
0.55
64
126
201
12
128
256
Training Epoch
Training Epoch
2000
Regression (Gaussian)
Classification (Categorical)
1.(0) Varisnes Sum
1000
0.20
0:10
000
0.05
32
128
206
256
Training Epoch
Training Epech
Figure I: Extended version of Fig. 1 with the sum of variance of FIM estimators over epochs.
2000
Regression (Gaussian)
6.76
Classification (Categorical)
Test Loos (NLL)
2000
0.70
Trijeturiss
NG wite 2t with
2000
Params
2000
64
123
268
52
126 Training Epoch
256
Training Epoch
1500
Regression (Gaussian)
Classification (Categorical)
1,00) Variance Sum
1200
1,00
0.75
300
0.50
0.35
0.6
52
125
256
32
128
256
Training
Training Epoch
Figure II: Fig. I over different randomizations (a).
250
Regression (Gaussian)
0.70
Classification (Categorical)
Test Loss (NLL)
NG with 2,
Trojettrie
100
Inst al Paramn
0.70
64
125
236
32
125 Epoch
256
Epoch
150
Regnession (Gaussian)
0.30
Classification (Categorica)
(0) Variance Sum
195
0.25
100
0.10
0.10
www
0.101
64
Training Epoch
Training Epoch
Figure III: Fig. over different randomizations (b).
16

---

For Eq. (4), we consider the following computation.
I(0;]] [x)
i log p y x; 0) a log ply x; 0)
E
plar 2.0)
00
d0
i (t T (y)ha(m)
(h)(()))
(x)))( ((ho(a)))
E
p(vl (x;)
00
00
(h)
T( ((y)
OF(h)
aF(h)
Tro (xr)
E
((y)
plul m;;))
00
0h
1h=ho())
8h
h==a((a)
30
(ah)(a:)
(ōho (Oho((m) (z))
(t(y) - n7xx)) (t(y) n>x)))
E plul m=
00
Tet
(a)
(t(y) ((x2)) (t(y) m(=)))]]
( 80
(Oho(x)
00
P(U m;;)
(ho(2z)
(ohg(x)
I(h| æ)
00
001
Using Einstein notation and restricting the partial derivative to component of 0 yields the desired result.
For Eq. (5), we shorthand d(y) t(y) -n().. Note that the (0; æ)) estimator can be written as follows:
(0, x) =
1
N
(F(h(()))
(x)
tu (yk))
N
00,
00,
1=
1
(x)
"a (x)
th (x)
tt txal
N
08;
00:
oh (h))))) (1))))1)))))
1.38
Regression (Gaussian)
0.74
(Categorical)
Tex (NLL)
Trajecunis
0.79
Trajetzaio
NG with 3t
Initall Params
0.70
True Porams
1.00
0.73
0.68
64
125 Training Epoch
256
52
218
Training Epoch
Regression (Grushian)
Classifiationn (Categorical)
(0) Variance Sum
0.5
300
0.4
200
0.2
0.1
MA
0-
0.
37
64
198
336
67
125
205
Training Epoch
Training Epoch
Figure IV: Fig. over different randomizations (c).
17

---

Thus, we have
(i (x)
= -Var
(Ir(0: x))
N
- Var
0h" (a
N
1
Var (z)" (1x
N
1
E [10M)(00)0000000000000000000)0)0
E (1) plu x1
N
2:0)
Let us compute each of these terms.
E ((N p(y 2:00
ah
Pla 0)
(h"
E [
pla 2
Oh*
And,
(x)an (2) (()) 2
p(y [(]|==:)) m;0)
(a) E
(5. (d.(z))((y)))
plu m=
(Oh" x))²
(*)) (z)) (z) a:) Ied (z) (I(h x) 60 I(h )
(h) (a: (a)00
Simplifying all term yields the result as required.
Finally, for Eq. (6) we consider the following simplification of the estimator.
1 (FF((()))
N
(x))
(0: xx)
tt(kk)
N
020
800
1
N
(1)
ma((e))
(x)
-ta(Vs)
N
a0;
00;
Y2A:
(h"(z)
@n (x)
(2)
(2))(z))
na(x)
to (a(1))
N
00,
00,
020,
020;
Ona(x)
(z)
(x)
6c((k))
N
00,
(0:h"()) (z) (.h Lat .)) - 0. ( 5y()))
N
(x)((()) 0,h² (x)) (as () Σ
h (x) 8, 8n))))
where the last line follows from [37, Lemma 2] (a result of ply x;; 0) following an exponential family, see [1]).
18

---

Notice that the first quantity is a constant w.r.t. the randomness of yv. As such, we can simplify the variance calculation as follows.
V22 (0; (x) - Var [Z2(0: m))
Var (x) ,,hh(aa) Iab a.)
Σ
(x))
6.(uke))
N k=1
(x) 6o(Bui))
Var
N
kml
Var (0fh((x) 6a(3))
N
E
da())) (09h (
E (o) 6d(w)]²)
N
ply m;
(z) (x) 61(3) 6z(3)] - (
E plu (=:))
[6 (1)))
N
Aply (x:0)
(0th"(a)) (2h)
E (v) 6n(y))
p
N 8 ha (a)) h Lab x))
where the second last line follows from the fact that 17(x) = Ep(u a: [ty)) and thus [d())
0. This yields the desired result.
Lemma 3.1 shows that, for the former, Vi (0, x)) only depends on 1st order derivatives; while (0, æ) only depends the 2nd order derivatives. For the latter, V1 (6; æ)) depends both the 2nd and 4th central moments of t(y); while (2(0;) | æ)) only depends on the 2nd central moments.
Given Tub (h x) and (xh" the computational complexity of all diagonal entries I(0; x) is O (f)) If KPP Kebbaatt ar) and ( are given, then the computational complexity of the variances in Eqs. (5) and (6) is respectively O dim(()) and O(T dim (0)). Each requires to evaluate T dim matrix, either 0,h" (æ)) or oe h² which can be expensive to calculate for the latter. This why we need efficient estimators and or bounds for the tensors on the LHS of Eqs. (4) to (6).
C Off-Diagonal Variance
We consider an off- off-diagonal version of the bound given by Theorem 4.1. Notice that in terms of the dependence neural network weights, the only change is splitting the "responsibility" of the i'th and j' ttt parameter norms.
Theorem C.I. Va E R,
Var (0)(0==))))
< -10,h(x:) [10,1()))3 Amax (M),
1
(19)
N
Var [2((12²)))) 2z)1)) s N
1
1 hx))11 Vmaa (I(h x)),
(20)
where
Fmax (M)
sup
w a Mabed
=:
Tmaax (M)
sup
2 v Mab-
||| =
Proof. The proof follows similarly to Appendices and K, where the primary difference just swapping the regular cigenvalue-like quantities with the y variational forms.
19

---

It should be noted that the corresponding lower bounds become trivial as the additional degree of freedom of having an inf over both and causes the corresponding "min definition have negative quantities. Although it is unclear what the "tensor-like" variational quantity Tmax (M) will be, for matrix, we have the following equivalence.
Lemma C.2. Tmax (A) 5max (A), where smax (A) the maximum singudar value of A.
Proof. The proof follows from optimizing over u and separately:
Ymax (A)
sup
sup
utt v.
Aab
u:Ilu == e
sup
sup
u Av
mill 2 0:||1/2=11
(Av) T Av
sup
||Av
w: v||==1
V Vo (AT AA)
sup e: e
This is equivalent to the square root of the maximal eigenvalue of A which is exactly the maximum singular value.
Hence for the 1g we have the following.
Corollary C.3. Va Rt,
Var (12(0(=))))
1 S N 100 h())( 8max (I(h x)))
(21)
D Bounding the Trace Variance by Full Spectrum Theorem D.I. For any Rt
(0h(z)) AT-141 (I(h) x)) s (I(e) (I(0)z))
T
$ (0h(z)) d (I(h [a))
(22)
1=1
T
2 (vJacch x)) -1- (M) s Vi x)
N
(=1
1
$ (vJac(h x)) de (M)
(23)
N
1=1
Σ
2 (dHes [x)) Ar-+41 (I(h) x)) I V2 x))
N
T
L(dHes(h) de (I(h)
(24)
where (?(A) >a A) denotes the i-th singular values, M is the "reshaped" matrix of M defined in Theorem 4.1 - i.e. there exists j. k such that Mjx = Mated for all a, b, d,
dHes(h) (+) = (diag(Hes(()) diiy....,iagHHe((tt x))),
and
vJac(h| x)) = (vec(0)h(z)0) (ert
20

---

Proof. The proof follows from generalized Ruhe's trace inequality [24]] Theorem D.2. For A, Rux Hermitian matrices, we have that
n Σ (((A) An-s+1 Str (AB) & Σ (((A) (B).
We prove the result for each equations. For readability, let j 0.h((a).
For Eq. (22):
One can notice that the trace of the FIM can exactly be expressed as the trace of two matrices.
dim
tr (I(O)=)))
Σ
(z))
1=1
dim(O)
Iab(h x))
Σ
jib
1=1
dim(O)
IIa x))
Ias (J J)ab
= tr ((J J)I(h x=)
Thus, noting that the eigenvalue of the "squared" matrix is the matrix's singular value Ae(JT J) sf()) with Theorem D.2, we have that:
T
(Oh(z)) AT A tr x))
s (0h(z)) de (I(h|=)).
For Eq. (23):
Noting that Mabed K2 (t - (h x) (cd(h x). Furthermore, have that vJac(h x)) (voc(0)h((m)()h((()) (e(())))_(((t)))))))aaa)))
Let us define the following 3D tensor with Jrab 0 (zhh(()) (a.h(a)0) (())))
dim (0)
(1 x))
Σ
0.h (w))
N
dim(
Mabedd
Jiab giod
N
1=1
dim(O)
Mabeed
Σ
giabgiac
a,b=1 c,d=1
dim(O)
Mjk
Σ vJac (h vacc (h x))
T T2
Σ
MvJac T (h [))Jac(h x))*
k=1
N tr (M(vJac T (h ([l=))Jac(hh)))) x)))
21

---

Thus, again simplifying the eigenvalue of the "squared" matrix, with Theorem D.2, we have that: of (vJac(h) x)) -I-++1 (M) s tr [1(0 x) S (vJac(h |x)) de (M)
For Eq. (24):
Similar to Eq. (22), we only need to rearrange the summation. Notice that
dHess(h æ)) (diag(Hes(h)) æ)),..., diag (Hes(hr)))) x))),
thus dlees " (h x)) 0 (ha xx).
dim (0)
V2(0 2:) = N
0 h (x) h (x:) Iab x)
dim(0) (
Iab (h| xx)
h (z))fhh
dim
dHes "h (h x x)HHes xx)
N
dim(o)
N Ias (t xx)
(dHes T y (h x)dHes xx)
Ias (h (dHHss (h ())HHss( x))
N (I(h) (dHes (hl (h)
Thus, again simplifying the eigenvalue the "squared" matrix, with Theorem D.2, we have that:
(dHHess [x)) A (I(h) x)) s tr (0 x)) ( A (I(h/)))
E
Second Central Moment of Categorical Distribution
Proof. We first notice that the exponential family density is given by, ply x)) = exp(h,, ((h())))
and thus also have
T
F(h(x)) = log Σ exp((tr(x))
The first order derivative follows as,
OF(h)
exp(h;(2x) (h(())
Oh, h=h(")
Ei=1 E (x)())(x))
As such, the second order derivatives also follow, F2F(h)
exp(h;(x))d, Σ exp(h, - -ese:h/ill)-epp/////eeeeee/ Öh,üh; h h(m)
Σ exp(h+(x)) 2
= (h(x)) 8ig - (b(())))
As such, we have that
I(h/ x)) = Diag(o(#)) ((())())
22

---

Empirical Results Continued
In the following section present additional details and results for the experimental verification conduct in Section
F.I Additional Details
We note that to calculate the diagonal Hessians required for the bounds and empirical FIM cal- culations, we utilize the BackPACK [6] for PyTorch. Additionally, to calculate the sufficient statistics moment's ' spectrum, we explicitly solve the minimum and maximum eigenvalues via their optimization problems. For 2D tensors matrices, we utilize numpy. linal 1 g. e For 4D tensors, we utilize PyTorch Minimize [9]. wrapper for SciPy's optimize function.
F.I Additional Plots
We present Figs. V to VIII which are the exact same experiment run in Section 5, but with different initial NN weights and random inputs.
Figures IX to XIII show the experimental results on 5-layer MLP and log-sigmoid activation function. In most the cases, the FIM and associated variances quickly go to zero in the first few epochs.
Param Group 1 (Linear 784 + 2561 Param Group 2 (Linear 256 - 2561 Param Group [Linear 256 1281
Parem Group [Lineer 120 - 10)
log V
20-
-40
Avg (ogtu/r9))
012345
10
15
20
012345
15
20
012345
10
15
20
012345
10
15
20
Training Epoch
2,
v/
UB
Figure V: The Fisher information, its variances and bounds of the variances w.r.t. MLP trained with different initialization and different input (a)
Param Group 1 (Linear 784 - 256) Param Group 2 [Linear 256 - 2561 Param Group [Lincar 256 -288
Param Group [Linear 128 10)
log 6.
20
-40
Aug (ogturri)
5.
012345
012345
15
012345
20
012345
10
20
Training Epoch
Figure VI: The Fisher information, its variances and bounds of the variances w.r.t a MLP trained with different initialization and a different input (b)
G "Empirical Fisher" Continued
Noting Lemma 6.1's characterization of the covariance, we are able to characterize the variance of the diagonal elements of i(0) æ). denoted as V(O, x) Vir x)).
Corollary G.1. For any E Rd
1 v(O, x)) N
0, (2) (x)), (x) (Katea(t) (t) x)) (h xx) Laahh/
Kateaa (x) N (a.h (ax) (Cov AH(=))
N
23

---

Group
Group
-20
10g
Avg (ngg)U865))
5.
Training Epoch
4
UB of
V
UB
Figure VII: The Fisher information, its variances and bounds of the variances w.r.t. a MLP trained with different initialization and a different input x: (c)
Param Group 1 (Linear 784 256) Param Group 2 (Linear 256 + 2561 Param Group [Linear 256 1281
Param Group [Linear 128 - 10)
log VT
-15
Avg logtumiis
012345
10
15
20
012345
15
012345
10
15.
20
012345
10
20
Training Epoch
4
UB of
v.
UB
Figure VIII: The Fisher information, its variances and bounds of the variances w.r.t. MLP trained with different initialization and a different input x (d)
where K(h) w) the 4th (non-central) moment of (t($) :n(x)) w.nt 9 [x)
As a result of the similarity of the functional forms of the empirical Fisher Î(0) and the FIM estimator Ż, (0), it is not surprising that Corollary G.1 is similar to the variance of Ê, (6; x)) Indeed, applying Lemma 6.1 will give the exact same functional form with the 2nd central moments of t(y) w.r.t. p(y æ)) exchanged with 2nd non-central moments of (t(ý) (x))) w.r.t. q( x). V(0, (z) therefore determined by the 2nd and the 4th moment of (f(9) n7 m())) up to the parameter transformation 0 + h. Subsequently, the bounds presented for V (0, x)) (Eq. (8) and Corollary 4.6) can be similarly adapted for [(0;| =x).
The extension of ((O, æ) to ((O)) can also be proven in similar manner to Theorem 4.7.
Corollary G.2. Given N₂ samples of x ~ ((x)) and N samples of y æ a(y) a:) for each sampled,
V(O.)
Var (I(0;| + N ((x)
E [V(0, x)).
(25)
where Var (I(0: x)) is the variance of 1(0, x)) w.nt. g(x).
If g(r y) = Nv E 8(x: xx)) 5(y ÿi.) for set of observations {( )1) k= then one can
directly evaluate the DFIM without sampling and achieve zero variance, i.e., i(() = I(0). In this scenario, there is a clear trade-off between the estimators of the FIM in Eq. (3) and the DFIM. The estimators of the FIM are unbiased, but have variance; while the DFIM has zero variance, but is biased approximation of the FIM.
24

---

PGroup PGroup
10g 16
-25
-50
Avg (og)U8855
15
Training Epoch
4 UB of
V
UB
Figure IX: The Fisher information, its variances and bounds of the variances w.r.t. a 5-layer MLP with log- sigmoid activation.
258) PGroup [Linear PGroup (Linear
(Linear -0)
log Vi
-25
Aug logrur/))
Training Epoch
4
v
UB of
v --- UB of
Figure X: The Fisher information, its variances and bounds of the variances w.r.f. 5-layer MLP with log-sigmoid activation.
PGroup
PGroup (Linear PGroup [Linear
PGroup (Linear 64)
PGroup
log
Avg logluerii
012345
10
15
20
012345
10
15
20
032345
10
15
20
012345
10
15
20
022345
10
15
20
Training Epoch
2
UB
UB
of
Figure XI: The Fisher information, its variances and bounds of the variances w.r.t. 5-layer MLP with lig-sigmoid activation.
H Derivation of Eq. (3) Using Log-Partition Function Derivatives
In what follows, we derive the alternative equations for (0.) and (2.).) presented Section That is, seek to derive the following equations:
OF(h(x)))
ah*(
to (81)) 2
2,(0,)
Σ
(26)
20;
00,
T=1
N
F(((xx)))
(
12.0.)
ta (())))
(27)
N
820:
800.
k=
We calculate the equations separately.
25

---

PGroup
PGroup
-25
10g
-50
AVg (og(UB9)))
5.0
0.0
15
Training Epoch
UB of
V
UB
Figure XII: The Fisher information, its variances and bounds of the variances w.r.t. a 5-layer MLP with log- sigmoid activation.
PGroup [Linear 256) PGroup (Linear PGroup 3 (Linear 128 PGroup (Linear
PGroup (Linear
log 1
-25
-50
Avg logtUnsi)
012345
10
15
012345
10
012345
15
20
012345
15
20
012345
10
15
20
Training Epoch
1.
UD of V'g
v/g
UB of
Figure XIII: The Fisher information, its variances and bounds of the variances w.r.t. 5-layer MLP with sigmoid activation.
H.I
Eq. (26)
Proof. For Eq. (26), note that
i log p(ys mx.)
(t (yk)h(zL) - F(h(zx)))
00,
00,
(xx) (1((())))
(xx)
tyk)))
00;
00:
(xx)
(xx)
-ta(y:)
-a (mkk)
00:
00,
th (xx(
(ta()u)) - na (xx))
00;
where we note that F. (h(x))) = 7. (2x) which follows from the connection to expected parameters and partition functions of exponential families, e.g. [37].
Then Eq. (26) follows immediately.
H.II Eq. (27)
Proof. For Eq. (27), we also calculate the derivative:
log p(y)
xx)
(t (yx))(((x)) - F(h(xx)))
00; 00,
(a.))
aF(h(an))
t. (yk))
00,
00,
Then
8 p(y: |wx)
ah
0F (F(((xx))
tu((y))
020;
00:
00:
00:
hh (mL)
8 FFF(((xx))
-ta(4))
820,
020;
26

---

Then Eq. (27) follows immediately.
Remark H.1. Although Eq. (27) useful in practice, ie., states an equation which can calculated via automatic differentiation, in the appendix and proofs we use alternative equation. In particular, we use
Σ [2(0.) - N
N
(mna(az)) - layn))
(ht((w))
820;
(a
0h ah"(
Iab (h xxk)
00,
00,
which follows from taking the derivative of 0 log p(y)) xx)) in the proof of Eq. (26) (above).
I Proof of Eq. (7)
We first begin by proving the follow lemma bound an R matrix. Lemma L.1. Let A E Rexx and R", then
| Amin S v v0 Aab S 11012 Amax (A).
Proof. The proof follows immediately from the Courant-Fischer min-max theorem [42]. That is,
Xmin (A)
inf " "uaAAaas
w::|aa/|=1
Amax (A)
sup u uuu" Aub-
w:|lu||=1
Thus it follows that:
o Aab = 11112 (1/1|=/22)
$ |1012 Amax The lower bound follows identically.
We note that this can be similarly proven via trace bounds, e.g., [43].
Now we can prove Eq. (7).
Proof. The proof follows from Lemma 3.1, Eq. (4), and directly applying Lemma I.I.
J Proof of Eq. (8)
Let us first define the maximum and minimum Z-eigenvalues of a 4-dimensional tensor K.
Xuia (K)
inf ua w:ll 1==1
(28)
Ámax (K) = sup u n " u
Kabodd
(29)
willu :=1
Now We first prove the following lemma regarding the Z-eigenvalues. Lemma J.1. Suppose K. is 4-dimensional tensor. Then we have
||ll Amin (XC < u I ||lll Amax (XC)
(30)
Proof. The proof follows similarly Lemma I.I. We simple use the following calculation: 0d 0" o o
Kaboo
< o0ll sup
n un Kabed
=:
= | 7max (AC).
The minimum case is proven identically (with the opposite inequality).
27

---

Now we can prove the bounds of Eq. (8)
Proof. From Lemma 3.1, we have that V1 - N
[Kabed - Iab(he x)) Icd (he x)];
where we shorthand Kabed = K at We bound two terms.
[0,h(())12 Amia - 110,h(>))1 Xmax (IC)
which follows directly from Lemma J.1
We now bound the second term in similar way, taking vn = (x) and noting that Ih ( x>) (I x))²
Ias x
which directly gives us.
(1u112 Amin [I(h|=))) v0 (aabh x Z<C (x)
$ (1l1112 Amax (I(h) x>))²²
which follows from Lemma 1.1. Thus, together these bounds prove Eq. (8).
K
Proof of Eq. (9)
From Lemma 3.1 have that,
1 V - N (f
Thus get
[00h(z)112 Amin (I(h)) < (z) -0fh(())) -Iv - [00 Amax (I(h)),
which follows from Lemma I.1. This immediately gives the bound as required.
L Proof of Corollary 4.2
Proof. The corollary holds from distributing the inf or sup and examining how the variational definition of the generalized 'eigenvalue' simplifies under tensor products.
Indeed, for the minimum case,
Xmin (tC (t) I(h x) 2 I(h|2) x))
inf u" "uuuur a (KP (Cpbaalt (he xx) [ha(h) x))
u: u/|/11
inf uuuu KP Reoeeat m²)
inf u u Iab
x)) (- Iea(h x²)
u: wull]|la=1
inf
un KP
Ceobel (t ax)
sup
(Iabll ( Ind (h x:))
(Iab(h)
willu||a=1
inf
Peeeaal (x))))
sup
u"' xx))
willu||a=1
inf w u ue "e KP a))
2 2 IIn (te xz)
sup willu|u211
28

---

where the last line holds from the fact that ( x)) is PSD (thus the inner Einstein summation is always positive).
Taking definitions of the types of eigenvalues, gives the statement.
We note that the 'max' case follows identically.
Additionally, for the lower bound, we can show the non-triviallity of the non-negativity of the minimum eigenvalue.
We note that K2 abcaa x>) = Eppuaebbeeal, where = =t()) 17(x))
Thus have that
Amin (XCP(t a)) - I(h @ I(h x))
inf 2t" u² "urur u (Katbcat (t x)) - [a x)) Ica(th) x)) 1=
inf E In''uu u (V. UbUOU Inb x)) [ea x))]
inf E (u''r (Uaus / }] 20. 0. =||||||2=1
Equality holds from simply looking the definition of (C a>) and I(h a)) (as moments).
M Proof of Proposition 4.3
Proof. Letting e = t(y) - n7(), we note that the maximum eigenvalue is given by, RK,
Amax (t²
sup u unu uu'uuu Kottod (t) ())
u:||u||==1
sup E u u u a u. U.
w:|lu|==11
sup E [(u v)
sup E [(u T ")²("" [2]]
w:|lu]|==1 F
sup E [[[|u]]| 1|0|12) (MTT)) w)
w:|lu|/==1 P
B. sup E (u T u)?]
u:||u[|2=1
11
B Amax (I(h) xx))
N Proof of Corollary 4.6
Proof. We split up the proof into the two arguments of the various min-function.
For the right term:
Suppose that we have bound such that (, x) S aB, Then,
dim(O)
x) Σ
(V;
x))²
dim(()
(a,3,)
dim(0)
a²
i=
29

---

Thus we have,
dim(
[1 æ)1l2 S a
Σ
8
Taking the appropriate a and from Eqs. (8) and (9) proves the case for Eqs. (13) and (15).
For Eq. (14), that taking
a N (X x))) min (I(h) x²))
B. = 100,((0)1122
Where we note that
d0m
dim(() Σ
T
272 2
(100,(>))
dim
T [2)h)
Σ ==1
dim (0) Σ
T
(2)h2
i=1
dim(0)
T
Σ
Σ
(x))
= [
For Eq. (15), that is taking
Amax (I(h) æ));
N
8 10 h |1
Where note that
dim
dim
((10)(==)1132
Σ
10 hr (2x))
2
@)
Σ
100 hr (x))
dim(O)
Σ
Σ
@h)())) 2
||dHes(h (h .)|F
For the left term:
We take the largest singular value of the network derivative term. We then further notice that
Smax (A) < ||A]]] F from norm ordering (of the matrix 2-norm).
30

---

To further elaborate on the Eq. (14) case, we further need to simplify the following:
max (vJac) I l/vJac(h x)
d
Σ
T
d)m(0)
Σ
(0)h)(=))(())((())²))))
a,b=1
==1
T
((0h (110))))))111(10))))))) (x))1(x)))11
a,b=1
[(0h((2)))112
(1@h " (x)
dim ()
(3))(())))
dim(@)
(())
where the last inequality follows from the norm ordering || S 11.
Proof of Theorem 4.7
To prove the Theorem, we will utilize the law of total variances. We note, that by the premise of the Theorem, sampling N₂ many samples from q(xr) and N many samples from g(y yy) for each initially sampled. To make this clear, the samples and sampling will be notated by:
D q(x)
y1 क4 ~ p(y [3g)
Note that using these samples, our empirical estimators for the FIM (for either estimator) will be of the form:
(1,)
1
Σ
1
Σ Bll
f(zk) (11
No
N
for an appropriately chosen f.
This also gives:
(0(0:
1
Σ
f(a, Di ))
N
Bil
31

---

Now, we simplify the variance as follows:
Var
1
Σ
1
Σ
f(akk
Mt ms)
Nx
N
f(o y1 mi)) 1)
Var
Σ
N2
N
Milm
Var
(8, xx)
(0: mxx)
Var 1 UN 1 [2 xx)) mx)1])
Varaa
+ EE
N2
M1 -. V.V az
Varaa
[I; (x:)
E (0; [xx)
N2
Var [I(O; [)] + (0, x3]
N
As required.
For Vi (0)
Proof.
1
Ö Ölog p(y xx)
i log p(y x))
2
(1(0.)= N
p((..))
00,
p( (,
00;
Let (a(x, v) = (t(y) -))x))
a logg(u .x)
p(m.y)
00,
10h((m (m)
0h (h)(a)) (x) (h)(x)
6a ()),.))),,, ()) (2. 9) v)
09;
a0,
00,
00,
p(m.y)
Oh (x) 0h (x)) (x)
(h((w)
E vim)
E
15, y)d (2. 2)0a(11))) v))
00,
00:
00,
00,
puu x))
(x) 0h (x)
0h
0h (x))
E
KP
Ka x))
00,
00,
a0,
00;
g(m)
32

---

And:
a log ply x))
00;
p(a.3)
(a)
0h
8 (x. 8x
p(a.))
00,
00,
[hh (x) 0h"(m)
(x)
E
E p(yl=)
(0 y z)]
((x)
00:
00,
Oh (x) 0h (x))
E
Tab (h) x))
(a))
00;
00;
(x)( (h)(a))
(x)
Varx ((
(ah(m)) 00:
T
@h(xx)
E (
Iab (he 2z)
I(h xx)
00;
00,
00:
(x)
(2)(2x)
E ((x)
Iab (h x))
Varx (I(O;
00;
00,
Together:
(0,)
Varx (I(0; x))
N 1
(x)) (x)) (h"(x) (x)
+
E
[Caboatt] [h x))
Ica x)]
N
((x)
00,
00;
00:
00;
For (e)
Proof.
V2(0,)
1
Var ((n((t)) - tv)))
1x)
+ +[( 1z)
N
00,00,
)2)
Var (na(xx) - tx(2))
+ Var (I(O,)
N
00,00,
(a)
(x)
+2 Cov
(x)( - ta(y))
I(O: xx)
00
(6)
33

---

(2)
(a) Var
(na (Na( (2)) - (x())
00,00;
(maa(1)) - to (3))
1 (x)
27
E p(a.y)
00,00;
8 ht (xx)
E p(a.
(Na(x)) (a(())
00;00:
(Ina(o)) - ((n((t))t((()) (z))
th (x))
27
E p(a.y)
00,00;
hhh (x)
E
E
[[na(x))
-ta))))(()))
g(w)
00,00,
p
( (m)((()) - (z))
0² 1t" (x)
27
E p(m.y)
0
00 00,00,
@h((a))
[Ph(ar)
(m(x) t(y)) (m(x) - t(v))
=
E
80,00,
E
00,00;
g(x)
ply m)
(PP((x))
(02h(z))
]
E
I(h| (x)
00,00,
00,00;
(())
8h
(b) Cov
(na(a)) (x))
I(0 ())
00,00:
8h (a:)
E p(a v)
(na( =a()))
I(O: xx)
00,00;
0 ( (x)
(na(xx) (1))
E p(m,y)
[I(0: a))
00,00;
p(x,y)
0,
which follows by taking the 'partial' expectation (y x)) for both terms.
Thus together,
1 V2((+) N
1
(
(OPA(()) 00.00.
(ah((z)
Var (=)) N o(x))
E
I(h| (x)
a0,00,
P Proof of Lemma 4.8
Proof. The lower bound holds from just considering the non-negativity of variance. For the upper bound, we utilize the bound directly consider the bounds of Eq. (7),
Var (I(e, x))
[I(e;
x)
[I(0,
(())
q(m)
< [[(;; x>
((m)
< [[]0,h())]1
X ((h]))))
((m)
34

---

Proof of Proposition 5.1
We first derive the statistics I(h] as) and (t æ) presented in "Regression: Isotropic Gaussian Distribution" Section 5, It follows that from the regression setting, we have that,
F F(h(=)) = log ( exp(t (y)h(z))
/ T(y) exp(y h())),
-oog
where notably, by definition, (y) is independent of learned parameter h(x)). As such, we have that:
J (Ty)) exp(t (y)h(x)) / T(y) (p)(t())))()) h,(x)) y.).
Fh)
0h;
h=h(=)
ply m)
Now we note that Ey x) [BA]] is exactly (x()) the parameter h(x) specifies the mean of the (isotropic) multivariate normal distribution. As such we have that,
0
F(h)
h(xx)
h=h(x)
8²
I(h x)
F(h)
= I.
Ohōh
h=h(w)
Furthermore, by [37, Lemma 5], we have that,
(1F(h)
(atbaatt x))
Oh. h=h(2)
Iau(h x²) Ica (h x)) + Iac (h x) Ioa(h xx) + Iad xx) Ibc (h x))
-0 + (r x)) Inalh x) + Iac(h x) Ina(h x)) + (hd(h] x)) Ipe (h x))
(h x²) Ind (h. æ:) Inc (z) ) Iha(h x)) + Ind hh x) Inc (h) x))
In summary, have,
1C" abboa (t x)) Iab x)) Ins x)) Inc ( |x) Ioa(th x))
+ Iad x)) Inc ( (h x))
(K"(t) I(h I(h anbed (he x)) Inal ( x) Ina(h x)) (he w))
Proof. The minimum and maximum eigenvalues of I(h x)) follows directly noting that the trace of a matrix the sum of eigenvalues. As such, from the statistics presented above we have that the minimum and eigenvalue must be 1.
The tensor eigenvalues of (C æ)) - I(h .)) 8 I(h a) Iae a)) Iha(h x²) + Toa az). The (h æ) follows from the variational definition Eq. (10). For instance, for the minimum eigenvalue,
inf ut un (Inc(h|a) [ha(h] ar) + Ina ()) Iae (h/ (h|2))
u: |x/|a==] 11
2 inf |/11/2
1
2.
The maximum eigenvalue proven identically.
R Proof of Theorem 5.2
We first prove the following corollary which connects the maximum eigenvalues of KC(t m)) to the maximum eigenvalues of (h æ).
35

---

Corollary R.1. Suppose that the exponential family in Eq. (1) is specified by a categorical distribution.
Then,
Xmax (K(t| S 2- Amax (I [)))))))...))))))))))()))))))(
(31)
Proof. As we are consider a categorical distribution we have that
1 (h) yf= = 0.
0 (h)
yf
Thus we have that |wy] S 1. Furthermore, note that the maximum f2-norm that can have is ||ll2 S 22. Note that this is tight when the positive and negative mass are placed only two distinct coordinates, Le., (-1, .....
Thus using Proposition 4.3, the result follows.
Now by using Corollaries 4.2 and R.I, the remainder of the proof, all we require is the bounding of Amax (I(h) x)).
Proof. The first term in the maximum eigenvalue follows from,
Amax (I(h) x))
= Amax (Diag(or(z)) - (xx)
s Amax (Diag(r(z))) Amin (a(x))((z)))
max (x))
The second term in the maximum follows from the trace of I(h x) being the sum of total eigenvalues.
S Proof of Lemma 6.1
Proof. The proof follows from the standard definition of covariance. Denoting (xx) =
Eg(y w) [t()], have:
(tv alu (2)
[tytt (v)] (()))((()))
Also expanding 1(h) æ):
I(h x) = E
(t(y) (())) [7{+)
< x))
E
[ty)t (y) (=x) [x)))((x) ni (x)m (x)) n())n])))))) (x). (x))
Thus we have
Cov Cov (t|2) = I(hl x)) + + (x) (z) (x) - n(x)n (x) n (x)
I(h xx) - (p(z))-(=))((((> [
As required.
T Proof of Corollary G.1
Proof. We calculate the variance:
1
a log p x)
log q x))
V(O,)
E
E
N
00;
00;
36

---

Each of the terms can be calculated: Let (x, (t(y) - -n))).
(
a log p(y ar)
IF p(a..a)
00,
1h (m) (m) (x) (x))))
(x. y)do(ar (0) 8)
< m)
00:
08,
00,
00,
0h (x)
Oh"(a) (x) (h)
E
(1, (,, 9) ( 4)
00;
00:
00:
00:
ol o x))
(h)
0h
(a) 0h (x)
(abed(t|a))
00,
00,
a0,
00;
And:
log p(y x))
IE p(u
00;
(x) (x)
E
(x y)d y)
p(y (x)
00:
00;
(x)
(x)
(a, g))x(( 9)
00,
00,
p
x)
(a)
oh
Lab (te x))
00,
00,
Together with Lemma 6.1 proves the theorem.
U
Proof of Corollary G.2
Proof. The proof follows identically to that of Theorem 4.7 with densities changed.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims are justified in the formal results of the paper and justified in the numerical experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of our results are expressed through out the paper. See for instance Section which discusses computation implications.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
38

---

Answer: [Yes]
Justification: All assumptions are stated when expressing theoretic results and proofs are provided in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details of experiments are presented in the main-text and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
39

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Only simple plots are presented in the paper. The first figure a toy example of natural gradient descent with the FIM estimators. The other plots consider variance estimation and bounds of the paper in small scale networks.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: //n:://iiii //ninppp... cc/ public cublic/un gu des /CodeSu'mm ss onnol i cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa: / /niis. cc wtce//bblccggguee gegedd////ee/eee CodeSuume ss onPol oiyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details provided.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification: Multiple runs presented. Error bars not they obscure plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
40

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Presented in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer:
[Yes]
Justification:
We conform to the code ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work mainly theoretic. The societal impact our work would be equivalent to any societal impact of machine learning as field.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
41

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No data or model release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: License and credit are given for MNIST.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, papersw's thoode. com / datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
42

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "I88fvCPBSm", "reviewer": "Reviewer_7Xmq", "summary": "Summary\n-------\nThe paper studies two estimators for estimating the diagonal of a Fisher information\nmatrix (FIM) of a parametric machine learning model. Both estimators are based on equivalent\nexpressions for the FIM. The first is the standard definition, expressed in terms of the\nderivatives of the log likelihood, and the latter, expressed in terms of the second\nderivative when the model is twice differentiable with respect to the parameters. The\nestimators replace the expectation with an average over the data. The authors provide\nbounds on the variance of both (unbiased) estimators in terms of the minimum/maximum\neigenvalues of the FIM, and then provide bounds on these eigenvalues under special cases.\nThey also provide a modest empirical evaluation.\n\n\nOverall, while I think there is a fair amount of technical meat in the paper, I am \nunconvinced about the problem. I will wait to hear from more expert reviewers in this \nfield before I make my decision. \n\n\nDetailed comments:\n------------------\n\nWhile I am theoretically inclined, I am not an expert in this sub-field. Therefore, I will\nkeep my comments at a high level.\n\nSignificance of problem: I struggled to understand when estimating the FIM will be useful.\n- One point raised by authors is that it is used in optimizing NNs. If this was true, it\n  would have been nice to see the estimators used in NN optimization empirically, or a\n  theoretical analysis showing how the bounds derived here percolate to bounds on\n  optimization.\n- Another explanation given is that it could be used to analyze the structure fo NNs.\n  But this is stated vaguely, and it wasn't clear to me that such an analysis would help\n  design better neural networks.\n\nSignificance of techniques: The proof in the appendix is substantial but the authors have \nnot stated which techniques used here are novel. Currently technical results are presented\none after the other. It might help the reader/reviewer\nappreciate the contributions of the paper if the authors can highlight the novel tools \nthey used to derive these bounds.\n\nPresentation and writing: The technical content was clear for the most part. However,\nthe writing was very equation-driven. I understand that this is in part due to the nature \nof the paper, but the authors should simplify the exposition further. One option is to\ncut down on some of the technical content.", "questions": "See above", "limitations": "See above", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "See above", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for praising the technical developments of the paper.\n\n> Re: *Usefulness of FIM estimation, e.g. in NN optimization*\n\nAs stated in the submitted paper, the FIM can be used (Line 117) \"to study the\nsingular structure of the neuromanifold [2,40], the curvature of the loss [8],\nto quantify model sensitivity [28], and to evaluate the quality of the local\noptimum [15,16].\" Other applications include continual learning as pointed out\nby **Reviewer wDHQ**.\n\nWe believe studying the estimation quality of the FIM in the general setting --\nwithout examining the specific application and implications of these use cases\n-- is meaningful on its own.\nFirst, the bias and variance always exists for tackling the computational\ncost of estimating the (diagonal) FIM, but has not been thoroughly analyzed.\nSecond, such a study can be potentially applied to different scenarios.\nFor example, in NN optimization, the bounds of the diagonal FIM leads to bounds\nof a learning step, which is determined by the product of the inverse of the\ndiagonal FIM and the gradient vector. In our toy example in Figure 1, we\ndemonstrated how optimization can be interfered with the variance of the FIM.\nTo apply our results into specific optimizers would be another independent\nwork.\n\n> Re: *FIM for analyzing NN structure*\n\nIn the paper, we mentioned that one application of the FIM is to study the\n\"intrinsic structure of the neuromanifold\". Here, \"structure\" does not refers\nto the NN structure but the Riemannian geometry structure in the space of\nneural networks (aka the neuromanifold), where the FIM plays the role of a local\nmetric tensor. Our main results are universal in the sense that they do not\nrely on specific NN structure. On the other hand, the FIM is indeed affected by\nthe NN structure, which in the simplest case includes the depth and width of\nneural networks [15,16]. This is beyond the scope of the current paper.\n\n> Re: *Organization of technical contents*\n\nA main reason for the technical density is space limitation. We tried hard\nto layout the main results while providing enough examples with intuitions.\nTo address the reviewer's concern, we propose to not further increase the\ndensity and use extra space, if provided, to remark on the connections between\nthe equations -- to improve the paper's self-contained narrative (also\nsuggested by **Reviewer qMTe**) -- and remark on takeaways in the conclusion\n(suggested by **Reviewer QMbw**).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hmGJqnm2Xf", "reviewer": "Reviewer_wDHQ", "summary": "In this paper, the authors analyzed two different estimators, $I_1(\\theta)$ and  $I_2(\\theta)$, for the diagonal elements \nof the Fisher information matrix of a parametric neural net model. These diagonal elements are approximations for the entire matrix,\nwhich is unfeasible to be calculated in real neural nets models. They identified the situations in which each of these estimators is preferable with respect to the other.", "questions": "The results are all given using $\\theta$ equal to its true value. In practice, we use the MLE estimator of $\\theta$. The results \nare not valid anymore as we have this additional source of variability. The authors may comment on the consequences of this?", "limitations": "This is a theoretical paper with no immediate connection to societal impact.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The Fisher information matrix is a major parameter driving the quality of the fit and highlighting important aspects of the interdependence between the parameters. It has been used in several papers of continual learning, the research area that I follow. \nIt focuses on the diagonal elements of the Fisher information matrix, offering approximations that are feasible to calculate in real neural net models, addressing a significant practical challenge given the unfeasibility of computing the entire matrix. he authors identified specific situations where each of the two estimators, $I_1(\\theta)$ and  $I_2(\\theta)$, is preferable. This practical guidance can help researchers and practitioners make informed decisions about which estimator to use in different scenarios.", "weaknesses": "This is a highly theoretical paper and I have difficulty following all the findings presented by the authors. One of the main problems with this paper is that it has a large intersection with Soen and Sun (2021) and this becomes clear while we read the paper. In the Related work, Soen and Sun (2021) is mentioned in passing and it is a major reference work for the current manuscript. It is worth to have a better explanation of the relative merits of these two papers. Both papers analyzed two different estimators, $I_1(\\theta)$ and  $I_2(\\theta)$, for the Fisher information matrix of a parametric neural net model. While the older paper focuses on the matrix as a whole, the present manuscript focuses on the diagonal elements. However, the work of Soen and Sun (2021) is repeatedly referred to during the technical development of the paper. The contribution of this paper seems marginal when compared to the previous paper but still relevant.   \n\nThe results are derived assuming $\\theta$ is equal to its true value. In practice, the maximum likelihood estimator (MLE) of \n$\\theta is used, which introduces additional variability. The paper does not adequately address the implications of this assumption or the consequences when using the MLE, which limits the practical applicability of the results.\n\nThe paper does not discuss the impact of model misspecification on the results. In real-world scenarios, the data might not follow the proposed model exactly, and the inverse-Fisher information matrix might not represent the correct asymptotic variance of MLE estimators. The authors should comment on how their findings would be affected in such cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our strength and potential usefulness\nin application areas including continual learning.\n\n> Re: *Significance vs [37]*\n\nWe kindly refer the reviewer to our global response.\n\n> Re: *Assumption of* $\\\\theta$ *being the true value*\n\nWe clarify that our result holds for any $\\\\theta$ in the parameter space\nof neural networks, aka the neuromanifold. Our formal statements and\ntheir proofs do *not* depend on setting $\\\\theta$ to the true value or its MLE,\nnor to be \"nearby\" to the MLE in any sense.\nSimilar to how the FIM $\\\\mathcal{I}(\\\\theta)$ is a PSD tensor when $\\\\theta$\nvaries on the neuromanifold, our central subjects\n$\\\\hat{\\\\mathcal{I}}_1(\\\\theta)$, $\\\\hat{\\\\mathcal{I}}_2(\\\\theta)$,\n$\\\\mathcal{V}_1(\\\\theta)$, and $\\\\mathcal{V}_2(\\\\theta)$ are all tensors defined\nfor any $\\\\theta$.\n\n> Re: *What are the effects of model miss-specification*\n\nModel miss-specification can be examined from the perspective of empirical data\ndeviating from the joint model distribution $p(\\\\mathbf{x}, \\\\mathbf{y};\n\\\\mathbf{\\\\theta}) = q(\\\\mathbf{x})p(\\\\mathbf{y} \\\\mid \\\\mathbf{x}; \\\\mathbf{\\\\theta})$\nin Line 24. Our analysis does provide convenient tools to study this\nphenomenon.\n\nWhen the observed samples $\\\\mathbf{x}$ deviates from the \"true\"\n$q(\\\\mathbf{x})$, the error in estimating the FIM is described by the first term\nin Eq. (13): a high variance of the conditional FIM $\\\\mathcal{I}(\\\\theta_i \\\\mid\n\\\\mathbf{x})$ leads to a large error in estimating the FIM.\n\nOn the other hand, when the observed $(\\\\mathbf{x},\\\\hat{\\\\mathbf{y}})$'s sampling\ndeviates from the predictive model $p(\\\\mathbf{y} \\\\mid \\\\mathbf{x};\n\\\\mathbf{\\\\theta})$, data also plays a larger role.\nIn this case, the empirical Fisher becomes a biased estimation of the FIM,\nwhich is described by Lemma 6.1 and its surrounding text. Our FIM estimators do\nnot depends on the observed $\\\\hat{\\\\mathbf{y}}$ in the dataset and is not\naffected by such a bias. Note the difference in sampling of the labels\n$\\\\mathbf{y}$ / $\\\\hat{\\\\mathbf{y}}$ in the data FIM versus the FIM.\n\nWhen the model is well specified, the data FIM and FIM are equivalent (see Line\n328).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fMM9dhBAJ2", "reviewer": "Reviewer_qMTe", "summary": "The paper analyses two estimators of the Fisher Information matrix and specifically their variances, which (from reference 37) have closed-form but non-practical expressions. The authors show a sequence of inequalities and derive practical bounds for these variances, both element-wise and trace-wise, both conditioned to x and not. Several remarks and trade-offs on these bounds clarify when an estimator is preferable over the other.\nThe results hold in the general exponential family likelihood but are also further concretized for regression and classification.", "questions": "In line 168. Shouldn't \"of Eqs. (4) and (6)\" instead be \"of Eqs. (4) and (5)\"? And consistently shouldn't \"$\\mathcal{V}_1(\\theta_i|x)$\" be \"$\\mathcal{V}_2(\\theta_i|x)$\"? \\\nI ask this because \"small shifts in parameter space yield large changes in the output\" to me refers to the magnitude of the network jacobian $||\\partial_i h(x)||$, and that appear in the bound for $V_1$, not for $V_2$. Am I misunderstanding something or is this a typo?\n\nIn line 246. Isn't the bound sample complexity $\\mathcal{O}(\\frac{1}{N_x} + \\frac{1}{N_y})$? I can't see why it should be the product, can you elaborate more on this derivation?", "limitations": "I think it would be nice to clarify better that the work is incremental on \"Soen et all - On the variance of the Fisher information for deep learning\" [reference 37] and their closed form expression reported in the appendix in Lemma B.1. (also it would be nice to move these closed form expressions to the main paper).", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is overall very well written and structured. \n\nNotation is clear, results are properly presented and every step is further clarified in the appendix. \\\nThe alternation between formal statements and explanations is great and makes the paper vey pleasant to read. I particularly like the amount of remarks and observations that discuss specific terms of the equations, explains them and connect them to each other. And the same is true for all the highlighted trade-offs (for example line 212-215)", "weaknesses": "I have found no major weaknesses in the paper. I report some minor things.\n\nI personally dislike the notation $\\hat{\\mathcal{I}}_1(\\theta_i)$ for the i-th element of the diagonal introduced in line 87. This notation suggest that this term only depends on $\\theta_i$, but it actually depends on the whole $\\theta$ vector. On the other hand, this is quite a light easy notations, and I can't think of alternatives which are not much heavier to read.\n\nAlso, I find the definition in line 120 \"is defined by $\\hat{\\mathcal{I}}_j(\\theta_i)$ with $x=x_1=...=x_N$\" to be yes understandable but not super clean. The alternatives are not as compact, but I'd recommend using the extra space to clarify this a bit better. At least you should write that $y_1,...,y_N$ are i.i.d. from $p(y|x;\\theta)$ and not from $p(y,x;\\theta)$ as in $\\hat{\\mathcal{I}}_j(\\theta_i)$.\n\nIt would be nice to explicitly write the proof for the variance closed expression in Lemma B.1, yet clarifying that it's not a paper contribution. This is for two reasons: (1) it would make the paper self contained, without the need for the reader to read reference 37 and (2) the notation is slightly different and it would be easier for the reader to be consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our contributions and praising our writing.\nWe respond to your remarks and suggestions as below.\n\n> Re: *The notation* $\\\\hat{\\\\mathcal{I}}_{1}(\\\\theta_i)$\n\nWe propose to explicitly define it as $\\\\hat{\\\\mathcal{I}}_{1}(\\\\theta_i) := (\\\\hat{\\\\mathcal{I}}_1(\\\\theta))\\_{i i}$ at the first appearance of this notation in\nthe beginning of Section 3, and add a sentence to remind the reader that\n$\\\\hat{\\\\mathcal{I}}_1(\\\\theta_i)$ is an abuse of notation and actually depends on\nthe whole $\\\\mathbf{\\\\theta}$ vector.\n\n> Re: *The notation* $\\\\hat{\\\\mathcal{I}}_j(\\\\theta_i \\\\mid \\\\mathbf{x})$ *with*\n> $\\\\mathbf{x}=\\\\mathbf{x}_1=\\\\cdots=\\\\mathbf{x}_N$\n\nWe will rewrite the sentence to clarify that $y_1,\\\\cdots,y_N$ are i.i.d. from\n$p(y \\\\mid \\\\mathbf{x}; \\\\mathbf{\\\\theta})$ as the reviewer has suggested.\n\n> Re: *Explicit proof of Lemma B.1 to make the paper self-contained*\n\nWe agree to include such a proof that is consistent with our current notation.\n\n> Re: *Line 168, Eqs. (4) and (6), the scale of* $\\\\mathcal{I}(\\\\theta_i \\\\mid\n> \\\\mathbf{x})$ *and* $\\\\mathcal{V}_2(\\\\theta_i \\\\mid \\\\mathbf{x})$\n\nYes, this is a typo and the corrections suggested by the reviewers are\ncorrect. Eqs. (4) and (5) are the bounds which depend on the network Jacobian.\n\n> Re: *Line 246, sample complexity should not be* $\\\\mathcal{O}(\\\\frac{1}{N_xN_y})$\n\nWe thank the reviewer for pointing out this mistake. By the total variance\ndecomposition in Eq. (13), the variance wrt $q(\\\\mathbf{x})$ scales inversely to\n$N_x$, the number of $\\\\mathbf{x}$-samples, and the variance wrt $p(y \\\\mid\n\\\\mathbf{x})$ scales inversely to $N_y$. Therefore the total variance is\nin the order of $\\\\mathcal{O}\\\\left(\\\\frac{1}{N_x}+\\\\frac{1}{N_y}\\\\right)$ as the\nreviewer suggested. We will make sure to correct this sentence and related\nplaces in this paragraph.\n\n> Re: *Relationship with reference [37]*\n\nPlease see our global response. We shifted the closed-form expression of\nthe variances to the appendix mainly due to space limitation. Should more space\nbe permitted, we agree to improve our self-contained narrative by moving them\nback into the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OahKtQqkuh", "reviewer": "Reviewer_QMbw", "summary": "The authors study two popular estimators of the Fisher information matrix with respect to neural network parameters. They derive upper and lower bounds for the variance of these estimators and showcase them in applications to regression and classification problems.", "questions": "- The authors should clarify clearly to what extent their submission improves upon previous works, particularly [Soen, Alexander, and Ke Sun. \"On the variance of the Fisher information for deep learning.\" Advances in Neural Information Processing Systems 34 (2021): 5708-5719]. How do their bounds compare to the ones previously derived?\n\n- In my opinion, the impact of the submission would be substantially improved if the authors state more clearly in what scenarios which of the two estimators will be preferred and provide a rough guideline for practitioners.\n\n- Could the authors provide a numerical example for regression similar to Fig. 2?", "limitations": "There is no further need for authors to address limitations.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Analyzing the convergence properties of Fisher information estimators is a timely and nontrivial endeavor, in part due to their use in natural gradient descent.\n- The paper is very extensive and mathematically sound.", "weaknesses": "- The paper is highly technical and the numerical examples are fairly toy.\n- The paper takes strong inspiration from a previous work [Soen, Alexander, and Ke Sun. \"On the variance of the Fisher information for deep learning.\" Advances in Neural Information Processing Systems 34 (2021): 5708-5719] on the topic. As such, I question the value the present submission adds to this line of research. In particular, I think the submission does a poor job in highlighting the central novel aspects of their work and reviewing the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our contributions and their soundness.\n\n> Re: *Significance vs [37]*\n\nPlease see our global response to all reviewers.\n\n> Re: *Better clarity on which estimator is preferred*\n\nThrough our analysis, We have identified certain scenarios where one estimator\nis superior to the other.\n\nFor example, ReLU networks can only apply $\\\\hat{\\\\mathcal{I}}_1$. As another\nexample, $\\\\hat{\\\\mathcal{I}}_2$ has zero variance in the last layer and is\nalways preferable than $\\\\hat{\\\\mathcal{I}}_1$. Similarly, in the second last\nlayer, $\\\\hat{\\\\mathcal{I}}_2$ has a simple closed form and preferable for\nneurons in their linear regions (see Remark 4.5).\n\nIn general, Theorem 4.1 suggests that the variance of both estimators depend on\nthree factors: (1) the number of samples; (2) the derivatives of the neural\nnetwork output wrt its parameters; (3) moments of the sufficient statistics\n$\\\\mathbf{t}(\\\\mathbf{y})$ (see Table 1). One has to incorporate these factors\nbased on their specific neural network and settings, to decide which estimator\nto use. We agree with the reviewer to recall these points in the conclusion\nsection.\n\n> Re: *Numerical examples are fairly toy*\n\nOur contributions are mainly theoretical and do not depend on the empirical\nresults. In the paper and appendix, we use MNIST MLP with different activations\n(sigmoid and log-sigmoid) and different number of layers (4 and 5) to showcase\nthe bounds of the variance and how the variance evolves over training time in\ndifferent layers of the neural network. These numerical examples mainly serves\nthe purpose of providing intuition. We do agree with the reviewer that more\nextensive empirical studies are meaningful as future work.\n\n> Re: *Numeric example for regression similar to Fig (2)*\n\nWe thank the reviewer for this suggestion. As remarked by Proposition 5.1, for\nregression, all bounds in Theorem 4.1 becomes equalities. Therefore it is less\nmeaningful to examine the case of regression, where all interested quantities\n$\\\\mathcal{I}$, $\\\\mathcal{V}_1$, and $\\\\mathcal{V}_2$ reduce to evaluation of the\nJacobian/Hessian of the parameter-output mapping. The limited time for the\nrebuttal does not allow for redesigning the pipeline of FIM estimation on a new\ndataset and learning task. We therefore hesitate to overstate our empirical\ndiscoveries or to promise experimental extensions without clearly understanding\ntheir implications.\n\nWe would also like to iterate that aim of this paper's contribution is\nprimarily theoretical. All though additional numerical experiments would be\nnice, a thorough independent empirical study is out of this paper's scope and\nis arguably would make an independent piece of work (examining various\nsettings, optimizers, and architectures etc).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our appreciation to all reviewers for their thoughtful reviews.
We are pleased to acknowledge the positive remarks on "[...] fair amount of technical
meat [...]" (**Reviewer 7Xmq**) and that the paper "[...] is very
extensive and mathematically sound" (**Reviewer QMbw**).
Despite the technical nature of the paper, we are glad to hear that the "[...]
alternation between formal statements and explanations is great and makes the
paper very pleasant to read" (**Reviewer qMTe**). We are also happy to learn about
other applications of the FIM, where it has been "[...] used in several papers of
continual learning [...]" (**Reviewer wDHQ**), which extends the potential
applicability of our results.

Please find the per-point rebuttals in each reviewer's response section.
Immediately below, we provide a shared response regarding our paper's relation to
reference [37]:

> [37] Soen and Sun. On the variance of the Fisher information for deep learning.
In Advances in Neural Information Processing Systems, 2021.


## Relationship to [37]
In response to **Reviewer wDHQ**, **Reviewer qMTe**, and **Reviewer QMbw**, we
hereby clarify our relationship with [37]. Indeed, both the current submission
and [37] study the same subject, and our developments rely on the closed-form
expression of the FIM variances, which is proved in [37]. Despite that, the
significance of the current work is explained as below.

- In [37], only the norm of the FIM and related variance tensors are bounded. By
  focusing on the diagonal elements, we derive novel bounds on the individual
  elements of these interested tensors, where the spectrum of sufficient
  statistics quantities naturally appear. In terms of proof techniques, [37]
  mostly utilize Holder's inequality whilst we utilize variational definitions /
  computations of eigenvalues.

- The main subjects in [37] are 4D tensors and their norms, which can not be
  easily computed. Their case studies are constrained to 1D exponential families.
  Our results leads to numerical algorithms that can be implemented through
  auto-differentiation. As a result, our bounds extend to typical learning
  settings (Section 5).

- We discussed not only variances but also bias in the estimation of the FIM,
  and clarified the relationship with empirical FIM (data FIM) that is widely
  used.

- [37] only considered the conditional FIM and associated objects in its study.
  Hence, their results only accounts for the sampling of $\\mathbf{y}$ with a
  fixed $\\mathbf{x}$.
  Our variance decomposition in Theorem 4.7 clarified how the variance is
  affected by both the sampling of $\\mathbf{x}$ and the sampling of
  $\\mathbf{y}$.

We are happy to include this discussion in Section 1 (around Line 42) and other relevant
places by utilizing additional space, if provided, to clarify the contributions
of our paper.

Meta-review: All reviewers lean towards accept with at least one reviewer advocating for a strong accept. This is a highly theoretical paper that is likely to be of interest to others interested in theory. 

However, there are several weaknesses discussed by the reviewers that should be addressed in the final version. In particular, more discussion on the differences between this work and [37] should be included. Also, the authors should work to better "remark on the connections between the equations" in the technical sections to make these sections more accessible and also further discuss when one estimator should be preferred over the other.
Decision: Accept (poster)