Title: Trade-Offs of Diagonal Fisher Information Matrix Estimators
Venue: NeurIPS 2024
Accepted: yes
Keywords: Fisher Information

Deep Learning

Information Geometry

Neuromanifold
Abstract: The Fisher information matrix can be used to characterize the local geometry of
the parameter space of neural networks. It elucidates insightful theories and
useful tools to understand and optimize neural networks. Given its high
computational cost, practitioners often use random estimators and evaluate only
the diagonal entries. We examine two popular estimators whose accuracy and sample
complexity depend on their associated variances. We derive bounds of the
variances and instantiate them in neural networks for regression and
classification. We navigate trade-offs for both estimators based on analytical
and numerical studies. We find that the variance quantities depend on the
non-linearity w.r.t. different parameter groups and should not be neglected when
estimating the Fisher information.

Trade-Offs of Diagonal Fisher Information Matrix Estimators

Abstract

The Fisher information matrix can be used to characterize the local geometry of the parameter space of neural networks. It elucidates insightful theories and useful tools to understand and optimize neural networks. Given its high computational cost, practitioners often use random estimators and evaluate only the diagonal entries. We examine two popular estimators whose accuracy and sample complexity depend on their associated variances. We derive bounds of the variances and instantiate them in neural networks for regression and classification. We navigate trade-offs for both estimators based on analytical and numerical studies. We find that the variance quantities depend on the non-linearity w.r.t. different parameter groups and should not be neglected when estimating the Fisher information.

1 Settings

In the parameter space of neural networks (NNs), i.e . the neuromanifold [1], the network weights and biases play the role of a coordinate system and the local metric tensor can be described by the Fisher Information Matrix (FIM). As a result, empirical estimation of the FIM helps reveal the geometry of the loss landscape and the intrinsic structure of the neuromanifold. Utilizing these insights has lead to efficient optimization algorithms, e.g ., the natural gradient [1] and Adam [16].

A NN with inputs x and stochastic outputs y can be specified by a conditional p.d.f. p ( y | x ; θ ) , where θ is the NN's weights and biases. This paper considers the general parametric form

where h θ : ℜ I → ℜ T maps I -dimensional inputs x to T -dimensional exponential family parameters, t ( y ) is a vector of sufficient statistics, π ( y ) is a base measure, and F ( · ) is the log-partition function (normalizing the exponential). For example, if y denotes class labels and t ( y ) maps to its corresponding one-hot vectors, then Eq. (1) is associated with a multi-class classification network.

Assuming that the marginal distribution q ( x ) is parameter-free, we define parametric joint distributions p ( x , y ; θ ) = q ( x ) p ( y | x ; θ ) . The (joint) FIM is defined as I ( θ ) . = E q ( x ) [ I ( θ | x )] , where

is the 'conditional FIM'. The second equality (*) holds if h θ 's activation functions are in C 2 ( ℜ ) ( i.e ., h θ is a sufficiently smooth NN). I ( θ | x ) does not have this equivalent expression (*) for NNs with ReLU activation functions [37]. Both I ( θ ) and I ( θ | x ) define dim( θ ) × dim( θ ) positive semi-definite (PSD) matrices. The distinction in notation is to emphasize that the joint FIM I ( θ ) (depending only on θ ) is simply the average over individual conditional FIMs I ( θ | x ) (depending on both θ and x ).

Ke Sun

0

8

32

64

128

256

1

2

3

Test Loss (NLL)

Regression (Gaussian)

Trajectories

0

8

32

64

128

256

0

.

65

0

.

70

Classification (Categorical)

Trajectories

NG with

ˆ

I

1

NG with

ˆ

I

2

Initial Params

ˆ

θ

0

True Params

θ

In practice, the FIM is typically computationally expensive and needs to be estimated. Given q ( x ) and a NN with weights and biases θ parameterizing p ( y | x ; θ ) , as per Eq. (1), we consider two commonly used estimators of the FIM [11, 37] given by

where p ( y k | x k ) . = p ( y k | x k ; θ ) and ( x 1 , y 1 ) , . . . , ( x N , y N ) are i.i.d. sampled from p ( x , y ; θ ) . A conditional variant of the estimators, denoted as ˆ I 1 ( θ | x ) and ˆ I 2 ( θ | x ) , can be defined by fixing x = x 1 = · · · = x N and sampling y 1 , . . . , y N independently from p ( y | x ) in Eq. (3) - details omitted for brevity.

Both estimators, ˆ I 1 ( θ ) and ˆ I 2 ( θ ) , are random matrices with the same shape as I ( θ ) . By Eq. (2), they are unbiased -for ˆ I 2 ( θ ) , this only holds if activations functions are in C 2 ( ℜ ) . Following Eq. (1)'s setting, the estimation variances of ˆ I 1 ( θ ) and ˆ I 2 ( θ ) can be expressed in closed form and upper bounded [37]. This provides an important, yet not widely discussed, tool for quantifying the estimators' accuracy [11] and hence insights for where / when different estimators should be used. Despite this, for deep NNs, neither these variances nor their bounds can be computed efficiently due to the huge dimensionality of θ .

This work focuses on estimating the diagonal entries of the FIM and their associated variances. Our results - including estimators of the FIM, their variances, and their variance bounds - can be implemented through automatic differentiation. These computational tools empower us to practically explore the trade-offs between the two estimators. For example, Fig. 1 shows natural gradient descent [1] for generalized linear models on a toy dataset, where ˆ I 2 ( θ ) is preferable (especially for regression) and ˆ I 1 ( θ ) suffers from high variance and an unstable learning curve. Our analytical results reveal how moments of the output exponential family and gradients of the NN in Eq. (1) affects the FIM estimators. We discover a general decomposition of the estimators' variances corresponding to the samples of x and y . We investigate different scenarios where each FIM estimator is the preferred one and then connect our analysis to the empirical FIM.

2 Related Work

Prior efforts aim to analyze the structure of the FIM of NNs with random weights [34, 14, 15, 3, 31]. This body of work hinges on utilizing tools from random matrix theory and spectral analysis, characterizing the behavior and statistics of the FIM. One insight is that randomly weighted NNs have FIMs with a majority of eigenvalues close to zero; with the other eigenvalues taking large values [14, 15]. In our work, the randomness stems from sampling from data distributions p ( x , y ) -which follows the principle of Monte Carlo (MC) information geometry [29] that approximates information geometric quantities via MC estimation. We examine a different subject on how the distribution of the FIM on a matrix manifold is affected by finite sampling of the data distribution.

In the literature of NN optimization, a main focus is on deriving a computationally friendly proxy for the FIM. One can consider the unit-wise FIM [30, 20, 39, 3] (also known as quasi-diagonal FIM [30]), where a block-diagonal approximation of the FIM is taken to capture intra-neuron curvature information. Or one can consider the block-diagonal layer-wise FIM where each block corresponds to parameters within a layer [19, 27, 32, 26, 12, 35, 13]. NN optimizers can approximate the inverse FIM [36] or approximate the product of the inverse FIM and the gradient vector [35].

Much less attention is paid to how related approximations deviate from the true FIM [11, 37] or how optimization is affected by such deviation [41]. For the univariate case, one can study the asymptotic variance of the Fisher information [11] with the central limit theorem. In deep NNs, the estimation variance of the FIM can be derived in closed form and bounded [37]. However, our former analysis [37] has two limitations: (1) the variance tensors are 4D and can not be easily computed; (2) only the norm of these tensors are bounded, and it is not clear how the variance is distributed among individual parameters. The current work tackles these limitations by focusing on the diagonal elements of the FIM. Our results can be computed numerically at a reasonable cost in typical learning settings. We provide novel bounds so that one can quantify the accuracy of the FIM computation w.r.t. individual parameters or subgroup of parameters.

Issues of utilizing the empirical FIM to approximate the FIM have been highlighted [32, 25]. For example, estimators of the FIM do not in general capture any second-order information about the log-likelihood [18]. The empirical FIM is a biased estimator and can be connected with our unbiased estimators via a generalized definition of the Fisher matrix in Section 6.

Alternative to the FIM, the Generalized Gauss-Newton (GGN) matrix - a Hessian approximator was originally motivated through the squared loss for non-linear models [25]. The GGN is equivalent to the FIM when a loss function is taken to be the empirical expectation of the negative log-likelihood of Eq. (1) [12, 32, 25].

3 Variance of Diagonal FIM Estimators

In our notations, all vectors such as x , y , and θ are column vectors. We use k to index random samples x and y and use i and j to index the NN weights and biases θ . We shorthand h . = h θ , p ( y | x ) . = p ( y | x ; θ ) , and p ( y , x ) . = p ( y , x ; θ ) whenever the parameters θ is clear from context. To be consistent, we use ' | x conditioning' to distinguish between jointly calculated values versus conditioned values with fixed x . By default, the derivatives are w.r.t. θ . For example, ∂ i h . = ∂ h /∂θ i and ∂ 2 i h . = ∂ 2 h /∂θ 2 i . We adopt Einstein notation to express tensor summations, so that an index appearing as both a subscript and a superscript in the same term indicates a summation. For example, x a y a denotes ∑ a x a y a . For clarity, we mix standard Σ -sum and Einstein notation. We denote the variance and covariance of random variables by Var( · ) and Cov( · ) , respectively.

Based on the parametric form of the model in Eq. (1), the diagonal entries of the FIM estimators in Eq. (3) can be written as 1 :

Correspondingly, the i 'th diagonal entry of the FIM I ( θ ) , which is the expected value of ˆ I 1 ( θ i ) and ˆ I 2 ( θ i ) , is denoted as I ( θ i ) . Notation is abused in I ( θ i ) , ˆ I 1 ( θ i ) , and ˆ I 2 ( θ i ) as they depend on the whole θ vector rather than solely on θ i . Clearly ˆ I 1 ( θ i ) ≥ 0 , while there is no guarantee for ˆ I 2 ( θ i ) which can be negative. Our results will be expressed in terms of the (central) moments of t ( y ) :

where ' ⊗ ' denotes the tensor product. We denote the covariance of t w.r.t. to p ( y | x ) as Cov p ( t | x ) -noting that I ( h | x ) = Cov p ( t | x ) . The 4D tensor K p ( t | x ) denotes the 4 th central moment of t ( y ) w.r.t. p ( y | x ) . These central moments correspond to the cumulants of t ( y ) , i.e . the derivatives of F w.r.t. the natural parameters h ( x ) of the exponential family. Therefore, the derivatives of F in ˆ I 1 ( θ i ) and ˆ I 2 ( θ i ) can further be written in terms of η ( x ) and I ( h | x ) following the chain rule. Practically, ˆ I 1 and ˆ I 2 involves computing the Jacobian ∂ h ( x ) /∂θ i and the Hessian ∂ 2 h ( x ) /∂ 2 θ i , respectively.

In practice, both estimators can be computed via automatic differentiation [33, 6]. In terms of complexity, by restricting to just the diagonal elements I ( θ i ) , we need to calculate O (dim( θ )) elements (originally O (dim( θ ) × dim( θ )) for the full FIM). Although the log-partition function for general exponential family distributions can be complicated, for the ones used in NNs (determined by the loss functions used in optimization) [37] the log-partition function F is usually in closed-form; and thus the cumulants η ( x ) and I ( h | x ) can be calculated efficiently.

Indeed, the primary cost of the estimators comes from evaluating the gradient information of the NN, given by ∂ h ( x ) /∂θ i and ∂ 2 h ( x ) /∂ 2 θ i . The former can be calculated easily. The latter is costly even when restricted to the diagonal elements of the FIM. With the Hessian's quadratic complexity, in practice approximations are used to reduce the computational overhead [4, 45, 46, 8]. In this case, additional error and (potentially) variance may be introduced as a result of the Hessian approximation. Note, the computational cost of the Hessian can still be manageable for the last few layers close to the output. By the chain rule, we only require a sub-computational graph from the output layer to a certain layer to compute the Hessian of that layer. Despite this, there is still a memory cost that scales quadratically with the number of parameters for non-linear activation functions [6].

The high cost of Hessian computation does not justify refraining from using ˆ I 2 . Depending on the setting (chosen loss function), an estimator's variance can outweigh the benefits of lower computational costs [37]. This is especially true when the FIM is used in an offline setting - where the Hessian's cost can be tolerated - to study, e.g ., the singular structure of the neuromanifold [2, 40], the curvature of the loss [7], to quantify model sensitivity [28], and to evaluate the quality of the local optimum [14, 15], etc .

To study the quality of ˆ I 1 ( θ ) and ˆ I 2 ( θ ) , it is natural to examine the variance of the estimators [37]: V j ( θ i | x ) . = Var( ˆ I j ( θ i | x )) , where ˆ I j ( θ i | x ) . = ( ˆ I j ( θ | x ) ) ii ( j ∈ { 1 , 2 } ) is the i 'th diagonal ˆ ˆ ˆ ˆ

element of I j ( θ | x ) . Similar to I 1 ( θ i ) and I 2 ( θ i ) , V j ( θ i | x ) and I j ( θ i | x ) depend on the vector θ and are abuses of notation. An estimator with a smaller variance indicates that it is more accurate and more likely to be close to the true FIM. Based on the variance, one can derive sample complexity bounds of the diagonal FIM via Chebyshev's inequality, see for instance [37, Section 3.4].

By its definition, V j ( θ i | x ) has a simple closed form, which was proved in [37] and is restated below.

Given a fixed x ∈ ℜ I , both V 1 ( θ i | x ) and V 2 ( θ i | x ) have an order of O (1 /N ) , with N denoting the number of samples of y k . They further depend on two factors: ① the derivatives of the parameteroutput mapping θ → h stored in a T × dim( θ ) matrix, either ∂ i h a ( x ) or ∂ 2 i h a ( x ) , where the latter can be expensive to calculate; and ② the central moments of t ( y ) , whose computation only scales with T (the number of output units) and is independent to dim( θ ) .

From an information geometry [1] perspective, I ( θ ) , V 1 ( θ ) , and V 2 ( θ ) are all pullback tensors of different orders. For example, I ( θ ) is the pullback tensor of I ( h ) and the singular semi-Riemannian metric [40]. They induce the geometric structures of the neuromanifold (parameterized by θ ) based on the corresponding low dimensional structures of the exponential family (parameterized by h ).

4 Practical Variance Estimation

To further understand the dependencies of the derivative and central moment terms, the FIM I ( θ i | x ) and variances of estimators ˆ I j ( θ i | x ) can be bounded to strengthen intuition and to provide a computationally convenient proxy of the interested quantities.

Theorem 4.1. ∀ x ∈ ℜ I ,

where M = K p ( t | x ) -I ( h | x ) ⊗I ( h | x ) ; λ min / λ max denotes the minimum / maximum matrix eigenvalue; and ˜ λ min , ˜ λ max : ℜ T × T × T × T →ℜ are defined as

To help ground Theorem 4.1, we summarize different sufficient statistics quantities for common learning settings in Table 1 - with further learning setting implications presented in Section 5. Note that Eqs. (8) and (9) (and many subsequent results) can be further generalized for off-diagonal elements. See Appendix C for details. Compared to prior work [37], Theorem 4.1 provides bounds for individual elements of the variance tensors, where the NN weights (the derivatives) and sufficient statistics (the eigenvalues) are neatly disentangled into a product. From a technical point of view, this comes from a difference in proof technique: we utilize variational definitions and computations of eigenvalues to establish bounds whereas [37] primarily applies Hölder's inequality.

We stress that ˜ λ min ( T ) and ˜ λ max ( T ) in Eq. (10) correspond to tensor eigenvalues iff T is a supersymmetric tensor [23] (a.k.a. totally symmetric tensor), i.e ., indices are permutation invariant. In this case, Eq. (10) is exactly the maximum and minimum Z-eigenvalues. These variational forms mirror the Courant-Fischer min-max theorem for symmetric matrices [42]. In the case of Eq. (8), with M = K p ( t | x ) - I ( h | x ) ⊗ I ( h | x ) , the tensor is not a supersymmetric tensor in general. Despite this, we note that the lower bound of Eq. (8) is non-trivial. A weaker bound than Eq. (8) can be established based on the Z-eigenvalue of the supersymmetric tensor K p ( t | x ) .

The tensor eigenvalue is typically expensive to calculate. However in our case, the eigenvalues ˜ λ min ( K p ( t | x )) and ˜ λ max ( K p ( t | x )) on the RHS of Eqs. (11) and (12) can be calculated via [17]'s method with O ( T 4 / 4!) complexity. In this paper, we assume T is reasonably bounded and are mainly concerned with the complexity w.r.t. dim( θ ) . From this perspective, all our bounds scale linearly w.r.t. dim( θ ) , and thus can be computed efficiently.

When t ( y ) -η ( x ) is bounded ( e.g . in classification), we can upper bound ˜ λ max ( K p ( t | x )) with λ max ( I ( h | x )) , which is easier to calculate.

Proposition 4.3. Suppose ∥ t ( y ) -η ( x ) ∥ 2 2 ≤ B . Then, ˜ λ max ( K p ( t | x )) ≤ Bλ max ( I ( h | x )) ≤ B 2 .

As long as the sufficient statistics t ( y ) has bounded norm ∥ t ∥ 2 , we have that ∥ t ( y ) -η ( x ) ∥ 2 2 ≤ 4 ∥ t ∥ 2 2 < ∞ . A similar lower bound can be established for the minimum tensor eigenvalue ˜ λ min ( K p ( t | x )) ≥ λ 2 min ( I ( h | x )) , but this ends up being trivial when applying Corollary 4.2's lower bound, Eq. (11).

Examining Theorem 4.1 reveals several trade-offs. An immediate observation is that the first order gradients of h ( x ) correspond to the robustness of h to parameter misspecification (w.r.t. an input

x ). As such, from the bounds in Eqs. (7) and (8), the scale of I ( θ i | x ) and V 1 ( θ i | x ) will be large when small shifts in parameter space yield large changes in the output h ( x ) . Another observation is how the spectrum of I ( h | x ) affects the scale of I ( θ i | x ) and the estimator variances. In particular, when λ min ( I ( h | x )) increases, the scale of V 1 ( θ i | x ) decreases but the scale of I ( θ i | x ) and V 2 ( θ i | x ) increases. When λ max ( I ( h | x )) decreases, then the opposite scaling occurs. With these two observations, there is a tension in how the scale of I ( θ i | x ) follows the different variances V 1 ( θ i | x ) and V 2 ( θ i | x ) . The element-wise FIM I ( θ i | x ) follows V 1 ( θ i | x ) in terms of the scale of NN derivatives ∥ ∂ i h ( x ) ∥ 2 ; at the same time, I ( θ i | x ) follows V 2 ( θ i | x ) in terms of the spectrum of sufficient statistics moment I ( h | x ) .

Remark 4.4. Typically, h is the linear output units: h ( x ) = W -1 h -1 ( x ) , where W -1 is the weights of the last layer, and h -1 ( x ) is the second last layer's output. We have V 2 ( θ i | x ) = 0 ≤ V 1 ( θ i | x ) for any θ i in W -1 . A smaller variance V 2 ( θ i | x ) is guaranteed for the last layer regardless of the choice of the exponential family in Eq. (1).

̸

Remark 4.5. h ( x ) = w j -1 ϕ ( h ⊤ -2 ( x ) w j -2 + C -2 ) + c -1 defines the NN mapping w.r.t. the j 'th neuron in the second last layer, where w j -2 and w j -1 are incoming and outgoing links of the interested neuron, respectively; h -2 ( x ) is the output of the third last layer; and ϕ is the activation function. The 'constants' C -2 and c -1 denote an aggregation of all terms which are independent of w j -2 and w j -1 in their respective layers. The Hessian of h k ( x ) w.r.t. w j -2 is ∂ 2 h k ( x ) = ( w j -1 ) k · ϕ ′′ ( h ⊤ -2 ( x ) w j -2 + C -2 ) · ( h -2 ( x ) h ⊤ -2 ( x )) . By Theorem 4.1, V 2 ( θ i | x ) can be arbitrarily small depending on ϕ ′′ ( h ⊤ -2 ( x ) w j -2 + C -2 ) . For example, if ϕ ( t ) = 1 / (1 + exp( -t )) , then ϕ ′′ ( t ) = ϕ ( t )(1 -ϕ ( t ))(1 -2 ϕ ( t )) . In this case, for a neuron in the second last layer, a sufficient condition for V 2 ( θ i | x ) = 0 (and having ˆ I 2 favored against ˆ I 1 ) is h ⊤ -2 ( x ) w j -2 + C -2 = 0 for the neuron's pre-activation. When the pre-activation value is saturated ( -∞ or ∞ ), we also have that V 1 ( θ i | x ) = V 2 ( θ i | x ) = 0 . Alternatively, suppose that ϕ ( t ) = SoftPlus ( t ) . = log(1 + exp( t )) , a continuous relaxation of ReLU , then ϕ ′′ ( t ) = ϕ ′ ( t )(1 -ϕ ′ ( t )) where ϕ ′ ( t ) = 1 / (1 + exp( -t )) . Then a sufficient condition for V 2 ( θ i | x ) = 0 with V 1 ( θ i | x ) = 0 for a neuron in the second last layer is h ⊤ -2 ( x ) w j -2 + C -2 → + ∞ .

These observations are further clarified by looking at related quantities over multiple parameters. So far we have only examined the variance of the FIM element-wise w.r.t. parameters θ i . To study all parameters θ jointly, we consider the trace variances of the FIM estimators: for any j ∈ { 1 , 2 } , V j ( θ | x ) denotes the trace of the covariance matrix of diag( ˆ I j ( θ | x )) , where diag( · ) extracts a matrix's diagonal elements into a column vector. We present upper bounds of these joint quantities.

Corollary 4.6. For any x ∈ ℜ I ,

where dHes( h | x ) . = (diag(Hes( h 1 | x )) , . . . , diag(Hes( h T | x ))) and ∥· ∥ F is the Frobenius norm.

This upper bound comes from integrating the parameter-wise variances in Theorem 4.1 and incorporating a trace variance bound which utilizes the full spectrum of the NN derivatives and sufficient statistics quantities. This is fully depicted in Theorem D.1. Lower bounds can also be derived in terms of singular values (deferred to the Appendix). Note the upper bounds in Corollary 4.6 can be improved by expressing the min function's first term with singular value quantities.

Having the min function in Corollary 4.6 is helpful as it shows a trade-off between two upper bounds: the scale of NN derivatives ∂ h ( x ) and ∂ 2 h ( x ) versus the spectrum of the sufficient statistic terms. In the case of Eqs. (13) and (15), the trace of I ( h | x ) is exactly the sum of all eigenvalues, including λ max ( I ( h | x )) . This can be helpful when the scale of the NN derivatives are not bounded by a small value. It should be noted that, by the chain rule, these NN derivatives scale with the overall sharpness / flatness [22] of the landscape of the loss, i.e ., the log-likelihood of Eq. (1). For NNs with large derivatives, the first term of the min could yield tight bounds of the variance, and one can therefore avoid dealing with the quadratic scaling of ∥ ∂ h ( x ) ∥ in the second term. On the other hand, if the sharpness of the NN h can be controlled, e.g . via sharpness aware minimization [10], then one can

benefit from the second term of the min and avoid computing the full spectrum of I ( h | x ) in the first term.

Joint FIM Estimators In the above, we considered the variance of conditional FIMs, which can scale differently depending on the input x . Prior work's analysis was limited to that of conditional FIMs (and their estimators) [37]. Nevertheless, the 'joint FIM' estimators ˆ I j ( θ i ) depend on sampling of x w.r.t. the data distribution q ( x ) . The bounds in Eq. (7) can be extended to the joint FIM I ( θ i ) . = E q ( x ) I ( θ i | x ) by simply taking an expectation E q ( x ) over the bounds. To analyze the variances of the joint FIM estimators V j ( θ i ) , we present the following theorem which connects the prior results established for V j ( θ i | x ) , e.g . Theorem 4.1, via the law of total variance.

Theorem 4.7. For any j ∈ { 1 , 2 } , given N x samples of x ∼ q ( x ) and N samples of y | x ∼ p ( y | x ) for each x sampled,

where Var( I ( θ i | x )) is the variance of I ( θ i | x ) w.r.t. q ( x ) .

The dependence on N , the number of samples of y for each fixed x , is hidden in V j ( θ i | x ) . When N = 1 , the hierarchical sampling described in the Theorem corresponds to an i.i.d. sampling of the joint distribution p ( x , y ) .

The variance incurred when estimating the FIM has two components. The first term on the RHS of Eq. (16) characterizes the randomness of the FIM w.r.t. q ( x ) , i.e ., the input randomness. It vanishes when the FIM is estimated by taking the expectation w.r.t. q ( x ) , or the number of samples x is large enough. The second term (although also depending on N x ) comes from the sampling of y | x according to p ( y | x ) , i.e ., the output randomness, which scales with the central moments of t ( y ) . If the NN is trained so that p ( y | x ) tends to be deterministic, this term will disappear leaving the first term to dominate. Eq. (16) can be further generalized using the law of total covariance to extend prior work considering conditional FIM covariances [37] to joint FIM covariances. Theorem 4.7 connects the variance of assuming a fixed input x with multiple samples y k with the variance of pairs of samples ( x k , y k ) . The V j ( θ i | x ) bounds in this section can thus be applied to the corresponding joint variance V j ( θ i ) by using this theorem. This is straightforward and omitted.

The first variance term in Theorem 4.7 is difficult to compute in practice: it relies on how the closed-form FIM varies w.r.t. q ( x ) . As such, it is useful to bound the first term into computable quantities.

This upper bound is very similar to the 4th central moment term ˜ λ max ( K p ( t | x )) when considering the variance upper bound V 2 ( θ i | x ) in Theorem 4.1 and Corollary 4.2. In general, the eigenvalue terms of Lemma 4.8 and Theorem 4.1 are distinct, i.e ., λ 2 max ( I ( h | x )) = ˜ λ max ( K p ( t | x )) . This is especially true for the classification and regression problems explored in this paper (see Table 1). However, the maximum eigenvalues can be related for exponential families with bounded sufficient statistic via Proposition 4.3, making both bounds depend only on λ max ( I ( h | x )) .

̸

The total number of samples of ( x k , y k ) is N x · N . In terms of sample complexity of the joint variance, using Theorem 4.7 and Lemma 4.8, the bound's rate is given by O (1 /N x +1 / ( N x · N )) .

5 Case Studies

To make our theoretic results more concrete, we consider regression and classification settings, which correspond to specifying the exponential family in Eq. (1) to an isotropic Gaussian distribution and a categorical distribution, respectively. We include an empirical analysis of NNs trained on MNIST. Notably, our analysis considers general multi-dimensional NN output. This extends the case studies of [37] which was limited to 1D distributions due to the limitations of their bounds (and their associated computational costs of dealing with a 4D tensor of the full covariance).

Regression: Isotropic Gaussian Distribution To characterize regression, we consider Gaussian distributions. As per Eq. (1), we have t ( y ) = y ∈ ℜ T and base measure π ( y ) ∝ exp( -1 2 y ⊤ y ) .

This corresponds to the case where h ( x ) is trained via the squared loss. In this case, I ( h | x ) = I and K p abcd ( t | x ) = I ab ( h | x ) · I cd ( h | x ) + I ac ( h | x ) · I bd ( h | x ) + I ad ( h | x ) · I bc ( h | x ) . The derivatives of the log-partition function F ( h ) yields these central moments [37, Lemma 5]. To apply Theorem 4.1, we examine the extreme eigenvalues of I ( h | x ) and K p ( t | x ) -I ( h | x ) ⊗I ( h | x ) .

Proposition 5.1. Suppose that Eq. (1) is an isotropic Gaussian distribution. Then:

Hence, for regression the eigenvalues of sufficient statistics quantities in our bounds are equal. As such, in this case, the bound for I ( θ i | x ) , V 1 ( θ i | x ) , and V 2 ( θ i | x ) in Theorem 4.1 are all equalities.

As I ( θ i | x ) can be written exactly in terms of the gradients of h , in practice one does not need to utilize a random estimator when computing the conditional FIM, which simplifies to a Gauss-Newton matrix for the squared loss [25]. When computing the FIM over a random sample x , a variance still appears due to Theorem 4.7. By Lemma 4.8 and Proposition 5.1, the variance over a joint distribution is bounded by a function of the derivative ∂ i h ( x ) over the marginal input distributions: V ( θ i ) ≤ E q ( x ) [ ∥ ∂ i h ( x ) ∥ 4 2 ] ≤ max x ∈ Supp( q ) ∥ ∂ i h ( x ) ∥ 4 2 . In other words, the overall variance to approximate the joint FIM is bounded by the gradients of the NN outputs.

Classification: Categorical Distribution For multi-class classification, we instantiate our exponential family with a categorical distribution (with π ( y ) = 1 ). This corresponds to training a classifier NN with the log-loss. Let Y = [ C ] for T = C classes defining the possible labels. Let t ( y ) = ( J y = 1 K , . . . J y = C K ) , where J r K = 1 when the predicate r is true and J r K = 0 otherwise. Noting our results do not depend on minimal sufficiency, this t ( y ) is sufficient but not minimal sufficient. In this setting, the NN outputs h correspond to the logits of the label probabilities. The resulting probabilities p ( y | x ) are the softmax values of h denoted by σ ( x ) . = SoftMax( h ( x )) ∈ [0 , 1] T .

Theorem 5.2. Suppose that Eq. (1) is a categorical distribution. With σ max ( x ) . = max k σ k ( x ) :

Under this setting, we have I ( h | x ) = Diag( σ ( x )) -σ ( x ) σ ( x ) ⊤ (where Diag( σ ( x )) is the diagonal matrix with its diagonal entries set to σ ( x ) ), whose eigenvalues do not follow a convenient pattern as C increases [44]. Likewise, the maximum eigenvalue of K p ( t | x ) is not available in simple closed form. As such, we provide upper bounds for the maximum eigenvalues of I ( h | x ) and K p ( t | x ) -I ( h | x ) ⊗I ( h | x ) using Corollary 4.2 and Proposition 4.3.

This upper bounds provides a tension. When the first term of m ( x ) is maximized, the second is minimized, and vice-versa. In particular, the dominating term depends on the uncertainty of the NN's output. When the NN's output is near random, e.g . at initialization, the first term will dominate with σ max ( x ) ≈ 1 /C . However, as the NN becomes more certain with its prediction, the second term will start dominating: a more deterministic output p ( y | x ) → 1 implies that λ max ( I ( h | x )) → 0 .

Empirical Verification: Classification Weexamine the MNIST classification task [21] (CC BY-SA 3.0) using multilayer perceptrons (MLP) with four densely connected layers, sigmoid activations, and a dropout layer. For classification, we consider a categorical distribution with C = 10 class labels. For a random x from the test set, we compute both estimators ˆ I 1 ( θ i | x ) and ˆ I 2 ( θ i | x ) using N = 5 , 000 samples. We record the variances of each estimator and compute their bounds based on Theorem 4.1. For all 20 training epochs, the Fisher information (FI) and their variances of individual parameters are aggregated via arithmetic averages over four parameter groups (corresponding to the four layers).

In Fig. 2, we present the variance scale of the estimators ˆ I 1 ( θ i | x ) and ˆ I 2 ( θ i | x ) in log-space; and the tightness of the bounds in Theorem 4.1 by consider the log-ratio log UB V I ( θ i | x ) , where UB is the upper bounds in Theorem 4.1. In this experiment, the UB is much tighter than the lower bound (LB), which is omitted in the figures for clarity. More experimental results are given in Appendix F.

We varied the NN's architecture and activation function. Across different settings, the proposed UB and LB are always valid. In Fig. 2, one can observe that the diagonal FIM and the associated

̂

̂

̂

-10

-5

log

√



i

Param Group 1 (Linear 784 → 256)

Param Group 2 (Linear 256 → 256)

Param Group 3 (Linear 256 → 128)

Param Group 4 (Linear 128 → 10)

012345

10

15

20

0.0

2.5

5.0

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

variances have a small magnitude. For example, in the first layer, V 1 ( θ i | x ) and V 2 ( θ i | x ) are roughly e -10 ≈ 5 × 10 -5 . The log-ratio log UB V 1 ( θ i | x ) ≈ 4 means that the UB is roughly 50 times larger than V 1 ( θ i | x ) . Comparatively, V 2 ( θ i | x ) has a tighter UB which is approximately 10 times larger than itself. The UB serves as a useful hint on the order of magnitude of the variances. In Appendix D, we present tighter bounds which are more expensive to compute.

In the first three layers of the MLP, V 1 ( θ i | x ) presents a smaller value than V 2 ( θ i | x ) , meaning that ˆ I 1 can more accurately estimate the diagonal FIM. Interestingly, this is not true for the last layer: V 2 ( θ i | x ) becomes zero while ˆ I 1 presents the largest variance across all parameter groups. Due to this, one should always prefer ˆ I 2 over ˆ I 1 for the last layer. In the last two layers, ˆ I 2 is in simple closed form and, hence, does not need automatic differentiation to calculate (see Remarks 4.4 and 4.5). The shape of the variance curves are sensitive to the choices of activation functions ϕ and inputs x . In general, the variance in the first few epochs presents more dynamics than the rest of the training process. If one uses log-sigmoid activations ϕ ( t ) = -log(1 + exp( -t )) (which is equivalent to ϕ ( t ) = -SoftPlus ( -t ) , as per Remark 4.4), the variances of ˆ I 1 and ˆ I 2 only appear in the randomly initialized NN and quickly vanish once training starts, as shown in Appendix F. In this case, the learner more easily approaches a nearly linear region of the loss landscape where local optima lie. In practice, one should estimate and examine the scale of variances - which should not be neglected as per Fig. 2 - before choosing a preferred diagonal FIM estimator.

6 Relationship with the 'Empirical Fisher'

In some scenarios, even the estimators of the diagonal FIM ˆ I 1 ( θ ) and ˆ I 2 ( θ ) can be prohibitively expensive. Part of the cost comes from requiring label samples y k for each x k , as per Eq. (3). For example, when the FIM is used in an iterative optimization procedure, y k 's need to be re-sampled at each learning step w.r.t. the current h alongside their backpropagation (accounting for sampling).

As such, alternative 'FIM-like' objects have been explored which replace the samples from p ( y | x ) with samples from an underlying true (but unknown) data distribution q ( y | x ) [20, 27]. We define the data's joint distribution as q ( x , y ) . = q ( x ) q ( y | x ) . Analogous to the FIM, the data Fisher information matrix (DFIM) can be defined as the PSD tensor I( θ ) . = E q ( x ) [I( θ | x )] , with

where I( h | x ) = E q ( ˆ y | x ) [ ( t ( ˆ y ) -η ( x ))( t ( ˆ y ) -η ( x )) ⊤ ] denotes the 2nd (non-central) moment of ( t ( ˆ y ) -η ( x )) w.r.t. q ( ˆ y | x ) , and ∂ h /∂ θ is the Jacobian of the map θ → h . In the special case that q ( y | x ) = p ( y | x ; θ ) , then I( θ | x ) becomes exactly I ( θ | x ) .

The DFIM I( θ | x ) in Eq. (17) is a more general definition. Compared to the FIM I ( θ | x ) , it yields a different PSD tensor on the θ parameter space (the neuromanifold) depending on a dis-

tribution q ( x , y ) , which is neither necessarily on the same neuromanifold nor necessarily parametric at all. The asymmetry in the true data distribution and the empirical one results in different geometric structures [5]. By definition, we have I( θ | x ) ⪰ ( ∂ KL /∂ θ ) ( ∂ KL /∂ θ ) ⊤ , where KL( θ ) . = ∫ q (ˆ y | x ) log q (ˆ y | x ) p (ˆ y | x ; θ ) dˆ y is the Kullback-Leibler (KL) divergence, or the loss in a parameter learning scenario. The DFIM can be regarded as a surrogate function of the squared gradient of the KL divergence. It is a symmetric covariant tensor and satisfies the same rule w.r.t. reparameterization as the FIM. Consider the reparameterization θ → ζ , the DFIM becomes I( ζ | x ) = ( ∂ θ /∂ ζ ) ⊤ I( θ | x ) ( ∂ θ /∂ ζ ) .

̸

If q ( x , y ) = 1 N ∑ N k =1 δ ( x -x k ) · δ ( y -y k ) is defined by the observed samples, DFIM gives the widely used ' Empirical Fisher ' [25], whose diagonal entries are

Notice that ˆ η ( x ) . = E q ( ˆ y | x ) [ t ( ˆ y )] = η ( x ) in general. As such, there will be a miss-match when utilizing I( h | x ) as a substitute for I ( h | x ) . However, as learning progresses and p ( ˆ y | x ) becomes more similar to the data's true labeling posterior q ( ˆ y | x ) , the DFIM will become closer to the FIM.

where ( x 1 , ˆ y 1 ) , . . . , ( x N , ˆ y N ) are i.i.d. sampled from q ( x , ˆ y ) . Similar to ˆ I 1 ( θ i | x ) , an estimator with a fixed input x can be considered, denoted as ˆ I( θ i | x ) .

Given the computational benefits of using the data directly - bypassing a separate sampling routine -manypopular optimization methods employ the empirical Fisher or its approximation. For instance, the Adam optimizer [16] uses the empirical Fisher to approximate the diagonal FIM. However, switching from sampling y k to ˆ y k is anything but superficial [25, Chapter 11] ˆ I( θ ) is not an unbiased estimator of I ( θ ) as I( h | x ) is different from I ( h | x ) .

The biased nature of the empirical Fisher affects the other moments as well. In particular, we do not have the same equivalence of covariance and the metric being pulled back by θ → h [38].

Lemma 6.1. Given the conditional data distribution q ( ˆ y | x ) , the covariance of t given x is given by

̸

As a result, although the variance of the estimator ˆ I( θ i | x ) takes a similar form to V 1 ( θ i | x ) ( i.e ., Eq. (8)), its sufficient statistic terms do not exclusively consist of central moments. Noting the miss-match in ˆ η ( x ) = η ( x ) , Lemma 6.1 reveals an additional term which shifts I( h | x ) away from the 2nd central moment of t ( ˆ y ) (w.r.t. q ( ˆ y | x ) ). Instead, these sufficient statistic terms correspond to non-central moments of t ( ˆ y ) -η ( x ) . Some corresponding empirical Fisher / DFIM bounds are characterized in Appendix G.

7 Conclusion

We have analyzed two different estimators ˆ I 1 ( θ ) and ˆ I 2 ( θ ) for the diagonal entries of the FIM. The variances of these estimators are determined by both the non-linearly of the neural network and the moments of the exponential family. We have identified distinct scenarios on which estimator is preferable. For example, ReLU networks can only apply ˆ I 1 ( θ ) due to a lack of smoothness. As another example, ˆ I 2 ( θ ) has zero variance in the last layer and thus is always preferable than ˆ I 1 ( θ ) . Similarly, in the second last layer, ˆ I 2 ( θ ) has a simple closed form and potentially preferable for neurons in their linear regions (see Remark 4.5). In general, one has to apply Theorem 4.1 based on their specific neural network and settings and choose the estimator with the smaller variance. Our results suggest that, from a variance perspective, uniformly utilizing one of the FIM estimators ˆ I j ( θ ) is often suboptimal in NNs. Our work has further extended from analyzing the conditional FIM estimators ˆ I j ( θ | x ) to the joint FIM estimators ˆ I j ( θ ) ; and we have examined the relationship between the investigated estimators and the empirical Fisher. Future directions include extending the analysis of the variance of FIM estimators to block diagonals ( e.g . [26, 35]) and adapting current NN optimizers ( e.g . [16]) to incorporate the variance of FIM estimators.

Acknowledgments and Disclosure of Funding

The authors thank Frank Nielsen, James C. Spall, and the anonymous reviewers for their insightful feedback and many constructive comments.

References

Shun-ichi Amari. Information Geometry and Its Applications , volume 194 of Applied Mathematical Sciences . Springer-Verlag, Berlin, 2016.

Shun-ichi Amari, Tomoko Ozeki, Ryo Karakida, Yuki Yoshida, and Masato Okada. Dynamics of learning in MLP: Natural gradient and singularity revisited. Neural Computation , 30(1): 1-33, 2018.

Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher information and natural gradient learning in random deep networks. In International Conference on Artificial Intelligence and Statistics , pages 694-702. PMLR, 2019.

Sue Becker, Yann Le Cun, et al. Improving the convergence of back-propagation learning with second order methods. In Proceedings of the 1988 connectionist models summer school , pages 29-37, 1988.

Frank Critchley, Paul Marriott, and Mark Salmon. Preferred point geometry and statistical manifolds. The Annals of Statistics , pages 1197-1224, 1993.

Felix Dangel, Frederik Kunstner, and Philipp Hennig. Backpack: Packing more into backprop. In International Conference on Learning Representations , 2020.

Bradley Efron. Curvature and inference for maximum likelihood estimates. The Annals of Statistics , 46(4):1664-1692, 2018.

Mohamed Elsayed and A Rupam Mahmood. Hesscale: Scalable computation of Hessian diagonals. arXiv preprint arXiv:2210.11639 , 2022.

Reuben Feinman. Pytorch-minimize: a library for numerical optimization with autograd, 2021. URL https://github.com/rfeinman/pytorch-minimize .

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.

Shenghan Guo and James C. Spall. Relative accuracy of two methods for approximating observed Fisher information. In Data-Driven Modeling, Filtering and Control: Methods and applications , pages 189-211. IET Press, London, 2019.

Tom Heskes. On 'natural' learning and pruning in multilayered perceptrons. Neural Computation , 12(4):881-901, 2000.

Ryo Karakida and Kazuki Osawa. Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks. Advances in neural information processing systems , 33:10891-10901, 2020.

Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of Fisher information in deep neural networks: Mean field approach. In International Conference on Artificial Intelligence and Statistics , pages 1032-1041. PMLR, 2019.

Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Pathological spectra of the Fisher information metric and its variants in deep neural networks. Neural Computation , 33(8): 2274-2307, 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Tamara G Kolda and Jackson R Mayo. An adaptive shifted power method for computing generalized tensor eigenpairs. SIAM Journal on Matrix Analysis and Applications , 35(4): 1563-1581, 2014.

Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems , pages 4133-4144. Curran Associates, Inc., 2020.

Takio Kurita. Iterative weighted least squares algorithms for neural networks classifiers. New generation computing , 12:375-394, 1994.

Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gradient algorithm. Advances in neural information processing systems , 20, 2007.

Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online] , 2, 2010.

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems , 31, 2018.

Lek-Heng Lim. Singular values and eigenvalues of tensors: a variational approach. In 1st IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing , pages 129-132. IEEE, 2005.

Albert W. Marshall, Ingram Olkin, and Barry C. Arnold. Inequalities: Theory of Majorization and its Applications , volume 143 of Springer Series in Statistics (SSS) . Springer, second edition, 2011.

James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research , 21(146):1-76, 2020.

James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning , pages 2408-2417. PMLR, 2015.

James Martens et al. Deep learning via Hessian-free optimization. In ICML , volume 27, pages 735-742, 2010.

Peter Nickl, Lu Xu, Dharmesh Tailor, Thomas Möllenhoff, and Mohammad Emtiyaz E Khan. The memory-perturbation equation: Understanding model's sensitivity to data. In Advances in Neural Information Processing Systems , volume 36, pages 26923-26949. Curran Associates, Inc., 2023.

Frank Nielsen and Gaëtan Hadjeres. Monte Carlo Information-Geometric Structures , pages 69-103. Springer International Publishing, 2019.

Yann Ollivier. Riemannian metrics for neural networks I: feedforward networks. Information and Inference: A Journal of the IMA , 4(2):108-153, 2015.

Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research , 21(252):1-64, 2020.

Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In International Conference on Learning Representations , 2014.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pages 8024-8035. Curran Associates, Inc., 2019.

Jeffrey Pennington and Pratik Worah. The spectrum of the Fisher information matrix of a single-hidden-layer neural network. In Advances in Neural Information Processing Systems , pages 5415-5424, 2018.

Yi Ren and Donald Goldfarb. Tensor normal training for deep learning models. In Advances in Neural Information Processing Systems , volume 34, pages 26040-26052. Curran Associates, Inc., 2021.

Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems , volume 33, pages 18098-18109. Curran Associates, Inc., 2020.

Alexander Soen and Ke Sun. On the variance of the Fisher information for deep learning. In Advances in Neural Information Processing Systems , volume 34, pages 5708-5719. Curran Associates, Inc., 2021.

Ke Sun. Information geometry for data geometry through pullbacks. In Deep Learning through Information Geometry (Workshop at NeurIPS 2020) , 2020.

Ke Sun and Frank Nielsen. Relative Fisher information and natural gradient for learning large modular models. In International Conference on Machine Learning , pages 3289-3298, 2017.

Ke Sun and Frank Nielsen. A geometric modeling of Occam's razor in deep learning. arXiv preprint arXiv:1905.11027 , 2019.

Shiqing Sun and James C. Spall. Connection of diagonal Hessian estimates to natural gradients in stochastic optimization. In Proceedings of the 55th Annual Conference on Information Sciences and Systems (CISS) , 2021.

Terence Tao. Topics in random matrix theory , volume 132. American Mathematical Soc., 2012.

Sheng-De Wang, Te-Son Kuo, and Chen-Fa Hsu. Trace bounds on the solution of the algebraic matrix riccati and lyapunov equation. IEEE Transactions on Automatic Control , 31(7):654-656, 1986.

Christopher S Withers and Saralees Nadarajah. The spectral decomposition and inverse of multinomial and negative multinomial covariances. Brazilian Journal of Probability and Statistics , pages 376-380, 2014.

Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the Hessian. In 2020 IEEE international conference on big data (Big data) , pages 581-590. IEEE, 2020.

Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence , volume 35, pages 10665-10673, 2021.

Supplementary Material

Abstract

This is the Supplementary Material to Paper "Trade-Offs of Diagonal Fisher Information Matrix Estimators". To differentiate with the numberings in the main file, the numbering of Theorems is letter-based (A, B, ...).

Table of Contents

→

A Natural Gradient Toy Data Example

The following section describes the data and models of Fig. 1. In general, the toy data and models constructed consists of taking 1D output setting presented by Section 5, where the NN h θ ( x ) is a linear function.

A.I Data

The 2D input data x ∈ ℜ 2 is sampled from a simple isotropic centered Gaussian N ( 0 , I ) . A linear response variable a ∈ ℜ is defined by the following:

The outputs of y for the cases of regression and classification are differentiated by how a is used in sampling:

where σ ( z ) = (1 + exp( -z )) is the logistic function.

A.II Model

The model h θ ( x ) = θ ⊤ x consists of a linear function; and the exponential family Eq. (1) is chosen to be a 1D isotropic Gaussian and binary multinomial distribution (Bernoulli) for regression and classification, respectively. This corresponds to Section 5 for 1D outputs. Notice that the model exactly matches the data generating function.

A.III Training

Natural gradient descent (NGD) is taken using both ˆ I 1 ( θ ) and ˆ I 2 ( θ ) . The estimated FIM utilize only a single y | x sample for each input x . We use a learning rate of η = 0 . 01 over 256 epochs. A training set of 256 data points are sampled. At each iteration of NGD, we sample 4 random points from the training set for the update. The test loss is evaluated on a test set of 4096 data points sampled.

A.IV Variance Plot of Example

Larger version of Fig. 1 with additional variance sum plotted over time is given by Fig. I. Note that variance sum is including off diagonals. Further note that the variance is calculated over joint sample in ( x , y ) .

A.V Other Seeds

We further present other random seed of the teaser plot in Figs. II to IV.

B The Conditional Variances in Closed Form

We consider the diagonal entries of the conditional FIM I ( θ i | x ) and the conditional variances V j ( θ i | x ) of its estimators in closed form.

Proof of Lemma 3.1. The proof directly follows from [37, Equation 6], [37, Theorem 4], and [37, Theorem 6]. In what follows, we provide a proof of the Lemma utilizing the notation of this paper for completeness. We prove the statement one equation at a time.

0 8

32

64

128

256

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

Test Loss (NLL)

Regression (Gaussian)

Trajectories

0 8

32

64

128

256

0

.

66

0

.

68

0

.

70

0

.

72

0

.

74

Classification (Categorical)

Trajectories

NG with

ˆ

I

1

NG with

ˆ

I

2

Initial Params

ˆ

θ

0

True Params

θ

0 8

32

64

128

256

0

500

1000

1500

2000

ˆ

I

j

(

θ

)

Regression (Gaussian)

0 8

32

64

128

256

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Classification (Categorical)

Regression (Gaussian)

Trajectories

128

Regression (Gaussian)

128

Classification (Categorical)

Trajectories

64

128

Classification (Categorical)

128

Regression (Gaussian)

Trajectories

128

Regression (Gaussian)

128

Classification (Categorical)

128

Classification (Categorical)

128

0

0

0

0

0

0

0

0

0

0

0

0

.

.

.

.

.

.

.

.

.

.

.

.

76

74

72

70

68

30

25

20

15

10

05

00

0

0

0

0

0

1

1

0

0

0

0

.

.

.

.

.

.

.

.

.

.

.

76

74

72

70

68

25

00

75

50

25

00

Test Loss (NLL)

)

θ

(

j

I

2500

2000

1500

1000

500

0

1500

1250

1000

750

500

250

0

250

Test Loss (NLL)

)

θ

(

j

I

ˆ

200

150

100

50

0

150

125

100

75

50

25

0

ˆ

0 8

0 8

0 8

0 8

32

32

32

32

64

64

64

64

256

256

256

256

0 8

0 8

0 8

0 8

32

32

32

32

64

Trajectories

64

64

NG with

1

ˆ

NG with

I

2

ˆ

Initial Params

I

θ

True Params

ˆ

0

θ

256

256

NG with

1

ˆ

NG with

I

2

ˆ

Initial Params

I

True Params

θ

ˆ

0

θ

256

256

For Eq. (4), we consider the following computation.

Using Einstein notation and restricting the partial derivative to a component of θ yields the desired result.

For Eq. (5), we shorthand δ ( y ) = t ( y ) -η ( x ) . Note that the ˆ I 1 ( θ i | x ) estimator can be written as follows:

0 8

32

64

128

256

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

Test Loss (NLL)

Regression (Gaussian)

Trajectories

0 8

32

64

128

256

0

.

66

0

.

68

0

.

70

0

.

72

0

.

74

Classification (Categorical)

Trajectories

NG with

ˆ

I

1

NG with

ˆ

I

2

Initial Params

ˆ

θ

0

True Params

θ

0 8

32

64

128

256

0

100

200

300

400

ˆ

I

j

(

θ

)

Regression (Gaussian)

0 8

32

64

128

256

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification (Categorical)

Thus, we have

Let us compute each of these terms.

And,

Simplifying all term yields the result as required.

Finally, for Eq. (6) we consider the following simplification of the estimator.

where the last line follows from [37, Lemma 2] (a result of p ( y | x ; θ ) following an exponential family, see [1]).

Notice that the first quantity is a constant w.r.t. the randomness of y k . As such, we can simplify the variance calculation as follows.

where the second last line follows from the fact that η ( x ) = E p ( y | x ; θ ) [ t ( y )] and thus E p ( y | x ; θ ) [ δ ( y )] = 0 . This yields the desired result.

Lemma 3.1 shows that, for the former, V 1 ( θ i | x ) only depends on 1st order derivatives; while V 2 ( θ i | x ) only depends on the 2nd order derivatives. For the latter, V 1 ( θ i | x ) depends on both the 2nd and 4th central moments of t ( y ) ; while V 2 ( θ i | x ) only depends on the 2nd central moments.

Given I ab ( h | x ) and ∂ i h a ( x ) , the computational complexity of all diagonal entries I ( θ i | x ) is O ( T 2 dim( θ )) . If K p abcd ( t | x ) and ∂ 2 i h a ( x ) are given, then the computational complexity of the variances in Eqs. (5) and (6) is respectively O ( T 4 dim( θ )) and O ( T 2 dim( θ )) . Each requires to evaluate a T × dim( θ ) matrix, either ∂ i h a ( x ) or ∂ 2 i h a ( x ) -which can be expensive to calculate for the latter. This is why we need efficient estimators and / or bounds for the tensors on the LHS of Eqs. (4) to (6).

C Off-Diagonal Variance

We consider an off-diagonal version of the bound given by Theorem 4.1. Notice that in terms of the dependence on neural network weights, the only change is splitting the 'responsibility' of the i 'th and j 'th parameter norms.

Theorem C.1. ∀ x ∈ ℜ I ,

where

Proof. The proof follows similarly to Appendices J and K, where the primary difference is just swapping the regular eigenvalue-like quantities with the γ variational forms.

It should be noted that the corresponding lower bounds become trivial as the additional degree of freedom of having an inf over both u and v causes the corresponding γ min definition to have negative quantities. Although it is unclear what the 'tensor-like' variational quantity ˜ γ max ( M ) will be, for a matrix, we have the following equivalence.

Lemma C.2. γ max ( A ) = s max ( A ) , where s max ( A ) is the maximum singular value of A .

Proof. The proof follows from optimizing over u and v separately:

This is equivalent to the square root of the maximal eigenvalue of A ⊤ A , which is exactly the maximum singular value.

Hence for the ˆ I 2 we have the following.

Corollary C.3. ∀ x ∈ ℜ I ,

D Bounding the Trace Variance by Full Spectrum

Theorem D.1. For any x ∈ ℜ I ,

where s 2 i ( A ) = λ i ( A ⊤ A ) denotes the i -th singular values, M is the 'reshaped' matrix of M defined in Theorem 4.1 i.e . there exists j, k such that M jk = M abcd for all a, b, c, d ,

and

Proof. The proof follows from a generalized Ruhe's trace inequality [24]:

Theorem D.2. For A,B ∈ ℜ n × n Hermitian matrices, we have that

We prove the result for each equations.

For readability, we let J ia = ∂ i h a ( x ) .

For Eq. (22):

One can notice that the trace of the FIM can exactly be expressed as the trace of two matrices.

Thus, noting that the eigenvalue of the 'squared' matrix is the matrix's singular value λ t ( J ⊤ J ) = s 2 t ( J ) , with Theorem D.2, we have that:

For Eq. (23):

Noting that M abcd = K p abcd ( t | x ) -I ab ( h | x ) · I cd ( h | x ) . Furthermore, we have that

Let us define the following 3D tensor with J iab = ∂ i h a ( x ) ∂ i h b ( x ) = ( ∂ i h ( x ) ∂ ⊤ i h ( x )) ab .

Thus, again simplifying the eigenvalue of the 'squared' matrix, with Theorem D.2, we have that:

For Eq. (24):

Similar to Eq. (22), we only need to rearrange the summation. Notice that

thus dHes ia ( h | x ) = ∂ 2 i ( h a | x ) .

Thus, again simplifying the eigenvalue of the 'squared' matrix, with Theorem D.2, we have that:

E Second Central Moment of Categorical Distribution

Proof. We first notice that the exponential family density is given by, and thus also have

The first order derivative follows as,

As such, the second order derivatives also follow,

As such, we have that

F Empirical Results Continued

In the following section we present additional details and results for the experimental verification we conduct in Section 5.

F.I Additional Details

We note that to calculate the diagonal Hessians required for the bounds and empirical FIM calculations, we utilize the BackPACK [6] for PyTorch . Additionally, to calculate the sufficient statistics moment's spectrum, we explicitly solve the minimum and maximum eigenvalues via their optimization problems. For 2D tensors / matrices, we utilize numpy.linalg.eig . For 4D tensors, we utilize PyTorch Minimize [9], a wrapper for SciPy 's optimize function.

F.II Additional Plots

We present Figs. V to VIII which are the exact same experiment run in Section 5, but with different initial NN weights and random inputs.

Figures IX to XIII show the experimental results on a 5-layer MLP and log-sigmoid activation function. In most of the cases, the FIM and its associated variances quickly go to zero in the first few epochs.

̂

-40

-20

log

√



i

Param Group 1 (Linear 784 → 256)

Param Group 2 (Linear 256 → 256)

Param Group 3 (Linear 256 → 128)

Param Group 4 (Linear 128 → 10)

012345

10

15

20

0

5

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

̂

-40

-20

0

log

√



i

Param Group 1 (Linear 784 → 256)

Param Group 2 (Linear 256 → 256)

Param Group 3 (Linear 256 → 128)

Param Group 4 (Linear 128 → 10)

012345

10

15

20

0

5

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

G 'Empirical Fisher' Continued

Noting Lemma 6.1's characterization of the covariance, we are able to characterize the variance of the diagonal elements of ˆ I( θ | x ) , denoted as V( θ i | x ) . = Var( ˆ I( θ i | x )) .

̂

̂

̂

̂

̂

̂

-40

-20

log

√



i

Param Group 1 (Linear 784 → 256)

Param Group 2 (Linear 256 → 256)

Param Group 3 (Linear 256 → 128)

Param Group 4 (Linear 128 → 10)

012345

10

15

20

0

5

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

-15

-10

-5

log

√



i

Param Group 1 (Linear 784 → 256)

Param Group 2 (Linear 256 → 256)

Param Group 3 (Linear 256 → 128)

Param Group 4 (Linear 128 → 10)

012345

10

15

20

0

5

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

where K( h | x ) the 4th (non-central) moment of ( t ( ˆ y ) -η ( x )) w.r.t. q ( ˆ y | x ) .

As a result of the similarity of the functional forms of the empirical Fisher ˆ I( θ ) and the FIM estimator ˆ I 1 ( θ ) , it is not surprising that Corollary G.1 is similar to the variance of ˆ I 1 ( θ i | x ) . Indeed, applying Lemma 6.1 will give the exact same functional form with the 2nd central moments of t ( y ) w.r.t. p ( y | x ) exchanged with 2nd non-central moments of ( t ( ˆ y ) -η ( x )) w.r.t. q ( ˆ y | x ) . V( θ i | x ) is therefore determined by the 2nd and the 4th moment of ( t ( ˆ y ) -η ( x )) up to the parameter transformation θ → h . Subsequently, the bounds presented for V 1 ( θ i | x ) (Eq. (8) and Corollary 4.6) can be similarly adapted for V( θ i | x ) .

The extension of V( θ i | x ) to V( θ i ) can also be proven in a similar manner to Theorem 4.7.

Corollary G.2. Given N x samples of x ∼ q ( x ) and N samples of y | x ∼ q ( y | x ) for each x sampled,

where Var(I( θ i | x )) is the variance of I( θ i | x ) w.r.t. q ( x ) .

If q ( x , y ) = 1 N ∑ N k =1 δ ( x -x k ) · δ ( y -ˆ y k ) for a set of observations { ( x k , ˆ y k ) } N k =1 , then one can directly evaluate the DFIM without sampling and achieve zero variance, i.e ., ˆ I( θ ) = I( θ ) . In this scenario, there is a clear trade-off between the estimators of the FIM in Eq. (3) and the DFIM. The estimators of the FIM are unbiased, but have a variance; while the DFIM has zero variance, but is a biased approximation of the FIM.

̂

̂

̂

̂

̂

-50

-25

0

log

√



i

PGroup 1 (Linear 784 → 256) PGroup 2 (Linear 256 → 128) PGroup 3 (Linear 128 → 128)

PGroup 4 (Linear 128 → 64)

PGroup 5 (Linear 64 → 10)

012345

10

15

20

0

1

2

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

̂

-50

-25

0

log

√



i

PGroup 1 (Linear 784 → 256) PGroup 2 (Linear 256 → 128) PGroup 3 (Linear 128 → 128)

PGroup 4 (Linear 128 → 64)

PGroup 5 (Linear 64 → 10)

012345

10

15

20

0

1

2

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

̂

-100

-50

0

log

√



i

PGroup 1 (Linear 784 → 256) PGroup 2 (Linear 256 → 128) PGroup 3 (Linear 128 → 128) PGroup 4 (Linear 128 → 64)

PGroup 5 (Linear 64 → 10)

012345

10

15

20

0

1

2

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

H Derivation of Eq. (3) Using Log-Partition Function Derivatives

In what follows, we derive the alternative equations for ˆ I 1 ( θ i ) and ˆ I 2 ( θ i ) presented in Section 3. That is, we seek to derive the following equations:

We calculate the equations separately.

̂

̂

̂

̂

-50

-25

0

log

√



i

PGroup 1 (Linear 784 → 256) PGroup 2 (Linear 256 → 128) PGroup 3 (Linear 128 → 128)

PGroup 4 (Linear 128 → 64)

PGroup 5 (Linear 64 → 10)

012345

10

15

20

0.0

2.5

5.0

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

-50

-25

0

log

√



i

PGroup 1 (Linear 784 → 256) PGroup 2 (Linear 256 → 128) PGroup 3 (Linear 128 → 128)

PGroup 4 (Linear 128 → 64)

PGroup 5 (Linear 64 → 10)

012345

10

15

20

0

1

2

Avg log(UB/



i

)

012345

10

15

20

012345

10

15

20

012345

10

15

20

012345

10

15

20



1

√



1

UB of

√



1

√



2

UB of

√



2

̂

̂

̂

̂

̂

H.I Eq. (26)

Proof. For Eq. (26), we note that

where we note that F ′ a ( h ( x k )) = η a ( x k ) which follows from the connection to expected parameters and partition functions of exponential families, see e.g . [37].

H.II Eq. (27)

Proof. For Eq. (27), we also calculate the derivative:

Then

Remark H.1. Although Eq. (27) is useful in practice, i.e ., it states an equation which can be calculated via automatic differentiation, in the appendix and proofs we use an alternative equation. In particular, we use

which follows from taking the derivative of ∂ i log p ( y k | x k ) in the proof of Eq. (26) (above).

I Proof of Eq. (7)

We first begin by proving the follow lemma to bound an ℜ n × n matrix.

Lemma I.1. Let A ∈ ℜ n × n and v ∈ ℜ n , then

Proof. The proof follows immediately from the Courant-Fischer min-max theorem [42]. That is,

Thus it follows that:

The lower bound follows identically.

We note that this can be similarly proven via trace bounds, e.g ., [43].

Now we can prove Eq. (7).

Proof. The proof follows from Lemma 3.1, Eq. (4), and directly applying Lemma I.1.

J Proof of Eq. (8)

Let us first define the maximum and minimum Z-eigenvalues of a 4-dimensional tensor K .

Now We first prove the following lemma regarding the Z-eigenvalues.

Lemma J.1. Suppose K is 4-dimensional tensor. Then we have

Proof. The proof follows similarly to Lemma I.1. We simple use the following calculation:

The minimum case is proven identically (with the opposite inequality).

Now we can prove the bounds of Eq. (8)

Proof. From Lemma 3.1, we have that

where we shorthand K abcd = K p abcd ( t | x )

We bound two terms.

which follows directly from Lemma J.1

We now bound the second term in a similar way, taking v a . = ∂ i h a ( x ) and noting that

which directly gives us,

which follows from Lemma I.1.

Thus, together these bounds prove Eq. (8).

K Proof of Eq. (9)

Thus we get

which follows from Lemma I.1. This immediately gives the bound as required.

L Proof of Corollary 4.2

Proof. The corollary holds from distributing the inf or sup and examining how the variational definition of the generalized 'eigenvalue' simplifies under tensor products.

Indeed, for the minimum case,

where the last line holds from the fact that I ab ( h | x ) is PSD (thus the inner Einstein summation is always positive).

Taking definitions of the types of eigenvalues, gives the statement.

We note that the 'max' case follows identically.

Additionally, for the lower bound, we can show the non-triviallity of the non-negativity of the minimum eigenvalue.

Thus we have that

Equality holds from simply looking at the definition of K p ( t | x ) and I ( h | x ) (as moments).

M Proof of Proposition 4.3

Proof. Letting v = t ( y ) -η ( y ) , we note that the maximum eigenvalue is given by,

N Proof of Corollary 4.6

Proof. We split up the proof into the two arguments of the various min -function.

For the right term:

Suppose that we have a bound such that V j ( θ i | x ) ≤ αβ i . Then,

Thus we have,

Where we note that

For Eq. (15), that is taking

Where we note that

For the left term:

We take the largest singular value of the network derivative term. We then further notice that s max ( A ) ≤ ∥ A ∥ F from norm ordering (of the matrix 2-norm).

Taking the appropriate α and β from Eqs. (8) and (9) proves the case for Eqs. (13) and (15). For Eq. (14), that is taking

To further elaborate on the Eq. (14) case, we further need to simplify the following:

where the last inequality follows from the norm ordering ∥ · ∥ 2 ≤ ∥ · ∥ 1 .

O Proof of Theorem 4.7

To prove the Theorem, we will utilize the law of total variances. We note, that by the premise of the Theorem, we are sampling N x many samples from q ( x ) and N many samples from q ( y | y ) for each y initially sampled. To make this clear, the samples and sampling will be notated by:

Note that using these samples, our empirical estimators for the FIM (for either estimator) will be of the form:

for an appropriately chosen f .

This also gives:

Now, we simplify the variance as follows:

As required.

For V 1 ( θ )

Proof.

Let δ a ( x , y ) . = ( t ( y ) -η ( x )) .

And:

Together:

For V 2 ( θ )

Proof.

which follows by taking the 'partial' expectation ( y | x ) for both terms. Thus together,

P Proof of Lemma 4.8

Proof. The lower bound holds from just considering the non-negativity of variance. For the upper bound, we utilize the bound directly consider the bounds of Eq. (7),

Q Proof of Proposition 5.1

We first derive the statistics I ( h | x ) and K p ( t | x ) presented in 'Regression: Isotropic Gaussian Distribution' Section 5. It follows that from the regression setting, we have that,

where notably, by definition, π ( y ) is independent of learned parameter h ( x ) .

As such, we have that:

Now we note that E p ( y | x )[ y i ] is exactly h i ( x ) as the parameter h ( x ) specifies the mean of the (isotropic) multivariate normal distribution. As such we have that,

Furthermore, by [37, Lemma 5], we have that,

In summary, we have,

Proof. The minimum and maximum eigenvalues of I ( h | x ) follows directly noting that the trace of a matrix is the sum of eigenvalues. As such, from the statistics presented above we have that the minimum and eigenvalue must be 1 .

The tensor eigenvalues of K p ( t | x ) -I ( h | x ) ⊗I ( h | x ) = I ac ( h | x ) · I bd ( h | x ) + I ad ( h | x ) · I bc ( h | x ) follows from the variational definition Eq. (10). For instance, for the minimum eigenvalue,

The maximum eigenvalue is proven identically.

R Proof of Theorem 5.2

We first prove the following corollary which connects the maximum eigenvalues of K ( t | x ) to the maximum eigenvalues of I ( h | x ) .

Corollary R.1. Suppose that the exponential family in Eq. (1) is specified by a categorical distribution. Then,

Proof. As we are consider a categorical distribution we have that

̸

Thus we have that | v i | ≤ 1 . Furthermore, note that the maximum ℓ 2 -norm that we can have is ∥ v ∥ 2 ≤ √ 2 . Note that this is tight when the positive and negative mass are placed only two distinct coordinates, i.e ., ( -1 , 1 , . . . ) .

Thus using Proposition 4.3, the result follows.

Now by using Corollaries 4.2 and R.1, the remainder of the proof, all we require is the bounding of λ max ( I ( h | x )) .

Proof. The first term in the maximum eigenvalue follows from,

The second term in the maximum follows from the trace of I ( h | x ) being the sum of total eigenvalues.

S Proof of Lemma 6.1

Proof. The proof follows from the standard definition of covariance. Denoting ˆ η ( x ) . = E q ( y | x ) [ t ( y )] , we have:

Also expanding I( h | x ) :

Thus we have

As required.

T Proof of Corollary G.1

Proof. We calculate the variance:

Each of the terms can be calculated: Let δ a ( x , y ) . = ( t ( y ) -η ( x )) .

And:

Together with Lemma 6.1 proves the theorem.

U Proof of Corollary G.2

Proof. The proof follows identically to that of Theorem 4.7 with densities changed.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims are justified in the formal results of the paper and justified in the numerical experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of our results are expressed through out the paper. See for instance Section 3 which discusses computation implications.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated when expressing theoretic results and proofs are provided in appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details of experiments are presented in the main-text and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Only simple plots are presented in the paper. The first figure is a toy example of natural gradient descent with the FIM estimators. The other plots consider variance estimation and bounds of the paper in small scale networks.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Multiple runs are presented. Error bars are not as they obscure plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Presented in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform to the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is mainly theoretic. The societal impact of our work would be equivalent to any societal impact of machine learning as a field.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No data or model release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: License and credit are given for MNIST.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors study two popular estimators of the Fisher information matrix with respect to neural network parameters. They derive upper and lower bounds for the variance of these estimators and showcase them in applications to regression and classification problems.

STRENGTHS:
- Analyzing the convergence properties of Fisher information estimators is a timely and nontrivial endeavor, in part due to their use in natural gradient descent.
- The paper is very extensive and mathematically sound.

WEAKNESSES:
- The paper is highly technical and the numerical examples are fairly toy.
- The paper takes strong inspiration from a previous work [Soen, Alexander, and Ke Sun. "On the variance of the Fisher information for deep learning." Advances in Neural Information Processing Systems 34 (2021): 5708-5719] on the topic. As such, I question the value the present submission adds to this line of research. In particular, I think the submission does a poor job in highlighting the central novel aspects of their work and reviewing the field.

QUESTIONS:
- The authors should clarify clearly to what extent their submission improves upon previous works, particularly [Soen, Alexander, and Ke Sun. "On the variance of the Fisher information for deep learning." Advances in Neural Information Processing Systems 34 (2021): 5708-5719]. How do their bounds compare to the ones previously derived?

- In my opinion, the impact of the submission would be substantially improved if the authors state more clearly in what scenarios which of the two estimators will be preferred and provide a rough guideline for practitioners.

- Could the authors provide a numerical example for regression similar to Fig. 2?

LIMITATIONS:
There is no further need for authors to address limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses two estimators of the Fisher Information matrix and specifically their variances, which (from reference 37) have closed-form but non-practical expressions. The authors show a sequence of inequalities and derive practical bounds for these variances, both element-wise and trace-wise, both conditioned to x and not. Several remarks and trade-offs on these bounds clarify when an estimator is preferable over the other.\nThe results hold in the general exponential family likelihood but are also further concretized for regression and classification.\n\nSTRENGTHS:\nThe paper is overall very well written and structured. \n\nNotation is clear, results are properly presented and every step is further clarified in the appendix. \\\nThe alternation between formal statements and explanations is great and makes the paper vey pleasant to read. I particularly like the amount of remarks and observations that discuss specific terms of the equations, explains them and connect them to each other. And the same is true for all the highlighted trade-offs (for example line 212-215)\n\nWEAKNESSES:\nI have found no major weaknesses in the paper. I report some minor things.\n\nI personally dislike the notation $\\hat{\\mathcal{I}}_1(\\theta_i)$ for the i-th element of the diagonal introduced in line 87. This notation suggest that this term only depends on $\\theta_i$, but it actually depends on the whole $\\theta$ vector. On the other hand, this is quite a light easy notations, and I can't think of alternatives which are not much heavier to read.\n\nAlso, I find the definition in line 120 \"is defined by $\\hat{\\mathcal{I}}_j(\\theta_i)$ with $x=x_1=...=x_N$\" to be yes understandable but not super clean. The alternatives are not as compact, but I'd recommend using the extra space to clarify this a bit better. At least you should write that $y_1,...,y_N$ are i.i.d. from $p(y|x;\\theta)$ and not from $p(y,x;\\theta)$ as in $\\hat{\\mathcal{I}}_j(\\theta_i)$.\n\nIt would be nice to explicitly write the proof for the variance closed expression in Lemma B.1, yet clarifying that it's not a paper contribution. This is for two reasons: (1) it would make the paper self contained, without the need for the reader to read reference 37 and (2) the notation is slightly different and it would be easier for the reader to be consistent.\n\nQUESTIONS:\nIn line 168. Shouldn't \"of Eqs. (4) and (6)\" instead be \"of Eqs. (4) and (5)\"? And consistently shouldn't \"$\\mathcal{V}_1(\\theta_i|x)$\" be \"$\\mathcal{V}_2(\\theta_i|x)$\"? \\\nI ask this because \"small shifts in parameter space yield large changes in the output\" to me refers to the magnitude of the network jacobian $||\\partial_i h(x)||$, and that appear in the bound for $V_1$, not for $V_2$. Am I misunderstanding something or is this a typo?\n\nIn line 246. Isn't the bound sample complexity $\\mathcal{O}(\\frac{1}{N_x} + \\frac{1}{N_y})$? I can't see why it should be the product, can you elaborate more on this derivation?\n\nLIMITATIONS:\nI think it would be nice to clarify better that the work is incremental on \"Soen et all - On the variance of the Fisher information for deep learning\" [reference 37] and their closed form expression reported in the appendix in Lemma B.1. (also it would be nice to move these closed form expressions to the main paper).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyzed two different estimators, $I_1(\\theta)$ and  $I_2(\\theta)$, for the diagonal elements \nof the Fisher information matrix of a parametric neural net model. These diagonal elements are approximations for the entire matrix,\nwhich is unfeasible to be calculated in real neural nets models. They identified the situations in which each of these estimators is preferable with respect to the other.\n\nSTRENGTHS:\nThe Fisher information matrix is a major parameter driving the quality of the fit and highlighting important aspects of the interdependence between the parameters. It has been used in several papers of continual learning, the research area that I follow. \nIt focuses on the diagonal elements of the Fisher information matrix, offering approximations that are feasible to calculate in real neural net models, addressing a significant practical challenge given the unfeasibility of computing the entire matrix. he authors identified specific situations where each of the two estimators, $I_1(\\theta)$ and  $I_2(\\theta)$, is preferable. This practical guidance can help researchers and practitioners make informed decisions about which estimator to use in different scenarios.\n\nWEAKNESSES:\nThis is a highly theoretical paper and I have difficulty following all the findings presented by the authors. One of the main problems with this paper is that it has a large intersection with Soen and Sun (2021) and this becomes clear while we read the paper. In the Related work, Soen and Sun (2021) is mentioned in passing and it is a major reference work for the current manuscript. It is worth to have a better explanation of the relative merits of these two papers. Both papers analyzed two different estimators, $I_1(\\theta)$ and  $I_2(\\theta)$, for the Fisher information matrix of a parametric neural net model. While the older paper focuses on the matrix as a whole, the present manuscript focuses on the diagonal elements. However, the work of Soen and Sun (2021) is repeatedly referred to during the technical development of the paper. The contribution of this paper seems marginal when compared to the previous paper but still relevant.   \n\nThe results are derived assuming $\\theta$ is equal to its true value. In practice, the maximum likelihood estimator (MLE) of \n$\\theta is used, which introduces additional variability. The paper does not adequately address the implications of this assumption or the consequences when using the MLE, which limits the practical applicability of the results.\n\nThe paper does not discuss the impact of model misspecification on the results. In real-world scenarios, the data might not follow the proposed model exactly, and the inverse-Fisher information matrix might not represent the correct asymptotic variance of MLE estimators. The authors should comment on how their findings would be affected in such cases.\n\nQUESTIONS:\nThe results are all given using $\\theta$ equal to its true value. In practice, we use the MLE estimator of $\\theta$. The results \nare not valid anymore as we have this additional source of variability. The authors may comment on the consequences of this?\n\nLIMITATIONS:\nThis is a theoretical paper with no immediate connection to societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary\n-------\nThe paper studies two estimators for estimating the diagonal of a Fisher information\nmatrix (FIM) of a parametric machine learning model. Both estimators are based on equivalent\nexpressions for the FIM. The first is the standard definition, expressed in terms of the\nderivatives of the log likelihood, and the latter, expressed in terms of the second\nderivative when the model is twice differentiable with respect to the parameters. The\nestimators replace the expectation with an average over the data. The authors provide\nbounds on the variance of both (unbiased) estimators in terms of the minimum/maximum\neigenvalues of the FIM, and then provide bounds on these eigenvalues under special cases.\nThey also provide a modest empirical evaluation.\n\n\nOverall, while I think there is a fair amount of technical meat in the paper, I am \nunconvinced about the problem. I will wait to hear from more expert reviewers in this \nfield before I make my decision. \n\n\nDetailed comments:\n------------------\n\nWhile I am theoretically inclined, I am not an expert in this sub-field. Therefore, I will\nkeep my comments at a high level.\n\nSignificance of problem: I struggled to understand when estimating the FIM will be useful.\n- One point raised by authors is that it is used in optimizing NNs. If this was true, it\n  would have been nice to see the estimators used in NN optimization empirically, or a\n  theoretical analysis showing how the bounds derived here percolate to bounds on\n  optimization.\n- Another explanation given is that it could be used to analyze the structure fo NNs.\n  But this is stated vaguely, and it wasn't clear to me that such an analysis would help\n  design better neural networks.\n\nSignificance of techniques: The proof in the appendix is substantial but the authors have \nnot stated which techniques used here are novel. Currently technical results are presented\none after the other. It might help the reader/reviewer\nappreciate the contributions of the paper if the authors can highlight the novel tools \nthey used to derive these bounds.\n\nPresentation and writing: The technical content was clear for the most part. However,\nthe writing was very equation-driven. I understand that this is in part due to the nature \nof the paper, but the authors should simplify the exposition further. One option is to\ncut down on some of the technical content.\n\nSTRENGTHS:\nSee above\n\nWEAKNESSES:\nSee above\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}