On the Variance of the Fisher Information for Deep Learning
Authors: Alexander Soen
Ke Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: Fisher information
natural gradient
Cramer-Rao Lower Bound
deep learning

Abstract: In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and build geometric learning theories. The exact FIM is either unavailable in closed form or too expensive to compute. In practice, it is almost always estimated based on empirical samples.  We investigate two such estimators based on two equivalent representations of the FIM --- both unbiased and consistent. Their estimation quality is naturally gauged by their variance given in closed form. We analyze how the parametric structure of a deep neural network can affect the variance. The meaning of this variance measure and its upper bounds are then discussed in the context of deep learning.

On the Variance of the Fisher Information for Deep Learning
Alexander Soen The Australian National University alexander. soene anu edu au
Ke Sun CSIRO's Data61, Australia The Australian National University Sunk@ieee.
Abstract
In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and build geometric learning theories. The exact FIM is either un- available in closed form or too expensive to compute. In practice, it is almost always estimated based empirical samples. We investigate two such estimators based on two equivalent representations of the FIM both unbiased and consis- tent. Their estimation quality is naturally gauged by their variance given in closed form. We analyze how the parametric structure of a deep neural network can af- feet the variance. The meaning of this variance measure and its upper bounds then discussed in the context of deep learning.
Introduction
The Fisher information one of the most fundamental concepts in statistical machine learning. Intuitively, it measures the amount of information carried by single random observation when the underlying model varies along certain directions the parameter space: if such a variation does not change the underlying model, then a corresponding observation contains zero (Fisher) information and non-informative regarding the varied parameter. Parameter estimation is impossible this case. Otherwise, if the variation significantly changes the model and has large information, then an observation informative and the parameter estimation can be more efficient as compared to parameters with small Fisher information. In machine learning, this basic concept is useful for defining intrinsic structures the parameter space, measuring model complexity, and performing gradient-based optimization.
Given a statistical model that is specified by parametric form p(z ()) and a continuous domain E M, the Fisher information matrix (FIM) is a 2D tensor varying with 0 E M, given by
Oe Oe 00
I(0) Ep(=
00
where (p(= (-), simply (.).) if the model p is clear from the context, denotes the expectation w.r.t. p(= 0), and & = log p(= 0) is the log-likelihood function. All vectors column vectors throughout this paper. Under weak conditions (see Lemma 5.3 in Lehmann and Casella [16] for the univariate case), the FIM has the equivalent expression I(0) Ep(z Ø) (-02:/00001) Given N i.idd. observations #1,.... &N, these two equivalent expressions of the FIM lead to two different estimators
N
de, de,
024
(0)
and 22
(2)
N
30 00T
N
0000
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

where l, == log p(z; ()) is the log-likeln log-likelihood of the i'th observation 2,. The notations (0) and (0) are abused for simplicity as they depend on both 0 and the random observations E₁.
These estimators are universal and independent to the parametric form p(z 0). They are expressed in terms of the 1st- or 2nd-order derivatives of the logllikeihood. Usually, we already have these derivatives to perform gradient-basee based learning. Therefore, we can save computational cost and reuse these derivatives to estimate the Fisher information, which in turn can be useful, e.g., to perform natural gradient optimization [1, 25J. Estimating the FIM especially meaningful for deep learning, where the computational overhead of the exact FIM can be significant.
is straightforward from the law of large numbers and the central limit theorem that both estimators in Eq. (2) are unbiased and consistent. This is formally stated as follows.
Proposition 1.
Ep(=10) (I1(0)) Ep(=10) (Iz(0) = I(0). (1(:(0) I(0)] + [2((0) - I(0) > >0)=0
Vc O, lim Prob N
=0;
where Probl.) denotes the probability of the parameter statement being true and nius norm of a tensor (with || 1/2 as the regular vector Toz-norm)
|| 1|1 is the Frobe-
The Fisher information can be zero for non-regular models infinite [7]. However, these properties may not be preserved by the empirical estimators.
How far can ( and (0) deviate from the "true FIM" I(0), and how fast can they converge to I(0) as the number of observations increases? To answer these questions, it is natural to think of the variance of Ż, (0) and (0). For example, estimator with large variance means the estimation does not accurately reflect I(0); and any procedure depending on the FIM consequently suffers from the estimation error. Through studying the variance, can control the estimation quality and reliably perform subsequent measurements or algorithms based on the FIM.
Towards this direction, we made the following contributions that will unfold in the following Sec- tions to
We review and rediscover two equivalent expression of the FIM in the context of deep feed-forward networks (Section 2);
We give in closed form the variance (extending to meaningful upper bounds) and discuss the convergence rate of the estimators Ż, (0) and (0) (Section 3);
We analyze how the 1st- and 2nd-order derivatives of the neural network can affect the estimation of the FIM (Section 4).
We discuss related work in Section 5 and conclude in Section
2 Feed-forward Networks with Exponential Family Output
This section realizes the concept of Fisher information in feed-forward network with exponentially family output and explains why its estimators useful in theory and practice.
Consider supervised learning with a neural network. The underlying statistical model is p(2 0) = p(x))p( az, 0). where = (xr, y). the random variable æ represents features, and y is the target variable. The marginal distribution p(æ) parameter- usually fixed as the empirical distribution p(x)= = Nे EM -(x= - x;) w.r.t. set of observations { where d(-) is the Dirac delta. In this paper, we consider w.l.o.g. M = 1 as the FIM w.r.t. observations JMM is simply the average over FIMs of each individual observation. All results generalize to multiple observations by taking the empirical average. The predictor p(y x. 0) is neural network with parameters 0 = -Wi11)

---

and exponential family output units, given by
pyy x) = exp (t T (y)hr
- F(hL));
h = Wi-ihe-1-
hi o oW oWi-ihh-1) (1= (1,1,..,,611
h: (hi ,
ho æ;
where (zy is the sufficient statisties of the prediction model, F(h) log exp(t (y)h)d Jy yy the log-partition function, and a Rt + R is element-wise non-linear activation function. Moreover, W, is a 14++ x (mi + 1) matrix, representing the neural network parameters (weights and biases) in the I'th layer, where ny == dim(h:) denotes the size of layer We use W,- for the nt41 ny weight matrix without the bias terms, obtained by removing the last column of W. h1 is learned representation of the input æ. All intermediate variables h; are extended to include a constant scalar 1 in h;, so that a linear layer can simply be expressed as Wi The last layer's output h with dimensionality n, specifies the natural parameter the exponential family.
We need the following Lemma which gives the FIM w.r.t. hr. which is a n. x nt, matrix in simple closed form for commonly used probability distributions.
Lemma 2. For the neural network model specified in Eq. (3).
Om
I(hr.) Cov(t(y))
Oh1
where Cov(.) denotes the covariance matrix w.E.. ply xx,0), 0). n := m(h)) 0F/0hz is the expectation parameters, and the vector-vector-deriveee ön/ dh, denotes the Jacobian matrix of the mapping hr, + 17.
The derivatives of the log-likeldhood ((O) == p(æ, y 0) characterize its landscape and are es- sential to compute the FIM. By Eq. (3), the score function (gradient of E) is
0h1
=
(t(y) (7r))) = 00
(t. no).
(4)
00
00
In this paper, mix the usual E-notation of summation with the Einstein notation: in the same term, an index appearing in both upper- and lower-positions indicates a sum over this index. For example, h a La h, Hence, in equations, upper- and lower-indexes have the he and ha mean the a'th element of h. For convenience and consistency, we take quantities w.r.t. upper indexed and other quantities as lower indexed, i.e., ( versus Iıp (hz). This mixed representation of sums helps to simplify our expressions without causing confusion. From Eq. (4) and Lemma the Hessian of l is given by
meaning: both
Ee 0000T (t. na) 0000
hh Th dna
the
dh 00
Oh 00
d0 T0T (t., na) 0000
Ins (hL)
Ian
(5)
Similar to the case of a general statistical model, the FIM equivalent to the expectation of the Hessian of -L as long as the activation function is smooth enough.
Theorem 3. Consider the neural network model in Eq. (3). For any activation function a E C3R(B)
(both o' (2) and (z) exist and are continuous), we have I(0) Ep 3900
82t
Remark 3.1. ReLU networks do not have this equivalent expression ReLU(z) is not differen- tiable at = 0.
Through the definition of the FIM, or alternatively its equivalent formula in Theorem 3, we arrive the same expression
[hh a0
Th Th TeT
th agT
I(0) =
I(hL)
00 I.b (h))
Equation (6) takes the form of generalized Gauss-Newton matrix [18]. This general expression of the FIM has been known in the literature [24, 25]. Under weak conditions, I(0) is pullback

---

metric [30] of (h1) in Lemma 2 associated with the mapping 0 + hL. To compute I(0) in closed form, one need first compute the Jacobian matrix of size nt, dim(O) then perform the matrix multiplicatiin in Eq. (6). The naive algorithm to evaluate Eq. (6) has a computational complexity of Onn), dim(0)+ dim (0)), where the term Onz dim" (0) is dominant as dim (0) >> n2 in deep architectures. Once the parameter 0 is updated, the FIM has to be recomputed. This is infeasible in practice for large networks where dim (0) can be millions or billions.
The two estimators Σ, (0) and Zg (0) in Eq. (2) provide a computationally inexpensive way to esti- mate the FIM. Given 0 and æ, one can draw i..dd samples yi..... Vv ~ p(y x 0). Both 00, and 04000000" can be evaluated directly through auto-diiferentiation (AD) that is highly opti- mized for modern GPUs. For In (0), we already have / to perform gradient descent. For (0), efficient methods to compute the Hessian are implemented in AD frameworks such as Py'Torch [26]. Using these derivatives, the computational cost only scales with the number N of samples but does not scale with n.
We rarely need the full FIM of size dim(O) dim (0). Most of the time, only its diagonal blocks are needed, where each block corresponds to subset of parameters, e.g., the neural network weights of a particular layer. Therefore the computation of both estimators can be further reduced.
If p(y x, 0) has the parametric form in Eq. (3), from Eqs. (4) and (5), the FIM estimators become
Öhg.
1
ohl 00 T
[0)
(t))
-ta((t()))) mb)-
00
N
N
8h
Sh
oh 00
(0) = na
tu (y:)
+
Iab (hL)
(8)
N
0000T
00
Recall that the notation of Ż, (0) and (0) is abused as they depend on æ and y -.. yn. Notably, in Eq. (7), 2, (0) is expressed in terms of the Jacobian matrix of the mapping 0 - h2, and the empirical variance of the minimal sufficient statistic t(y:) of the output exponential family. In Eq. (8), (0) depends on both the Jacobian and Hessian of O + h, and the empirical average of tyy:). The second term the right-hand side (RHS) of Eq. (8) is exactly the FIM, and therefore the first term serves as a bias term. Eqs. (7) and (8) are only for the case with exponential family output. If the output units belong to non-exponential families, egg., statistical mixture model, one falls back to the general formulac, i.e., Eq. (2) for the FIM.
As application of the Fisher information, the Cramér-Rao lower bound (CRLB) states that any unbiased estimator θ of the parameters 0 satisfies Cov ž [Z())]- For example, in Lemma 2, the FIM is w.r.t. the output of the neural network. As such, I(hrr) can be used to study the estimation covariance of h based on random samples yı ... yN drawn from ply ar, O). Similarly for Eq. (6), we can consider unbiased estimators of the weights of the neural network. In any case, to apply the CRLB, one needs an accurate estimation of (0).
the scale of I(0) relatively small when compared to its covariance, its estimation (() is more likely to be small positive value (or even worse, zero or negative). The empirical computation of the CRLB is not meaningful in this case.
3 The Variance of the FIM Estimators
Based on the deep learning architecture specified in Eq. (3), we measure the quality of the two estimators (0) and (0) given by their variances. Given the same sample size N, a smaller variance is preferred as the estimator is more accurate and likely to be closer to the true FIM I(0). We study how the structure of the exponential family has an impact on the variance.
3.1 Variance in closed form
We first consider (0) and (o) in Eq. (2) as real matrices of dimension dim (0) x dim(A). As
Ż, (0) is square matrix, the corresponding covariance is 4D tensor [Cov ( (0) fikt
of dimen- sion dim(@) dim(@) x dim (0) x dim((), representing the covariance between the two elements

---

(0) and (0). The element-win wise variance of (0) is matrix with the same size of (0).
which we denote Var (0)). Thus,
Var(2, = [Cov (Z1(0))
Similarly, the covariance and element-wise variance of I2( are denoted as [Cov (Ez(0)) Var( respectively.
iike
and
As the samples yı...... UN are i..dd, we have
and Cov( Cov ( de
0
(ov( N
Cov
(10)
00 a0T
N
0000T
Both Cov( ( (0)) and Cov(Iz(0)) have an order of O(1/N) For the neural network model in Eq. (3), we further have those covariance tensors in closed form.
Theorem 4.
[oov (Z)) 1961
de 0e
00 de
Cov
N
00:
00;"
00, 00:
n
(Kabea(t) Int(ha) Ioa(h)))
N
where the 4D tensor
Kt)ea(t) E [[t, no (z)) no((h((2))
the 4th (unscaled) central moment t(y) and 0 (x) == 0hg (x))
Remark 4.1. The 4D) tensor (Kabed (t) (hi() Ind (h1.)) is the covariance of the random trix
ae a8 (t(y) (t(r)) - n1)
oh, Th
where y ~ ~ply ht). This random matrix is an estimator of I i.c the FIM w.Et. the nat- ural parameters ht- Theorem 4 describes how the covariance tensor adapts w.r.t. the coordinate transformation h2 6.
Notably, t(y) is the sufficient statistics of exponential family, the derivatives of the log- partition function F(h) w.r.t. the natural parameters are equivalent to the cumulants t(y). The cumulants correspond to the coefficients of the Taylor expansion of the logarithm of the moment generating function [20]. Importantly, the cumulants of order 3 and below are equivalent to the cen- tral moments (see e.g. Lemma 2). However, this is not the case for the 4th central moment which must be expressed as a combination of the 2nd and 4th cumulants, as stated the following Lemma.
Lemma 5.
(tace(t)) = Naabe + Iat (ha( + Iac(he) Isa(he) + Iad(hL) Ise Ibc())),
where
8*
Kabed
Ōh, Öh, h=hu(
Remark 5.1. In the ID case, the 4th central moment simplifies to K((t) = F""(hL) + (hF"(hL))
For the second estimator i2(0), the covariance only depends on the 2nd central moment of t(y). Theorem 6.
[Cov (Ia(0))) iskt
8
0
Cov
B (x)
00,00,
00.00,
where (x) 8h 000,002
'The kurtosis of random variable is defined by its 4th standardized (both centered and normalized) moment. Here, XC(-) denotes the 4th central moment but not the kurtosis.
In this paper, the derivatives are by default taken w.r.t. 0. Therefore, d: == 30,
2 and 0G 50.50,
8²
5

---

Remark 6.1. By Lemma 2, the matrix (as(h is the covariance of the sufficient statistic t(y). Hence, the covariance of L2 (6) scales with the covariance t(y) If tly) tends be deterministic, then the covariance of I2 (0) shrinks towards and its estimation of the FIM becomes accurate.
The covariance in Theorems 4 and 6 has two different components: ① the derivatives of the deep neural network; and the central (unscaled) moments of fyy. The tensor Kabea (t) and the 2D FIM I (hr) correspond to the 4th and 2nd central moments of t(y), respectively. Intuitively, the larger the scale the Jacobian or the Hessian of the neural network mapping + h2 and/or the larger the central moments of the exponential family, the lower the accuracy when estimating the FIM.
3.2 Variance Bounds
We aim to derive meaningful upper bounds of the covariances presented in Theorems and 6. Using the
inequality, we can "decouple" the derivatives the neural network mapping and the central moments of the exponential family into different terms. This provides various bounds on the scale of covariance quantities.
Theorem 7.
Cov (Z1(0)
oh.. 00
||(((t) - I(hx) @ I(h
N
where & is the tensor-product: (I(hL) @ [(hL))aaoo == (hr) Ica
The scale Coor (0) measures how much the estimator Ż (0) deviates from I(0). Theo- rem 7 says that this deviation is bounded by the scale of the Jacobian matrix 0h2/00 well as the scale of (K((t) I(h,,) @ I(hx))- Recall from Remark 4.1 the latter measures the variance when estimating the FIM I(hr) of the exponential family. Theorem allows study these two different factors separately. Similarly, we have an upper bound on the scale of the covariance of (0).
Theorem 8.
Cov (I,(0))
1
h (x))
S N
8000 (I
On the RHS, the Hessian &h (2r 00000 is 3D tensor of shape ny, x dim(O) x dim(O). There- fore, the variance of (0) bounded by the scale of the Hessian, well as the scale of the FIM I(hz) of the output exponential family.
We consider upper bound to further simplify related terms in Theorems and 8. Lemma 9.
n2
[|C(t) I(h,) @ I(hr)ll
/2
Σ
VKaaao (t) Ina Ina(tt())))))
[I(h))
Iaa (.z))
Remark 9.1. Using Lemma 9, it is straighfforward to bound the scale of the covariance tensors with the size of the Jacobian/Hessian, as well the central moments Kaeaa (t) and Iaa (hL). These bounds are meaningful but omitted for brevity.
Remark 9.2. By Lemma 9, [[((()) - I(h2) @ I(l is in the order of O and [[Z(h)))l is in the order of O(n)).
The scale of the tensors XC(L) I(h,,) 8 I(hi.) and I(hr.) ) is bounded by the diagonal elements of KC(t) and Thh,), or the element-wise central moments of E(y). Understanding the scale of these 1D central moments helps to understand the scale of the moment terms in our key statements.

---

Table 1: Cumulants of univariate exponential family distributions, given by derivatives of the log- partition function. P. μ and À denote the mean the Bernoulli, normal, and Poisson distributions, respectively. The normal distribution has unit standard deviation (a 1).
DIST.
F(h)
h
8 Fh))
04 F())
BERNOULLI NORMAL#
log(1 + exp(h))
loge/1-p
p(1 p)
p(1 pp)p6p2 6p + +1)
//2
0
POISSON
exp(h)
log A
2,00
800
.2 (8) Var
0.24
600
Var(t)
400
0.11
1.25
200
0.0
1.00
0.50
-1.0
-0.5
0.5
p
(a) Berouuli.
(h) Normal (a = 1).
(c) Poisson.
Figure 1: The scale of (C) - (t) and Var(t) for the exponential family distributions in Table
Table presents some ID exponential families and their cumulants. Figure displays K.(t) ((r))Var(u (1) and Var(t) against the mean of these distributions. Based on Fig. 1a, if the neural network has Bernoulli output units, then the scale of Kaaaaa (t) - (Iaa (hz))²² is smaller than Iea (hr)) regardless of h2. Notably, when = 0.5, the variance of the first estimator 2, (0) is 0 - regardless of hL. For normal distribution output units (corresponding to the mean squared error loss) Fig. 1b, both central moment quantities are constant. For Poisson output units in Fig. Ic, Ian (h..) increases linearly with the average number of events X, while Knoza (t) - (Ina (hz))² increases quadratically. Thus, the upper bound [Cov(I
increases faster than the upper bound of ([z))( as h& enlarges. In this case, one may prefer (0) rather than Ż₃ (0) and/or control the scale of ht. In general, hg desired be in certain regions in the parameter space of the exponential family to control the estimation variance of the FIM. Techniques to achieve this include regularization on the scale of h2; temperature scaling [12]: or normalization layers [4, 28]. Of course, they could inversely increase the scale of the derivatives of the neural network, which can be controlled by imposing additional constraints, i.e., Lipschitz requirements.
3.3 Positive Definiteness
By definition, the FIM of any statistical model is positive semidefinite (p.s.d.). The first estimator (0) is naturally on the p.s.d. manifold (space of p.s.d. matrices). On the other hand, (²) can "fall off" the p.s.d. manifold. It is important to examine the likelihood for (0) having a negative spectrum and the corresponding scale, so that any algorithm (e.g. natural gradient) relying of the FIM being p.s.d. can be adapted.
Eq. (8) can be re-expressed as the sum of p.s.d. matrix and a linear combination of nı symmetric matrices. We provide the likelihood for (0) staying on the p.s.d. manifold given conditions on the spectrum of the Hessian.
Theorem 10. Let Amia(.), Amax((), and p(-) denote the smallest eigenvalue, the largest eigen- value, and the spectral radius (largest absolute value of the spectrum), respectively. Let == (p(8h))) ((00))))....p))""
h02h).)) )). If A'i (I()) > O, then with probability at least
1 1 nz ||e| Amase ([(r))
N X2 (I(O))
the estimator (0) with N samples is p.s.d. matrix.
The bound becomes uninformative as the output layer size ny increases, as the spectrum of the Hessian of h scales up. or the spectrum of the FIM I(hz) enlarges. On the other hand, as the
7

---

minimal eigenvalue of the FIM I(0) increases, Theorem 10 can give meaningful lower bounds. In particular, with sample rate ((N-1), estimator (0) will be p.s.d. matrix. In practice for over- parametrized networks, Amiu (0)) is close to or equals 0 and Theorem 10 not meaningful. In any case, we need to consider the scale of the negative spectrum of (0). Theorem 11.
N Σ ta (v:) N
Amiu (Ix(0) 2 ~p(02h;
na
Theorem 11 guarantees that in the worst case, the scale of the negative spectrum of (0) is con- trolled. By Lemma 2, Var(n , Σ to (y:)) _ (hz). Therefore, as N increases or _ (hr.) decreases, the negative spectrum of (0) will shrink. Further analysis on the spectrum of I (0) and (0) can utilize the geometric structure of the p.s.d. manifold. This is left for future work.
3.4 Convergence Rate
The rate of convergence for each of the estimators is of particular interest when considering their practical viability. Through generalized Chebyshev inequality [8], we can get simple Frobenius norm convergence rate.
Lemma 12. Let < Then
dim
Ol de
S
Var
VeN
00, 00,
holds with probability at least - andd
dim
|20( - S
Σ 4==1
Var
8
VeN
00,00,
hold with probability at least -
Each of these convergence rates only depends the element-wiss variance of the estimator terms in Eq. (9). Moreover, each of the estimators has convergence rate of (NN-1/2). The rate's constants are determined by the variance of the estimators given by Theorems and 6, which are influenced by the derivatives of the neural network and the moments of the output exponential family.
4 Effect of Neural Network Derivatives
The derivatives of the deep learning network can affect the estimation variance of the FIM. By Theorem 4, the variance of the first estimator Ż, (0) scales with the Jacobian of the neural network mapping + (x). By Theorem 6, the variance of Zz(0) scales with the Hessian of - (z)). The larger the scale the Jacobian the Hessian, the larger the estimation variance. In this section, we examine these derivatives in more detail.
We give the closed form gradient of the log-likelihood and the last layer's output h5 w.r.t. the neural network parameters.
Lemma 13.
ai D Ot
OE
dh D.B
T
BT (U(y) m/hr))))
aw,
Sh1t1
Th
ow:
where en is the ah standard basis vector, B and D; are recursively defined by
B: B D,W,,
Ir-1=I1, D. diag (o' (WWhh)))
I is the identity matrix, and diag(.) means diagonal matrix with given diagonal entries.

---

By Lemma 13, we can estimate the FIM w.r.t. the hidden representations h through
1 N N 2m1 Oh dh,
de, de,
N N E (t(u.) m(hL)) (t(y,) - (hh))) Bi. (11)
[hi) =
B
As B; is recursively evaluated from the last layer to previous layers, the FIM can also be recur- sively estimated based on I, (0). It is similar to pack-propagation, except that the FIMs are back-
propagated instead of gradients of the network. This is similar the backpropagated metric [23].
To investigate how the first estimator I (0) is affected by the loss landscape, we bound the Frobenius norm of the parameter-outpe Jacobian 0/0/
Lemma 14. If the activation function has bounded gradient and Vz E R, lo' S 1, then
th. dw
L-1
= | Dille [|lll22 -
II ==1+1
W 1F |
(12)
where @hr/ow; is a 3D tensor.
"0h,"" /owi] the derivative of vector w.r.t. matrix that
Given Lemma 14, that the gradient @hW/OW, scales with both the neural network weights W; and the gradient of the activation function D,. Common activation functions have both bounded outputs and 1st-order derivatives; tion functions. During training, regularizing the scale the neural network weights is a sufficient condition for bounding the variance of Σ (0).
least are locally Lipschitz, i.e., sigmoid and ReLU activa-
An alternative bound can be established which depends on the maximum singular values of the weight matrices.
Lemma 15. Suppose that the activation function has bounded gradient Vz E R, [o'(=)) S 1. Then
Oh. aw
h-1
Smax (; (:))
II ii=++
-1hull22
(13)
2,
where 8max (-) denotes the maximum singular value and [|T 12, denotes the tensor spectral norm for a 31) tensor T, defined by
ITT1,2 max {(T @ @ |||||2 = || l| 1}.
Therefore, regularizing Smax (W,). the spectral norm of the weight matrices, also helps to im- prove the estimation accuracy of the FIM.
We further reveal the relationship between the loss landscape and the FIM estimators. For given target ÿ, the logelikeli is denoted as ī := log p( 0). Furthermore, let us define AŽ, (0) == (ai (ai/a0
I (0) and AŻ, ()) -00000000 -īz By Eqs. (4) and (5),
dh 00T
(Z)
(t. (g) nn) (+) nb)
(L (y:) mn)((o(y))) m))
00
N
82
Ai₂(0)-
Σ
ta to (ū)
0000T
Hence, the difference between Ż, (0) (resp. Iz)) and the squared gradient (resp. Hessian) of the loss -/ depends how y: differs from ÿ. If the network 0 is trained, then the random samples y: ply æx 0) are close to the given target z. In this case, (0) corresponds to the squared gradient, and (0) corresponds to the Hessian. This is not true untrained neural networks with random weights.

---

5 Related Work
The two estimators 2,(0) and (0) are not new as one usually utilizes one of them to compute the FIM. Guo and Spall [1I] analyzed their accuracy for univariate symmetric density functions based on the central limit theorem. Fisher information estimation is also examined in latent variable models [9]. Under the same topic, our work analyzes the factors affecting the variance of Z (0) and (0) for deep neural networks.
large body of work tries to approximate the FIM or define similar curvature tensors for per- forming natural gradient descent [1, 18, 19, 14, 23, 31j. If the loss is empirical expectation of log p x; f), its Hessian w.r.t. hy is exactly [h).). Then, the FIM in Eq. (6) is in the form of a Generalized Gauss-Newton matrix (GNN) [29, 18]. Ollivier [23] provided algorithm proce- dures to compute the unit-wise FIM and discusses Monte Carlo natural gradient. The FIM can be computed locally [31, 3] based a joint distribution representation of the neural network. Efficient computational methods are developed to evaluate the FIM inverse [24].
The estimator Ż, (0) not the "empirical Fisher" (see e.g. [18, Section I1]) as y; randomly sam- pled from pyy x f), making Σg (0) an unbiased estimator. The difference between the empirical Fisher and the FIM is clarified [15]. Similarly, the estimator (0) is not the Hessian of the loss, as y: is randomly sampled rather than fixed to the given target.
Recently, the structure of the FIM (or its partial approximations) are examined in deep learning. The FIM of randomized networks is analyzed [2], where the weights of the neural network assumed to be random. Often the analysis of randomized networks uses spectral analysis and random matrix theory [27]. An insight from this body of work is that most of the eigenvalues of the FIM are close to O; while the high end of spectrum has large values [13]. In this paper, the estimators Σn (0) and I2(0) are random matrices due to the sampling y; p(y 2:) (the weights are considered fixed).
In information geometry [1, 22]. the FIM serves as a Riemannian metric in the space of probabil- ity distributions. The FIM is a covariant tensor and is invariant to diffeomorphism on the sample space [22]. Higher order tensors are used to describe the intrinsic structure in this space. For exam- ple, the Riemannian curvature a 4D tensor, while the Ricci curvature is 2D. The third cumulants of the sufficient statistics give an affine connection (belonging to the a-connections or the Amari- Čensov tensor) of the exponential family. The FIM is generalized to one-parameter family [21].
In statistics, our estimator (2) usually evaluated at the maximum likelihood estimation Ô. Higher order moments of the maximum likelihood estimator (MLE) were discussed (see e.g. Bowman and Shenton [5]). These moments are associated with parameter estimators and differ from the concept of the FIM estimators. Similar concepts are examined in higher-order asymptotic theory [1, Chapter 7].
(f) is Efron and Hinkley [10]'s "observed Fisher information", which is
6 Conclusion
The FIM is covariant p.s.d. tensor revealing the intrinsic geometric structure of the parameter manifold. yields useful practical methods such as the natural gradient. practice, the true FIM I(0) is usually expensive or impossible to obtain. Estimators of the FIM based on empirical samples used in the deep learning practice. We analyzed two different estimators Σ, (0) and (0) of the FIM of a deep neural network. These estimators are convenient to compute using auto-differentiation frameworks but randomly deviates from I(O). Our central results, Theorems 4 and 6, present the variance of ī, (0) and (0) in closed form, which is further extended to upper bounds in simpler forms. Two factors affecting the estimation variance are ① the derivatives of neural network output h, w.r.t. the weight parameters 0; and @ the property of hr, as an exponential family distribution.
A large scale of the 1st- and/or 2nd-order derivatives leads to large variance when estimating the FIM. Our analytical results can useful to measure the quality of the estimated FIM and could lead to variance reduction techniques.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous NeurIPS reviewers for their constructive comments. We thank Frank
Nielsen for the insightful feedback. We thank James C. Spall for pointing us to related work.
References
[1] Shun-ichi Amari. Information Geometry and Its Applications, volume 194 of Applied Mathe- matical Sciences. Springer- Verlag, Berlin, 2016.
[2] Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher information and natural gradi- ent learning in random deep networks. In International Conference on Artificial Intelligence and Statistics, pages 694-702. PMLR, 2019.
[3] Nihat Ay. On the locality of the natural gradient for learning in deep Bayesian networks. Information Geomerry, pages 1-49, 2020.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXiv:1607.064333 2016.
[5] K. O. Bowman and L. R. Shenton. Properties of Estimators for the Gamma Distribution. Marcel Dekker, New York, 1988.
[6] Bilian Chen and Zhening Li. On the tensor spectral p-norm and its dual norm via partitions. Computational Optimization and Applications, 75(3):609-628, 2020.
[7] Jiahua Chen and Pengfei Li. Hypothesis test for normal mixture models: The EM approach. The Annals of Statistics, 37(5A):2523-2552 2009.
[8] Xinjia Chen. A new generalization of Chebyshev inequality for random vectors. arXiv preprint arXiv:0707.000 0805, 2007.
[9] Maud Delattre and Estelle Kuhn. Estimating Fisher information matrix in latent variable mod- els based on the score function. In European Meeting of Statisticians (EMS), 2019.
[10] Bradley Efron and David V. Hinkley. Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika, 65(3):457-422, 1978.
[1I] Shenghan Guo and James C. Spall. Relative accuracy of two methods for approximating ob- served Fisher information. In Data-Driven Modeling, Filtering and Control: Methods and applications, pages 189-211. IET Press, London, 2019.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 553.0253/1 2015.
[13] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of Fisher informa- tion in deep neural networks: Mean field approach. In International Conference on Artificial Intelligence and Statistics, pages 1032-1041. PMLR, 2019.
[14] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[15] Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems, pages 4133-4144. Curran Associates, Inc., 2020.
[16] Erich L. Lehmann and George Casella. Theory of Point Estimation. Springer- Verlag New York, second edition, 1998.
[17] Lek-Heng Lim. Singular values and eigenvalues of tensors: variational approach. In 1st IEEE International Workshop on Computational Advances Multi-Sensor Adaptive Process- ing, 2005., pages 129-132. IEFF, 2005.
[18) James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76, 2020.
[19] James Martens and Roger Grosse. Optimizing neural networks with Kronecker- factored ap- proximate curvature. In International Conference on Machine Learning, pages 2408-2417. PMLR, 2015.
[20] Peter McCullagh. Tensor methods in statistics. Courier Dover Publications, 2018

---

[21]
Frank Nielsen. The a-representations of the Fisher information matrix, 2017. URL https: //rankniiesen. github. ht./blcg/appha......... IM/ modex.htm...
[22] Frank Nielsen. An elementary introduction to information geometry. Entropy, 22(10), 2020.
[23] Yann Ollivier. Riemannian metrics for neural networks I: feedforward networks. Information
and Inference: A Journal of the IMA, 4(2): 10:-153,,.. 2015.
[24] Hyeyoung Park, Shun-ichi Amari, and Kenji Fukumizu. Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, (3(7):5557764 2000.
[25] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In Inter- national Conference on Learning Representations, 2014.
[26]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035. Curran Associates, Inc., 2019.
[27] Jeffrey Pennington and Pratik Worah. The spectrum of the Fisher information matrix of single-hidden-layer neural network. In Advances in Neural Information Processing Systems, pages 5415-5424, 2018.
[28] Tim Salimans and Durk P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[29] Nicol N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1723-1736 2002.
[30] Ke Sun. Information geometry for data geometry through pullbacks. Deep Learning through Information Geometry (Workshop at NeurIPS 2020), 2020.
[31] Ke Sun and Frank Nielsen. Relative Fisher information and natural gradient for learning large modular models. In International Conference on Machine Learning, pages 3289-3298. PMLR, 2017.
12


Reviews:
[{"review_id": "Q0udI5TVCIq", "reviewer": "Reviewer_muyd", "summary": "For two commonly used approximations to the Fisher Information Matrix, this paper derives bounds on the variance of these estimates. They give examples of how these bounds change for different types of exponential family neural network output distributions.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "After looking at the author responses, I am raising my score to an 8 to reflect that I already believed the paper was good enough for acceptance, but is now stronger.\n\n----------------------------\n\nOverall this is a well-done theory paper. I believe that these bounds will be useful references for practitioners deriving optimization preconditioners or other methods to compute local loss surface information via the FIM, which in my opinion is an important ongoing area of research. Additionally, the possibility of negative eigenvalues is underappreciated in practice, and so section 3.3 is a nice consideration of this situation.\n\nCorrectness:\nI briefly reviewed the proofs in the appendix, and while they appear correct I could have missed a detail due to the sometimes complicated algebra.\n\nWriting:\nThe paper is well written, with arguments and theorems clearly set up and presented. Minor nits:\n-line 52, exponentially -> exponential\n-line 189 “neural network” -> “the neural network”\n\nPrior work:\nTo my knowledge no such bounds have been previously derived, and thus this is a sufficiently novel paper.\n\nAdditionally, I’m not sure if there is such a relation (the forms look similar enough I wanted to ask), but is there any relationship between your equation (8) and the Generalized Gauss-Newton Matrix (see sec. 8 of https://arxiv.org/abs/1412.1193)? If so that would be a nice connection to make.\n\nAdditional feedback:\nThis is by no means a requirement, as theory-only papers are perfectly acceptable, but while the examples in Table/Figure 1 are useful and compelling, the paper would be further strengthened by small-scale feed-forward neural net experiments where you could analytically compute the FIM and the approximations presented in this paper.\n\nAt the end of section 2.3, you say “control the scale of neural network output h_L”. Could this be possible by using temperature scaling or adding a normalization layer to the top of the network (although the latter may introduce complexities in your assumptions about the NN structure h_i)? It could be nice to at least mention that in practice it is easily possible/common to control the scale of h_L.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I agree with the authors that I do not believe there are relevant societal impacts of this work that are useful to discuss at this time.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8m2kj30Eawx", "reviewer": "Reviewer_Q3bA", "summary": "This paper investigates the Fisher information matrix (FIM) theoretically. Two equivalent expressions of FIM and their estimators are discussed (rediscovered) for deep neural network with exponential family output units. The closed-form variance (and simpler-form upper bounds) for the two estimators is proposed; the convergence rate of the positiveness definiteness are also analyzed. Finally, the variance is connected with the structure of the corresponding deep neural networks.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nWhile the two estimators are not proposed originally, their variance has not been analyzed theoretically in machine learning, claimed by the authors. Besides, positive definiteness, convergence rates of the two estimators and the connection with the structure of the deep neural networks are also analyzed in details.\n\n**Clarity**\n\nThis paper is clearly presented and readable. Although I am not familiar with this research area, I could keep up with the storyline without much effort.\n\n**Significance**\n\nThe discussed estimators are very intuitive and make sense to me. In fact, I feel a little surprised about the \"fact\" that the two estimators have not been analyzed.\n\n**Concerns**\n\nIn spite of some mentioned related work, more comparison of existing estimators with the two estimators should be proposed. It is confusing about the position in literature.\n\nMoreover, since the two esitmators are quite simple, it is suggested to add some application cases (maybe toy cases) about them.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "-63G53bl5-z", "reviewer": "Reviewer_Wuqv", "summary": "Update: I thank the authors for their response, and I will keep the score as is. It would be nice to have a simple empirical analysis in the final supplemental material.\n\n\nThe paper discusses the covariance of two FIM estimators, which is useful in deep learning where the exact FIM over the full dataset can be hard to compute. One is based on the variance of the score (type 1) and the other is based on the Hessian of the relative entropy (type 2). Comparing these covariances can depend on the exponential family used in the model, type 1 has a higher variance for Normal and Poisson whereas type 2 has a higher variance for Bernoulli. Convergence rate results are given which is $O(N^{-0.5))$ with constant determined by the variances. These results provide some theoretical justification as to which estimator of FIM to use when using it to perform natural gradient in deep learning. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is theoretical but is highly related to deep learning practice, especially when natural gradients are used. The paper compared two variants of FIM estimators, computed their covariances (Theorem 4, and Theorem 6), upper bounds (Corollaries 7 and 8), when these estimators are PSD (theorem 9, theorem 10), convergence rate (Lemma 11), and the structure of the deep learning network (Lemma 13).\n\nI think the paper is clearly written, the results are solid, and the problem investigated is important to machine learning and natural gradient methods. The variance results clearly indicate what type of FIM estimator is preferred in NG in principle.\n\nMy only concern is that there are not any empirical results that corroborate with the theory. One very simple example is to take a very small neural network (just a two-layer one would suffice) on a small task, and evaluate their FIM with the same number of samples using the two FIM estimators. I also think it is interesting to see if these choices affect natural gradient optimization (although I do not think this is necessary to the paper).\n\nAdditional comment: can we gain insights regarding the spectrum of the estimated FIM, not just the smallest eigenvalue? This also seems to be related to stochastic optimization, because a small condition number for the true FIM does not necessarily indicate a small condition number of the estimated FIM?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed.\nSocietal impact is N/A because the paper is theoretical.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this is a solid theoretical paper that makes a real contribution. The paper is well-written relative to the complexity of what it's presenting.
Decision: Accept (Poster)