Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds

## Abstract

Abstract The high dimensional parameter space of modern deep neural networks
— the neuromanifold — is endowed with a unique metric tensor
defined by the Fisher information, estimating which is crucial
for both theory and practical methods in deep learning.
To analyze this tensor for classification networks, we return
to a low dimensional space of probability distributions
— the core space — and carefully analyze the spectrum of
its Riemannian metric. We extend our discoveries there into
deterministic bounds of the metric tensor on the neuromanifold.
We introduce an unbiased random estimate of the metric tensor
and its bounds based on Hutchinson’s trace estimator.
It can be evaluated efficiently through a single backward pass
and can be used to estimate the diagonal, or block diagonal,
or the full tensor. Its quality is guaranteed with a standard
deviation bounded by the true value up to scaling.

## 1   Introduction

Deep learning can be considered as a trajectory through *the space of neural networks* (*neuromanifold* [2]) where each point is a neural network instance with a prescribed architecture but different parameters.
This work investigates classifier models in the form $p(y\,|\,x,\theta)$, where $x$ is the input features, $y$ is the class label in a discrete domain, and $\theta\in\Theta$ is the network weights and biases. Given an unlabeled dataset $\mathcal{D}_{x}=\{x_{1},x_{2},\cdots\}$,
the intrinsic structure of $\Theta$ is uniquely specified
by the Fisher Information Matrix (FIM), defined as:

$$ $\mathcal{F}(\theta)\coloneqq\sum_{x\in\mathcal{D}_{x}}\mathop{\mathbb{E}}_{p(y \,|\,x)}\left[\frac{\partial\log p(y\,|\,x,\theta)}{\partial\theta}\frac{ \partial\log p(y\,|\,x,\theta)}{\partial\theta^{\top}}\right]=\sum_{x\in \mathcal{D}_{x}}\mathop{\mathbb{E}}_{p(y\,|\,x)}\left[\frac{\partial\ell_{xy}} {\partial\theta}\frac{\partial\ell_{xy}}{\partial\theta^{\top}}\right],$ (1) $$

where $\ell_{xy}(\theta)\coloneqq\log p(y\,|\,x,\theta)$ denotes the likelihood of the pair $(x,y)$ with respect to (w.r.t.) $p(y\,|\,x,\theta)$.
This is based on a supervised model $x\to{y}$. For unsupervised models, one can treat $x$
as constant and apply the same formula.
Under regularity conditions, $\mathcal{F}(\theta)$ is a
$\dim(\theta)\times\dim(\theta)$ positive semi-definite (psd) matrix varying smoothly with $\theta\in\Theta$.
Following Hotelling [8], and independently Rao [26],
$\mathcal{F}(\theta)$ is used as a metric tensor on $\Theta$,
representing a local degenerate inner product and underpinning the *information geometry* of $\Theta$ [2].
In the machine learning literature, $\mathcal{F}(\theta)$ is sometimes referred to as a curvature matrix [16]
but actually is a *singular semi-Riemannian metric* [34] in rigorous terms.
One can use $\mathcal{F}(\theta)$ to build better learning trajectories and efficient optimization
with variants of the natural gradient [1, 23, 13, 38].

The existing work has focused on addressing the computational challenges through approximations of $\mathcal{F}(\theta)$,
thereby enabling its practical applications in real-world scenarios,
where $\dim(\theta)$ ranges from millions to billions.
For example, the empirical FIM (eFIM, a.k.a. empirical Fisher)

$$ $\hat{\mathcal{F}}(\theta)\coloneqq\sum_{(x,y)\in\mathcal{D}}\left[\frac{ \partial\ell_{xy}}{\partial\theta}\frac{\partial\ell_{xy}}{\partial\theta^{ \top}}\right],$ $$

where $E_{p(y\,|\,x)}\left[\frac{\partial\ell_{xy}}{\partial\theta}\frac{\partial\ell
_{xy}}{\partial\theta^{\top}}\right]$ in [Eq. 1](https://arxiv.org/html/2505.13614v1#S1.E1) is replaced by
$\left[\frac{\partial\ell_{xy}}{\partial\theta}\frac{\partial\ell_{xy}}{
\partial\theta^{\top}}\right]$, and $\mathcal{D}=\{(x_{1},y_{1}),(x_{2},y_{2}),\cdots\}$ is a labeled dataset,
is widely used [16] as a computationally friendly proxy of $\mathcal{F}(\theta)$.
Intuitively, when the network is trained,
computations based on the given labels is close to the expectation w.r.t. ${p(y\,|\,x)}$.
However, eFIM is biased. As we will show latter, the bias of $\hat{\mathcal{F}}(\theta)$
can be enlarged if $y$ is set adversarially.

Aiming at estimates of $\mathcal{F}(\theta)$ with guaranteed quality,
this paper advances in two directions:
①
a novel deterministic analysis of the FIM based on matrix perturbation theory,
and ② a family of random estimators using Hutchinson’s trick [10], following the procedure outlined below.
First, compute the scalar-valued function

$$ $\displaystyle\mathfrak{h}(\mathcal{D}_{x},\theta)$ $\displaystyle\coloneqq\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\sqrt{\tilde{p}( y\,|\,x,\theta)}\ell_{xy}(\theta)\xi_{xy},$ (2) $$

where $\xi_{xy}$ is a standard multivariate Gaussian vector of size $C|\mathcal{D}|$ or a Rademacher vector,
and $\tilde{p}(y\,|\,x,\theta)$ has the same value as $p(y\,|\,x,\theta)$ but is *non-differentiable*, meaning its gradient is always zero, preventing error from
back-propagating through $\tilde{p}(y\,|\,x,\theta)$. This $\tilde{p}$ can be implemented by Tensor.detach() in PyTorch [24] or similar functions in other auto-differentiation (AD) frameworks.
Second, the gradient vector

$$ $\frac{\partial\mathfrak{h}}{\partial\theta}=\sum_{x\in\mathcal{D}_{x}}\sum_{y= 1}^{C}\sqrt{p(y\,|\,x,\theta)}\frac{\partial\ell_{xy}}{\partial\theta}xi_{xy}$ $$

can be evaluated by AD, *e.g*. by $\mathfrak{h}$.backward() in Pytorch.
Third, the random psd matrix
$\mathbb{F}(\theta)\coloneqq\frac{\partial\mathfrak{h}}{\partial\theta}\frac{
\partial\mathfrak{h}}{\partial\theta^{\top}}$, which we refer to as the “Hutchinson’s estimate” (of the FIM), can be used to estimate $\mathcal{F}(\theta)$.
By straightforward derivations,

$$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}\left(\mathbb{F}(\theta)\right)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\sum_{x^{\prime}\in \mathcal{D}_{x}}\sum_{y^{\prime}=1}^{C}\sqrt{p(y\,|\,x,\theta)}\sqrt{p(y^{ \prime}\,|\,x^{\prime},\theta)}\frac{\partial\ell_{xy}}{\partial\theta}\frac{ \partial\ell_{x^{\prime}y^{\prime}}}{\partial\theta^{\top}}\mathop{\mathbb{E}} _{p(\xi)}\left[\xi_{xy}\xi_{x^{\prime}y^{\prime}}\right]=\mathcal{F}(\theta).$ (3) $$

The last “=” is because $\mathop{\mathbb{E}}_{p(\xi)}(\xi_{xy}\xi_{x^{\prime}y^{\prime}})=1$ if $x=x^{\prime}$ and $y=y^{\prime}$,
and $\mathop{\mathbb{E}}_{p(\xi)}(\xi_{xy}\xi_{x^{\prime}y^{\prime}})=0$ otherwise.
Considering $\frac{\partial\mathfrak{h}}{\partial\theta}$ as an implicit representation
of the FIM, its computational cost is ① evaluating the $\mathfrak{h}$ function,
② the backward pass to compute the gradient of $\mathfrak{h}$.
The cost is same as evaluating the classification loss
$-\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\ell_{xy}(\theta)$ (observe how similar [Eq. 2](https://arxiv.org/html/2505.13614v1#S1.E2) is to the loss)
and its gradient.
Moreover, $\mathfrak{h}$ can reuse the logits already computed during the forward pass.
Therefore $\frac{\partial\mathfrak{h}}{\partial\theta}$ requires merely one additional backward
pass, making it practical for large scale networks.
Note that a sample of the random matrix $\mathbb{F}(\theta)$
is always rank-1: $\operatorname*{\mathrm{rank}}{\mathbb{F}(\theta)}=1\leq\operatorname*{\mathrm{
rank}}{\mathcal{F}(\theta)}$,
but the expectation of $\mathbb{F}(\theta)$ has the same rank as $\mathcal{F}(\theta)$.
In summary, $\mathbb{F}(\theta)$ is a *universal estimator* of
$\mathcal{F}(\theta)$ for general statistical model,
which is independent of neural network architectures and applicable to non-neural network models.

In the rest of this section, we introduce our notations. [Section 2](https://arxiv.org/html/2505.13614v1#S2) develops fundamental bounds and estimates in low dimensional spaces of probability distributions.
[Section 3](https://arxiv.org/html/2505.13614v1#S3) extends the deterministic bounds into the high dimensional neuromanifold.
[Section 4](https://arxiv.org/html/2505.13614v1#S4) revisits Hutchinson’s FIM estimator introduced above and discusses its theoretical properties with numerical simulation on DistilBERT [27].
[Section 5](https://arxiv.org/html/2505.13614v1#S5) positions our work into the literature.
[Section 6](https://arxiv.org/html/2505.13614v1#S6) concludes. We provide outline derivation steps in the main text and leave detailed proofs in the appendix.

**Table 1: Metric tensors used in this paper. Both empirical FIM and Monte Carlo FIM are denoted as $\hat{\mathcal{I}}$ / $\hat{\mathcal{F}}$ for reducing notation overload. We use $\mathcal{I}$ / $\hat{\mathcal{I}}$ / $\mathbb{I}$ for simple low-dimensional statistical manifolds and use $\mathcal{F}$ / $\hat{\mathcal{F}}$ / $\mathbb{F}$ for neural networks. We optionally use superscripts to indicate the associated parameter space. For example, $\mathcal{I}^{\Delta}$ and $\mathcal{F}^{\Delta}$ denote the metric tensor of the statistical simplex and the space of neural networks with simplex-valued outputs, respectively.**
| FIM | empirical FIM | Monte Carlo FIM | Hutchinson FIM |
| --- | --- | --- | --- |
| $\mathcal{I}(z)$ / $\mathcal{F}(\theta)$ | $\hat{\mathcal{I}}(z)$ / $\hat{\mathcal{F}}(\theta)$ | $\hat{\mathcal{I}}(z)$ / $\hat{\mathcal{F}}(\theta)$ | $\mathbb{I}(z)$ / $\mathbb{F}(\theta)$ |

#### Notations and Conventions

We use lowercase letters such as $\lambda$ or $a$ for both vectors and scalars, which should be distinguished based on context, and capital letters such as $A$ for matrices. All vectors are column vectors.
A scalar-vector or vector-scalar derivative such as $\partial\ell/\partial\theta$ yields a gradient vector of the same shape as the vector.
A vector-vector derivative such as ${\partial z}/{\partial\theta}$ denotes the $\dim(z)\times\dim(\theta)$ Jacobian matrix of the mapping $\theta\to{z}$, with its rows and columns corresponding to the dimensions of $z$ and $\theta$, respectively.
$\|\cdot\|$ denote the Euclidean norm for vectors or Frobenius norm for matrices.
$\|\cdot\|_{\sigma}$ denotes the spectral norm (maximum singular value) of matrices.
The metric tensors (variants of FIM) are listed in [table 1](https://arxiv.org/html/2505.13614v1#S1.T1).

## 2   Geometry of Low-dimensional Core Spaces

Consider a classifier network $p(y\,|\,x,\theta)\coloneqq p(y\,|\,z(x,\theta))$, where $z(x,\theta)$ is last layer’s linear output.
Due to the chain rule, we can plug
$\frac{\partial\ell_{xy}}{\partial\theta}=\left(\frac{\partial z}{\partial
\theta}\right)^{\top}\frac{\partial\ell_{xy}}{\partial z}$ into [Eq. 1](https://arxiv.org/html/2505.13614v1#S1.E1).
After some simple derivations, we get

$$ $\mathcal{F}(\theta)=\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial \theta}\right)^{\top}\cdot\mathcal{I}(z(x,\theta))\cdot\frac{\partial z}{ \partial\theta},$ (4) $$

which is in the form of a Gauss-Newton matrix [17],
or a pullback metric tensor [32](^1^11Strictly speaking, the pullback tensor requires the Jacobian of $\theta\to{z}$ have full column rank everywhere, which is not satisfied in typical settings of deep neural networks. This leads to singular metric tensors.) from a low dimensional statistical manifold, where the metric is $\mathcal{I}(z)$, to the much higher dimensional neuromanifold, where the metric is $\mathcal{F}(\theta)$.
Therefore it is important to review the geometrical structure of the
low dimensional statistical manifold, which we refer to as the *core space*, or simply the *core*.

In multi-class classification, $y$ (given a feature vector $x$) follows a category distribution $p(y=i\,|\,x,\theta)=p_{i}(x,\theta)$, $i=1,\cdots,C$.
All possible category distributions over $\{1,\cdots,C\}$ form a closed statistical simplex

$$ $\Delta^{C-1}\coloneqq\left\{(p_{1},\cdots,p_{C})\;:\;\sum_{i=1}^{C}p_{i}=1;\; \forall{i},p_{i}\geq 0\right\}.$ $$

The superscript $C-1$ denotes the dimensionality of $\Delta$ and can be omitted.
If $p\in\mathrm{int}(\Delta^{C-1})$ (interior of $\Delta^{C-1}$),
we can reparameterize $p=\operatorname*{\mathrm{SoftMax}}(z)$, where $z\in\Re^{C}$ is the logits.
The core $\Delta^{C-1}$ is a curved space,
where $p$ or $z$ serves as a coordinate system in the sense that different choices of $p$ or $z$ yield different distributions.
By [Eq. 1](https://arxiv.org/html/2505.13614v1#S1.E1), the FIM is:

$$ $\mathcal{I}^{\Delta}(z)=\mathop{\mathbb{E}}\left[(e_{y}-p)(e_{y}-p)^{\top} \right]=\mathrm{diag}\left(p\right)-pp^{\top},$ (5) $$

where $\mathrm{diag}\left(\cdot\right)$ means the diagonal matrix constructed with a given diagonal vector. In below, depending on context, $\mathrm{diag}\left(\cdot\right)$ also denotes a diagonal vector extracted from a square matrix.
$e$ (without subscripts) denotes a vector of all ones, $e_{y}$ denotes the one-hot vector with only the $y$’th bit activated, and $e_{ij}$ denotes the binary matrix with only the $ij$’th entry set to 1.
Note $z$ is a redundant coordinate system
as $\dim(z)=C>C-1$. If $z\in\mathrm{int}(\Delta^{C-1})$,
$\mathcal{I}^{\Delta}(z)$ has a one-dimensional kernel: it is not difficult to verify $\mathcal{I}^{\Delta}(z)(te)=0$ for all $t\in\Re$.

By noting that $\mathcal{I}^{\Delta}(z)$ is a rank-1 perturbation of the diagonal matrix $\mathrm{diag}\left(p\right)$, we can apply Cauchy’s interlacing theorem and study
the spectral properties of $\mathcal{I}^{\Delta}(z)$.

###### Theorem 1 (Spectrum of Simplex FIM) .

Assume the spectral decomposition
$\mathcal{I}^{\Delta}(z)=\sum_{i=1}^{C}\lambda_{i}v_{i}v_{i}^{\top}$, where
$\lambda_{1}\leq\cdots\leq\lambda_{C}$.
Then
$\lambda_{1}=0$; $v_{1}=e/\|e\|$;
$\sum_{i=1}^{C}\lambda_{i}=1-\|p\|^{2}$;
and

$$ $\max\left\{p_{i}(1-p_{i})\right\}\cup\left\{p_{(C-1)},\frac{1-\|{p}\|^{2}}{C-1 }\right\}\leq\lambda_{C}\leq\min\left\{p_{(C)},2\max_{i}(p_{i}(1-p_{i})),1-\|p \|^{2}\right\},$ $$

where $p_{(C-1)}$ and $p_{(C)}$ denote the second-largest and the largest elements of $p$, respectively.

The largest eigenvalue of $\mathcal{I}^{\Delta}(z)$, denoted as $\lambda_{C}$, and its associated eigenvector correspond to the “most informative” direction at any $z\in\Delta^{C-1}$.
By [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1), $\lambda_{C}$ can be bounded from above and below.
The bound gap is at most
$\min\{p_{(C)}-p_{(C-1)},\max_{i}(p_{i}(1-p_{i}))\}$.
We have found through numerical simulations that, in practice, the bounds in [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1) are quite tight and can provide an estimate of $\lambda_{C}$ within a narrow range.
The lemma below gives lower and upper bounds of $\mathcal{I}^{\Delta}(z)$,
both with a simpler structure than $\mathcal{I}^{\Delta}(z)$,
in the space of psd matrices based on Löwner partial order.

###### Lemma 2 .

Assume the spectral decomposition
$\mathcal{I}^{\Delta}(z)=\sum_{i=1}^{C}\lambda_{i}v_{i}v_{i}^{\top}$, where
$\lambda_{1}\leq\cdots\leq\lambda_{C-1}<\lambda_{C}$.
Then,

$$ $\lambda_{C}v_{C}v_{C}^{\top}\preceq\mathcal{I}^{\Delta}(z)\preceq\mathrm{diag} \left(p\right).$ $$

Moreover,
$\lambda_{C}v_{C}v_{C}^{\top}$ is the best rank-1 representation of $\mathcal{I}^{\Delta}(z)$ in the sense that no rank-1 matrix $B\neq\lambda_{C}v_{C}v_{C}^{\top}$ satisfies $\lambda_{C}v_{C}v_{C}^{\top}\preceq B\preceq\mathcal{I}^{\Delta}(z)$.
Meanwhile,
$\mathrm{diag}\left(p\right)$ is the best diagonal representation of $\mathcal{I}^{\Delta}(z)$ in the sense that no diagonal matrix
$D\neq\mathrm{diag}\left(p\right)$ satisfies $\mathcal{I}^{\Delta}(z)\preceq{}D\preceq\mathrm{diag}\left(p\right)$.

The simplex FIM is upper-bounded by a diagonal matrix and lower bounded by a rank-1 matrix.
By [lemma 2](https://arxiv.org/html/2505.13614v1#Thmtheorem2), $\lambda_{C}v_{C}v_{C}^{\top}$ is the *lower-envelop* (greatest lower bound) of $\mathcal{I}^{\Delta}(z)$ in rank-1 matrices, and $\mathrm{diag}\left(p\right)$ is the *upper-envelop* (least upper bound) of $\mathcal{I}^{\Delta}(z)$ in diagonal matrices.
If the bounds in [lemma 2](https://arxiv.org/html/2505.13614v1#Thmtheorem2)
are used as a deterministic estimate of $\mathcal{I}^{\Delta}(z)$, the error can be controlled, as shown below.

###### Lemma 3 .

We have $\forall{z}\in\Delta$,
$\|\mathcal{I}^{\Delta}(z)-\mathrm{diag}\left(p\right)\|=\|p\|^{2}\geq\frac{1}{C}$; meanwhile,

$$ $\|\mathcal{I}^{\Delta}(z)-\lambda_{C}v_{C}v_{C}^{\top}\|\leq\min\left\{1-\|p\| -p_{(C-1)},\sqrt{\sum_{i=2}^{C-1}p_{(i)}^{2}}\right\},$ $$

where $p_{(i)}$ denote the entries of $p$ sorted in ascending order.

Note $\sqrt{\sum_{i=2}^{C-1}p_{(i)}^{2}}$ is the Euclidean norm of *trimmed* $p$, *i.e*. the vector obtained by removing $p$’s smallest and largest elements.
By [lemma 3](https://arxiv.org/html/2505.13614v1#Thmtheorem3), the upper bound $\mathrm{diag}\left(p\right)$ always incurs an error of at least $1/C$.
Depending on $p$, the lower bound $\lambda_{C}v_{C}v_{C}^{\top}$ can more accurately estimate $\mathcal{I}^{\Delta}(z)$ as the error can go to zero.

Alternatively, one can use random matrices to estimate $\mathcal{I}^{\Delta}(z)$. By [Eq. 5](https://arxiv.org/html/2505.13614v1#S2.E5),
the rank-1 matrix $R(y)=(e_{y}-p)(e_{y}-p)^{\top}$ is an unbiased estimator of $\mathcal{I}^{\Delta}(z)$.
The eFIM of $\Delta$ is given by $\hat{\mathcal{I}}^{\Delta}(z)=R(y)$, where $y$ is a given empirical sample
of the distribution specified by $z$. The lemma below shows the worst case error of using
eFIM to estimate $\mathcal{I}^{\Delta}(z)$.

###### Lemma 4 .

$\forall{z}\in\Delta^{C-1}$, $\exists y\in\{1,\cdots,C\}$, such that $\|R(y)-\mathcal{I}^{\Delta}(z)\|\geq 1+\|p\|^{2}-\lambda_{C}-2p_{(1)}\\
\geq 2\|p\|^{2}-2p_{(1)}$.

The first “$\geq$” is tighter but the second ”$\geq$ is easier to interpret. The term $\|p\|$ can be as large as 1 (when $p$ is close to one-hot), In such cases, using $R(y)$ to estimate $\mathcal{I}^{\Delta}(z)$ may incur significant error if $y$ is adversarially chosen.

The lemma below gives the average error (variance) of using $R(y)$ to estimate $\mathcal{I}^{\Delta}(z)$, where $y$ is a random variable distributed according to $p(y\,|\,z)$.

###### Lemma 5 .

The element-wise variance of the random matrix $R(y)$,
denoted by $\mathrm{Var}(R_{ij})$, is given by

$$ $\mathrm{Var}(R_{ij})=\left\{\begin{array}[]{ll}p_{i}(1-p_{i})(1-4p_{i}(1-p_{i} ))&\text{if $i=j$};\\ p_{i}p_{j}(p_{i}+p_{j}-4p_{i}p_{j})&\text{otherwise}.\end{array}\right.$ $$

$\forall{i,j}$, $\mathrm{Var}(R_{ij})\leq 1/16$.
For both diagonal and off-diagonal entries,
the coefficient of variation (CV) ${\mathrm{Std}(R_{ij})}/{|\mathcal{I}^{\Delta}_{ij}(z)|}$ can be arbitrarily large,
where $\mathrm{Std}(\cdot)$ means standard deviation.

Throughout our analysis, the *CV is a key indicator*
of the quality of a FIM estimator,
as a bounded CV for a random variable $X$ ensures
the random estimator’s probability mass within $[0,\alpha\mu]$,
where $\alpha>1$ and $\mu\geq 0$ is the mean of $X$.
If $\mathrm{CV}=\frac{\mathrm{Std}{X}}{\mu}\leq{K}$,
then by Cantelli inequality (one-sided Chebyshev), we have

$$ $\mathbb{P}(X\geq\alpha\mu)=\mathbb{P}(X\geq\mu+(\alpha-1)\mu)\leq\mathbb{P}(X \geq\mu+\frac{\alpha-1}{K}\mathrm{Std}{X})\leq\frac{1}{1+\left(\frac{\alpha-1} {K}\right)^{2}}.$ $$

By [lemma 5](https://arxiv.org/html/2505.13614v1#Thmtheorem5),
when using the rank-1 matrix $R(y)$ as an estimator of $\mathcal{I}^{\Delta}(z)$, the absolute error
is bounded, but the relative error given by the CV is unbounded. One may alternatively
use the rank-2 random matrix $R^{\prime}(y)=e_{yy}-pp^{\top}$ to estimate $\mathcal{I}^{\Delta}(z)$.
Obviously we have
$\mathop{\mathbb{E}}(R^{\prime}(y))=\mathrm{diag}\left(p\right)-pp^{T}=\mathcal
{I}^{\Delta}(z)$ and thus $R^{\prime}(y)$ is unbiased.
The variance appears only on the diagonal while all off-diagonal entries are deterministic with zero-variance.
This $R^{\prime}(y)$ is not used in our developments but is of theoretical interest.

In classification tasks with multiple binary labels,
we assume $p(y_{i}=1\,|\,x)=p_{i}$ ($i=1,\cdots,C$) and that all dimensions of $y$ are
conditional independent given $x$. All such distributions form a $C$-dimensional hypercube

$$ $\mathcal{C}^{C}(p)=\left\{(p_{1},\cdots,p_{C})\,:\,\forall{i},0\leq{}p_{i}\leq 1 \right\},$ $$

which is the product space of 1-dimensional simplexes.
Consider $p_{i}=\sigma(z_{i})\coloneqq 1/(1+\exp(-z_{i}))$ for $i=1,\cdots,C$.
In this case, the FIM is a diagonal matrix, given by

$$ $\mathcal{I}^{\mathcal{C}}(z)=\mathrm{diag}\left((p_{1}(1-p_{1}),\cdots,p_{C}(1 -p_{C}))\right)=\mathrm{diag}\left(\sigma^{\prime}(z_{i}),\cdots,\sigma^{ \prime}(z_{C})\right).$ (6) $$

In what follows, unless stated otherwise, our results pertain to the core $\Delta$ as it is more commonly used and has a more complex FIM as compared to $\mathcal{C}$.

###### Theorem 1 (Spectrum of Simplex FIM) .

###### Lemma 2 .

###### Lemma 3 .

###### Lemma 4 .

###### Lemma 5 .

## 3   FIM for Classifier Networks — Deterministic Analysis

### 3.1   Deterministic Lower and Upper Bounds

By [Eq. 4](https://arxiv.org/html/2505.13614v1#S2.E4), the neuromanifold FIM $\mathcal{F}(\theta)$ is determined by both the core space and the parameter-output Jacobian $\frac{\partial z}{\partial\theta}$.
Similar to [lemma 2](https://arxiv.org/html/2505.13614v1#Thmtheorem2), we can have lower and upper bounds of $\mathcal{F}^{\Delta}(\theta)$ in the space of psd matrices (although these bounds are not envelops as in [lemma 2](https://arxiv.org/html/2505.13614v1#Thmtheorem2)).

###### Proposition 6 .

If $p(y\,|\,x,\theta)\in\Delta^{C-1}$ is categorical, then $\forall\theta\in\Theta$, we have

$$ $\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z}{\partial\theta} \right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial\theta}\preceq \mathcal{F}^{\Delta}(\theta)\preceq\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p(y \,|\,x,\theta)\frac{\partial z_{y}}{\partial\theta}\left(\frac{\partial z_{y}} {\partial\theta}\right)^{\top},$ $$

where $\lambda_{C}\coloneqq\lambda_{C}(x,\theta)$ and
$v_{C}\coloneqq v_{C}(x,\theta)$ denote the largest eigenvalue and its associated eigenvector of $\mathcal{I}(z(x,\theta))$.

###### Remark .

The LHS is a sum of $|\mathcal{D}_{x}|$ (number of samples in $\mathcal{D}_{x}$) matrices of rank-1. Its rank is at most $|\mathcal{D}_{x}|$.
The RHS is a sum of $C|\mathcal{D}_{x}|$ matrices of rank-1 and potentially has a larger rank.

If $p(y\,|\,x)$ is in $\mathcal{C}$, then
$\mathcal{I}^{\mathcal{C}}(z(x,\theta))$ is diagonal as in [Eq. 6](https://arxiv.org/html/2505.13614v1#S2.E6). By [Eq. 4](https://arxiv.org/html/2505.13614v1#S2.E4), we have
$\mathcal{F}^{\mathcal{C}}(\theta)=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p_{y
}(1-p_{y})\frac{\partial z_{y}}{\partial\theta}\left(\frac{\partial z_{y}}{
\partial\theta}\right)^{\top}$,
which is similar to the upper bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6).
In summary, $\mathcal{F}(\theta)$ can be bounded or computed
using the Jacobian $\frac{\partial{z}}{\partial\theta}$
as well as the output probabilities $p(y\,|\,x,\theta)$.
The following analysis depends on the spectral properties of $\frac{\partial{z}}{\partial\theta}$.
Across our formal statements, we denote the singular values of $\frac{\partial{z}}{\partial\theta}$,
sorted in ascending order, as $\sigma_{1}(x,\theta)\leq\cdots\leq\sigma_{C}(x,\theta)$.
In [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6), by taking the trace on all sides, the trace of the FIM can be bounded from above and below.

###### Corollary 7 .

If $p(y\,|\,x,\theta)\in\Delta^{C-1}$ is categorical, then
it holds for all $\theta\in\Theta$ that

$$ $\sum_{x\in\mathcal{D}_{x}}\lambda_{C}(x,\theta)\sigma^{2}_{1}(x,\theta)\leq \sum_{x\in\mathcal{D}_{x}}\sum_{i=2}^{C}\lambda_{i}(x,\theta)\sigma^{2}_{C+1-i }(x,\theta)\leq\mathrm{tr}(\mathcal{F}^{\Delta}(\theta))\leq\sum_{x\in\mathcal {D}_{x}}\sum_{y=1}^{C}p(y\,|\,x,\theta)\left\|\frac{\partial z_{y}}{\partial \theta}\right\|^{2}.$ $$

These bounds are useful to get the overall scale of $\mathcal{F}^{\Delta}(\theta)$ without computing its exact value.
The proposition below gives the error of the upper bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6) in terms of Frobenius norm.

###### Proposition 8 .

We have $\forall\theta\in\Theta$ that

$$ $\sqrt{\sum_{x\in\mathcal{D}_{x}}\left\|\left(\frac{\partial z}{\partial\theta} \right)^{\top}p(x,\theta)\right\|^{4}}\leq\left\|\sum_{x\in\mathcal{D}_{x}} \sum_{y=1}^{C}p(y\,|\,x,\theta)\left(\frac{\partial z_{y}}{\partial\theta} \right)^{\top}\frac{\partial z_{y}}{\partial\theta}-\mathcal{F}^{\Delta}( \theta)\right\|\leq\sum_{x\in\mathcal{D}_{x}}\|p(x,\theta)\|^{2}\sigma_{C}^{2} (x,\theta),$ $$

where $p(x,\theta)=\operatorname*{\mathrm{SoftMax}}(z(x,\theta))$ denotes the output probability vector.

We use Frobenius norm for matrices but it is not difficult to bound the spectral norm using similar techniques.
By [proposition 8](https://arxiv.org/html/2505.13614v1#Thmtheorem8), the error of the upper bound scales with the 2-norm (maximum singular value) of the parameter-output Jacobian $\frac{\partial z}{\partial\theta}$.
Similar to what happens in the core space, using the upper bound of the FIM *always incurs an error*. For example, let $p$ tend to be one-hot, the LHS in [proposition 8](https://arxiv.org/html/2505.13614v1#Thmtheorem8) does not vanish but scales with certain rows of $\frac{\partial z}{\partial\theta}$ corresponding to the predicted $y$.
Naturally, we also want to examine the error of the lower bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6), as detailed below.

###### Proposition 9 .

We have $\forall\theta\in\Theta$ that

$$ $\left\|\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z}{\partial \theta}\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial\theta}- \mathcal{F}^{\Delta}(\theta)\right\|\leq\sum_{x\in\mathcal{D}_{x}}\sqrt{\sum_{ i=2}^{C-1}\sigma_{i+1}^{4}(x,\theta)p_{(i)}^{2}(x,\theta)}.$ $$

Clearly, as $p$ approaches a one-hot vector, all elements in the trimmed vector $p_{(i)}$,
for $i=2,\cdots,C-1$, tend to zero, and
the error approaches zero since its upper bound on the RHS goes to zero.
From this view, the lower bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6) is a better estimate as compared to the upper bound.

###### Remark .

By noting that $0\leq\sigma_{i}(x,\theta)\leq\sigma_{C}(x,\theta)$, we can relax the bound in [proposition 9](https://arxiv.org/html/2505.13614v1#Thmtheorem9) to be comparable to [proposition 8](https://arxiv.org/html/2505.13614v1#Thmtheorem8):

$$ $\left\|\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z}{\partial \theta}\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial\theta}- \mathcal{F}^{\Delta}(\theta)\right\|\leq\sum_{x\in\mathcal{D}_{x}}\sqrt{\sum_{ i=2}^{C-1}p_{(i)}^{2}(x,\theta)}\cdot\sigma_{C}^{2}(x,\theta).$ $$

The estimation error of
$\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z}{\partial\theta}
\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial\theta}$
is controlled by the norms of the Jacobian and the trimmed probabilities $(p_{(2)},\cdots,p_{(C-1)})$.
The latter is upper bounded by $p_{(C-1)}(x,\theta)$, the second largest probability of each sample $x$. By comparing with [proposition 8](https://arxiv.org/html/2505.13614v1#Thmtheorem8), one can easily observe that [proposition 9](https://arxiv.org/html/2505.13614v1#Thmtheorem9) is tighter in general.

###### Proposition 6 .

###### Remark .

###### Corollary 7 .

###### Proposition 8 .

###### Proposition 9 .

###### Remark .

### 3.2   Empirical FIM (eFIM)

Besides the proposed bounds, a commonly used deterministic approximation of the FIM is the eFIM.
By simple derivations, $\hat{\mathcal{F}}(\theta)$ defined in [section 1](https://arxiv.org/html/2505.13614v1#S1) can be written as

$$ $\hat{\mathcal{F}}(\theta)=\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{ \partial\theta}\right)^{\top}\cdot R(y)\cdot\frac{\partial z}{\partial\theta}.$ $$

Observe that it is similar to [Eq. 4](https://arxiv.org/html/2505.13614v1#S2.E4), except
$\mathcal{I}(z(x,\theta))$ is replaced by its empirical counterpart $R(y)$.
If the neural network output is in the simplex, the error of eFIM can be bounded, as stated below.

###### Proposition 10 .

$\forall{\theta}\in\Theta$, $\forall{y}$,
we have
$\|\mathcal{F}^{\Delta}(\theta)-\hat{\mathcal{F}}^{\Delta}(\theta)\|_{\sigma}
\leq\sum_{x\in\mathcal{D}_{x}}(1+\|p(x,\theta)\|^{2})\sigma_{C}^{2}(x,\theta)$.

Here we need to switch to the spectral norm $\|\cdot\|_{\sigma}$ to get a simple expression of the upper bound.
The approximation error in terms of the spectral norm is controlled by the spectral norm of the parameter-output Jacobian. The error by Frobenius norm is even larger.
The bound is loose as compared to [propositions 8](https://arxiv.org/html/2505.13614v1#Thmtheorem8) and [9](https://arxiv.org/html/2505.13614v1#Thmtheorem9).

We have found in [lemma 4](https://arxiv.org/html/2505.13614v1#Thmtheorem4) that using $R(y)$ to approximate
$\mathcal{I}^{\Delta}(z)$ suffers from a large error if $y$ is chosen in a tricky way.
The same principle applies to using $\hat{\mathcal{F}}(\theta)$ to approximate $\mathcal{F}(\theta)$.

###### Proposition 11 .

$\forall{\theta}\in\Theta$, $\forall{x}$, $\exists{y}$, such that

$$ $\left\|\left(\frac{\partial z}{\partial\theta}\right)^{\top}\mathcal{I}^{ \Delta}(z(x,\theta))\frac{\partial z}{\partial\theta}-\left(\frac{\partial z}{ \partial\theta}\right)^{\top}R(y)\frac{\partial z}{\partial\theta}\right\|_{ \sigma}\geq\sigma_{1}^{2}(x,\theta)\left|1+\|p(x,\theta)\|^{2}-\lambda_{C}(x, \theta)-2p_{(1)}(x,\theta)\right|.$ $$

In the above inequality, the LHS is the error of $\hat{\mathcal{F}}(\theta)$ for one single $x\in\mathcal{D}_{x}$. Therefore, when $y$ is set unfavorably,
the eFIM suffers from an approximation error that scales
with the smallest singular value of $\frac{\partial z}{\partial\theta}$.
Among all the investigated deterministic approximations, the lower bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6)
provides the smallest guaranteed error but is relatively expensive to compute.
We solve the computational issues in the next section.

###### Proposition 10 .

###### Proposition 11 .

## 4   Hutchinson’s Estimate of the FIM

### 4.1   From Monte Carlo To Hutchinson

From the definition of $\mathcal{F}(\theta)$ in [Eq. 1](https://arxiv.org/html/2505.13614v1#S1.E1),
it is natural to consider the Monte Carlo (MC) estimator

$$ $\hat{\mathcal{F}}(\theta)=\frac{1}{m}\sum_{\hat{x},\hat{y}}\frac{\partial\ell_ {\hat{x}\hat{y}}}{\partial\theta}\frac{\partial\ell_{\hat{x}\hat{y}}}{\partial \theta^{\top}},$ (7) $$

where $\hat{x}$, $\hat{y}$ are random samples drawn from
$\mathcal{D}_{x}$ and $p(y\,|\,{}\hat{x})$, respectively, $m$ is the number of $\hat{y}$ samples generated for each $\hat{x}$, and $\hat{\mathcal{F}}(\theta)$ is an abuse of notation and also denotes the eFIM (see [table 1](https://arxiv.org/html/2505.13614v1#S1.T1)).
The variance of $\hat{\mathcal{F}}(\theta)$ is analyzed for neural network models [7, 29, 30].

We show that the quality of the MC estimate can be arbitrarily bad.
Consider the single neuron model $z=\theta x$ for binary classification, where $z,\theta,x$ are all scalars, and $\theta$ is close to zero. Then $p\approx\frac{1}{2}$ is a fair Bernoulli distribution.
$\mathcal{I}(z)=p(1-p)\approx\frac{1}{4}$.
The Jacobian is simply
$\frac{\partial z}{\partial\theta}=x$.
and
$\mathcal{F}(\theta)=\mathop{\mathbb{E}}_{p(x)}\left[\frac{\partial z}{\partial
\theta}\mathcal{I}(z)\frac{\partial z}{\partial\theta}\right]\approx\frac{1}{4
}\mathop{\mathbb{E}}_{p(x)}[x^{2}]$.
A basic MC estimator takes the form
$\hat{\mathcal{F}}(\theta)=\frac{1}{4m}\sum_{i=1}^{m}x_{i}^{2}$,
where $x_{i}$’s are independently and identically distributed according to $p(x)$.
Its variance is
$\mathrm{Var}(\hat{\mathcal{F}})=\frac{1}{4m}[\mathop{\mathbb{E}}_{p(x)}(x^{4})
-\mathop{\mathbb{E}}^{2}_{p(x)}(x^{2})]$.
We let $p(x)$ be a heavy tailed distribution,
*e.g*. Student’s t-distribution with $\nu>4$ degrees of freedom,
so that
$\mathrm{Var}(\hat{\mathcal{F}})$ is large while $\mathcal{F}(\theta)$ is small.
Then $\mathop{\mathbb{E}}_{p(x)}(x^{2})=\frac{\nu}{\nu-2}$
and $\mathop{\mathbb{E}}_{p(x)}(x^{4})=\frac{3\nu^{2}}{(\nu-2)(\nu-4)}$.
The ratio
$\frac{\mathop{\mathbb{E}}_{p(x)}(x^{4})}{(\mathop{\mathbb{E}}_{p(x)}x^{2})^{2}
}=\frac{3(\nu-2)}{\nu-4}$ can be arbitrarily large when $\nu\to 4^{+}$.
Therefore the CV $\mathrm{Std}(\hat{\mathcal{F}})/F(\theta)$ is unbounded.
The general case is more complicated, but follows a similar idea.
The variance of MC estimators depends on the 4th moment of the Jacobian
$\frac{\partial z}{\partial\theta}$ w.r.t. $p(x)$
while the mean value $\mathcal{F}(\theta)$ only depends on the 2nd moment of $\frac{\partial z}{\partial\theta}$.
The ratio of the variance and $\mathcal{F}^{2}(\theta)$, or the CV $\mathrm{Std}(\hat{\mathcal{F}})/\mathcal{F}(\theta)$, is unbounded without further assumption on $p(x)$.
One can increase the number of samples $m$ to reduce variance. However, this is computationally expensive especially in online settings.

In contrast, Hutchinson’s estimate in [section 1](https://arxiv.org/html/2505.13614v1#S1) has guaranteed quality, which is
formally established below.

###### Proposition 12 .

$\mathop{\mathbb{E}}_{p(\xi)}\left(\mathbb{F}(\theta)\right)=\mathcal{F}(\theta)$.
If $p(\xi)$ is standard multivariate Gaussian,
then $\mathrm{Var}(\mathbb{F}_{ii}(\theta))=2\mathcal{F}_{ii}(\theta)^{2}$;
if $p(\xi)$ is standard multivariate Rademacher,
$\mathrm{Var}(\mathbb{F}_{ii}(\theta))=2\mathcal{F}_{ii}(\theta)^{2}-2\sum_{x
\in\mathcal{D}_{x}}\sum_{y=1}^{C}p^{2}(y\,|\,x)(\frac{\partial\ell_{xy}}{
\partial\theta_{i}})^{4}$.

It is know that Rademacher distribution yields smaller variance
for Hutchinson’s estimator compared to the Gaussian distribution.
In what follows, $p(\xi)$ is Rademacher by default.
By [proposition 12](https://arxiv.org/html/2505.13614v1#Thmtheorem12), $\mathrm{Std}(\mathbb{F}_{ii}(\theta))\leq\sqrt{2}\mathcal{F}_{ii}(\theta)$. Thus the CV
$\mathrm{Std}(\mathbb{F}_{ii}(\theta))/\mathcal{F}_{ii}(\theta)$ is bounded by $\sqrt{2}$.
We only investigate the diagonal of Hutchinson’s estimate because the diagonal FIM is widely used, but our results can be readily extended to off-diagonal entries.

###### Remark .

Taking trace on both sides of $\mathop{\mathbb{E}}_{p(\xi)}\left(\mathbb{F}(\theta)\right)=\mathcal{F}(\theta)$, we get
$\mathop{\mathbb{E}}_{p(\xi)}(\|\frac{\partial\mathfrak{h}}{\partial\theta}\|^{
2})=\mathrm{tr}(\mathcal{F}(\theta))$.
The squared Euclidean-norm of
$\frac{\partial\mathfrak{h}}{\partial\theta}$
is an unbiased estimate of the trace of the FIM.
This is useful for computing related regularizers [25].

In theory, one needs to compute the numerical average of more than one $\mathbb{F}(\theta)$ samples to reduce variance
and have sufficient rank.
Due to computational constraints in deep learning practice, much fewer (*e.g*. 1) samples are used to estimate $\mathcal{F}(\theta)$.
Instead, accumulated statistics on the learning path $\theta_{1}\to\theta_{2}\to\cdots$
can be used to compute $\mathcal{F}(\theta_{t})$ at each training step $t$.

###### Proposition 12 .

###### Remark .

### 4.2   Diagonal Core

For multi-label classification, and for computing the upper bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6), the core matrix is diagonal, in the form $\mathcal{I}^{\mathrm{DG}}(z(x,\theta))=\mathrm{diag}\left(\zeta_{1}(x,\theta),
\cdots,\zeta_{C}(x,\theta)\right)$,
and the associated FIM is
$\mathcal{F}^{\mathrm{DG}}(\theta)=\sum_{x\in\mathcal{D}_{x}}\left(\frac{
\partial z}{\partial\theta}\right)^{\top}\cdot\mathcal{I}^{\mathrm{DG}}(z(x,
\theta))\cdot\frac{\partial z}{\partial\theta}$.
In the former case, $\zeta_{y}(x,\theta)=p(y\,|\,x,\theta)(1-p(y\,|\,x,\theta))$;
in the latter case, $\zeta_{y}(x,\theta)=p(y\,|\,x,\theta)$.
Here, the tensor superscript — *e.g*., “$\mathrm{DG}$” for diagonal or “$\mathrm{LR}$” for low-rank — indicates the parametric form of the core FIM, in contrast to denoting the core space as in $\mathcal{I}^{\Delta}$.
We define the scalar valued function

$$ $\mathfrak{h}^{\mathrm{DG}}(\theta)\coloneqq\sum_{x\in\mathcal{D}_{x}}\sum_{y=1 }^{C}\sqrt{\tilde{\zeta}_{y}(x,\theta)}z_{y}(x,\theta)\xi_{xy},$ (8) $$

where $\xi_{xy}$ are standard Rademacher samples that are independent across all $x$ and $y$.
Similar to the derivation steps in [section 1](https://arxiv.org/html/2505.13614v1#S1),
we first compute the random vector
$\frac{\partial\mathfrak{h}^{\mathrm{DG}}}{\partial\theta}$ through AD,
and then compute
$\mathbb{F}^{\mathrm{DG}}(\theta)\coloneqq\frac{\partial\mathfrak{h}^{\mathrm{
DG}}}{\partial\theta}\frac{\partial\mathfrak{h}^{\mathrm{DG}}}{\partial\theta^
{\top}}$ (or its diagonal blocks) to estimate $\mathcal{F}^{\mathrm{DG}}(\theta)$.

###### Proposition 13 .

The random matrix $\mathbb{F}^{\mathrm{DG}}(\theta)$ is an unbiased estimator of $\mathcal{F}^{\mathrm{DG}}(\theta)$.
The variance of its diagonal elements is
$\mathrm{Var}\left(\mathbb{F}^{\mathrm{DG}}_{ii}(\theta)\right)=2(\mathcal{F}_{
ii}^{\mathrm{DG}}(\theta))^{2}-2\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\zeta_
{y}^{2}(x,\theta)(\frac{\partial z_{y}}{\partial\theta_{i}})^{4}$.

For computing the upper bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6),
$\tilde{\zeta}_{y}(x,\theta)=\tilde{p}_{y}(x,\theta)$, then we find that [Eq. 2](https://arxiv.org/html/2505.13614v1#S1.E2) and [Eq. 8](https://arxiv.org/html/2505.13614v1#S4.E8) are similar.
The only difference is that,
the “raw” logits $z_{y}$ in [Eq. 8](https://arxiv.org/html/2505.13614v1#S4.E8) is replaced by
$\ell_{xy}(\theta)=z_{y}-\log\sum_{y}\exp(z_{y})$ in [Eq. 2](https://arxiv.org/html/2505.13614v1#S1.E2).
Compared to $\frac{\partial z}{\partial\theta}$,
the gradient $\frac{\partial\ell_{xy}}{\partial\theta}=\frac{\partial z_{y}}{\partial\theta}
-\sum_{y}p(y\,|\,x,\theta)\frac{\partial z_{y}}{\partial\theta}$ is centered.
Due to their computational similarity, in practice, one should use [Eq. 2](https://arxiv.org/html/2505.13614v1#S1.E2) instead of [Eq. 8](https://arxiv.org/html/2505.13614v1#S4.E8) and get an unbiased estimate of $\mathcal{F}^{\Delta}(\theta)$.
[Eq. 8](https://arxiv.org/html/2505.13614v1#S4.E8) is useful when the dimensions of $y$ are conditional independent given $x$, *e.g*. for computing $\mathcal{F}^{\mathcal{C}}(\theta)$.

###### Proposition 13 .

### 4.3   Low Rank Core

By [lemma 2](https://arxiv.org/html/2505.13614v1#Thmtheorem2), the FIM of the core space $\Delta$ has a rank-1 lower-bound
$\mathcal{I}^{\Delta}(z)\succeq\mathcal{I}^{\mathrm{LR}}(z)\coloneqq\lambda_{C}
v_{C}v_{C}^{\top}$.
By [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6),
$\mathcal{F}^{\Delta}(\theta)\succeq\mathcal{F}^{\mathrm{LR}}(\theta)\coloneqq
\sum_{x\in\mathcal{D}_{x}}\lambda_{C}(x,\theta)\left(\frac{\partial z}{
\partial\theta}\right)^{\top}v_{C}(x,\theta)v_{C}^{\top}(x,\theta)\frac{
\partial z}{\partial\theta}$.
We define

$$ $\mathfrak{h}^{\mathrm{LR}}(\theta)=\sum_{x\in\mathcal{D}_{x}}\sqrt{\tilde{ \lambda}_{C}(x,\theta)}\tilde{v}_{C}^{\top}(x,\theta)z(x,\theta)\xi_{x},$ (9) $$

where $\xi_{x}$ are independent standard Rademacher samples. For computing $\mathfrak{h}^{\mathrm{LR}}(\theta)$, we only need $|\mathcal{D}_{x}|$ Rademacher samples, as compared to
$C|\mathcal{D}_{x}|$ samples for computing $\mathfrak{h}(\theta)$ and $\mathfrak{h}^{\mathrm{DG}}(\theta)$.
Correspondingly,
$\mathbb{F}^{\mathrm{LR}}(\theta)\coloneqq\frac{\partial\mathfrak{h}^{\mathrm{
LR}}}{\partial\theta}\frac{\partial\mathfrak{h}^{\mathrm{LR}}}{\partial\theta^
{\top}}$ is used to estimate $\mathcal{F}^{\mathrm{LR}}(\theta)$.
Note that $\mathfrak{h}$, $\mathfrak{h}^{\mathrm{DG}}$ and $\mathfrak{h}^{\mathrm{LR}}$ can be computed solely based on the neural network output logits $z(x,\theta)$ for each $x\in\mathcal{D}_{x}$.

###### Proposition 14 .

$\mathbb{F}^{\mathrm{LR}}(\theta)$ is an unbiased estimate of $\mathcal{F}^{\mathrm{LR}}(\theta)$;
the variance of its diagonal elements is
$\mathrm{Var}\left(\mathbb{F}^{\mathrm{LR}}_{ii}(\theta)\right)=2(\mathcal{F}_{
ii}^{\mathrm{LR}}(\theta))^{2}-2\sum_{x\in\mathcal{D}_{x}}\lambda_{C}^{2}(x,
\theta)\left(v_{C}^{\top}(x,\theta)\frac{\partial z}{\partial\theta_{i}}\right
)^{4}$.

We have $\mathrm{Std}(\mathbb{F}^{\mathrm{DG}}_{ii}(\theta))/\mathcal{F}_{ii}^{\mathrm{
DG}}(\theta)\leq\sqrt{2}$ by [proposition 13](https://arxiv.org/html/2505.13614v1#Thmtheorem13),
and at the same time,
we have $\mathrm{Std}(\mathbb{F}^{\mathrm{LR}}_{ii}(\theta))/\mathcal{F}_{ii}^{\mathrm{
LR}}(\theta)\leq\sqrt{2}$ by [proposition 14](https://arxiv.org/html/2505.13614v1#Thmtheorem14).
Their estimation quality is guaranteed.

We remain to solve $\lambda_{C}(x,\theta)$ and $v_{C}(x,\theta)$ for each $x\in\mathcal{D}_{x}$.
They can be conveniently computed based on the power iteration.
By [Eq. 5](https://arxiv.org/html/2505.13614v1#S2.E5),
starting from a random unit vector $v_{C}^{0}$,
we compute

$$ $v_{C}^{t+1}=\frac{\mathcal{I}^{\Delta}(z)v_{C}^{t}}{\|\mathcal{I}^{\Delta}(z)v _{C}^{t}\|}=\frac{p\circ v_{C}^{t}-p^{\top}v_{C}^{t}p}{\|p\circ v_{C}^{t}-p^{ \top}v_{C}^{t}p\|},$ $$

for $t=1,2,\cdots$, until convergence or until a fixed number of iterations is reached.
Then,

$$ $\lambda_{C}=p^{\top}(v_{C}\circ v_{C})-(p^{\top}v_{C})^{2}.$ $$

The overall computational complexity to compute $\lambda_{C}$ and $v_{C}$
for all $x\in\mathcal{D}_{x}$ is $\mathcal{O}(MC|\mathcal{D}_{x}|)$,
where $M$ (*e.g*. $M=10$) is the maximum number of iteration steps.

###### Proposition 14 .

### 4.4   Numerical Simulations

To provide intuition, we compute the diagonal FIM of DistilBERT [27],
pretrained by Hugging Face [36] (^2^22Available as distilbert-base-uncased in the Hugging Face library.)
combined with a randomly initialized classification head (two dense layers) for AG News [39] topic classification ($C=4$ classes).
Another representative case is provided in [appendix \AlphAlph](https://arxiv.org/html/2505.13614v1#A15), where DistilBERT is
fine-tuned on the Stanford Sentiment Treebank v2 (SST-2) [28], and the FIM is
computed in regions of $\Theta$ corresponding to a more confident model.
[Figure 1](https://arxiv.org/html/2505.13614v1#S4.F1) shows the normalized density plots of $\mathbb{F}^{\mathrm{DG}}_{ii}(\theta)$
(Hutchinson’s estimate of the upper bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6)),
$\mathbb{F}_{ii}(\theta)$ (Hutchinson’s unbiased estimate),
$\mathbb{F}^{\mathrm{LR}}_{ii}(\theta)$ (Hutchinson’s estiamte of the lower bound in [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6)),
and the empirical FIM $\hat{\mathcal{F}}_{ii}(\theta)$.
All estimators use the first 128 data samples to compute the FIM.
All Hutchinson estimators use $10$ samples for variance reduction.
Due to the pathological structure [12] of the FIM, all densities
exhibit a spike near zero and become sparse on large Fisher information values.
For example, all layers have more than $20\%$ of their parameters with $\mathbb{F}_{ii}<10^{-5}$.
The visualization is smoothed out on a logarithmic y-axis.
The mean values of these densities are reflected on the low-right corner of the subplots (up to a scaling factor).
Across the layers, the classification head has the largest scale of Fisher information and the embedding layer has the lowest scale.
In general, the deeper layers (close to the input) have smaller values of $\mathbb{F}_{ii}$.
The scale of $\mathbb{F}^{\mathrm{DG}}_{ii}$ appears larger than $\mathbb{F}_{ii}$,
which in turn is larger than $\mathbb{F}^{\mathrm{LR}}_{ii}$. This makes sense as
the expected values of $\mathbb{F}^{\mathrm{DG}}_{ii}$ and $\mathbb{F}^{\mathrm{LR}}_{ii}$ are upper and lower bounds of the expected values of $\mathbb{F}_{ii}$, respectively.
The scale of $\hat{\mathcal{F}}_{ii}$ is not informative as the others regarding $\mathcal{F}_{ii}$ because it is biased.
The classification head is not trained and hence has large gradient values, leading to large values of $\hat{\mathcal{F}}_{ii}$.

Figure: Figure 1: Density plots (based on kernel density estimation with a small bandwidth) of diagonal FIM elements based on different approximations (rows) across different layers (columns) of DistilBERT on the AG News dataset. The four rows, from top to bottom, represent Hutchinson’s estimates $\mathbb{F}^{\mathrm{DG}}(\theta)$, $\mathbb{F}(\theta)$, $\mathbb{F}^{\mathrm{LR}}(\theta)$, and the eFIM $\hat{\mathcal{F}}(\theta)$. The columns are arranged from layers close to the input (left) to those near the output (right). In each subplot, the maximum value of the x-axis (number on the bottom right corner) shows the mean value of the FIM multiplied by 2,000. The y-axis means probability density in log-scale.
Refer to caption: x1.png

## 5   Related Work

A prominent application of Fisher information in deep learning
is the natural gradient [1] and its variants.
The Adam optimizer [13] uses the empirical diagonal FIM.
Efforts have been made to obtain more accurate approximations
of $\mathcal{F}(\theta)$ at the expense of higher computational cost,
such as modeling the diagonal blocks of $\mathcal{F}(\theta)$ with Kronecker product [16]
of component-wise FIM [22, 33],
or computing $\mathcal{F}(\theta)$ through low rank
approximations [15, 3].
The FIM can be alternatively defined on a sub-model [33] instead of the
global mapping $x\to{y}$ or based on $\alpha$-embeddings of a parametric
family [20].
AdaHessian [38] uses Hutchinson probes to approximate the diagonal Hessian.

From theoretical perspectives,
the quality of Kronecker approximation is discussed [18] with its error bounded.
It is well known that the eFIM differs from $\mathcal{F}(\theta)$ [23, 16, 14]
and leads to distinct optimization paths.
The accuracy of two different MC approximations of $\mathcal{F}(\theta)$
is analyzed [7, 29, 30, 35],
which lie in the framework of MC information geometry [21].
By our analysis, the Hutchinson’s estimate $\mathbb{F}(\theta)$
has unique advantages over both MC and the eFIM.
Notably, the MC estimate in [section 4.1](https://arxiv.org/html/2505.13614v1#S4.SS1) needs to compute
$\frac{\partial\ell_{\hat{x}\hat{y}}}{\partial\theta}$ for
each $x\in\mathcal{D}_{x}$, while $\mathbb{F}(\theta)$ only needs to
evaluate one gradient vector $\frac{\partial\mathfrak{h}}{\partial\theta}$.
Our bounds improves over existing bounds, *e.g*. those of $\mathcal{F}(\theta)$ [30],
through carefully analyzing the core space.

The Hutchinson’s stochastic trace estimator is used to estimate the trace of the FIM [11],
or the FIM for Gaussian processes [31, 6] where the FIM entries are in the form of a trace.
Closely related to this is computations around the Hessian,
where Hutchinson’s trick is applied to compute the Hessian trace [9],
or the principal curvature [4],
or related regularizers [25].
The Hessian trace estimator is implemented
in deep learning libraries [5, 37]
and usually relies on the Hessian-vector product.
As a natural yet important next step, our estimators leverage
both Hutchinson’s trick and AD’s interfaces (detach(), etc.),
avoid the need for expensive Hessian computations/approximations,
and are well-suited in scalable settings.
In [Eq. 3](https://arxiv.org/html/2505.13614v1#S1.E3), we perform a double contraction of a high dimensional
tensor indexed by $x$, $y$, $x^{\prime}$, $y^{\prime}$, $i$ and $j$
($i$ and $j$ are indices of the FIM)
and thereby obtain an unbiased estimator of the full metric tensor $\mathcal{F}(\theta)$
including its substructures and trace.
Our estimator can be applied to different classification networks regardless of
the network architecture.

## 6   Conclusion

We explore the FIM $\mathcal{F}(\theta)$ of classifier networks, focusing on the case of multi-class classification.
We provide deterministic lower and upper bounds of the FIM based on related bounds in the low dimensional core space.
We discover a new family of random estimators $\mathbb{F}(\theta)$ based on Hutchinson’s trace estimator.
Their estimate has guaranteed quality with bounded variance and can be computed efficiently through auto-differentiation.
We analyze the metric tensor of the statistical simplex, which is useful in related theory and applications.
Advanced variance reduction techniques [19] that could improve our proposed random estimator $\mathbb{F}(\theta)$
remain to be investigated.
More thorough numerical experiments and related methodologies on real applications,
*e.g*. new deep learning optimizer, are not developed here and left as meaningful future work.

## References

- [1]
Shun-ichi Amari.
Natural gradient works efficiently in learning.
Neural Comput., 10(2):251–276, 1998.
- [2]
Shun-ichi Amari.
Information Geometry and Its Applications, volume 194 of Applied Mathematical Sciences.
Springer-Verlag, Berlin, 2016.
- [3]
Aleksandar Botev, Hippolyt Ritter, and David Barber.
Practical Gauss-Newton optimisation for deep learning.
In Doina Precup and Yee Whye Teh, editors, Proceedings of the
34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 557–565. PMLR, 2017.
- [4]
Lucas Böttcher and Gregory Wheeler.
Visualizing high-dimensional loss landscapes with Hessian
directions.
Journal of Statistical Mechanics: Theory and Experiment,
2024(2):023401, 2024.
- [5]
Felix Dangel, Frederik Kunstner, and Philipp Hennig.
BackPACK: Packing more into backprop.
In International Conference on Learning Representations (ICLR),
2020.
- [6]
Christopher J. Geoga, Mihai Anitescu, and Michael L. Stein.
Scalable gaussian process computations using hierarchical matrices.
Journal of Computational and Graphical Statistics,
29(2):227–237, 2020.
- [7]
Shenghan Guo and James C. Spall.
Relative accuracy of two methods for approximating observed Fisher
information.
In Data-Driven Modeling, Filtering and Control: Methods and
applications, pages 189–211. IET Press, London, 2019.
- [8]
Harold Hotelling.
Spaces of statistical parameters.
American Mathematical Society Meeting, 1929.
(unpublished. Presented orally by O. Ore during the meeting).
- [9]
Zheyuan Hu, Zekun Shi, George Em Karniadakis, and Kenji Kawaguchi.
Hutchinson trace estimation for high-dimensional and high-order
physics-informed neural networks.
Computer Methods in Applied Mechanics and Engineering,
424:116883, 2024.
- [10]
M.F. Hutchinson.
A stochastic estimator of the trace of the influence matrix for
Laplacian smoothing splines.
Communications in Statistics - Simulation and Computation,
19(2):433–450, 1990.
- [11]
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg, Huan
Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J Geras.
Catastrophic Fisher explosion: Early phase Fisher matrix impacts
generalization.
In Marina Meila and Tong Zhang, editors, Proceedings of the 38th
International Conference on Machine Learning, volume 139 of Proceedings
of Machine Learning Research, pages 4772–4784. PMLR, 18–24 Jul 2021.
- [12]
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari.
Pathological spectra of the Fisher information metric and its
variants in deep neural networks.
Neural Computation, 33(8):2274–2307, 2021.
- [13]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In International Conference on Learning Representations (ICLR),
2015.
- [14]
Frederik Kunstner, Lukas Balles, and Philipp Hennig.
Limitations of the empirical Fisher approximation for natural
gradient descent.
In Advances in Neural Information Processing Systems (NeurIPS),
pages 4133–4144. Curran Associates, Inc., 2020.
- [15]
Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio.
Topmoumoute online natural gradient algorithm.
In Advances in neural information processing systems,
volume 20, pages 849–856. Curran Associates, Inc., 2007.
- [16]
James Martens.
New insights and perspectives on the natural gradient method.
Journal of Machine Learning Research, 21(146):1–76, 2020.
- [17]
James Martens et al.
Deep learning via Hessian-free optimization.
In International Conference on Machine Learning (ICML),
volume 27, pages 735–742, 2010.
- [18]
James Martens and Roger Grosse.
Optimizing neural networks with Kronecker-factored approximate
curvature.
In International Conference on Machine Learning (ICML), pages
2408–2417. PMLR, 2015.
- [19]
Raphael A. Meyer, Cameron Musco, Christopher Musco, and David P. Woodruff.
Hutch++: Optimal stochastic trace estimation.
In Proceedings of the 4th Symposium on Simplicity in Algorithms
(SOSA), pages 142–155, 2021.
- [20]
Frank Nielsen.
The $\alpha$-representations of the Fisher information matrix,
2017.
[https://franknielsen.github.io/blog/alpha-FIM/index.html](https://franknielsen.github.io/blog/alpha-FIM/index.html).
- [21]
Frank Nielsen and Gaëtan Hadjeres.
Monte Carlo information-geometric structures.
In Frank Nielsen, editor, Geometric Structures of Information,
pages 69–103. Springer International Publishing, Cham, 2019.
- [22]
Yann Ollivier.
Riemannian metrics for neural networks I: feedforward networks.
Information and Inference: A Journal of the IMA, 4(2):108–153,
2015.
- [23]
Razvan Pascanu and Yoshua Bengio.
Revisiting natural gradient for deep networks.
In International Conference on Learning Representations, 2014.
- [24]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban
Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan
Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith
Chintala.
PyTorch: An imperative style, high-performance deep learning
library.
In Advances in Neural Information Processing Systems, pages
8024–8035. Curran Associates, Inc., 2019.
[https://pytorch.org/](https://pytorch.org/).
- [25]
William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio
Torralba.
The Hessian penalty: A weak prior for unsupervised disentanglement.
In Proceedings of (ECCV) European Conference on Computer
Vision, pages 581 – 597, August 2020.
- [26]
C. R. Rao.
Information and the accuracy attainable in the estimation of
statistical parameters.
Bulletin of the Calcutta Mathematical Society, 37(3):81–91,
1945.
- [27]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper
and lighter.
CoRR, abs/1910.01108, 2019.
[http://arxiv.org/abs/1910.01108](http://arxiv.org/abs/1910.01108).
- [28]
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning,
Andrew Y. Ng, and Christopher Potts.
Recursive deep models for semantic compositionality over a sentiment
treebank.
In Empirical Methods in Natural Language Processing (EMNLP),
pages 1631–1642, 2013.
- [29]
Alexander Soen and Ke Sun.
On the variance of the Fisher information for deep learning.
In Advances in Neural Information Processing Systems (NeurIPS),
volume 34, pages 5708–5719. Curran Associates, Inc., 2021.
- [30]
Alexander Soen and Ke Sun.
Trade-Offs of diagonal Fisher information matrix estimators.
In Advances in Neural Information Processing Systems (NeurIPS),
volume 37, pages 5870–5912. Curran Associates, Inc., 2024.
- [31]
Michael L. Stein, Jie Chen, and Mihai Anitescu.
Stochastic approximation of score functions for Gaussian processes.
The Annals of Applied Statistics, 7(2):1162 – 1191, 2013.
- [32]
Ke Sun.
Information geometry for data geometry through pullbacks.
In Deep Learning through Information Geometry (Workshop at
NeurIPS 2020), 2020.
- [33]
Ke Sun and Frank Nielsen.
Relative Fisher information and natural gradient for learning large
modular models.
In International Conference on Machine Learning (ICML),
volume 70 of Proceedings of Machine Learning Research, pages
3289–3298. PMLR, 2017.
- [34]
Ke Sun and Frank Nielsen.
A geometric modeling of Occam’s razor in deep learning.
arXiv preprint arXiv:1905.11027, 2019.
[https://arxiv.org/abs/1905.11027](https://arxiv.org/abs/1905.11027). Formerly titled “Lightlike
neuromanifolds, Occam’s razor and deep learning”.
- [35]
Shiqing Sun and James C. Spall.
Connection of diagonal Hessian estimates to natural gradients in
stochastic optimization.
In Proceedings of the 55th Annual Conference on Information
Sciences and Systems (CISS), 2021.
- [36]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander Rush.
Transformers: State-of-the-art natural language processing.
In Qun Liu and David Schlangen, editors, Empirical Methods in
Natural Language Processing (EMNLP): System Demonstrations, pages 38–45.
Association for Computational Linguistics, 2020.
[https://huggingface.co](https://huggingface.co).
- [37]
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney.
PyHessian: Neural networks through the lens of the Hessian.
In IEEE international conference on big data (Big Data),
pages 581–590. IEEE, IEEE Computer Society, 2020.
- [38]
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and
Michael Mahoney.
AdaHessian: An adaptive second order optimizer for machine
learning.
In Proceedings of the AAAI conference on artificial
intelligence, volume 35, pages 10665–10673, 2021.
- [39]
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun.
Character-level convolutional networks for text classification.
CoRR, abs/1509.01626, 2015.

## Appendix \AlphAlph Proof of Theorem 1

###### Proof.

We already know the closed form FIM

$$ $\mathcal{I}^{\Delta}(z)=\mathrm{diag}\left(p\right)-pp^{\top}.$ $$

Therefore

$$ $\mathcal{I}^{\Delta}(z)e=(\mathrm{diag}\left(p\right)-pp^{\top})e=p-\left(\sum _{i=1}^{C}p_{i}\right)p=p-p=0.$ $$

Therefore $te$, $t\in\Re$ is a one-dimensional kernel of $\mathcal{I}^{\Delta}(z)$. Since $\mathcal{I}^{\Delta}(z)\succeq 0$, we must have $\lambda_{1}=0$, and $v_{1}=e/\|e\|$.

To show the sum of the eigenvalues of $\mathcal{I}^{\Delta}(z)$, we have

$$ $\sum_{i=1}^{C}\lambda_{i}=\mathrm{tr}(\mathcal{I}^{\Delta}(z))=\mathrm{tr}( \mathrm{diag}\left(p\right))-\mathrm{tr}(pp^{\top})=1-\mathrm{tr}(p^{\top}p)=1 -p^{\top}p=1-\|p\|^{2}.$ $$

In below, we consider the maximum eigenvalue $\lambda_{C}$.
We know that

$$ $\lambda_{C}=\sup_{\|u\|=1}u^{\top}\mathcal{I}^{\Delta}(z)u.$ $$

Therefore

$$ $\forall{i},\quad\lambda_{c}\geq e_{i}\mathcal{I}^{\Delta}(z)e_{i}=\mathcal{I}^ {\Delta}_{ii}(z)=p_{i}(1-p_{i}).$ $$

Therefore $\lambda_{C}\geq\max_{i}p_{i}(1-p_{i})$.
At the same time, because $\lambda_{1}=0$, we have

$$ $\sum_{i=1}^{C}\lambda_{i}=\lambda_{2}+\lambda_{3}+\cdots+\lambda_{C}\leq(C-1) \lambda_{C}.$ $$

Therefore

$$ $\lambda_{C}\geq\frac{\sum_{i=1}^{C}\lambda_{i}}{C-1}=\frac{1-\|p\|^{2}}{C-1}.$ $$

Because

$$ $\mathrm{diag}\left(p\right)=\mathcal{I}^{\Delta}(z)+pp^{\top}.$ $$

By the Cauchy’s interlacing theorem, we have

$$ $\lambda_{C-1}\leq p_{(C-1)}\leq\lambda_{C}\leq p_{(C)}.$ $$

It remains to prove the upper bounds of $\lambda_{C}$.
First, we have

$$ $\displaystyle\lambda_{C}$ $\displaystyle=\sup_{\|u\|=1}u^{\top}\mathcal{I}^{\Delta}(z)u.=\sup_{\|u\|=1} \left(\sum_{i=1}^{C}p_{i}u_{i}^{2}-(p^{\top}u)^{2}\right)$ $\displaystyle\leq\sup_{\|u\|=1}\sum_{i=1}^{C}p_{i}u_{i}^{2}=\max_{i}p_{i}=p_{( C)},$ $$

which has just been proved using Cauchy’s interlacing theorem.

By the Gershgorin circle theorem,
$\lambda_{C}$ must lie in one of the Gershgorin discs,
given by the closed intervals

$$ $\left[p_{i}(1-p_{i})-\sum_{j\neq{i}}p_{i}p_{j},\;p_{i}(1-p_{i})+\sum_{j\neq{i} }p_{i}p_{j}\right],\quad i=1,\cdots,C.$ $$

Therefore

$$ $\displaystyle\lambda_{C}$ $\displaystyle\leq\max_{i}\left(p_{i}(1-p_{i})+\sum_{j\neq{i}}p_{i}p_{j}\right)$ $\displaystyle=\max_{i}\left(p_{i}(1-p_{i})+p_{i}(1-p_{i})\right)=2\max_{i}p_{i }(1-p_{i}).$ $$

Because $\mathcal{I}^{\Delta}(z)\succeq 0$,

$$ $\lambda_{C}\leq\sum_{i=1}^{C}\lambda_{i}=1-\|p\|^{2}.$ $$

The statement follows immediately by combining the above lower and upper bounds of $\lambda_{C}$.
∎

###### Proof.

## Appendix \AlphAlph Proof of Lemma 2

###### Proof.

Because $\mathcal{I}^{\Delta}(z)\succeq 0$. All its eigenvalues are greater or equal to 0.
We have

$$ $\mathcal{I}^{\Delta}(z)-\lambda_{C}v_{C}v_{C}^{\top}=\sum_{i=1}^{C-1}\lambda_{ i}v_{i}v_{i}^{\top}\succeq 0.$ $$

To show that $\lambda_{C}v_{C}v_{C}^{\top}$ is the best rank-1 representation. Assume that
$\exists u\neq 0$, such that $\mathcal{I}^{\Delta}(z)\succeq uu^{\top}\succeq\lambda_{C}v_{C}v_{C}^{\top}$.
Then

$$ $v_{C}^{\top}\mathcal{I}^{\Delta}(z)v_{C}=\lambda_{C}\geq(v_{C}^{\top}u)^{2} \geq\lambda_{C}.$ $$

Therefore

$$ $v_{C}^{\top}u=\pm\sqrt{\lambda_{C}}.$ $$

Assume that
$u=\sum_{i=1}^{C}\alpha_{i}v_{i}$, then
$\alpha_{C}=v_{C}^{\top}u=\pm\sqrt{\lambda_{C}}$.
Moreover, we have

$$ $\lambda_{C}\geq\frac{u^{\top}}{\|u\|}\mathcal{I}^{\Delta}(z)\frac{u}{\|u\|} \geq\frac{u^{\top}}{\|u\|}uu^{\top}\frac{u}{\|u\|}=\|u\|^{2}=\sum_{i=1}^{C} \alpha_{i}^{2}.$ $$

Therefore $\forall{i}\neq{C}$, $\alpha_{i}=0$.
In summary, $u=\pm\sqrt{\lambda_{C}}v_{C}$.
Hence, $uu^{\top}=\lambda_{C}v_{C}v_{C}^{\top}$.

We have

$$ $\mathrm{diag}\left(p\right)-\mathcal{I}^{\Delta}(z)=\mathrm{diag}\left(p\right )-(\mathrm{diag}\left(p\right)-pp^{\top})=pp^{\top}\succeq 0.$ $$

Therefore $\mathrm{diag}\left(p\right)\succeq\mathcal{I}^{\Delta}(z)$. Assume that $\mathrm{diag}\left(q\right)$ satisfies

$$ $\mathcal{I}^{\Delta}(z)\preceq\mathrm{diag}\left(q\right)\preceq\mathrm{diag} \left(p\right).$ $$

Then

$$ $\mathrm{diag}\left(p\right)-\mathcal{I}^{\Delta}(z)=pp^{\top}\succeq\mathrm{ diag}\left(q\right)-\mathcal{I}^{\Delta}(z)\succeq 0.$ $$

Therefore

$$ $\mathrm{diag}\left(q\right)-\mathcal{I}^{\Delta}(z)=\beta pp^{\top}(\beta\leq 1).$ $$

Consequently,

$$ $\mathrm{diag}\left(q\right)=\mathcal{I}^{\Delta}(z)+\beta pp^{\top}=\mathrm{ diag}\left(p\right)-pp^{\top}+\beta pp^{\top}=\mathrm{diag}\left(p\right)+( \beta-1)pp^{\top}.$ $$

Therefore all off-diagonal entries of $(\beta-1)pp^{\top}$ are zero.
We must have $\beta=1$ and thus $\mathrm{diag}\left(q\right)=\mathrm{diag}\left(p\right)$.
∎

###### Proof.

## Appendix \AlphAlph Proof of Lemma 3

###### Proof.

$$ $\displaystyle\|\lambda_{C}v_{C}v_{C}^{\top}-\mathcal{I}^{\Delta}(z)\|$ $\displaystyle=\|\sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\|=\sqrt{\sum_{i=1 }^{C-1}\lambda_{i}^{2}}\leq\sqrt{(\sum_{i=1}^{C-1}\lambda_{i})^{2}}$ $\displaystyle=\sum_{i=1}^{C-1}\lambda_{i}=\mathrm{tr}(\mathcal{I}^{\Delta}(z)) -\lambda_{C}=1-\|p\|^{2}-\lambda_{C}.$ $$

By [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1), we have $\lambda_{C}\geq p_{(C-1)}$. Therefore

$$ $\|\lambda_{C}v_{C}v_{C}^{\top}-\mathcal{I}^{\Delta}(z)\|\leq 1-\|p\|^{2}-p_{(C -1)}.$ $$

By Cauchy’s interlacing theorem (see our proof of [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1)),
we have

$$ $\forall{i}\in\{1,\cdots,C-1\},\quad\lambda_{i}\leq p_{(i)}.$ $$

Hence

$$ $\displaystyle\|\lambda_{C}v_{C}v_{C}^{\top}-\mathcal{I}^{\Delta}(z)\|=\sqrt{ \sum_{i=1}^{C-1}\lambda_{i}^{2}}=\sqrt{\sum_{i=2}^{C-1}\lambda_{i}^{2}}\leq \sqrt{\sum_{i=2}^{C-1}p_{(i)}^{2}}.$ $$

The statement follows immediately by combining the above upper bounds.
∎

###### Proof.

## Appendix \AlphAlph Proof of Lemma 4

###### Proof.

The spectrum of $R(y)$ is

$$ $0\leq\cdots\leq 0\leq\|e_{y}-p\|^{2}$ $$

The spectrum of $\mathcal{I}^{\Delta}(z)$, by our assumption, is

$$ $\lambda_{1}\leq\cdots\leq\lambda_{C-1}\leq\lambda_{C}$ $$

By Hoffman-Wielandt inequality, we have $\forall{z}\in\Delta^{C-1}$, $y\in\{1,\cdots,C\}$

$$ $\displaystyle\|R(y)-\mathcal{I}^{\Delta}(z)\|$ $\displaystyle\geq\sqrt{\sum_{i=1}^{C-1}\lambda_{i}^{2}+(\lambda_{C}-\|e_{y}-p \|^{2})^{2}}$ $\displaystyle\geq|\lambda_{C}-\|e_{y}-p\|^{2}|$ $\displaystyle=|\lambda_{C}-e_{y}^{\top}e_{y}-p^{\top}p+2e_{y}^{\top}p|$ $\displaystyle=|\lambda_{C}-1-\|p\|^{2}+2p_{y}|$ $\displaystyle=\max\{\lambda_{C}-1-\|p\|^{2}+2p_{y},\;1+\|p\|^{2}-\lambda_{C}-2 p_{y}\}.$ $$

By [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1), we have
$\lambda_{C}\leq 1-\|p\|^{2}$.
One can choose $y$ so that $p_{y}=p_{(1)}$, then

$$ $\displaystyle\|R(y)-\mathcal{I}^{\Delta}(z)\|$ $\displaystyle\geq 1+\|p\|^{2}-\lambda_{C}-2p_{(1)}$ $\displaystyle\geq 1+\|p\|^{2}-(1-\|p\|^{2})-2p_{(1)}$ $\displaystyle=2\|p\|^{2}-2p_{(1)}.$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Lemma 5

###### Proof.

We first look at the diagonal entries of $R$.
We have

$$ $R_{ii}=(\llbracket y=i\rrbracket-p_{i})^{2}=\left\{\begin{array}[]{ll}(1-p_{i} )^{2}&\text{if $y=i$;}\\ p_{i}^{2}&\text{otherwise.}\\ \end{array}\right.$ $$

Therefore

$$ $\mathop{\mathbb{E}}(R_{ii})=p_{i}(1-p_{i})^{2}+(1-p_{i})p_{i}^{2}=p_{i}(1-p_{i })=\mathcal{I}^{\Delta}_{ii}(z).$ $$

This shows that $R_{ii}$ is an unbiased estimator of the diagonal entries of $\mathcal{I}^{\Delta}(z)$. We have

$$ $\displaystyle\mathop{\mathbb{E}}(R_{ii}^{2})$ $\displaystyle=p_{i}(1-p_{i})^{4}+(1-p_{i})p_{i}^{4}=p_{i}(1-p_{i})\left[(1-p_{ i})^{3}+p_{i}^{3}\right]$ $\displaystyle=p_{i}(1-p_{i})\left[(1-p_{i})^{2}-p_{i}(1-p_{i})+p_{i}^{2}\right].$ $$

Therefore

$$ $\displaystyle\mathrm{Var}(R_{ii})$ $\displaystyle=\mathop{\mathbb{E}}(R_{ii}^{2})-(\mathop{\mathbb{E}}(R_{ii}))^{2}$ $\displaystyle=p_{i}(1-p_{i})\left[(1-p_{i})^{2}-p_{i}(1-p_{i})+p_{i}^{2}\right ]-p_{i}^{2}(1-p_{i})^{2}$ $\displaystyle=p_{i}(1-p_{i})\left[(1-p_{i})^{2}-2p_{i}(1-p_{i})+p_{i}^{2}\right]$ $\displaystyle=p_{i}(1-p_{i})(1-4p_{i}(1-p_{i}))$ $\displaystyle=\mathcal{I}^{\Delta}_{ii}(z)(1-4\mathcal{I}^{\Delta}_{ii}(z))$ $\displaystyle=-4\left(\mathcal{I}^{\Delta}_{ii}(z)-\frac{1}{8}\right)^{2}+ \frac{1}{16}\leq\frac{1}{16}.$ $$

The coefficient of variation (CV)

$$ $\displaystyle\frac{\sqrt{\mathrm{Var}(R_{ii})}}{\mathcal{I}^{\Delta}_{ii}(z)}= \sqrt{\frac{\mathcal{I}^{\Delta}_{ii}(z)(1-4\mathcal{I}^{\Delta}_{ii}(z))}{ \mathcal{I}^{\Delta}_{ii}(z)^{2}}}=\sqrt{\frac{1}{\mathcal{I}^{\Delta}_{ii}(z) }-4}$ $$

is unbounded. As $\mathcal{I}^{\Delta}_{ii}(z)\to 0$, the CV can take arbitrarily large value.

Next, we consider the off-diagonal entries of R.
For $i\neq{j}$, we have

$$ $\displaystyle R_{ij}$ $\displaystyle=(\llbracket y=i\rrbracket-p_{i})(\llbracket y=j\rrbracket-p_{j})$ $\displaystyle=p_{i}p_{j}-\llbracket y=i\rrbracket p_{j}-\llbracket y=j \rrbracket p_{i}.$ $$

Hence,

$$ $\mathop{\mathbb{E}}(R_{ij})=p_{i}p_{j}-p_{j}p_{j}-p_{j}p_{i}=-p_{i}p_{j}= \mathcal{I}^{\Delta}_{ij}(z).$ $$

At the same time,

$$ $\displaystyle\mathop{\mathbb{E}}(R_{ij}^{2})$ $\displaystyle=\mathop{\mathbb{E}}\left(p_{i}p_{j}-\llbracket y=i\rrbracket p_{ j}-\llbracket y=j\rrbracket p_{i}\right)^{2}$ $\displaystyle=p_{i}^{2}p_{j}^{2}+\mathop{\mathbb{E}}\left(\llbracket y=i \rrbracket p_{j}^{2}+\llbracket y=j\rrbracket p_{i}^{2}-2\llbracket y=i \rrbracket p_{i}p_{j}^{2}-2\llbracket y=j\rrbracket p_{i}^{2}p_{j}\right)$ $\displaystyle=p_{i}^{2}p_{j}^{2}+p_{i}p_{j}^{2}+p_{j}p_{i}^{2}-2p_{i}^{2}p_{j} ^{2}-2p_{i}^{2}p_{j}^{2}$ $\displaystyle=p_{i}p_{j}^{2}+p_{i}^{2}p_{j}-3p_{i}^{2}p_{j}^{2}$ $\displaystyle=p_{i}p_{j}(p_{i}+p_{j}-3p_{i}p_{j}).$ $$

Therefore

$$ $\displaystyle\mathrm{Var}(R_{ij})$ $\displaystyle=\mathop{\mathbb{E}}(R_{ij}^{2})-(\mathop{\mathbb{E}}(R_{ij}))^{2}$ $\displaystyle=p_{i}p_{j}(p_{i}+p_{j}-3p_{i}p_{j})-p_{i}^{2}p_{j}^{2}$ $\displaystyle=p_{i}p_{j}(p_{i}+p_{j}-4p_{i}p_{j})$ $\displaystyle\leq p_{i}p_{j}(1-4p_{i}p_{j})$ $\displaystyle=-4\left(p_{i}p_{j}-\frac{1}{8}\right)^{2}+\frac{1}{16}\leq\frac{ 1}{16}.$ $$

The coefficient of variation

$$ $\frac{\sqrt{\mathrm{Var}(R_{ij})}}{|\mathcal{I}^{\Delta}_{ij}(z)|}=\sqrt{\frac {p_{i}p_{j}(p_{i}+p_{j}-4p_{i}p_{j})}{p_{i}^{2}p_{j}^{2}}}=\sqrt{\frac{1}{p_{i }}+\frac{1}{p_{j}}-4}$ $$

is unbounded. As either $p_{i}\to 0$, or $p_{j}\to 0$, the CV can take arbitrarily large value.
∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 6

###### Proof.

By [lemma 2](https://arxiv.org/html/2505.13614v1#Thmtheorem2), we have

$$ $\lambda_{C}v_{C}v_{C}^{\top}\preceq\mathcal{I}^{\Delta}(z)\preceq\mathrm{diag} \left(p\right).$ $$

Therefore

$$ $\forall{x},\theta\quad\left(\frac{\partial z}{\partial\theta}\right)^{\top} \lambda_{C}v_{C}v_{C}^{\top}\frac{\partial z}{\partial\theta}\preceq\left( \frac{\partial z}{\partial\theta}\right)^{\top}\mathcal{I}^{\Delta}(z(x,\theta ))\frac{\partial z}{\partial\theta}\preceq\left(\frac{\partial z}{\partial \theta}\right)^{\top}\mathrm{diag}\left(p\right)\frac{\partial z}{\partial \theta}.$ $$

Therefore

$$ $\displaystyle\forall\theta\quad\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left( \frac{\partial z}{\partial\theta}\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z }{\partial\theta}$ $\displaystyle\preceq\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial \theta}\right)^{\top}\mathcal{I}^{\Delta}(z(x,\theta))\frac{\partial z}{ \partial\theta}\preceq\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\frac{ \partial z_{i}}{\partial\theta}\frac{\partial z_{i}}{\partial\theta^{\top}}.$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Corollary 7

###### Proof.

We first prove the upper bound. By [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6), we have

$$ $\mathcal{F}^{\Delta}(\theta)\preceq\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{ i}\frac{\partial z_{i}}{\partial\theta}\frac{\partial z_{i}}{\partial\theta^{ \top}}.$ $$

Taking trace on both sides, we get

$$ $\displaystyle\mathrm{tr}(\mathcal{F}^{\Delta}(\theta))$ $\displaystyle\leq\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\mathrm{tr}\left (\frac{\partial z_{i}}{\partial\theta}\frac{\partial z_{i}}{\partial\theta^{ \top}}\right)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\mathrm{tr}\left( \frac{\partial z_{i}}{\partial\theta^{\top}}\frac{\partial z_{i}}{\partial \theta}\right)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\frac{\partial z_{i }}{\partial\theta^{\top}}\frac{\partial z_{i}}{\partial\theta}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\left\|\frac{ \partial z_{i}}{\partial\theta}\right\|^{2}.$ $$

The lower bound is not straightforward from [proposition 6](https://arxiv.org/html/2505.13614v1#Thmtheorem6). By [Eq. 4](https://arxiv.org/html/2505.13614v1#S2.E4), we have

$$ $\mathrm{tr}(\mathcal{F}^{\Delta}(\theta))=\sum_{x\in\mathcal{D}_{x}}\mathrm{tr }\left[\left(\frac{\partial z}{\partial\theta}\right)^{\top}\mathcal{I}^{ \Delta}(z)\frac{\partial z}{\partial\theta}\right]=\sum_{x\in\mathcal{D}_{x}} \mathrm{tr}\left[\frac{\partial z}{\partial\theta}\left(\frac{\partial z}{ \partial\theta}\right)^{\top}\mathcal{I}^{\Delta}(z)\right].$ $$

Note that $\frac{\partial z}{\partial\theta}\left(\frac{\partial z}{\partial\theta}\right
)^{\top}$
is a $C\times C$ matrix with sorted eigenvalues $\sigma_{1}^{2}(x,\theta)\leq\cdots\leq\sigma^{2}_{C}(x,\theta)$.
By [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1),
$\mathcal{I}^{\Delta}(z)$ is another $C\times C$ matrix with sorted eigenvalues
$0=\lambda_{1}(x,\theta)\leq\cdots\leq\lambda_{C}(x,\theta)$.
Applying the Von Neumann trace inequality, we get

$$ $\mathrm{tr}(\mathcal{F}^{\Delta}(\theta))\geq\sum_{x\in\mathcal{D}_{x}}\sum_{i =2}^{C}\lambda_{i}(x,\theta)\sigma^{2}_{C-i+1}(x,\theta)\geq\sum_{x\in\mathcal {D}_{x}}\lambda_{C}(x,\theta)\sigma^{2}_{1}(x,\theta).$ $$

The last “$\geq$” is because all terms $\lambda_{i}(x,\theta)\sigma^{2}_{C-i+1}(x,\theta)$ are non-negative.
∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 8

###### Proof.

Denote the singular values of
$\frac{\partial z}{\partial\theta}$
as $0\leq\sigma_{1}\leq\cdots\leq\sigma_{C}$.
Then the eigenvalues of the $C\times C$ Hermitian matrix
$\frac{\partial z}{\partial\theta}\left(\frac{\partial z}{\partial\theta}\right
)^{\top}$
is $\sigma_{1}^{2}\leq\cdots\leq\sigma_{C}^{2}$.

To prove the upper bound, we have

$$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\left(\frac{ \partial z_{i}}{\partial\theta}\right)^{\top}\frac{\partial z_{i}}{\partial \theta}-\mathcal{F}^{\Delta}(\theta)\right\|$ $\displaystyle=$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial \theta}\right)^{\top}\left(\mathrm{diag}\left(p\right)-\mathrm{diag}\left(p \right)+pp^{\top}\right)\frac{\partial z}{\partial\theta}\right\|$ $\displaystyle=$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial \theta}\right)^{\top}pp^{\top}\frac{\partial z}{\partial\theta}\right\|$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\sqrt{\mathrm{tr}\left[\left(\frac{ \partial z}{\partial\theta}\right)^{\top}pp^{\top}\frac{\partial z}{\partial \theta}\left(\frac{\partial z}{\partial\theta}\right)^{\top}pp^{\top}\frac{ \partial z}{\partial\theta}\right]}$ $\displaystyle=$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\sqrt{\mathrm{tr}\left[p^{\top}\frac{ \partial z}{\partial\theta}\left(\frac{\partial z}{\partial\theta}\right)^{ \top}pp^{\top}\frac{\partial z}{\partial\theta}\left(\frac{\partial z}{ \partial\theta}\right)^{\top}p\right]}$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\sqrt{\left[p^{\top}\frac{\partial z}{ \partial\theta}\left(\frac{\partial z}{\partial\theta}\right)^{\top}p\right]^{ 2}}$ $\displaystyle=$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}p^{\top}\frac{\partial z}{\partial \theta}\left(\frac{\partial z}{\partial\theta}\right)^{\top}p$ $\displaystyle=$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\|p\|^{2}\cdot\frac{p^{\top}}{\|p\|} \frac{\partial z}{\partial\theta}\left(\frac{\partial z}{\partial\theta}\right )^{\top}\frac{p}{\|p\|}$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\|p\|^{2}\sigma_{C}^{2}.$ $$

Now we are ready to prove the lower bound. From the above, we have

$$ $\left\|\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\left(\frac{\partial z_{i} }{\partial\theta}\right)^{\top}\frac{\partial z_{i}}{\partial\theta}-\mathcal{ F}^{\Delta}(\theta)\right\|=\left\|\sum_{x\in\mathcal{D}_{x}}\left(\frac{ \partial z}{\partial\theta}\right)^{\top}pp^{\top}\frac{\partial z}{\partial \theta}\right\|.$ $$

Denote $\omega(x)\coloneqq\left(\frac{\partial z}{\partial\theta}\right)^{\top}p$. Then

$$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{i}\left(\frac{ \partial z_{i}}{\partial\theta}\right)^{\top}\frac{\partial z_{i}}{\partial \theta}-\mathcal{F}^{\Delta}(\theta)\right\|$ $\displaystyle=\left\|\sum_{x\in\mathcal{D}_{x}}\omega(x)\omega(x)^{\top}\right\|$ $\displaystyle=\sqrt{\mathrm{tr}\left(\left(\sum_{x\in\mathcal{D}_{x}}\omega(x) \omega(x)^{\top}\right)^{2}\right)}$ $\displaystyle\geq\sqrt{\sum_{x\in\mathcal{D}_{x}}(\omega(x)^{\top}\omega(x))^{ 2}}$ $\displaystyle=\sqrt{\sum_{x\in\mathcal{D}_{x}}\|\omega(x)\|^{4}}.$ $$

The last “$\geq$” is due to

$$ $\mathrm{tr}\left(\omega(x)\omega(x)^{\top}\omega(x^{\prime})\omega(x^{\prime}) ^{\top}\right)=\mathrm{tr}\left(\omega(x^{\prime})^{\top}\omega(x)\omega(x)^{ \top}\omega(x^{\prime})\right)=(\omega(x^{\prime})^{\top}\omega(x))^{2}\geq 0.$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 9

###### Proof.

We can first have a loose bound:

$$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z }{\partial\theta}\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial \theta}-\mathcal{F}^{\Delta}(\theta)\right\|$ $\displaystyle=$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z }{\partial\theta}\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial \theta}-\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial\theta} \right)^{\top}\mathcal{I}^{\Delta}(z)\frac{\partial z}{\partial\theta}\right\|$ $\displaystyle=$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial \theta}\right)^{\top}\left(\sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\right) \frac{\partial z}{\partial\theta}\right\|$ $\displaystyle\leq$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}p_{(C-1)}\left(\frac{\partial z} {\partial\theta}\right)^{\top}\frac{\partial z}{\partial\theta}\right\|\quad \text{(Due to that $\sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\preceq p_{(C- 1)}I$)}$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}p_{(C-1)}\left\|\frac{\partial z}{ \partial\theta}\left(\frac{\partial z}{\partial\theta}\right)^{\top}\right\|.$ $$

The eigenvalues of
$\left(\frac{\partial z}{\partial\theta}\left(\frac{\partial z}{\partial\theta}
\right)^{\top}\right)^{2}$ are
$\sigma_{1}^{4}\leq\cdots\leq\sigma_{C}^{4}$.
We have

$$ $\displaystyle\left\|\left(\frac{\partial z}{\partial\theta}\right)^{\top}\left (\sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\right)\frac{\partial z}{\partial \theta}\right\|^{2}$ $\displaystyle=$ $\displaystyle\mathrm{tr}\left[\left(\frac{\partial z}{\partial\theta}\right)^{ \top}\left(\sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\right)\frac{\partial z }{\partial\theta}\left(\frac{\partial z}{\partial\theta}\right)^{\top}\left( \sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\right)\frac{\partial z}{\partial \theta}\right]$ $\displaystyle=$ $\displaystyle\mathrm{tr}\left[\left(\frac{\partial z}{\partial\theta}\left( \frac{\partial z}{\partial\theta}\right)^{\top}\left(\sum_{i=1}^{C-1}\lambda_{ i}v_{i}v_{i}^{\top}\right)\right)^{2}\right]$ $\displaystyle\leq$ $\displaystyle\mathrm{tr}\left[\left(\frac{\partial z}{\partial\theta}\left( \frac{\partial z}{\partial\theta}\right)^{\top}\right)^{2}\left(\sum_{i=1}^{C- 1}\lambda_{i}^{2}v_{i}v_{i}^{\top}\right)\right]\quad\text{Due to $\mathrm{tr} (AB)^{2}\leq\mathrm{tr}(A^{2}B^{2})$}$ $\displaystyle\leq$ $\displaystyle\sum_{i=2}^{C-1}\sigma_{i+1}^{4}\lambda_{i}^{2}.$ $$

The last “$\leq$” is due to Von Neumann’s trace inequality, and that the smallest two eigenvalues of the matrix
$\sum_{i=1}^{C-1}\lambda_{i}^{2}v_{i}v_{i}^{\top}$ are both zero.
We also have the Cauchy interlacing

$$ $\lambda_{2}\leq p_{(2)}\leq\lambda_{3}\leq p_{(3)}\leq\cdots\leq\lambda_{C-1} \leq p_{(C-1)}.$ $$

To sum up,

$$ $\displaystyle\left\|\sum_{x\in\mathcal{D}_{x}}\lambda_{C}\left(\frac{\partial z }{\partial\theta}\right)^{\top}v_{C}v_{C}^{\top}\frac{\partial z}{\partial \theta}-\mathcal{F}^{\Delta}(\theta)\right\|$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\left\|\left(\frac{\partial z}{\partial \theta}\right)^{\top}\left(\sum_{i=1}^{C-1}\lambda_{i}v_{i}v_{i}^{\top}\right) \frac{\partial z}{\partial\theta}\right\|$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\sqrt{\sum_{i=2}^{C-1}\sigma_{i+1}^{4} \lambda_{i}^{2}}$ $\displaystyle\leq$ $\displaystyle\sum_{x\in\mathcal{D}_{x}}\sqrt{\sum_{i=2}^{C-1}\sigma_{i+1}^{4}p ^{2}_{(i)}}.$ $$

If one relax $\forall{i}\in\{2,\cdots,C-1\}$, $p_{(i)}\leq p_{(C-1)}$,
then we get the loose bound proved earlier.

∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 10

###### Proof.

$$ $\displaystyle\|\mathcal{F}(\theta)-\hat{\mathcal{F}}^{\Delta}(\theta)\|_{\sigma}$ $\displaystyle=\left\|\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{ \partial\theta}\right)^{\top}\cdot\mathcal{I}(z(x,\theta))\cdot\frac{\partial z }{\partial\theta}-\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial \theta}\right)^{\top}(e_{y}-p)(e_{y}-p)^{\top}\frac{\partial z}{\partial\theta }\right\|_{\sigma}$ $\displaystyle=\left\|\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{ \partial\theta}\right)^{\top}\left[\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y }-p)(e_{y}-p)^{\top}\right]\frac{\partial z}{\partial\theta}\right\|_{\sigma}$ $\displaystyle\leq\sum_{x\in\mathcal{D}_{x}}\left\|\left(\frac{\partial z}{ \partial\theta}\right)^{\top}\left[\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y }-p)(e_{y}-p)^{\top}\right]\frac{\partial z}{\partial\theta}\right\|_{\sigma}$ $\displaystyle\leq\sum_{x\in\mathcal{D}_{x}}\left\|\frac{\partial z}{\partial \theta}\right\|_{\sigma}\left\|\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p) (e_{y}-p)^{\top}\right\|_{\sigma}\left\|\frac{\partial z}{\partial\theta} \right\|_{\sigma}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sigma_{C}^{2}\left\|\mathrm{diag} \left(p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{\top}\right\|_{\sigma}.$ $$

Now we examine the matrix
$\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{\top}$.
By [theorem 1](https://arxiv.org/html/2505.13614v1#Thmtheorem1), the spectrum of $\mathrm{diag}\left(p\right)-pp^{\top}$ is

$$ $\lambda_{1}=0\leq\lambda_{2}\leq\cdots\leq\lambda_{C}.$ $$

By Cauchy interlacing theorem,
the spectrum of
$\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{\top}$, given by
$\lambda^{\prime}_{1},\cdots,\lambda^{\prime}_{C}$, must satisfy

$$ $\lambda^{\prime}_{1}\leq\lambda_{1}=0\leq\lambda^{\prime}_{2}\leq\lambda_{2} \leq\cdots\leq\lambda^{\prime}_{C}\leq\lambda_{C}.$ $$

with at least one eigenvalue that is not positive: $\lambda_{1}^{\prime}\leq 0$.
Therefore

$$ $\left\|\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{\top}\right\| _{\sigma}\leq\max\{-\lambda_{1}^{\prime},\lambda_{C}\}.$ $$

We also have

$$ $\displaystyle\lambda^{\prime}_{1}$ $\displaystyle=\inf_{u:\|u\|=1}u^{\top}\left[\mathrm{diag}\left(p\right)-pp^{ \top}-(e_{y}-p)(e_{y}-p)^{\top}\right]u$ $\displaystyle\geq\inf_{u:\|u\|=1}-u^{\top}\left[(e_{y}-p)(e_{y}-p)^{\top} \right]u$ $\displaystyle=-(e_{y}-p)^{\top}(e_{y}-p)$ $\displaystyle=-(1+p^{\top}p-2p_{y})$ $\displaystyle=2p_{y}-1-\|p\|^{2}.$ $$

Therefore

$$ $\displaystyle\left\|\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{ \top}\right\|_{\sigma}$ $\displaystyle\leq\max\{1+\|p\|^{2}-2p_{y},\lambda_{C}\}$ $\displaystyle\leq\max\{1+\|p\|^{2}-2p_{y},1-\|p\|^{2}\}$ $\displaystyle\leq 1+\|p\|^{2}.$ $$

In summary,

$$ $\|\mathcal{F}(\theta)-\hat{\mathcal{F}}^{\Delta}(\theta)\|_{\sigma}\leq\sum_{x \in\mathcal{D}_{x}}\sigma_{C}^{2}(1+\|p\|^{2}).$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 11

###### Proof.

$$ $\displaystyle\left\|\left(\frac{\partial z}{\partial\theta}\right)^{\top}\cdot \mathcal{I}^{\Delta}(z(x,\theta))\cdot\frac{\partial z}{\partial\theta}-\left( \frac{\partial z}{\partial\theta}\right)^{\top}\cdot\hat{\mathcal{I}}^{\Delta} (z(x,\theta))\cdot\frac{\partial z}{\partial\theta}\right\|_{\sigma}$ $\displaystyle\geq$ $\displaystyle\left\|\left(\frac{\partial z}{\partial\theta}\right)^{\top}\cdot \left[\mathcal{I}^{\Delta}(z(x,\theta))-\hat{\mathcal{I}}^{\Delta}(z(x,\theta) )\right]\cdot\frac{\partial z}{\partial\theta}\right\|_{\sigma}$ $\displaystyle=$ $\displaystyle\left\|\left(\frac{\partial z}{\partial\theta}\right)^{\top}\cdot \left[\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{\top}\right] \cdot\frac{\partial z}{\partial\theta}\right\|_{\sigma}$ $\displaystyle=$ $\displaystyle\sup_{u:\|u\|=1}\left|\left(\frac{\partial z}{\partial\theta}u \right)^{\top}\cdot\left[\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p)(e_{y} -p)^{\top}\right]\cdot\left(\frac{\partial z}{\partial\theta}u\right)\right|$ $\displaystyle\geq$ $\displaystyle\sup_{v:\|v\|=1}\left|\sigma_{(1)}v\cdot\left[\mathrm{diag}\left( p\right)-pp^{\top}-(e_{y}-p)(e_{y}-p)^{\top}\right]\cdot\sigma_{(1)}v\right|$ $\displaystyle\geq$ $\displaystyle\sigma_{(1)}^{2}\|\mathrm{diag}\left(p\right)-pp^{\top}-(e_{y}-p) (e_{y}-p)^{\top}\|_{\sigma}$ $\displaystyle\geq$ $\displaystyle\sigma_{(1)}^{2}\left|\left(\frac{e_{y}-p}{\|e_{y}-p\|}\right)^{ \top}\left((e_{y}-p)(e_{y}-p)^{\top}-\lambda_{C}\right)\frac{e_{y}-p}{\|e_{y}- p\|}\right|$ $\displaystyle=$ $\displaystyle\sigma_{(1)}^{2}\left|\|e_{y}-p\|^{2}-\lambda_{C}\right|$ $\displaystyle=$ $\displaystyle\sigma_{(1)}^{2}\left|1+\|p\|^{2}-\lambda_{C}-2p_{y}\right|.$ $$

We choose $p_{y}=p_{(1)}$, therefore $\exists y$, such that

$$ $\displaystyle\left\|\left(\frac{\partial z}{\partial\theta}\right)^{\top}\cdot \mathcal{I}^{\Delta}(z(x,\theta))\cdot\frac{\partial z}{\partial\theta}-\left( \frac{\partial z}{\partial\theta}\right)^{\top}\cdot\hat{\mathcal{I}}^{\Delta} (z(x,\theta))\cdot\frac{\partial z}{\partial\theta}\right\|_{\sigma}$ $\displaystyle\geq$ $\displaystyle\sigma_{(1)}^{2}\left|1+\|p\|^{2}-\lambda_{C}-2p_{(1)}\right|.$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 12

###### Proof.

From the derivations in the main text, we already know that $\mathop{\mathbb{E}}_{p(\xi)}\mathbb{I}(\theta)=\mathcal{I}(\theta)$.
To show the estimator variance, we first consider the case when $p(\xi)$ is a standard multivariate Gaussian distribution.
First we note that both $\mathfrak{h}(\mathcal{D}_{x},\theta)$ and $\partial{\mathfrak{h}}/\partial\theta_{i}$
are in the form of a sum of independent Gaussian random variables. Hence,

$$ $\frac{\partial\mathfrak{h}}{\partial\theta_{i}}=\sum_{x\in\mathcal{D}_{x}}\sum _{y=1}^{C}\sqrt{p(y\,|\,x,\theta)}\frac{\partial\ell_{xy}}{\partial\theta_{i}} \xi_{xy}\sim G\left(0,\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p(y\,|\,x,\theta )\left(\frac{\partial\ell_{xy}}{\partial\theta_{i}}\right)^{2}\right).$ $$

Therefore

$$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}}{ \partial\theta_{i}}\right)^{2}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p(y\,|\,x,\theta)\left( \frac{\partial\ell_{xy}}{\partial\theta_{i}}\right)^{2}=\mathcal{I}_{ii}( \theta);$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}}{ \partial\theta_{i}}\right)^{4}$ $\displaystyle=3\mathcal{I}_{ii}^{2}(\theta).$ $$

Therefore

$$ $\mathrm{Var}(\mathbb{I}(\theta_{i}))=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{ \partial\mathfrak{h}}{\partial\theta_{i}}\right)^{4}-\mathcal{I}_{ii}^{2}( \theta)=2\mathcal{I}_{ii}^{2}(\theta).$ $$

We now consider that $p(\xi)$ is Rademacher.

$$ $\displaystyle\mathrm{Var}(\mathbb{I}(\theta_{i}))$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}}{ \partial\theta_{i}}\right)^{4}-\left(\mathop{\mathbb{E}}\left(\frac{\partial \mathfrak{h}}{\partial\theta_{i}}\right)^{2}\right)^{2}$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}}{ \partial\theta_{i}}\right)^{4}-\mathcal{I}_{ii}^{2}(\theta)$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\sum_{x\in\mathcal{D}_{x}}\sum _{y=1}^{C}\sqrt{p(y\,|\,x,\theta)}\frac{\partial\ell_{xy}}{\partial\theta_{i}} \xi_{xy}\right)^{4}-\mathcal{I}_{ii}^{2}(\theta)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p^{2}(y\,|\,x,\theta) \left(\frac{\partial\ell_{xy}}{\partial\theta_{i}}\right)^{4}$ $\displaystyle\quad+3\sum_{(x,y)\neq(x^{\prime},y^{\prime})}p(y\,|\,x,\theta) \left(\frac{\partial\ell_{xy}}{\partial\theta_{i}}\right)^{2}p(y^{\prime}\,|\, x^{\prime},\theta)\left(\frac{\partial\ell_{x^{\prime}y^{\prime}}}{\partial \theta_{i}}\right)^{2}-\mathcal{I}_{ii}^{2}(\theta).$ $$

Note that

$$ $\displaystyle\mathcal{I}_{ii}^{2}(\theta)$ $\displaystyle=\left(\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p(y\,|\,x,\theta) \left(\frac{\partial\ell_{xy}}{\partial\theta_{i}}\right)^{2}\right)^{2}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}p^{2}(y\,|\,x,\theta) \left(\frac{\partial\ell_{xy}}{\partial\theta_{i}}\right)^{4}+\sum_{(x,y)\neq( x^{\prime},y^{\prime})}p(y\,|\,x,\theta)\left(\frac{\partial\ell_{xy}}{ \partial\theta_{i}}\right)^{2}p(y^{\prime}\,|\,x^{\prime},\theta)\left(\frac{ \partial\ell_{x^{\prime}y^{\prime}}}{\partial\theta_{i}}\right)^{2}.$ $$

Hence,

$$ $\displaystyle\mathrm{Var}(\mathbb{I}(\theta_{i}))$ $\displaystyle=3\mathcal{I}_{ii}^{2}(\theta)-2\sum_{x\in\mathcal{D}_{x}}\sum_{y =1}^{C}p^{2}(y\,|\,x,\theta)\left(\frac{\partial\ell_{xy}}{\partial\theta_{i}} \right)^{4}-\mathcal{I}_{ii}^{2}(\theta)$ $\displaystyle=2\mathcal{I}_{ii}^{2}(\theta)-2\sum_{x\in\mathcal{D}_{x}}\sum_{y =1}^{C}p^{2}(y\,|\,x,\theta)\left(\frac{\partial\ell_{xy}}{\partial\theta_{i}} \right)^{4}.$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 13

###### Proof.

$$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}(\mathbb{F}^{\mathrm{DG}}(\theta))$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{DG}}}{\partial\theta}\frac{\partial\mathfrak{h}^{\mathrm{DG}}}{ \partial\theta^{\top}}\right)$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\sum_{x\in\mathcal{D}_{x}}\sum _{y=1}^{C}\sqrt{\zeta_{y}(x,\theta)}\frac{\partial z_{y}}{\partial\theta}\xi_{ xy}\sum_{x^{\prime}\in\mathcal{D}_{x}}\sum_{y^{\prime}=1}^{C}\sqrt{\zeta_{y^{ \prime}}(x^{\prime},\theta)}\frac{\partial z_{y^{\prime}}}{\partial\theta^{ \top}}\xi_{x^{\prime}y^{\prime}}\right)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\sum_{x^{\prime}\in \mathcal{D}_{x}}\sum_{y^{\prime}=1}^{C}\sqrt{\zeta_{y}(x,\theta)}\sqrt{\zeta_{ y^{\prime}}(x^{\prime},\theta)}\frac{\partial z_{y}}{\partial\theta}\frac{ \partial z_{y^{\prime}}}{\partial\theta^{\top}}\mathop{\mathbb{E}}_{p(\xi)} \left(\xi_{xy}\xi_{x^{\prime}y^{\prime}}\right)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\zeta_{y}(x,\theta)\frac {\partial z_{y}}{\partial\theta}\frac{\partial z_{y}}{\partial\theta^{\top}}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\left(\frac{\partial z}{\partial\theta }\right)^{\top}\mathcal{I}^{\mathrm{DG}}(z(x,\theta))\frac{\partial z}{ \partial\theta}$ $\displaystyle=\mathcal{F}^{\mathrm{DG}}(\theta).$ $$

Therefore,

$$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}\left(\mathbb{F}_{ii}^{\mathrm{DG}}( \theta)\right)$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{DG}}}{\partial\theta_{i}}\right)^{2}=\sum_{x\in\mathcal{D}_{x}}\sum_{y =1}^{C}\zeta_{y}(x,\theta)\left(\frac{\partial z_{y}}{\partial\theta_{i}} \right)^{2}=\mathcal{F}_{ii}^{\mathrm{DG}}(\theta).$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{DG}}}{\partial\theta_{i}}\right)^{4}$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\sum_{x\in\mathcal{D}_{x}}\sum _{y=1}^{C}\sqrt{\zeta_{y}(x,\theta)}\frac{\partial z_{y}}{\partial\theta_{i}} \xi_{xy}\right)^{4}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\sum_{y=1}^{C}\zeta_{y}^{2}(x,\theta) \left(\frac{\partial z_{y}}{\partial\theta_{i}}\right)^{4}+3\sum_{(x,y)\neq(x^ {\prime},y^{\prime})}\zeta_{y}(x,\theta)\left(\frac{\partial z_{y}}{\partial \theta_{i}}\right)^{2}\zeta_{y^{\prime}}(x^{\prime},\theta)\left(\frac{ \partial z_{y^{\prime}}}{\partial\theta_{i}}\right)^{2}$ $\displaystyle=3(\mathcal{F}_{ii}^{\mathrm{DG}}(\theta))^{2}-2\sum_{x\in \mathcal{D}_{x}}\sum_{y=1}^{C}\zeta_{y}^{2}(x,\theta)\left(\frac{\partial z_{y }}{\partial\theta_{i}}\right)^{4}.$ $$

Hence,

$$ $\displaystyle\mathrm{Var}(\mathbb{F}_{ii}^{\mathrm{DG}}(\theta))$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{DG}}}{\partial\theta_{i}}\right)^{4}-(\mathcal{F}_{ii}^{\mathrm{DG}}( \theta))^{2}$ $\displaystyle=2(\mathcal{F}_{ii}^{\mathrm{DG}}(\theta))^{2}-2\sum_{x\in \mathcal{D}_{x}}\sum_{y=1}^{C}\zeta_{y}^{2}(x,\theta)\left(\frac{\partial z_{y }}{\partial\theta_{i}}\right)^{4}.$ $$

∎

###### Proof.

## Appendix \AlphAlph Proof of Proposition 14

###### Proof.

The proof is similar to [proposition 13](https://arxiv.org/html/2505.13614v1#Thmtheorem13) and is also based on the Hutchinson’s trick.

$$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}(\mathbb{F}^{\mathrm{LR}}(\theta))$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{LR}}}{\partial\theta}\frac{\partial\mathfrak{h}^{\mathrm{LR}}}{ \partial\theta^{\top}}\right)$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\sum_{x\in\mathcal{D}_{x}} \sqrt{\lambda_{C}(x,\theta)}\left(\frac{\partial z}{\partial\theta}\right)^{ \top}v_{C}(x,\theta)\xi_{x}\sum_{x^{\prime}\in\mathcal{D}_{x}}\sqrt{\lambda_{C }(x^{\prime},\theta)}v_{C}(x^{\prime},\theta)^{\top}\left(\frac{\partial z}{ \partial\theta}\right)\xi_{x^{\prime}}\right)$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\lambda_{C}(x,\theta)\left(\frac{ \partial z}{\partial\theta}\right)^{\top}v_{C}(x,\theta)v_{C}(x,\theta)^{\top} \left(\frac{\partial z}{\partial\theta}\right)$ $\displaystyle=\mathcal{F}^{\mathrm{LR}}(\theta).$ $$

Therefore

$$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}(\mathbb{F}^{\mathrm{LR}}_{ii}(\theta))$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\lambda_{C}(x,\theta)\left(\left(\frac {\partial z}{\partial\theta_{i}}\right)^{\top}v_{C}(x,\theta)\right)^{2}= \mathcal{F}^{\mathrm{LR}}_{ii}(\theta);$ $\displaystyle\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{LR}}}{\partial\theta_{i}}\right)^{4}$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\sum_{x\in\mathcal{D}_{x}} \sqrt{\lambda_{C}(x,\theta)}v_{C}^{\top}(x,\theta)\frac{\partial z}{\partial \theta_{i}}\xi_{x}\right)^{4}$ $\displaystyle=\sum_{x\in\mathcal{D}_{x}}\lambda_{C}^{2}(x,\theta)\left(v_{C}^{ \top}(x,\theta)\frac{\partial z}{\partial\theta_{i}}\right)^{4}$ $\displaystyle\quad+3\sum_{x\neq{x^{\prime}}}\lambda_{C}(x,\theta)\left(v_{C}^{ \top}(x,\theta)\frac{\partial z}{\partial\theta_{i}}\right)^{2}\lambda_{C}(x^{ \prime},\theta)\left(v_{C}^{\top}(x^{\prime},\theta)\frac{\partial z}{\partial \theta_{i}}\right)^{2}$ $\displaystyle=3(\mathcal{F}^{\mathrm{LR}}_{ii}(\theta))^{2}-2\sum_{x\in \mathcal{D}_{x}}\lambda_{C}^{2}(x,\theta)\left(v_{C}^{\top}(x,\theta)\frac{ \partial z}{\partial\theta_{i}}\right)^{4}.$ $$

Hence,

$$ $\displaystyle\mathrm{Var}\left(\mathbb{F}^{\mathrm{LR}}_{ii}(\theta)\right)$ $\displaystyle=\mathop{\mathbb{E}}_{p(\xi)}\left(\frac{\partial\mathfrak{h}^{ \mathrm{LR}}}{\partial\theta_{i}}\right)^{4}-(\mathcal{F}^{\mathrm{LR}}_{ii}( \theta))^{2}$ $\displaystyle=2(\mathcal{F}^{\mathrm{LR}}_{ii}(\theta))^{2}-2\sum_{x\in \mathcal{D}_{x}}\lambda_{C}^{2}(x,\theta)\left(v_{C}^{\top}(x,\theta)\frac{ \partial z}{\partial\theta_{i}}\right)^{4}.$ $$

∎

###### Proof.

## Appendix \AlphAlph Experiments on SST-2

We compute the diagonal FIM of DistilBERT [27], which is fine-tuned
on the Stanford Sentiment Treebank v2 (SST-2) [28]
for binary sentiment classification.
The model is available as distilbert-base-uncased-finetuned-sst-2-english in the Hugging Face library [36].
The density of diagonal FIM entries are shown in [fig. 2](https://arxiv.org/html/2505.13614v1#A15.F2).
There are two differences with the AG News experiment in the main text:
(1) The number of classes has reduced to $C=2$; (2) The model is already fine-tuned
and the Fisher information is evaluated on a different region in the parameter space
compared to the AG News case.
Note $\mathbb{F}^{\mathrm{LR}}_{ii}$ is very close to and sometimes larger than the value of
$\mathbb{F}_{ii}$. This is because when $C=2$, the core matrix is already rank-1.
And $\mathbb{F}$ and $\mathbb{F}^{\mathrm{LR}}$ are essentially different (unbiased) estimators
of $\mathcal{F}$. The scale of the upper bound $\mathbb{F}^{\mathrm{DG}}_{ii}$ is much larger
than $\mathbb{F}_{ii}$ showing that the bound is loose.
All numerical results presented here are performed on a MacBook Pro
with Apple M1 CPU and 16GB RAM.

Figure: Figure 2: Density plots of diagonal FIM elements based on different approximations (rows) across different layers (columns) on DistilBERT fine-tuned on the SST-2 dataset. The maximum value of the x-axis (number on the bottom right corner) shows the mean value of the FIM multiplied by 2,000. The y-axis means probability density in log-scale.
Refer to caption: x2.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Technical Concerns

Computational complexity underspecified: Power iteration convergence for computing λ_C, v_C could be slow near uniform distributions (small spectral gap). The O(MC|𝒟_x|) cost needs quantification.
Implementation gaps: No guidance on choosing probe count for target accuracy, mini-batch handling, or numerical stability considerations.

Minor Issues

Notation overload (multiple uses of F̂ for both empirical and MC Fisher)
Some bounds are loose (e.g., diagonal upper bound always has error ≥ 1/C). Correct me if im wrong/misinterpret

Questions for Authors

What is the wall-clock time comparison with K-FAC for equivalent accuracy?
Can you demonstrate improved optimization performance in a practical setting?
How does mini-batching affect the variance bounds?

Verdict

The paper makes solid theoretical contributions with rigorous mathematical development. However, it lacks the experimental validation needed to demonstrate practical value. The theory is great and extensive, but the work feels a little missing in terms of comprehensive experiments showing real optimization improvements. But I am also open to understanding if this mainly geared to a paper of more theoretical build up value (or step in that direction) than a fully practical ready usage.
soundness: 3
strengths: Solid mathematical foundation: All major theorems have complete proofs in the appendix. The spectral analysis of the simplex FIM (Theorem 1) and envelope characterizations (Lemma 2) are rigorously established.

Novel computational approach: The Hutchinson estimator with “detach-and-mix” construction provides unbiased FIM estimates with provably bounded diagonal CV ≤ √2, addressing variance issues in Monte Carlo methods.

Comprehensive theoretical analysis: The paper provides both deterministic bounds and stochastic estimates with detailed variance analysis for Gaussian and Rademacher distributions.
summary: The paper analyzes the Fisher Information Matrix (FIM) for neural classifiers through deterministic bounds based on spectral analysis and proposes a novel Hutchinson-based estimator. The theoretical development is rigorous with complete proofs in the appendix, though experimental validation remains limited.
weaknesses: Experiments: Only DistilBERT tested on AG News and SST-2. Missing: (i) comparison with K-FAC, diagonal empirical Fisher (Adam), exact Gauss-Newton; (ii) optimization performance metrics; (iii) other architectures (CNNs, ResNets). Just I guess a general expansion in this area would be nice, though you note each formulation may require a from-scratch derivation for each. Please comment on this

Unclear practical advantage: No demonstration of how the estimator improves optimization or learning dynamics compared to existing methods.

Terminology issue: “Singular semi-Riemannian metric” is incorrect I believe - the FIM is degenerate PSD, not indefinite.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Q1. It would be helpful to include a comparison with a Monte Carlo approach in terms of both computational complexity and accuracy.

Q2. Is it correct that the LB estimator typically performs better when the predicted distribution approaches one of the corners of the simplex (see Lemma 3)?

Q3. I suggest including the true Fisher information in the experimental comparison, at least in settings where it is feasible.

Q4. A simple synthetic example where the full Fisher information matrix can be computed and visualized would help demonstrate the behavior of the estimator.
soundness: 3
strengths: - The problem of approximating the Fisher metric in neural network settings is timely and very important. The proposed estimators are simple and can be easily computed in practice.
- The technical part of the paper appears to be sound and the theoretical claims correct, but I have not followed every proof in detail. The experiments support the theoretical analysis.
- The paper is generally well written and accessible.
summary: The authors propose an unbiased estimator for the Fisher metric in the parameter space of a neural network, in the setting of multiclass classification, where the underlying geometry is determined by the simplex. The main estimator, along with two additional approximations, is straightforward to compute and leverages automatic differentiation. The authors provide a theoretical analysis to analyze the accuracy and some properties of the proposed estimators. The theoretical claims are further supported by empirical demonstrations.
weaknesses: - Since the paper proposes an estimator for the Fisher metric, I would expect additional empirical demonstrations regarding the estimator’s accuracy and efficiency. I acknowledge that the theoretical results constitute the primary contribution, but I believe that further benchmark experiments would help demonstrate the properties of the estimator and also consider comparisons to related methods. The current experimental section provides some insight, but additional comparisons would strengthen the paper.
- As the authors mention in Section 5, Hutchinson-type estimators have been used in related contexts. While the extension to neural networks is appreciated and well motivated, as a non-expert in Fisher information analysis, I cannot argue which of the theoretical contributions are novel and which may already be known.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: see wk
soundness: 3
strengths: - Provides a unified and theoretically grounded framework for deterministic and stochastic FIM estimation with clear variance guarantees.  
- The Hutchinson estimator is efficient and easy to integrate into deep learning pipelines, enabling scalable information-geometric analysis.
summary: This paper proposes deterministic bounds and Hutchinson-based stochastic estimators for the Fisher Information Matrix (FIM) on neural manifolds. The method provides unbiased, bounded-variance FIM estimates requiring only one backward pass, making it scalable to large networks. Experiments on DistilBERT show feasible and stable approximations compared with empirical and Monte Carlo estimators.
weaknesses: - The paper does not validate the estimator on analytically known probability distributions, especially in high-dimensional settings. This makes it difficult to verify the claimed accuracy of the proposed bounds and stochastic estimates.
- The practical impact of improved FIM accuracy on downstream tasks (e.g., optimization, generalization) is not demonstrated. In some real applications [a], approximation may even be preferable to precision, since neural networks themselves are inherently approximate. The authors should strengthen the discussion and empirical evidence on how their estimator benefits real learning tasks.

[a] Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: None
soundness: 4
strengths: 1. Theoretical results are sufficient and rigorous.

2. The ease of implementation in standard deep learning frameworks, which is evident by the empirical validation with better MAE while keeping computational costs similar. The framework's independence from architecture and its ability to apply to both diagonal and low-rank approximations offer flexible trade-offs between accuracy and computational cost.
summary: This work presents a new approach along with theoretical results for estimating the Fisher Information Matrix (FIM) in deep neural networks. The deterministic upper and lower bounds were examined via the geometric properties in low-dimensional probability simplex spaces, followed by extending to high-dimensional parameter spaces using pullback metrics. This estimator can be computed with just a single backward pass, providing significant computational benefits over existing Monte Carlo and empirical FIM methods for large networks like DistilBERT.
weaknesses: 1. Empirical validation relies solely on DistilBERT and the results from two text classification datasets. How computational overhead scales with network depth and width, memory needs for large models?


Decision: Accept (Poster)