SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\psi (z) = \lim  _ {n _ {0} \rightarrow \infty} \frac {1}{n _ {0}} \tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\Sigma} _ {0} \left(z \mathbf {I} _ {n _ {0}} + \tilde {\Sigma} _ {0}\right) ^ {- 1} \tilde {\mathbf {w}} _ {*}. \tag {A.50}
$$
To obtain the average generalization error for the Gibbs estimator in the zero-temperature limit, we must account for the effect of  $q_{0}$  in the regime  $\alpha_{\ell} > 1$ , as in all other regimes it is  $q_{0} \sim \mathcal{O}(1 / \beta)$ . But, we recognize that
$$
q _ {0} = \prod_ {j = 0} ^ {L} \frac {- 1}{\alpha_ {j}} M _ {\hat {\mathbf {S}} _ {j}} ^ {- 1} \left(\frac {- 1}{\alpha_ {j}}\right) = \prod_ {\ell = 0} ^ {L} \frac {\kappa_ {\ell}}{\alpha_ {\ell}} \tag {A.51}
$$
from the definition above, hence we conclude that
$$
\epsilon_ {\mathrm {B R F M}} = \epsilon_ {\text {r i d g e l e s s}} + \left\{ \begin{array}{l l} \prod_ {\ell = 0} ^ {L} \frac {\kappa_ {\ell}}{\alpha_ {\ell}}, & \alpha_ {0}, \alpha_ {\min } > 1 \\ 0, & \alpha_ {\min } <   1, \alpha_ {\min } <   \alpha_ {0} \\ 0, & \alpha_ {0} <   1, \alpha_ {0} <   \alpha_ {\min }. \end{array} \right. \tag {A.52}
$$
A.4 Physical interpretation of the order parameters and thermal bias-variance decomposition
With these results in hand, we now comment on the interpretation of the replica uniform and replica non-uniform contributions to
$$
\mathbf {C} _ {0} = q _ {0} \mathbf {I} _ {m} + c _ {0} \mathbf {1} _ {m} \mathbf {1} _ {m} ^ {\top}. \tag {A.53}
$$
At the saddle point, we have
$$
\left(C _ {0}\right) ^ {a b} = \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \left(\mathbf {F} \mathbf {v} ^ {a} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \mathbf {v} ^ {b} - \mathbf {w} _ {*}\right) \right\rangle_ {\beta}, \tag {A.54}
$$
where  $\langle \cdot \rangle_{\beta}$  denotes the expectation with respect to the replicated Gibbs measure at inverse temperature  $\beta$ . Under the replica-symmetric Ansatz, considering off-diagonal elements  $a \neq b$ , we can use the fact that the replicas are initially uncoupled and identical to write
$$
\begin{array}{l} c _ {0} = C _ {0} ^ {a b} (A.55) \\ = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left(\mathbf {F} \langle \mathbf {v} ^ {a} \rangle_ {\beta} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \langle \mathbf {v} ^ {b} \rangle_ {\beta} - \mathbf {w} _ {*}\right) (A.56) \\ = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left(\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}\right) (A.57) \\ = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}) \| ^ {2}. (A.58) \\ \end{array}
$$
$$
\begin{array}{l} q _ {0} = C _ {0} ^ {a a} - C _ {0} ^ {a b} (A.59) \\ = \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \left(\mathbf {F} \mathbf {v} ^ {a} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \mathbf {v} ^ {a} - \mathbf {w} _ {*}\right) \right\rangle_ {\beta} - c _ {0} (A.60) \\ = \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \left(\mathbf {F} \delta \mathbf {v} + \mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \delta \mathbf {v} + \mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}\right) \right\rangle_ {\beta} - c _ {0} (A.61) \\ = \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \left(\mathbf {F} \delta \mathbf {v}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \delta \mathbf {v}\right) \right\rangle_ {\beta} + \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left(\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}\right) - c _ {0} (A.62) \\ = \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \left(\mathbf {F} \delta \mathbf {v}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \delta \mathbf {v}\right) \right\rangle_ {\beta} (A.63) \\ = \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \delta \mathbf {v} \| ^ {2} \right\rangle_ {\beta}, (A.64) \\ \end{array}
$$
where we write  $\delta \mathbf{v} = \mathbf{v} - \langle \mathbf{v}\rangle_{\beta}$ . Therefore, at the saddle point,  $c_{0}$  and  $q_{0}$  correspond exactly to the bias and variance terms in the thermal bias-variance decomposition of the generalization error:
$$
\mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \mathbf {v} - \mathbf {w} _ {*}) \| ^ {2} \right\rangle_ {\beta} = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}) \| ^ {2} + \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \delta \mathbf {v} \| ^ {2} \right\rangle_ {\beta}. \tag {A.65}
$$
This makes concrete an argument which was presented only intuitively in [13]. As a result, if one considered the Bayesian MMSE estimator  $\hat{\mathbf{v}} = \langle \mathbf{v}\rangle_{\beta}$ , the zero-temperature generalization error would simply coincide with that for the ridgeless estimator.
B Properties of the inverse generating functions
Here, we record a few useful properties of the inverse spectral generating functions
$$
\frac {1}{\alpha_ {\ell}} = - M _ {\tilde {\mathbf {S}} _ {\ell}} (- \kappa_ {\ell}) = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right] \tag {B.1}
$$
and their relatives
$$
\mu_ {\ell} = - \alpha_ {\ell} \kappa_ {\ell} M _ {\boldsymbol {\Sigma} _ {\ell}} ^ {\prime} (- \kappa_ {\ell}) = 1 - \alpha_ {\ell} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right]. \tag {B.2}
$$
These results are used in the proofs of Lemmas
B.1 Dependence on width
Implicitly differentiating the self-consistent equation defining  $\kappa_{\ell}$ , we have
$$
\frac {d \kappa_ {\ell}}{d \left(1 / \alpha_ {\ell}\right)} = - \frac {1}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\left(\kappa_ {\ell} + \tilde {\sigma} _ {\ell}\right) ^ {2}} \right]}, \tag {B.3}
$$
showing that  $\kappa_{\ell}$  is a decreasing function of  $1 / \alpha_{\ell}$ . As  $1 / \alpha_{\ell} \downarrow 0$ , we should have  $\kappa_{\ell} \uparrow \infty$ , while as  $1 / \alpha_{\ell} \uparrow 1$ , we should have  $\kappa_{\ell} \downarrow 0$ .
B.2 Behavior under rescaling
Consider the re-scaling  $\tilde{\Sigma}_{\ell}^{\prime} = \tau_{\ell}\tilde{\Sigma}_{\ell}$  for  $\tau_{\ell} > 0$ . Then, we have  $\kappa_{\ell}$  and  $\tilde{\kappa}_{\ell}^{\prime}$  given by
$$
\frac {1}{\alpha_ {\ell}} = - M _ {\tilde {\Sigma} _ {\ell}} (- \kappa_ {\ell}) = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right] \tag {B.4}
$$
$$
\frac {1}{\alpha_ {\ell}} = - M _ {\tilde {\mathbf {S}} _ {\ell} ^ {\prime}} \left(- \kappa_ {\ell} ^ {\prime}\right) = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tau_ {\ell} \tilde {\sigma} _ {\ell}}{\kappa_ {\ell} ^ {\prime} + \tau_ {\ell} \tilde {\sigma} _ {\ell}} \right] \tag {B.5}
$$
respectively. We can then see that we should have
$$
\kappa_ {\ell} ^ {\prime} = \tau_ {\ell} \kappa_ {\ell}. \tag {B.6}
$$
B.3 Bound on  $\kappa_{\ell}$  in terms of isotropic spectrum
We now prove that
$$
\kappa_ {\ell} \leq (\alpha_ {\ell} - 1) \mathbb {E} [ \tilde {\sigma} _ {\ell} ] \tag {B.7}
$$
in the relevant regime  $\alpha_{\ell} > 1$  . For any  $z > 0$
$$
\tilde {\sigma} _ {\ell} \mapsto \frac {\tilde {\sigma} _ {\ell}}{(z + \tilde {\sigma} _ {\ell})} \tag {B.8}
$$
is a concave function of  $\tilde{\sigma}_{\ell} \geq 0$ , hence Jensen's inequality implies that
$$
\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{(z + \tilde {\sigma} _ {\ell})} \right] \leq \frac {\mathbb {E} [ \tilde {\sigma} _ {\ell} ]}{z + \mathbb {E} [ \tilde {\sigma} _ {\ell} ]}. \tag {B.9}
$$
Then, note that
$$
z \mapsto \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{z + \tilde {\sigma} _ {\ell}} \right] \tag {B.10}
$$
$$
z \mapsto \frac {\mathbb {E} [ \tilde {\sigma} _ {\ell} ]}{z + \mathbb {E} [ \tilde {\sigma} _ {\ell} ]} \tag {B.11}
$$
are both decreasing functions of  $z \geq 0$ , and both are equal to 1 when  $z = 0$ . Thus, if  $\kappa_{\ell} > 0$  solves
$$
\frac {1}{\alpha_ {\ell}} = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right] \tag {B.12}
$$
as specified by its definition and  $\bar{\kappa}_{\ell} > 0$  solves
$$
\frac {1}{\alpha_ {\ell}} = \frac {\mathbb {E} [ \tilde {\sigma} _ {\ell} ]}{\bar {\kappa} _ {\ell} + \mathbb {E} [ \tilde {\sigma} _ {\ell} ]}, \tag {B.13}
$$
we must have
$$
\kappa_ {\ell} \leq \bar {\kappa} _ {\ell}. \tag {B.14}
$$
But, we can easily see that  $\bar{\kappa}_{\ell} = (\alpha_{\ell} - 1)\mathbb{E}[\bar{\sigma}_{\ell}]$ , hence the claim follows.
B.4 Bound on  $\mu_{\ell}$  terms of isotropic spectrum
We next prove that
$$
\mu_ {\ell} \leq 1 - \frac {1}{\alpha_ {\ell}} \tag {B.15}
$$
in the relevant regime  $\alpha_{\ell} > 1$ . By definition, we have
$$
\mu_ {\ell} = 1 - \alpha_ {\ell} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right]. \tag {B.16}
$$
By Jensen's inequality and the definition of  $\kappa_{\ell}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right] \geq \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right] ^ {2} (B.17) \\ = \frac {1}{\alpha_ {\ell} ^ {2}}. (B.18) \\ \end{array}
$$
As  $\alpha_{\ell} > 1$  by assumption, this bound is always positive. Therefore, we conclude the desired claim.
C Simplifying the generalization error for fixed data
In this appendix, we show how the ridgeless generalization error can be simplified in each regime for fixed data. Using the solution to the ridge regression problem (6),
$$
\hat {\mathbf {v}} = \frac {1}{\sqrt {n _ {0}}} \left(\lambda \boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \frac {1}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y}, \tag {C.1}
$$
$$
\begin{array}{l} \epsilon = \lim  _ {\lambda \downarrow 0} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \hat {\mathbf {v}} - \mathbf {w} _ {*}) \| ^ {2} (C.2) \\ = \lim  _ {\lambda \downarrow 0} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left\| \frac {1}{\sqrt {n _ {0}}} \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \left(\lambda \boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \frac {1}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \right\| ^ {2}. (C.3) \\ \end{array}
$$
Following our discussion in the main text, we may set  $\Gamma_{L + 1} = \mathbf{I}_{n_L}$  without loss of generality, as otherwise we may re-define  $\Sigma_L$ . Then, we have
$$
\epsilon = \lim  _ {\lambda \downarrow 0} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left\| \frac {1}{\sqrt {n _ {0}}} \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \left(\lambda \mathbf {I} _ {n _ {L}} + \frac {1}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \right\| ^ {2}. \tag {C.4}
$$
In the subsequent sections, we will simplify this expression in each regime.
For the Gibbs estimator, we must account for the additional contribution to the generalization error from thermal variance. Following our previous work [13], we may compute the bias and variance terms directly from the posterior moment generating function of the readout weight vector,
$$
\begin{array}{l} \mathcal {Z} (\mathbf {j}) \propto \int d \mathbf {v} \exp \left(- \frac {\beta}{2} \| n _ {0} ^ {- 1 / 2} \mathbf {X F v} - \mathbf {y} \| ^ {2} - \frac {1}{2} \| \boldsymbol {\Gamma} _ {L + 1} ^ {- 1 / 2} \mathbf {v} \| ^ {2} + \mathbf {j} ^ {\top} \mathbf {v}\right) (C.5) \\ \propto \exp \left(\beta n _ {0} ^ {- 1 / 2} \mathbf {y} ^ {\top} \mathbf {X F} \left(\boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \beta n _ {0} ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X F}\right) ^ {- 1} \mathbf {j} \right. \\ \left. + \frac {1}{2} \mathbf {j} ^ {\top} \left(\boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \beta n _ {0} ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X F}\right) ^ {- 1} \mathbf {j}\right), (C.6) \\ \end{array}
$$
$$
\langle \mathbf {v} \rangle_ {\beta} = \frac {1}{\sqrt {n _ {0}}} \left(\frac {1}{\beta} \boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \frac {1}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y} \tag {C.7}
$$
$$
\left\langle \mathbf {v} \mathbf {v} ^ {\top} \right\rangle_ {\beta} - \left\langle \mathbf {v} \right\rangle_ {\beta} \left\langle \mathbf {v} \right\rangle_ {\beta} ^ {\top} = \left(\boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \frac {\beta}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1}. \tag {C.8}
$$
We then can see that
$$
\langle \mathbf {v} \rangle_ {\beta} = \hat {\mathbf {v}} \Bigg | _ {\lambda = 1 / \beta}, \tag {C.9}
$$
which is precisely in agreement with the conversion in Appendix A.2. Considering the thermal bias-variance decomposition of the generalization error for the Gibbs estimator,
$$
\mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \mathbf {v} - \mathbf {w} _ {*}) \| ^ {2} \right\rangle_ {\beta} = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \langle \mathbf {v} \rangle_ {\beta} - \mathbf {w} _ {*}) \| ^ {2} + \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \delta \mathbf {v} \| ^ {2} \right\rangle_ {\beta}, \tag {C.10}
$$
we can then see that the bias term at zero temperature coincides exactly with the generalization error of the ridgeless estimator, as we found in Appendix A. The variance term is
$$
\lim  _ {\beta \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \delta \mathbf {v} \| ^ {2} \right\rangle_ {\beta} = \lim  _ {\beta \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \left(\boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \frac {\beta}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \right]. \tag {C.11}
$$
In both the bias and variance terms, we can see that we may set  $\Gamma_{L + 1} = \mathbf{I}_{n_L}$  without loss of generality, as otherwise we may simply re-scale  $\pmb{\Sigma}_{L}$  as discussed in Lemma 2.1. Then, we need only consider the thermal variance term
$$
\lim  _ {\beta \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \delta \mathbf {v} \| ^ {2} \right\rangle_ {\beta} = \lim  _ {\beta \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \left(\mathbf {I} _ {n _ {L}} + \frac {\beta}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \right]. \tag {C.12}
$$
Here, we leave the thermodynamic limit implicit to allow the expression to fit on a single line.
C.1 The overparameterized regime
First, consider the regime  $p < \min \{n_0,\dots ,n_L\}$ . Here, we expect the kernel
$$
\mathbf {K} = \frac {1}{n _ {0}} \mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \tag {C.13}
$$
to be invertible with probability one in the thermodynamic limit, and with overwhelming probability at large but finite size [50]. Applying the push-through identity and passing to the ridgeless limit, we have
$$
\begin{array}{l} \epsilon = \lim  _ {\lambda \downarrow 0} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left\| \frac {1}{\sqrt {n _ {0}}} \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \left(\lambda \mathbf {I} _ {p} + \frac {1}{n _ {0}} \mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top}\right) ^ {- 1} \mathbf {y} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \right\| ^ {2} (C.14) \\ = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left\| \sqrt {n _ {0}} \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \left(\mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top}\right) ^ {- 1} \mathbf {y} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \right\| ^ {2}. (C.15) \\ \end{array}
$$
Averaging over label noise, we have
$$
\begin{array}{l} \epsilon = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \left\| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \left(\mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top}\right) ^ {- 1} \mathbf {X} \mathbf {w} _ {*} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \right\| ^ {2} \\ + \eta^ {2} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \left\| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \left(\mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top}\right) ^ {- 1} \right\| ^ {2}. \tag {C.16} \\ \end{array}
$$
Turning our attention to the Gibbs estimator, we can use the Woodbury identity to write the thermal variance term as
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \left\langle \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} \delta \mathbf {v} \| ^ {2} \right\rangle_ {\beta} \\ = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \left(\mathbf {I} _ {n _ {L}} + \frac {\beta}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \right] (C.17) \\ = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \mathbf {F} ^ {\top} \right] - \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \left(\beta^ {- 1} \mathbf {I} _ {n _ {L}} + \frac {1}{n _ {0}} \mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top}\right) ^ {- 1} \mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \right] (C.18) \\ = \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \mathbf {F} ^ {\top} \right] - \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \left(\frac {1}{n _ {0}} \mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top}\right) ^ {- 1} \mathbf {X} \mathbf {F} \mathbf {F} ^ {\top} \right] + \mathcal {O} (\beta^ {- 1}), (C.19) \\ \end{array}
$$
where the thermodynamic limit is implied [13]. Therefore, in this regime we do not expect the thermal variance term to vanish, consistent with Proposition 6.1.
C.2 The bottlenecked regime
If  $\min \{n_1,\dots ,n_L\} <  \min \{n_0,p\}$  , then the situation is slightly more complicated. Let
$$
\ell_ {\min } = \underset {\ell} {\arg \min } n _ {\ell} \tag {C.20}
$$
be the index of the narrowest hidden layer. Then, let
$$
\mathbf {F} _ {1} = \frac {1}{\sqrt {n _ {1} \cdots n _ {\ell_ {\operatorname* {m i n}}}}} \mathbf {U} _ {1} \dots \mathbf {U} _ {\ell_ {\min }} \in \mathbb {R} ^ {n _ {0} \times n _ {\min }} \tag {C.21}
$$
$$
\mathbf {F} _ {2} = \frac {1}{\sqrt {n _ {\ell_ {\operatorname* {m i n}} + 1} \cdots n _ {L}}} \mathbf {U} _ {\ell_ {\operatorname* {m i n}} + 1} \dots \mathbf {U} _ {L} \in \mathbb {R} ^ {n _ {\min } \times n _ {L}} \tag {C.22}
$$
$$
\mathbf {F} = \mathbf {F} _ {1} \mathbf {F} _ {2}. \tag {C.23}
$$
Then, the matrices  $\mathbf{F}_1^\top \mathbf{X}^\top \mathbf{X}\mathbf{F}_1$  and  $\mathbf{F}_2\mathbf{F}_2^\top$  are invertible with probability one, and upon passing to the ridgeless limit we have
$$
\epsilon = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \sqrt {n _ {0}} \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} _ {1} \left(\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}\right) ^ {- 1} \mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \| ^ {2}. \tag {C.24}
$$
Averaging over the label noise,
$$
\begin{array}{l} \epsilon = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} _ {1} (\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}) ^ {- 1} \mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {w} _ {*} - \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {w} _ {*} \| ^ {2} \\ + \eta^ {2} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} _ {1} \left(\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}\right) ^ {- 1} \mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \| ^ {2} \tag {C.25} \\ \end{array}
$$
Focusing on the label noise term, we have
$$
\mathbb {E} _ {\mathcal {D}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \mathbf {F} _ {1} \left(\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}\right) ^ {- 1} \mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \| ^ {2} = \mathbb {E} _ {\mathcal {D}} \operatorname {t r} \left[ \mathbf {F} _ {1} ^ {\top} \boldsymbol {\Sigma} _ {0} \mathbf {F} _ {1} \left(\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}\right) ^ {- 1} \right]. \tag {C.26}
$$
Then, using the fact that
$$
\mathbf {X} ^ {\top} \mathbf {X} \sim \mathcal {W} _ {n _ {0}} (\Sigma_ {0}, p), \tag {C.27}
$$
$$
\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1} \sim \mathcal {W} _ {n _ {\min }} \left(\mathbf {F} _ {1} ^ {\top} \boldsymbol {\Sigma} _ {0} \mathbf {F} _ {1}, p\right). \tag {C.28}
$$
Then, as we expect the matrix  $\mathbf{F}_1^\top \pmb{\Sigma}_0\mathbf{F}_1$  to be invertible with overwhelming probability, the standard formula for the mean of an inverse-Wishart distribution [50] gives
$$
\mathbb {E} _ {\mathcal {D}} \left(\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}\right) ^ {- 1} = \frac {1}{p - n _ {\min } - 1} \left(\mathbf {F} _ {1} ^ {\top} \boldsymbol {\Sigma} _ {0} \mathbf {F} _ {1}\right) ^ {- 1}, \tag {C.29}
$$