SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \operatorname {t r} \left[ \mathbf {F} _ {1} ^ {\top} \boldsymbol {\Sigma} _ {0} \mathbf {F} _ {1} \left(\mathbf {F} _ {1} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F} _ {1}\right) ^ {- 1} \right] = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \frac {n _ {\min }}{p - n _ {\min } - 1} (C.30) \\ = \frac {\alpha_ {\min }}{1 - \alpha_ {\min }}. (C.31) \\ \end{array}
$$
This proves that, in this regime, the label noise term does not depend on data structure, matching the result of our replica computation.
Considering the Gibbs estimator, we can see immediately that the thermal variance term is  $\mathcal{O}(\beta^{-1})$  because of the fact that  $\mathbf{F}_1^\top \mathbf{X}^\top \mathbf{X}\mathbf{F}_1$  and  $\mathbf{F}_2\mathbf{F}_2^\top$  are invertible with probability one. This is consistent with Proposition 6.1.
C.3 The overdetermined regime
Finally, consider the regime in which  $n_0 < \min\{p, n_1, \ldots, n_L\}$ . Then, both  $\mathbf{X}^\top \mathbf{X}$  and  $\mathbf{F}\mathbf{F}^\top$  are invertible with probability one, and we can easily compute
$$
\begin{array}{l} \epsilon = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \lim  _ {\lambda \downarrow 0} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \left(\lambda \mathbf {I} _ {n _ {L}} + \frac {1}{n _ {0}} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \frac {1}{\sqrt {n _ {0}}} \mathbf {F} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y} - \mathbf {w} _ {*} \| ^ {2} (C.32) \\ = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \mathbf {X} ^ {\top} \boldsymbol {\xi} \| ^ {2} (C.33) \\ = \eta^ {2} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \right]. (C.34) \\ \end{array}
$$
$$
\left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \sim \mathcal {W} _ {n _ {0}} ^ {- 1} \left(\boldsymbol {\Sigma} _ {0} ^ {- 1}, p\right), \tag {C.35}
$$
so using the formula for the mean of the inverse-Wishart [50] we have
$$
\epsilon = \eta^ {2} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \right] \tag {C.36}
$$
$$
= \eta^ {2} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \frac {n _ {0}}{p - n _ {0} - 1} \tag {C.37}
$$
$$
= \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}, \tag {C.38}
$$
as we found using replicas.
Here, again, we can see that the thermal variance term for the Gibbs estimator is  $\mathcal{O}(\beta^{-1})$ , matching Proposition 6.1.
D A notational dictionary
In this appendix, we show that special cases of our general result recover the results reported in previous works. This is largely a matter of translating notation, as the conventions used in different communities are often at odds with each other.
D.1 Shallow ridgeless regression
In the shallow case  $L = 0$ , our general result for a fixed target (23) reduces to
$$
\epsilon = \left\{ \begin{array}{l l} - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} \left(\kappa_ {0}\right) + \frac {1 - \mu_ {0}}{\mu_ {0}} \eta^ {2}, & \alpha_ {0} > 1 \\ \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}, & \alpha_ {0} <   1 \end{array} \right. \tag {D.1}
$$
where, writing expectation with respect to the limiting spectral distribution of  $\Sigma_0$  as  $\mathbb{E}_{\sigma_0}$ , we recall that  $\kappa_0$  is determined by the implicit equation
$$
\frac {1}{\alpha_ {0}} = - M _ {\boldsymbol {\Sigma} _ {0}} (- \kappa_ {0}) = \mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0}}{\kappa_ {0} + \sigma_ {0}} \right], \tag {D.2}
$$
in terms of which we have
$$
\mu_ {0} = 1 - \alpha_ {0} \mathbb {E} _ {\sigma_ {0}} \left[ \left(\frac {\sigma_ {0}}{\kappa_ {0} + \sigma_ {0}}\right) ^ {2} \right], \tag {D.3}
$$
$$
\psi (z) = \lim  _ {n _ {0} \rightarrow \infty} \frac {1}{n _ {0}} \mathbf {w} _ {*} ^ {\top} \boldsymbol {\Sigma} _ {0} \left(z \mathbf {I} _ {n _ {0}} + \boldsymbol {\Sigma} _ {0}\right) ^ {- 1} \mathbf {w} _ {*}. \tag {D.4}
$$
Working in the eigenbasis of  $\Sigma_0$  and assuming that  $\| \mathbf{w}_* \|^2 = n_0$ , we introduce the weighted density
$$
\rho \left(\sigma_ {0}\right) = \lim  _ {n _ {0} \rightarrow \infty} \frac {1}{n _ {0}} \sum_ {j = 1} ^ {n _ {0}} \left(w _ {*}\right) _ {j} ^ {2} \delta \left(\sigma_ {0} - \sigma_ {j}\right) \tag {D.5}
$$
$$
\psi (z) = \mathbb {E} _ {\sigma_ {0} \sim \rho} \left[ \frac {\sigma_ {0}}{z + \sigma_ {0}} \right] \tag {D.6}
$$
$$
- \psi^ {\prime} (z) = \mathbb {E} _ {\sigma_ {0} \sim \rho} \left[ \frac {\sigma_ {0}}{(z + \sigma_ {0}) ^ {2}} \right]. \tag {D.7}
$$
We can now make contact with the result of Hastie et al. [5]. We note that those authors use an opposite definition for  $p$  and  $n$ : following the convention in the statistics literature, they use  $p$  for
the dimensionality and  $n$  for the number of examples, while we follow the convention in the physics literature of using  $n_0$  for the dimensionality and  $p$  for the number of examples. Then, Hastie et al. [5]'s  $\gamma$ , defined such that, in our terms,  $n_0 / p \to \gamma$ , is precisely our  $\alpha_0$ . Moreover, they use  $H(z)$  to denote the limiting spectral law of  $\Sigma_0$ , and  $G(z)$  to denote the law corresponding to the weighted density we define above as  $\rho$ . We note also that their  $\sigma^2$  is our  $\eta^2$ . In these terms, their Theorem 2 gives the generalization error in the overparameterized regime  $\alpha_0 > 1$  as
$$
\epsilon = \left\{1 + \alpha_ {0} c _ {0} \frac {\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0} ^ {2}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]}{\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]} \right\} \mathbb {E} _ {\sigma_ {0} \sim \rho} \left[ \frac {\sigma_ {0}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right] + \eta^ {2} \alpha_ {0} c _ {0} \frac {\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0} ^ {2}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]}{\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0}}{\left( 1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]} (D. 8)
$$
where  $c_{0}$  is defined by the implicit equation
$$
1 - \frac {1}{\alpha_ {0}} = \mathbb {E} _ {\sigma_ {0}} \left[ \frac {1}{1 + c _ {0} \alpha_ {0} \sigma_ {0}} \right]. \tag {D.9}
$$
Subtracting one from both sides, the implicit equation for  $c_{0}$  gives
$$
\frac {1}{\alpha_ {0}} = \mathbb {E} _ {\sigma_ {0}} \left[ \frac {c _ {0} \alpha_ {0} \sigma_ {0}}{1 + c _ {0} \alpha_ {0} \sigma_ {0}} \right] \tag {D.10}
$$
from which we can see that
$$
c _ {0} \alpha_ {0} = \frac {1}{\kappa_ {0}}. \tag {D.11}
$$
$$
\begin{array}{l} \mathbb {E} _ {\sigma_ {0} \sim \rho} \left[ \frac {\sigma_ {0}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right] = \kappa_ {0} ^ {2} \mathbb {E} _ {\sigma_ {0} \sim \rho} \left[ \frac {\sigma_ {0}}{\left(\kappa_ {0} + \sigma_ {0}\right) ^ {2}} \right] (D.12) \\ = - \kappa_ {0} ^ {2} \psi^ {\prime} (\kappa_ {0}) (D.13) \\ \end{array}
$$
$$
\begin{array}{l} \alpha_ {0} c _ {0} \frac {\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0} ^ {2}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]}{\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]} = \frac {\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\left(\alpha_ {0} c _ {0} \sigma_ {0}\right) ^ {2}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]}{\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\alpha_ {0} c _ {0} \sigma_ {0}}{\left(1 + c _ {0} \alpha_ {0} \sigma_ {0}\right) ^ {2}} \right]} (D.14) \\ = \frac {\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0} ^ {2}}{(\kappa_ {0} + \sigma_ {0}) ^ {2}} \right]}{\mathbb {E} _ {\sigma_ {0}} \left[ \frac {\kappa_ {0} \sigma_ {0}}{(\kappa_ {0} + \sigma_ {0}) ^ {2}} \right]} (D.15) \\ = \frac {1 - \mu_ {0}}{\alpha_ {0} \mathbb {E} _ {\sigma_ {0}} \left[ \frac {\kappa_ {0} \sigma_ {0}}{(\kappa_ {0} + \sigma_ {0}) ^ {2}} \right]} (D.16) \\ = \frac {1 - \mu_ {0}}{\alpha_ {0} \mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0}}{\kappa_ {0} + \sigma_ {0}} \right] - \alpha_ {0} \mathbb {E} _ {\sigma_ {0}} \left[ \frac {\sigma_ {0} ^ {2}}{(\kappa_ {0} + \sigma_ {0}) ^ {2}} \right]} (D.17) \\ = \frac {1 - \mu_ {0}}{\mu_ {0}}, (D.18) \\ \end{array}
$$
which proves the equivalence of our results. This also shows that we recover the results of other works on ridgeless kernel interpolation [4, 6, 7, 16, 17] that are in this setting equivalent to the results of Hastie et al. [5].
D.2 Two-layer linear random feature models with unstructured weights and isotropic targets
Another special case in which we can make contact with prior work is that of a single hidden layer  $(L = 1)$  and with target averaging. In this case, our general result (27) reduces to
$$
\bar {\epsilon} = \left\{ \begin{array}{l l} \left(1 + \frac {1}{\alpha_ {1} - 1}\right) \chi \left(\alpha_ {0}\right) + \left(\frac {1 - \mu_ {0}}{\mu_ {0}} + \frac {1}{\alpha_ {0} - 1}\right) \eta^ {2} & \alpha_ {0}, \alpha_ {1} > 1 \\ \frac {1}{1 - \alpha_ {1}} \chi \left(\frac {\alpha_ {0}}{\alpha_ {1}}\right) + \frac {\alpha_ {1}}{1 - \alpha_ {1}} \eta^ {2} & \alpha_ {1} <   1, \alpha_ {1} <   \alpha_ {0} \\ \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2} & \alpha_ {0} <   1, \alpha_ {0} <   \alpha_ {1} \end{array} \right. \tag {D.19}
$$
where in this case we find it convenient to write  $\kappa_0 / \alpha_0$  and  $\alpha_0\kappa_{\mathrm{min}} / \alpha_{\mathrm{min}}$  in terms of  $\chi (z)$ , which solves
$$
\begin{array}{l} 1 = - z M _ {\tilde {\Sigma} _ {0}} [ - z \chi (z) ] (D.20) \\ = \mathbb {E} _ {\tilde {\sigma} _ {0}} \left[ \frac {\tilde {\sigma} _ {0}}{\chi (z) + z ^ {- 1} \tilde {\sigma} _ {0}} \right]. (D.21) \\ \end{array}
$$
It is then easy to show that our result agrees with that of Maloney et al. [19]. Their notation is:
$$
M = n _ {0} \tag {D.22}
$$
$$
N = n _ {1} \tag {D.23}
$$
$$
T = p. \tag {D.24}
$$
When  $M > N, T$ , their result is, in the absence of label noise,
$$
\bar {\epsilon} = \frac {1}{M} \left\{ \begin{array}{l l} \frac {1}{1 - N / T} \Delta_ {- 1} (N, M), & N <   T \\ \frac {1}{1 - T / N} \Delta_ {- 1} (T, M), & N > T, \end{array} \right. \tag {D.25}
$$
where  $\Delta_{-1}(N,M)$  solves
$$
1 = \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \left(\Delta_ {- 1} (N, M) \mathbf {I} _ {M} + N \boldsymbol {\Sigma} _ {0}\right) ^ {- 1} \right] \tag {D.26}
$$
and similarly for  $\Delta_{-1}(T,M)$ . To map this to our results, let us re-define
$$
\bar {\Delta} _ {- 1} (N, M) \equiv \frac {1}{M} \Delta_ {- 1} (N, M), \tag {D.27}
$$
which then satisfies
$$
1 = \frac {1}{M} \operatorname {t r} \left[ \boldsymbol {\Sigma} _ {0} \left(\bar {\Delta} _ {- 1} (N, M) \mathbf {I} _ {M} + (N / M) \boldsymbol {\Sigma} _ {0}\right) ^ {- 1} \right], \tag {D.28}
$$
$$
\frac {N}{M} = - M _ {\Sigma_ {0}} \left(- \frac {M}{N} \bar {\Delta} _ {- 1} (N, M)\right) \tag {D.29}
$$
Then, we can see that, in our notation,
$$
\bar {\Delta} _ {- 1} (N, M) = \chi \left(\frac {M}{N}\right) = \chi \left(\frac {\alpha_ {0}}{\alpha_ {1}}\right), \tag {D.30}
$$
$$
\bar {\Delta} _ {- 1} (T, M) = \chi \left(\frac {M}{T}\right) = \chi \left(\alpha_ {0}\right). \tag {D.31}
$$
Then, noting that
$$
\frac {1}{1 - T / N} = \frac {1}{1 - 1 / \alpha_ {1}} = \frac {\alpha_ {1}}{\alpha_ {1} - 1} = 1 + \frac {1}{\alpha_ {1} - 1}, \tag {D.32}
$$
we can see that we recover their result in these regimes. We can also map their  $\Delta_0$  to our  $\mu_0$ . For  $T < M$ , they let
$$
\begin{array}{l} \frac {\Delta_ {0}}{1 + \Delta_ {0}} = \sum_ {j = 1} ^ {M} \frac {T \sigma_ {j} ^ {2}}{\left(T \sigma_ {j} + \Delta_ {- 1}\right) ^ {2}} (D.33) \\ = \frac {1}{T} \sum_ {j = 1} ^ {M} \frac {\sigma_ {j} ^ {2}}{\left(\sigma_ {j} + M / T \bar {\Delta} _ {- 1}\right) ^ {2}}, (D.34) \\ \end{array}
$$
hence we can see that
$$
\frac {\Delta_ {0}}{1 + \Delta_ {0}} = 1 - \mu_ {0}. \tag {D.35}
$$
Figure D.1: Phase diagrams in different parameterizations of the thermodynamic limit. (a). Phase diagram in the  $(\alpha_0, \alpha_1)$  plane. Region 1 (orange) is the overparameterized regime, Region 2 (yellow) is the bottlenecked regime, and Region 3 (green) is the overdetermined regime. (b). As in (a), but in the  $(1 / \alpha_0, \alpha_1 / \alpha_0)$  plane, matching the parameterization used in our previous work [13]. Note that the plane is divided identically, but the locations of the phases are swapped.
This mapping also enables our application of their interpolating approximate solutions for  $\Delta_{-1}$  and  $\Delta_0$  in the case of power law spectra. For a finite-size spectrum
$$
\sigma_ {j} = \frac {\sigma_ {+}}{j ^ {1 + \omega}} \quad (j = 1, \dots , M), \tag {D.36}
$$
$$
\sigma_ {+} = M ^ {1 + \omega} \sigma_ {-}, \tag {D.37}
$$
where we denote the exponent by  $\omega$  rather than  $\alpha$  as Maloney et al. [19] do to avoid clashing with our notation elsewhere, they obtain the approximate solution
$$
\frac {1}{M} \Delta_ {- 1} (N, M) = \left\{ \begin{array}{l l} \sigma_ {-} \left\{k \left[ \left(\frac {M}{N}\right) ^ {\omega} - 1 \right] + [ 2 + \omega (1 - k) ] \left(1 - \frac {N}{M}\right) \right\}, & N <   M \\ 0 & N > M \end{array} \right. \tag {D.38}
$$
$$
k = \left[ \frac {\frac {\pi}{1 + \omega}}{\sin \left(\frac {\pi}{1 + \omega}\right)} \right] ^ {1 + \omega} = \left[ \frac {1}{\operatorname {s i n c} \left(\frac {\pi}{1 + \omega}\right)} \right] ^ {1 + \omega}, \tag {D.39}
$$
which leads to the expression
$$
\chi (z) = \left\{ \begin{array}{l l} \sigma_ {-} \left\{k \left(z ^ {\omega} - 1\right) + [ 2 + \omega (1 - k) ] \left(1 - \frac {1}{z}\right) \right\}, & z > 1 \\ 0 & z <   1. \end{array} \right. \tag {D.40}
$$
Moreover, for  $T < M$ , they give the approximate solution
$$
\Delta_ {0} (T, M) = \omega + \frac {1}{M / T - 1}. \tag {D.41}
$$
By applying these results, we obtain the result claimed in the main text, (31). We note that we fix  $\sigma_{-}$  to be constant rather than  $\sigma_{+}$  as Maloney et al. [19] do, which ensures normalizability of the limiting eigenvalue distribution at the expense of diverging moments.
D.3 Deep linear models with unstructured weights and data
In [13], we studied deep Bayesian linear models with unstructured features and data. There, and in very recent work by Schröder et al. [20], a different parameterization for the thermodynamic limit
$^{2}$ In [13], we focused on the Gibbs estimator rather than on the ridgeless maximum-likelihood estimator (MLE). However, given the average generalization error for the Gibbs estimator, it is easy to obtain the generalization error for the MLE. We discuss this point in detail in Appendix A.
was used:
$$
p, n _ {0}, \dots , n _ {L} \rightarrow \infty , \quad \text {w i t h} \quad \frac {p}{n _ {0}} \rightarrow \tilde {\alpha}, \quad \frac {n _ {\ell}}{n _ {0}} \rightarrow \tilde {\gamma} _ {\ell} \quad (\ell = 1, \dots , L), \tag {D.42}
$$
where we decorate  $\tilde{\alpha}$  and  $\tilde{\gamma}_{\ell}$  with tildes to avoid confusion with parameters used elsewhere in the present work. The conversion to the parameterization used in the present work and in [30] is then given by
$$
\tilde {\alpha} = \frac {1}{\alpha_ {0}}, \tag {D.43}
$$
$$
\tilde {\gamma} _ {\ell} = \frac {\alpha_ {\ell}}{\alpha_ {0}}, \quad (\ell = 1, \dots , L). \tag {D.44}
$$
Though these parameterizations are mathematically equivalent, it is important to distinguish between them as they give phase diagrams that divide the plane identically but swap the locations of the phases, as is shown in Figure D.1. Moreover, though the parameterization used here is more convenient for the replica computation [30], that given in (D.42) is conceptually useful, as it is closer to what one does in practical machine learning settings: the input dimension  $n_0$  is fixed by the task, and one can vary the dataset size  $p$  and the network widths  $n_\ell$ . This is why we plot the phase diagrams in Figure 1 in the  $(1 / \alpha_0, \alpha_1 / \alpha_0)$  plane.
E Large-width expansions
In this appendix, we consider the limit of large width, i.e., the limit in which  $\alpha_{1},\ldots ,\alpha_{L}\to \infty$  for fixed  $\alpha_0$ . Our first task is to determine how the quantities  $\kappa_{\ell}$  behave in this limit, as it is through these inverse generating functions that the hidden layer widths enter the generalization error.
Starting from the defining equation
$$
\frac {1}{\alpha_ {\ell}} = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right] \tag {E.1}
$$
we can see that  $\kappa_{\ell}$  should tend to infinity linearly with  $\alpha_{\ell}$  as  $\alpha_{\ell}\to \infty$  . In particular, we should have
$$
\frac {\kappa_ {\ell}}{\alpha_ {\ell}} \rightarrow \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] \tag {E.2}
$$
at large widths. Then,  $\mu_{\ell}$  has limiting behavior
$$
\mu_ {\ell} = 1 - \alpha_ {\ell} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right] \tag {E.3}
$$
$$
\rightarrow 1 \tag {E.4}
$$
From this, we can see that in the infinite-width limit the generalization error of the random feature model in Proposition 3.2 reduces to that of shallow ridgeless regression as in Corollary 3.1, as we would expect.
We now want to compute the leading correction to this result. In the unstructured case, this is easy, because we have  $\kappa_{\ell} = (\alpha_{\ell} - 1)\tilde{\sigma}_{\ell}$ , hence there is an  $\mathcal{O}(1)$  correction and nothing else. More generally, we assume Laurent series behavior of the form
$$
\kappa_ {\ell} = \alpha_ {\ell} \kappa_ {\ell} ^ {1} + \kappa_ {\ell} ^ {0} + \frac {1}{\alpha_ {\ell}} \kappa_ {\ell} ^ {- 1} + \dots . \tag {E.5}
$$
Expanding, we have
$$
\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} = \frac {\tilde {\sigma} _ {\ell}}{\alpha_ {\ell} \kappa_ {\ell} ^ {1}} - \frac {\tilde {\sigma} _ {\ell} \left(\tilde {\sigma} _ {\ell} + \kappa_ {\ell} ^ {0}\right)}{\alpha_ {\ell} ^ {2} \left(\kappa_ {\ell} ^ {1}\right) ^ {2}} + \mathcal {O} \left(\alpha_ {\ell} ^ {- 3}\right) \tag {E.6}
$$
hence, if we integrate term-by-term, we have
$$
\frac {1}{\alpha_ {\ell}} = \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ]}{\alpha_ {\ell} \kappa_ {\ell} ^ {1}} - \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ^ {2} ] + \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] \kappa_ {\ell} ^ {0}}{\alpha_ {\ell} ^ {2} (\kappa_ {\ell} ^ {1}) ^ {2}} + \mathcal {O} \left(\alpha_ {\ell} ^ {- 3}\right). \tag {E.7}
$$
If we solve order-by-order, we again find that
$$
\kappa_ {\ell} ^ {1} = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] \tag {E.8}
$$
while the coefficients of all higher-order terms in  $1 / \alpha_{\ell}$  must vanish. In particular, this gives
$$
\kappa_ {\ell} ^ {0} = - \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} ^ {2} \right]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} \right]}. \tag {E.9}
$$
This computation assumes that the spectrum has finite moments, which is not the case for the heavy-tailed power law spectra considered in Corollary 5.1.
$$
\begin{array}{l} \mu_ {\ell} = 1 - \alpha_ {\ell} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right] (E.10) \\ = 1 - \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} ^ {2} \right]}{\alpha_ {\ell} \left(\kappa_ {\ell} ^ {1}\right) ^ {2}} + 2 \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} ^ {3} \right] + \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} ^ {2} \right] \kappa_ {\ell} ^ {0}}{\alpha_ {\ell} ^ {2} \left(\kappa_ {\ell} ^ {1}\right) ^ {3}} + \mathcal {O} \left(\alpha_ {\ell} ^ {- 3}\right) (E.11) \\ = 1 - \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} ^ {2} \right]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} \right] ^ {2}} \frac {1}{\alpha_ {\ell}} + \mathcal {O} \left(\alpha_ {\ell} ^ {- 2}\right). (E.12) \\ \end{array}
$$
Collecting our results, we have
$$
\kappa_ {\ell} = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] \alpha_ {\ell} \left(1 - \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ^ {2} ]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] ^ {2}} \frac {1}{\alpha_ {\ell}} + \mathcal {O} \left(\alpha_ {\ell} ^ {- 2}\right)\right) \tag {E.13}
$$
$$
\mu_ {\ell} = 1 - \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} ^ {2} \right]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \tilde {\sigma} _ {\ell} \right] ^ {2}} \frac {1}{\alpha_ {\ell}} + \mathcal {O} \left(\alpha_ {\ell} ^ {- 2}\right). \tag {E.14}
$$
Each term in these expansions has the expected behavior under rescaling: if we let  $\tilde{\Sigma}_{\ell}^{\prime} = \tau_{\ell}\tilde{\Sigma}_{\ell}$  for  $\tau_{\ell} > 0$ , we have  $\kappa_{\ell}^{\prime} = \tau_{\ell}\kappa_{\ell}$  and  $\mu_{\ell}^{\prime} = \mu_{\ell}$ .
Then, substituting these expansions into (23), we find that the generalization error of an RFM in the ridgeless limit expands at large widths as
$$
\begin{array}{l} \epsilon = - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} (\kappa_ {0}) + \frac {1 - \mu_ {0}}{\mu_ {0}} \eta^ {2} \\ + \left(\sum_ {\ell = 1} ^ {L} \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ^ {2} ]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] ^ {2}} \frac {1}{\alpha_ {\ell}}\right) (\kappa_ {0} \psi (\kappa_ {0}) + \eta^ {2}) \\ + \mathcal {O} \left(\alpha_ {1} ^ {- 2}, \dots , \alpha_ {L} ^ {- 2}\right) \tag {E.15} \\ \end{array}
$$
For an RFM trained using the Gibbs estimator, as considered in Proposition 6.1, we find that
$$
\begin{array}{l} \epsilon_ {\mathrm {B R F M}} = - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} (\kappa_ {0}) + \frac {1 - \mu_ {0}}{\mu_ {0}} \eta^ {2} + \frac {\kappa_ {0}}{\alpha_ {0}} \varsigma^ {2} \\ + \left(\sum_ {\ell = 1} ^ {L} \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ^ {2} ]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] ^ {2}} \frac {1}{\alpha_ {\ell}}\right) \left(\kappa_ {0} \psi (\kappa_ {0}) + \eta^ {2} - \frac {\kappa_ {0}}{\alpha_ {0}} \zeta^ {2}\right) \\ + \mathcal {O} \left(w ^ {- 2}\right) \tag {E.16} \\ \end{array}
$$
$$
\varsigma^ {2} \equiv \prod_ {\ell = 1} ^ {L} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ], \tag {E.17}
$$
upon expanding the thermal variance term
$$
\begin{array}{l} \prod_ {\ell = 0} ^ {L} \frac {\kappa_ {\ell}}{\alpha_ {\ell}} = \frac {\kappa_ {0}}{\alpha_ {0}} \left[ \prod_ {\ell = 1} ^ {L} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] \right] \\ - \frac {\kappa_ {0}}{\alpha_ {0}} \left[ \prod_ {\ell = 1} ^ {L} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] \right] \sum_ {\ell = 1} ^ {L} \frac {\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ^ {2} ]}{\mathbb {E} _ {\tilde {\sigma} _ {\ell}} [ \tilde {\sigma} _ {\ell} ] ^ {2}} \frac {1}{\alpha_ {\ell}} \\ + \mathcal {O} \left(w ^ {- 2}\right). \tag {E.18} \\ \end{array}
$$
Here, we denote by  $\mathcal{O}(w^{-2})$  all terms of  $\mathcal{O}(\alpha_{\ell}^{-2})$  for a given layer  $\ell = 1,\ldots ,L$  or terms of  $\mathcal{O}(\alpha_{\ell}^{-1}\alpha_{\ell^{\prime}}^{-1})$  for two different layers  $\ell ,\ell^{\prime}$
F Numerical methods