SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[42] Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. Learning curves of generic features maps for realistic datasets with a teacher-student model. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, 2021. URL https://proceedings.neurips.cc/paper/2021/bit/9704a4fc48ae88598cbcdf57f3def-Abstract.html.
[43] Jacob A. Zavatone-Veth and Cengiz Pehlevan. Learning curves for deep structured Gaussian feature models. arXiv, 2023. doi:https://doi.org/10.48550/arXiv.2303.00564.
[44] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distribution generalization in kernel regression. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 12600-12612. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/691dcb1d65f31967a874d18383b9da75-Paper.pdf.
[45] Jacob A Zavatone-Veth, Abdulkadir Canatar, Benjamin S Ruben, and Cengiz Pehlevan. Asymptotics of representation learning in finite Bayesian neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2022(11):114008, nov 2022. doi:10.1088/1742-5468/ac98a6. URL https://dx.doi.org/10.1088/1742-5468/ac98a6.
[46] Jacob A Zavatone-Veth and Cengiz Pehlevan. Depth induces scale-averaging in overparameterized linear Bayesian neural networks. In Asilomar Conference on Signals, Systems, and Computers, volume 55, 2021. doi:10.1109/IEEECONF53345.2021.9723137.
[47] S. Ariosto, R. Pacelli, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo. Statistical mechanics of deep learning beyond the infinite-width limit. arXiv, 2022. doi:10.48550/ARXIV.2209.04882. URL https://arxiv.org/abs/2209.04882.
[48] Jacob A. Zavatone-Veth, Sheng Yang, Julian A. Rubinfien, and Cengiz Pehlevan. Neural networks learn to magnify areas near decision boundaries. arXiv, 2023. doi:10.48550/ARXIV.2301.11375. URL https://arxiv.org/abs/2301.11375.
[49] Lukas Braun, Clémentine Carla Juliette Dominé, James E Fitzgerald, and Andrew M Saxe. Exact learning dynamics of deep linear networks with prior knowledge. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, volume 35, 2022. URL https://openreview.net/forum?id=1Jx2vng-KiC.
[50] Robb J Muirhead. Aspects of Multivariate Statistical Theory. John Wiley & Sons, 2009. doi:10.1002/9780470316559.
A Derivation of Proposition 3.2
In this Appendix, we sketch our replica-theory approach to computing the learning curves, which leads to Proposition 3.2. Many of the steps of this calculation are all but identical to our previous works on replica approaches to the spectra of product Wishart random matrices [30], and on unstructured deep Gaussian random feature models [13], so we will sketch the major steps rather than spelling out all the details of the algebra.
A.1 Gibbs distribution and replica free energy
We start by introducing a Gibbs distribution at fictitious inverse temperature  $\beta$  associated with the ridge regression loss
$$
L = \frac {1}{2} \left\| \frac {1}{\sqrt {n _ {0}}} \mathbf {X F v} - \mathbf {y} \right\| ^ {2} + \frac {\lambda}{2} \| \boldsymbol {\Gamma} _ {L + 1} ^ {- 1 / 2} \mathbf {v} \| _ {2} ^ {2}, \tag {A.1}
$$
with partition function
$$
Z (\beta , \mathcal {D}) \propto \int d \mathbf {v} e ^ {- \beta L (\mathbf {v}, \mathcal {D})}, \tag {A.2}
$$
where we denote by  $\mathcal{D}$  all randomness in the problem. For any  $\lambda > 0$ , in the zero-temperature limit  $\beta \to \infty$ , this Gibbs distribution concentrates around the unique minimum of the loss  $E$  [29, 33].
For the purpose of the replica computation, it is convenient to consider instead the partition function of the posterior of a related Bayesian model, which corresponds to absorbing  $\beta \lambda$  into a redefinition of  $\Gamma_{L + 1}$ , and treating the ridge penalty as a Gaussian prior
$$
\mathbf {v} \sim_ {\text {p r i o r}} \mathcal {N} (\mathbf {0}, \boldsymbol {\Gamma} _ {L + 1}). \tag {A.3}
$$
We can then recover the partition function of the ridge regression model by undoing the rescaling:  $\Gamma_{L + 1}\gets \Gamma_{L + 1} / (\beta \lambda)$ . Without this re-scaling—i.e., in the case in which the prior variance is held fixed as the temperature goes to zero—this is the Gibbs estimator in the zero-temperature limit, i.e., a Bayesian model with Gaussian likelihood of vanishing variance [13, 34, 37, 45, 46].
This gives us the partition function
$$
Z = \mathbb {E} _ {\mathbf {v} \sim \mathcal {N} (\mathbf {0}, \mathbf {r} _ {L + 1})} \exp \left[ - \frac {\beta}{2} \sum_ {\mu = 1} ^ {p} [ g (\mathbf {x} _ {\mu}; \mathbf {v}, \mathbf {F}) - y _ {\mu} ] ^ {2} \right], \tag {A.4}
$$
which is the extension to structured priors of the Gibbs estimator partition function considered in [13]. By standard arguments, we expect the quenched free energy
$$
f = - \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \frac {1}{p} \log Z, \tag {A.5}
$$
to be self-averaging in the thermodynamic limit, i.e.,  $f = \mathbb{E}_{\mathcal{D}}f$  almost surely [29, 33]. To compute the limiting quenched average, we use the replica trick, and write
$$
f = - \lim  _ {m \rightarrow 0} \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \frac {1}{p m} \log \mathbb {E} _ {\mathcal {D}} Z ^ {m}, \tag {A.6}
$$
where we evaluate the moments  $\mathbb{E}_{\mathcal{D}}Z^{m}$  for positive integer  $m$ , and assume that they can be analytically continued to  $m\to 0$
Following previous work [13, 30], we can compute the quenched averages and integrate out the weights by introducing order parameters
$$
\left(C _ {0}\right) ^ {a b} = \frac {1}{n _ {0}} \left(\mathbf {F} \mathbf {v} ^ {a} - \mathbf {w} _ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {0} \left(\mathbf {F} \mathbf {v} ^ {b} - \mathbf {w} _ {*}\right), \tag {A.7}
$$
for  $\ell = 0$
$$
\left(C _ {\ell}\right) ^ {a b} = \frac {1}{n _ {\ell} \cdots n _ {L}} \left(\mathbf {v} ^ {a}\right) ^ {\top} \mathbf {U} _ {L} ^ {\top} \dots \mathbf {U} _ {\ell + 1} ^ {\top} \boldsymbol {\Sigma} _ {\ell} \mathbf {U} _ {\ell + 1} \dots \mathbf {U} _ {L} \mathbf {v} ^ {b} \tag {A.8}
$$
for  $\ell = 1,\ldots ,L - 1$  and
$$
\left(C _ {L}\right) ^ {a b} = \frac {1}{n _ {L}} \left(\mathbf {v} ^ {a}\right) ^ {\top} \boldsymbol {\Sigma} _ {L} \mathbf {v} ^ {b}, \tag {A.9}
$$
along with corresponding Lagrange multipliers  $\hat{\mathbf{C}}_{\ell}$ , which yields
$$
\mathbb {E} _ {\mathcal {D}} Z ^ {m} = \int \frac {d \mathbf {C} _ {0} d \hat {\mathbf {C}} _ {0}}{(4 \pi i / n _ {0}) ^ {m (m + 1) / 2}} \int \frac {d \mathbf {C} _ {1} d \hat {\mathbf {C}} _ {1}}{(4 \pi i / n _ {1}) ^ {m (m + 1) / 2}} \dots \int \frac {d \mathbf {C} _ {L} d \hat {\mathbf {C}} _ {L}}{(4 \pi i / n _ {L}) ^ {m (m + 1) / 2}} \exp \left[ - \frac {p m}{2} S \right] \tag {A.10}
$$
$$
\begin{array}{l} m S = \log \det (\mathbf {I} _ {m} + \beta \mathbf {C} _ {0} + \beta \eta^ {2} \mathbf {1} _ {m} \mathbf {1} _ {m} ^ {\top}) \\ - \alpha_ {0} \frac {1}{n _ {0}} \mathrm {v} (\tilde {\mathbf {w}} _ {*} \mathbf {1} _ {m} ^ {\top}) ^ {\top} [ \hat {\mathbf {C}} _ {0} \otimes \tilde {\mathbf {S}} _ {0} ] [ \mathbf {I} _ {m n _ {0}} - \mathbf {C} _ {1} \hat {\mathbf {C}} _ {0} \otimes \tilde {\mathbf {S}} _ {0} ] ^ {- 1} \mathrm {v} (\tilde {\mathbf {w}} _ {*} \mathbf {1} _ {m} ^ {\top}) \\ + \sum_ {\ell = 0} ^ {L} \alpha_ {\ell} \left[ \operatorname {t r} \left(\mathbf {C} _ {\ell} \hat {\mathbf {C}} _ {\ell}\right) + \frac {1}{n _ {\ell}} \log \det  \left(\mathbf {I} _ {m n _ {\ell}} - \mathbf {C} _ {\ell + 1} \hat {\mathbf {C}} _ {\ell} \otimes \tilde {\mathbf {S}} _ {\ell}\right) \right], \tag {A.11} \\ \end{array}
$$
where we let  $\mathbf{C}_{L + 1} = \mathbf{I}_m$  and
$$
\tilde {\boldsymbol {\Sigma}} _ {\ell} = \boldsymbol {\Gamma} _ {\ell + 1} ^ {1 / 2} \boldsymbol {\Sigma} _ {\ell} \boldsymbol {\Gamma} _ {\ell} ^ {1 / 2} \tag {A.12}
$$
for  $\ell = 0,\ldots ,L$ . We note that  $\otimes$  here denotes the Kronecker product, and we use the convention that the standard matrix product has higher precedence than the Kronecker product, i.e.,  $\mathbf{AB}\otimes \mathbf{C} = (\mathbf{AB})\otimes \mathbf{C}$ . Importantly, the quantity of interest—the generalization error—is simply given by the diagonal elements of  $\mathbf{C}_0$ , i.e.,  $\epsilon = (C_0)^{aa}$ . Therefore, if we can solve for the order parameters at zero temperature, we will obtain the generalization error.
In the thermodynamic limit, the integral over these order parameters can be evaluated using the method of steepest descent. We make a replica symmetric Ansatz, and seek saddle points of the form
$$
\mathbf {C} _ {\ell} = q _ {\ell} \mathbf {I} _ {m} + c _ {\ell} \mathbf {1} _ {m} \mathbf {1} _ {m} ^ {\top}, \tag {A.13}
$$
$$
\hat {\mathbf {C}} _ {\ell} = \hat {q} _ {\ell} \mathbf {I} _ {m} + \hat {c} _ {\ell} \mathbf {1} _ {m} \mathbf {1} _ {m} ^ {\top}. \tag {A.14}
$$
Under this Ansatz, we have
$$
\begin{array}{l} S = \log (1 + \beta q _ {0}) + \frac {\beta (c _ {0} + \eta^ {2})}{1 + \beta q _ {0}} \\ - \alpha_ {0} \frac {1}{n _ {0}} (\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\mathbf {\Sigma}} _ {0} (\mathbf {I} _ {n _ {0}} - q _ {1} \hat {q} _ {0} \tilde {\mathbf {\Sigma}} _ {0}) ^ {- 1} \tilde {\mathbf {w}} _ {*}) \hat {q} _ {0} \\ + \sum_ {\ell = 0} ^ {L} \alpha_ {\ell} \left(q _ {\ell} \hat {q} _ {\ell} + q _ {\ell} \hat {c} _ {\ell} + c _ {\ell} \hat {q} _ {\ell} + \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \log \left(1 - q _ {\ell + 1} \hat {q} _ {\ell} \tilde {\sigma} _ {\ell}\right) \right. \\ \left. - \left(q _ {\ell + 1} \hat {c} _ {\ell} + c _ {\ell + 1} \hat {q} _ {\ell}\right) \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{1 - q _ {\ell + 1} \hat {q} _ {\ell} \tilde {\sigma} _ {\ell}} \right]\right) \\ + \mathcal {O} (m) \tag {A.15} \\ \end{array}
$$
to leading order in  $m$ , where we recall the boundary condition  $q_{L + 1} = 1$ ,  $c_{L + 1} = 0$  [30]. The resulting saddle point equations can be simplified to give a closed system for the replica non-uniform components,
$$
\alpha_ {0} \hat {q} _ {0} = - \frac {\beta}{1 + \beta q _ {0}} \tag {A.16}
$$
$$
\alpha_ {\ell} \hat {q} _ {\ell} = \alpha_ {\ell - 1} \hat {q} _ {\ell - 1} \mathbb {E} _ {\tilde {\sigma} _ {\ell - 1}} \left[ \frac {\tilde {\sigma} _ {\ell - 1}}{1 - q _ {\ell} \hat {q} _ {\ell - 1} \tilde {\sigma} _ {\ell - 1}} \right] \quad (\ell = 1, \dots , L) \tag {A.17}
$$
$$
q _ {\ell} = q _ {\ell + 1} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{1 - q _ {\ell + 1} \hat {q} _ {\ell} \tilde {\sigma} _ {\ell}} \right] \quad (\ell = 0, \dots , L) \tag {A.18}
$$
with the boundary condition  $q_{L + 1} = 1$ , and a linear system for the replica uniform components,
$$
\begin{array}{l} \alpha_ {0} \hat {c} _ {0} = \frac {\beta^ {2} \left(c _ {0} + \eta^ {2}\right)}{\left(1 + \beta q _ {0}\right) ^ {2}} (A.19) \\ \alpha_ {1} \hat {c} _ {1} = \alpha_ {0} \frac {1}{n _ {0}} (\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\Sigma} _ {0} ^ {2} (\mathbf {I} _ {n _ {0}} - q _ {1} \hat {q} _ {0} \tilde {\Sigma} _ {0}) ^ {- 2} \tilde {\mathbf {w}} _ {*}) \hat {q} _ {0} ^ {2} \\ + \alpha_ {0} \left(\hat {c} _ {0} \mathbb {E} _ {\tilde {\sigma} _ {0}} \left[ \frac {\tilde {\sigma} _ {0}}{1 - q _ {1} \hat {q} _ {0} \tilde {\sigma} _ {0}} \right] + \left(q _ {1} \hat {c} _ {0} + c _ {1} \hat {q} _ {0}\right) \hat {q} _ {0} \mathbb {E} _ {\tilde {\sigma} _ {0}} \left[ \left(\frac {\tilde {\sigma} _ {0}}{1 - q _ {1} \hat {q} _ {0} \tilde {\sigma} _ {0}}\right) ^ {2} \right]\right) (A.20) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\alpha_ {\ell}}{\alpha_ {\ell - 1}} \hat {c} _ {\ell} = \hat {c} _ {\ell - 1} \mathbb {E} _ {\tilde {\sigma} _ {\ell - 1}} \left[ \frac {\tilde {\sigma} _ {\ell - 1}}{1 - q _ {\ell} \hat {q} _ {\ell - 1} \tilde {\sigma} _ {\ell - 1}} \right] \\ + \left(q _ {\ell} \hat {c} _ {\ell - 1} + c _ {\ell} \hat {q} _ {\ell - 1}\right) \hat {q} _ {\ell - 1} \mathbb {E} _ {\tilde {\sigma} _ {\ell - 1}} \left[ \left(\frac {\tilde {\sigma} _ {\ell - 1}}{1 - q _ {\ell} \hat {q} _ {\ell - 1} \tilde {\sigma} _ {\ell - 1}}\right) ^ {2} \right] \quad (\ell = 2, \dots , L) \tag {A.21} \\ \end{array}
$$
$$
\begin{array}{l} c _ {0} = \frac {1}{n _ {0}} (\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\boldsymbol {\Sigma}} _ {0} (\mathbf {I} _ {n _ {0}} - q _ {1} \hat {q} _ {0} \tilde {\boldsymbol {\Sigma}} _ {0}) ^ {- 2} \tilde {\mathbf {w}} _ {*}) \\ + \left(c _ {1} \mathbb {E} _ {\tilde {\sigma} _ {0}} \left[ \frac {\tilde {\sigma} _ {0}}{1 - q _ {1} \hat {q} _ {0} \tilde {\sigma} _ {0}} \right] + \left(q _ {1} \hat {c} _ {0} + c _ {1} \hat {q} _ {0}\right) q _ {1} \mathbb {E} \left[ \left(\frac {\tilde {\sigma} _ {0}}{1 - q _ {1} \hat {q} _ {0} \tilde {\sigma} _ {0}}\right) ^ {2} \right]\right) \tag {A.22} \\ \end{array}
$$
$$
\begin{array}{l} c _ {\ell} = c _ {\ell + 1} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{1 - q _ {\ell + 1} \hat {q} _ {\ell} \tilde {\sigma} _ {\ell}} \right] \\ + \left(q _ {\ell + 1} \hat {c} _ {\ell} + c _ {\ell + 1} \hat {q} _ {\ell}\right) q _ {\ell + 1} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{1 - q _ {\ell + 1} \hat {q} _ {\ell} \tilde {\sigma} _ {\ell}}\right) ^ {2} \right] \quad (\ell = 1, \dots , L) \tag {A.23} \\ \end{array}
$$
with the boundary condition  $c_{L + 1} = 0$ .
A.2 Converting between the Gibbs and maximum-likelihood estimators
As our primary aim is to study ridge regression, we must now account for the fact that the prior over the readout weights scales with the inverse temperature  $\beta$ . In particular, we have a prior with scaled covariance  $\Gamma_{L + 1} / (\beta \lambda)$ , where  $\Gamma_{L + 1}$  does not scale with  $\beta$ . If we perform this rescaling in (A.16) and (A.16), we can see that the re-scaled order parameters
$$
\bar {q} _ {\ell} = \beta \lambda q _ {\ell} \tag {A.24}
$$
$$
\bar {\hat {q}} _ {\ell} = \frac {1}{\beta \lambda} \hat {q} _ {\ell} \tag {A.25}
$$
$$
\bar {c} _ {\ell} = c _ {\ell} \tag {A.26}
$$
$$
\hat {\tilde {c}} _ {\ell} = \frac {1}{(\beta \lambda) ^ {2}} \hat {c} _ {\ell} \tag {A.27}
$$
obey an identical system of equations to the original order parameters in the Bayesian case at inverse temperature
$$
\beta = \frac {1}{\lambda}. \tag {A.28}
$$
Therefore, if we can solve the saddle point equations for the Gibbs estimator in the zero-temperature limit, we can simply read off the corresponding result for the ridge regression estimator in the ridgeless limit. The important difference is that the replica nonuniform component  $q_{0}$  of  $\mathbf{C}_0$  is  $\mathcal{O}(1 / \beta)$  in the ridge regression case, hence only the replica uniform component  $c_{0}$  contributes to the generalization error. We note that this allows one to read off the generalization error of a deep linear RFM with unstructured features from the results of our previous work [13] simply by setting the Bayesian prior variance  $\sigma^2$  to zero.
A.3 Solutions for the generalization error
The replica-symmetric saddle point equations in (A.16) and (A.19) are nearly identical to those analyzed our computation of the maximum eigenvalue of a structured Wishart product matrix [30],
which in turn are related to those in our original paper on unstructured deep linear RFMs [13] by the replacement of the spectral moment generating function of the identity matrix with the appropriate spectral generating functions. Given this similarity, and the fact that we have provided extensive exposition of how to solve such systems in those previous works, we will merely state the results for the order parameters relevant to the computation of the generalization error.
$$
M _ {\tilde {\mathbf {S}} _ {\ell}} (z) = \lim  _ {n _ {\ell} \rightarrow \infty} \frac {1}{n _ {\ell}} \operatorname {t r} \left[ \tilde {\mathbf {S}} _ {\ell} \left(z \mathbf {I} _ {n _ {\ell}} - \tilde {\mathbf {S}} _ {\ell}\right) ^ {- 1} \right] \tag {A.29}
$$
be the moment generating function of  $\tilde{\Sigma}_{\ell}$ , with functional inverse  $M_{\tilde{\Sigma}_{\ell}}^{-1}(z)$ . Then, at finite temperature, after eliminating the Lagrange multipliers, the replica nonuniform components of the order parameters are given by
$$
q _ {\ell} = \prod_ {j = \ell} ^ {L} \frac {A}{\alpha_ {j}} M _ {\tilde {\mathbf {S}} _ {j}} ^ {- 1} \left(\frac {A}{\alpha_ {j}}\right) \tag {A.30}
$$
for  $\ell = 0,\dots ,L$  where
$$
A = q _ {0} \hat {q} _ {0} = - \frac {\beta q _ {0}}{1 + \beta q _ {0}} \tag {A.31}
$$
satisfies the closed equation
$$
- \frac {1}{\beta} = \frac {A + 1}{A} \prod_ {\ell = 0} ^ {L} \frac {A}{\alpha_ {\ell}} M _ {\tilde {\Sigma} _ {\ell}} ^ {- 1} \left(\frac {A}{\alpha_ {\ell}}\right). \tag {A.32}
$$
From [30], we recognize this as the self-consistent equation for the moment generating function  $M = A$  of the feature kernel  $\mathbf{K} = \mathbf{X}\mathbf{F}\mathbf{F}^{\top}\mathbf{X}^{\top} / n_{0}$ , evaluated at  $-1 / \beta$ . Even in the unstructured case, this equation must in general be solved numerically at finite temperature [30].
Given a solution to this equation, we can solve the system of linear equations (A.19) for  $c_{0}$ , mirroring the computation of the extremal eigenvalues of structured product Wishart matrices in [30]. After eliminating the Lagrange multipliers, this calculation boils down to solving a three-term recurrence relation, which is detailed in previous works [13, 30]. We therefore simply state the result of this computation here. Let  $\zeta = -A$ , which then satisfies
$$
\lambda = \frac {1 - \zeta}{\zeta} \prod_ {\ell = 0} ^ {L} \frac {- \zeta}{\alpha_ {\ell}} M _ {\hat {\Sigma} _ {\ell}} ^ {- 1} \left(- \frac {\zeta}{\alpha_ {\ell}}\right). \tag {A.33}
$$
For  $\ell = 0,\dots ,L$  , let
$$
\kappa_ {\ell} = - M _ {\bar {\Sigma} _ {\ell}} ^ {- 1} \left(- \frac {\zeta}{\alpha_ {\ell}}\right) \tag {A.34}
$$
so that  $\kappa_{\ell}$  satisfies
$$
\frac {\zeta}{\alpha_ {\ell}} = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right]. \tag {A.35}
$$
Viewing  $\kappa_{\ell}$  as a function of  $\zeta$ , we may alternatively write the self-consistent equation for  $\zeta$  as
$$
\frac {1}{\beta} = \frac {1 - \zeta}{\zeta} \prod_ {\ell = 0} ^ {L} \frac {\zeta}{\alpha_ {\ell}} \kappa_ {\ell} (\zeta) \tag {A.36}
$$
$$
\begin{array}{l} \mu_ {\ell} = - \frac {\alpha_ {\ell}}{\zeta} \kappa_ {\ell} M _ {\boldsymbol {\Sigma} _ {\ell}} ^ {\prime} (- \kappa_ {\ell}) (A.37) \\ = 1 - \frac {\alpha_ {\ell}}{\zeta} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right] (A.38) \\ \end{array}
$$
We then finally have
$$
\begin{array}{l} \left[ 1 + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) (1 - \zeta) \right] c _ {0} = \frac {1}{\mu_ {0}} \frac {1}{n _ {0}} (\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\mathbf {\Sigma}} _ {0} (\kappa_ {0} \mathbf {I} _ {n _ {0}} + \tilde {\mathbf {\Sigma}} _ {0}) ^ {- 2} \tilde {\mathbf {w}} _ {*}) \kappa_ {0} ^ {2} \\ + \left(\sum_ {\ell = 1} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \frac {1}{n _ {0}} \left(\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\mathbf {S}} _ {0} \left(\kappa_ {0} \mathbf {I} _ {n _ {0}} + \tilde {\mathbf {S}} _ {0}\right) ^ {- 1} \tilde {\mathbf {w}} _ {*}\right) \kappa_ {0} \\ + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \zeta \eta^ {2}. \tag {A.39} \\ \end{array}
$$
Using the mapping of Appendix A.2 and again defining the weighted generating function
$$
\psi (z) = \lim  _ {n _ {0} \rightarrow \infty} \frac {1}{n _ {0}} \tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\Sigma} _ {0} \left(z \mathbf {I} _ {n _ {0}} + \tilde {\Sigma} _ {0}\right) ^ {- 1} \tilde {\mathbf {w}} _ {*}. \tag {A.40}
$$
this yields Proposition 3.1.
We now want to extract the zero-temperature/ridgeless limit. As  $\beta \to \infty$ , the self-consistent equation for  $\zeta$  admits the solution
$$
\zeta = 1, \tag {A.41}
$$
valid for  $\alpha_{\ell} > 1$  for all  $\ell$ , which gives  $q_0 \sim \mathcal{O}(1)$ , the solution
$$
\zeta = \alpha_ {0}, \tag {A.42}
$$
valid for  $\alpha_0 < 1, \alpha_0 < \alpha_1, \ldots, \alpha_L$ , which gives  $q_0 \sim \mathcal{O}(1 / \beta)$ , and, for  $\ell_* = 0, \ldots, L$ , the solutions
$$
\zeta = \alpha_ {\ell_ {*}}, \tag {A.43}
$$
valid for  $\alpha_{\ell_*} < 1$ ,  $\alpha_{\ell_*} < \alpha_0$ ,  $\alpha_{\ell_*} < \alpha_\ell$  for all  $\ell \neq \ell_*$ , which also give  $q_0 \sim \mathcal{O}(1 / \beta)$ . These solutions mirror those found in the unstructured setting [13]. We remark that, as in [13], we can determine the regimes in which each solution is physical by demanding that the order parameters  $q_\ell$  are non-negative.
For the  $\zeta \rightarrow 1$  solution, we immediately have
$$
\begin{array}{l} c _ {0} = \frac {1}{\mu_ {0}} \frac {1}{n _ {0}} (\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\boldsymbol {\Sigma}} _ {0} (\kappa_ {0} \mathbf {I} _ {n _ {0}} + \tilde {\boldsymbol {\Sigma}} _ {0}) ^ {- 2} \tilde {\mathbf {w}} _ {*}) \kappa_ {0} ^ {2} \\ + \left(\sum_ {\ell = 1} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \frac {1}{n _ {0}} \left(\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\Sigma} _ {0} \left(\kappa_ {0} \mathbf {I} _ {n _ {0}} + \tilde {\Sigma} _ {0}\right) ^ {- 1} \tilde {\mathbf {w}} _ {*}\right) \kappa_ {0} \\ + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \zeta \eta^ {2}, \tag {A.44} \\ \end{array}
$$
where by a minor abuse of notation we simply write  $\kappa_{\ell}$  and  $\mu_{\ell}$  for the corresponding quantities evaluated at  $\zeta = 1$ .
If  $\zeta \rightarrow \alpha_{\ell}$ , then  $\kappa_{\ell} \downarrow 0$  and  $\mu_{\ell} \downarrow 0$ . We can then apply L'Hôpital's rule to evaluate the limit in A.39, which corresponds to extracting the most divergent terms on each side of A.39. For the  $\zeta = \alpha_0$  solution, one finds that
$$
c _ {0} = \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}. \tag {A.45}
$$
Finally, for the solutions with  $\zeta = \alpha_{\ell_*}$  for  $\ell_{*} = 1,\ldots ,L$  , one finds that
$$
\begin{array}{l} c _ {0} = \frac {1}{1 - \alpha_ {\ell_ {*}}} \frac {1}{n _ {0}} (\tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\mathbf {S}} _ {0} (\kappa_ {0} \mathbf {I} _ {n _ {0}} + \tilde {\mathbf {S}} _ {0}) ^ {- 1} \tilde {\mathbf {w}} _ {*}) \kappa_ {0} \\ + \frac {\alpha_ {\ell_ {*}}}{1 - \alpha_ {\ell_ {*}}} \eta^ {2}, \tag {A.46} \\ \end{array}
$$
where we must be careful to recall that  $\kappa_0$  now satisfies
$$
\frac {\alpha_ {\ell *}}{\alpha_ {0}} = \mathbb {E} _ {\tilde {\sigma} _ {0}} \left[ \frac {\tilde {\sigma} _ {0}}{\kappa_ {0} + \tilde {\sigma} _ {0}} \right]. \tag {A.47}
$$
But, we recognize that  $\alpha_{\ell_*} = \alpha_{\min} = \min \{\alpha_1,\ldots ,\alpha_L\}$ , so we will write
$$
\kappa_ {\min } = \kappa_ {0} \Bigg | _ {\zeta = \alpha_ {\min }} \tag {A.48}
$$
to avoid clashing with our notation for the  $\zeta = 1$  solution.
Therefore, recalling from Appendix A.2 that the generalization error for the ridge regression estimator in the ridgeless limit is simply given by  $c_{0}$ , we have
$$
\epsilon_ {\text {r i d g e l e s s}} = \left\{ \begin{array}{l l} \left(\sum_ {\ell = 1} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \kappa_ {0} \psi \left(\kappa_ {0}\right) - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} \left(\kappa_ {0}\right) + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \eta^ {2}, & \alpha_ {0}, \alpha_ {\min } > 1 \\ \frac {\kappa_ {\min } \psi \left(\kappa_ {\min }\right)}{1 - \alpha_ {\min }} + \frac {\alpha_ {\min }}{1 - \alpha_ {\min }} \eta^ {2}, & \alpha_ {\min } <   1, \alpha_ {\min } <   \alpha_ {0} \\ \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}, & \alpha_ {0} <   1, \alpha_ {0} <   \alpha_ {\min }, \end{array} \right. \tag {A.49}
$$
as reported in (23), where we again define the weighted generating function