SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Highly overparameterized models where  $\psi_{1}\rightarrow \infty$ . The number of parameters grows faster than any constant multiple of the number of samples.
2. Large sample models where  $\psi_{2}\rightarrow \infty$ . The number of samples grows faster than any constant multiple of the number of parameters.
The first regime has been studied as the relevant regime in modern deep learning. The second regime is closer to the classical regime of asymptotic statistics where only the number of samples  $n$  diverges. Below, we re-state the asymptotic formulae for  $R_{RF}$  of Mei and Montanari [2022] in these special cases, which admits simplifications relative to when both  $\psi_1, \psi_2$  are finite.
Proposition 3. (Theorem 4-5, Mei and Montanari [2022]) Under the notation of Proposition 1, define a function
$$
\omega \equiv \omega (\zeta , \psi , \bar {\lambda}) = - \frac {(\psi \zeta^ {2} - \dot {\zeta} ^ {2} - 1) + \sqrt {\psi \zeta^ {2} - \zeta^ {2} - 1}}{2 (\bar {\lambda} \psi + 1)}.
$$
In the highly overparameterized regime, where  $\psi_1\to \infty$  , asymptotic risk is given as
$$
\mathcal {R} _ {w i d e} (\rho , \zeta , \psi_ {2}, \bar {\lambda}) = \lim  _ {\psi_ {1} \rightarrow \infty} \lim  _ {d \rightarrow \infty} \mathbb {E} R _ {R F} (\boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}, \lambda) = \frac {(F _ {1} ^ {2} + F _ {*} ^ {2} + \tau^ {2}) (\psi_ {2} \rho + \omega_ {2} ^ {2})}{(1 + \rho) (\psi_ {2} - 2 \omega_ {2} \psi_ {2} + \omega_ {2} ^ {2} \psi_ {2} - \omega_ {2} ^ {2})} + F _ {*} ^ {2}
$$
with  $\omega_{2} = \omega (\zeta ,\psi_{2},\lambda /\mu_{*}^{2})$
In the large sample regime, where  $\psi_{2}\rightarrow \infty$ , asymptotic risk is given as
$$
\mathcal {R} _ {l s a m p} (\zeta , \psi_ {1}, \bar {\lambda}) = \lim _ {\psi_ {2} \to \infty} \lim _ {d \to \infty} \mathbb {E} R _ {R F} (\pmb {y}, \pmb {X}, \pmb {\Theta}, \lambda) = \frac {F _ {1} ^ {2} (\psi_ {1} \zeta^ {2} + \omega_ {1} ^ {2})}{\zeta^ {2} (\psi_ {1} - 2 \omega_ {1} \psi_ {1} + \omega_ {1} ^ {2} \psi_ {1} - \omega_ {1} ^ {2})} + F _ {*} ^ {2}
$$
with  $\omega_{1} = \omega (\zeta ,\psi_{1},\lambda /\mu_{*}^{2})$
A simple question, mentioned in Section 2.2, was whether the two quantities agree. It turns out that in the second regime, at least, the two formulae converge to the same limit, which is the main content of the next Proposition. In the first regime, whether the limits agree is determined by the signal-to-noise ratio  $\rho$ . Mei and Montanari [2022] showed that if  $\rho$  is larger than a certain critical threshold  $\rho_{*}$ , which depends on  $(\psi_2,\zeta)$ , the optimal regularization is achieved by  $\lambda \rightarrow 0^{+}$ , whereas if  $\rho$  is smaller than  $\rho_{*}$ , there exists an optimal regularization  $\lambda_{*}$  bounded away from 0. This phase transition also determines the agreement of the risk and the PPV in the limit.
Proposition 4. (Proposition 5.2, Mei and Montanari [2022] +  $\alpha$ ) Define quantities
$$
\rho_ {*} (\zeta , \psi_ {2}) = \frac {\omega_ {0 , 2} ^ {2} - \omega_ {0 , 2}}{(1 - \psi_ {2}) \omega_ {0 , 2} + \psi_ {2}},
$$
$$
\omega_ {0, 2} = \omega (\zeta , \psi_ {2}, 0)
$$
under the notation of Proposition 3.
1. If  $\rho < \rho_{*}$ , then  $\min_{\bar{\lambda} \geq 0} \mathcal{R}_{wide}(\rho, \zeta, \psi_2, \bar{\lambda})$ , is attained at some  $\lambda^{opt} > 0$ :
$$
\lambda^ {o p t} := \arg \min _ {\bar {\lambda} \geq 0} \mathcal {R} _ {w i d e} (\rho , \zeta , \psi_ {2}, \bar {\lambda}) = \frac {\zeta^ {2} \psi_ {2} - \zeta^ {2} \omega_ {*} \psi_ {2} + \zeta^ {2} \omega_ {*} + \omega_ {*} - \omega_ {*} ^ {2}}{(\omega_ {*} ^ {2} - \omega_ {*}) \psi_ {2}},
$$
$$
\omega_ {*} := \omega (\sqrt {\rho}, \psi_ {2}, 0).
$$
Furthermore,  $\mathcal{R}_{wide}(\rho ,\zeta ,\psi_2,\lambda^{opt}) = \lim_{\psi_1\to \infty}\mathcal{S}^2 (\psi_1,\psi_2,\lambda^{opt}) - \tau^2$
If, on the other hand,  $\rho >\rho_{*}$ , then  $\arg \min_{\bar{\lambda}\geq 0}\mathcal{R}_{wide}(\rho ,\zeta ,\psi_2,\bar{\lambda}) = 0$ . Furthermore,  $\mathcal{R}_{wide}(\rho ,\zeta ,\psi_2,0) < \lim_{\psi_1\to \infty}\mathcal{S}^2 (\psi_1,\psi_2,0) - \tau^2$
2. arg  $\min_{\bar{\lambda}\geq 0}\mathcal{R}_{lsamp}(\zeta ,\psi_1,\bar{\lambda}) = 0$  and  $\mathcal{R}_{lsamp}(\zeta ,\psi_1,0) = \lim_{\psi_2\to \infty}\mathcal{S}^2 (\psi_1,\psi_2,0) - \tau^2.$
Note the subtraction of the noise level  $\tau^2$  from  $S^2$ . This is because  $S_{RF}^2$  is computed based on the training error in Proposition 1, which includes both the approximation error and the variance of data, whereas  $R_{RF}$  is computed based only on the approximation error in the test set.
3.3 Numerical Simulations
In this section, we first compare evaluations of the asymptotic formulae for  $R_{RF}$  and  $S_{RF}^{2}$  for varying  $(\psi_1, \psi_2, \lambda)$ . We highlight their difference for finite  $(\psi_1, \psi_2)$  at the optimal choice of  $\lambda$  for the MAP risk. We also present numerical simulations, which both validate the formulae (they concentrate fast) and empirically exhibit further interesting distributional properties (suggesting the need for second-order asymptotics). MATLAB codes used to produce simulation results are included in the Supplementary Materials.
Figure 1 shows the dramatic mismatch between  $R_{RF}$  and  $S_{RF}^{2}$  in the low-noise regime. It turns out the conservativeness of the credible ball persists for the choice of  $\lambda$  that depends on  $(\psi_1, \psi_2)$ . The situation becomes more delicate in the high-noise regime because there exists a phase transition in the optimal choice of  $\lambda$  that depends on (19), which decreases with the noise variance  $\tau^2$ . Figures 2.2a and 2b compare the two curves,  $\mathcal{R}$  and  $S$ , for the "optimally tuned"  $\lambda$  at which the best possible frequentist risk is attained, for a fixed pair of  $(\psi_1, \psi_2)$ . In a low-noise task, with  $\rho = 5$ , the ratio of the frequentist risk of the posterior mean predictor to the width of the posterior predictive credible ball is less than 1 for a wide range of  $\psi_1$ . The situation is more nuanced and possibly more favorable for the Bayesian case in the high-noise task with  $\rho = 1/5$ .
While Figure 1 validates good concentration properties of  $R_{RF}$  and  $S_{RF}^{2}$  with respect to their asymptotic formulae, we may want to investigate the rate of fluctuations for these quantities. Figure
(a)  $\lambda = 1\mathrm{e} - 8$
(b)  $\lambda = 1\mathrm{e} - 3$
Figure 1: Comparison of asymptotic formula and 20 instances of  $S_{RF}^{2}$  (12). Data are generated via noiseless linear model  $y = \langle \pmb{x},\pmb{\beta}\rangle$  ( $|\beta|| = 1$ ,  $\rho = \infty$ ). Activation is ReLU:  $\sigma(x) = \max\{0, x\}$ .  $d$  and  $n$  are fixed to 100 and 300, respectively. The asymptotic formula for  $R_{RF}$  (8) is plotted for comparison (red, dashed).
(a)  $\psi_{2} = 3$
(b)  $\psi_{1} = 3$
Figure 2: Ratio of  $\mathcal{R}(\lambda^{opt})$  to  $S^2 (\lambda^{opt}) - \tau^2$  as a function of  $\psi_{1}$  (2a) and of  $\psi_{2}$  (2b). In each plot,  $\psi_{2}$  and  $\psi_{1}$  are respectively fixed to 3, while  $F_{1} = 1$ ,  $F_{*} = 0$ , and  $\rho = 1 / \tau^2$  for noise variance  $\tau^2\in \{.2,5\}$ .
Figure 3: Histograms of  $1\mathrm{e} + 4$  draws of  $R_{RF}$  and  $S_{RF}^{2} - \tau^{2}$  under low-noise linear model  $y = \langle \pmb {x},\pmb {\beta}\rangle +\tau^2$  with  $\tau^2 = 1 / 5$
3 suggests an interesting phenomenon: both quantities appear Gaussian, but are nearly orthogonal precisely near the "interpolation boundary" where  $R_{RF}$  exhibits double descent (2nd subplot). Our
empirical results should be compared with the results of Freedman [1999] and Johnstone [2010]. The Gaussianity of  $R_{RF}$  and  $S_{RF}^{2}$ , if true, strengthens the agreement between the expected width  $S_{RF}^{2}$  and expected risk  $R_{RF}$  immediately transfers to the agreement between the frequentist coverage of  $(1 - \alpha) - \%$  Bayes credible ball around the mean and the nominal coverage. Again, such claims are generally true in finite-dimensional settings, but not true for high-dimensional settings. For instance, Freedman [1999] showed that asymptotically, the posterior variance fluctuates like a Gaussian random variable with a variance strictly smaller than that of the frequentist risk. Extracting such information is possible only if we study second-order information of  $R_{RF}$  and  $S_{RF}^{2}$ ; in particular, suggests a Gaussian central limit for these quantities. A rigorous proof of a central limit theorem for  $R_{RF}$  and  $S_{RF}^{2}$  goes beyond the scope of this work. Nevertheless, we conjecture the following for the fluctuations of  $R_{RF}$  and  $S_{RF}^{2}$ :
1. Both  $d(R_{RF} - \mathbb{E}R_{RF})$  and  $d(S_{RF}^2 - \mathbb{E}S_{RF}^2)$  weakly converge to Gaussian distribution with appropriate variances. The faster convergence rate of  $d$  rather than  $\sqrt{d}$  is known to be common in central limit theorems for linear spectral statistics [Lytova and Pastur, 2009].
2. When  $\psi_{1} \leq \psi_{2}$ , asymptotic variance of  $S_{RF}^{2}$  is smaller than  $R_{RF}$ . When  $\psi_{1} = \psi_{2}$ , the two distributions are nearly orthogonal. For large enough  $\psi_{1}$ , the asymptotic variances are of the same order.
These conjectures are of independent interest and pose interesting theoretical challenges. We must note that second-order asymptotics has received less attention in the deep learning community. Only recently did Li et al. [2021] study the asymptotic normality of prediction risk achieved by a min-norm least squares interpolator. Their result relies on a central limit theorem for linear statistics of eigenvalues of large sample covariance matrices, established by Bai and Silverstein [2004]. Many central limit theorems in random matrix theory seem insufficient to handle kernel learning or learning with random features.
In sum, our calculations and empirical findings suggest that the RF model still has some interesting discoveries to offer, particularly for those interested in the implications of Bayesian deep learning. The asymptotics of the posterior predictive summaries can be very different from that of the generalization error. Numerical simulations suggest that some version of the central limit holds so that depending on the particular scaling of dimensions, the frequentist coverage of the posterior predictive distribution can be arbitrarily low or high. We now conclude with a discussion of several conjectures and the next directions suggested by these findings.
Heuristic Implications. While our technical calculations are only applicable to the RF model, we believe the general perspective of comparing frequentist versus Bayesian estimation approaches to the deep learning model can offer useful heuristics explaining empirically observed phenomena in training Bayesian deep learning models.
One suggestion is that the different regimes of asymptotics may explain away the "cold posterior effect", first empirically observed by Wenzel et al. [2020]. The authors forcefully suggested that raising the posterior of a deep learning model to an even larger power (i.e., more concentrated) can improve generalization. That the posteriors can be too wide in highly overparameterized, near-noiseless regimes may explain this issue. Even though in this simple RF model setup, the posterior predictive mean is left unchanged from that of the posterior, it is plausible that for more complicated models this need not be so. When that posterior is much wider than the expected risk of the mean, we will mix over too many bad predicting weights, so it will be actually worse to average over the posterior than not.
Technical Improvements. The technically most interesting but quite challenging direction, suggested by our numerical simulations in Section 3.3, is to study second-order fluctuations of random matrix quantities often encountered in deep learning. While at this point a vast amount of literature exists that systematically overcomes the non-standard definitions of quantities like generalization error from the viewpoint of random matrix theory, we are aware of no study as of yet that overcomes the same issues to show a central limit-type result. On the other hand, central limit theorems for linear spectral statistics remain an active area of research in random matrix theory and are generally acknowledged to require more advanced, subtler arguments that are fundamentally different from first-order results
(i.e., convergence of empirical spectral distribution). For a start, a separate work on the central limit theorem for linear spectral statistics of kernel matrices is in preparation by the authors.
Another interesting direction is to extend the comparison between frequentist and Bayesian modeling approaches to more complex models of training dynamics. For instance, Adlam and Pennington [2020] have derived precise asymptotics of generalization error in the neural tangent kernel (NTK) regime, in which the features  $\Theta$  are "learned" through a linearized approximation of the gradient flow in training. Mei et al. [2018, 2019] have suggested a different asymptotic model, in which  $\Theta$  evolve through nonlinear dynamics. An interesting challenge, both technically and conceptually, is to adapt their analysis to the Bayesian setting, where the gradient flow on the space of posterior measures of  $\Theta$  is approximated and specific projections are tracked.
Making the "Right" Comparison. A possible objection is that the comparison between two quantities is not appropriate: if interested in the coverage of function  $f_{d}$  with our credible set, then one must compare the expected width of the interval of the function space posterior,  $\widehat{f}$ , instead of that of the Bayesian model average. While the objection is sensible, the fact is that the average variance of the function posterior will be an even worse uncertainty estimate for underparameterized models, whereas it will make little difference for overparameterized models. The reason is that the variance of the posterior of  $\widehat{f}$  only differs from that of the posterior predictive, in (11), by the training set error term (14). The training error dominates the PPV precisely for underparameterized models  $(\psi_{1} < \psi_{2})$ , so the curves of Figure 1 will now be non-decreasing and asymptotically convergent to a similar constant in expectation. This fact illustrates that a simple interpretation of the width of the posterior is complicated in even simplistic high-dimensional models.
Another possible objection is that the same value of  $\lambda$  need not be optimal for the posterior predictive. Since the asymptotic curve for  $S^2$  is non-increasing in  $\lambda$ , this implies one must choose even smaller  $\lambda$  than  $\lambda^{opt}$  to obtain less conservative credible intervals, regardless of the SNR  $\rho$ . This seems an odd consequence of using overparameterized models, where a good frequentist coverage is obtained by choosing a narrower prior than that with good "contraction property" (i.e., minimal  $L^2$  risk of posterior mean).
The final objection is that the RF model is simply not a very realistic model of practical Bayesian deep learning. We point out that quantifying the behavior of RF models as approximations of kernel learning methods remains a valid concern for Bayesian practitioners.
All authors would like to thank Matthew M. Engelhard, Boyao Li, David Page, and Alexander J Thomson for the helpful discussion around the theoretical work.
Samuel I. Berchuck would like to acknowledge support from the National Eye Institute of the National Institutes of Health under Award Number K99EY033027. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Sayan Mukherjee would like to acknowledge partial funding from HFSP RGP005, NSF DMS 17-13012, NSF BCS 1552848, NSF DBI 1661386, NSF IIS 15-46331, NSF DMS 16-13261, as well as high-performance computing partially supported by grant 2016-IDG-1013 from the North Carolina Biotechnology Center as well as the Alexander von Humboldt Foundation, the BMBF and the Saxony State Ministry for Science.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-986, 2022.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49(2):1029-1054, 2021.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 (4):667-766, 2022.
A Lytova and L Pastur. Central limit theorem for linear eigenvalue statistics of random matrices with independent entries. Annals of Probability, 37(5):1778-1840, 2009.
Zeng Li, Chuanlong Xie, and Qinwen Wang. Asymptotic normality and confidence intervals for prediction risk of the min-norm least squares estimator. In International Conference on Machine Learning, pages 6533-6542. PMLR, 2021.
ZD Bai and Jack W Silverstein. Clt for linear spectral statistics of large-dimensional sample covariance matrices. The Annals of Probability, 32(1A):553-605, 2004.
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Świkatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020.
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pages 74-84. PMLR, 2020.
Proof of Proposition 1
The necessary calculations follow immediately from the proof of Mei and Montanari [2022]. Since we do not wish to replicate the long proof with only minor readjustments, we only sketch the main argument. The expected PPV consists of two random variables, denoted by
$$
\widehat {\phi^ {- 1}} \left(f _ {d}, \boldsymbol {X}, \boldsymbol {\Theta}, \lambda\right) = \frac {\langle \boldsymbol {y} , \boldsymbol {y} - \widehat {f} (\boldsymbol {x}) \rangle}{n}, r (\boldsymbol {X}, \boldsymbol {\Theta}, \lambda) := \mathbb {E} _ {\boldsymbol {x}} \left[ \boldsymbol {\sigma} (\boldsymbol {x}) ^ {T} \left(\boldsymbol {Z} ^ {T} \boldsymbol {Z} + \lambda \psi_ {1 d} \psi_ {2 d} \mathbf {I} _ {N}\right) ^ {- 1} \boldsymbol {\sigma} (\boldsymbol {x}) / d \right]. \tag {23}
$$
It suffices to show the two quantities individually converge in probability to some limits. The first of the two is a training set error, for which an asymptotic formula was derived in Section 6, Mei and Montanari [2022]:
$$
\lim  _ {d \rightarrow \infty} \mathbb {E} \widehat {\phi^ {- 1}} = - i \nu_ {2} \left(i \sqrt {\psi_ {1} \psi_ {2} \lambda} / \mu_ {*}\right) \sqrt {\frac {\lambda \psi_ {1}}{\psi_ {2} \mu_ {*} ^ {2}}} \left(\frac {F _ {1} ^ {2}}{1 - \chi \zeta^ {2}} + F _ {*} ^ {2} + \tau^ {2}\right), \tag {24}
$$
using the definition of quantities in Proposition 1, for  $\chi \equiv \chi (\pmb {i}\sqrt{\psi_1\psi_2\lambda} /\mu_*) = \nu_1\nu_2(\pmb {i} = \sqrt{-1})$  . At this point, rearranging the fixed point equations (20) and (21) with  $\xi \equiv i\sqrt{\psi_1\psi_2\lambda} /\mu_{*}$  , we obtain:
$$
\begin{array}{l} - \psi_ {1} - \mathbf {i} \sqrt {\lambda \psi_ {1} \psi_ {2}} / \mu_ {*} \nu_ {1} = - \psi_ {2} - \mathbf {i} \sqrt {\lambda \psi_ {1} \psi_ {2}} / \mu_ {*} \nu_ {2} = - \chi - \frac {\zeta^ {2} \chi}{1 - \zeta^ {2} \chi}. (25) \\ \Longrightarrow - i \sqrt {\frac {\lambda \psi_ {1}}{\psi_ {2} \mu_ {*} ^ {2}}} \nu_ {2} = 1 + \frac {\chi}{\psi_ {2}} \left(\frac {\zeta^ {2}}{1 - \zeta^ {2} \chi} + 1\right). (26) \\ \end{array}
$$
A similar formula for  $\mathbb{E}r$  can also be derived, using almost exactly the same steps required for simplifying asymptotic formulae of  $R_{RF}$  and  $\widehat{\phi^{-1}}$ . Assumption 3 on the nonlinear component of the generating model,  $f_{d}^{NL}$ , implies that it admits an explicit decomposition into spherical harmonics that form a basis on  $\mathbb{S}^{d - 1}$  and coefficients with Gaussian distributions (appropriately scaled). With many simplifications, also sketched out in Appendix E, Mei and Montanari [2022], we can write
$$
\mathbb {E} r = \mu_ {1} ^ {2} \operatorname {T r} \left(\left(\boldsymbol {Z} ^ {T} \boldsymbol {Z} + \lambda \psi_ {1} \psi_ {2} \mathbf {I} _ {N}\right) ^ {- 1} \left(\boldsymbol {\Theta} \boldsymbol {\Theta} ^ {T} / d\right)\right) + \mu_ {*} ^ {2} \operatorname {T r} \left(\left(\boldsymbol {Z} ^ {T} \boldsymbol {Z} + \lambda \psi_ {1} \psi_ {2} \mathbf {I} _ {N}\right) ^ {- 1}\right) + o _ {d} (1). \tag {27}
$$
Next, form an  $(N + n) \times (N + n)$ -dimensional block matrix  $\mathbf{A}$ , indexed by a 5-d real parameter  $\mathbf{q} = (s_1, s_2, t_1, t_2, p)$ :
$$
\boldsymbol {A} (\boldsymbol {q}) = \left[ \begin{array}{l l} s _ {1} \mathbf {I} _ {N} + s _ {2} \boldsymbol {\Theta} \boldsymbol {\Theta} / d & \boldsymbol {Z} ^ {T} + p \boldsymbol {\Theta} \boldsymbol {X} ^ {T} / \sqrt {d} \\ \boldsymbol {Z} + p \boldsymbol {X} \boldsymbol {\Theta} ^ {T} / \sqrt {d} & t _ {2} \mathbf {I} _ {n} + t _ {2} \boldsymbol {X} \boldsymbol {X} / d \end{array} \right]. \tag {28}
$$
Denoting by  $\xi \in \mathbb{C}_{+}$  the log-determinant of the resolvent matrix  $A(\pmb {q}) - \xi \mathbf{I}_{N + n}$  as  $G_{d}(\xi ;\pmb {q})$  straightforward calculus and plugging in the figures reveals
$$
\mathbb {E} r = - \mu_ {1} ^ {2} \cdot \boldsymbol {i} \frac {1}{\sqrt {\psi_ {1} \psi_ {2}} \lambda} \mathbb {E} [ \partial_ {s _ {1}} G _ {d} (\boldsymbol {i} \sqrt {\lambda \psi_ {1} \psi_ {2}}; 0) ] - \mu_ {*} ^ {2} \cdot \boldsymbol {i} \frac {1}{\sqrt {\psi_ {1} \psi_ {2}} \lambda} \mathbb {E} [ \partial_ {s _ {2}} G _ {d} (\boldsymbol {i} \sqrt {\lambda \psi_ {1} \psi_ {2}}; 0) ] + o _ {d} (1). \tag {29}
$$
At this point, we pass on to the limit  $d\to \infty$  and plug in appropriate asymptotic limits of the partial derivatives. Since the  $\xi$ -derivative of  $G_{d}$  is the trace of  $\mathbf{A}(\pmb {q}) - \xi \mathbf{I}_{N + n}$ , this is possible by characterizing the asymptotics of the Stieltjes transform of the empirical spectral distribution of  $\mathbf{A}(\pmb {q})$ , which is done using Proposition 8.3, Mei and Montanari [2022]. Integrating the limit over path stretching in the imaginary axis ( $iK$  with  $K\rightarrow \infty$ ), one then rigorously passes onto the limit by Proposition 8.4, Mei and Montanari [2022]. Final simplifications using Lemma 8.1, Mei and Montanari [2022] yield the formula
$$
\lim  _ {d \rightarrow \infty} \mathbb {E} r = - i \nu_ {1} \left(i \sqrt {\psi_ {1} \psi_ {2} \lambda} / \mu_ {*}\right) \frac {\mu_ {*}}{\sqrt {\lambda} \psi_ {1} \psi_ {2}} \left(\frac {\zeta^ {2}}{1 - \zeta^ {2} \chi} + 1\right). \tag {30}
$$
At this point, we may conclude, using (23),(30) and (26):
$$
\widehat {\phi^ {- 1}} (1 + r) \stackrel {P} {\rightarrow} \frac {F _ {1} ^ {2}}{1 - \zeta^ {2} \chi} + F _ {*} ^ {2} + \tau^ {2}.
$$
Proof of Proposition 2
The first point is a consequence of the fact that  $S_{RF}^2$  is decreasing in  $\lambda$  and passing onto the limits. The second point is immediately seen by taking the limit of  $\chi$  as  $\lambda \to 0$ , a formula for which is provided by Theorem 3, Mei and Montanari [2022].
Proof of Proposition 4
We want to check whether equality between asymptotic risk and the expected PPV holds. Using Proposition 3: In the highly overparameterized regime, we demand equality
$$
\mathcal {R} _ {w i d e} = \frac {F _ {1} ^ {2} \psi_ {2} + (F _ {*} ^ {2} + \tau^ {2}) \omega_ {2} ^ {2}}{\psi_ {2} - 2 \omega_ {2} \psi_ {2} + \omega_ {2} ^ {2} \psi_ {2} - \omega_ {2} ^ {2}} \stackrel {*} {=} \frac {F _ {1} ^ {2}}{1 - \omega_ {2}}.
$$
$$
\Longleftrightarrow \omega_ {2} ^ {3} + (\rho \psi_ {2} - \rho - 1) \omega_ {2} ^ {2} - \psi_ {2} \rho \omega_ {2} \stackrel {*} {=} 0.
$$