SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Youngsoo Baek
Department of Statistical Science
Durham, NC 27705
youngsoo. baek@duke.edu
Samuel I. Berchuck
Department of Biostatistics & Bioinformatics
sib2@duke.edu
Sayan Mukherjee*†
Center for Scalable Data Analysis and Artificial Intelligence
Leipzig 04105
sayan.mukherjee@mis.mpg.de
In this paper, we compare and contrast the behavior of the posterior predictive distribution to the risk of the maximum a posteriori (MAP) estimator for the random features regression model in the overparameterized regime. We will focus on the variance of the posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grows faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, which is of independent theoretical interest.
One of the most surprising empirical observations in deep learning is the generalization of overparameterized models that can perfectly interpolate the data. The "double descent" curve, referring to the test error first increasing then decreasing with model complexity, has been both empirically and theoretically validated for linear [Hastie et al., 2022] and nonlinear models [Ghorbani et al., 2021, Mei and Montanari, 2022, Hu and Lu, 2022]. Mei and Montanari [2022] showed that the generalization error of the random features (RF) model proposed by Rahimi and Recht [2007] does demonstrate double descent. Perhaps more surprisingly, they also showed that vanishingly small regularization can yield optimal generalization in a nearly noiseless learning task. These findings highlight the recent trend of explaining the success of machine learning through the prism of beneficial interpolation and overparameterization [Belkin, 2021].
There exists, however, another approach to overparameterized learning problems, which is the Bayesian posterior predictive distribution. In Bayesian practice, one can define the training objective
*Departments of Mathematics, Computer Science, Biostatistics & Bioinformatics, Duke University, NC
†Max Planck Institute for Mathematics in the Sciences, Leipzig
as the negative log-likelihood of a probabilistic model. The ridge regularized predictor for the RF model studied by Mei and Montanari [2022] is the maximum a posteriori (MAP) estimator of this probabilistic model. A "truly Bayesian" approach, on the other hand, is to derive the posterior predictive distribution which averages over different predictors and summarizes one's uncertainty in prediction. The posterior predictive distribution is also referred to as the "Bayesian model average" in the literature [Ovadia et al., 2019, Fortuin et al., 2021]. A fundamental question in Bayesian statistics is whether Bayesian credible sets, defined as high probability regions of the posterior, are also valid confidence sets in the frequentist sense [Kleijn and van der Vaart, 2012]. While this is true in finite-dimensional settings and known as Bernstein von-Mises theorem, the behavior of "frequentist" and "Bayesian" confidence sets can be drastically different in high-to-infinite dimensions, as revealed by Cox [1993], Freedman [1999], Johnstone [2010].
In this work, we examine the frequentist properties of the Bayesian random features model. The focus is on whether the variance of the posterior predictive distribution (PPV) demonstrates similar asymptotics as the risk of the MAP estimator studied by Mei and Montanari [2022]. The simplicity of the probabilistic model simplifies the study, as the posterior predictive distribution is available in closed form and is centered around the MAP estimator, for which we know the asymptotic risk. Due to this simplicity, agreement between the risk of the MAP estimator versus the PPV directly implies good coverage property of a natural confidence set formed by the posterior predictive. However, in light of previous work of Johnstone [2010], it is to be expected that such agreement can be hard to reach in high-dimensional learning. Previous literature relevant to our problem is reviewed in Section 2.3.
In this work, we show:
1. Absence of double descent phenomenon in the PPV with vanishing regularization (vague prior). Numerical instability does arise at the "interpolation boundary" for any finite dimensions.
2. In low noise problems, the expected width of the Bayesian credible ball for highly overparameterized models can be much wider than the true risk of the MAP predictor.
3. The expected width of a credible ball asymptotically agrees with the risk when the sample size grows much faster than the model dimensions. Numerical results show the required growth rate of the sample size is unreasonable for many learning problems.
4. Numerical simulations demonstrate finer distributional properties that are currently beyond the reach of theoretical analysis. They suggest the results of Freedman [1999] and Johnstone [2010] in the Gaussian sequence model are applicable to the more complicated RF model.
The rest of the paper is organized as follows. Section 2 reviews the basic setup of the RF model and its probabilistic reformulation. Section 3 summarizes our main results based on asymptotic formulae and numerical simulations. Section 4 concludes with a discussion of the implications and future directions.
2 Background on Random Features Model
Let inputs  $\boldsymbol{x}_i \in \mathbb{R}^d$ ,  $i = 1,2,\dots,n$  be drawn i.i.d. from a uniform measure (denoted  $\tau$ ) on a  $d$ -dimensional sphere with respect to the conventional Euclidean norm:
$$
\mathbb {S} ^ {d - 1} (\sqrt {d}) := \left\{\boldsymbol {x} \in \mathbb {R} ^ {d}: | | \boldsymbol {x} | | = \sqrt {d} \right\}. \tag {1}
$$
Let outputs  $y_{i}$  be generated by the following model:
$$
y _ {i} = f _ {d} \left(\boldsymbol {x} _ {i}\right) + \epsilon_ {i}, f _ {d} (\boldsymbol {x}) = \beta_ {d, 0} + \langle \boldsymbol {x}, \boldsymbol {\beta} _ {d} \rangle + f _ {d} ^ {N L} (\boldsymbol {x}). \tag {2}
$$
The model is decomposed into a linear component and a nonlinear component,  $f_{d}^{NL}$ . The random error  $\epsilon_{i}$ 's are assumed to be i.i.d. with mean zero, variance  $\tau^2$ , and finite fourth moment. We allow  $\tau^2 = 0$ , in which case the generating model is noiseless. In the analysis, these quantities will be assumed to obey further conditions, including an appropriate scaling so that all  $d$ -dependent quantities are  $O_{d}(1)$ .
2.1 Training with Ridge Regularization
We focus on learning the optimal random features model that best fits the training data. This is a class of functions
$$
\mathcal {F} := \left\{f: f (\boldsymbol {x}) = \sum_ {j = 1} ^ {N} a _ {j} \sigma \left(\left\langle \boldsymbol {x}, \boldsymbol {\theta} _ {j} \right\rangle / \sqrt {d}\right) \right\}, \tag {3}
$$
which is dependent on  $N$  random features,  $(\theta_{j})_{j = 1}^{N}$ , which are drawn i.i.d. from  $\mathbb{S}^{d - 1}$ , and a nonlinear activation function,  $\sigma$ . The training objective solves a regularized least squares problem for the linear coefficients  $\pmb{a} \equiv (a_{j})_{j = 1}^{N}$ :
$$
\min  _ {\boldsymbol {a} \in \mathbb {R} ^ {N}} \left\{\sum_ {i = 1} ^ {n} \left(y _ {i} - \sum_ {j = 1} ^ {N} a _ {j} \sigma \left(\left\langle \boldsymbol {x} _ {i}, \theta_ {j} \right\rangle / \sqrt {d}\right)\right) ^ {2} + d \psi_ {1, d} \psi_ {2, d} \lambda \| \boldsymbol {a} \| ^ {2} \right\}, \tag {4}
$$
where  $\psi_{1,d} = N / d$ ,  $\psi_{2,d} = n / d$ , and  $\lambda > 0$ . The optimal weights,  $\widehat{\pmb{a}} \equiv \widehat{\pmb{a}}(\lambda)$ , determine an optimal ridge predictor denoted by  $\widehat{f} \equiv f(\cdot; \widehat{\pmb{a}}(\lambda))$ . The dependence of the trained predictor on the dataset  $(X, y)$  and features  $\Theta$  are suppressed in notation.
There exist both practical and theoretical motivations for studying RF regression. On the practical side, RF regression has been suggested by Rahimi and Recht [2007] as a randomized approximation scheme for training kernel ridge regression (KRR) models for large datasets. On the theoretical side, (4) describes a "lazy training" algorithm for a 2-layer neural network with activation function  $\sigma$ . Previous studies have focused on the approximation power of such a function class [Jacot et al., 2018, Chizat et al., 2019] and the optimization landscape involved in training problems of type (4) [Mei et al., 2018, 2019].
Given a new input feature  $\pmb{x}$ , the ridge regularized predictor for the unknown output  $y$  has the form
$$
\widehat {f} (\boldsymbol {x}) \equiv f (\boldsymbol {x}; \widehat {\boldsymbol {a}}) = \sigma \left(\boldsymbol {x} ^ {T} \boldsymbol {\Theta} / \sqrt {d}\right) \widehat {\boldsymbol {a}}, \tag {5}
$$
with the optimal ridge weights  $\hat{\pmb{a}}$  and a resolvent matrix that defines the joint posterior of the weights:
$$
\widehat {\boldsymbol {a}} \equiv \widehat {\boldsymbol {a}} (\lambda) := \widehat {\boldsymbol {\Sigma}} (\lambda) \boldsymbol {Z} ^ {T} \boldsymbol {y} / \sqrt {d}, \tag {6}
$$
$$
\hat {\boldsymbol {\Sigma}} (\lambda) := \left(\boldsymbol {Z} ^ {T} \boldsymbol {Z} + \psi_ {1 d} \psi_ {2 d} \lambda \mathbf {I} _ {N}\right) ^ {- 1}. \tag {7}
$$
Here,  $Z \coloneqq \sigma(X\Theta/\sqrt{d})/\sqrt{d}$  for input design matrix  $X \in \mathbb{R}^{n \times d}$  and output vector  $\pmb{y} \equiv (y_i)_{i=1}^n$ . Similarly, we write  $\pmb{\sigma}(\pmb{x}) \coloneqq \sigma(\pmb{x}^T\pmb{\Theta}/\sqrt{d})$ . The  $L^2 \equiv L^2(\mathbb{S}^{d-1}(\sqrt{d});\tau)$  generalization error of predictor  $\widehat{f}$  is defined by
$$
R _ {R F} (\boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}, \lambda) := \mathbb {E} _ {\boldsymbol {x}} \| f _ {d} (\boldsymbol {x}) - \widehat {f} (\boldsymbol {x}) \| ^ {2} \equiv \| f _ {d} - \widehat {f} \| _ {L ^ {2}} ^ {2}, \tag {8}
$$
We emphasize that (8) is a random quantity, as it depends on  $(\pmb {y},\pmb {X},\Theta)$ .
2.2 RF as Bayesian Model
The objective function (4) can be interpreted as a MAP estimation problem for an equivalent Bayesian model. Formally, we adopt a  $d$ -dependent weight prior distribution, denoted  $p(\mathbf{a})$ , and also a normal likelihood, denoted  $p(\mathbf{y}|\mathbf{X}, \boldsymbol{\Theta}, \mathbf{a})$ , centered around a function in the class (3) with variance  $\phi^{-1}$ .
$$
\boldsymbol {a} \sim \operatorname {N o r m a l} \left(0, \phi^ {- 1} \frac {\psi_ {1 , d} \psi_ {2 , d} \lambda}{d} \mathbf {I} _ {N}\right)
$$
$$
\boldsymbol {y} \mid \boldsymbol {X}, \boldsymbol {\Theta}, \boldsymbol {a} \sim \mathrm {N o r m a l} \left(\sigma (\boldsymbol {X} \boldsymbol {\Theta} / \sqrt {d}) \boldsymbol {a}, \phi^ {- 1} \mathbf {I} _ {n}\right)
$$
The normal likelihood model need not agree with the generating process (2), as is often the case for Bayesian deep learning. Technically and inferentially, it can be justified as a coherent update of belief given a squared error loss [Bissiri et al., 2016]. The joint likelihood of  $(\pmb{y},\pmb{a})$  is defined conditional on both the random features  $\Theta$  and an "inverse temperature"  $\phi$ . The choice to condition on  $\Theta$  instead of learning them can be unnatural in certain settings and is mainly for the convenience of analysis.
However, we do note that the RF model has been used by Crawford et al. [2018] as an approximation of fully Bayesian kernel learning.
The posterior predictive distribution, or Bayesian model average over the posterior of  $\mathbf{a}$ , is defined as
$$
p (y \mid \boldsymbol {x}, \boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}) := \int_ {\mathbb {R} ^ {N}} p (y \mid \boldsymbol {x}, \boldsymbol {\Theta}, \boldsymbol {a}) p (\boldsymbol {a} \mid \boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}) d \boldsymbol {a}, \tag {9}
$$
where the posterior of  $\pmb{a}$  is a probability distribution placing higher mass near settings of  $\pmb{a}$  minimizing ridge objective (4):
$$
p (\boldsymbol {a} \mid \boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}) \propto \exp \left[ - \frac {\phi}{2} \left\{\sum_ {i = 1} ^ {n} \left(y _ {i} - f (\boldsymbol {x}; \boldsymbol {a})\right) ^ {2} + \frac {\psi_ {1 , d} \psi_ {2 , d} \lambda}{d} | | \boldsymbol {a} | | ^ {2} \right\} \right]. \tag {10}
$$
This is a Gaussian measure centered around  $\widehat{a}$  (6) and covariance matrix  $\widehat{\Sigma} (\lambda) / d$ . Thus the posterior predictive at new input  $x$  is also a normal distribution centered around  $\widehat{f}$ , with variance
$$
s ^ {2} (\boldsymbol {x}) \equiv s ^ {2} (\boldsymbol {x}; \lambda) := \phi^ {- 1} (1 + \boldsymbol {\sigma} (\boldsymbol {x}) ^ {T} \hat {\boldsymbol {\Sigma}} (\lambda) \boldsymbol {\sigma} (\boldsymbol {x}) / d). \tag {11}
$$
We refer to (11) as the PPV. This quantity dictates the width of the uncertainty interval centered around (4) evaluated at  $\pmb{x}$ . The expected PPV over  $\pmb{x}$  is defined by
$$
S _ {R F} ^ {2} (\boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}, \lambda) := \mathbb {E} _ {\boldsymbol {x}} \left[ s ^ {2} (\boldsymbol {x}) \right] \equiv \mathbb {E} _ {\boldsymbol {x}} \left[ \mathbb {V} [ y \mid \boldsymbol {x}, \boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta} ] \right]. \tag {12}
$$
The expectation over  $\mathbf{x}$ , similarly as in (8), yields the radius of the posterior credible ball for a posterior Gaussian process  $f$  centered around the optimal predictor  $\widehat{f}$ . Furthermore, the Gaussian likelihood model simplifies the expected PPV into a decomposition of this radius and the inverse temperature of the posterior:
$$
S _ {R F} ^ {2} = \int | | f - \widehat {f} | | _ {L ^ {2}} ^ {2} d p (f \mid \mathbf {y}, \mathbf {X}, \boldsymbol {\Theta}) + \phi^ {- 1}, \tag {13}
$$
where  $p(f \mid y, X, \Theta)$  is the law of Gaussian process  $f$  induced by the weight posterior  $p(\boldsymbol{a} \mid \boldsymbol{y}, \boldsymbol{X}, \Theta)$ . Thus, both summands in the display depend on the training features and are random. It is worth contrasting this definition with (12); in the next section, we explain in detail the motivation for comparing the two quantities.
The extra quantity introduced in the Bayesian formulation,  $\phi$ , governs how much the resulting posterior should concentrate around the ridge predictor (6). Since we are interested in the regime where  $N, d \to \infty$ , it is reasonable to assume the scale of the likelihood,  $\phi$ , must appropriately decrease with  $d$ , similar to how the prior on  $\pmb{a}$  is rescaled by  $1 / d$ . Practitioners may adopt some prior distribution on this parameter and perform hierarchical inference. We adopt a simpler, empirical Bayes approach and choose it to maximize the marginal likelihood:
$$
\widehat {\phi} ^ {- 1} \equiv \widehat {\phi} ^ {- 1} (\lambda) := \frac {\langle \mathbf {y} , \mathbf {y} - \widehat {f} (\mathbf {X}) \rangle}{n}. \tag {14}
$$
This choice coincides with the training set error attained by predictor (4), so it will be decreasing as  $N \to \infty$ . If  $N > n$  and  $\lambda \to 0^{+}$ , the training error vanishes as the model can perfectly interpolate the training set. We note the precise asymptotics of the training error has been already characterized by Mei and Montanari [2022] (Section 6).
A fundamental question here is whether  $R_{RF}$  and  $S_{RF}^{2}$  have similar asymptotics, as both quantities summarize uncertainty about our prediction in different ways.  $R_{RF}$  is the "frequentist's true risk," which requires assumptions about the unknown generative process.  $S_{RF}^{2}$ , on the other hand, is the "Bayesian's risk" that can be actually computed without any model assumptions. Its asymptotics does depend on model assumptions, as both the prior and likelihood need not capture the generative process (2). In particular, it is desired that it agrees with  $R_{RF}$  in some limiting sense. Throughout the rest of this work, we probe the question: Do  $R_{RF}$  and  $S_{RF}^{2}$  converge to the same value as  $d, n, N \to \infty$ ?
2.3 Previous Works
We have introduced our problem of comparing two different quantities,  $R_{RF}$  and  $S_{RF}^{2}$ , and provided their respective interpretations. Such comparison, between the frequentist risk and the variance of
the posterior, was done by Freedman [1999] in a white noise model, where one observes an infinite square-summable sequence with Gaussian noise. The key finding is that the distributions of two corresponding statistics, re-normalized, have different variances. They are in fact radically different distributions in that they are essentially orthogonal. Knapik et al. [2011] clarified the situation by showing frequentist coverage of credible ball depends heavily on the smoothness of the prior covariance operator. Johnstone [2010] have extended the results to a sequence of finite but growing length, which is a setup more similar to ours. Our study for the RF model addresses a much simpler question, as the theoretical results in Section 3 only address whether two statistics converge to the same limits. Unlike in the work by previous authors, the identity of limits cannot be taken for granted. A key feature driving the different asymptotics of the two quantities is that the Bayesian's prior and the likelihood are "mismatched" with respect to the data generating process; i.e.,  $f_{d}$  need not belong to the RF class (3). In Section 3.3, we demonstrate some distributional properties of  $R_{RF}$  and  $S_{RF}^{2}$  empirically observed in numerical simulations.
Uncertainty quantification in Bayesian learning has recently garnered much attention in the theoretical deep learning literature. We highlight, among many works: Clarté et al. [2023a], characterizing exact asymptotic calibration of a Bayes optimal classifier and a classifier obtained from the generalized approximate message passing algorithm, and Clarté et al. [2023b], demonstrating double descent-like behavior of the Bayes calibration curve in RF models. In particular, quantities  $R_{RF}$  and  $S_{RF}^{2}$  studied by our work can be related, respectively, to the "Bayes-optimal classifier" and "empirical risk classifier" of the latter work (Section 2). The recent work of Guionnet et al. [2023], on the other hand, explicitly addresses the model mismatch in Bayesian inference by deriving the exact, universal asymptotic formula for generalization of a misspecified Bayes estimator in a rank-1 signal detection problem.
3.1 Asymptotic Characterization
We present the main results and illustrate them through numerical simulations. We operate under assumptions identical to those of Mei and Montanari [2022], stated below. The proofs are collected in the Supplementary Materials.
Assumption 1. Define  $\psi_{1,d} = N / d$  and  $\psi_{2,d} = n / d$ . We assume  $\psi_{1,d} \to \psi_1 < +\infty$  and  $\psi_{2,d} \to \psi_2 < +\infty$  as  $d \to \infty$ .
Assumption 2. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be weakly differentiable and  $|\sigma(x)|, |\sigma'(x)| < c_0 e^{c_1 |x|}$  for some  $c_0, c_1 < +\infty$ . For  $G \sim \mathrm{Normal}(0,1)$ , the coefficients:
$$
\mu_ {0} = \mathbb {E} [ \sigma (G) ], \mu_ {1} = \mathbb {E} [ G \sigma (G) ], \mu_ {*} ^ {2} = E [ \sigma^ {2} (G) ] - (\mu_ {0} ^ {2} + \mu_ {1} ^ {2}) \tag {15}
$$
are assumed to be greater than 0 (ruling out a linear  $\sigma$ ) and finite.
Assumption 3. The generating model (2) satisfies
$$
\beta_ {0, d} ^ {2} \rightarrow F _ {0} ^ {2} <   + \infty , \left\| \beta_ {d} \right\| ^ {2} \rightarrow F _ {1} ^ {2} <   + \infty ; \tag {16}
$$
Furthermore,  $f_{NL,d}(\cdot)$  is a centered Gaussian process on  $\mathbb{S}^{d - 1}(\sqrt{d})$ , whose covariance function has the form:
$$
\mathbb {E} \left[ f _ {N L, d} \left(\boldsymbol {x} _ {1}\right) f _ {N L, d} \left(\boldsymbol {x} _ {2}\right) \right] = \Sigma_ {d} \left(\left\langle \boldsymbol {x} _ {1}, \boldsymbol {x} _ {2} \right\rangle / d\right). \tag {17}
$$
The kernel  $\Sigma_{d}$  satisfies
$$
\mathbb {E} \left[ \Sigma_ {d} \left(x _ {(1)} / \sqrt {d}\right)\right] = 0, \mathbb {E} \left[ x _ {(1)} \Sigma_ {d} \left(x _ {(1)} / \sqrt {d}\right)\right] = 0 a n d \Sigma_ {d} (1) \rightarrow F _ {*} ^ {2} <   + \infty , \tag {18}
$$
where  $x_{(1)}$  is the first entry of  $\mathbf{x} \sim \mathrm{Unif}(\mathbb{S}^{d - 1}(\sqrt{d}))$ . The signal-to-noise ratio (SNR) of the model is defined by
$$
\rho = \frac {F _ {1} ^ {2}}{F _ {*} ^ {2} + \tau^ {2}}. \tag {19}
$$
In this "linear proportional" asymptotic regime, we derive an asymptotic formula for the expected PPV akin to that of Mei and Montanari [2022] for the generalization error. Analysis shows Definition 1 of Mei and Montanari [2022], characterizing the asymptotics of  $R_{RF}$ , can be straightforwardly applied to also characterize the asymptotics of  $S_{RF}^2$ .
Proposition 1. Denote by  $\mathbb{C}_+$  the upper half complex plane:  $\{a + bi : i = \sqrt{-1}, b > 0\}$ . Let functions  $\nu_1, \nu_2 : \mathbb{C}_+ \to \mathbb{C}_+$  be uniquely defined by the conditions: on  $\mathbb{C}_+$ ,  $\nu_1(\xi), \nu_2(\xi)$  are analytic and uniquely satisfy the equations
$$
\nu_ {1} = \psi_ {1} \left(- \xi - \nu_ {2} - \frac {\zeta^ {2} \nu_ {2}}{1 - \zeta^ {2} \nu_ {1} \nu_ {2}}\right) ^ {- 1}, \tag {20}
$$
$$
\nu_ {2} = \psi_ {2} \left(- \xi - \nu_ {1} - \frac {\zeta^ {2} \nu_ {1}}{1 - \zeta^ {2} \nu_ {1} \nu_ {2}}\right) ^ {- 1}, \tag {21}
$$
when  $|\nu_{1}(\xi)|\leq \psi_{1} / \mathrm{Im}(\xi)$  and  $|\nu_{2}(\xi)|\leq \psi_{2} / \mathrm{Im}(\xi)$ , with  $\operatorname {Im}(\xi) > C$  for sufficiently large constant  $C$ . Define
$$
\chi = \nu_ {1} \left(i \sqrt {\psi_ {1} \psi_ {2} \lambda} / \mu_ {*}\right) \nu_ {2} \left(i \sqrt {\psi_ {1} \psi_ {2} \lambda} / \mu_ {*}\right), \zeta = \frac {\mu_ {1}}{\mu_ {*}}. \tag {22}
$$
Under Assumptions 1-3 and for  $\lambda >0$
$$
S _ {R F} ^ {2} (\boldsymbol {y}, \boldsymbol {X}, \boldsymbol {\Theta}, \lambda) \stackrel {{P}} {{\to}} \mathcal {S} ^ {2} = \frac {F _ {1} ^ {2}}{1 - \chi \zeta^ {2}} + F _ {*} ^ {2} + \tau^ {2} a s d, n, N \rightarrow \infty .
$$
Note that  $S^2$  depends on  $(\psi_1,\psi_2,\lambda)$  only through function  $\chi$ . The following facts follow from this fact and the asymptotic characterization of function  $\chi$  when  $\lambda \to 0^{+}$ .
Proposition 2. The following holds for the map  $(\psi_1,\psi_2,\lambda)\mapsto S^2$  defined in Proposition 1:
1. It is non-decreasing in  $\lambda$ .
2.  $\lim_{\lambda \to 0^{+}}S^{2} < + \infty$  when  $\psi_{1} = \psi_{2}$
Item 2, Proposition 2 deserves special attention, as it seems to suggest that there is no "double descent" in the asymptotics of  $S^2$  when using an improper prior. This fact does not possess an operational meaning at any finite  $N = n$ , due to the ordering of the limits  $N, n \to \infty$  and  $\lambda \to 0^+$ . In fact, one may expect that the distribution of the least singular value of the relevant matrix  $Z$  causes numerical instability for small  $\lambda$  when  $N = n$ . In Section 3.3, this hypothesis is empirically validated through simulations. Subtly, the theoretical prediction does accurately describe the numerical simulations for small choices of  $\lambda$ . On the other hand, the asymptotic prediction for the frequentist risk does diverge when  $\lambda \to 0^+$  and  $\psi_1 = \psi_2$ .
3.2 Comparison with Generalization Error
The asymptotics of (8), the "frequentist" risk for our comparison, was characterized by Mei and Montanari [2022] through a theorem similar to Proposition 1. Our main interest lies in comparing the risk against the Bayesian variance in two limiting cases: