Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression
Authors: Youngsoo Baek
Samuel Berchuck
Sayan Mukherjee
Venue: NeurIPS 2023 poster (2023)
Keywords: asymptotics
random features model
Bayesian inference

Abstract: In this paper we compare and contrast the behavior of the posterior predictive distribution to the risk of the 
the maximum a posteriori estimator for the random features regression model in the overparameterized regime.  We will focus on the variance of the  posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grow faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, this is of independent theoretical interest.

Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression
Youngsoo Baek
Samuel Berchuck
Department of Statistical Science
Department of Biostatistics & Bioinformatics
Duke University
Duke University
Durham, NC 27705
Durham, NC 27705
youngsoo. baek@duke.ccoooo edu
sib2@duke.
Sayan Mukherjee
Center for Scalable Data Analysis and Artificial Intelligence
Universität Leipzig
Leipzig 04105
sayan Dayan.mukinn mukher jeeeeeemii wew.emmimmmmm......mme..m.. mpg.
Abstract
In this paper, we compare and contrast the behavior of the posterior predictive distribution the risk of the maximum posteriori (MAP) estimator for the random features regression model the overparameterized regime. We will focus on the variance of the posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grows faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, which is of independent theoretical interest.
Introduction
One of the most surprising empirical observations in deep learning the generalization of overpa- rameterized models that can perfectly interpolate the data. The "double descent" curve, referring to the error first increasing then decreasing with model complexity, has been both empirically and theoretically validated for linear (Hastie et al. 2022) and nonlinear models [Ghorbani et al. 2022 Meij
and Montanari 2022 Hu and Lu 2022 Mei and Montanari [2022] showed that the generalization eтror of the random features (RF) model proposed by Rahimi and Recht [2007) does demonstrate double descent. Perhaps more surprisingly, they also showed that vanishingly small regularization can yield optimal generalization in nearly noiseless learning task. These findings highlight the recent trend of explaining the success of machine learning through the prism beneficial interpolation and
overparameterization [Belkin] 2021).
There exists, however, another approach overparameterized learning problems, which is the Bayesian posterior predictive distribution. In Bayesian practice, one can define the training objective
Depaameents of Mathematics, Computer Science, Biostatistics & Bioinformatics, Duke University, NC Max Planck Institute for Mathematics in the Sciences, Leipzig
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

as the negative log-likelihood of a probabilistic model. The ridge regularized predictor for the RF model studied by Mei and Montanari [202]] the maximum a posterioni (MAP) estimator of this probabilistic model. A "truly Bayesian" approach, on the other hand, is to derive the posterior predictive distribution which averages over different predictors and summarizes one's uncertainty in prediction. The posterior predictive distribution is also referred to as the "Bayesian model average" in the literature [Ovadia et al., 2019 Fortuin et al. 2021]. A fundamental question in Bayesian statistics is whether Bayesian credible sets, defined high probability regions of the posterior, are also valid confidence sets the frequentist sense [Kleijn and van der Vaart 2012]. While this is true in settings and known as Bernstein von-Miser theorem, the behavior of "frequentist" and "Bayesian" " confidence sets can be drastically different high-to-infinice dimensions, as revealed by Cox [1993]. Freedman [1999], Johnstone (2010).
In this work, we examine the frequentist properties the Bayesian random features model. The focus is on whether the variance of the posterior predictive distribution (PPV) demonstrates similar asymptoties as the risk of the MAP estimator studied by Mei and Montanari (2022). The simplicity of the probabilistic model simplifies the study, the posterior predictive distribution is available in closed form and is centered around the MAP estimator, for which we know the asymptotic risk. Due to this simplicity, agreement between the risk of the MAP estimator versus the PPV directly implies good coverage property of a natural confidence set formed by the posterior predictive. However, in light of previous work Johnstone [2010], it is to be expected that such agreement can be hard to reach in high-dimensions learning. Previous literature relevant to our problem reviewed in Section
In this work, we show:
1. Absence of double descent phenomenon in the PPV with vanishing regularization (vague prior). Numerical instability does arise at the "interpolation boundary" for any finite dimensions.
2. In low noise problems, the expected width of the Bayesian credible ball for highly overpa- rameterized models can be much wider than the true risk of the MAP predictor.
3. The expected width of credible ball asymptotically agrees with the risk when the sample size grows much faster than the model dimensions. Numerical results show the required growth rate of the sample size is unreasonable for many learning problems.
4. Numerical simulations demonstrate finer distributional properties that are currently beyond the reach of theoretical analysis. They suggest the results of Freedman [1999] Johnstone (210) the Gaussian sequence model are applicable to the more complicated RF model.
The rest of the paper is organized as follows. Section 2 reviews the basic setup of the RF model and its probabilistic reformulation. Section 3 summarizes our main results based on asymptotic formulae and numerical simulations. Section concludes with a discussion of the implications and future directions.
2 Background on Random Features Model
Let inputs E Rd, i = 2. n be drawn i.i.d. from uniform measure (denoted 7) on d-dimensional sphere with respect to the conventional Euclidean norm:
s ( = (a E Rd: ||ح|||
vd).
Let outputs yi be generated by the following model:
y/ = fa(x,)) + 6, fa(z) = 33,0 + (x. Ba) + + (x))
(2)
The model is decomposed into a linear component and a nonlinear component, f The random error e are assumed to be i.i.d. with mean zero, variance ?2, and finite fourth moment. We allow ?? = 0, in which case the generating model is noiseless. In the analysis, these quantities will be assumed to obey further conditions, including an appropriate scaling so that all d-dependent quantities are (1))

---

2.1 Training with Ridge Regularization
We focus on learning the optimal random features model that best fits the training data. This is a class of functions
N F f(z( = Ea aao (aa(()==ao(ttte 0j)/va)
(3)
which is dependent on N random features, (0z) which drawn i..dd from gd-1, - and nonlinear activation function, d. The training objective solves regularized least squares problem for the linear coefficients (az))=1;
a (ao )
2
1
ARR
Σ
y/1
+ dih av
(4)
where 1/1 = N/d, 4/2, n/d, and X > 0. The optimal weights, @(), determine an optimal ridge predictor denoted by f f(; @()). The dependence of the trained predictor on the dataset
(X, y) and features @ are suppressed in notation.
There exist both practical and theoretical motivations for studying regression. the practical side, RF regression has been suggested by Raaiim and Recht [2007] randomized approximation scheme for training kernel ridge regression (KRR) models for large datasets. On the theoretical side,
(4) describes a "lazy training" algorithm for 2-layer neural network with activation function Previous studies have focused on the approximation power such a function class [Jacot et al. 2018
Chizat et al. 2019] and the optimization landscape involved in training problems of type Mei
ct al.
2018 2019].
Given new input feature z, the ridge regularized predictor for the unknown output has the form
f(x) f(x; @) o (o==0//vD))
(5)
with the optimal ridge weights à and resolvent matrix that defines the joint posterior of the weights:
à == @(A) = 5(A)2"//ve
Σ(A) : Z Z ViaarzaAIN
Here, Z: (X=/X//V2 for input design matrix Rud and output vector y = (1:))_1-
Similarly we write o (a) == ((="0/Va The L (P(5-1(Vd))1) generalization error of predictor f is defined by
RaF (y, X, θ A) = B= M())11 IIʃa Í1
We emphasize that 8 is random quantity, as it depends on (y. X, O).
2.2 RF as Bayesian Model
The objective function (4) can be interpreted a MAP estimation problem for an equivalent Bayesian model. Formally, we adopt a d-dependent weight prior distribution, denoted p(a). and also a normal likelihood, denoted p( X, O a), centered around function in the class (3) with variance $-1
Normal (0.0-1 DIaVaa IN
X. O. Normal (x(0///)) IIn)
The normal likelihood model need not agree with the generating process (2). as is often the case for Bayesian deep learning. Technically and inferentially, it can be justified as coherent update of belief given a squared error loss (Bissiri et al. 2016). The joint likelihood of (y, a) is defined conditional on both the random features θ and an "inverse temperature" ø. The choice to condition on θ instead of learning them can be unnatural in certain settings and is mainly for the convenience of analysis.
3

---

However, we do note that the RF model has been used by Crawford et al. 2018 as an approximation of fully Bayesian kernel learning.
The posterior predictive distribution, or Bayesian model average over the posterior of a, is defined as
ply y. y. X 6) | ply x a) y, X, ②) da,
where the posterior of a is probubility distribution placing higher mass near settings of a minimizing ridge objective (4
}|
(3 f(x;; a))² + V1aaaad
p(a X & exp 2
|al??
(10)
d
This is Gaussian measure centered around ā and covariance matrix (A))/d. Thus the posterior predictive at new input x is also a normal distribution centered around f. with variance
s² (x) = (x; A) o (x)) (())))))))
(11)
We refer to 111 as the PPV. This quantity dictates the width of the uncertainty interval centered around 4 evaluated at æ. The expected PPV over æ is defined by
R X, ²) = E. (x)) = E. VVly | x, @I].
(12)
The expectation over æ, similarly as in (8). yields the radius of the posterior credible ball for posterior Gaussian process centered around the optimal predictor f. Furthermore, the Gaussian
likelihood model simplifies the expected PPV into decomposition of this radius and the inverse temperature of the posterior:
S f---1115 115 115- dp(f y.
(13)
where p(f y, X) () is the law of Gaussian process f induced by the weight posterior p(a y, X, ②). Thus, both summands in the display depend on the training features and are random. It is worth contrasting this definition with (2); in the next section, we explain in detail the motivation for comparing the two quantities.
The extra quantity introduced in the Bayesian formulation, o. governs how much the resulting posterior should concentrate around the ridge predictor (6). Since we are interested in the regime where N, d + oc, is reasonable decrease with d, similar to how the prior on is rescaled by 1/d. Practitioners may adopt some prior distribution on this parameter and perform hierarchical inference. We adopt a simpler, empirical Bayes approach and choose it to maximize the marginal likelihood:
assume the scale of the likelihood, c, must appropriately
(y. f(X))
(14)
72
This choice coincides with the training set error attained by predictor (4) so it will be decreasing as N - 00. If N > n and X - 0+, the training error vanishes as the model can perfectly interpolate the training set. We note the precise asymptotics of the training error has been already characterized by Mei and Montanari 2002 (Section 6).
A fundamental question here is whether RaF and 5 have similar asymptotics, as both quantities summarize uncertainty about our prediction in different ways. REF is the "frequentist's true risk," " which requires assumptions about the unknown generative process. SRF, the other hand, is the "Bayesian's 's risk" that can be actually computed without any model assumptions. Its asymptotics does depend on model assumptions, as both the prior and likelihood need not capture the generative process (2. In particular, it is desired that it agrees with Ra in some limiting sense. Throughout the rest of this work, we probe the question: Do REF and 52 RF converge the same value d, n, N + oo?
2.3 Previous Works
We have introduced our problem of comparing two different quantities, RAF and Sip, and provided their respective interpretations. Such comparison, between the frequentist risk and the variance of

---

the posterior, was done by Freedman 1999 in white noise model, where one observes an infinite suaumbblee sequence with Gaussian noise. The key finding that the distributions of two corresponding statistics, re-normalized, have different variances. They are in fact radically different distributions in that they are essentially orthogonal. Knapik et al. [201T] clarified the situation by showing frequentist coverage of credible ball depends heavily on the smoothness of the prior covariance operator. Johnstone [2010) have extended the results to a sequence of finite but growing length, which is setup more similar to ours. Our study for the RF model addresses much simpler question, as the theoretical results in Section 33 only address whether two statistics converge to the same limits. Unlike in the work by previous authors, the identity of limits cannot be taken for granted. key feature driving the different asymptotics of the two quantities is that the Bayesian's 's prior and the likelihood are "mismatched" with respect to the data generating process: i... fa need not belong
to the RF class (3) In Section 3.3 we demonstrate some distributional properties of RE and S empirically observed in numerical simulations.
Uncertainty quantification in Bayesian learning has recently garnered much attention the theoretical deep learning literature. We highlight, among many works: Clarté et al. [2023a]. characterizing exact asymptotic calibration of a Bayes optimal classifier and a classifier obtained from the generalized approximate message passing algorithm, and Clarté et al. [202bb] demonstrating double descent- like behavior of the Bayes calibration curve in RF models. In particular, quantities REF and RF studied by our work can be related, respectively, to the "Bayes-optimal classifier" and "empirical risk classifier" of the latter work (Section 2). The recent work of Guionnet et al. [2023), on the other hand, explicitly addresses the model mismatch in Bayesian inference by deriving the exact, universal asymptotic formula for generalization of a misspecified Bayes estimator in a rank- signal detection problem.
3 Results
3.1 Asymptotic Characterization
We present the main results and illustrate them through numerical simulations. We operate under assumptions identical to those of and Montanari 2022], stated below. The proofs are collected in the Supplementary Materials.
Assumption 1. Define whi,d = N/d and 1/2,a = n/d. We assume vhidd - w/i < +o0 and v22,d + 4/2 < 400 as d Co.
Assumption 2. Let a : R + R be weakly differentiable and (o()))))((<<) < Cgectlal (ul for some Co C1 < +00. For G Normal(O, 1), the coefficients:
Pg = ([)( p1 = (G), 22 = E (Hā +
(15)
assumed to be greater than 0 (ruling out a linear σ) and finite. Assumption 3. The generating model (2) satisfies
Bi + F3 < +00, 113a11 - F? < +00;
(16)
furthermore, JNL.a(.) is centered Gaussian process 3d-1 (6-1(V[ (Vd), whose covariance function has the form:
T [(ULL((::))xxx((mms)) = E,( 22)/d).
(17)
The kernel Ea satisfies
(VEl) O, B( Vāj and Ea(1) - F? +00,
(18)
where 211) is the first entry of x ( The signal-to-noise ratio (SNR) of the model defined by
F?
p = F? +
(19)
In this "linear proportional" asymptotic regime, we derive an asymptotic formula for the expected PPV akin to that of Mei and Montanari (002) for the generalization error. Analysis shows Definition 1 of Mei and Montanari [2022], characterizing the asymptotics of RAF, can be straightforwardly
applied to also characterize the asymptotics of S
5

---

Proposition 1. Denote by C+ the upper half complex plane: {a + bi : = /-1, b> 0}. Let
functions v1, V2 =C+ - C+ be uniquely defined by the conditions: on C++ (E) v2(E) are analytic and uniquely satisfy the equations
22
41 = 1/1
- 12
(20)
(2
2n
-1
U/ = 12
- - V1 1- --22
(21)
when [v1(E)] S /1/Im(E) and (22(E)) S 2/2 (m)) with Im(E) > C for sufficiently large constant C. Define
x vi 11 )22 (102////1))))
=
P1
(22)
p.
Under Assumptions
3 and for X >> 0,
F?
1 x - F? + as n. S θ A) 4 8²
Note that S depends on (+/1, 2/2, X) only through function X. The following facts follow from this fact and the asymptotic characterization of function when À - 0+.
Proposition 2. The following holds for the map (111 1/2, A) + S² defined in Proposition !
1. It is non-decreasing in X.
2. lim 8 +00 when 401 1/2.
Item 2, Proposition deserves special attention, as it seems to suggest that there is no "double descent" in the asymptotics of 5 when using an improper prior. This fact does not possess an operational meaning at any finite N = n, due to the ordering of the limits N, + oo and X - 0+. In fact, one may expect that the distribution of the least singular value of the relevant matrix Z causes numerical instability for small when N = n. In Section 3.3 this hypothesis is empirically validated through simulations. Subtly, the theoretical prediction does accurately describe the numerical simulations for small choices of X. On the other hand, the asymptotic prediction for the frequentist risk does diverge when À 0+ and 4/1 1/2
3.2 Comparison with Generalization Error
The asymptotics of 8 the "frequentist" risk for our comparison, was characterized by Mei and Montanari [2022) through theorem similar to Proposition Our main interest lies in comparing the risk against the Bayesian variance in two limiting cases:
1. Highly overparameterizeed models where vi - xc. The number of parameters grows faster than any constant multiple of the number of samples.
2. Large sample models where 4/2 constant multiple of the number of parameters.
The number of samples grows faster than any
The first regime has been studied the relevant regime in modern deep learning. The second regime closer to the classical regime of asymptotic statistics where only the number of samples n diverges. Below, we re-state the asymptotic formulae for RRF of Mei and Montanari (2022) in these special cases, which admits simplifications relative to when both v1, 1/2 are finite.
Proposition 3. (Theorem 4-5, Mei and Montanari define a function
2022
Under the notation of Proposition 7
= w(C (C.4,))) (WCC 6 + V 1
2 2(xx +1)
In the highly overparameterizee regime, where the + 0c, asymptotic risk is given as
(Ff + 73 ) + w
Rovide (p. Ç, 1/2, A) lim lim ERaF (v, X, x, (1 (1/2 2 1/2 w////2 wa)
+ F2
t 000 d-+00

---

with We = w(S, V2 x/p2).
In the large sample regime, where 1/2 + co, axymptotic risk is given as
F + wi) Risamp (S; 11, A) = lim lim (R <2(1/1 2w1 + wfthi wff 422+00 d
+ F²
with W1 = w(<, 11 A/1E),
A simple question, mentioned in Section 2.2 was whether the two quantities agree. It turns out that in the second regime, at least, the two formulae converge to the same limit, which is the main content of the next Proposition. In the first regime, whether the limits agree is determined by the signal-to-noise ratio p. Mei and Montanari [022] showed that if is larger than a certain critical threshold pu, which depends (t/2. {), the optimal regularization is achieved by À - 0+, whereas if p is smaller than p., there exists an optimal regularization X bounded away from O. This phase transition also determines the agreement of the risk and the PPV in the limit.
Proposition 4. (Proposition M.i and Montanari
2022 +a) Define quantities
wh.2 w0.2
Pa(5,422) = (1 1/,2) + 1/2
W0.2 = w(S, 222 0)
under the notation of Proposition 3
1. If < p., then minx20 Rwvide (p. C, 12, X), is attained at some o > 0:
Aopt == arg Ruvide 1/2, <21/22 (2 + (2 + w. w (w? - Wa 1122
w. w.:=w(VP 1)2, 0).
Farthermore, Ruiide (pr S, 1/2, Aopt) lim -o S2 (1 th2, (oo1)- - -22
If. on the other hand, P > P.. then arg min R (P, 1/,2,,, = 0. Furthermore, Rwide (p. S, 2/2 0) < lim S2 2/2 0) 7
2. arg minx Risamp (C, 1/, A) = 0 and Risamp (C, 1/ 0) 1 S2 th 0)
Note the subtraction of the noise level 72 from S² This is because S is computed based on the training error in Proposition I which includes both the approximation error and the variance of data, whereas RR is computed based only on the approximation error in the test set.
3.3 Numerical Simulations
In this section, first compare evaluations of the asymptotic formulae for RRF and S for varying (th1, t/2, A). We highlight their difference for finite (11, 4/22) at the optimal choice of X for the MAP risk. We also present numerical simulations, which both validate the formulae (they concentrate fast) and empirically exhibit further interesting distributional properties (suggesting the need for second-order asymptotics). MATLAB codes used to produce simulation results included in the Supplementary Materials.
Figure | shows the dramatic mismatch between RaF and Shr in the low-noise regime. It turns out the conservativeness of the credible ball persists for the choice of that depends on (v),, v22). The situation becomes more delicate in the high-noise regime because there exists a phase transition in the optimal choice of À that depends on 19). which decreases with the noise variance 7², Figures and 2b compare the two curves, R and S, for the "optimally tuned" À at which the best possible frequentist risk is attained, for a fixed pair of (1/1, 4/2). a low-noise task, with p = 5, the ratio of the frequentist risk of the posterior mean predictor to the width of the posterior predictive eredible ball is less than for a wide range of th The situation is more nuanced and possibly more favorable for the Bayesian case in the high-noise task with p = 1/5.
2 and
While Figure D validates good concentration properties of RRF and Phr with respect to their asymptotic formulac, may want to investigate the rate of fluctuations for these quantities. Figure
7

---

2-5.001
Sp Paamrr MTearssoo
1 X. Puneica
1.6
(Thsostac))
16-
E PPV
122
0.0
04-
0--
0.2
25
45
N4h
Nhh
(a) A = 1e-8
(b) A 1c-3
Figure 1: Comparison of asymptotic formula and 20 instances of R (2). Data are generated via
noiseless linear model y = (x, 33) (I|3 = 1, P = oo). Activation is ReLU: (z) max x). d and are fixed to 100 and 300, respectively. The asymptotic formula for R is plotted for comparison (red, dashed).
0.96
0.96
0.6
0..94
30
0.92
0.8
0.75
0.88
6.7
0.86
086
0.84
88
0.02
15
10
(a) t/2 3
(b) 1/1 3
Figure 2: Ratio of R(10Mt to S2(( -2 as a function of thi 2a and of x/
2b In each plot,
4/g and 1i are respectively fixed to while Fi 1, F. 0, and p 1 T 7² eel 5...
for noise variance
M
5n
700
300
800
600
700
700
500
800
300
600
200
200
100
0.4 3.55
1.2
0.5
0.05
0.1
0.15
0.2
022
0.3
Figure 3: Histograms of le+4 draws of RRF and S 72 under low-noise linear model y
(x. 3) + with 7² = 1/5.
3 sugggests an interesting phenomenon: both quantities appear Gaussian, but are nearly orthogonal preciscly near the "interpolation boundary" where REF exhibits double descent (2nd subplot). Our
8

---

empirical results should be compared with the results of Freedman [1999] and Johnstone (2010) The Gaussianity of R and Sk.. if true, strengthens the agreement between the expected width RF and expected risk Rur immediately transfers to the agreement between the frequentist coverage of (1-o) ax) % Bayes credible ball around the mean and the nominal coverage. Again, such claims are generally true in fnite-dimensional settings, but not true for high-dimensional settings. For instance, [999] showed that asymptotically, the posterior variance fluctuates like a Gaussian random variable with variance strictly smaller than that of the frequentist risk. Extracting such information is possible only if we study second-order information of RRF and ShFi in particular, suggests a Gaussian central limit for these quantities. A rigorous proof of central limit theorem for RaF and RF goes beyond the scope of this work. Nevertheless, we conjecture the following for the fluctuations of RRF and Rp:
Freedman
1. Both d(RRF= - ERRF) and d( - RF) weakly converge to Gaussian distribution with appropriate variances. The faster convergence rate of d rather than vd is known to be common in central limit theorems for linear spectral statistics [Lytova and Pastur 2009].
2. When 1/1 S 4/2, asymptotic variance of fuF is smaller than Rnr. When 4/1 1/2, the two distributions are nearly orthogonal. For large enough 1 the asymptotic variances are of the same order.
These conjectures are of independent interest and pose interesting theoretical challenges. We must note that second-order asymptoties has received less attention in the deep learning community. Only recently did Li et al. [2021] study the asymptotic normality of prediction risk achieved by a min- norm least squares interpolator. Their result relies on a central limit theorem for linear statistics of eigenvalues of large sample covariance matrices, established by Bai and Silverstein (2004). Many central limit theorems in random matrix theory seem insufficient to handle kernel learning or learning with random features.
4 Discussion
In sum, our calculations and empirical findings suggest that the RF model still has some interesting discoveries to offer, particularly for those interested in the implications of Bayesian deep learning. The asymptotics of the posterior predictive summaries can be very different from that of the generalization error. Numerical simulations suggest that some version of the central limit holds so that depending on the particular scaling of dimensions, the frequentist coverage of the posterior predictive distribution can be arbitrarily low high. We now conclude with discussion of several conjectures and the next directions suggested by these findings.
Heuristic Implications. While our technical calculations are only applicable to the RF model, we believe the general perspective of comparing frequentist versus Bayesian estimation approaches to the deep learning model can offer useful heuristics explaining empirically observed phenomena in training Bayesian deep learning models.
One suggestion is that the different regimes of asymptotics may explain away the "cold posterior effect", first empirically observed by Wenzel et al. 2020). The authors forcefully suggested that raising the posterior of a deep learning model to an even larger power (i.e., more concentrated) can improve generalization. That the posteriors can be too wide in highly overparameterizee, near- noiseless regimes may explain this issue. Even though in this simple RF model setup, the posterior predictive mean left unchanged from that of the posterior, is plausible that for more complicated models this need not be so. When that posterior is much wider than the expected risk of the mean, we will mix over too many bad predicting weights, so will be actually worse to average over the posterior than not.
Technical Improvements. The technically most interesting but quite challenging direction, suggested by our numerical simulations in Section 3.3 is to study second-order fluctuations of random matrix quantities often encountered in deep learning. While at this point a vast amount of literature exists that systematically overcomes the non-standard definitions of quantities like generalization error from the viewpoint of random matrix theory, we are aware of no study as of yet that overcomes the same issues to show a central limit-type result. On the other hand, central limit theorems for linear spectral statisties remain an active area of research random matrix theory and are generally acknowledger to require more advanced, subtler arguments that are fundamentally different from first-order results

---

(i.e., convergence of empirical spectral distribution). For start, a separate work on the central limit theorem for linear spectral statistics of kernel matrices is in preparation by the authors.
Another interesting direction is to extend the comparison between frequentist and Bayesian modeling approaches more complex models of training dynamies. For instance, AAdlam and Pennington [2020) have derived precise asymptotics of generalization error in the neural tangent kernl (NTK) regime, in which the features ④ are "learned" through a linearized approximation of the gradient flow in training. MMi et al. 2018 2019] have suggested a different asymptotic model, in which O evolve through nonlimear dynamics. An interesting challenge, both technically and conceptually, is adapt their analysis to the Bayesian setting, where the gradient flow on the space of posterior measures of O is approximated and specific projections are tracked.
Making the "Right" Comparison. A possible objection is that the comparison between two quantities is not appropriate: if interested in the coverage of function fa with our credible set, then one must compare the expected width of the interval of the function space posterior, f. instead of that of the Bayesian model average. While the objection is sensible, the fact is that the average variance of the function posterior will be an even worse uncertainty estimate for underparameterizeed models, whereas it will make little difference for overparameterized models. The reason is that the variance of the posterior of f only differs from that of the posterior predictive, in (II), by the training set error term (1... The training error dominates the PPV precisely for underparameterized models (1/1 < t22), so the curves of Figure [ will now be non-decreasing and asymptotically convergent to similar constant in expectation. This fact illustrates that a simple interpretation of the width of the posterior complicated in even simplistic high-dimensional models.
Another possible objection is that the same value of À need not be optimal for the posterior predictive. Since the asymptotic curve for S2 is non- non-increasing in X, this implies must choose even smaller than A to obtain less conservative credible intervals, regardless of the SNR p. This seems an odd consequence of using overparameterized models, where a good frequentist coverage is obtained by choosing a narrower prior than that with good "contraction property" (i.e., minimal 2 risk of posterior mean).
The final objection is that the RF model is simply not a very realistic model of practical Bayesian deep learning. We point out that quantifying the behavior of RF models as approximations of kemel learning methods remains a valid concern for Bayesian practitioners.
Acknowledgments
All authors would like thank Matthew M. Engelhard, Boyao Li, David Page, and Alexander J Thomson for the helpful discussion around the theoretical work.
Samuel I. Berchuck would like to acknowledge support from the National Eye Institute of the National Institutes of Health under Award Number K99EY033027. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Sayan Mukherjee would like to acknowledge partial funding from HFSP RGPOOS, NSF DMS 17-13 NSF BCS 1552848, NSF DBI 1661386, NSF IIS 15-46331, NSF DMS 16-13261, as well as
computing partially supported by grant 2016- IDG-1013 from the North Carolina Biotechnology Center well the Alexander von Humboldt Foundation, the BMBF and the Saxony State Ministry for Science.
References
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-986 2022.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49(2):1029-1054, 2021.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 (4):667-766, 2022.
10

---

Hong Hu and Yue M Lu. Sharp asymptotics of kernel ridge regression beyond the linear regime. arXiv preprint arXi:22005.06798, 2022.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-244 248, 2021.
Yaniv Ovadia, Emily Fertig. Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Vincent Fortuin, Adrià Garriga-Alonse Sebastian W Ober, Florian Wenzel, Gunnar Rätsch, Richard E Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. arXiv preprint arXiv:2 102.06711, 2021.
BJK Kleijn and AW van der Vaart. The bernseiin-oon-mises theorem under misspecification. Elec- tronic Journal of Statistics, 6:354-381, 2012.
Dennis D Cox.
analysis of bayesian inference for nonparametric regression. The Annals of Statistics, pages 903-923, 1993.
David Freedman. Wald lecture: On the bernstein-von mises theorem with infinite-dimensional parameters. The Annals of Statistics, 27(4):111-11141, 1999.
lain Johnstone, High dimensional bernstein-vor von mises: simple examples. Institute of Mathematical
Statistics Collections, 6:87, 2010.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training differentiable programming. Advances in neural information processing systems, 32, 2019.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665-E766 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-fe free bounds and kernel limit. In Conference on Learning Theory, pages PMLR, 2019.
Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78 (5):1103-1133) 2016.
Lorin Crawford, Kris C. Wood, Xiang Zhou, and Sayan Mukherjee. Bayesian approximate kernel regression with variable selection. Journal of the American Statistical Association, 113(524): 1710-1721, 2018.
BT Knapik, AW van der Vaart, and JH van Zanten. Bayesian inverse problems with gaussian priors. The Annals of Statistics, 2965)2266-2657, 2011.
Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Theoretical characteriza- tion of uncertainty in high-dimensional linear classification. Machine Learning: Science and Technology, 4(2):025029, 2023a.
Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. On double-descent in uncertainty quantification in overparametrized models. In International Conference Artificial Intelligence and Statistics, pages 7089-7125. PMLR, 2023b.
Alice Guionnet, Justin Ko, Florent Krzakala, and Lenka Zdeborová. Estimating rank-one matrices with mismatched prior and noise: universality and large deviations, 2023.

---

A Lytova and L Pastur. Central limit theorem for linear eigenvalue statistics of random matrices with independent entries. Annals of Probability, 37(5):1778-1840, 2009.
Zeng Li, Chuanlong Xie, and Qinwen Wang. Asymptotic normality and confidence intervals for prediction risk of the min-norm least squares estimator. In International Conference Machine Learning, pages 5533-6542 PMLR, 2021.
ZD Bai and Jack W Silverstein, Clt for linear spectral statistics of large-dimensione sample covariance matrices. The Annals of Probability, 32(1A):553 605, 2004.
Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swikatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020.
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. International Conference on Machine Learning, pages 74-84. PMLR, 2020.
12

---

A Proofs
Proof of Proposition
The necessary calculations follow immediately from the proof of Mei and Montanari [2022). Since we do not wish to replicate the long proof with only minor readjustments, we only sketch the main argument. The expected PPV consists of two random variables, denoted by
(fa: yy. y f(x)) rXX, O, A) == (x) Z A o (x)/d.
(23)
It suffices to show the two quantities individually converge in probability some limits. The first of the two is a training set error, for which an asymptotic formula was derived in Section 6, Mei and
Montanari 2022
/ (iwad/ u.) lim EØ (
A
F?
+ F??
+ +2)
(24)
1/2
1 X²2
using the definition of quantities in Proposition I for x x(iv 12122//H11) 1i (i =i= V-11. At this point, rearranging the fixed point equations (20) and
(i=v-1).
21 with & =8V0'122 8/2\ Hr. we obtain:
-2-2 iv Xeoi /2/ = 1 53x
- / 10/12//1111
(25)
32x
X 4/2
(1-xx+1) +1)
5² Tx Gx
1+ V2
(26)
similar formula for Er can also be derived, using almost exactly the same steps required for simplifying asymptotic formulae of R and Φ-1 Assumption 3 on the nonlinear component of the generating model. J.N implies that it admits an explicit decomposition into spherical harmonics that form a basis gd-1 and coefficients with Gaussian distributions (appropriately scaled). With many simplifications, also sketched out in Appendix E. Mei and Montanari 2202). we can write
Er = 14 T t Z Ax /4) + TTT"( Z + X 11) + oz((). (27)
Next, form an (N + +n) x (N + m)ddimensional block matrix A, indexed by a 5-d real parameter q (s1, 82, 11, p):
ssIi + 00/d z 2"+ /Va
A(g)
(28)
Z + pxp /Vd teh, 1xXX/d
Denoting by & C+ the log-determinant of the resolvent matrix A(q) +IN+ as GalE; q). straightforward calculus and plugging in the figures reveals
Er = -47 - VINUA
D(O., Ga(iv Adira 0)] 22
EIO,, Abh 0)] + oa(1).
(29)
At this point, we pass on to the limit d + oc and plug in appropriate asymptotic limits of the partial derivatives. Since the E-derivative of Go is the trace of A(g) {IN+n. this possible by characterizing the asymptotics of the Stieltjes transform of the empirical spectral distribution of A(q), which done using Proposition 8.3, Mei and Montanari [2022]. Integrating the limit over path stretching in the imaginary axis (iK with K - oc), one then rigorously passes onto the limit by Proposition 8.4, MMi and Montanari (2022). Final simplifications using Lemma 8.1, Mei and
Montanari 2022 yield the formula
H1.
g2 (1-xx+1)
+1).
lim Er = in Un (1122/p.)) d0000
Vediz (1-
(30)
At this point, we may conclude, using
30 and 26
)1t)))
F?
+ + +72
33
13

---

Proof of Proposition| 2
The first point is consequence of the fact that Sür is decreasing in À and passing onto the limits. The second point is immediately seen by taking the limit of X as À - 0, formula for which is provided by Theorem 3, Mei and Montanari [2022]
Proof of Proposition
We want to check whether equality between asymptotic risk and the expected PPV holds. Using Proposition 3 In the highly overparameterized regime, we demand equality
F/ + (F? 7 Jw3
F?
Rwide
1/2 2022/22 + w 1/2 W2
-22
W2 (pv2 - p 1) 122 - 0.
By Proposition 13.1, Mei and Montanari 2022]. this equation met precisely by choosing Ā= X only possible if p < p² per the definition of Proposition Otherwise (i.e., if >> p²). the equality is unsatisfiable, and furthermore,
the property of the map w w v, () in Lemma 13.1,
Mei and Montanari 2022]. to show that w + (pr52 - P1)w3 w/apwr >
In the large sample regime, we similarly want check the equality
wF/5² + 1h
V1 - 21/2wi w/w w/
1 1- w/
w/ + (5 - 2 1) 1/1 22 - 0.
In this case, the demanded equality does not have p in it. Utilizing symmetry and the same argument as in Proposition Me. and Montanari [2022]. one can show that the absence of implies the
equality is always satisfiable by choosing À 0, which also corresponds to the minimizer of the asymptotic risk: arg Rismp = 0.
14


Reviews:
[{"review_id": "KOto9PSN0S", "reviewer": "Reviewer_KMbc", "summary": "The main focus of the paper is on the comparison between the posterior predictive distribution and the frequentist risk associated with the maximum a posteriori estimator for random features ridge regression model. The target function is assumed to be a sum of a linear model, a non-linear function given by a Gaussian process with specific kernel (see assumptions), and an iid noise model with mean zero, constant variance, and bounded fourth moment (see Eq. 1). The technical part starts with a review of frequentist risk and empirical estimator, given in Eq. (7). This is followed by a Bayesian model that assumes normal prior on the weights and likelihood model that conditions on the random features and weights of the linear model (see Section 2.2). The posterior distribution is the Gaussian centered at the maximum a posteriori estimator with covariance matrix given in Eq. (6). This is standard for the weight space derivation of Gaussian processes. The posterior predictive at an instance x is given in Eq. (8) and is just variance at that point. The expected posterior predictive (EPP) is given in Eq. (9) and is obtained by taking the expectation relative to the instance space. The paper then provides an asymptotic characterization of EPP relative to prior work by Mei & Montanari (2022) on frequentist risk. Numerical simulations illustrate the difference between the two concepts and show the lack of double-descent phenomena in the Bayesian setup.", "questions": "Are the instances $x_i$ from the interior of the d-dimensional sphere or from its surface (\"around\" is not precise enough)? If the latter, why is this assumption necessary and to what level it affects the results? Normally, one assumes that instances are contained within a ball of constant radius and this appears to be a bit stronger assumption.\n\nIn Eq. (1), I see $f_d^{NL}$ defined but not $f_d$. The latter appears in Eq. (7) and my understanding is that it refers to the function described in Eq. (1). Could you please do a pass over the notation?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is a nicely presented work with an interesting insight into random features and the differences between Bayesian and frequentist notions of uncertainty. The theoretical result is cumbersome to parse but still provides an interesting illustration and side-by-side comparison with Mei & Montanari (2022). Numerical simulations also support the theoretical conclusions. The discussion is also valuable, pointing at some of the shortcomings and directions for future research.", "weaknesses": "The assumptions might be constraining the problem setting strongly but still the results provide an interesting insight on what happens asymptotically. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n- We agree that the assumptions can be quite restrictive. We address the input distribution below. \nBesides the input distribution, we also assume a Gaussian prior structure on the weights and a form of analytic expansion of both the learned function class and the activation function.\nExtending the former is an interesting technical challenge. The case of log-concave priors is likely manageable with existing theoretical tools. Similarly, we also believe more technical work can be done to cover different likelihoods, such as that for logistic regression. \nThe latter assumption, while technical, does cover the ReLU activation, so it is flexible enough to allow non-smooth functions.\n\n## Questions\n\n- Inputs are distributed on the **surface** of the sphere (we will rephrase the sentence at the camera-ready stage). Input distribution can affect the results to some extent if it is not \"generic'' enough. \nFor example, when it is Gaussian, one should expect in high dimensions many results should similarly hold due to the concentration of measure (e.g., see Cheng and Singer [2013]). \nWhen there is dependence within input or if the input is distributed non-uniformly around the sphere/in the ball, it is harder to predict possible changes in the results. \n\n- $f_d$ is, as pointed out, meant to be a fixed function used for generating data in equation (1). We will clarify this at the camera-ready stage."}, {"review_id": "w3fJlD2hiG", "reviewer": "Reviewer_gLVj", "summary": "This paper considers the random feature model. Two objects are studied: (1) The posterior predictive distribution (in particular, the variance), and (2) The MAP estimator. This paper gives asymptotic formulas for these two quantities under the proportional regime. Comparison between these two quantities are made, in terms of both theory and experiments. Their results suggest that the posterior predictive summaries can be very different from that of the generalization error. ", "questions": "1. Line 312: \"worse an uncertainty estimate -> worse than uncertainty estimate\"\n2. It is not clear why they consider averaged posterior predictive variance versus generalization error. The authors did present an explanation in the end. However, this explanation, at least to me, is like simply stating \"these two quantities are different and identical in several regimes\". I think a more convincing argument should be provided. \n", "limitations": "Limitations have been adressed, and societal impact not applicable. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "This paper is among the few works that address the following problem: Are the Bayesian credible sets also valide confidence sets in the frequentist sense. This is true in the finite dimensional setting with n \\to \\infty, while unclear in the high-dimensional setting. They give precise expressions for the two targeted quantities: generalization error and posterior predictive variance. They give rigorous results to show that these two quantities agree and disagree in different regimes. The presentation is nice and clear. Extensive simulation is conducted to justify several unproven claims.   ", "weaknesses": "1. Their setting is a little bit restrictive: Only one specific prior structure is considered. Having said that, I understand that choosing this prior is for getting a precise asymptotic expression for the quantities of interest. \n2. Their theoretical results seem like direct consequences of [Mei and Montanari 2022], perhaps the authors should highlight their technical contributions comparing to previous works. This is my major concern, and I will raise my score if this question is well addressed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n1. We agree that the Gaussian prior structure is restrictive. Incorporating more diverse priors (log-concave, scale mixture of Gaussian, etc.) is an interesting technical challenge. Extension to log-concave priors seems to be the natural immediate next step.\n\n2. We agree that our manuscript relies heavily on the technical tools of Mei and Montanari (2020). Our main goal was not to add to this technical machinery, but to use it to ask a question about the confidence properties of Bayesian procedures that were not in the interest of previous authors. \nWe also identified new technical directions in the manuscript that will be pursued by the authors in future works. Overall, it is of interest that the technical tools of Mei and Montanari (2020) can be used to analyze Bayesian procedures and derive different asymptotic properties. \n\n## Questions\n\n1. Line 312 is *not* a typo. However, we will change the phrase to \"an even worse uncertainty estimate,\" as it reads much more clearly without possible confusion.\n\n2. We believe there are two angles on why expected PPV is a meaningful comparison point.\nFirst, it is a knowable, computable quantity from the researcher's point of view without knowledge of the true data-generating mechanism $f_d$. The expected PPV investigated in the manuscript is computable for a given instantiation of the training data and provided as a statistic for UQ. Its asymptotic average behavior *does* depend on the assumptions we make on $f_d$.\nTherefore, our analysis should be read as highlighting whether the two quantities, standing for Mei and Montanari's \"unknowable risk\" and the \"Bayesian's risk for UQ,\" behave similarly under the same technical assumptions on the data generating mechanism.\nSecond, as we explain in the Discussion (lines 308-318), it is possible to either consider the posterior of $f$ induced by the weight posterior or the posterior predictive obtained by marginalizing out the weights. Both are Gaussian processes indexed by $\\boldsymbol{x}$, which is randomly drawn. Other than the technical point explained in the manuscript, we also believe that prediction for future data is of high interest in most supervised learning tasks, and UQ should also target predictive uncertainty in this case. Hence, we choose the variance of the posterior predictive distribution as a comparison point for the generalization risk."}, {"review_id": "JKHyGo1lrh", "reviewer": "Reviewer_dJw1", "summary": "The authors compare in this paper the asymptotic posterior variance of the predictive risk in the random feature model  associated with a Gaussian prior on the weights with the  asymptotic \\textit{frequentist risk} which had been derived by Mei and Montanari.  The asymptotic here is in terms of the dimension d of the covariates, in a context where the number of features $N$ and the number of observations $n$ are comparable with $d$. This regime is often studied to understand the phenomenon of double descent. \nThey derive under the same assumptions as in Mei and Montanari the asymptotic behaviour of the posterior predictive risk and show that in the \"wide\" regime, i.e. when N/d converges to infinity with low signal noise ratio or when n/d goes to infinity both risks (under the optimal choice of the hyperparameter $\\lambda$ for the frequentist risk) have the same limit while when N/d goes to infinity and the \"SNR \" $\\rho$ is larger than some threshold then posterior variance is asymptotically larger than the frequentist risk which is 0. The authors then perform a simulation study to understand more precisely the role of $\\lambda$ and the different asymptotic regimes. ", "questions": "The paper has a number of typos and inconsistencies which makes it a bit cryptic  and which I will list below. In particular the notion of proba under which expectations re calculated are not always clearly stated . In the context of this paper it is particularly important to be precise in this because these expectations are somewhat non standard. \n\n1. p4 line 110: The comment on the credible radius is cryptic or incorrect. By definition a credible radius depends on the coverage (Bayesian) of the credible ball. So it cannot be equal to the posterior variance for all coverage. This sentence makes no sense to me. \n\n2. In the definition of the Bayesian model in line 95, the variance of the noise is denoted $\\phi$ while in the previous section it was $\\tau$. Is this a typo or is there some confusion there or is it that the model may also be mispecified in the variance component? If it is purely a notation inconsistency an $\\phi $ also denotes the variance  of the noise, then the paragraph in lines 118-123 is puzzling. It reads as if it was a quantity chosen by the analyst , while it is driven by the data. It is not completely clear to me why $\\phi$ should go to 0 while in the subsequent computations $\\tau$ is held fixed.\n\n3. Proposition 1. What is $\\mathbb C_+$? Also what does the sentence \"and for expectation over the randomness of $(y, X, \\Theta)$  mean? There is no expectation in the statement below. \n\n4. Proposition 3: Recall what is the Expectation (with respect to which variables)\n\n5. Do the activations functions considered in the simulated examples satisfy the assumptions ? Provide some examples which satisfy the assumptions. \n\n6. What are the different curves in figure 2?\n\n7. line 233 : What do you mean by \" conservativeness of the credible ball \" ?\n\n8. What do you mean by \"he posterior variance fluctuates like a Gaussian random variable with a variance strictly smaller than that of the frequentist risk\"?  (l 250) The posterior variance is deterministic in a Bayesian Gaussian - Gaussian model (as in Freedman). \n\n9. What does the sentence l l286 mean : \", the posterior\n286 predictive mean is left unchanged from that of the posterior, \"? Similarly I don't understand the last sentence of that paragraph. \n\n10. l310 : What do you mean by \"interval of the function space posterior, $\\hat f$\"? abd by the variance of the posterior of $\\hat f$ a few lines below?\n", "limitations": "The authors have interesting remarks on the open questions raised by the paper and what are the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The comparison of the frequentist and the Bayesian approaches in  the double descent  types of regimes is a very natural question and treating the random feature model seems like the obvious first choice to attack the problem. The results derived by the authors seem correct and the simulations shed light on the results. The proofs are a consequence of the proofs of Mei and Montanari, but remain technical (at least the proof of Proposition 1). \n\n The posterior variance , as studied by the authors is strongly related to the radius of credible regions therefore understanding if the posterior variance is similar to the frequentist risk is a first step to understand if credible balls are confidence balls, as hinted by the authors. As noted by the authors this is not a full statement on coverage of credible balls and higher order asymptotics is required  to understand the frequentist coverage  of credible balls. ", "weaknesses": "What I am not so clear about is what is the conclusion/ consequence of the results obtained in this paper. In particular, the \\textit{frequentist} risk of Montanari is not really frequentist and in my opinion, it  has already a Bayesian flavour. The risk $\\|f_d - \\hat f\\|_2^2$ (which is in fact a loss function) is viewed in Montanari as stochastic both in terms of the data $X,Y$ appearing in $\\hat f$ but also in $f_d$ (not to speak of the randomness $\\Theta$) and its asymptotic behaviour is under this regime. Hence, it would be interesting to have a proper discussion of the different notions of risk which are considered in the paper. I also think that the authors do not comment enough on why in some regimes the two risks are equivalent and not in others. Some description in terms on $\\tau$ (noise variance ) being large or small in the wide regime is provided but no real interpretation. \n\n\n\n That being said I still find the results interesting, but I am uncertain after reading as of to what ot make of them. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n- As pointed out, the risk $\\mid f_d - \\widehat{f}\\mid^2$ appearing in Mei and Montanari's work is viewed as stochastic in the input. Even for a specific instantiation of the training data $(\\boldsymbol{y,X,\\Theta})$, however, the quantity depends on the \"truth\"\n$f_d$ and is unknown to the researcher.\nMei and Montanari's theory provides an explanation of the average (and high probability) behavior of the risk of the learning procedure under certain assumptions on $f_d$.\nThe expected PPV investigated in the manuscript is computable for a given instantiation of the training data and provided as a statistic for UQ. Its asymptotic average behavior *does* depend on the assumptions we make on $f_d$. \nTherefore, our analysis should be read as highlighting whether the \"unknowable risk\" and the \"Bayesian's risk for UQ\" behave similarly under the same technical assumptions on the data-generating mechanism. This was exactly done in the problem of infinite-length Gaussian denoising by Freedman, hence our repeated reference to his work.\n\n- The interpretation of the role of $\\tau$ is subtle. We elaborate a little on the left panel of Figure 2 for one such interpretation. The blue curve shows that for \"near-noiseless'' problems, where there is little noise and the error primarily comes from approximation error with respect to $f_d$, the actual generalization risk decreases faster than the posterior predictive variance as the model becomes more overparameterized. \nThus, overparameterization is primarily beneficial for \"noiseless\" learning problems where the posterior mean provides a near-perfect prediction. This regime can be precisely pinned down by the phase transition of Mei and Montanari (Proposition 4). The Bayesian's posterior predictive distribution centered around the mean, roughly speaking, does not contract as it should with increasing dimensions, so uncertainty estimate based on its variance will be \"conservative\" than it should be for future predictions. The formulae suggest the possible existence of \"sweet spots\" of noise level, scaling with dimensions, where the ratio is nearly 1.\n\n## Questions\n\n1. We note that we did not provide a precise definition of the \"credible ball,'' which led to the confusion. In our manuscript, *$(1-\\alpha)$-\\% credible ball* is simply synonymous with the $L^2$-ball centered around the posterior (predictive) mean containing $(1-\\alpha)$\\% posterior (predictive) mass, so it need not have the nominal coverage. We will explicitly define the term at the camera-ready stage to eliminate this confusion.\n\n2. $\\tau^2$ is the variance of the noise corrupting the observations. $\\phi$ can be interpreted as essentially estimating $\\tau^2$. Hence, the specific choice of (10) is well-justified. We used the more generic notation $\\phi$ to emphasize that (10) is not the only choice, and that it must scale appropriately with the relevant dimensions $d,n,N$ for our asymptotics to be meaningful.\n\n3. $\\mathbb{C}_+$ is the upper half complex plane containing all complex numbers with positive imaginary part. The phrase \"expectation over'' should be omitted, which we will do at the camera-ready stage.\n\n4. Expectation is with respect to all random quantities needed to define $R_{RF}$; hence, it is an integral over the joint measure of $\\boldsymbol{y,X,\\Theta}$.\n\n5. The main example, ReLU, satisfies the assumptions. Other examples include mixed-order polynomial functions.\n\n6. The appropriate legends were omitted from Figures 1 and 2. In Figure 2, the two curves correspond respectively to the ratio described in the caption for high $\\rho$ (blue) versus low $\\rho$ (orange). New figures with legends have been included in the PDF attached to the general rebuttal.\n\n7. This confusion, we believe, is also caused by our non-definition of the term \"credible ball.\" Since the $L^2$-ball around the posterior mean need not have the nominal coverage, its \"conservativeness\" implies that its actual coverage is strictly larger than the posterior mass it contains.\n\n8. As pointed out, the phrase \"posterior variance'' is incorrect and should be revised. We would like to elaborate on what is meant by this phrase and why we make the comparison to Freedman's result. \nIn Freedman (1999), a simpler problem is studied, where an infinite-length sequence $\\beta\\in\\mathbb{R}^\\infty$ is observed with additive Gaussian noise (with known variance). The phrase \"posterior variance'' must be changed into the \"Bayesian pivot'' in equation (4), Freedman (1999): $\\mid\\mid\\beta-\\widehat\\beta\\mid\\mid^2$,\nwhere $\\widehat{\\beta}$ is the posterior mean of $\\beta$ under a Gaussian prior. Unlike in our setting, Freedman's has essentially only two sources of randomness, contributed by a prior distribution and observational noise. Hence, the expectation of the \"pivot'' with respect to the posterior of $\\beta$ is deterministic. On the other hand, the expected PPV in our study is random, as it depends on a given instantiation of the training data $(\\boldsymbol{y,X,\\Theta})$.\nThe key feature we wanted to emphasize was the similarity between Figure 3 and the near-orthogonality of the laws of \"frequentist'' and \"Bayesian'' expectations of the pivot found by Freedman. While it would have been best to formally expand on the notations used by Freedman in his study, we could not due to lack of space and organization.\n\n9. Line 286 means that the posterior predictive distribution, $p(y|\\boldsymbol{x,y,X,\\Theta})$, is centered around the same mean as the posterior distribution, $p(f(\\boldsymbol{x})|\\boldsymbol{x,y,X,\\Theta})$. This need not be true in the case when there is no Gaussian-Gaussian conjugacy structure as in our manuscript.\n\n10. Line 310 means that one may compare not the posterior predictive distribution of $f$, but its posterior, as described above."}, {"review_id": "jsqU1fCgTO", "reviewer": "Reviewer_Stco", "summary": "The authors study random feature (RF) regression in the high-dimensional setting. \nThey focus on comparing the variance of the Bayesian posterior distributions (a measure of uncertaintiy in the Bayesian setting) to the variance of the maximum-a-posteriori estimator (a measure of uncertainty in the frequentist sense, and the generalisation error, already studied in previous work), motivated by previous work showing a discrepancy in the two quantities in the high-dimensional setting. They ask whether the two quantities have the same high-dimensional asymptotics.\n\nTha authors compute the high-dimensional asymptotic of the variance of the Bayesian posterior distribution. \nThey find that it equals the variance of the maximum-a-posteriori estimator only in the region where the optimal L2 regularisation is not vanishing in the strongly overparametrised case, and that they always match in the large sample complexity regime, close to the classical low-dimensional limit.\n\nThe authors study numerically the fluctuations of the two observables studied, and conjecture that they are Gaussian distributed, and such distribution have non-overlapping support around the interpolation threshold.\n\nComment: \nThe main proof seem a minor variation on reference \"Mei and Montanari, 2022\". \nA reader familiar with that work (which I am not) should be able to easily tell whether the proof of the results of the authors is correct. \n", "questions": "- The authors observe empirically and conjecture near-orthogonality of $R$ and $S^2$ at the interpolation threshold $N=n$. Is there any intuition why this should be the case? And what are the possible consequences of this fact?\n\n## Suggestions for manuscript improvement\n\n- this is more of a personal preference, but I like to number all equations in my manuscripts. This makes easier to readers (and reviewers) to refer to a given equation when providing feedback.\n\n- Eq between line 103 and 104: $p(y | x, \\Theta, a)$ is not clearly defined. Do the authors mean that this is the same distribution as the second line of the equation between line 95-96? This should be specified. \n\n- Eq between line 105 and 106: missing subscript $i$ for the datapoint $x$. \n\n- line 106: possibly wrong reference to Eq 5. Maybe the authors are referring to Eq 4, or want to say that the Gaussian measure is centered around $\\hat{a}$.\n\n- equation between line 113 and 114: the authors could derive this equation (even if trivial) either in the main text or in the appendix. For example, I do not see at first sight how to derive it by averaging over $x$ in Eq.8, which by Eq. 9 is the definition of $S_{RF}^2$. Line 115-116 seems to imply that indeed this equation is not derived from Eq. 8. Where does it come from? Over which Gaussian process is the average performed?\n\n- line 123: where is the marginal likelihood defined? Where does Eq 10 comes from?\n\n- line 183-190: this paragraph is not clear to me. Is \"The second point\" the full Proposition 2, or just point 2 of Proposition 2? Why does this suggest that there is not double descent? I only see monotonicity in the L2 regularisation parameter and boundedness.\n\n- line 209-211: it is not apparent at first sight that in the second regime equation between line 207-208 and the one between line 176-177 coincide. Is there an easy way to see it? If not, this should be done explicitly, or better justified, somewhere.\n\n- Figure 1: the caption is incomplete (and a legend is missing). One can guess what the circles and the black curve are, but it would be clearer if the authors wrote it explicitly. Also, the authors should specify how the error bars are computed.\n\n- Figure 2: apart from the dashed line at 1, I see two curves here, orange and blue. It seems that each curve is for a different value of $\\rho$, but it is not clear which one is which. I guess from lines 237-240 orange is smaller rho, but this should be clearly specified in a legend.\n\n- line 241: Figure 1 shows concentration only for $S^2$, and not for $R$ for which simulations are not provided, contrary to what this phrase states.\n\n- Figure 3: it would be clearer if the authors listed here all the sources of disorder causing the fluctuations of the quantities plotted \n\n- line 285-287: doesn't the author find in Proposition 4, point 1, that there exists an overparametrised regime where $S^2 > R$, i.e. the posterior is unecessarily wide? Maybe I am misunderstanding this phrase.\n\n- line 216: possible typo $+\\alpha$", "limitations": "The authors discuss nicely limitations/shortcomings of their choice of observables to compute in Section 4.\n", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The authors compare Bayesian and frequentist notions of variance-like observables, computing a novel characterisation for one such variable. They also provide interesting numerical observations of the flucuations of such variables, proposing a number of new conjectures.", "weaknesses": "- The paper is not badly written overall, but many details (see list below) could be improved.\n- The authors provide no code to reproduce their figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n* We thank the reviewer's suggestions for manuscript improvements. We respond to specific items below.\n\n* We have provided access to an anonymized code used to produce the Figures to the AC.\n\n## Questions\n\n* At the \"interpolation threshold\" of $N=n$, two main features are notable. \nFirst, there exists a numerical instability for both quantities plotted that we believe (but do not exactly prove) is mainly driven by the near-non-invertibility when $N=n$. The justification is the literature on the minimum singular value of square random matrices (e.g., see Tao and Vu [2009]). \nSecond, the two risks appear \"nearly orthogonal\" in law, which was a crucial finding of Freedman in the Gaussian sequence estimation problem. We do not have a theoretical explanation as to why this regime is important in observing this phenomenon. As remarked in the manuscript, second-order asymptotics of the two risks seem key to understanding it.\n\n## Suggestions for manuscript improvement\n\n1. We agree with the reviewer's point about numbering. All equations will be later renumbered.\n\n2. The distributions specified in lines 95-6 also should be denoted as $p(\\boldsymbol{a})$ and $p(\\boldsymbol{y}|\\boldsymbol{X,\\Theta,a})$, as suggested. These will be made more explicit.\n\n3. Line 106 is meant to read \"centered around $\\widehat{\\boldsymbol{a}}$\" and *not* $\\widehat{\\boldsymbol{f}}$. This will be corrected.\n\n4. We will incorporate the suggested change.\n\n5. The decomposition of (9) follows from (8), distributing $\\phi^{-1}$ inside the parentheses, along with the posterior on the weights given above line 106. The weight posterior induces a posterior distribution on the Gaussian process $\\{f(\\boldsymbol{x}\\}$ indexed by $\\boldsymbol{x}$ uniformly drawn from the sphere, centered around $\\widehat{f}$. The expectation in (9) is the expectation over this posterior Gaussian process, as stated in lines 110-1. The $L^2$ notation is an abbreviation of averaging over $\\boldsymbol{x}$. We will change it into $L^2(\\mathbb{S}^{d-1}(\\sqrt{d}))$, so that it is more strongly suggestive.\n\n6. The marginal likelihood is not explicitly stated, but derivable as that of a Gaussian distribution induced by the prior on weights $\\boldsymbol{a}$ and the Gaussian likelihood model (referred to in item 2 above).\n\n7. \"Second point\" indicates **bullet 2** of Proposition 2. By \"no double descent,\" we indicate that there is \"no blow-up of risk that separates the underparameterized and overparameterized regime.\" In this sense, it cannot happen because of the boundedness.\n\n8. Lines 209-211 summarize in words the mathematical statement that follows (Proposition 4). It is not meant to be immediate at first sight and is proven in the supplement. We will add a sentence pointing the reader to the proof at the camera-ready stage.\n\n9. Figures with appropriate legends included are now in the PDF. The error bars plot 1 standard error interval around the average of numerical simulations at each point of $\\psi_1$. We will incorporate the Figures and changes in the caption into the manuscript at the camera-ready stage. \n\n10. The point is addressed by adding legends in Figure 2.\n\n11. As pointed out, numerical simulations are not explicitly shown for $R_{RF}$. Concentration was already demonstrated in Figure 1, Mei and Montanari (2022).\n\n12. We describe the sources of randomness contributing to the histograms in Figure 3. The randomness is due to a specific instantiation of the training data $(\\boldsymbol{y,X,\\Theta})$. The true data-generating mechanism, represented by function $f_d$, is fixed. The two risks for a given instantiation of the training data are averages over $\\boldsymbol{x}$ uniformly drawn from the sphere, which in numerical simulations is approximated by Monte Carlo averaging. We will elaborate on the sources of randomness for interpreting plots at the camera-ready stage.\n\n13. We are here addressing a general situation with not necessarily Gaussian prior on the weights and a possibly different likelihood. In this scenario, we expect qualitatively similar findings about the posterior distribution, but this is not exactly proven in the manuscript.\n\n14. \"$+\\alpha$\" is *not* a typo, but rather a notation to indicate that we are re-stating a theorem from the previous paper, along with our additional result. Because our point is to make a comparison between the asymptotics of two different quantities, it seems reasonable to not precisely separate the two statements."}]

Author rebuttal:
We thank all reviewers for the careful and constructive feedback. We list below a general re-emphasis of the main point of our work and propose changes at the camera-ready stage that directly address the reviewers' concerns.

## General Rebuttal

- Broadly speaking this paper examines what is similar and what is different between the MAP setting versus marginalizing the posterior. This question is of fundamental importance in understanding the difference between penalized frequentist procedures and fully Bayesian frameworks. There are two interesting results in this paper:

    1. there is a difference between the two inference procedures;
    
    2. the mathematical analysis in the paper by Mei and Montanari (2022) can be adapted to analyze a fully Bayesian procedure.

## Proposed changes at the camera-ready stage

- Mathematical notations and descriptions of the main propositions will be reviewed and revised according to the reviewers' suggestions.

- The appropriate legends have not been included in Figures 1 and 2. **We have included the revised Figures in the attached PDF file.** They will be incorporated into the manuscript later.

Meta-review: This manuscript discusses some interesting asymptotic relationships between the behavior of the MAP estimator for Gaussian process regression and the full Bayesian posterior that marginalizes hyperparameters against their posterior.

The reviewers expressed interest in the results of the paper and overall expressed support in favor of its acceptance. The results appear correct and likely to be of interest to the NeurIPS community. This work would be a welcome addition to the conference program.

I want to stress that the reviews did reveal a number of confusing elements in the manuscript as submitted, which the authors were able to successfully address during the author--reviewer discussion period. I strongly recommend that the authors take this feedback and discussion into account in preparing an updated version of the manuscript.
Decision: Accept (poster)