Title: Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: asymptotics

random features model

Bayesian inference
Abstract: In this paper we compare and contrast the behavior of the posterior predictive distribution to the risk of the 
the maximum a posteriori estimator for the random features regression model in the overparameterized regime.  We will focus on the variance of the  posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grow faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, this is of independent theoretical interest.

Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression

Abstract

In this paper, we compare and contrast the behavior of the posterior predictive distribution to the risk of the maximum a posteriori (MAP) estimator for the random features regression model in the overparameterized regime. We will focus on the variance of the posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grows faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, which is of independent theoretical interest.

1 Introduction

One of the most surprising empirical observations in deep learning is the generalization of overparameterized models that can perfectly interpolate the data. The 'double descent' curve, referring to the test error first increasing then decreasing with model complexity, has been both empirically and theoretically validated for linear [Hastie et al., 2022] and nonlinear models [Ghorbani et al., 2021, Mei and Montanari, 2022, Hu and Lu, 2022]. Mei and Montanari [2022] showed that the generalization error of the random features (RF) model proposed by Rahimi and Recht [2007] does demonstrate double descent. Perhaps more surprisingly, they also showed that vanishingly small regularization can yield optimal generalization in a nearly noiseless learning task. These findings highlight the recent trend of explaining the success of machine learning through the prism of beneficial interpolation and overparameterization [Belkin, 2021].

There exists, however, another approach to overparameterized learning problems, which is the Bayesian posterior predictive distribution. In Bayesian practice, one can define the training objective

∗

as the negative log-likelihood of a probabilistic model. The ridge regularized predictor for the RF model studied by Mei and Montanari [2022] is the maximum a posteriori (MAP) estimator of this probabilistic model. A 'truly Bayesian' approach, on the other hand, is to derive the posterior predictive distribution which averages over different predictors and summarizes one's uncertainty in prediction. The posterior predictive distribution is also referred to as the 'Bayesian model average' in the literature [Ovadia et al., 2019, Fortuin et al., 2021]. A fundamental question in Bayesian statistics is whether Bayesian credible sets, defined as high probability regions of the posterior, are also valid confidence sets in the frequentist sense [Kleijn and van der Vaart, 2012]. While this is true in finite-dimensional settings and known as Bernstein von-Mises theorem, the behavior of 'frequentist' and 'Bayesian' confidence sets can be drastically different in high-to-infinite dimensions, as revealed by Cox [1993], Freedman [1999], Johnstone [2010].

In this work, we examine the frequentist properties of the Bayesian random features model. The focus is on whether the variance of the posterior predictive distribution (PPV) demonstrates similar asymptotics as the risk of the MAP estimator studied by Mei and Montanari [2022]. The simplicity of the probabilistic model simplifies the study, as the posterior predictive distribution is available in closed form and is centered around the MAP estimator, for which we know the asymptotic risk. Due to this simplicity, agreement between the risk of the MAP estimator versus the PPV directly implies good coverage property of a natural confidence set formed by the posterior predictive. However, in light of previous work of Johnstone [2010], it is to be expected that such agreement can be hard to reach in high-dimensional learning. Previous literature relevant to our problem is reviewed in Section 2.3.

In this work, we show:

Absence of double descent phenomenon in the PPV with vanishing regularization (vague prior). Numerical instability does arise at the 'interpolation boundary' for any finite dimensions.

In low noise problems, the expected width of the Bayesian credible ball for highly overparameterized models can be much wider than the true risk of the MAP predictor.

The expected width of a credible ball asymptotically agrees with the risk when the sample size grows much faster than the model dimensions. Numerical results show the required growth rate of the sample size is unreasonable for many learning problems.

Numerical simulations demonstrate finer distributional properties that are currently beyond the reach of theoretical analysis. They suggest the results of Freedman [1999] and Johnstone [2010] in the Gaussian sequence model are applicable to the more complicated RF model.

The rest of the paper is organized as follows. Section 2 reviews the basic setup of the RF model and its probabilistic reformulation. Section 3 summarizes our main results based on asymptotic formulae and numerical simulations. Section 4 concludes with a discussion of the implications and future directions.

2 Background on Random Features Model

Let inputs x i ∈ R d , i = 1 , 2 , . . . , n be drawn i.i.d. from a uniform measure (denoted τ ) on a d -dimensional sphere with respect to the conventional Euclidean norm:

Let outputs y i be generated by the following model:

The model is decomposed into a linear component and a nonlinear component, f NL d . The random error glyph[epsilon1] i 's are assumed to be i.i.d. with mean zero, variance τ 2 , and finite fourth moment. We allow τ 2 = 0 , in which case the generating model is noiseless . In the analysis, these quantities will be assumed to obey further conditions, including an appropriate scaling so that all d -dependent quantities are O d (1) .

2.1 Training with Ridge Regularization

We focus on learning the optimal random features model that best fits the training data. This is a class of functions

which is dependent on N random features, ( θ j ) N j =1 , which are drawn i.i.d. from S d -1 , and a nonlinear activation function, σ . The training objective solves a regularized least squares problem for the linear coefficients a ≡ ( a j ) N j =1 :

where ψ 1 ,d = N/d , ψ 2 ,d = n/d , and λ > 0 . The optimal weights, ̂ a ≡ ̂ a ( λ ) , determine an optimal ridge predictor denoted by ̂ f ≡ f ( · ; ̂ a ( λ )) . The dependence of the trained predictor on the dataset ( X , y ) and features Θ are suppressed in notation.

There exist both practical and theoretical motivations for studying RF regression. On the practical side, RF regression has been suggested by Rahimi and Recht [2007] as a randomized approximation scheme for training kernel ridge regression (KRR) models for large datasets. On the theoretical side, (4) describes a 'lazy training' algorithm for a 2-layer neural network with activation function σ . Previous studies have focused on the approximation power of such a function class [Jacot et al., 2018, Chizat et al., 2019] and the optimization landscape involved in training problems of type (4) [Mei et al., 2018, 2019].

Given a new input feature x , the ridge regularized predictor for the unknown output y has the form

with the optimal ridge weights ̂ a and a resolvent matrix that defines the joint posterior of the weights:

Here, Z := σ ( X Θ / √ d ) / √ d for input design matrix X ∈ R n × d and output vector y ≡ ( y i ) n i =1 . Similarly we write σ ( x ) := σ ( x T Θ / √ d ) . The L 2 ≡ L 2 ( S d -1 ( √ d ); τ ) generalization error of predictor ̂ f is defined by

We emphasize that (8) is a random quantity, as it depends on ( y , X , Θ ) .

2.2 RF as Bayesian Model

The objective function (4) can be interpreted as a MAP estimation problem for an equivalent Bayesian model. Formally, we adopt a d -dependent weight prior distribution, denoted p ( a ) , and also a normal likelihood, denoted p ( y | X , Θ , a ) , centered around a function in the class (3) with variance φ -1 .

The normal likelihood model need not agree with the generating process (2), as is often the case for Bayesian deep learning. Technically and inferentially, it can be justified as a coherent update of belief given a squared error loss [Bissiri et al., 2016]. The joint likelihood of ( y , a ) is defined conditional on both the random features Θ and an 'inverse temperature' φ . The choice to condition on Θ instead of learning them can be unnatural in certain settings and is mainly for the convenience of analysis.

However, we do note that the RF model has been used by Crawford et al. [2018] as an approximation of fully Bayesian kernel learning.

The posterior predictive distribution, or Bayesian model average over the posterior of a , is defined as

where the posterior of a is a probability distribution placing higher mass near settings of a minimizing ridge objective (4):

This is a Gaussian measure centered around ̂ a (6) and covariance matrix ̂ Σ ( λ ) /d . Thus the posterior predictive at new input x is also a normal distribution centered around ̂ f , with variance

We refer to (11) as the PPV. This quantity dictates the width of the uncertainty interval centered around (4) evaluated at x . The expected PPV over x is defined by

The expectation over x , similarly as in (8), yields the radius of the posterior credible ball for a posterior Gaussian process f centered around the optimal predictor ̂ f . Furthermore, the Gaussian likelihood model simplifies the expected PPV into a decomposition of this radius and the inverse temperature of the posterior:

where p ( f | y , X , Θ ) is the law of Gaussian process f induced by the weight posterior p ( a | y , X , Θ ) . Thus, both summands in the display depend on the training features and are random. It is worth contrasting this definition with (12); in the next section, we explain in detail the motivation for comparing the two quantities.

The extra quantity introduced in the Bayesian formulation, φ , governs how much the resulting posterior should concentrate around the ridge predictor (6). Since we are interested in the regime where N,d → ∞ , it is reasonable to assume the scale of the likelihood, φ , must appropriately decrease with d , similar to how the prior on a is rescaled by 1 /d . Practitioners may adopt some prior distribution on this parameter and perform hierarchical inference. We adopt a simpler, empirical Bayes approach and choose it to maximize the marginal likelihood:

This choice coincides with the training set error attained by predictor (4), so it will be decreasing as N →∞ . If N > n and λ → 0 + , the training error vanishes as the model can perfectly interpolate the training set. We note the precise asymptotics of the training error has been already characterized by Mei and Montanari [2022] (Section 6).

A fundamental question here is whether R RF and S 2 RF have similar asymptotics, as both quantities summarize uncertainty about our prediction in different ways. R RF is the 'frequentist's true risk,' which requires assumptions about the unknown generative process. S 2 RF , on the other hand, is the 'Bayesian's risk' that can be actually computed without any model assumptions. Its asymptotics does depend on model assumptions, as both the prior and likelihood need not capture the generative process (2). In particular, it is desired that it agrees with R RF in some limiting sense. Throughout the rest of this work, we probe the question: Do R RF and S 2 RF converge to the same value as d, n, N →∞ ?

2.3 Previous Works

We have introduced our problem of comparing two different quantities, R RF and S 2 RF , and provided their respective interpretations. Such comparison, between the frequentist risk and the variance of

the posterior, was done by Freedman [1999] in a white noise model, where one observes an infinite square-summable sequence with Gaussian noise. The key finding is that the distributions of two corresponding statistics, re-normalized, have different variances. They are in fact radically different distributions in that they are essentially orthogonal. Knapik et al. [2011] clarified the situation by showing frequentist coverage of credible ball depends heavily on the smoothness of the prior covariance operator. Johnstone [2010] have extended the results to a sequence of finite but growing length, which is a setup more similar to ours. Our study for the RF model addresses a much simpler question, as the theoretical results in Section 3 only address whether two statistics converge to the same limits. Unlike in the work by previous authors, the identity of limits cannot be taken for granted. A key feature driving the different asymptotics of the two quantities is that the Bayesian's prior and the likelihood are 'mismatched' with respect to the data generating process; i.e., f d need not belong to the RF class (3). In Section 3.3, we demonstrate some distributional properties of R RF and S 2 RF empirically observed in numerical simulations.

Uncertainty quantification in Bayesian learning has recently garnered much attention in the theoretical deep learning literature. We highlight, among many works: Clarté et al. [2023a], characterizing exact asymptotic calibration of a Bayes optimal classifier and a classifier obtained from the generalized approximate message passing algorithm, and Clarté et al. [2023b], demonstrating double descentlike behavior of the Bayes calibration curve in RF models. In particular, quantities R RF and S 2 RF studied by our work can be related, respectively, to the 'Bayes-optimal classifier' and 'empirical risk classifier' of the latter work (Section 2). The recent work of Guionnet et al. [2023], on the other hand, explicitly addresses the model mismatch in Bayesian inference by deriving the exact, universal asymptotic formula for generalization of a misspecified Bayes estimator in a rank-1 signal detection problem.

3 Results

3.1 Asymptotic Characterization

We present the main results and illustrate them through numerical simulations. We operate under assumptions identical to those of Mei and Montanari [2022], stated below. The proofs are collected in the Supplementary Materials.

Assumption 1. Define ψ 1 ,d = N/d and ψ 2 ,d = n/d . We assume ψ 1 ,d → ψ 1 < + ∞ and ψ 2 ,d → ψ 2 < + ∞ as d →∞ .

Assumption 2. Let σ : R → R be weakly differentiable and | σ ( x ) | , | σ ′ ( x ) | < c 0 e c 1 | x | for some c 0 , c 1 < + ∞ . For G ∼ Normal(0 , 1) , the coefficients:

are assumed to be greater than 0 (ruling out a linear σ ) and finite.

Assumption 3. The generating model (2) satisfies

furthermore, f NL,d ( · ) is a centered Gaussian process on S d -1 ( √ d ) , whose covariance function has the form:

The kernel Σ d satisfies where x (1) is the first entry of x ∼ Unif( S d -1 ( d )) . The signal-to-noise ratio (SNR) of the model is defined by

√

In this 'linear proportional' asymptotic regime, we derive an asymptotic formula for the expected PPV akin to that of Mei and Montanari [2022] for the generalization error. Analysis shows Definition 1 of Mei and Montanari [2022], characterizing the asymptotics of R RF , can be straightforwardly applied to also characterize the asymptotics of S 2 RF .

Proposition 1. Denote by C + the upper half complex plane: { a + b i : i = √ -1 , b > 0 } . Let functions ν 1 , ν 2 : C + → C + be uniquely defined by the conditions: on C + , ν 1 ( ξ ) , ν 2 ( ξ ) are analytic and uniquely satisfy the equations

when | ν 1 ( ξ ) | ≤ ψ 1 / Im( ξ ) and | ν 2 ( ξ ) | ≤ ψ 2 / Im( ξ ) , with Im( ξ ) > C for sufficiently large constant C . Define

Under Assumptions 1-3 and for λ > 0 ,

Note that S 2 depends on ( ψ 1 , ψ 2 , λ ) only through function χ . The following facts follow from this fact and the asymptotic characterization of function χ when λ → 0 + .

Proposition 2. The following holds for the map ( ψ 1 , ψ 2 , λ ) ↦→S 2 defined in Proposition 1:

It is non-decreasing in λ .

Item 2, Proposition 2 deserves special attention, as it seems to suggest that there is no 'double descent' in the asymptotics of S 2 when using an improper prior. This fact does not possess an operational meaning at any finite N = n , due to the ordering of the limits N,n →∞ and λ → 0 + . In fact, one may expect that the distribution of the least singular value of the relevant matrix Z causes numerical instability for small λ when N = n . In Section 3.3, this hypothesis is empirically validated through simulations. Subtly, the theoretical prediction does accurately describe the numerical simulations for small choices of λ . On the other hand, the asymptotic prediction for the frequentist risk does diverge when λ → 0 + and ψ 1 = ψ 2 .

3.2 Comparison with Generalization Error

The asymptotics of (8), the 'frequentist' risk for our comparison, was characterized by Mei and Montanari [2022] through a theorem similar to Proposition 1. Our main interest lies in comparing the risk against the Bayesian variance in two limiting cases:

Highly overparameterized models where ψ 1 →∞ . The number of parameters grows faster than any constant multiple of the number of samples.

Large sample models where ψ 2 → ∞ . The number of samples grows faster than any constant multiple of the number of parameters.

The first regime has been studied as the relevant regime in modern deep learning. The second regime is closer to the classical regime of asymptotic statistics where only the number of samples n diverges. Below, we re-state the asymptotic formulae for R RF of Mei and Montanari [2022] in these special cases, which admits simplifications relative to when both ψ 1 , ψ 2 are finite.

Proposition 3. (Theorem 4-5, Mei and Montanari [2022]) Under the notation of Proposition 1, define a function

In the highly overparameterized regime, where ψ 1 →∞ , asymptotic risk is given as

with ω 2 = ω ( ζ, ψ 2 , λ/µ 2 ∗ ) .

In the large sample regime, where ψ 2 →∞ , asymptotic risk is given as

A simple question, mentioned in Section 2.2, was whether the two quantities agree. It turns out that in the second regime, at least, the two formulae converge to the same limit, which is the main content of the next Proposition. In the first regime, whether the limits agree is determined by the signal-to-noise ratio ρ . Mei and Montanari [2022] showed that if ρ is larger than a certain critical threshold ρ ∗ , which depends on ( ψ 2 , ζ ) , the optimal regularization is achieved by λ → 0 + , whereas if ρ is smaller than ρ ∗ , there exists an optimal regularization λ ∗ bounded away from 0. This phase transition also determines the agreement of the risk and the PPV in the limit.

Proposition 4. (Proposition 5.2,Mei and Montanari [2022]+ α ) Define quantities

under the notation of Proposition 3.

If ρ < ρ ∗ , then min ¯ λ ≥ 0 R wide ( ρ, ζ, ψ 2 , ¯ λ ) , is attained at some λ opt > 0 :

If, on the other hand, ρ > ρ ∗ , then arg min ¯ λ ≥ 0 R wide ( ρ, ζ, ψ 2 , ¯ λ ) = 0 . Furthermore, R wide ( ρ, ζ, ψ 2 , 0) < lim ψ 1 →∞ S 2 ( ψ 1 , ψ 2 , 0) -τ 2 .

Note the subtraction of the noise level τ 2 from S 2 . This is because S 2 RF is computed based on the training error in Proposition 1, which includes both the approximation error and the variance of data, whereas R RF is computed based only on the approximation error in the test set.

3.3 Numerical Simulations

In this section, we first compare evaluations of the asymptotic formulae for R RF and S 2 RF for varying ( ψ 1 , ψ 2 , λ ) . We highlight their difference for finite ( ψ 1 , ψ 2 ) at the optimal choice of λ for the MAP risk. We also present numerical simulations, which both validate the formulae (they concentrate fast) and empirically exhibit further interesting distributional properties (suggesting the need for second-order asymptotics). MATLAB codes used to produce simulation results are included in the Supplementary Materials.

Figure 1 shows the dramatic mismatch between R RF and S 2 RF in the low-noise regime. It turns out the conservativeness of the credible ball persists for the choice of λ that depends on ( ψ 1 , ψ 2 ) . The situation becomes more delicate in the high-noise regime because there exists a phase transition in the optimal choice of λ that depends on (19), which decreases with the noise variance τ 2 . Figures 2.2a and 2b compare the two curves, R and S , for the 'optimally tuned' λ at which the best possible frequentist risk is attained, for a fixed pair of ( ψ 1 , ψ 2 ) . In a low-noise task, with ρ = 5 , the ratio of the frequentist risk of the posterior mean predictor to the width of the posterior predictive credible ball is less than 1 for a wide range of ψ 1 . The situation is more nuanced and possibly more favorable for the Bayesian case in the high-noise task with ρ = 1 / 5 .

While Figure 1 validates good concentration properties of R RF and S 2 RF with respect to their asymptotic formulae, we may want to investigate the rate of fluctuations for these quantities. Figure

(a)

λ

=

1e-8

(b)

λ

=

1e-3

入=1e-08

入=0.001

2

s2 (Numerical)

2

sRF (Numerical)

1.8

SRF (Theoretical)

1.8

Ss (Theoretical)

1.6

RRF (Theoretical)

1.6

RRF (Theoretical)

1.4

1.4

Expected PPV

1.2

Expected PPV

1.2

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

N/n

N/n

(a)

ψ

2

= 3

(b)

ψ

1

= 3

1.05

1.02

Highp

Highp

Lowp

1

Lowp

0.95

0.98

0.96

0.9

0.94

0.85

2

2

S

0.92

0.8

R

R

0.9

0.75

0.88

0.7

0.86

0.65

0.84

0.6

0.82

0

5

10

15

0

5

10

15

N = n/5

N =n

N =5n

700

800

800

R

RF

R

RF

600

S2F

700

700

S25

RF

600

600

500

500

500

400

400

400

300

300

300

200

200

200

100

100

100

0

0

0

0.4

0.6

0.8

1

1.2

0

0.5

1

1.5

2

0.05

0.1

0.15

0.2

0.25

0.3

3 suggests an interesting phenomenon: both quantities appear Gaussian, but are nearly orthogonal precisely near the 'interpolation boundary' where R RF exhibits double descent (2nd subplot). Our

empirical results should be compared with the results of Freedman [1999] and Johnstone [2010]. The Gaussianity of R RF and S 2 RF , if true, strengthens the agreement between the expected width S 2 RF and expected risk R RF immediately transfers to the agreement between the frequentist coverage of (1 -α ) -% Bayes credible ball around the mean and the nominal coverage. Again, such claims are generally true in finite-dimensional settings, but not true for high-dimensional settings. For instance, Freedman [1999] showed that asymptotically, the posterior variance fluctuates like a Gaussian random variable with a variance strictly smaller than that of the frequentist risk. Extracting such information is possible only if we study second-order information of R RF and S 2 RF ; in particular, suggests a Gaussian central limit for these quantities. A rigorous proof of a central limit theorem for R RF and S 2 RF goes beyond the scope of this work. Nevertheless, we conjecture the following for the fluctuations of R RF and S 2 RF :

Both d ( R RF -E R RF ) and d ( S 2 RF -E S 2 RF ) weakly converge to Gaussian distribution with appropriate variances. The faster convergence rate of d rather than √ d is known to be common in central limit theorems for linear spectral statistics [Lytova and Pastur, 2009].

When ψ 1 ≤ ψ 2 , asymptotic variance of S 2 RF is smaller than R RF . When ψ 1 = ψ 2 , the two distributions are nearly orthogonal. For large enough ψ 1 , the asymptotic variances are of the same order.

These conjectures are of independent interest and pose interesting theoretical challenges. We must note that second-order asymptotics has received less attention in the deep learning community. Only recently did Li et al. [2021] study the asymptotic normality of prediction risk achieved by a minnorm least squares interpolator. Their result relies on a central limit theorem for linear statistics of eigenvalues of large sample covariance matrices, established by Bai and Silverstein [2004]. Many central limit theorems in random matrix theory seem insufficient to handle kernel learning or learning with random features.

4 Discussion

In sum, our calculations and empirical findings suggest that the RF model still has some interesting discoveries to offer, particularly for those interested in the implications of Bayesian deep learning. The asymptotics of the posterior predictive summaries can be very different from that of the generalization error. Numerical simulations suggest that some version of the central limit holds so that depending on the particular scaling of dimensions, the frequentist coverage of the posterior predictive distribution can be arbitrarily low or high. We now conclude with a discussion of several conjectures and the next directions suggested by these findings.

Heuristic Implications. While our technical calculations are only applicable to the RF model, we believe the general perspective of comparing frequentist versus Bayesian estimation approaches to the deep learning model can offer useful heuristics explaining empirically observed phenomena in training Bayesian deep learning models.

One suggestion is that the different regimes of asymptotics may explain away the 'cold posterior effect', first empirically observed by Wenzel et al. [2020]. The authors forcefully suggested that raising the posterior of a deep learning model to an even larger power (i.e., more concentrated) can improve generalization. That the posteriors can be too wide in highly overparameterized, nearnoiseless regimes may explain this issue. Even though in this simple RF model setup, the posterior predictive mean is left unchanged from that of the posterior, it is plausible that for more complicated models this need not be so. When that posterior is much wider than the expected risk of the mean, we will mix over too many bad predicting weights, so it will be actually worse to average over the posterior than not.

Technical Improvements. The technically most interesting but quite challenging direction, suggested by our numerical simulations in Section 3.3, is to study second-order fluctuations of random matrix quantities often encountered in deep learning. While at this point a vast amount of literature exists that systematically overcomes the non-standard definitions of quantities like generalization error from the viewpoint of random matrix theory, we are aware of no study as of yet that overcomes the same issues to show a central limit-type result. On the other hand, central limit theorems for linear spectral statistics remain an active area of research in random matrix theory and are generally acknowledged to require more advanced, subtler arguments that are fundamentally different from first-order results

(i.e., convergence of empirical spectral distribution). For a start, a separate work on the central limit theorem for linear spectral statistics of kernel matrices is in preparation by the authors.

Another interesting direction is to extend the comparison between frequentist and Bayesian modeling approaches to more complex models of training dynamics. For instance, Adlam and Pennington [2020] have derived precise asymptotics of generalization error in the neural tangent kernel (NTK) regime, in which the features Θ are 'learned' through a linearized approximation of the gradient flow in training. Mei et al. [2018, 2019] have suggested a different asymptotic model, in which Θ evolve through nonlinear dynamics. An interesting challenge, both technically and conceptually, is to adapt their analysis to the Bayesian setting, where the gradient flow on the space of posterior measures of Θ is approximated and specific projections are tracked.

Making the 'Right' Comparison. A possible objection is that the comparison between two quantities is not appropriate: if interested in the coverage of function f d with our credible set, then one must compare the expected width of the interval of the function space posterior, ̂ f , instead of that of the Bayesian model average. While the objection is sensible, the fact is that the average variance of the function posterior will be an even worse uncertainty estimate for underparameterized models, whereas it will make little difference for overparameterized models. The reason is that the variance of the posterior of ̂ f only differs from that of the posterior predictive, in (11), by the training set error term (14). The training error dominates the PPV precisely for underparameterized models ( ψ 1 < ψ 2 ), so the curves of Figure 1 will now be nondecreasing and asymptotically convergent to a similar constant in expectation. This fact illustrates that a simple interpretation of the width of the posterior is complicated in even simplistic high-dimensional models.

Another possible objection is that the same value of λ need not be optimal for the posterior predictive. Since the asymptotic curve for S 2 is non-increasing in λ , this implies one must choose even smaller λ than λ opt to obtain less conservative credible intervals, regardless of the SNR ρ . This seems an odd consequence of using overparameterized models, where a good frequentist coverage is obtained by choosing a narrower prior than that with good 'contraction property' (i.e., minimal L 2 risk of posterior mean).

The final objection is that the RF model is simply not a very realistic model of practical Bayesian deep learning. We point out that quantifying the behavior of RF models as approximations of kernel learning methods remains a valid concern for Bayesian practitioners.

Acknowledgments

All authors would like to thank Matthew M. Engelhard, Boyao Li, David Page, and Alexander J Thomson for the helpful discussion around the theoretical work.

Samuel I. Berchuck would like to acknowledge support from the National Eye Institute of the National Institutes of Health under Award Number K99EY033027. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Sayan Mukherjee would like to acknowledge partial funding from HFSP RGP005, NSF DMS 17-13012, NSF BCS 1552848, NSF DBI 1661386, NSF IIS 15-46331, NSF DMS 16-13261, as well as high-performance computing partially supported by grant 2016- IDG-1013 from the North Carolina Biotechnology Center as well as the Alexander von Humboldt Foundation, the BMBF and the Saxony State Ministry for Science.

References

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics , 49(2):1029-1054, 2021.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75 (4):667-766, 2022.

Hong Hu and Yue M Lu. Sharp asymptotics of kernel ridge regression beyond the linear regime. arXiv preprint arXiv:2205.06798 , 2022.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248, 2021.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems , 32, 2019.

Vincent Fortuin, Adrià Garriga-Alonso, Sebastian W Ober, Florian Wenzel, Gunnar Rätsch, Richard E Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. arXiv preprint arXiv:2102.06571 , 2021.

BJK Kleijn and AW van der Vaart. The bernstein-von-mises theorem under misspecification. Electronic Journal of Statistics , 6:354-381, 2012.

Dennis D Cox. An analysis of bayesian inference for nonparametric regression. The Annals of Statistics , pages 903-923, 1993.

David Freedman. Wald lecture: On the bernstein-von mises theorem with infinite-dimensional parameters. The Annals of Statistics , 27(4):1119-1141, 1999.

Iain M Johnstone. High dimensional bernstein-von mises: simple examples. Institute of Mathematical Statistics Collections , 6:87, 2010.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems , 32, 2019.

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of twolayer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory , pages 2388-2464. PMLR, 2019.

Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 78 (5):1103-1130, 2016.

Lorin Crawford, Kris C. Wood, Xiang Zhou, and Sayan Mukherjee. Bayesian approximate kernel regression with variable selection. Journal of the American Statistical Association , 113(524): 1710-1721, 2018.

BT Knapik, AW van der Vaart, and JH van Zanten. Bayesian inverse problems with gaussian priors. The Annals of Statistics , 39(5):2626-2657, 2011.

Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Theoretical characterization of uncertainty in high-dimensional linear classification. Machine Learning: Science and Technology , 4(2):025029, 2023a.

Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. On double-descent in uncertainty quantification in overparametrized models. In International Conference on Artificial Intelligence and Statistics , pages 7089-7125. PMLR, 2023b.

Alice Guionnet, Justin Ko, Florent Krzakala, and Lenka Zdeborová. Estimating rank-one matrices with mismatched prior and noise: universality and large deviations, 2023.

A Lytova and L Pastur. Central limit theorem for linear eigenvalue statistics of random matrices with independent entries. Annals of Probability , 37(5):1778-1840, 2009.

Zeng Li, Chuanlong Xie, and Qinwen Wang. Asymptotic normality and confidence intervals for prediction risk of the min-norm least squares estimator. In International Conference on Machine Learning , pages 6533-6542. PMLR, 2021.

ZDBai and Jack W Silverstein. Clt for linear spectral statistics of large-dimensional sample covariance matrices. The Annals of Probability , 32(1A):553-605, 2004.

Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub ´ Swikatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405 , 2020.

Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning , pages 74-84. PMLR, 2020.

A Proofs

Proof of Proposition 1

The necessary calculations follow immediately from the proof of Mei and Montanari [2022]. Since we do not wish to replicate the long proof with only minor readjustments, we only sketch the main argument. The expected PPV consists of two random variables, denoted by

It suffices to show the two quantities individually converge in probability to some limits. The first of the two is a training set error, for which an asymptotic formula was derived in Section 6, Mei and Montanari [2022]:

using the definition of quantities in Proposition 1, for χ ≡ χ ( i √ ψ 1 ψ 2 λ/µ ∗ ) = ν 1 ν 2 ( i = √ -1 ). At this point, rearranging the fixed point equations (20) and (21) with ξ ≡ i √ ψ 1 ψ 2 λ/µ ∗ , we obtain:

A similar formula for E r can also be derived, using almost exactly the same steps required for simplifying asymptotic formulae of R RF and ̂ φ -1 . Assumption 3 on the nonlinear component of the generating model, f NL d , implies that it admits an explicit decomposition into spherical harmonics that form a basis on S d -1 and coefficients with Gaussian distributions (appropriately scaled). With many simplifications, also sketched out in Appendix E, Mei and Montanari [2022], we can write

Next, form an ( N + n ) × ( N + n ) -dimensional block matrix A , indexed by a 5-d real parameter q = ( s 1 , s 2 , t 1 , t 2 , p ) :

Denoting by ξ ∈ C + the log-determinant of the resolvent matrix A ( q ) -ξ I N + n as G d ( ξ ; q ) , straightforward calculus and plugging in the figures reveals

At this point, we pass on to the limit d → ∞ and plug in appropriate asymptotic limits of the partial derivatives. Since the ξ -derivative of G d is the trace of A ( q ) -ξ I N + n , this is possible by characterizing the asymptotics of the Stieltjes transform of the empirical spectral distribution of A ( q ) , which is done using Proposition 8.3, Mei and Montanari [2022]. Integrating the limit over path stretching in the imaginary axis ( i K with K →∞ ), one then rigorously passes onto the limit by Proposition 8.4, Mei and Montanari [2022]. Final simplifications using Lemma 8.1, Mei and Montanari [2022] yield the formula

At this point, we may conclude, using (23),(30) and (26):

Proof of Proposition 2

The first point is a consequence of the fact that S 2 RF is decreasing in λ and passing onto the limits. The second point is immediately seen by taking the limit of χ as λ → 0 , a formula for which is provided by Theorem 3, Mei and Montanari [2022].

Proof of Proposition 4

We want to check whether equality between asymptotic risk and the expected PPV holds. Using Proposition 3: In the highly overparameterized regime, we demand equality

By Proposition 13.1, Mei and Montanari [2022], this equation is met precisely by choosing ¯ λ ≡ λ opt , only possible if ρ < ρ ∗ per the definition of Proposition 4. Otherwise (i.e., if ρ > ρ ∗ ), the equality is unsatisfiable, and furthermore, one can use the property of the map ω ≡ ω ( ¯ λ, ψ, ζ ) in Lemma 13.1, Mei and Montanari [2022], to show that ω 3 2 +( ρψ 2 -ρ 1 ) ω 2 2 -ψ 2 ρω 2 > 0 .

In the large sample regime, we similarly want to check the equality

In this case, the demanded equality does not have ρ in it. Utilizing symmetry and the same argument as in Proposition 13.1, Mei and Montanari [2022], one can show that the absence of ρ implies the equality is always satisfiable by choosing ¯ λ = 0 , which also corresponds to the minimizer of the asymptotic risk: arg min ¯ λ ≥ 0 R lsamp = 0 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study random feature (RF) regression in the high-dimensional setting. \nThey focus on comparing the variance of the Bayesian posterior distributions (a measure of uncertaintiy in the Bayesian setting) to the variance of the maximum-a-posteriori estimator (a measure of uncertainty in the frequentist sense, and the generalisation error, already studied in previous work), motivated by previous work showing a discrepancy in the two quantities in the high-dimensional setting. They ask whether the two quantities have the same high-dimensional asymptotics.\n\nTha authors compute the high-dimensional asymptotic of the variance of the Bayesian posterior distribution. \nThey find that it equals the variance of the maximum-a-posteriori estimator only in the region where the optimal L2 regularisation is not vanishing in the strongly overparametrised case, and that they always match in the large sample complexity regime, close to the classical low-dimensional limit.\n\nThe authors study numerically the fluctuations of the two observables studied, and conjecture that they are Gaussian distributed, and such distribution have non-overlapping support around the interpolation threshold.\n\nComment: \nThe main proof seem a minor variation on reference \"Mei and Montanari, 2022\". \nA reader familiar with that work (which I am not) should be able to easily tell whether the proof of the results of the authors is correct.\n\nSTRENGTHS:\n- The authors compare Bayesian and frequentist notions of variance-like observables, computing a novel characterisation for one such variable. They also provide interesting numerical observations of the flucuations of such variables, proposing a number of new conjectures.\n\nWEAKNESSES:\n- The paper is not badly written overall, but many details (see list below) could be improved.\n- The authors provide no code to reproduce their figures.\n\nQUESTIONS:\n- The authors observe empirically and conjecture near-orthogonality of $R$ and $S^2$ at the interpolation threshold $N=n$. Is there any intuition why this should be the case? And what are the possible consequences of this fact?\n\n## Suggestions for manuscript improvement\n\n- this is more of a personal preference, but I like to number all equations in my manuscripts. This makes easier to readers (and reviewers) to refer to a given equation when providing feedback.\n\n- Eq between line 103 and 104: $p(y | x, \\Theta, a)$ is not clearly defined. Do the authors mean that this is the same distribution as the second line of the equation between line 95-96? This should be specified. \n\n- Eq between line 105 and 106: missing subscript $i$ for the datapoint $x$. \n\n- line 106: possibly wrong reference to Eq 5. Maybe the authors are referring to Eq 4, or want to say that the Gaussian measure is centered around $\\hat{a}$.\n\n- equation between line 113 and 114: the authors could derive this equation (even if trivial) either in the main text or in the appendix. For example, I do not see at first sight how to derive it by averaging over $x$ in Eq.8, which by Eq. 9 is the definition of $S_{RF}^2$. Line 115-116 seems to imply that indeed this equation is not derived from Eq. 8. Where does it come from? Over which Gaussian process is the average performed?\n\n- line 123: where is the marginal likelihood defined? Where does Eq 10 comes from?\n\n- line 183-190: this paragraph is not clear to me. Is \"The second point\" the full Proposition 2, or just point 2 of Proposition 2? Why does this suggest that there is not double descent? I only see monotonicity in the L2 regularisation parameter and boundedness.\n\n- line 209-211: it is not apparent at first sight that in the second regime equation between line 207-208 and the one between line 176-177 coincide. Is there an easy way to see it? If not, this should be done explicitly, or better justified, somewhere.\n\n- Figure 1: the caption is incomplete (and a legend is missing). One can guess what the circles and the black curve are, but it would be clearer if the authors wrote it explicitly. Also, the authors should specify how the error bars are computed.\n\n- Figure 2: apart from the dashed line at 1, I see two curves here, orange and blue. It seems that each curve is for a different value of $\\rho$, but it is not clear which one is which. I guess from lines 237-240 orange is smaller rho, but this should be clearly specified in a legend.\n\n- line 241: Figure 1 shows concentration only for $S^2$, and not for $R$ for which simulations are not provided, contrary to what this phrase states.\n\n- Figure 3: it would be clearer if the authors listed here all the sources of disorder causing the fluctuations of the quantities plotted \n\n- line 285-287: doesn't the author find in Proposition 4, point 1, that there exists an overparametrised regime where $S^2 > R$, i.e. the posterior is unecessarily wide? Maybe I am misunderstanding this phrase.\n\n- line 216: possible typo $+\\alpha$\n\nLIMITATIONS:\nThe authors discuss nicely limitations/shortcomings of their choice of observables to compute in Section 4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors compare in this paper the asymptotic posterior variance of the predictive risk in the random feature model  associated with a Gaussian prior on the weights with the  asymptotic \\textit{frequentist risk} which had been derived by Mei and Montanari.  The asymptotic here is in terms of the dimension d of the covariates, in a context where the number of features $N$ and the number of observations $n$ are comparable with $d$. This regime is often studied to understand the phenomenon of double descent. \nThey derive under the same assumptions as in Mei and Montanari the asymptotic behaviour of the posterior predictive risk and show that in the \"wide\" regime, i.e. when N/d converges to infinity with low signal noise ratio or when n/d goes to infinity both risks (under the optimal choice of the hyperparameter $\\lambda$ for the frequentist risk) have the same limit while when N/d goes to infinity and the \"SNR \" $\\rho$ is larger than some threshold then posterior variance is asymptotically larger than the frequentist risk which is 0. The authors then perform a simulation study to understand more precisely the role of $\\lambda$ and the different asymptotic regimes.\n\nSTRENGTHS:\nThe comparison of the frequentist and the Bayesian approaches in  the double descent  types of regimes is a very natural question and treating the random feature model seems like the obvious first choice to attack the problem. The results derived by the authors seem correct and the simulations shed light on the results. The proofs are a consequence of the proofs of Mei and Montanari, but remain technical (at least the proof of Proposition 1). \n\n The posterior variance , as studied by the authors is strongly related to the radius of credible regions therefore understanding if the posterior variance is similar to the frequentist risk is a first step to understand if credible balls are confidence balls, as hinted by the authors. As noted by the authors this is not a full statement on coverage of credible balls and higher order asymptotics is required  to understand the frequentist coverage  of credible balls.\n\nWEAKNESSES:\nWhat I am not so clear about is what is the conclusion/ consequence of the results obtained in this paper. In particular, the \\textit{frequentist} risk of Montanari is not really frequentist and in my opinion, it  has already a Bayesian flavour. The risk $\\|f_d - \\hat f\\|_2^2$ (which is in fact a loss function) is viewed in Montanari as stochastic both in terms of the data $X,Y$ appearing in $\\hat f$ but also in $f_d$ (not to speak of the randomness $\\Theta$) and its asymptotic behaviour is under this regime. Hence, it would be interesting to have a proper discussion of the different notions of risk which are considered in the paper. I also think that the authors do not comment enough on why in some regimes the two risks are equivalent and not in others. Some description in terms on $\\tau$ (noise variance ) being large or small in the wide regime is provided but no real interpretation. \n\n\n\n That being said I still find the results interesting, but I am uncertain after reading as of to what ot make of them.\n\nQUESTIONS:\nThe paper has a number of typos and inconsistencies which makes it a bit cryptic  and which I will list below. In particular the notion of proba under which expectations re calculated are not always clearly stated . In the context of this paper it is particularly important to be precise in this because these expectations are somewhat non standard. \n\n1. p4 line 110: The comment on the credible radius is cryptic or incorrect. By definition a credible radius depends on the coverage (Bayesian) of the credible ball. So it cannot be equal to the posterior variance for all coverage. This sentence makes no sense to me. \n\n2. In the definition of the Bayesian model in line 95, the variance of the noise is denoted $\\phi$ while in the previous section it was $\\tau$. Is this a typo or is there some confusion there or is it that the model may also be mispecified in the variance component? If it is purely a notation inconsistency an $\\phi $ also denotes the variance  of the noise, then the paragraph in lines 118-123 is puzzling. It reads as if it was a quantity chosen by the analyst , while it is driven by the data. It is not completely clear to me why $\\phi$ should go to 0 while in the subsequent computations $\\tau$ is held fixed.\n\n3. Proposition 1. What is $\\mathbb C_+$? Also what does the sentence \"and for expectation over the randomness of $(y, X, \\Theta)$  mean? There is no expectation in the statement below. \n\n4. Proposition 3: Recall what is the Expectation (with respect to which variables)\n\n5. Do the activations functions considered in the simulated examples satisfy the assumptions ? Provide some examples which satisfy the assumptions. \n\n6. What are the different curves in figure 2?\n\n7. line 233 : What do you mean by \" conservativeness of the credible ball \" ?\n\n8. What do you mean by \"he posterior variance fluctuates like a Gaussian random variable with a variance strictly smaller than that of the frequentist risk\"?  (l 250) The posterior variance is deterministic in a Bayesian Gaussian - Gaussian model (as in Freedman). \n\n9. What does the sentence l l286 mean : \", the posterior\n286 predictive mean is left unchanged from that of the posterior, \"? Similarly I don't understand the last sentence of that paragraph. \n\n10. l310 : What do you mean by \"interval of the function space posterior, $\\hat f$\"? abd by the variance of the posterior of $\\hat f$ a few lines below?\n\nLIMITATIONS:\nThe authors have interesting remarks on the open questions raised by the paper and what are the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the random feature model. Two objects are studied: (1) The posterior predictive distribution (in particular, the variance), and (2) The MAP estimator. This paper gives asymptotic formulas for these two quantities under the proportional regime. Comparison between these two quantities are made, in terms of both theory and experiments. Their results suggest that the posterior predictive summaries can be very different from that of the generalization error.\n\nSTRENGTHS:\nThis paper is among the few works that address the following problem: Are the Bayesian credible sets also valide confidence sets in the frequentist sense. This is true in the finite dimensional setting with n \\to \\infty, while unclear in the high-dimensional setting. They give precise expressions for the two targeted quantities: generalization error and posterior predictive variance. They give rigorous results to show that these two quantities agree and disagree in different regimes. The presentation is nice and clear. Extensive simulation is conducted to justify several unproven claims.\n\nWEAKNESSES:\n1. Their setting is a little bit restrictive: Only one specific prior structure is considered. Having said that, I understand that choosing this prior is for getting a precise asymptotic expression for the quantities of interest. \n2. Their theoretical results seem like direct consequences of [Mei and Montanari 2022], perhaps the authors should highlight their technical contributions comparing to previous works. This is my major concern, and I will raise my score if this question is well addressed.\n\nQUESTIONS:\n1. Line 312: \"worse an uncertainty estimate -> worse than uncertainty estimate\"\n2. It is not clear why they consider averaged posterior predictive variance versus generalization error. The authors did present an explanation in the end. However, this explanation, at least to me, is like simply stating \"these two quantities are different and identical in several regimes\". I think a more convincing argument should be provided.\n\nLIMITATIONS:\nLimitations have been adressed, and societal impact not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main focus of the paper is on the comparison between the posterior predictive distribution and the frequentist risk associated with the maximum a posteriori estimator for random features ridge regression model. The target function is assumed to be a sum of a linear model, a non-linear function given by a Gaussian process with specific kernel (see assumptions), and an iid noise model with mean zero, constant variance, and bounded fourth moment (see Eq. 1). The technical part starts with a review of frequentist risk and empirical estimator, given in Eq. (7). This is followed by a Bayesian model that assumes normal prior on the weights and likelihood model that conditions on the random features and weights of the linear model (see Section 2.2). The posterior distribution is the Gaussian centered at the maximum a posteriori estimator with covariance matrix given in Eq. (6). This is standard for the weight space derivation of Gaussian processes. The posterior predictive at an instance x is given in Eq. (8) and is just variance at that point. The expected posterior predictive (EPP) is given in Eq. (9) and is obtained by taking the expectation relative to the instance space. The paper then provides an asymptotic characterization of EPP relative to prior work by Mei & Montanari (2022) on frequentist risk. Numerical simulations illustrate the difference between the two concepts and show the lack of double-descent phenomena in the Bayesian setup.\n\nSTRENGTHS:\nIt is a nicely presented work with an interesting insight into random features and the differences between Bayesian and frequentist notions of uncertainty. The theoretical result is cumbersome to parse but still provides an interesting illustration and side-by-side comparison with Mei & Montanari (2022). Numerical simulations also support the theoretical conclusions. The discussion is also valuable, pointing at some of the shortcomings and directions for future research.\n\nWEAKNESSES:\nThe assumptions might be constraining the problem setting strongly but still the results provide an interesting insight on what happens asymptotically.\n\nQUESTIONS:\nAre the instances $x_i$ from the interior of the d-dimensional sphere or from its surface (\"around\" is not precise enough)? If the latter, why is this assumption necessary and to what level it affects the results? Normally, one assumes that instances are contained within a ball of constant radius and this appears to be a bit stronger assumption.\n\nIn Eq. (1), I see $f_d^{NL}$ defined but not $f_d$. The latter appears in Eq. (7) and my understanding is that it refers to the function described in Eq. (1). Could you please do a pass over the notation?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}