Title: Sub-optimality of the Naive Mean Field approximation for proportional high-dimensional Linear Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Variational Bayes; Naive Mean Field; Gaussian comparison inequalities; High-dimensional statistics; Proportional asymptotic.
Abstract: The Naïve Mean Field (NMF) approximation is widely employed in modern Machine Learning due to the huge computational gains it bestows on the statistician. Despite its popularity in practice, theoretical guarantees for high-dimensional problems are only available under strong structural assumptions (e.g. sparsity). Moreover, existing theory often does not explain empirical observations noted in the existing literature. 
 
In this paper, we take a step towards addressing these problems by deriving sharp asymptotic characterizations for the NMF approximation in high-dimensional linear regression. Our results apply to a wide class of natural priors and allow for model mismatch (i.e. the underlying statistical model can be different from the fitted model).  We work under an iid Gaussian design and the proportional asymptotic regime, where the number of features and number of observations grow at a proportional rate. As a consequence of our asymptotic characterization, we establish two concrete corollaries: (a) we establish the inaccuracy of the NMF approximation for the log-normalizing constant in this regime, and (b) we provide theoretical results backing the empirical observation that the NMF approximation can be overconfident in terms of uncertainty quantification.

Our results utilize recent advances in the theory of Gaussian comparison inequalities. To the best of our knowledge, this is the first application of these ideas to the analysis of Bayesian variational inference problems. Our theoretical results are corroborated by numerical experiments. Lastly, we believe our results can be generalized to non-Gaussian designs and provide empirical evidence to support it.

Sub-optimality of the Naive Mean Field approximation for proportional high-dimensional Linear Regression

Abstract

The Naïve Mean Field (NMF) approximation is widely employed in modern Machine Learning due to the huge computational gains it bestows on the statistician. Despite its popularity in practice, theoretical guarantees for high-dimensional problems are only available under strong structural assumptions (e.g., sparsity). Moreover, existing theory often does not explain empirical observations noted in the existing literature.

In this paper, we take a step towards addressing these problems by deriving sharp asymptotic characterizations for the NMF approximation in high-dimensional linear regression. Our results apply to a wide class of natural priors and allow for model mismatch (i.e., the underlying statistical model can be different from the fitted model). We work under an iid Gaussian design and the proportional asymptotic regime, where the number of features and the number of observations grow at a proportional rate. As a consequence of our asymptotic characterization, we establish two concrete corollaries: (a) we establish the inaccuracy of the NMF approximation for the log-normalizing constant in this regime, and (b) we provide theoretical results backing the empirical observation that the NMF approximation can be overconfident in terms of uncertainty quantification.

Our results utilize recent advances in the theory of Gaussian comparison inequalities. To the best of our knowledge, this is the first application of these ideas to the analysis of Bayesian variational inference problems. Our theoretical results are corroborated by numerical experiments. Lastly, we believe our results can be generalized to non-Gaussian designs and provide empirical evidence to support it.

1 Introduction

The Naive Mean Field (NMF) approximation is widely employed in modern Machine Learning as an approximation to the actual intractable posterior distribution. The NMF approximation is attractive as (a) it bestows huge computational gains, and (b) it is naturally interpretable and can provide access to easily interpretable summaries of the posterior distribution e.g., credible intervals. However, these two advantages may be overshadowed by the following limitations: (a) it is a priori unclear whether this strategy of using a product distribution as a proxy for the true posterior will result in a 'good' approximation, and (b) it has been empirically observed that NMF often tends to be significantly over-confident, especially when the feature dimension p is not negligible compared to the sample size n . In the traditional asymptotic regime ( p fixed and n →∞ ), significant progress was made in understanding these two problems for different statistical models, see for instance [8] and references therein. On the other hand, in the complementary high-dimensional regime where both n and p are growing, Ghorbani et al. [7] recently established an instability result for the topic model under the

Approximation: log normalizing constant (alpha=1, q=0.1)

Credible intervals generated by NMF (q = 0.2)

2.00

MSE (q = 0.1; alpha = 1)

1.3

NMF approximation

Limiting log Z

0.94

1.75

NMF:CGMT Theory Prediction

1.2

0.92

1.50

AMP

NMF: Gradient Decent Simulation

1.1

Rate

0.90

1.25

buzieou

1.0

0.88

MSE

1.00

0.9

0.86

alpha = 1

0.75

0.8

0.84

0.82

alpha = 4

alpha = 2

alpha = 7

alpha = 11

0.50

alpha = 16

0.25

9°0

0.80

0.95

0.00

0

1

2

3

4

5

0.5

1.0

1.5

2.0

2.5

3.0

0

1

2

delta

3

4

5

delta~2

delta~2

proportional asymptotics, i.e., n = Θ( p ) . In fact, in this regime, based on non-rigorous physics arguments, it is conjectured and partially established that instead of the NMF free energy one should optimize the TAP free energy. For linear regression in particular, please see [13, 18]. On the other hand, positive results of NMF for high-dimensional linear regression were recently established in [16] when p = o ( n ) .

In this paper, we investigate the performance of NMF approximation for linear regression under the proportional asymptotics regime. As a consequence of our asymptotic characterization, we establish two concrete corollaries: (a) we establish the inaccuracy of the NMF approximation for the log-normalizing constant in this regime, and (b) provide theoretical results backing the empirical observation that NMF can be overconfident in constructing Bayesian credible regions.

Before proceeding further, we formalize the setup under investigation. Given data { ( y i , x i ) : 1 ≤ i ≤ n } , y i ∈ R , x i ∈ R p , the scientist fits a linear model

where ϵ i iid ∼ N (0 , σ 2 ) and β ⋆ ∈ S p is a p -dimensional latent signal. We consider either S = R or S = [ -1 , 1] . In fact, unless explicitly specified otherwise, S = R . Most of our results generalize to bounded support naturally. To recover the latent signal, the scientist adapts a Bayesian approach. She puts an iid prior on β i 's, namely, d π 0 ( β ) = ∏ p i =1 d π ( β i ) and then constructs the posterior distribution of β ,

with normalization constant

Our results are established assuming that the design matrix X is randomly sampled from an iid Gaussian ensemble, i.e., X ij iid ∼ N (0 , 1 /n ) , while we provide empirical evidence for more general classes of X that has iid entries with mean 0 and variance 1 /n . Moreover, we assume n/p → α ∈ (0 , ∞ ) as n, p →∞ .

Definition 1 (Exponential tilting) . For any γ := ( γ 1 , γ 2 ) ∈ ¯ R × R + and probability distribution π on S , we define π γ as

Note that c ( · ) depends on the distribution π and is usually referred to as the cumulant generating function in the statistics literature.

Using this definition of exponential tilts, the ( X T X ) ii β 2 i terms in (2) can be absorbed into the base measure

where π i := π (0 ,d i ) and d i := ( X T X ) ii σ 2 . By the classical Gibbs variational principle (see for instance [27]), the log-normalizing constant can be expressed as a variational form,

where the inf is taken over all probability distribution on S p . While the infimum is always attained if and only if Q = µ , the Naive Mean Field (NMF) approximation restricts the variational domain to product distributions only and renders a natural upper bound,

It can be shown that the product distribution ˆ Q that achieves this infimum is exactly the one closest to µ , in terms of KL-divergence. Before moving forward, we need some additional definitions and basic properties of exponential tilts. The first lemma establishes that instead of using ( γ 1 , γ 2 ) we can also use ( u, γ 2 ) = ( E U ∼ π γ U, γ 2 ) to parameterize the tilted distribution.

Lemma 1 (Basic properties of the cumulant generating function c ( · ) ) . Let c ( · ) be as in Definition 1. Let supp( π ) denote the support of π . If m ( π ) := inf supp( π ) < 0 and M ( π ) := sup supp( π ) > 0 , then the following conclusions hold.

(b) For any u ∈ ( m ( π ) , M ( π )) , there always exists a unique h ( u, γ 2 ) ∈ R such that ˙ c ( h ( u, γ 2 ) , γ 2 ) = u .

Definition 2 (Naive mean field variational objective) . With d i := ( X T X ) ii /σ 2 , we define M p ( u ) : [ m ( π ) , M ( π )] p → ¯ R as

where G is defined as a possibly extended real valued function on [ m ( π ) , M ( π )] × R ,

in which h ( · , · ) was defined in Lemma 1 and π ∞ and π -∞ are degenerate distributions which assigns all measure to M ( π ) and m ( π ) respectively.

Note that under product distributions, the E Q ( · ) term in (3) is parameterized by the mean vector u := E Q β and exponential tilts of π i 's minimize the KL-divergence term. Therefore, the scaled log-normalizing function, which is also referred to as the average free energy in statistical physics parlance and (log) evidence in Bayesian statistics, is bounded by the following variational form,

The right-hand side is equal to (3) and is also referred to as the evidence lower bound (ELBO) or NMF free energy, which can be used as a model selection criterion, see for instance [14]. Asymptotically, the second term is nothing but a constant since it concentrates around c (0 , 1 /σ 2 ) as n, p →∞ .

The main theoretical question of interest here is whether this bound in (4) is asymptotically tight or not, which serves as the fundamental first step towards answering the question of whether NMF distribution is a good approximation of the target posterior. Please see [4, 27] for comprehensive surveys on variational inference, including but not limited to NMF approximation.

To derive sharp asymptotics for the NMF approximation, the key observation is that the optimization problem is convex under certain priors. We then employ the Convex Gaussian Min-max Theorem (CGMT). CGMT is a generalization of the classical Gordon's Gaussian comparison inequality [9], which allows one to reduce a minimax problem parameterized by a Gaussian process to another (tractable) minimax Gaussian optimization problem. This idea was pioneered by Stojnic [21] and then applied to many different statistical problems, including regularized regression, M-estimation, and so on, see for instance [15, 24]. Unfortunately, concentration results derived from CGMT require both Gaussianity and convexity. This is exactly why we need the Gaussian design assumption in our analysis. In the meantime, though we do not pursue this front theoretically, we provide empirical evidence for more general design matrices in the Supplementary Material. It is worth noting that there is a parallel line of research that aims to develop universality results for these comparison techniques. We refer the interested reader to [11] and references within.

Let us emphasize that our main conceptual concern is not investigating whether (4) as a convex optimization procedure gives a good point estimator, but instead evaluating whether NMF as a strategy or product distributions as a family of distributions can provide 'close to correct' approximation for the true posterior. Nevertheless, this optimizer's asymptotic mean square error can also be characterized as a by-product of our main theorem.

Regarding the accuracy of variational approximations in general, certain contraction rates and asymptotic normality results were established in the traditional fixed p large n regime [28, 17, 10]. However, under the high-dimensional setting and scaling we consider in the current paper, without extra structural assumptions (e.g., sparsity), both the true posterior and its variational approximation are not expected to contract towards the true signal, which also explains why one is instead interested in whether the log-normalizing constant can be well approximated, as a weaker standard of 'correctness'. Ray et al. [19] studied a pre-specified class of mean field approximation in sparse high-dimensional logistic regression. Recently, the first known results on mean and covariance approximation error of Gaussian Variational Inference (GVI) in terms of dimension and sample size were obtained in [12].

2 Results

This section starts with some necessary definitions and our main assumptions. Then, we present our main theorem and one natural corollary. Finally, we identify a wide class of priors that would ensure the convexity of the NMF objective, which plays a crucial role in our analysis.

2.1 Notations and main assumptions

Notations: We use the usual Bachmann-Landau notation O ( · ) , o ( · ) , Θ( · ) for sequences. For a sequence of random variables { X p : p ≥ 1 } , we say that X p = o p (1) if X p P → 0 as p → ∞ and X p = o p ( f ( p )) if X p /f ( p ) = o p (1) . We use C, C 1 , C 2 · · · to denote positive constants independent of n, p . Further, these constants can change from line to line. For any square symmetric matrix A , ∥ A ∥ op and ∥ A ∥ F denote the matrix operator norm and the Frobenius norm, respectively.

Assumption 1 (Proportional asymptotics) . We assume n/p → α ∈ (0 , ∞ ) , as n, p →∞ .

Assumption 2 (Gaussian features) . For all our theoretical results, we assume the design matrix X is randomly sampled from an iid Gaussian ensemble, i.e., X ij iid ∼ N (0 , 1 /n ) .

Definition 3. Define F : ( m ( π ) , M ( π )) → R as

Definition 4 (The NMF point estimator) . Recalling the NMF objective M p ( · ) as in Definition 2, let ˆ u = ˆ β NMF := arg min u ∈ [ -1 , 1] p M p ( u ) be the NMF point estimator, which is also the mean vector of the product distribution ( ˆ Q ) that best approximates the posterior in terms of KL-divergence. We refer to this optimal product distribution as the NMF distribution.

Assumption 3 (Convexity of F ( · ) ) . We assume F ( · ) is strongly convex on S o := S \ ∂S .

As alluded to, our analysis relies on the convexity of the 'penalty' term F ( · ) . Please note that the definition of F ( · ) only depends on the prior π chosen by the statistician, rather than the 'true prior' π ⋆ . Therefore, to support this assumption, we provide a few sufficient conditions that identify a broad class of priors that ensure (strong) convexity of F ( · ) in Section 2.3.

Throughout, we work under a partially well-specified situation, i.e., model (1) is assumed to be correct, but β ⋆ i 's may not have been a priori sampled iid from π . Instead, we assume the empirical distribution of β ⋆ i 's converges in L 2 to a probability distribution π ⋆ supported on S . In addition, the noise level σ 2 is fixed and known to the statistician. Last but not least, π ⋆ is assumed to have finite second moment and let s 2 := E T ∼ π ⋆ [ T 2 ] < ∞ .

2.2 Main results

From now on, we always assume Assumption 1, 2, and 3. Next, we introduce a scalar denoising function, which is just the proximal operator of F ( · ) .

Definition 5 (Scalar denoising function) . For x ∈ R and t > 0 ,

Since F ( · ) is strongly convex, this one-dimensional optimization has a unique minimizer. Note that when S = [ -1 , 1] , since lim w →± 1 d F d w ( w ) = lim w →± 1 h ( w, 1 /σ 2 ) ∓ 1 σ 2 = ±∞ , the minimum is never achieved on the boundary of S . Similarly, when S = R , lim w →±∞ d F d w ( w ) = ±∞ . Therefore, the minimum is always achieved at a stationary point. Lastly, η (0 , t ) = 0 if π is symmetric. In fact, throughout this paper, we only consider symmetric priors.

Before stating our main result and its implications, we first introduce a two-dimensional optimization problem, which will play a central role in our later discussion,

where F ( · ) was defined in Definition 3 and the E is taken over ( B,Z ) ∼ π ⋆ ⊗N (0 , 1) . In the next lemma, we gather some additional characterizations of this min-max problem.

Lemma 2. The max-min in (5) is achieved at some ( b ⋆ , τ ⋆ ) ∈ (0 , ∞ ) × ( σ, ∞ ) . In fact, b ⋆ is unique. In addition, ( b ⋆ , τ ⋆ ) is also a solution to the following fixed point equation,

where ∂x

Definition 6. We use ν ⋆ = ν ⋆ π,π ⋆ to denote the distribution of ( η ( τ ⋆ Z + B, τ ⋆ σ 2 b ⋆ ) , B ) , in which ( B,Z ) ∼ π ⋆ ⊗N (0 , 1) . We denote by ˆ ν the empirical distribution of { (ˆ u i , β ⋆ i ) } p i =1 .

We are ready to state our main result, which provides a sharp asymptotic characterization of ˆ ν .

Theorem 1. Suppose the max-min problem in (5) has a unique optimizer ( b ⋆ , τ ⋆ ) , or the fixed point equation in (7) has a unique solution ( b ⋆ , τ ⋆ ) . Then for all ε > 0 , as n, p →∞ ,

where W 2 ( · , · ) stands for order 2 Wasserstein distance.

Remark 1. This result indicates the NMF estimator ˆ u should be asymptotically roughly iid among different coordinates, which is different from the NMF distributions being product distributions.

Corollary 1. Suppose the hidden true signal β ⋆ was a priori sampled iid from a probability distribution π ⋆ with finite second moment. Note that π ⋆ can differ from the prior π that the Bayesian statistician chose. In addition, suppose the max-min problem in (5) has a unique optimizer ( b ⋆ , τ ⋆ ) , or the fixed point equation in (7) has a unique solution ( b ⋆ , τ ⋆ ) , then for all ε > 0 ,

in which ν ⋆ was defined in Definition 6 .

We provide a proof sketch in Section 6 and all the detailed proofs are deferred to the Supplementary Materials.

2.3 Convexity of F ( · )

In this section, we present a few lemmas that would ensure the validity of Assumption 3. In fact, if conditions of any of these lemmas are satisfied, Assumption 3 holds.

Lemma 3 (Condition to ensure convexity of F ( · ) : nice prior) . Suppose π is absolutely continuous with respect to Lebesgue measure and

for some V : support( π ) → R . In addition, suppose either of the following two conditions is true,

support( π ) = R ; V ( x ) is continuously differentiable almost everywhere; V ( x ) is unbounded above at infinity.

support( π ) = [ -a, a ] , for some 0 < a < ∞ ; V ( x ) is continuously differentiable almost everywhere.

Then if V ( x ) is even, non-decreasing in | x | and V ′ ( x ) is convex, F ( · ) is always strongly convex, regardless of the value of σ 2 .

Lemma 4 (Condition to ensure convexity of F ( · ) : discrete prior) . Suppose π is a symmetric discrete distribution supported on {-1 , 0 , 1 } ,

for q ∈ (2 / 3 , 1) . Then F ( · ) is always strongly convex, regardless of the value of σ 2 .

Proofs of Lemma 3 and 4 crucially utilize the Griffiths-Hurst-Sherman (GHS) inequality [5, 6], which arose from the study of correlation structure in spin systems. The following two lemmas give examples of some other families of priors for which convexity of F ( · ) depends on the noise level σ 2 , while those in Lemma 3 and 4 do not.

Lemma 5 (Condition to ensure convexity of F ( · ) : low signal-to-noise ratio) . Suppose support( π ) ⊂ [ -a, a ] for some a > 0 . Then as long as σ 2 > a 2 , F ( u ) = F π ( u, σ 2 ) , as a function of u , is always strongly convex on S , regardless of the exact choice of π and value of σ 2 .

Lemma 6 (Condition to ensure convexity of F ( · ) : Spike and Slab prior) . Consider a spike and slab prior to the following form,

which is just a mixture of a point mass at 0 and a Normal distribution of mean 0 and variance ∆ 2 . Suppose

Approximation:lognormalizing constant(delta~2=0.6,alpha=1)

0.725

NMF approximation

Limiting log Z

0.700

constant

0.675

0.650

0.625

0.600

601

0.575

0.550

0.525

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

b

NMF approximation

2.0

Limiting log Z

1.5

1.0

0.5

0.0

2

4

6

8

10

12

14

16

alpha

where π ˜ q, ˜ ∆ 2 is again a Gaussian spike and slab mixture,

Then, F ( u ) is strongly convex. In addition, one easier-to-check sufficient condition for (8) is

Remark 2. It is easy to see that for large enough σ ( q and ∆ fixed), or small enough q ( ∆ and σ fixed), or small enough ∆ ( q and σ fixed), (9) is always satisfied. In other words, F ( · ) is strongly convex for low signal-to-noise ratio or high temperature in physics parlance.

3 Log normalizing constant: sub-optimality of NMF

As alluded, as implications of Theorem 1, we develop asymptotics of both log Z NMF p and mean square error (MSE) of the NMF point estimator ˆ u in terms of ( b ⋆ , τ ⋆ ) .

Corollary 2 (MSE) . When conditions of Corollary 1 hold, as n, p →∞ ,

Corollary 3 (Log normalizing constant) . When conditions of Corollary 1 hold, as n, p →∞ ,

̸

Though all our main theorems and corollaries apply to the case when π ⋆ = π , for simplicity and clarity, from now on, we only consider the 'nicest' setting, i.e, when assumptions of Corollary 1 are satisfied and in addition π = π ⋆ . By doing so, we would like to convey that even if there were no model mismatch at all, NMF still would not be 'correct'.

Concentration and limiting values of both the optimal Bayesian mean square error (i.e., E µ ∥ β -E [ β ⋆ | X,y ] ∥ 2 /p ) and the actual log-normalizing constant were conjectured and rigorously established

CredibleintervalsgeneratedbyNMF(delta~2=2)

Credibleintervalsgenerated byNMF(alpha=4)

0.950

0.94

0.925

0.900

CoverageRate

0.92

Rate

0.875

Coverage

0.90

0.850

q = 0.1

q = 0.1

q = 0.2

0.825

q = 0.2

0.88

q = 0.3

q = 0.3

q=0.4

0.800

q = 0.4

q = 0.5

q =0.5

0.95

0.775

0.95

2

4

6

8

10

12

14

16

0

1

2

3

4

5

delta~2

alpha

under additional regularity conditions, which provides us the 'correct answers' to compare with. Please see [2, 20].

Please see Figure 2 for numerical evaluations of Corollary 3, which suggests the bound in (4) is not tight for Gaussian Spike and Slab prior. Since, in general, both F ( · ) and η ( · , · ) lack analytical forms, it is hard to provide a universal guarantee on whether (5) has a unique optimizer or the fixed point equation (7) has a unique solution. In fact, our numerical experiments suggest it is possible for (7) to have multiple fixed points. Therefore, how to exactly realize and evaluate the asymptotic predictions in these two corollaries (so as Corollary 4 in the next section) is challenging in general and can only be done in a case-by-case basis and usually involves numerically solving (7). In light of this observation, we use the Gaussian Spike and Slab prior as defined in Lemma 6 for presentation purposes. Since it is both non-trivial and of practical interest, though, we do emphasize that the same framework and workflow also apply to other priors. Without loss of generality, we also take σ 2 = 1 . This choice renders Figure 2 and 3 in the next section. Details of how to generate these plots are deferred to the Supplementary Material.

4 Uncertainty quantification: the average coverage rate

To study uncertainty quantification properties of NMF approximation, we consider the average coverage rate of symmetric Bayesian credible regions (of level 1 -ζ ) suggested by the NMF distributions, i.e, R p,ζ := 1 p ∑ p i =1 ✶ { β ⋆ i ∈ [ˆ q i,ζ/ 2 , ˆ q i, 1 -ζ/ 2 ] } , where ˆ q i,t is the t -th quantile of π ( h (ˆ u i ,d i ) ,d i ) . In order to study asymptotic behavior of R p,ζ , we define an ( m ( π ) , M ( π )) × S →{ 0 , 1 } indicator function

The following corollary of Theorem 1 establishes the asymptotic convergence of R p,ζ . Numerically evaluating it for the Gaussian Spike and Slab prior renders Figure 3, which shows NMF credible regions can not achieve the nominal coverage, in this case, 95%, and also provides an exhibition of how large the gaps are for different hyper-parameters.

Corollary 4. Suppose conditions of Corollary 1 hold. In addition, assume the quantile function of π is continuous. Then as n, p →∞ ,

On the other hand, based on the asymptotic joint distribution of ˆ u and β ⋆ as stated in Corollary1, we can in fact identify a strategy of constructing asymptotically exact Bayesian credible regions based on ˆ u . Let q t ( x ) be the t -th quantile of conditional distribution of B given η ( τ ⋆ Z + B,τ ⋆ σ 2 /b ⋆ ) = x . This way, the following Corollary ensures [ q ζ/ 2 (ˆ u i ) , q 1 -ζ/ 2 (ˆ u i )] is asymptotically of at least 1 -ζ coverage.

Corollary 5. Suppose conditions of Corollary 4 hold, then for any ε > 0 ,

5 Discussion: Extensions and Limitations

̸

In order to provide some intuition on why the NMF approximation is loose in the current setting, it is worth noting that in comparison with the proportional asymptotics regime we consider here, positive results of NMF for high-dimensional linear regression were recently established in [16] when p = o ( n ) . Using terminology from Austin [1], Mukherjee and Sen [16] (when restricted to designs with iid Gaussian features) essentially proved, when p/n → 0 , the eigenvalue concentration behavior of X T X leads to the Hamiltonian being of 'low complexity'. On the other hand, when p = Θ( n ) , tr( A 2 ) = o ( p ) , where A = A ( X ) is defined as the off-diagonal part of X T X , which violates [16, Equation (5)]. Roughly speaking, when the eigenstructure of A is not 'dominated' by a few top eigenvalues, the Hamiltonian can not be covered by an efficient net and thus is not of 'low complexity'. Please see [1, 16, 3] for more details.

We want to be clear about the fact that, technically, we did not 'prove' the sub-optimality of NMF. Instead, we rigorously derived asymptotic characterizations of NMF approximation through the solution of a fixed point equation. But this fixed point equation can only be solved numerically on a case-by-case basis and is not guaranteed a unique solution. All our plots are based on iteratively solving the fixed point equation. As a matter of fact, for instance, when q is close to 1 for the Gaussian Spike and Slab prior we considered, the fixed point equation is clearly not converging to the right fixed point, as demonstrated in the Supplementary Material. It could also just not converge for very small α . Nevertheless, all the plots we are showing in the main text are backed by a numerical simulation using simple gradient descent to optimize the NMF objective, i.e. inf M p ( u ) , for n = 8000 . All in all, it is probably more accurate to say we provided a tool for establishing the sub-optimality of NMF for a general class of priors rather than proving it for good.

Another obvious limitation is we can only handle priors that guarantee convexity of the KL-divergence term in terms of the mean parameters. Though it is indeed a broad class of distributions covering some of the most commonly used symmetric priors (e.g., Gaussian, Laplace, and so on), little is known about the asymptotic behavior of NMF when the convexity assumption is violated.

We note that, in theory, in order to carry out the analysis using CGMT, the additive noise ϵ as defined in (1) does not have to be Gaussian. Instead, as long as it has log-concave density, the same proof idea applies, though we intentionally chose to stick with Gaussian noise as it renders much cleaner results and a more comprehensive presentation. In addition, we expect stronger uniform convergence results (e.g., uniform in σ 2 ) could also be established, which can be crucial for applications like hyperparameters selection. Please see [15] for an example in which results of this flavor were obtained.

6 Proof strategy

This section gives a proof outline of Theorem 1. More details can be found in the Supplementary Material. Replacing all d i 's in M p with E d i = 1 /σ 2 , we define N p as

Lemma 7. Let ˆ u N := arg min u [ N p ( u )] . Then for some C s ∈ R + , as n, p →∞ ,

Lemma 8. For any ε > 0 , as n, p →∞ , with C s as defined in Lemma 7,

According to Lemma 8 and 7, N p ( · ) and M p ( · ) are with high probability uniformly close. Thus, from now on, we focus on using Gaussian comparison to analyze ˆ u N and N p (ˆ u N ) in place of ˆ u and M p (ˆ u ) . Since F ( · ) is strongly convex, ˆ w := ˆ u N -β ⋆ is the unique minimizer of

By introducing a dual vector s , we get

By CGMT (see for instance [22, Theorem 3.3.1] or [15, Theorem 5.1]), it suffices now to study

where g ∼ N (0 , I p ) and h ∼ N (0 , I n ) and they are independent. Note that the min and max can be flipped due to convex-concavity. By optimizing with respect to s/ ∥ s ∥ and introducing

Under minor regularity conditions, as n, p →∞ , it converges to

with ( B,Z ) ∼ π ⋆ ⊗N (0 , 1) , which is how we got ϕ ( · , · ) as in (5). Furthermore, by differentiating ϕ ( b, τ ) with respect to τ and b , we arrive at the fixed point equation in Lemma 2. Last but not least, note that arg min w { b 2 τ w 2 -bZw + σ 2 F ( w + B ) } = η ( τZ + B,τσ 2 /b ) -B , which explains why the joint empirical distribution of ( ˆ w i , β ⋆ i ) 's converges to the law of ( η ( τ ⋆ Z + B,τ ⋆ σ 2 /b ⋆ ) -B,B ) . Finally, we note that similar proof arguments were made in [15, 22].

References

T. Austin. The structure of low-complexity gibbs measures on product spaces. The Annals of Probability , 47(6):4002-4023, 2019.

J. Barbier, N. Macris, M. Dia, and F. Krzakala. Mutual information and optimality of approximate message-passing in random linear estimation. IEEE Transactions on Information Theory , 66(7):4270-4303, 2020.

A. Basak and S. Mukherjee. Universality of the mean-field for the potts model. Probability Theory and Related Fields , 168:557-600, 2017.

D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association , 112(518):859-877, 2017.

R. S. Ellis and J. L. Monroe. A simple proof of the ghs and further inequalities. Communications in Mathematical Physics , 41(1):33-38, 1975.

R. S. Ellis, J. L. Monroe, and C. M. Newman. The ghs and other correlation inequalities for a class of even ferromagnets. Communications in Mathematical Physics , 46(2):167-182, 1976.

B. Ghorbani, H. Javadi, and A. Montanari. An instability in variational inference for topic models. In International conference on machine learning , pages 2221-2231. PMLR, 2019.

R. J. Giordano, T. Broderick, and M. I. Jordan. Linear response methods for accurate covariance estimates from mean field variational bayes. Advances in neural information processing systems , 28, 2015.

Y. Gordon. Some inequalities for gaussian processes and applications. Israel Journal of Mathematics , 50:265-289, 1985.

P. Hall, T. Pham, M. P. Wand, and S. S. Wang. Asymptotic normality and valid inference for gaussian variational approximation. 2011.

Q. Han and Y. Shen. Universality of regularized regression estimators in high dimensions. arXiv preprint arXiv:2206.07936 , 2022.

A. Katsevich and P. Rigollet. On the approximation accuracy of gaussian variational inference. arXiv preprint arXiv:2301.02168 , 2023.

F. Krzakala, A. Manoel, E. W. Tramel, and L. Zdeborová. Variational free energies for compressed sensing. In 2014 IEEE International Symposium on Information Theory , pages 1499-1503. IEEE, 2014.

C. A. McGrory and D. Titterington. Variational approximations in bayesian model selection for finite mixture distributions. Computational Statistics & Data Analysis , 51(11):5352-5367, 2007.

L. Miolane and A. Montanari. The distribution of the lasso: Uniform control over sparse balls and adaptive parameter tuning. The Annals of Statistics , 49(4):2313-2335, 2021.

S. Mukherjee and S. Sen. Variational inference in high-dimensional linear regression. arXiv preprint arXiv:2104.12232 , 2021.

D. Pati, A. Bhattacharya, and Y. Yang. On statistical optimality of variational bayes. In International Conference on Artificial Intelligence and Statistics , pages 1579-1588. PMLR, 2018.

J. Qiu and S. Sen. The tap free energy for high-dimensional linear regression. arXiv preprint arXiv:2203.07539 , 2022.

K. Ray, B. Szabó, and G. Clara. Spike and slab variational bayes for high dimensional logistic regression. Advances in Neural Information Processing Systems , 33:14423-14434, 2020.

G. Reeves and H. D. Pfister. The replica-symmetric prediction for compressed sensing with gaussian matrices is exact. In 2016 IEEE International Symposium on Information Theory (ISIT) , pages 665-669. IEEE, 2016.

M. Stojnic. A framework to characterize performance of lasso algorithms. arXiv preprint arXiv:1303.7291 , 2013.

C. Thrampoulidis. Recovering structured signals in high dimensions via non-smooth convex optimization: Precise performance analysis . PhD thesis, California Institute of Technology, 2016.

C. Thrampoulidis, S. Oymak, and B. Hassibi. Regularized linear regression: A precise analysis of the estimation error. In Conference on Learning Theory , pages 1683-1709. PMLR, 2015.

C. Thrampoulidis, E. Abbasi, and B. Hassibi. Precise error analysis of regularized m -estimators in high dimensions. IEEE Transactions on Information Theory , 64(8):5592-5628, 2018.

J. v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen , 100(1):295-320, 1928.

R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference . Now Publishers Inc, 2008.

F. Zhang and C. Gao. Convergence rates of variational posterior distributions. 2020.

Supplementary Materials

A Technical lemmas and basic facts

Lemma 9. Let ˙ c ( h, d ) := ∂c ∂h ( h, d ) and ¨ c ( h, d ) := ∂ 2 c ∂h 2 ( h, d ) . We have, for u ∈ ( m ( π ) , M ( π )) and d ∈ R ,

Lemma10 (von Neumann's minimax theorem, [25]) . Let S w ⊂ R n and S s ⊂ R m be compact convex sets. If f : S w × S s → R is a continuous function that is convex-concave, i.e., f ( · , s ) : S w → R is convex for fixed s , and f ( w, · ) : S s → R is concave for fixed w . Then we have that

Theorem 2 (CGMT, [23, 22, 15]) . Let S w ⊂ R p and S s ⊂ R n be two compact sets and let Q : S w × S s → R be a continuous function. Let G = ( G ij ) 1 ≤ i ≤ n, 1 ≤ j ≤ p iid ∼ N (0 , 1) , g ∼ N (0 , I p ) , h ∼ N (0 , I n ) be independent standard Gaussian vectors. Denote

Then we have

For all t ∈ R ,

If both S w and S s are convex and if Q ( · , · ) is convex-concave, then for all t ∈ R ,

Remark 3. The most important message of this theorem is essentially whenever Ψ( g, h ) concentrates around a certain value t , Φ( G ) will also concentrate around t , assuming Q ( · , · ) is convex-concave.

B Proofs

Proof of Lemma 1 and 9 can be found in for instance [16].

B.1 Convexity of F ( · )

Proof of Lemma 3. We only prove part (1) here, as proof of part (2) is almost exactly the same. For any h, d ∈ R + , by GHS inequality [6, Equation 1.4],

Together with the assumption that V is even, we have for any h ∈ R and d ≥ 0 ,

Consider now a family of parametric distributions {P θ : θ ≥ 0 } as a generalization of π (0 ,d ) , with

Note that P θ =1 = π (0 ,d ) . Since V ( · ) is even and increasing,

which ensures Var B ∼ π ( h ( u, 1 /σ 2 ) , 1 /σ 2 ) ( B ) ≤ σ 2 and therefore d 2 F d u 2 ( u ) ≥ 0 by (11). Note that as long as π is a valid probability distribution, F ( · ) is not only convex but always strongly convex, as Var B ∼ π ( h ( u, 1 /σ 2 ) , 1 /σ 2 ) ( B ) = σ 2 if and only if V ( · ) is a constant function and the support of π is the whole real line.

The same proof idea also applies to Lemma 4; therefore, we omit its proof to avoid redundancy.

Proof of Lemma 5. The conclusion follows by noting

as π ( h,d ) is a distribution on [ -a, a ] and thus its variance is at most a 2 , which is assumed to be smaller than σ 2 .

For Lemma 6, since Var B ∼ π ( h,d ) ( B ) can be analytically computed for the Gaussian Spike and Slab prior, its proof is nothing but elementary calculation and then checking for (11).

B.2 Replacing d i with E d i

Proof of Lemma 7. We focus on only ∥ ˆ u ∥ since almost exactly the same argument also applies to ˆ u N . We first collect a few high-probability claims, proofs of which are just direct applications of basic standard random matrix results (see, for instance, [26]).

There exist positive constants C 1 and C 2 (only depend on α ), such that for any ε > 0 , S 1 := {| λ max ( X T X ) -C 1 | < ε } and S 2 := {| λ min ( X T X ) -C 2 | < ε } are both of high probability.

Recall the additive noise ϵ ∼ N (0 , σ 2 I n ) . For any ε > 0 , S 3 := {|∥ ϵ ∥ 2 /n -σ 2 | < ε } is of high probability.

For any ε > 0 , S 4 = { ∣ ∣ ϵ T Xβ ⋆ /p ∣ ∣ < ε } is of high probability.

Let S 0 = S 1 ∩ S 2 ∩ S 3 ∩ S 4 , which is again an event of approaching 1 probability. Note that since the empirical distribution of β ⋆ i 's converge in L 2 to π ⋆ , one has ∥ β ⋆ ∥ 2 < 1 . 01 ps 2 for large enough p . When S 0 happens, if ∥ u ∥ 2 /p > (1 + C s ) s 2 (with C s > 0 to be chosen later, but large enough such that ∥ Xu ∥ > ∥ Y ∥ ),

On the other hand,

Upon C s being large enough, we have N p ( u ) > N p ( → 0 ) for any u such that ∥ u ∥ 2 /p > (1 + C s ) s 2 . Therefore, ∥ ˆ u N ∥ 2 /p < (1 + C 2 ) s 2 on S 0 .

Proof of Lemma 8. If S = [ -1 , 1] , by Lemma 9, ∣ ∣ ∣ ∂G ( u,d ) ∂d ( u, d ) ∣ ∣ ∣ ≤ 1 2 for any u, d , thus

Since X ji 's are iid with variance 1 /n , we know E d i = 1 σ 2 E [ ∑ n j =1 X 2 ji ] = 1 σ 2 , d i n →∞ -→ 1 σ 2 , and all d i 's are iid , which guarantee RHS of the previous display goes to 0 in probability as n, p →∞ . On the other hand, if S = R , note that for any δ ∈ (0 , 1 / (2 σ 2 )) , P (max 1 ≤ i ≤ p | d i -1 /σ 2 | > δ ) → 0 as n, p → ∞ . In addition, when max 1 ≤ i ≤ p | d i -1 /σ 2 | ≤ δ is true, which is of approaching 1 probability,

where δ i ∈ ( min(0 , d i -1 /σ 2 ) , max(0 , d i -1 /σ 2 ) ) . By Lemma 9, it is further smaller than

Lastly, note that when conditions of one of Lemma 3, 4, 5 and 6 are true, for ˜ d close enough to 1 /σ 2 , we have Var X ∼ π ( ˜ h, ˜ d ) ( X ) < 2 σ 2 for any ˜ h ∈ R . Therefore, upon choosing small enough δ such that all d i 's are close enough to 1 /σ 2 , the display above is controlled by

Lastly, further requiring δ < ε 4 σ 2 +(1+ C s ) s 2 renders Lemma 8.

B.3 Regarding the fixed point equation

Proof of Lemma 2. First of all, recall the definition of ϕ ( · , · ) in (6),

Note that for any fixed x , | x -η ( x, t ) | is always strictly increasing with respect to t , we have

Therefore, for any fixed τ , ϕ ( · , τ ) is 1 -strongly concave. Define ψ ( b ) := min τ ≥ σ ϕ ( b, τ ) . Since ψ ( · ) is the minimum of a collection of 1 -strongly concave functions, it is 1 -strongly concave itself and which further leads to

must have a unique maximizer b ⋆ over [0 , ∞ ) . In addition, by definition of η , lim t →∞ η ( x, t ) = 0 , dominated convergence theorem gives

Therefore, for any fixed τ ,

̸

Together with Lemma 11 and continuity of ϕ ( · , · ) , it ensures b ⋆ = 0 . On the other hand, for any b > 0 ,

Together with Lemma 11, we have min τ ≥ σ ϕ ( b ⋆ , τ ) has at least one minimizer τ ⋆ ∈ ( σ, ∞ ) . Finally, since b ⋆ and τ ⋆ are not on the boundary, we have ∂ϕ ∂b ( b ⋆ , τ ⋆ ) = ∂ϕ ∂τ ( b ⋆ , τ ⋆ ) = 0 , which gives rise to the fixed point equation as in (7).

Lemma 11. Recall the definition of ϕ in (6) . For any fixed b ∈ (0 , ∞ ) ,

Therefore, min τ ϕ ( b, τ ) admits at least one minimizer.

Proof. Since E [ B 2 ] = s 2 < ∞ , E min w ∈ S { b 2 τ w 2 -bZw + σ 2 F ( w + B ) -σ 2 F ( B ) } is decreasing in τ and always finite for any ( b, τ ) ∈ (0 , ∞ ) × [ σ, ∞ ) . Therefore lim τ →∞ ( b, τ ) = ∞ .

B.4 Proof of the main results

We devote this subsection to proving Theorem 1, while we note Corollary 1, 2, 3, 4 and 5 are all direct consequences of it. We first prove Theorem 1 while introducing some necessary lemmas. Then, we prove these lemmas at the end of this subsection. Whenever the optimization domains for w and s are omitted throughout this subsection, they are understood to be R p and R n , respectively. We use ˆ ν to denote empirical distribution in general.

Since F ( · ) is strongly convex, ˆ w := ˆ u N -β ⋆ is the unique minimizer of

By introducing a dual vector s , we get

Following the recipe in Theorem 2, we define

where g ∼ N (0 , I p ) and h ∼ N (0 , I n ) and they are independent. Note that with a deliberate abuse of notations, we use Φ and Ψ to denote these two functions to indicate their resemblance to those in the statement of Theorem 2. By Theorem 2, it suffices now to study min w max s Ψ g,h ( w,s ) in place of min w max s Φ X ( w,s ) , which is made rigorous by the following lemma.

Lemma 12. Let D be any close set.

We have for all t ∈ R

If D is in addition convex, then we have for all t ∈ R

We defer the proof of Lemma 12 to the end of this section and proceed with proving Theorem 1. Due to strong convexity, ˆ w Ψ := arg min w max s Ψ g,h ( w,s ) always exists and is unique. Note that the min and max can be flipped due to convex-concavity (Lemma 10). By optimizing with respect to s/ ∥ s ∥ and introducing

min w max s Ψ g,h ( w,s ) can be further reduced to max b ≥ 0 min τ ≥ σ Γ g,h ( b, τ )

in the sense that (i) the optimizers ˆ w Ψ and ˆ w Γ are close, i.e, for any κ > 0 ,

and (ii) the optimum value is preserved with arbitrarily small error with high probability. The next lemma ensures empirical distribution of ( ˆ w Ψ , β ⋆ ) is close to the distribution of ( η ( τ ⋆ Z + B, τ ⋆ σ 2 b ⋆ ) -B,B ) , which we denote as ν ⋆ ( w ⋆ ,π ⋆ ) , where ( B,Z ) ∼ π ⋆ ⊗N (0 , 1) .

Lemma 13. Suppose all conditions of Theorem1 are satisfied. For any ε > 0 , there exists C ( ε ) ∈ (0 , ε ) , such that as p, n →∞ ,

In the meantime,

Again, for now, we proceed assuming Lemma 13 and prove it later at the end of this section. Building upon Lemma 12 and Lemma 13, we now prove the empirical distribution of (ˆ u N , β ⋆ ) = ( β ⋆ + ˆ w,β ⋆ ) is close to ν ⋆ as defined in Definition 6. For ε > 0 , define D ε =

it suffices to show with high probability, for some δ ( ε ) > 0 ,

On the one hand, by applying both (1) and (2) of Lemma 12 to D = R p , together with Lemma 13, we have

where the ' lim ' is understood to be convergence in probability. It further leads to

On the other hand, applying (1) of Lemma 12 to D = D ε , together with Lemma 13, we have

which establishes (15) with δ ( ε ) = C ( ε ) + ε , where C ( ε ) > 0 was defined in Lemma 13. Therefore, we have the empirical distribution of (ˆ u N , β ⋆ ) is close to the target distribution ν ⋆ , i.e.,

Finally, according to Lemma 8 and 7, N p ( · ) and M p ( · ) are with high probability uniformly close. Together with strong convexity of N p ( · ) , we have for any κ > 0

Theorem 1 is therefore given by (16) and (17).

Proof of Lemma 12. In order to prove Lemma 12 using Theorem 2, one only needs to establish that the optimizer of Φ X ( w,s ) always has a bounded norm with high probability. In fact, Lemma 7 ensures boundedness of ˆ w = arg min w max s Φ X ( w,s ) while the boundedness of ˆ s := arg max s Φ X ( ˆ w,s ) can be established by a similar argument.

Proof of Lemma 13. Define

By definition, ˜ Γ g,h ( b, τ ) ≥ Γ g,h ( b, τ ) for any ( b, τ ) deterministically. Recall the definition of Φ g,h ( · ) in (12),

Therefore,

where ˜ τ ( b ⋆ ) := arg min τ ≥ σ ˜ Γ g,h ( b ⋆ , τ ) . We claim that the gaps resulting from (i) and (ii) can not be both negligible. Namely, there exists ϱ > 0 such that

In order to establish (19), we will proceed with proof by contradiction. Suppose (19) is NOT true, equivalently, for any ϱ > 0 ,

Recall the definition of ϕ ( · ) in (6). Since ( b ⋆ , τ ⋆ ) is the unique optimizer of ϕ ( · ) and Γ g,h ( b, τ ) converges to ϕ ( b, τ ) uniformly on any compact subset of [0 , ∞ ) × [ σ, ∞ ) , (21) is only possible if | ˜ τ ( b ⋆ ) -τ ⋆ | P. → 0 as n, p →∞ . On the other hand, by definition of D ε , there exists γ > 0 such that with high probability, for any w ∈ D ε ,

where ¯ w i is sampled independently from the distribution of η ( τ ⋆ Z + β ⋆ i , τ ⋆ σ 2 b ⋆ -β ⋆ i ) with Z ∼ N (0 , 1) . Therefore, with high probability,

where

By triangle inequality,

In addition, note that if | ˜ τ ( b ⋆ ) -τ ⋆ | P. → 0 as n, p →∞ , then

Putting them together, we have

The display above is in contradiction to (20), which means (19) is thus established.

Finally, note that Lemma 13 is equivalent to

and it is therefore a direct consequence of (19) and (18).

C Numerical simulations

All source code can be found in a separate zip file in the Supplementary Materials.

1.8

1.6

+

NMF: Gradient Decent simulation with Gaussian design

NMF: Gradient Decent simulation with Laplace design

NMF: CGMT Theory Prediction

q = 0.1; delta ~2 = 3.

2.00

1.75

NMF: Gradient Decent simulation with Gaussian design

NMF: Gradient Decent simulation with Laplace design

NMF: CGMT Theory Prediction

MSE (q = 0.1; alpha = 1)

0.35

0.40

X

alpha = 1; delta~2 = 0.6.

1.50

1.4

1.25

0.30

MSE

1.2

MSE

1.00

MSE

0.25

1.0 -

0.75

0.50

0.20

F 8°0

0.6

X

0.25

0.00

0.15

+

NMF: CGMT Theory Prediction

NMF: Gradient Decent simulation with Gaussian design

NMF: Gradient Decent simulation with Laplace design

2

4

6

8

alpha

10

12

14

16

0

1

2

delta

3

4

5

0.1

0.2

'0

0.4

q

0.5

0.6

0.7

C.1 Universality: non-Gaussian design matrix

Instead of assuming X ij iid ∼ N (0 , 1 /n ) , we now present empirical evidence of universality, i.e., Theorem 1 holds for a broader class of design matrix that has iid entries with variance 1 /n . Since it is impossible to exhaust all possible distributions, we will stick with a representative example X ij iid ∼ Laplace ( √ 2 / 2) and the Gaussian spike and slab prior. We use Gradient Decent to optimize M p ( u ) and then demonstrate that the empirical MSE of its optimizer coincides with the prediction of Corollary 2. Please see Figure 4 for a visual summary.

For more comprehensive and rigorous results on the universality of Gaussian comparison inequalities, we refer interested readers to [11] and references within.

C.2 Fixed point equation

alpha =2; delta~2 =0.4.

NMF: CGMT Theory Prediction

0.8

NMF: Gradient Decent Simulation

PriorVariance

AMP

0.6

MSE

0.4

0.2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

q

As alluded to in the main text, all our plots are generated by iteratively solving the fixed point equation (7). However, this naive strategy might not give the right fixed point, i.e., the ( b ⋆ , τ ⋆ ) that minimizes ϕ ( b, τ ) , or it could just do not converge. Please see Figure 5 for an empirical example. Since either F ( · ) or η ( · , · ) lacks analytical forms for most natural priors, unlike other applications of CGMT (e.g., asymptotic analysis of lasso [15]), it is hard to determine whether (7) has a unique solution analytically. Fortunately, there are two possible remedies. First, which is the option we took, one could solve min u M p ( u ) for some large n and check if the empirical MSE matches the prediction given by the fixed point. Alternatively, one could adapt a more brute-force way to find the actual optimizer of max b min τ ϕ ( b, τ ) , e.g., grid search or iteratively solving (7) with multiple initializations. After all, it is only a two-dimensional scalar optimization problem. We followed the first way simply because we wanted to use empirical simulations to corroborate our theoretical predictions anyway.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives precise asymptotic characterization for the naive mean field approximation in high-dimensional linear regression under the proportional limit. Two concrete corollaries are obtained including the inaccuracy of the NMF approximation for the log-normalizing constant and the overconfidence of the NMF approximation in terms of uncertainty quantification.\n\nSTRENGTHS:\n1. The naive mean field approximation, the problem studied in the paper, is widely used in modern machine learning. The paper contributes  to understanding the utility of NMF.\n\n2. The paper is mathematically solid and all the technicalities are precisely presented. The proof techniques utilized are quite novel in the area of variational inference.\n\nWEAKNESSES:\nI do not find any major weakness of the paper but there are some presentation issues that can be possibly improved. Since the paper is overall very technical and mathematical, I suggest the reviewers to setup a table or a subsection exclusively for all notations. Also, I suggest to considering to formalize conditions utilized throughout the paper such as Gaussian features, noises with special environments of assumptions instead of stating them within texts. The current presentation is a bit challenging for me to follow. Also, there are some typos that need to be fixed. For example, in line 78. I think the definition should be $G(u,d):=D_{KL}(\\pi^{(h(u,d),d)}\\Vert\\pi^{(0,d)})=\\cdots$; in line 129, the right most term should be $G(u,\\frac{1}{\\sigma^2})-\\frac{u^2}{2\\sigma^2}$.\n\nQUESTIONS:\n1. In line 28, why the map $M_p(u)$ has to be defined over $[-1,1]^p$? It is a bit weird to me because $u=\\mathbb{E}[\\beta]$ but $\\beta$ can be outside the interval.\n\n2. The authors indeed adopt the parametrization based on exponential tilts $\\pi^{(h(u,d),d)}$ on the basis of NMF. Does $\\pi^{(h(u,d),d)}$ conceptually cover all candidates of continuous distributions? If not, is it the standard setup to facilitate analysis in variational inference?\n\nLIMITATIONS:\nSee the weakness stated above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide an asymptotic characterization of naive mean field (NMF) for linear regression under certain priors (e.g., those which guarantee strong convexity).  Under these priors, this characterization can be used to demonstrate the sub-optimality of NMF (asymptotically).\n\nThis is primarily a theoretical work, but some experimental observations of the fixed-point scheme are provided.\n\nSTRENGTHS:\nThe approach seems novel and fills gaps in the existing literature.  \n\nThe high-level ideas were easy to follow (though I didn't check all of the proofs in detail).\n\nThe authors suggest lots of limitations which may lead to further investigation.\n\nWEAKNESSES:\nIt isn't immediately clear what practical utility this work would have.  This isn't necessarily required in a theoretical work, but it would bolster the case for acceptance.\n\nAlthough the main ideas were novel (to me at least), the conditions required to draw the conclusions are difficult to assess (see questions below).  \n\nThe writing needs a lot of work.  Some notation is undefined (though I could guess what it meant).\n\nQUESTIONS:\n- Could you provide examples of possible practical applications of this work?  The approach seems novel and interesting to me, but I can't ever recall using NMF and thinking that it would yield a highly accurate answer.\n\n- Could you comment on the conditions in Theorem 1 -- are these likely to be true in many instances?  Let's suppose that neither condition is satisfied, is it possible to conclude anything from the fixed point iteration?\n\n\nMinor typos:\n- \"i.e.\" and \"e.g.\" must always be followed by commas.\n- \"using product\" -> \"using a product\"\n- \"when feature\" -> \"when the feature\"\n-  \"progresses were\" -> \"progress was\"\n- Stylistically, it is better not to use a citation as a noun.  You should use So and so et al. [citation] instead.\n... too many more to log here.  Serious proofreading is recommended, and please, please, please remove the phrase \"not gonna be correct.\"\n\nLIMITATIONS:\nYes, the authors clearly describe the limitations of the current work, which could suggest interesting avenues for future research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Naive Mean-Field Approximation (NMF) for the posterior distribution of a simple linear regression problem with Gaussian design. It provides an exact asymptotic result for the joint distribution between the NMF approximation $\\hat u$ and the regression vector $\\beta^\\star$. This asymptotic is then used to provide results on the log-partition function associated to the NMF, the mean square error between the NMF estimator and the ground truth, as well as the associated confidence intervals around $\\beta^\\star$. Those asymptotics depend on a two-dimensional fixed-point equation, which can only be numerically solved; nevertheless, the authors compare this numerical solution with the full posterior distribution, to quantify how the NMF approximates specific quantities of interest of the posterior.\n\nSTRENGTHS:\nThis paper is an interesting contribution to the study of Naive Mean-Field estimators. It complements well the results of Sen and Mukherjee on the data-rich setting, by showing that the NMF approximation is not even correct to the first order in the proportional regime. The methods used are fairly standard in the field of exact asymptotics, which is acknowledged in the paper, but they are applied to obtain relevant insights on a new problem. As far as I checked, these methods are applied rigorously and the paper is correct.\n\nWEAKNESSES:\nSome of the exposition feels however a bit too obscure: it feels like some very important points (e.g. the definition of the NMF objective and its explanation, or the main theorem) are drowned in very technical considerations (e.g. almost a full page of conditions to ensure the convexity of $F$, or the computation part of the CGMT). I think the paper could achieve a better balance between technical content and explanation, to reach communities that may be familiar with either exact asymptotics or Bayesian approximations, but not both.\n\nSome explanations that would maybe need to be highlighted include:\n- why you can replace (3) by the objective in Definition 2, and why you need the $\\frac{d_i}2\\beta_i^2$ correction term for,\n- how the CGMT also imply that the minimizers for both objectives are close (i.e. the part below Lemma 13)\n\nMinor remarks:\n- l.38: unfinished sentence ?\n- Eq. (3): $\\sum_{i=1^p}$\n- your log-normalizing constants are sometimes denoted with $Z$ (e.g. eq (2), (4)) and sometimes with $\\mathcal{Z}$ (most of the time)\n- contrary to what is claimed l.203-207, the expressions for $\\mathcal Z_p$ and related quantities are absent from the Supplementary Material\n- Eq. (9): I'm not sure redefining $F$ here is necessary\n- Fig. 5: there is no right panel\n\nQUESTIONS:\n- How well do you expect the conditions of Theorem 1 to hold ? I read your Appendix C, so I understand why it would be complicated, but do you have at least a few examples of unicity ?\n- What's the use of Corollary 1 ? Isn't it strictly weaker than the conditions in l.120-124 ?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the naive mean field approximation (NMF) in high dimensional linear regression in the limit where p/n goes to a constant. While the NMF is known to be consistent when n >>p, the authors show that in the high-dimensional case, this is not true. As a result, this paper gives nice theoretic evidence for empirical observations showing that the NMF is no longer accurate in high dimensional regimes.  Moreover, the paper also discusses the consequences on the uncertainty quantification.\n\nSTRENGTHS:\nThe research question is important and the results should definitely be published at a top ML conference.\n\nThe paper is well-written and easily understandable.  The proof sketch is clear and the theoretical results are nicely illustrated with figures.\n\nWEAKNESSES:\nWhile I recognize that this paper meets the standards for NeurIPS, my reluctance in awarding a higher grade stems from two significant limitations:\n\n1) The proofs' methodological contribution is rather limited. Despite the elegant use of the CGMT, which is commonly seen within the high-dimensional statistics literature, the paper doesn't break new ground in terms of methodology. Although the paper commendably presents the proof in a digestible manner, and the choice of the CGMT is likely the most efficient way to prove the result, this doesn't warrant a significant methodological contribution deserving of a high grade (8+).\n\n2) The insights gained from the results are fairly limited and expected. This is also generally a shortcoming of proofs relying on the CGMT. While such proofs usually yield the precise  asymptotic limit, the resulting theorem statements  are challenging to interpret.\n\nQUESTIONS:\nI am currently missing an intuition in the paper for why the NMF approximation is loose in the high dimensional limit. Can the authors elaborate more on this point?\n\nTo me, the main intuition arises from the fact that when $n>>p$, the eigenvalues of $1/n X'X$ concentrate around $1$, which is clearly not the case when $p/n \\to c$. As a result, when $n>>p$, the measure $\\mu$ is approximately symmetric around its mode (and thus a NMF approximation should be accurate). On the other hand, when $p/n \\to c$, the measure $\\mu$ is \"elliptic\" and strongly depends on the small eigenvectors of $X'X$. However, these eigenvectors cannot be captured by the NMF approximation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}