SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {h \in \mathbb {R}} \operatorname {V a r} _ {X \sim \pi_ {\tilde {q}, \Delta^ {2}}} (X) <   \sigma^ {2} \tag {8}
$$
Figure 2: These two figures demonstrate the existence of a gap between  $\lim_{p\to \infty}(\mathcal{Z}_p) / p$  and  $\lim_{p\to \infty}(\log \mathcal{Z}_p^{\mathrm{NMF}}) / p$  when  $\pi = \pi^{\star}$  is a Gaussian Spike and Slab distribution. The left panel features the observation that the gap gets smaller as  $q$  (prior sparsity) increases, while the right panel shows as  $\alpha \coloneqq n / p$  gets large, the gap seems to converge to 0, which is consistent with the results established in [16] when  $p = o(n)$ .
where  $\pi_{\tilde{q},\tilde{\Delta}^2}$  is again a Gaussian spike and slab mixture,
$$
\pi (\mathrm {d} x) = \tilde {q} \delta (x) + \frac {1 - \tilde {q}}{\sqrt {2 \pi \tilde {\Delta} ^ {2}}} e ^ {- \frac {x ^ {2}}{2 \tilde {\Delta} ^ {2}}} \mathrm {d} x
$$
$$
w i t h \quad \tilde {q} = \frac {q}{q + (1 - q) (1 + \Delta^ {2} / \sigma^ {2}) ^ {- 1 / 2}} a n d \tilde {\Delta} ^ {2} = \frac {\sigma^ {2} \Delta^ {2}}{\sigma^ {2} + \Delta^ {2}}.
$$
Then,  $F(u)$  is strongly convex. In addition, one easier-to-check sufficient condition for (8) is
$$
\left(1 + \frac {2 q}{1 - q} \sqrt {1 + \frac {\Delta^ {2}}{\sigma^ {2}}}\right) \frac {\Delta^ {2}}{\sigma^ {2} + \Delta^ {2}} <   1. \tag {9}
$$
Remark 2. It is easy to see that for large enough  $\sigma$  ( $q$  and  $\Delta$  fixed), or small enough  $q$  ( $\Delta$  and  $\sigma$  fixed), or small enough  $\Delta$  ( $q$  and  $\sigma$  fixed), (9) is always satisfied. In other words,  $F(\cdot)$  is strongly convex for low signal-to-noise ratio or high temperature in physics parlance.
3 Log normalizing constant: sub-optimality of NMF
As alluded, as implications of Theorem 1, we develop asymptotics of both  $\log \mathcal{Z}_p^{\mathrm{NMF}}$  and mean square error (MSE) of the NMF point estimator  $\hat{u}$  in terms of  $(b^{\star},\tau^{\star})$ .
Corollary 2 (MSE). When conditions of Corollary 1 hold, as  $n,p\to \infty$
$$
\frac {1}{p} \| \hat {u} - \beta^ {\star} \| ^ {2} \xrightarrow {P} \mathbb {E} _ {(B, Z) \sim \pi^ {\star} \otimes \mathcal {N} (0, 1)} \left[ \left(\eta \left(\tau^ {\star} Z + B, \frac {\tau^ {\star} \sigma^ {2}}{b ^ {\star}}\right) - B\right) ^ {2} \right] = \alpha (\tau^ {\star 2} - \sigma^ {2}).
$$
Corollary 3 (Log normalizing constant). When conditions of Corollary 1 hold, as  $n,p\to \infty$
$$
- \frac {1}{p} \log \mathcal {Z} _ {p} ^ {N M F} = \frac {1}{p} \left[ M _ {p} (\hat {u}) - \sum_ {i = 1} ^ {p} c (0, d _ {i}) \right] \xrightarrow {P} \frac {\alpha b ^ {\star 2}}{2 \sigma^ {2}} + \mathbb {E} F \left(\eta \left(B + \tau^ {\star} Z, \tau^ {\star} / b ^ {\star}\right)\right) - c (0, 1 / \sigma^ {2}).
$$
Though all our main theorems and corollaries apply to the case when  $\pi^{\star} \neq \pi$ , for simplicity and clarity, from now on, we only consider the "nicest" setting, i.e., when assumptions of Corollary 1 are satisfied and in addition  $\pi = \pi^{\star}$ . By doing so, we would like to convey that even if there were no model mismatch at all, NMF still would not be "correct".
Concentration and limiting values of both the optimal Bayesian mean square error (i.e.,  $\mathbb{E}_{\mu} \| \beta - \mathbb{E}[\beta^{\star}|X,y]\|^{2} / p$ ) and the actual log-normalizing constant were conjectured and rigorously established
Figure 3: These two figures show that estimated credible regions given by NMF do not achieve the nominal coverage (95%) when  $\pi = \pi^{\star}$  is a Gaussian Spike and Slab distribution. Recall that  $\alpha = n / p$  and please see Lemma 6 for exact definitions of the hyper-parameters  $q$  and  $\Delta^2$ .
under additional regularity conditions, which provides us the "correct answers" to compare with. Please see [2, 20].
Please see Figure 2 for numerical evaluations of Corollary 3, which suggests the bound in (4) is not tight for Gaussian Spike and Slab prior. Since, in general, both  $F(\cdot)$  and  $\eta(\cdot, \cdot)$  lack analytical forms, it is hard to provide a universal guarantee on whether (5) has a unique optimizer or the fixed point equation (7) has a unique solution. In fact, our numerical experiments suggest it is possible for (7) to have multiple fixed points. Therefore, how to exactly realize and evaluate the asymptotic predictions in these two corollaries (so as Corollary 4 in the next section) is challenging in general and can only be done in a case-by-case basis and usually involves numerically solving (7). In light of this observation, we use the Gaussian Spike and Slab prior as defined in Lemma 6 for presentation purposes. Since it is both non-trivial and of practical interest, though, we do emphasize that the same framework and workflow also apply to other priors. Without loss of generality, we also take  $\sigma^2 = 1$ . This choice renders Figure 2 and 3 in the next section. Details of how to generate these plots are deferred to the Supplementary Material.
4 Uncertainty quantification: the average coverage rate
To study uncertainty quantification properties of NMF approximation, we consider the average coverage rate of symmetric Bayesian credible regions (of level  $1 - \zeta$ ) suggested by the NMF distributions, i.e.,  $R_{p,\zeta} := \frac{1}{p}\sum_{i=1}^{p}\mathbb{1}_{\{\beta_i^\star \in [\hat{q}_{i,\zeta/2}, \hat{q}_{i,1 - \zeta/2}]\}}$ , where  $\hat{q}_{i,t}$  is the  $t$ -th quantile of  $\pi^{(h(\hat{u}_i,d_i),d_i)}$ . In order to study asymptotic behavior of  $R_{p,\zeta}$ , we define an  $(m(\pi),M(\pi)) \times S \to \{0,1\}$  indicator function
$$
\psi_ {\zeta} (u _ {0}, \beta_ {0}) = \mathbb {1} _ {\left\{\beta_ {0} \in \left[ q _ {\pi (h (u _ {0}, 1 / \sigma^ {2}), 1 / \sigma^ {2})}, _ {\zeta / 2}, q _ {\pi (h (u _ {0}, 1 / \sigma^ {2}), 1 / \sigma^ {2})}, _ {1 - \zeta / 2} \right] \right\}}.
$$
The following corollary of Theorem 1 establishes the asymptotic convergence of  $R_{p,\zeta}$ . Numerically evaluating it for the Gaussian Spike and Slab prior renders Figure 3, which shows NMF credible regions can not achieve the nominal coverage, in this case,  $95\%$ , and also provides an exhibition of how large the gaps are for different hyper-parameters.
Corollary 4. Suppose conditions of Corollary 1 hold. In addition, assume the quantile function of  $\pi$  is continuous. Then as  $n,p\to \infty$
$$
R _ {p, \zeta} \xrightarrow {P} \mathbb {E} _ {(B, Z) \sim \pi^ {\star} \otimes \mathcal {N} (0, 1)} \left[ \psi_ {\zeta} \left(\eta \left(\tau^ {\star} Z + B, \frac {\tau^ {\star} \sigma^ {2}}{b ^ {\star}}\right), B\right) \right].
$$
On the other hand, based on the asymptotic joint distribution of  $\hat{u}$  and  $\beta^{\star}$  as stated in Corollary1, we can in fact identify a strategy of constructing asymptotically exact Bayesian credible regions based on  $\hat{u}$ . Let  $q_{t}(x)$  be the  $t$ -th quantile of conditional distribution of  $B$  given  $\eta (\tau^{\star}Z + B,\tau^{\star}\sigma^{2} / b^{\star}) = x$ . This way, the following Corollary ensures  $[q_{\zeta /2}(\hat{u}_i),q_{1 - \zeta /2}(\hat{u}_i)]$  is asymptotically of at least  $1 - \zeta$  coverage.
Corollary 5. Suppose conditions of Corollary 4 hold, then for any  $\varepsilon >0$
$$
\lim _ {p \to \infty} \mathbb {P} \left(\frac {1}{p} \sum_ {i = 1} ^ {p} \mathbb {1} _ {\{\beta_ {i} ^ {*} \in [ q _ {\zeta / 2} (\hat {u} _ {i}), q _ {1 - \zeta / 2} (\hat {u} _ {i}) ] \}} <   1 - \zeta - \varepsilon\right) = 0.
$$
5 Discussion: Extensions and Limitations
In order to provide some intuition on why the NMF approximation is loose in the current setting, it is worth noting that in comparison with the proportional asymptotics regime we consider here, positive results of NMF for high-dimensional linear regression were recently established in [16] when  $p = o(n)$ . Using terminology from Austin [1], Mukherjee and Sen [16] (when restricted to designs with iid Gaussian features) essentially proved, when  $p / n \to 0$ , the eigenvalue concentration behavior of  $X^T X$  leads to the Hamiltonian being of "low complexity". On the other hand, when  $p = \Theta(n)$ ,  $\mathrm{tr}(A^2) \neq o(p)$ , where  $A = A(X)$  is defined as the off-diagonal part of  $X^T X$ , which violates [16, Equation (5)]. Roughly speaking, when the eigenstructure of  $A$  is not "dominated" by a few top eigenvalues, the Hamiltonian can not be covered by an efficient net and thus is not of "low complexity". Please see [1, 16, 3] for more details.
We want to be clear about the fact that, technically, we did not "prove" the sub-optimality of NMF. Instead, we rigorously derived asymptotic characterizations of NMF approximation through the solution of a fixed point equation. But this fixed point equation can only be solved numerically on a case-by-case basis and is not guaranteed a unique solution. All our plots are based on iteratively solving the fixed point equation. As a matter of fact, for instance, when  $q$  is close to 1 for the Gaussian Spike and Slab prior we considered, the fixed point equation is clearly not converging to the right fixed point, as demonstrated in the Supplementary Material. It could also just not converge for very small  $\alpha$ . Nevertheless, all the plots we are showing in the main text are backed by a numerical simulation using simple gradient descent to optimize the NMF objective, i.e.  $\inf M_p(u)$ , for  $n = 8000$ . All in all, it is probably more accurate to say we provided a tool for establishing the sub-optimality of NMF for a general class of priors rather than proving it for good.
Another obvious limitation is we can only handle priors that guarantee convexity of the KL-divergence term in terms of the mean parameters. Though it is indeed a broad class of distributions covering some of the most commonly used symmetric priors (e.g., Gaussian, Laplace, and so on), little is known about the asymptotic behavior of NMF when the convexity assumption is violated.
We note that, in theory, in order to carry out the analysis using CGMT, the additive noise  $\epsilon$  as defined in (1) does not have to be Gaussian. Instead, as long as it has log-concave density, the same proof idea applies, though we intentionally chose to stick with Gaussian noise as it renders much cleaner results and a more comprehensive presentation. In addition, we expect stronger uniform convergence results (e.g., uniform in  $\sigma^2$ ) could also be established, which can be crucial for applications like hyperparameters selection. Please see [15] for an example in which results of this flavor were obtained.
6 Proof strategy
This section gives a proof outline of Theorem 1. More details can be found in the Supplementary Material. Replacing all  $d_{i}$ 's in  $M_p$  with  $\mathbb{E}d_{i} = 1 / \sigma^{2}$ , we define  $N_{p}$  as
$$
N _ {p} (u) = \frac {1}{2 \sigma^ {2}} \| Y - X u \| _ {2} ^ {2} + \sum_ {i = 1} ^ {p} \left[ G (u _ {i}, 1 / \sigma^ {2}) - \frac {u _ {i} ^ {2}}{2 \sigma^ {2}} \right] = \frac {1}{2 \sigma^ {2}} \| Y - X u \| _ {2} ^ {2} + \sum_ {i = 1} ^ {p} F (u _ {i}).
$$
Lemma 7. Let  $\hat{u}_N\coloneqq \arg \min_u[N_p(u)]$ . Then for some  $C_s\in \mathbb{R}^+$ , as  $n,p\to \infty$
$$
\mathbb {P} \left(\frac {1}{p} \max (\| \hat {u} \| ^ {2}, \| \hat {u} _ {N} \| ^ {2}) > (1 + C _ {s}) s _ {2}\right) \longrightarrow 0.
$$
Lemma 8. For any  $\varepsilon >0$ , as  $n,p\to \infty$ , with  $C_s$  as defined in Lemma 7,
$$
\mathbb {P} \left(\frac {1}{p} \sup  _ {\| u \| ^ {2} / p \leq (1 + C _ {s}) s _ {2}} \left| \sum_ {i = 1} ^ {p} \left[ G \left(u _ {i}, 1 / \sigma^ {2}\right) - \frac {u _ {i} ^ {2}}{2 \sigma^ {2}} \right] - \left[ G \left(u _ {i}, d _ {i}\right) - \frac {d _ {i} u _ {i} ^ {2}}{2} \right] \right| > \varepsilon\right) \longrightarrow 0. \tag {10}
$$
According to Lemma 8 and 7,  $N_{p}(\cdot)$  and  $M_{p}(\cdot)$  are with high probability uniformly close. Thus, from now on, we focus on using Gaussian comparison to analyze  $\hat{u}_N$  and  $N_{p}(\hat{u}_{N})$  in place of  $\hat{u}$  and  $M_p(\hat{u})$ . Since  $F(\cdot)$  is strongly convex,  $\hat{w} \coloneqq \hat{u}_N - \beta^\star$  is the unique minimizer of
$$
L (w) := \frac {1}{2 n} \| X w - \epsilon \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} \left(F (w _ {i} + \beta_ {i} ^ {\star}) - F (\beta_ {i} ^ {\star})\right).
$$
By introducing a dual vector  $s$ , we get
$$
\min _ {w} L (w) = \min _ {w \in \mathbb {R} ^ {p}} \max _ {s \in \mathbb {R} ^ {n}} \frac {1}{n} s ^ {T} (X w - \epsilon) - \frac {1}{2 n} \| s \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} \left(F (w _ {i} + \beta_ {i} ^ {\star}) - F (\beta_ {i} ^ {\star})\right).
$$
By CGMT (see for instance [22, Theorem 3.3.1] or [15, Theorem 5.1]), it suffices now to study
$$
\min _ {w \in \mathbb {R} ^ {p}} \max _ {s \in \mathbb {R} ^ {n}} \frac {1}{n ^ {3 / 2}} \| s \| g ^ {T} w + \frac {1}{n ^ {3 / 2}} \| w \| h ^ {T} u - \frac {1}{n} s ^ {T} \epsilon - \frac {1}{2 n} \| s \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} (F (w _ {i} + \beta_ {i} ^ {\star}) - F (\beta_ {i} ^ {\star}))
$$
where  $g \sim \mathcal{N}(0, I_p)$  and  $h \sim \mathcal{N}(0, I_n)$  and they are independent. Note that the min and max can be flipped due to convex-concavity. By optimizing with respect to  $s / \|s\|$  and introducing  $\sqrt{\frac{\|w\|^2}{n} + \sigma^2} = \min_{\tau \geq \sigma} \left\{ \frac{\frac{\|w\|^2}{n} + \sigma^2}{2\tau} + \frac{\tau}{2} \right\}$ , it can be further reduced to
$$
\operatorname * {m a x} _ {b \geq 0} \operatorname * {m i n} _ {\tau \geq \sigma} \frac {b}{2} (\frac {\sigma^ {2}}{\tau} + \tau) - \frac {b ^ {2}}{2} + \frac {1}{\alpha} \operatorname * {m i n} _ {w \in \mathbb {R} ^ {p}} \sum_ {i = 1} ^ {p} \left[ \frac {1}{p} \left\{\frac {b}{2 \tau} w _ {i} ^ {2} - b g _ {i} w _ {i} + \sigma^ {2} F (w _ {i} + \beta_ {i} ^ {\star}) - \sigma^ {2} F (\beta_ {i} ^ {\star}) \right\} \right].
$$
Under minor regularity conditions, as  $n,p\to \infty$  , it converges to
$$
\max  _ {b \geq 0} \min  _ {\tau \geq \sigma} \frac {b}{2} \left(\frac {\sigma^ {2}}{\tau} + \tau\right) - \frac {b ^ {2}}{2} + \frac {1}{\alpha} \mathbb {E} _ {B, Z} \min  _ {w \in \mathbb {R}} \left\{\frac {b}{2 \tau} w ^ {2} - b Z w + \sigma^ {2} F (w + B) - \sigma^ {2} F (B) \right\}
$$
with  $(B,Z)\sim \pi^{\star}\otimes \mathcal{N}(0,1)$ , which is how we got  $\phi (\cdot ,\cdot)$  as in (5). Furthermore, by differentiating  $\phi (b,\tau)$  with respect to  $\tau$  and  $b$ , we arrive at the fixed point equation in Lemma 2. Last but not least, note that  $\arg \min_{w}\left\{\frac{b}{2\tau} w^{2} - bZw + \sigma^{2}F(w + B)\right\} = \eta (\tau Z + B,\tau \sigma^{2} / b) - B$ , which explains why the joint empirical distribution of  $(\hat{w}_i,\beta_i^\star)$ 's converges to the law of  $\left(\eta (\tau^{\star}Z + B,\tau^{\star}\sigma^{2} / b^{\star}) - B,B\right)$ . Finally, we note that similar proof arguments were made in [15, 22].
I am grateful to Subhabrata Sen for some very insightful conversations and encouragement throughout the process. Along with Subhabrata Sen and three anonymous referees, they also made valuable suggestions about earlier versions of this paper. The author was partially supported by a Harvard Dean's Competitive Fund Award to Subhabrata Sen and NSF DMS CAREER 2239234.
[1] T. Austin. The structure of low-complexity gibbs measures on product spaces. The Annals of Probability, 47(6):4002-4023, 2019.
[2] J. Barbier, N. Macris, M. Dia, and F. Krzakala. Mutual information and optimality of approximate message-passing in random linear estimation. IEEE Transactions on Information Theory, 66(7):4270-4303, 2020.
[3] A. Basak and S. Mukherjee. Universality of the mean-field for the potts model. Probability Theory and Related Fields, 168:557-600, 2017.
[4] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859-877, 2017.
[5] R. S. Ellis and J. L. Monroe. A simple proof of the ghs and further inequalities. Communications in Mathematical Physics, 41(1):33-38, 1975.
[26] R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010.
[27] M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.
[28] F. Zhang and C. Gao. Convergence rates of variational posterior distributions. 2020.
A Technical lemmas and basic facts
Lemma 9. Let  $\dot{c}(h, d) \coloneqq \frac{\partial c}{\partial h}(h, d)$  and  $\ddot{c}(h, d) \coloneqq \frac{\partial^2 c}{\partial h^2}(h, d)$ . We have, for  $u \in (m(\pi), M(\pi))$  and  $d \in \mathbb{R}$ ,
$$
\begin{array}{l} \frac {\partial G}{\partial u} (u, d) = h (u, d), \frac {\partial G}{\partial \bar {d}} = \frac {1}{2} \int_ {S} z ^ {2} \mathrm {d} \pi^ {(h (u, d), d)} (z) - \frac {1}{2} \int_ {S} z ^ {2} \mathrm {d} \pi^ {(0, d)} (z). \\ \frac {\partial^ {2} G}{\partial^ {2} u} (u, d) = \frac {1}{\ddot {c} (h (u , d) , d)} = \frac {1}{\mathrm {V a r} _ {X \sim \pi^ {(h (u , d) d)}} (X)} > 0. \\ \end{array}
$$
Lemma 10 (von Neumann's minimax theorem, [25]). Let  $S_w \subset \mathbb{R}^n$  and  $S_s \subset \mathbb{R}^m$  be compact convex sets. If  $f: S_w \times S_s \to \mathbb{R}$  is a continuous function that is convex-concave, i.e.,  $f(\cdot, s): S_w \to \mathbb{R}$  is convex for fixed  $s$ , and  $f(w, \cdot): S_s \to \mathbb{R}$  is concave for fixed  $w$ . Then we have that
$$
\min  _ {w \in S _ {w}} \max  _ {s \in S _ {s}} f (w, s) = \max  _ {s \in S _ {s}} \min  _ {w \in S _ {w}} f (w, s).
$$
Theorem 2 (CGMT, [23, 22, 15]). Let  $S_w \subset \mathbb{R}^p$  and  $S_s \subset \mathbb{R}^n$  be two compact sets and let  $Q: S_w \times S_s \to \mathbb{R}$  be a continuous function. Let  $G = (G_{ij})_{1 \leq i \leq n, 1 \leq j \leq p} \stackrel{iid}{\sim} \mathcal{N}(0,1)$ ,  $g \sim \mathcal{N}(0,I_p)$ ,  $h \sim \mathcal{N}(0,I_n)$  be independent standard Gaussian vectors. Denote
$$
\begin{array}{l} \Phi (G) = \min_{w\in S_{w}}\max_{s\in S_{s}}s^{T}Gw + Q(w,s), \\ \Psi (g,h) = \min_{w\in S_{w}}\max_{s\in S_{s}}\| s\| g^{T}w + \| w\| h^{T}s + Q(w,s). \\ \end{array}
$$
1. For all  $t\in \mathbb{R}$
$$
\mathbb {P} \left(\Phi (G) \leq t\right) \leq 2 \mathbb {P} (\Psi (g, h) \leq t).
$$
2. If both  $S_w$  and  $S_s$  are convex and if  $Q(\cdot, \cdot)$  is convex-concave, then for all  $t \in \mathbb{R}$ ,
$$
\mathbb {P} \left(\Phi (G) \geq t\right) \leq 2 \mathbb {P} (\Psi (g, h) \geq t).
$$
Remark 3. The most important message of this theorem is essentially whenever  $\Psi(g, h)$  concentrates around a certain value  $t$ ,  $\Phi(G)$  will also concentrate around  $t$ , assuming  $Q(\cdot, \cdot)$  is convex-concave.
Proof of Lemma 1 and 9 can be found in for instance [16].
B.1 Convexity of  $F(\cdot)$
Proof of Lemma 3. We only prove part (1) here, as proof of part (2) is almost exactly the same. For any  $h,d\in \mathbb{R}^+$ , by GHS inequality [6, Equation 1.4],
$$
\frac {\partial \left[ \operatorname {V a r} _ {B \sim \pi^ {(h , d)}} (B) \right]}{\partial h} = \mathbb {E} [ B ^ {3} ] - 3 \mathbb {E} B \mathbb {E} [ B ^ {2} ] + 2 \left(\mathbb {E} B\right) ^ {3} \stackrel {\mathrm {G H S}} {\leq} 0,
$$
Together with the assumption that  $V$  is even, we have for any  $h \in \mathbb{R}$  and  $d \geq 0$ ,
$$
\operatorname {V a r} _ {B \sim \pi^ {(h, d)}} (B) \leq \operatorname {V a r} _ {B \sim \pi^ {(0, d)}} (B).
$$
Consider now a family of parametric distributions  $\{\mathcal{P}_{\theta}:\theta \geq 0\}$  as a generalization of  $\pi^{(0,d)}$ , with
$$
\frac {\mathrm {d} \mathcal {P} _ {\theta}}{\mathrm {d} x} (x) \propto \exp (- \theta V (x)) \exp \left(- d x ^ {2} / 2\right).
$$
Note that  $\mathcal{P}_{\theta = 1} = \pi^{(0,d)}$ . Since  $V(\cdot)$  is even and increasing,
$$
\begin{array}{l} \operatorname {V a r} _ {B \sim \pi^ {(0, d)}} (B) = \operatorname {V a r} _ {S \sim \mathcal {P} _ {\theta = 1}} (S) \leq \operatorname {V a r} _ {S \sim \mathcal {P} _ {\theta = 0}} (S) \\ = \frac {\int_ {\mathbb {R}} z ^ {2} e ^ {- d z ^ {2} / 2} \mathrm {d} z}{\int_ {\mathbb {R}} e ^ {- d z ^ {2} / 2} \mathrm {d} z} \\ = \frac {1}{d} \frac {\int_ {\mathbb {R}} z ^ {2} e ^ {- z ^ {2} / 2} \mathrm {d} z}{\int_ {\mathbb {R}} e ^ {- z ^ {2} / 2} \mathrm {d} z} \\ \leq \frac {1}{d} \operatorname {V a r} _ {S \sim \mathcal {N} (0, 1)} (S) = \frac {1}{d}, \\ \end{array}
$$
which ensures  $\operatorname{Var}_{B \sim \pi^{(h(u,1/\sigma^2),1/\sigma^2)}}(B) \leq \sigma^2$  and therefore  $\frac{\mathrm{d}^2F}{\mathrm{d}u^2}(u) \geq 0$  by (11). Note that as long as  $\pi$  is a valid probability distribution,  $F(\cdot)$  is not only convex but always strongly convex, as  $\operatorname{Var}_{B \sim \pi^{(h(u,1/\sigma^2),1/\sigma^2)}}(B) = \sigma^2$  if and only if  $V(\cdot)$  is a constant function and the support of  $\pi$  is the whole real line.
The same proof idea also applies to Lemma 4; therefore, we omit its proof to avoid redundancy.
Proof of Lemma 5. The conclusion follows by noting
$$
\frac {\mathrm {d} ^ {2} F}{\mathrm {d} u ^ {2}} (u) = \frac {1}{\operatorname {V a r} _ {B \sim \pi^ {(h (u , 1 / \sigma^ {2}) , 1 / \sigma^ {2})}} (B)} - \frac {1}{\sigma^ {2}} > 0, \tag {11}
$$
as  $\pi^{(h,d)}$  is a distribution on  $[-a,a]$  and thus its variance is at most  $a^2$ , which is assumed to be smaller than  $\sigma^2$ .
For Lemma 6, since  $\operatorname{Var}_{B \sim \pi^{(h,d)}}(B)$  can be analytically computed for the Gaussian Spike and Slab prior, its proof is nothing but elementary calculation and then checking for (11).
B.2 Replacing  $d_{i}$  with  $\mathbb{E}d_{i}$
Proof of Lemma 7. We focus on only  $\| \hat{u} \|$  since almost exactly the same argument also applies to  $\hat{u}_N$ . We first collect a few high-probability claims, proofs of which are just direct applications of basic standard random matrix results (see, for instance, [26]).
1. There exist positive constants  $C_1$  and  $C_2$  (only depend on  $\alpha$ ), such that for any  $\varepsilon > 0$ ,  $S_1 := \{|\lambda_{\max}(X^T X) - C_1| < \varepsilon\}$  and  $S_2 := \{|\lambda_{\min}(X^T X) - C_2| < \varepsilon\}$  are both of high probability.
2. Recall the additive noise  $\epsilon \sim \mathcal{N}(0, \sigma^2 I_n)$ . For any  $\varepsilon > 0$ ,  $S_3 \coloneqq \{\| \epsilon \|^2 / n - \sigma^2 | < \varepsilon\}$  is of high probability.
3. For any  $\varepsilon > 0$ ,  $S_4 = \left\{\left|\epsilon^T X\beta^\star / p\right| < \varepsilon \right\}$  is of high probability.
Let  $S_0 = S_1 \cap S_2 \cap S_3 \cap S_4$ , which is again an event of approaching 1 probability. Note that since the empirical distribution of  $\beta_i^\star$ 's converge in  $L_2$  to  $\pi^\star$ , one has  $\| \beta^\star \|^2 < 1.01ps_2$  for large enough  $p$ . When  $S_0$  happens, if  $\| u \|^2 / p > (1 + C_s)s_2$  (with  $C_s > 0$  to be chosen later, but large enough such that  $\| Xu \| > \| Y \|$ ),
$$
\begin{array}{l} N _ {p} (u) \geq \frac {1}{2 \sigma^ {2}} \| Y - X u \| ^ {2} \geq \frac {1}{2 \sigma^ {2}} \left(\| X u \| - \| Y \|\right) ^ {2} \geq \frac {p}{2 \sigma^ {2}} \left[ \sqrt {(C _ {2} - \varepsilon) (1 + C _ {s}) s _ {2}} - \| X \beta^ {\star} + \epsilon \| / p \right] ^ {2} \\ \geq \frac {p}{2 \sigma^ {2}} \left[ \sqrt {(C _ {2} - \varepsilon) (1 + C _ {s}) s _ {2}} - \sqrt {2 (C _ {1} + \varepsilon) \cdot 2 s _ {2} + 2 \alpha (\sigma^ {2} + \epsilon)} \right] ^ {2}. \\ \end{array}
$$
$$
N _ {p} \left(\vec {0}\right) = \frac {1}{2 \sigma^ {2}} \| Y \| ^ {2} \leq \frac {p}{2 \sigma^ {2}} \left[ (C _ {1} \varepsilon) \cdot 2 p s _ {2} + \alpha (\sigma^ {2} - \varepsilon) + 2 \varepsilon \right].
$$
Upon  $C_s$  being large enough, we have  $N_p(u) > N_p(\vec{0})$  for any  $u$  such that  $\| u \|^2 / p > (1 + C_s)s_2$ . Therefore,  $\| \hat{u}_N \|^2 / p < (1 + C_2)s_2$  on  $S_0$ .
Proof of Lemma 8. If  $S = [-1,1]$ , by Lemma 9,  $\left|\frac{\partial G(u,d)}{\partial d}(u,d)\right| \leq \frac{1}{2}$  for any  $u,d$ , thus