SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaze Qiu
jiazeqiu@g.harvard.edu
The Naïve Mean Field (NMF) approximation is widely employed in modern Machine Learning due to the huge computational gains it bestows on the statistician. Despite its popularity in practice, theoretical guarantees for high-dimensional problems are only available under strong structural assumptions (e.g., sparsity). Moreover, existing theory often does not explain empirical observations noted in the existing literature.
In this paper, we take a step towards addressing these problems by deriving sharp asymptotic characterizations for the NMF approximation in high-dimensional linear regression. Our results apply to a wide class of natural priors and allow for model mismatch (i.e., the underlying statistical model can be different from the fitted model). We work under an iid Gaussian design and the proportional asymptotic regime, where the number of features and the number of observations grow at a proportional rate. As a consequence of our asymptotic characterization, we establish two concrete corollaries: (a) we establish the inaccuracy of the NMF approximation for the log-normalizing constant in this regime, and (b) we provide theoretical results backing the empirical observation that the NMF approximation can be overconfident in terms of uncertainty quantification.
Our results utilize recent advances in the theory of Gaussian comparison inequalities. To the best of our knowledge, this is the first application of these ideas to the analysis of Bayesian variational inference problems. Our theoretical results are corroborated by numerical experiments. Lastly, we believe our results can be generalized to non-Gaussian designs and provide empirical evidence to support it.
The Naive Mean Field (NMF) approximation is widely employed in modern Machine Learning as an approximation to the actual intractable posterior distribution. The NMF approximation is attractive as (a) it bestows huge computational gains, and (b) it is naturally interpretable and can provide access to easily interpretable summaries of the posterior distribution e.g., credible intervals. However, these two advantages may be overshadowed by the following limitations: (a) it is a priori unclear whether this strategy of using a product distribution as a proxy for the true posterior will result in a "good" approximation, and (b) it has been empirically observed that NMF often tends to be significantly over-confident, especially when the feature dimension  $p$  is not negligible compared to the sample size  $n$ . In the traditional asymptotic regime ( $p$  fixed and  $n \to \infty$ ), significant progress was made in understanding these two problems for different statistical models, see for instance [8] and references therein. On the other hand, in the complementary high-dimensional regime where both  $n$  and  $p$  are growing, Ghorbani et al. [7] recently established an instability result for the topic model under the
Figure 1: These three figures serve as a visual summary of our main results when the Gaussian Spike and Slab prior is adopted, i.e., NMF does not provide up to leading order correct approximation to the log-normalizing constant (left), and the estimated credible regions suggested by the NMF distribution do not achieve the nominal coverage (middle), even when NMF could achieve close to optimal MSE. Please see Lemma 6 for definitions of the Gaussian Spike and Slab prior and the hyper-parameters  $q$  and  $\Delta^2$ .
proportional asymptotics, i.e.,  $n = \Theta(p)$ . In fact, in this regime, based on non-rigorous physics arguments, it is conjectured and partially established that instead of the NMF free energy one should optimize the TAP free energy. For linear regression in particular, please see [13, 18]. On the other hand, positive results of NMF for high-dimensional linear regression were recently established in [16] when  $p = o(n)$ .
In this paper, we investigate the performance of NMF approximation for linear regression under the proportional asymptotics regime. As a consequence of our asymptotic characterization, we establish two concrete corollaries: (a) we establish the inaccuracy of the NMF approximation for the log-normalizing constant in this regime, and (b) provide theoretical results backing the empirical observation that NMF can be overconfident in constructing Bayesian credible regions.
Before proceeding further, we formalize the setup under investigation. Given data  $\{(y_i,x_i):1\leq i\leq n\}$ ,  $y_{i}\in \mathbb{R}$ ,  $x_{i}\in \mathbb{R}^{p}$ , the scientist fits a linear model
$$
Y = X \beta^ {\star} + \epsilon , \tag {1}
$$
where  $\epsilon_i \stackrel{iid}{\sim} \mathcal{N}(0, \sigma^2)$  and  $\beta^\star \in S^p$  is a  $p$ -dimensional latent signal. We consider either  $S = \mathbb{R}$  or  $S = [-1, 1]$ . In fact, unless explicitly specified otherwise,  $S = \mathbb{R}$ . Most of our results generalize to bounded support naturally. To recover the latent signal, the scientist adapts a Bayesian approach. She puts an iid prior on  $\beta_i$ 's, namely,  $\mathrm{d}\pi_0(\beta) = \prod_{i=1}^{p} \mathrm{d}\pi(\beta_i)$  and then constructs the posterior distribution of  $\beta$ ,
$$
\frac {\mathrm {d} \mu}{\mathrm {d} \pi_ {0}} (\beta) = \frac {\mathrm {d} \mu_ {X , y}}{\mathrm {d} \pi_ {0}} (\beta) \propto e ^ {- \frac {1}{2 \sigma^ {2}} \| Y - X \beta \| ^ {2}},
$$
with normalization constant
$$
\mathcal {Z} _ {p} = \mathcal {Z} _ {p} (X, Y) = \int_ {S ^ {p}} e ^ {- \frac {1}{2 \sigma^ {2}} \| Y - X \beta \| ^ {2}} \pi_ {0} (\mathrm {d} \beta). \tag {2}
$$
Our results are established assuming that the design matrix  $X$  is randomly sampled from an iid Gaussian ensemble, i.e.,  $X_{ij}\stackrel {iid}{\sim}\mathcal{N}(0,1 / n)$ , while we provide empirical evidence for more general classes of  $X$  that has iid entries with mean 0 and variance  $1 / n$ . Moreover, we assume  $n / p\to \alpha \in (0,\infty)$  as  $n,p\rightarrow \infty$ .
Definition 1 (Exponential tilting). For any  $\gamma \coloneqq (\gamma_1,\gamma_2)\in \bar{\mathbb{R}}\times \mathbb{R}^+$  and probability distribution  $\pi$  on  $S$ , we define  $\pi^{\gamma}$  as
$$
\frac {\mathrm {d} \pi^ {\gamma}}{\mathrm {d} \pi} (x) := \exp \left(\gamma_ {1} x - \frac {\gamma_ {2}}{2} x ^ {2} - c (\gamma)\right), c (\gamma) = c _ {\pi} (\gamma) := \log \int_ {S} \exp \left(\gamma_ {1} x - \frac {\gamma_ {2}}{2} x ^ {2}\right) \pi (\mathrm {d} x).
$$
Note that  $c(\cdot)$  depends on the distribution  $\pi$  and is usually referred to as the cumulant generating function in the statistics literature.
Using this definition of exponential tilts, the  $(X^T X)_{ii}\beta_i^2$  terms in (2) can be absorbed into the base measure
$$
\mu (\mathrm {d} \beta) \propto e ^ {- \frac {1}{2 \sigma^ {2}} \| y - X \beta \| ^ {2} + \sum_ {i = 1} ^ {p} \frac {d _ {i}}{2} \beta_ {i} ^ {2}} \prod_ {i = 1} ^ {p} \pi_ {i} (\mathrm {d} \beta_ {i}),
$$
where  $\pi_i\coloneqq \pi^{(0,d_i)}$  and  $d_{i}\coloneqq \frac{(X^{T}X)_{ii}}{\sigma^{2}}$ . By the classical Gibbs variational principle (see for instance [27]), the log-normalizing constant can be expressed as a variational form,
$$
\begin{array}{l} - \log \mathcal {Z} _ {p} = \inf  _ {Q} \left(\mathbb {E} _ {Q} \left[ \frac {1}{2 \sigma^ {2}} \| y - X \beta \| ^ {2} \right] + \mathrm {D} _ {K L} (Q \| \pi_ {0})\right) \\ = \inf _ {Q} \left( \right.\mathbb {E} _ {Q} \left[ \frac {1}{2 \sigma^ {2}} \| y - X \beta \| ^ {2} - \sum_ {i = 1} ^ {p} \frac {d _ {i}}{2} \beta_ {i} ^ {2} \right] + \mathrm {D} _ {K L} \left( \right.Q \left\| \right. \prod_ {i = 1} ^ {p} \pi_ {i}\left. \right)\left. \right) - \sum_ {i = 1} ^ {p} c (0, d _ {i}), \\ \end{array}
$$
where the inf is taken over all probability distribution on  $S^p$ . While the infimum is always attained if and only if  $Q = \mu$ , the Naive Mean Field (NMF) approximation restricts the variational domain to product distributions only and renders a natural upper bound,
$$
\inf  _ {Q = \prod_ {i = 1} ^ {p} Q _ {i}} \left[ \mathbb {E} _ {Q} \left(\frac {1}{2 \sigma^ {2}} \| y - X \beta \| ^ {2} - \sum_ {i = 1} ^ {p} \frac {d _ {i}}{2} \beta_ {i} ^ {2}\right) + \mathrm {D} _ {K L} \left(Q \left\| \prod_ {i = 1} ^ {p} \pi_ {i}\right) - \sum_ {i = 1} ^ {p} c (0, d _ {i}) \right]. \right. \tag {3}
$$
It can be shown that the product distribution  $\hat{Q}$  that achieves this infimum is exactly the one closest to  $\mu$ , in terms of KL-divergence. Before moving forward, we need some additional definitions and basic properties of exponential tilts. The first lemma establishes that instead of using  $(\gamma_{1},\gamma_{2})$  we can also use  $(u,\gamma_{2}) = (\mathbb{E}_{U\sim \pi^{\gamma}}U,\gamma_{2})$  to parameterize the tilted distribution.
Lemma 1 (Basic properties of the cumulant generating function  $c(\cdot)$ ). Let  $c(\cdot)$  be as in Definition 1. Let  $\operatorname{supp}(\pi)$  denote the support of  $\pi$ . If  $m(\pi) := \inf \operatorname{supp}(\pi) < 0$  and  $M(\pi) := \sup \operatorname{supp}(\pi) > 0$ , then the following conclusions hold.
(a)  $\dot{c}\left(\gamma_1,\gamma_2\right):= \frac{\partial c(\gamma_1,\gamma_2)}{\partial\gamma_1} = \mathbb{E}_{X\sim \pi^{\gamma}}(X)$  is strictly increasing in  $\gamma_{1}$ , for every  $\gamma_{2}\in \mathbb{R}$ .
(b) For any  $u \in (m(\pi), M(\pi))$ , there always exists a unique  $h(u, \gamma_2) \in \mathbb{R}$  such that  $\dot{c}(h(u, \gamma_2), \gamma_2) = u$ .
Definition 2 (Naive mean field variational objective). With  $d_{i} \coloneqq (X^{T}X)_{ii} / \sigma^{2}$ , we define  $M_{p}(u): [m(\pi), M(\pi)]^{p} \to \bar{\mathbb{R}}$  as
$$
M _ {p} (u) := \frac {1}{2 \sigma^ {2}} \left\| y - X u \right\| ^ {2} + \sum_ {i = 1} ^ {p} \left[ G (u _ {i}, d _ {i}) - \frac {d _ {i} u _ {i} ^ {2}}{2} \right],
$$
where  $G$  is defined as a possibly extended real valued function on  $[m(\pi), M(\pi)] \times \mathbb{R}$ ,
$$
\begin{array}{l} G (u, d) := \mathrm {D} _ {\mathrm {K L}} (\pi^ {(h (u, d), d)} \| \pi^ {(0, d)}) = u h (u, d) - c (h (u, d), d) + c (0, d) \quad i f u \in (m (\pi), M (\pi)), d \in \mathbb {R}, \\ := \mathrm {D} _ {\mathrm {K L}} \left(\pi_ {\infty} \| \pi^ {(0, d)}\right) \quad i f u = M (\pi) <   \infty , d \in \mathbb {R}, \\ := \mathrm {D} _ {\mathrm {K L}} \left(\pi_ {- \infty} \| \pi^ {(0, d)}\right) \quad \text {i f} u = m (\pi) > - \infty , d \in \mathbb {R}, \\ \end{array}
$$
in which  $h(\cdot, \cdot)$  was defined in Lemma 1 and  $\pi_{\infty}$  and  $\pi_{-\infty}$  are degenerate distributions which assigns all measure to  $M(\pi)$  and  $m(\pi)$  respectively.
Note that under product distributions, the  $\mathbb{E}_Q(\cdot)$  term in (3) is parameterized by the mean vector  $u\coloneqq \mathbb{E}_Q\beta$  and exponential tilts of  $\pi_i$ 's minimize the KL-divergence term. Therefore, the scaled log-normalizing function, which is also referred to as the average free energy in statistical physics parlance and (log) evidence in Bayesian statistics, is bounded by the following variational form,
$$
- \frac {1}{p} \log \mathcal {Z} _ {p} \leq \frac {1}{p} \inf  _ {u \in [ m (\pi), M (\pi) ] ^ {p}} M _ {p} (u) - \frac {1}{p} \sum_ {i = 1} ^ {p} c (0, d _ {i}) = - \frac {1}{p} \log \mathcal {Z} _ {p} ^ {\mathrm {N M F}}. \tag {4}
$$
The right-hand side is equal to (3) and is also referred to as the evidence lower bound (ELBO) or NMF free energy, which can be used as a model selection criterion, see for instance [14]. Asymptotically, the second term is nothing but a constant since it concentrates around  $c(0, 1 / \sigma^2)$  as  $n, p \to \infty$ .
The main theoretical question of interest here is whether this bound in (4) is asymptotically tight or not, which serves as the fundamental first step towards answering the question of whether NMF distribution is a good approximation of the target posterior. Please see [4, 27] for comprehensive surveys on variational inference, including but not limited to NMF approximation.
To derive sharp asymptotics for the NMF approximation, the key observation is that the optimization problem is convex under certain priors. We then employ the Convex Gaussian Min-max Theorem (CGMT). CGMT is a generalization of the classical Gordon's Gaussian comparison inequality [9], which allows one to reduce a minimax problem parameterized by a Gaussian process to another (tractable) minimax Gaussian optimization problem. This idea was pioneered by Stojnic [21] and then applied to many different statistical problems, including regularized regression, M-estimation, and so on, see for instance [15, 24]. Unfortunately, concentration results derived from CGMT require both Gaussianity and convexity. This is exactly why we need the Gaussian design assumption in our analysis. In the meantime, though we do not pursue this front theoretically, we provide empirical evidence for more general design matrices in the Supplementary Material. It is worth noting that there is a parallel line of research that aims to develop universality results for these comparison techniques. We refer the interested reader to [11] and references within.
Let us emphasize that our main conceptual concern is not investigating whether (4) as a convex optimization procedure gives a good point estimator, but instead evaluating whether NMF as a strategy or product distributions as a family of distributions can provide "close to correct" approximation for the true posterior. Nevertheless, this optimizer's asymptotic mean square error can also be characterized as a by-product of our main theorem.
Regarding the accuracy of variational approximations in general, certain contraction rates and asymptotic normality results were established in the traditional fixed  $p$  large  $n$  regime [28, 17, 10]. However, under the high-dimensional setting and scaling we consider in the current paper, without extra structural assumptions (e.g., sparsity), both the true posterior and its variational approximation are not expected to contract towards the true signal, which also explains why one is instead interested in whether the log-normalizing constant can be well approximated, as a weaker standard of "correctness". Ray et al. [19] studied a pre-specified class of mean field approximation in sparse high-dimensional logistic regression. Recently, the first known results on mean and covariance approximation error of Gaussian Variational Inference (GVI) in terms of dimension and sample size were obtained in [12].
This section starts with some necessary definitions and our main assumptions. Then, we present our main theorem and one natural corollary. Finally, we identify a wide class of priors that would ensure the convexity of the NMF objective, which plays a crucial role in our analysis.
2.1 Notations and main assumptions
Notations: We use the usual Bachmann-Landau notation  $O(\cdot)$ ,  $o(\cdot)$ ,  $\Theta(\cdot)$  for sequences. For a sequence of random variables  $\{X_p : p \geq 1\}$ , we say that  $X_p = o_p(1)$  if  $X_p \stackrel{P}{\to} 0$  as  $p \to \infty$  and  $X_p = o_p(f(p))$  if  $X_p / f(p) = o_p(1)$ . We use  $C, C_1, C_2 \cdots$  to denote positive constants independent of  $n, p$ . Further, these constants can change from line to line. For any square symmetric matrix  $A$ ,  $\|A\|_{\mathrm{op}}$  and  $\|A\|_F$  denote the matrix operator norm and the Frobenius norm, respectively.
Assumption 1 (Proportional asymptotics). We assume  $n / p \to \alpha \in (0, \infty)$ , as  $n, p \to \infty$ .
Assumption 2 (Gaussian features). For all our theoretical results, we assume the design matrix  $X$  is randomly sampled from an iid Gaussian ensemble, i.e.,  $X_{ij} \stackrel{iid}{\sim} \mathcal{N}(0,1/n)$ .
Definition 3. Define  $F:(m(\pi),M(\pi))\to \mathbb{R}$  as
$$
F (u) = F _ {\pi , \sigma^ {2}} (u) := G (u, \mathbb {E} d _ {1}) - \frac {u ^ {2} \mathbb {E} d _ {1}}{2} = G \left(u, \frac {1}{\sigma^ {2}}\right) - \frac {u ^ {2}}{2 \sigma^ {2}}.
$$
Definition 4 (The NMF point estimator). Recalling the NMF objective  $M_p(\cdot)$  as in Definition 2, let  $\hat{u} = \hat{\beta}_{NMF} \coloneqq \arg \min_{u \in [-1,1]^p} M_p(u)$  be the NMF point estimator, which is also the mean vector of the product distribution  $(\hat{Q})$  that best approximates the posterior in terms of KL-divergence. We refer to this optimal product distribution as the NMF distribution.
Assumption 3 (Convexity of  $F(\cdot)$ ). We assume  $F(\cdot)$  is strongly convex on  $S^0 \coloneqq S \setminus \partial S$ .
As alluded to, our analysis relies on the convexity of the "penalty" term  $F(\cdot)$ . Please note that the definition of  $F(\cdot)$  only depends on the prior  $\pi$  chosen by the statistician, rather than the "true prior"  $\pi^{\star}$ . Therefore, to support this assumption, we provide a few sufficient conditions that identify a broad class of priors that ensure (strong) convexity of  $F(\cdot)$  in Section 2.3.
Throughout, we work under a partially well-specified situation, i.e., model (1) is assumed to be correct, but  $\beta_{i}^{\star}$ 's may not have been a priori sampled iid from  $\pi$ . Instead, we assume the empirical distribution of  $\beta_{i}^{\star}$ 's converges in  $L_{2}$  to a probability distribution  $\pi^{\star}$  supported on  $S$ . In addition, the noise level  $\sigma^{2}$  is fixed and known to the statistician. Last but not least,  $\pi^{\star}$  is assumed to have finite second moment and let  $s_{2} := \mathbb{E}_{T \sim \pi^{\star}}[T^{2}] < \infty$ .
2.2 Main results
From now on, we always assume Assumption 1, 2, and 3. Next, we introduce a scalar denoising function, which is just the proximal operator of  $F(\cdot)$ .
Definition 5 (Scalar denoising function). For  $x \in \mathbb{R}$  and  $t > 0$ ,
$$
\eta (x, t) := \arg \min _ {w \in S} \left\{\frac {1}{2 t} (w - x) ^ {2} + F (w) \right\} \in S ^ {\mathrm {o}}
$$
Since  $F(\cdot)$  is strongly convex, this one-dimensional optimization has a unique minimizer. Note that when  $S = [-1, 1]$ , since  $\lim_{w \to \pm 1} \frac{\mathrm{d}F}{\mathrm{d}w}(w) = \lim_{w \to \pm 1} h(w, 1 / \sigma^2) \mp \frac{1}{\sigma^2} = \pm \infty$ , the minimum is never achieved on the boundary of  $S$ . Similarly, when  $S = \mathbb{R}$ ,  $\lim_{w \to \pm \infty} \frac{\mathrm{d}F}{\mathrm{d}w}(w) = \pm \infty$ . Therefore, the minimum is always achieved at a stationary point. Lastly,  $\eta(0, t) = 0$  if  $\pi$  is symmetric. In fact, throughout this paper, we only consider symmetric priors.
Before stating our main result and its implications, we first introduce a two-dimensional optimization problem, which will play a central role in our later discussion,
$$
\max  _ {b \geq 0} \min  _ {\tau \geq \sigma} \phi (b, \tau) \tag {5}
$$
$$
\phi (b, \tau) := \frac {b}{2} \left(\frac {\sigma^ {2}}{\tau} + \tau\right) - \frac {1}{2} b ^ {2} + \frac {1}{\alpha} \mathbb {E} \min  _ {w \in S} \left\{\frac {b}{2 \tau} w ^ {2} - b Z w + \sigma^ {2} F (w + B) - \sigma^ {2} F (B) \right\}, \tag {6}
$$
where  $F(\cdot)$  was defined in Definition 3 and the  $\mathbb{E}$  is taken over  $(B,Z)\sim \pi^{\star}\otimes \mathcal{N}(0,1)$ . In the next lemma, we gather some additional characterizations of this min-max problem.
Lemma 2. The max-min in (5) is achieved at some  $(b^{\star},\tau^{\star})\in (0,\infty)\times (\sigma ,\infty)$ . In fact,  $b^{\star}$  is unique. In addition,  $(b^{\star},\tau^{\star})$  is also a solution to the following fixed point equation,
$$
\tau^ {2} = \sigma^ {2} + \frac {1}{\alpha} \mathbb {E} \left[ \left(\eta (\tau Z + B, \frac {\tau \sigma^ {2}}{b}) - B\right) ^ {2} \right] \tag {7}
$$
$$
b = \tau - \frac {1}{\alpha} \mathbb {E} \left[ Z \cdot \eta \left(\tau Z + B, \frac {\tau \sigma^ {2}}{b}\right) \right] = \tau \left(1 - \frac {1}{\alpha} \mathbb {E} \left[ \eta^ {\prime} \left(\tau Z + B, \frac {\tau \sigma^ {2}}{b}\right) \right]\right),
$$
where  $\eta^{\prime}(x,t)\coloneqq \frac{\partial\eta}{\partial x} (x,t)$
Definition 6. We use  $\nu^{\star} = \nu_{\pi, \pi^{\star}}^{\star}$  to denote the distribution of  $\left(\eta\left(\tau^{\star} Z + B, \frac{\tau^{\star} \sigma^{2}}{b^{\star}}\right), B\right)$ , in which  $(B, Z) \sim \pi^{\star} \otimes \mathcal{N}(0,1)$ . We denote by  $\hat{\nu}$  the empirical distribution of  $\{(\hat{u}_i, \beta_i^{\star})\}_{i=1}^{p}$ .
We are ready to state our main result, which provides a sharp asymptotic characterization of  $\hat{\nu}$ .
Theorem 1. Suppose the max-min problem in (5) has a unique optimizer  $(b^{\star},\tau^{\star})$ , or the fixed point equation in (7) has a unique solution  $(b^{\star},\tau^{\star})$ . Then for all  $\varepsilon >0$ , as  $n,p\to \infty$
$$
\mathbb {P} \left(W _ {2} (\nu^ {\star}, \hat {\nu}) ^ {2} \geq \varepsilon\right)\rightarrow 0,
$$
where  $W_{2}(\cdot, \cdot)$  stands for order 2 Wasserstein distance.
Remark 1. This result indicates the NMF estimator  $\hat{u}$  should be asymptotically roughly iid among different coordinates, which is different from the NMF distributions being product distributions.
Corollary 1. Suppose the hidden true signal  $\beta^{\star}$  was a priori sampled iid from a probability distribution  $\pi^{\star}$  with finite second moment. Note that  $\pi^{\star}$  can differ from the prior  $\pi$  that the Bayesian statistician chose. In addition, suppose the max-min problem in (5) has a unique optimizer  $(b^{\star},\tau^{\star})$  or the fixed point equation in (7) has a unique solution  $(b^{\star},\tau^{\star})$ , then for all  $\varepsilon >0$
$$
\mathbb {P} \left(W _ {2} \left(\nu_ {\pi , \pi^ {*}} ^ {\star}, \hat {\nu}\right) ^ {2} \geq \varepsilon\right)\rightarrow 0, \quad a s n, p \rightarrow \infty ,
$$
in which  $\nu^{\star}$  was defined in Definition 6.
We provide a proof sketch in Section 6 and all the detailed proofs are deferred to the Supplementary Materials.
2.3 Convexity of  $F(\cdot)$
In this section, we present a few lemmas that would ensure the validity of Assumption 3. In fact, if conditions of any of these lemmas are satisfied, Assumption 3 holds.
Lemma 3 (Condition to ensure convexity of  $F(\cdot)$ : nice prior). Suppose  $\pi$  is absolutely continuous with respect to Lebesgue measure and
$$
\frac {\mathrm {d} \pi}{\mathrm {d} x} (x) \propto e ^ {- V (x)}, \forall x \in \operatorname {s u p p o r t} (\pi),
$$
for some  $V:\mathrm{supp o r t}(\pi)\to \mathbb{R}$  . In addition, suppose either of the following two conditions is true,
1.  $\mathrm{support}(\pi) = \mathbb{R}$ ;  $V(x)$  is continuously differentiable almost everywhere;  $V(x)$  is unbounded above at infinity.
2.  $\mathrm{support}(\pi) = [-a, a]$ , for some  $0 < a < \infty$ ;  $V(x)$  is continuously differentiable almost everywhere.
Then if  $V(x)$  is even, non-decreasing in  $|x|$  and  $V'(x)$  is convex,  $F(\cdot)$  is always strongly convex, regardless of the value of  $\sigma^2$ .
Lemma 4 (Condition to ensure convexity of  $F(\cdot)$ : discrete prior). Suppose  $\pi$  is a symmetric discrete distribution supported on  $\{-1,0,1\}$ ,
$$
\pi (\mathrm {d} x) = q \delta (x) + \frac {1 - q}{2} \delta (x - 1) + \frac {1 - q}{2} \delta (x + 1),
$$
for  $q \in (2/3, 1)$ . Then  $F(\cdot)$  is always strongly convex, regardless of the value of  $\sigma^2$ .
Proofs of Lemma 3 and 4 crucially utilize the Griffiths-Hurst-Sherman (GHS) inequality [5, 6], which arose from the study of correlation structure in spin systems. The following two lemmas give examples of some other families of priors for which convexity of  $F(\cdot)$  depends on the noise level  $\sigma^2$ , while those in Lemma 3 and 4 do not.
Lemma 5 (Condition to ensure convexity of  $F(\cdot)$ : low signal-to-noise ratio). Suppose support  $(\pi) \subset [-a, a]$  for some  $a > 0$ . Then as long as  $\sigma^2 > a^2$ ,  $F(u) = F_{\pi}(u, \sigma^2)$ , as a function of  $u$ , is always strongly convex on  $S$ , regardless of the exact choice of  $\pi$  and value of  $\sigma^2$ .
Lemma 6 (Condition to ensure convexity of  $F(\cdot)$ : Spike and Slab prior). Consider a spike and slab prior to the following form,
$$
\pi (\mathrm {d} x) = q \delta (x) + \frac {1 - q}{\sqrt {2 \pi \Delta^ {2}}} e ^ {- \frac {x ^ {2}}{2 \Delta^ {2}}} \mathrm {d} x
$$
which is just a mixture of a point mass at 0 and a Normal distribution of mean 0 and variance  $\Delta^2$ . Suppose