SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \text {L H S} (1 0) \leq \sup  _ {u} \left[ \sum_ {i = 1} ^ {p} \left| G \left(u _ {i}, d _ {i}\right) - G \left(u _ {i}, 1 / \sigma^ {2}\right) \right| + \sum_ {i = 1} ^ {p} \left| \frac {u _ {i} ^ {2}}{2 \sigma^ {2}} - \frac {d _ {i} u _ {i} ^ {2}}{2} \right| \right] \\ \leq \sup _ {u} \left[ \sum_ {i = 1} ^ {p} \left| \frac {\partial G (u , d)}{\partial d} (u _ {i}, 1 / \sigma^ {2}) (d _ {i} - 1 / \sigma^ {2}) \right| \right] + \frac {1}{2} \sum_ {i = 1} ^ {p} | d _ {i} - 1 / \sigma^ {2} | \\ \leq \sum_ {i = 1} ^ {p} \left| d _ {i} - \frac {1}{\sigma^ {2}} \right|. \\ \end{array}
$$
Since  $X_{ji}$ 's are iid with variance  $1/n$ , we know  $\mathbb{E}d_i = \frac{1}{\sigma^2}\mathbb{E}\left[\sum_{j=1}^{n}X_{ji}^2\right] = \frac{1}{\sigma^2}, d_i \stackrel{n\to\infty}{\longrightarrow} \frac{1}{\sigma^2}$ , and all  $d_i$ 's are iid, which guarantee RHS of the previous display goes to 0 in probability as  $n, p \to \infty$ . On the other hand, if  $S = \mathbb{R}$ , note that for any  $\delta \in (0,1/(2\sigma^2))$ ,  $\mathbb{P}(\max_{1\leq i\leq p}|d_i - 1/\sigma^2| > \delta) \to 0$  as  $n, p \to \infty$ . In addition, when  $\max_{1\leq i\leq p}|d_i - 1/\sigma^2| \leq \delta$  is true, which is of approaching 1 probability,
$$
\begin{array}{l} \frac {1}{p} \cdot \mathrm {L H S} \left. \text {o f (1 0)} \leq \sup  _ {u: \| u \| / p <   (1 + C _ {s}) s _ {2}} \left[ \sum_ {i = 1} ^ {p} \left| G \left(u _ {i}, d _ {i}\right) - G \left(u _ {i}, 1 / \sigma^ {2}\right) \right| + \sum_ {i = 1} ^ {p} \left| \frac {u _ {i} ^ {2}}{2 \sigma^ {2}} - \frac {d _ {i} u _ {i} ^ {2}}{2} \right| \right] \right. \\ \leq \frac {1}{p} \cdot \sup  _ {u: \| u \| / p <   (1 + C _ {s}) s _ {2}} \left[ \sum_ {i = 1} ^ {p} \left| \frac {\partial G (u , d)}{\partial d} \left(u _ {i}, 1 / \sigma^ {2} + \delta_ {i}\right) \left(d _ {i} - 1 / \sigma^ {2}\right) \right| \right] + \frac {1}{2} \sum_ {i = 1} ^ {p} | d _ {i} - 1 / \sigma^ {2} | u _ {i} ^ {2}, \\ \end{array}
$$
where  $\delta_i\in \bigl (\min (0,d_i - 1 / \sigma^2),\max (0,d_i - 1 / \sigma^2)\bigr)$ . By Lemma 9, it is further smaller than
$$
\frac{1}{p}\cdot \sup_{u:\| u\| /p <   (1 + C_{s})s_{2}}\left\{\frac{1}{2}\sum_{i = 1}^{p}\bigg|d_{i} - \frac{1}{\sigma^{2}}\bigg|\cdot \left[\operatorname{Var}_{X\sim \pi^{(h(u_{i},1 / \sigma^{2} + \delta_{i}),1 / \sigma^{2})}}(X) + u_{i}^{2} + \operatorname{Var}_{X\sim \pi^{(0,1 / \sigma^{2} + \delta_{i})}}(X) + u_{i}^{2}\right]\right\} .
$$
Lastly, note that when conditions of one of Lemma 3, 4, 5 and 6 are true, for  $\tilde{d}$  close enough to  $1 / \sigma^2$ , we have  $\operatorname{Var}_{X \sim \pi^{(\tilde{h}, \tilde{d})}}(X) < 2\sigma^2$  for any  $\tilde{h} \in \mathbb{R}$ . Therefore, upon choosing small enough  $\delta$  such that all  $d_i$ 's are close enough to  $1 / \sigma^2$ , the display above is controlled by
$$
\begin{array}{l} \frac {1}{p} \cdot \sup  _ {u: \| u \| / p <   2 s _ {2}} \left\{\frac {1}{2} \sum_ {i = 1} ^ {p} \left[ \left| d _ {i} - \frac {1}{\sigma^ {2}} \right| \cdot \left(4 \sigma^ {2} + 2 u _ {i} ^ {2}\right) \right] \right\} \\ \leq \max _ {1 \leq i \leq p} | d _ {i} - 1 / \sigma^ {2} | \cdot \sup _ {u: \| u \| / p <   2 s _ {2}} \left[ 4 \sigma^ {2} + \frac {\| u \| ^ {2}}{p} \right] \\ \leq \delta \cdot \left(4 \sigma^ {2} + \left(1 + C _ {s}\right) s _ {2}\right), \\ \end{array}
$$
Lastly, further requiring  $\delta < \frac{\varepsilon}{4\sigma^2 + (1 + C_s)s_2}$  renders Lemma 8.
B.3 Regarding the fixed point equation
Proof of Lemma 2. First of all, recall the definition of  $\phi (\cdot ,\cdot)$  in (6),
$$
\frac {\partial \phi}{\partial b} (b, \tau) = \frac {1}{2} \left(\sigma^ {2} / \tau + \tau\right) - b - \frac {\tau}{2 \alpha} + \frac {1}{2 \tau \alpha} \mathbb {E} \left[ \left(\tau Z + B - \eta (\tau Z + B, \tau \sigma^ {2} / b)\right) ^ {2} \right].
$$
Note that for any fixed  $x$ ,  $|x - \eta(x, t)|$  is always strictly increasing with respect to  $t$ , we have
$$
\frac {\partial \left\{\mathbb {E} \left[ (\tau Z + B - \eta (\tau Z + B , \tau \sigma^ {2} / b)) ^ {2} \right] \right\}}{\partial b} <   0,
$$
$$
\frac {\partial^ {2} \phi}{\partial b ^ {2}} (b, \tau) <   - 1, \quad \forall b, \tau .
$$
Therefore, for any fixed  $\tau$ ,  $\phi(\cdot, \tau)$  is 1-strongly concave. Define  $\psi(b) := \min_{\tau \geq \sigma} \phi(b, \tau)$ . Since  $\psi(\cdot)$  is the minimum of a collection of 1-strongly concave functions, it is 1-strongly concave itself and
must have a unique maximizer  $b^{\star}$  over  $[0, \infty)$ . In addition, by definition of  $\eta$ ,  $\lim_{t \to \infty} \eta(x, t) = 0$ , dominated convergence theorem gives
$$
\lim _ {b \to 0 ^ {+}} \mathbb {E} \left[ (\tau Z + B - \eta (\tau Z + B, \tau \sigma^ {2} / b)) ^ {2} \right] = \mathbb {E} \left[ (\tau Z + B) ^ {2} \right] = \tau^ {2} + \mathbb {E} [ B ^ {2} ].
$$
Therefore, for any fixed  $\tau$
$$
\lim _ {b \to 0} \inf \frac {\partial \phi}{\partial b} (b, \tau) = \frac {1}{2} (\sigma^ {2} / \tau + \tau) + \frac {\mathbb {E} [ B ^ {2} ]}{2 \tau \alpha} > 0.
$$
Together with Lemma 11 and continuity of  $\phi(\cdot, \cdot)$ , it ensures  $b^{\star} \neq 0$ . On the other hand, for any  $b > 0$ ,
$$
\frac {\partial \phi}{\partial \tau} (b, \tau) = \frac {b}{2 \tau^ {2}} \left[ \tau^ {2} - \left(\sigma^ {2} + \frac {1}{\alpha} \mathbb {E} \left[ (\eta (\tau Z + B, \tau \sigma^ {2} / b) - B) ^ {2} ]\right)\right) \right],
$$
$$
\frac {\partial \phi}{\partial \tau} (b, \tau = \sigma) <   0.
$$
Together with Lemma 11, we have  $\min_{\tau \geq \sigma} \phi(b^{\star}, \tau)$  has at least one minimizer  $\tau^{\star} \in (\sigma, \infty)$ . Finally, since  $b^{\star}$  and  $\tau^{\star}$  are not on the boundary, we have  $\frac{\partial \phi}{\partial b}(b^{\star}, \tau^{\star}) = \frac{\partial \phi}{\partial \tau}(b^{\star}, \tau^{\star}) = 0$ , which gives rise to the fixed point equation as in (7).
Lemma 11. Recall the definition of  $\phi$  in (6). For any fixed  $b\in (0,\infty)$
$$
\lim  _ {\tau \to \infty} \phi (b, \tau) = \infty .
$$
Therefore,  $\min_{\tau}\phi (b,\tau)$  admits at least one minimizer.
Proof. Since  $\mathbb{E}[B^2] = s_2 < \infty$ ,  $\mathbb{E}\min_{w\in S}\left\{\frac{b}{2\tau} w^2 - bZw + \sigma^2 F(w + B) - \sigma^2 F(B)\right\}$  is decreasing in  $\tau$  and always finite for any  $(b,\tau)\in (0,\infty)\times [\sigma ,\infty)$ . Therefore  $\lim_{\tau \to \infty}(b,\tau) = \infty$ .
B.4 Proof of the main results
We devote this subsection to proving Theorem 1, while we note Corollary 1, 2, 3, 4 and 5 are all direct consequences of it. We first prove Theorem 1 while introducing some necessary lemmas. Then, we prove these lemmas at the end of this subsection. Whenever the optimization domains for  $w$  and  $s$  are omitted throughout this subsection, they are understood to be  $\mathbb{R}^p$  and  $\mathbb{R}^n$ , respectively. We use  $\hat{\nu}$  to denote empirical distribution in general.
Since  $F(\cdot)$  is strongly convex,  $\hat{w} \coloneqq \hat{u}_N - \beta^*$  is the unique minimizer of
$$
L (w) := \frac {1}{2 n} \| X w - \epsilon \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} \left(F \left(w _ {i} + \beta_ {i} ^ {\star}\right) - F \left(\beta_ {i} ^ {\star}\right)\right)
$$
$$
\min _ {w} L (w) = \min _ {w \in \mathbb {R} ^ {p}} \max _ {s \in \mathbb {R} ^ {n}} \frac {1}{n} s ^ {T} (X w - \epsilon) - \frac {1}{2 n} \| s \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} (F (w _ {i} + \beta_ {i} ^ {\star}) - F (\beta_ {i} ^ {\star})) := \min _ {w} \max _ {s} \Phi_ {X} (w, s)
$$
Following the recipe in Theorem 2, we define
$$
\Psi_ {g, h} (w, s) := \frac {1}{n ^ {3 / 2}} \| s \| g ^ {T} w + \frac {1}{n ^ {3 / 2}} \| w \| h ^ {T} s - \frac {1}{n} s ^ {T} \epsilon - \frac {1}{2 n} \| s \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} \left(F \left(w _ {i} + \beta_ {i} ^ {\star}\right) - F \left(\beta_ {i} ^ {\star}\right)\right), \tag {12}
$$
where  $g \sim \mathcal{N}(0, I_p)$  and  $h \sim \mathcal{N}(0, I_n)$  and they are independent. Note that with a deliberate abuse of notations, we use  $\Phi$  and  $\Psi$  to denote these two functions to indicate their resemblance to those in the statement of Theorem 2. By Theorem 2, it suffices now to study  $\min_w \max_s \Psi_{g,h}(w,s)$  in place of  $\min_w \max_s \Phi_X(w,s)$ , which is made rigorous by the following lemma.
Lemma 12. Let  $D$  be any close set.
1. We have for all  $t \in \mathbb{R}$
$$
\mathbb{P}\left(\min_{w\in D}\max_{s}\Phi_{X}(w,s)\leq t\right)\leq 2\mathbb{P}\left(\min_{w\in D}\max_{s}\Psi_{g,h}(w,s)\leq t\right).
$$
2. If  $D$  is in addition convex, then we have for all  $t \in \mathbb{R}$
$$
\mathbb{P}\left(\min_{w\in D}\max_{s}\Phi_{X}(w,s)\geq t\right)\leq 2\mathbb{P}\left(\min_{w\in D}\max_{s}\Psi_{g,h}(w,s)\geq t\right).
$$
We defer the proof of Lemma 12 to the end of this section and proceed with proving Theorem 1. Due to strong convexity,  $\hat{w}_{\Psi} \coloneqq \arg \min_{w} \max_{s} \Psi_{g,h}(w,s)$  always exists and is unique. Note that the min and max can be flipped due to convex-concavity (Lemma 10). By optimizing with respect to  $s / \| s\|$  and introducing
$$
\sqrt {\frac {\| w \| ^ {2}}{n} + \sigma^ {2}} = \min  _ {\tau \geq \sigma} \left\{\frac {\frac {\| w \| ^ {2}}{n} + \sigma^ {2}}{2 \tau} + \frac {\tau}{2} \right\}, \tag {13}
$$
$\min_w \max_s \Psi_{g,h}(w,s)$  can be further reduced to
$$
\max_{b\geq 0}\min_{\tau \geq \sigma}\Gamma_{g,h}(b,\tau)
$$
$$
\Gamma_ {g, h} (b, \tau) := \frac {b}{2} (\frac {\sigma^ {2}}{\tau} + \tau) - \frac {b ^ {2}}{2} + \frac {1}{\alpha} \min _ {w \in \mathbb {R} ^ {p}} \sum_ {i = 1} ^ {p} \left[ \frac {1}{p} \left\{\frac {b}{2 \tau} w _ {i} ^ {2} - b g _ {i} w _ {i} + \sigma^ {2} F (w _ {i} + \beta_ {i} ^ {\star}) - \sigma^ {2} F (\beta_ {i} ^ {\star}) \right\} \right],
$$
in the sense that (i) the optimizers  $\hat{w}_{\Psi}$  and  $\hat{w}_{\Gamma}$  are close, i.e, for any  $\kappa > 0$
$$
\mathbb {P} \left(\frac {1}{p} \| \hat {w} _ {\Psi} - \hat {w} _ {\Gamma} \| ^ {2} > \kappa\right)\rightarrow 0, \tag {14}
$$
and (ii) the optimum value is preserved with arbitrarily small error with high probability. The next lemma ensures empirical distribution of  $(\hat{w}_{\Psi},\beta^{\star})$  is close to the distribution of  $\left(\eta \left(\tau^{\star}Z + B,\frac{\tau^{\star}\sigma^{2}}{b^{\star}}\right) - B,B\right)$ , which we denote as  $\nu_{(w^{\star},\pi^{\star})}^{\star}$ , where  $(B,Z)\sim \pi^{\star}\otimes \mathcal{N}(0,1)$ .
Lemma 13. Suppose all conditions of Theorem1 are satisfied. For any  $\varepsilon >0$ , there exists  $C(\varepsilon)\in (0,\varepsilon)$ , such that as  $p,n\to \infty$
$$
\mathbb {P} \left(\exists \tilde {w} \in \mathbb {R} ^ {p} s u c h t h a t W _ {2} \left(\hat {\nu} _ {(\tilde {w}, \beta^ {\star})}, \nu_ {(w ^ {\star}, \pi^ {\star})} ^ {\star}\right) ^ {2} \geq \varepsilon a n d \max  _ {s} \Psi_ {g, h} (\tilde {w}, s) <   \min  _ {w} \max  _ {s} \Psi_ {g, h} (w, s) + C (\varepsilon)\right)\rightarrow 0.
$$
In the meantime,
$$
\min  _ {w} \max  _ {s} \Psi_ {g, h} (w, s) \xrightarrow {P} \frac {\alpha b ^ {\star 2}}{2 \sigma^ {2}} + \mathbb {E} F (\eta (B + \tau^ {\star} Z, \tau^ {\star} / b ^ {\star})).
$$
Again, for now, we proceed assuming Lemma 13 and prove it later at the end of this section. Building upon Lemma 12 and Lemma 13, we now prove the empirical distribution of  $(\hat{u}_N,\beta^\star) = (\beta^\star +\hat{w},\beta^\star)$  is close to  $\nu^{\star}$  as defined in Definition 6. For  $\varepsilon >0$  , define  $D_{\varepsilon} =$ $\left\{w\in \mathbb{R}^p:W_2\left(\hat{\nu}_{(w,\beta^{\star})},\nu_{(w^{\star},\pi^{\star})}^{\star}\right)^2\geq \varepsilon \right\}$  . To establish
$$
\mathbb {P} \left(W _ {2} (\hat {\nu} _ {(\hat {w}, \beta^ {\star})}, \nu_ {(w ^ {\star}, \pi^ {\star})} ^ {\star}) ^ {2} > \epsilon\right) \to 0,
$$
it suffices to show with high probability, for some  $\delta (\varepsilon) > 0$
$$
\min  _ {w \in D _ {\varepsilon}} \max  _ {s} \Phi_ {X} (w, s) \geq \min  _ {w \in \mathbb {R} ^ {p}} \max  _ {s} \Phi_ {X} (w, s) + \delta (\varepsilon). \tag {15}
$$
On the one hand, by applying both (1) and (2) of Lemma 12 to  $D = \mathbb{R}^p$ , together with Lemma 13, we have
$$
\lim _ {n, p \to \infty} \min _ {w} \max _ {s} \Phi_ {X} (w, s) = \lim _ {n, p \to \infty} \min _ {w} \max _ {s} \Psi_ {g, h} (w, s) = \frac {\alpha b ^ {\star 2}}{2 \sigma^ {2}} + \mathbb {E} F (\eta (B + \tau^ {\star} Z, \tau^ {\star} / b ^ {\star})),
$$
where the "lim" is understood to be convergence in probability. It further leads to
$$
\mathbb {P} \left(\left| \min  _ {w} \max  _ {s} \Phi_ {X} (w, s) - \min  _ {w} \max  _ {s} \Psi_ {g, h} (w, s) \right| > \varepsilon\right)\rightarrow 0.
$$
On the other hand, applying (1) of Lemma 12 to  $D = D_{\varepsilon}$ , together with Lemma 13, we have
$$
\mathbb {P} \left(\min  _ {w \in D _ {\varepsilon}} \max  _ {s} \Phi_ {X} (w, s) > \min  _ {w} \max  _ {s} \Phi_ {X} (w, s) + C (\varepsilon) + \varepsilon\right) \to 0,
$$
which establishes (15) with  $\delta(\varepsilon) = C(\varepsilon) + \varepsilon$ , where  $C(\varepsilon) > 0$  was defined in Lemma 13. Therefore, we have the empirical distribution of  $(\hat{u}_N, \beta^\star)$  is close to the target distribution  $\nu^\star$ , i.e.,
$$
W _ {2} \left(\hat {\nu} _ {\left(\hat {u} _ {N}, \beta^ {\star}\right)}, \nu^ {\star}\right) \xrightarrow {\mathrm {P}} 0. \tag {16}
$$
Finally, according to Lemma 8 and 7,  $N_{p}(\cdot)$  and  $M_{p}(\cdot)$  are with high probability uniformly close. Together with strong convexity of  $N_{p}(\cdot)$ , we have for any  $\kappa > 0$
$$
\mathbb {P} \left(\frac {1}{p} \| \hat {u} - \hat {u} _ {N} \| ^ {2} <   \kappa\right)\rightarrow 0. \tag {17}
$$
Theorem 1 is therefore given by (16) and (17).
Proof of Lemma 12. In order to prove Lemma 12 using Theorem 2, one only needs to establish that the optimizer of  $\Phi_X(w,s)$  always has a bounded norm with high probability. In fact, Lemma 7 ensures boundedness of  $\hat{w} = \arg \min_w\max_s\Phi_X(w,s)$  while the boundedness of  $\hat{s}\coloneqq \arg \max_s\Phi_X(\hat{w},s)$  can be established by a similar argument.
$$
\tilde {\Gamma} _ {g, h} (b, \tau) := \frac {b}{2} (\frac {\sigma^ {2}}{\tau} + \tau) - \frac {b ^ {2}}{2} + \frac {1}{\alpha} \min _ {w \in D _ {\varepsilon}} \sum_ {i = 1} ^ {p} \left[ \frac {1}{p} \left\{\frac {b}{2 \tau} w _ {i} ^ {2} - b g _ {i} w _ {i} + \sigma^ {2} F (w _ {i} + \beta_ {i} ^ {\star}) - \sigma^ {2} F (\beta_ {i} ^ {\star}) \right\} \right].
$$
By definition,  $\tilde{\Gamma}_{g,h}(b,\tau) \geq \Gamma_{g,h}(b,\tau)$  for any  $(b,\tau)$  deterministically. Recall the definition of  $\Phi_{g,h}(\cdot)$  in (12),
$$
\min  _ {w \in D _ {\varepsilon}} \max  _ {s} \Psi_ {g, h} (w, s)
$$
$$
= \min  _ {w \in D _ {\varepsilon}} \max  _ {\| s \|} \frac {1}{n ^ {3 / 2}} \| s \| g ^ {T} w + \frac {1}{n} \left\| \frac {\| w \| h}{\sqrt {n}} - \epsilon \right\| \cdot \| s \| - \frac {1}{2 n} \| s \| ^ {2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} (F (w _ {i} + \beta_ {i} ^ {*}) - F (\beta_ {i} ^ {*}))
$$
$$
\stackrel {b = \frac {\| s \|}{\sqrt {n}}} {=} \min  _ {w \in D _ {\varepsilon}} \max  _ {b \geq 0} \left[ \frac {1}{n} b g ^ {T} w + b \sqrt {\frac {\| w \| ^ {2}}{n} + \sigma^ {2}} - \frac {b ^ {2}}{2} + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {p} \left(F \left(w _ {i} + \beta_ {i} ^ {*}\right) - F \left(\beta_ {i} ^ {*}\right)\right) \right] + o _ {n} (1)
$$
$$
\stackrel {(1 3)} {=} \min _ {w \in D _ {\varepsilon}} \max _ {b \geq 0} \min _ {\tau \geq \sigma} \frac {b}{2} (\frac {\sigma^ {2}}{\tau} + \tau) - \frac {b ^ {2}}{2} + \frac {1}{\alpha} \sum_ {i = 1} ^ {p} \left[ \frac {1}{p} \left\{\frac {b}{2 \tau} w _ {i} ^ {2} - b g _ {i} w _ {i} + \sigma^ {2} F (w _ {i} + \beta_ {i} ^ {\star}) - \sigma^ {2} F (\beta_ {i} ^ {\star}) \right\} \right] + o _ {n} (1)
$$
$$
\stackrel {\text {L e m m a} 1 0} {\geq} \max  _ {b \geq 0} \min  _ {\tau \geq \sigma} \tilde {\Gamma} _ {g, h} (b, \tau) + o _ {n} (1).
$$
$$
\begin{array}{l} \min  _ {w \in D _ {\varepsilon}} \max  _ {s} \Psi_ {g, h} (w, s) \geq \max  _ {b \geq 0} \min  _ {\tau \geq \sigma} \tilde {\Gamma} _ {g, h} (b, \tau) + o _ {n} (1) \\ \geq \min  _ {\tau \geq \sigma} \tilde {\Gamma} _ {g, h} (b ^ {\star}, \tau) + o _ {n} (1) \\ = \tilde {\Gamma} _ {g, h} \left(b ^ {\star}, \tilde {\tau} \left(b ^ {\star}\right)\right) + o _ {n} (1) \\ \geq \Gamma_ {g, h} \left(b ^ {\star}, \tilde {\tau} \left(b ^ {\star}\right)\right) + o _ {n} (1) \tag {18} \\ \stackrel {(i i)} {\geq} \min  _ {\tau \geq \sigma} \Gamma_ {g, h} (b ^ {\star}, \tau) + o _ {n} (1) \\ = \Gamma_ {g, h} \left(b ^ {\star}, \tau^ {\star}\right) + o _ {n} (1) \\ = \min  _ {w \in \mathbb {R} ^ {p}} \max  _ {s} \Psi_ {g, h} (w, s) + o _ {n} (1), \\ \end{array}
$$
where  $\tilde{\tau}(b^{\star}) \coloneqq \arg \min_{\tau \geq \sigma} \tilde{\Gamma}_{g,h}(b^{\star}, \tau)$ . We claim that the gaps resulting from (i) and (ii) can not be both negligible. Namely, there exists  $\varrho > 0$  such that
$$
\lim  _ {n, p \rightarrow \infty} \sup  _ {\varrho > \varrho} \mathbb {P} (T _ {1} + T _ {2} > \varrho) > 0, \tag {19}
$$
$$
T _ {1} := \tilde {\Gamma} _ {g, h} \left(b ^ {\star}, \tilde {\tau} \left(b ^ {\star}\right)\right) - \Gamma_ {g, h} \left(b ^ {\star}, \tilde {\tau} \left(b ^ {\star}\right)\right) \geq 0,
$$
$$
T _ {2} := \Gamma_ {g, h} (b ^ {\star}, \tilde {\tau} (b ^ {\star})) - \Gamma_ {g, h} (b ^ {\star}, \tau^ {\star}) \geq 0.
$$
In order to establish (19), we will proceed with proof by contradiction. Suppose (19) is NOT true, equivalently, for any  $\varrho > 0$ ,
$$
\lim  _ {n, p \rightarrow \infty} \mathbb {P} \left(T _ {1} > \varrho\right) = 0, \tag {20}
$$
$$
\lim  _ {n, p \rightarrow \infty} \mathbb {P} \left(T _ {2} > \varrho\right) = 0. \tag {21}
$$
Recall the definition of  $\phi(\cdot)$  in (6). Since  $(b^{\star},\tau^{\star})$  is the unique optimizer of  $\phi(\cdot)$  and  $\Gamma_{g,h}(b,\tau)$  converges to  $\phi(b,\tau)$  uniformly on any compact subset of  $[0,\infty) \times [\sigma,\infty)$ , (21) is only possible if  $|\tilde{\tau}(b^{\star}) - \tau^{\star}| \xrightarrow{P_{\varepsilon}} 0$  as  $n,p \to \infty$ . On the other hand, by definition of  $D_{\varepsilon}$ , there exists  $\gamma > 0$  such that with high probability, for any  $w \in D_{\varepsilon}$ ,
$$
\begin{array}{l} \sum_ {i = 1} ^ {p} \left[ \frac {1}{p} \left\{\frac {b ^ {\star}}{2 \tau^ {\star}} w _ {i} ^ {2} - b ^ {\star} g _ {i} w _ {i} + \sigma^ {2} F (w _ {i} + \beta_ {i} ^ {\star}) - \sigma^ {2} F (\beta_ {i} ^ {\star}) \right\} \right] \\ \left. \right. > \sum_ {i = 1} ^ {p} \left[ \frac {1}{p} \left\{\frac {b ^ {\star}}{2 \tau^ {\star}} \bar {w} _ {i} ^ {2} - b ^ {\star} g _ {i} \bar {w} _ {i} + \sigma^ {2} F (\bar {w} _ {i} + \beta_ {i} ^ {\star}) - \sigma^ {2} F (\beta_ {i} ^ {\star}) \right\}\right] + \gamma \varepsilon , \\ \end{array}
$$
where  $\bar{w}_i$  is sampled independently from the distribution of  $\eta\left(\tau^{\star}Z + \beta_i^{\star}, \frac{\tau^{\star}\sigma^2}{b^{\star}} - \beta_i^{\star}\right)$  with  $Z \sim \mathcal{N}(0,1)$ . Therefore, with high probability,
$$
\tilde {\Gamma} _ {g, h} \left(b ^ {\star}, \tau^ {\star}\right) - \Gamma_ {g, h} \left(b ^ {\star}, \tau^ {\star}\right) > \gamma \varepsilon .
$$
By triangle inequality,
$$
T _ {1} \geq \left[ \tilde {\Gamma} _ {g, h} (b ^ {\star}, \tau^ {\star}) - \Gamma_ {g, h} (b ^ {\star}, \tau^ {\star}) \right] - \left| \tilde {\Gamma} _ {g, h} (b ^ {\star}, \tilde {\tau} (b ^ {\star})) - \tilde {\Gamma} _ {g, h} (b ^ {\star}, \tau^ {\star}) \right| - \left| \Gamma_ {g, h} (b ^ {\star}, \tilde {\tau} (b ^ {\star})) - \Gamma_ {g, h} (b ^ {\star}, \tau^ {\star}) \right|.
$$
In addition, note that if  $|\tilde{\tau}(b^{\star}) - \tau^{\star}| \stackrel{\mathrm{P}}{\to} 0$  as  $n, p \to \infty$ , then
$$
\tilde {\Gamma} _ {g, h} \left(b ^ {\star}, \tilde {\tau} \left(b ^ {\star}\right)\right) - \tilde {\Gamma} _ {g, h} \left(b ^ {\star}, \tau^ {\star}\right) \xrightarrow {\mathrm {P}} 0 \quad \text {a n d} \quad \Gamma_ {g, h} \left(b ^ {\star}, \tilde {\tau} \left(b ^ {\star}\right)\right) - \Gamma_ {g, h} \left(b ^ {\star}, \tau^ {\star}\right) \xrightarrow {\mathrm {P}} 0.
$$
$$
\mathbb {P} \left(T _ {1} > \gamma \varepsilon / 2\right)\rightarrow 1.
$$
The display above is in contradiction to (20), which means (19) is thus established.
Finally, note that Lemma 13 is equivalent to
$$
\mathbb{P}\left(\min_{w\in D_{\varepsilon}}\max_{s}\Psi_{g,h}(w,s) - \min_{w\in \mathbb{R}^{p}}\max_{s}\Psi_{g,h}(w,s) > C(\varepsilon)\right)\to 1
$$
and it is therefore a direct consequence of (19) and (18).
C Numerical simulations
All source code can be found in a separate zip file in the Supplementary Materials.
Figure 4: iid Gaussian design versus iid Laplace design (with Gaussian spike and slab prior): These three plots showcase the empirical observation that prediction of Corollary 2 seem to be valid for a design matrix with iid entries that have sub-exponential tails.
C.1 Universality: non-Gaussian design matrix
Instead of assuming  $X_{ij} \stackrel{iid}{\sim} \mathcal{N}(0,1/n)$ , we now present empirical evidence of universality, i.e., Theorem 1 holds for a broader class of design matrix that has  $iid$  entries with variance  $1/n$ . Since it is impossible to exhaust all possible distributions, we will stick with a representative example  $X_{ij} \stackrel{iid}{\sim} \mathrm{Laplace}(\sqrt{2}/2)$  and the Gaussian spike and slab prior. We use Gradient Decent to optimize  $M_p(u)$  and then demonstrate that the empirical MSE of its optimizer coincides with the prediction of Corollary 2. Please see Figure 4 for a visual summary.
For more comprehensive and rigorous results on the universality of Gaussian comparison inequalities, we refer interested readers to [11] and references within.
C.2 Fixed point equation
Figure 5: As we can see, when  $q$  is large ( $q = 0.8$  or 0.9 in the figure above), our initialization did not lead to the right fixed point.
As alluded to in the main text, all our plots are generated by iteratively solving the fixed point equation (7). However, this naive strategy might not give the right fixed point, i.e., the  $(b^{\star},\tau^{\star})$  that minimizes  $\phi (b,\tau)$ , or it could just do not converge. Please see Figure 5 for an empirical example. Since either  $F(\cdot)$  or  $\eta (\cdot ,\cdot)$  lacks analytical forms for most natural priors, unlike other applications of CGMT (e.g., asymptotic analysis of lasso [15]), it is hard to determine whether (7) has a unique solution analytically. Fortunately, there are two possible remedies. First, which is the option we took, one could solve  $\min_uM_p(u)$  for some large  $n$  and check if the empirical MSE matches the prediction given by the fixed point. Alternatively, one could adapt a more brute-force way to find the actual optimizer of  $\max_b\min_\tau \phi (b,\tau)$ , e.g., grid search or iteratively solving (7) with multiple initializations. After all, it is only a two-dimensional scalar optimization problem. We followed the first way simply because we wanted to use empirical simulations to corroborate our theoretical predictions anyway.