SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \left(\pmb {U} ^ {\perp}\right) ^ {\mathsf {T}} \pmb {\theta} _ {0} \right\| _ {\ell_ {2}} ^ {2} = \left\| \pmb {\theta} _ {0} \right\| _ {\ell_ {2}} ^ {2} - \left\| \pmb {U} ^ {\mathsf {T}} \pmb {\theta} _ {0} \right\| _ {\ell_ {2}} ^ {2} = (1 - \rho) r _ {\mathrm {s}} ^ {2} + r _ {\mathrm {n s}} ^ {2}.
$$
We next proceed to compute the risk of estimator in terms of  $\gamma$ ,  $\gamma_0$ . We use equation (B.6), which for the min-norm estimator with no look-alike clustering, reduces to
$$
\operatorname {R i s k} (\widehat {\boldsymbol {\theta}}) = \sigma^ {2} + \gamma_ {0} ^ {2} + \gamma^ {\top} (I + \mu^ {2} \operatorname {d i a g} (\pi)) \gamma . \tag {B.18}
$$
This concludes the proof. Note that in the theorem statement we made the change of variables  $\gamma_0\rightarrow \tilde{\gamma}_0$  and  $\gamma \rightarrow \tilde{\alpha}$ , for an easier comparison with the risk of look-alike estimator.)
B.4 Proof of Proposition 3.4
Consider singular value decompositions  $\boldsymbol{X}_L = \boldsymbol{U} \boldsymbol{\Sigma} \boldsymbol{V}^T$  and  $\tilde{\boldsymbol{X}}_L = \widetilde{\boldsymbol{U}} \widetilde{\boldsymbol{\Sigma}} \widetilde{\boldsymbol{V}}^T$ . We then can write the estimators  $\widetilde{\boldsymbol{\theta}}_L$  and  $\widehat{\boldsymbol{\theta}}_L$  as follows:
$$
\widehat {\boldsymbol {\theta}} _ {L} = \boldsymbol {U} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {V} ^ {\mathsf {T}} \boldsymbol {y}, \quad \widetilde {\boldsymbol {\theta}} _ {L} = \widetilde {\boldsymbol {U}} \widetilde {\boldsymbol {\Sigma}} ^ {- 1} \widetilde {\boldsymbol {V}} ^ {\mathsf {T}} \boldsymbol {y}.
$$
We first bound  $\| \widehat{\pmb{\theta}}_L - \widetilde{\pmb{\theta}}_L\|$ . We write
$$
\left\| \widehat {\boldsymbol {\theta}} _ {L} - \widetilde {\boldsymbol {\theta}} _ {L} \right\| \leq \left\| \boldsymbol {U} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {V} ^ {\mathsf {T}} - \widetilde {\boldsymbol {U}} \widetilde {\boldsymbol {\Sigma}} ^ {- 1} \widetilde {\boldsymbol {V}} ^ {\mathsf {T}} \right\| \| \boldsymbol {y} \|. \tag {B.19}
$$
$$
\left\| \boldsymbol {y} \right\| = \left\| \boldsymbol {X} ^ {\top} \boldsymbol {\theta} _ {0} + \varepsilon \right\| = \left\| \boldsymbol {\Lambda} ^ {\top} \boldsymbol {M} ^ {\top} \boldsymbol {\theta} _ {0} + \boldsymbol {Z} ^ {\top} \boldsymbol {\theta} _ {0} + \varepsilon \right\|.
$$
Note that  $Z^{\top} \pmb{\theta}_0 + \varepsilon \stackrel{(d)}{=} \sqrt{\|\pmb{\theta}_0\|^2 + \sigma^2} \pmb{g}$  where  $g \sim \mathsf{N}(0, I_n)$ . In addition,
$$
\begin{array}{l} \frac {1}{n} \left\| \boldsymbol {\Lambda} ^ {\mathrm {T}} \boldsymbol {M} ^ {\mathrm {T}} \boldsymbol {\theta} _ {0} \right\| ^ {2} = \frac {1}{n} \boldsymbol {\theta} _ {0} ^ {\mathrm {T}} \boldsymbol {M} \boldsymbol {\Lambda} \boldsymbol {\Lambda} ^ {\mathrm {T}} \boldsymbol {M} ^ {\mathrm {T}} \boldsymbol {\theta} _ {0} \\ = \boldsymbol {\theta} _ {0} ^ {\mathsf {T}} \boldsymbol {M} \operatorname {d i a g} \left(\frac {n _ {1}}{n}, \dots , \frac {n _ {k}}{n}\right) \boldsymbol {M} ^ {\mathsf {T}} \boldsymbol {\theta} _ {0} \xrightarrow {p} \boldsymbol {\theta} _ {0} ^ {\mathsf {T}} \boldsymbol {M} \operatorname {d i a g} \left(\pi_ {1}, \dots , \pi_ {k}\right) \boldsymbol {M} ^ {\mathsf {T}} \boldsymbol {\theta} _ {0}. \\ \end{array}
$$
Therefore by using concentration of Lipschitz functions of Gaussian vectors, we get
$$
\frac {1}{\sqrt {n}} \| \boldsymbol {y} \| \stackrel {p} {\rightarrow} \sqrt {\boldsymbol {\theta} _ {0} ^ {\top} M \mathrm {d i a g} (\boldsymbol {\pi}) M ^ {\top} \boldsymbol {\theta} _ {0} + \| \boldsymbol {\theta} _ {0} \| ^ {2} + \sigma^ {2}}.
$$
This shows that
$$
\frac {1}{\sqrt {n}} \| \boldsymbol {y} \| \rightarrow C \leq \sqrt {\left(\mu + 1\right)\left(r _ {\mathrm {s}} ^ {2} + r _ {\mathrm {n s}} ^ {2}\right) + \sigma^ {2}}. \tag {B.20}
$$
We next use the result of [28, Theorem 3.3], by which we obtain
$$
\left\| \boldsymbol {U} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {V} ^ {\mathsf {T}} - \widetilde {\boldsymbol {U}} \widetilde {\boldsymbol {\Sigma}} ^ {- 1} \widetilde {\boldsymbol {V}} ^ {\mathsf {T}} \right\| \leq \frac {1 + \sqrt {5}}{2} \max  \left(\frac {1}{\sigma_ {\min } (\boldsymbol {\Sigma}) ^ {2}}, \frac {1}{\sigma_ {\min } (\widetilde {\boldsymbol {\Sigma}}) ^ {2}}\right) \| \boldsymbol {U} \boldsymbol {\Sigma} \boldsymbol {V} ^ {\mathsf {T}} - \widetilde {\boldsymbol {U}} \widetilde {\boldsymbol {\Sigma}} \widetilde {\boldsymbol {V}} ^ {\mathsf {T}} \|. \tag {B.21}
$$
$$
\left\| \boldsymbol {U} \boldsymbol {\Sigma} \boldsymbol {V} ^ {\mathsf {T}} - \widetilde {\boldsymbol {U}} \widetilde {\boldsymbol {\Sigma}} \widetilde {\boldsymbol {V}} ^ {\mathsf {T}} \right\| = \left\| \boldsymbol {X} _ {L} - \widetilde {\boldsymbol {X}} _ {L} \right\| = \left\| \boldsymbol {M} _ {\mathrm {s}} \boldsymbol {\Lambda} - \widetilde {\boldsymbol {M}} _ {s} \widetilde {\boldsymbol {\Lambda}} \right\| \leq \delta \sqrt {n}, \tag {B.22}
$$
by the assumption of the theorem statement. We next lower bound  $\sigma_{\mathrm{min}}(\Sigma) = \sigma_{\mathrm{min}}(X_L)$ . Recall that  $X_L^\top = (M\Lambda)^\top + [\mathbf{0}_{n \times p}, Z_{n \times (d - p)}]$ , with  $Z$  having i.i.d  $\mathsf{N}(0,1)$  entries.
Next suppose that Condition  $(i)$  holds true, namely  $\delta < \sqrt{1 - (\psi_d - \psi_p)} - \sqrt{\psi_d - \psi_p}$ , with  $\psi_d - \psi_p < 0.5$ . Using the result of [31, Theorem 2.1], we have with probability at least  $1 - n^{-1}$ ,
$$
\sigma_ {\min } \left(\boldsymbol {X} _ {L}\right) \geq \sqrt {n} \left(\sqrt {\psi_ {d} - \psi_ {p} - 1} - 1 - \sqrt {\frac {2 \log n}{n}}\right).
$$
$$
\begin{array}{l} \sigma_ {\min } \left(\tilde {\boldsymbol {X}} _ {L}\right) \geq \sigma_ {\min } \left(\boldsymbol {X} _ {L}\right) - \left\| \boldsymbol {X} _ {L} - \tilde {\boldsymbol {X}} _ {L} \right\| \\ \geq \sqrt {n} \left(\sqrt {1 - (\psi_ {d} - \psi_ {p})} - \sqrt {\psi_ {d} - \psi_ {p}} - \sqrt {\frac {2 \log n}{n}} - \delta\right) \\ \geq c ^ {\prime} \sqrt {n} \left(\sqrt {1 - (\psi_ {d} - \psi_ {p})} - \sqrt {\psi_ {d} - \psi_ {p}}\right), \\ \end{array}
$$
using the assumption on the estimation error rate  $\delta$ . Therefore, using the above bound along with (B.22) in (B.21) we get
$$
\| \boldsymbol {U} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {V} ^ {\mathsf {T}} - \widetilde {\boldsymbol {U}} \widetilde {\boldsymbol {\Sigma}} ^ {- 1} \widetilde {\boldsymbol {V}} ^ {\mathsf {T}} \| \leq \frac {1 + \sqrt {5}}{2 c ^ {\prime 2}} \frac {1}{\sqrt {n} \left(\sqrt {1 - \left(\psi_ {d} - \psi_ {p}\right)} - \sqrt {\psi_ {d} - \psi_ {p}}\right) ^ {2}} \delta .
$$
Combining the above bound with (B.20), we get
$$
\left\| \widehat {\boldsymbol {\theta}} _ {L} - \widetilde {\boldsymbol {\theta}} _ {L} \right\| \leq \frac {1 + \sqrt {5}}{2 c ^ {\prime 2}} \frac {C}{\left(\sqrt {1 - \left(\psi_ {d} - \psi_ {p}\right)} - \sqrt {\psi_ {d} - \psi_ {p}}\right) ^ {2}} \delta . \tag {B.23}
$$
We next note that by triangle inequality, the above bound implies that
$$
\left\| \widetilde {\boldsymbol {\theta}} _ {L} - \boldsymbol {\theta} _ {0} \right\| - \left\| \widehat {\boldsymbol {\theta}} _ {L} - \boldsymbol {\theta} _ {0} \right\| \leq \left\| \widehat {\boldsymbol {\theta}} _ {L} - \widetilde {\boldsymbol {\theta}} _ {L} \right\| = O (\delta).
$$
Therefore, by invoking Lemma 2.1, we obtain the desired result on  $\operatorname{Risk}(\widetilde{\theta}_L)$ .
Next suppose that Condition  $(ii)$  holds, namely  $\delta < \sqrt{\psi_d - \psi_p - 1} - 1$  with  $\psi_d - \psi_p > 2$ . Using the result of [31, Theorem 2.1] for  $X^{\top}$ , we have with probability at least  $1 - n^{-1}$ ,
$$
\sigma_ {\min} (\boldsymbol {X} _ {L}) \geq \sqrt {n} \left(\sqrt {\psi_ {d} - \psi_ {p} - 1} - 1 - \sqrt {\frac {2 \log n}{n}}\right).
$$
By following a similar argument we prove the claim under Condition  $(ii)$ .
B.5 Proof of Theorem 5.1
We use Theorem 3.3 (b) to characterize  $\operatorname{Risk}(\widehat{\pmb{\theta}})$  in the regime of  $\psi_d \geq 1$ . Specializing to the case of balanced cluster priors, the risk depends on  $\tilde{\alpha}$  only through its norm  $\tilde{\alpha} \coloneqq \| \tilde{\alpha} \|_{\ell_2}$ , and is given by
$$
\begin{array}{l} \operatorname {R i s k} (\widehat {\boldsymbol {\theta}}) \stackrel {\mathcal {P}} {\rightarrow} \sigma^ {2} + \tilde {\gamma} _ {0} ^ {2} + \left(\frac {\mu^ {2}}{k} + 1\right) \tilde {\alpha} ^ {2} \\ = \frac {\psi_ {d}}{\psi_ {d} - 1} \left(\sigma^ {2} + \left(\frac {\mu^ {2}}{k} + 1\right) \tilde {\alpha} ^ {2}\right) + \left(1 - \frac {1}{\psi_ {d}}\right) \left((1 - \rho) r _ {\mathrm {s}} ^ {2} + r _ {\mathrm {n s}} ^ {2}\right), \\ \end{array}
$$
$$
\tilde {\alpha} = \left(1 + \frac {\frac {\mu^ {2}}{k} + 1}{\psi_ {d} - 1}\right) ^ {- 1} \sqrt {\rho} r _ {\mathrm {s}}.
$$
In addition, by Theorem 3.1 we have
$$
\mathrm {R i s k} (\widehat {\pmb {\theta}} _ {L}) \stackrel {\mathcal {P}} {\rightarrow} \frac {\sigma^ {2} + r _ {\mathrm {s}} ^ {2}}{1 - \psi_ {d} + \psi_ {p}} - \rho r _ {\mathrm {s}} ^ {2}.
$$
Note that  $\operatorname{Risk}(\widehat{\theta}_L)$  in this regime does not depend on  $\mu^2 / k$ . Also, it is easy to verify that  $\operatorname{Risk}(\widehat{\theta})$  is decreasing in  $\mu^2 / k$ . Therefore the gain  $\Delta$  is decreasing in  $\mu^2 / k$ .
Also observe that  $\operatorname{Risk}(\widehat{\pmb{\theta}})$  is increasing in  $r_{\mathrm{ns}}$ , while  $\operatorname{Risk}(\widehat{\theta}_L)$  does not depend on  $r_{\mathrm{ns}}$ . Therefore, the gain  $\Delta$  is increasing in  $r_{\mathrm{ns}}$ .
To understand the dependence of  $\Delta$  on  $\rho$ , we write
$$
\begin{array}{l} \Delta - 1 = \frac {\operatorname {R i s k} (\widehat {\boldsymbol {\theta}})}{\operatorname {R i s k} (\widehat {\boldsymbol {\theta}} _ {L})} - 1 \\ = \frac {\frac {\psi_ {d}}{\psi_ {d} - 1} \left(\sigma^ {2} + \left(\frac {\mu^ {2}}{k} + 1\right) \left(1 + \frac {\mu^ {2} / k + 1}{\psi_ {d} - 1}\right) ^ {- 2} \rho r _ {\mathrm {s}} ^ {2}\right) + \left(1 - \frac {1}{\psi_ {d}}\right) \left((1 - \rho) r _ {\mathrm {s}} ^ {2} + r _ {\mathrm {n s}} ^ {2}\right)}{\frac {\sigma^ {2} + r _ {\mathrm {s}} ^ {2}}{1 - \psi_ {d} + \psi_ {p}} - \rho r _ {\mathrm {s}} ^ {2}} - 1 \\ = \frac {\frac {\psi_ {d}}{\psi_ {d} - 1} \left(\sigma^ {2} + \left(\frac {\mu^ {2}}{k} + 1\right) \left(1 + \frac {\mu^ {2} / k + 1}{\psi_ {d} - 1}\right) ^ {- 2} \rho r _ {\mathrm {s}} ^ {2}\right) + \left(1 - \frac {1}{\psi_ {d}}\right) \left(r _ {\mathrm {s}} ^ {2} + r _ {\mathrm {n s}} ^ {2}\right) - \frac {\sigma^ {2} + r _ {\mathrm {s}} ^ {2}}{1 - \psi_ {d} + \psi_ {p}} + \frac {\rho r _ {\mathrm {s}} ^ {2}}{\psi_ {d}}}{\frac {\sigma^ {2} + r _ {\mathrm {s}} ^ {2}}{1 - \psi_ {d} + \psi_ {p}} - \rho r _ {\mathrm {s}} ^ {2}} \\ \end{array}
$$
As we see the numerator is increasing in  $\rho$  and denominator is decreasing in  $\rho$ , which implies that the gain  $\Delta$  is increasing in  $\rho$ .
We next show that  $\Delta \geq 1$  if condition (5.1) holds. Since  $\Delta$  is decreasing in  $\mu^2 /k$  and increasing in  $\rho$ , it suffices to show the claim assuming  $\mu^2 /k\to \infty$  and  $\rho = 0$ . In this case we have  $\left(\frac{\mu^2}{k} +1\right)\tilde{\alpha}^2\rightarrow 0$  and so
$$
\begin{array}{l} \Delta \rightarrow \frac {\frac {\sigma^ {2} \psi_ {d}}{\psi_ {d} - 1} + \left(1 - \frac {1}{\psi_ {d}}\right)\left(r _ {\mathrm {s}} ^ {2} + r _ {\mathrm {n s}} ^ {2}\right)}{\frac {\sigma^ {2} + r _ {\mathrm {s}} ^ {2}}{1 - \psi_ {d} + \psi_ {p}}} \\ \geq \frac {\frac {\sigma^ {2} \psi_ {d}}{\psi_ {d} - 1} + \left(1 - \frac {1}{\psi_ {d}}\right) r _ {\mathrm {s}} ^ {2}}{\frac {\sigma^ {2} + r _ {\mathrm {s}} ^ {2}}{1 - \psi_ {d} + \psi_ {p}}} \\ = \frac {\frac {\psi_ {d}}{\psi_ {d} - 1} + \left(1 - \frac {1}{\psi_ {d}}\right) \mathrm {S N R} ^ {2}}{\frac {1 + \mathrm {S N R} ^ {2}}{1 - \psi_ {d} + \psi_ {p}}} \geq 1, \\ \end{array}
$$
where the last step follows from condition (5.1).