SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \lambda_ {F} ^ {2} \min  \left\{\mu^ {2}, 1 \right\} \cdot \min  \left\{1, \mu^ {- 1} \right\} o \left(n ^ {r + 1 - t - 2 p}\right) \leq \mu^ {- 2} \min  \left\{\mu^ {2}, \mu^ {- 1} \right\} o \left(n ^ {r - t - 1}\right) (158) \\ \leq \min  \left\{1, \mu^ {- 3} \right\} o \left(n ^ {r - t - 1}\right) (159) \\ \leq \min  \left\{1, \mu^ {- 2} \right\} o \left(n ^ {r - t - 1}\right). (160) \\ \end{array}
$$
Turning back to the first term, we are now in a position to apply Bernstein. Note that  $(\mathbf{A}_{-s}^{-1},\mathbf{y}_{\alpha},\mathbf{y}_{\beta})$  are independent of  $Z_{R}$ . Hence conditioned on  $\mathbf{A}_{-s}^{-1},\mathbf{y}_{\alpha}$ , and  $\mathbf{y}_{\beta}$ ,  $\langle Z_R^\top A_{-s}^{-1}\mathbf{y}_\alpha ,Z_R^\top A_{-s}^{-1}\mathbf{y}_\beta \rangle$  is a sum of  $s - 2$  subexponential variables, and by Lemma 2.7.7 of Vershynin (2018) each of these random variables conditionally has subexponential norm at most  $\| \pmb{A}_{-s}^{-1}\pmb {y}_\alpha \| _2\| \pmb{A}_{-s}^{-1}\pmb {y}_\beta \| _2$  and conditional mean  $\langle A_{-s}^{-1}\pmb {y}_\alpha ,A_{-s}^{-1}\pmb {y}_\beta \rangle$
$$
\left\| \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha} \right\| _ {2} \left\| \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta} \right\| _ {2} \leq O \left(n ^ {1 - t - 2 p}\right). \tag {161}
$$
Let us now compute the Hanson-Wright bound for  $\langle A_{-s}^{-1}\pmb{y}_{\alpha},A_{-s}^{-1}\pmb{y}_{\beta}\rangle$  . Note that  $A_{-s}^{-1}$  is independent of  $\pmb{y}_{\alpha}$  and  $\pmb{y}_{\beta}$  , so we can condition on  $A_{-s}^{-1}$  and conclude that with probability at least  $1 - O(1 / nk)$
$$
\left\langle \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle \leq \mathbb {E} \left[ \left\langle \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle \mid \boldsymbol {A} _ {- s} ^ {- 1} \right] + c _ {6} n ^ {\frac {1 - t}{2}} \left\| \boldsymbol {A} _ {- s} ^ {- 2} \right\| _ {2} \sqrt {\log (n k)}. \tag {162}
$$
We can rewrite the expectation as
$$
\operatorname {T r} \left(\boldsymbol {A} _ {s} ^ {- 2} \mathbb {E} \left[ \boldsymbol {y} _ {\beta} \boldsymbol {y} _ {\alpha} ^ {\top} \right]\right). \tag {163}
$$
Clearly,  $\mathbb{E}[\pmb{y}_{\beta}\pmb{y}_{\alpha}^{\top}]$  is diagonal, and each diagonal entry is equal to. Let  $\rho = \frac{1}{k}$ . Then since  $\pmb{y}_{\alpha} = 1 - \rho$  implies  $\pmb{y}_{\beta} = -\rho$  and vice versa, we get
$$
\begin{array}{l} \mathbb {E} \left[ \boldsymbol {\mathcal {Y}} _ {\alpha} [ i ] \boldsymbol {\mathcal {Y}} _ {\beta} [ i ] \right] = 2 (1 - \rho) (- \rho) \Pr \left[ \boldsymbol {\mathcal {Y}} _ {\alpha} [ i ] = 1 - \rho \right] + (- \rho) ^ {2} \Pr \left[ \boldsymbol {\mathcal {Y}} _ {\alpha} [ i ] = \boldsymbol {\mathcal {Y}} _ {\beta} [ i ] = - \rho \right] (164) \\ \leq - 2 \rho^ {2} (1 - \rho) + \rho^ {2} \Pr [ \mathbf {y} _ {\alpha} [ i ] = - \rho ] (165) \\ \leq - \rho^ {2} (1 - \rho). (166) \\ \end{array}
$$
In other words, the expectation is negative, so we can neglect it in our upper bound.
On the other hand, the deviation term is with very high probability at most
$$
c _ {6} n ^ {\frac {1 - t}{2}} \left\| \boldsymbol {A} _ {- s} ^ {- 2} \right\| _ {2} \sqrt {\log (n k)} \leq c _ {6} n ^ {\frac {1 - t}{2} - 2 p} \sqrt {\log (n k)}. \tag {167}
$$
Combining all of our bounds, Bernstein yields
$$
\begin{array}{l} \left\langle \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle \leq \left(\sum_ {j \in [ s ] \backslash \{\alpha , \beta \}} c _ {6} n ^ {\frac {1 - t}{2} - 2 p} \sqrt {\log (n k)}\right) + n ^ {\frac {r}{2} + 1 - t - 2 p} (168) \\ \leq n ^ {r + \frac {1 - t}{2} - 2 p} \sqrt {\log (n k)} + n ^ {\frac {r}{2} - t + 1 - 2 p}. (169) \\ \end{array}
$$
Again, let's trace all the way back to Eq. (148) and then the favored correlation bound. We have shown that  $\langle Z_R^\top A_{-s}^{-1}\pmb{y}_\alpha, Z_R^\top A_{-s}^{-1}\pmb{y}_\beta \rangle$  's contribution to the favored correlation is at most
$$
\begin{array}{l} c _ {1 7} \lambda_ {F} ^ {2} \min  \left\{\mu^ {2}, 1 \right\} \left(n ^ {r + \frac {1 - t}{2} - 2 p} \sqrt {\log (n k)} + n ^ {\frac {r}{2} - t + 1 - 2 p}\right) (170) \\ = c _ {1 7} \mu^ {- 2} \min  \left\{\mu^ {2}, 1 \right\} \left(n ^ {r - \frac {1 + t}{2} - 1} \sqrt {\log (n k)} + n ^ {\frac {r}{2} - t - 1}\right) (171) \\ \leq c _ {1 7} \min  \left\{1, \mu^ {- 2} \right\} \left(n ^ {r - \frac {1 + t}{2} - 1} \sqrt {\log (n k)} + n ^ {\frac {r}{2} - t - 1}\right) (172) \\ \leq c _ {1 7} \min  \left\{1, \mu^ {- 2} \right\} o \left(n ^ {r - t - 1}\right), (173) \\ \end{array}
$$
where the last line follows because  $0 < t < r < 1$ , and  $c_{17}$  is a positive constant.
F.3.2 Bounding the unfavored correlation
Now, let us show that the unfavored correlation  $\mathsf{COR}_{\alpha,\beta,U}$  is negligible; more precisely, we'll show that it's min  $\{1,\mu^{-2}\} o(n^{r-t-1}) + o(n^{1-t-p})$ . We can rewrite  $\mathsf{COR}_{\alpha,\beta,U}$  as
$$
\lambda_ {U} \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {A} _ {U} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\beta},
$$
and play the same game with using Woodbury to replace  $\mathbf{A}^{-1}$  with  $\mathbf{A}^{-1} - \mathbf{A}_U^{-1}\mathbf{M}_s\mathbf{A}_U^{-1}$ , where we recall that
$$
\boldsymbol {M} _ {s} \triangleq \boldsymbol {W} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \in \mathbb {R} ^ {n \times n}.
$$
$$
\boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta} - 2 \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} M _ {s} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta} + \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} M _ {s} \boldsymbol {A} _ {U} ^ {- 1} M _ {s} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta}. \tag {174}
$$
Let us first focus on the first term of Eq. (174). Hanson-Wright implies that with probability at least  $1 - O(1/n)$ ,
$$
\left| \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta} - \mathbb {E} \left[ \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta} \mid \boldsymbol {A} _ {U} ^ {- 1} \right] \right| \leq n ^ {\frac {1 - t}{2}} \left\| \boldsymbol {A} _ {U} ^ {- 1} \right\| _ {2} \sqrt {\log n}. \tag {175}
$$
Also,  $\mathbb{E}[y_{\alpha}^{\top}\pmb{A}_{U}^{-1}\pmb{y}_{\beta}|\pmb{A}_{U}^{-1}] = \mathrm{Tr}\big(\pmb{A}_{U}^{-1}\mathbb{E}[\pmb{y}_{\beta}\pmb{y}_{\alpha}^{\top}]\big) = \Theta (n^{1 - 2t - p})$  with high probability, and  $\| \pmb{A}_U^{-1}\| _2\leq n^{-p}$  with extremely high probability. Hence we see that  $\pmb{y}_{\alpha}^{\top}\pmb{A}_{U}^{-1}\pmb{y}_{\beta}\leq O(n^{\frac{1 - t}{2} -p})\leq o(n^{1 - t - p})$  as  $t > 0$
Next, let's turn to the second and third terms of Eq. (174). We claim that only the second term will be relevant to bound asymptotically, and moreover that they are both min  $\{1,\mu^{-2}\} o(n^{r - t - 1})$ . Since
$$
\begin{array}{l} M _ {s} \boldsymbol {A} _ {U} ^ {- 1} M _ {s} = \boldsymbol {W} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {H} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \\ = \boldsymbol {W} _ {s} \left(\left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 1} - \left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 2}\right) \boldsymbol {W} _ {s} ^ {\top}, \\ \end{array}
$$
the second and third term can be rewritten as
$$
\begin{array}{l} - 2 \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {W} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta} \\ + \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {W} _ {s} \left(\left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 1} - \left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 2}\right) \boldsymbol {W} _ {s} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {y} _ {\beta}. \\ \end{array}
$$
As we are going to use Hanson-Wright to bound the entries of  $\mathbf{Z}_s^\top \mathbf{A}_U^{-1} \mathbf{y}_\alpha$ , it follows that only the second term of Eq. (174) is relevant asymptotically.
To bound the second term, we will use Cauchy-Schwarz. We see that
$$
\begin{array}{l} \left. \mathbf {y} _ {\alpha} ^ {\top} \mathbf {A} _ {U} ^ {- 1} \mathbf {M} _ {s} \mathbf {A} _ {U} ^ {- 1} \mathbf {y} _ {\beta} \leq \lambda_ {F} \right\| \left(\mathbf {I} _ {s} + \mathbf {H} _ {s}\right) ^ {- 1} \left\| _ {2} \right\| \left. \mathbf {Z} _ {s} ^ {\top} \mathbf {A} _ {U} ^ {- 1} \mathbf {y} _ {\alpha} \right\| _ {2} \left\| \mathbf {Z} _ {s} ^ {\top} \mathbf {A} _ {U} ^ {- 1} \mathbf {y} _ {\beta} \right\| _ {2} (176) \\ \leq \lambda_ {F} \min  \left\{\mu , 1 \right\} O \left(n ^ {r - t + 1 - 2 p}\right) \log (n s) (177) \\ \leq \mu^ {- 1} \min  \left\{\mu , 1 \right\} O \left(n ^ {r - t - p}\right) \log (n s) (178) \\ \leq \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {r - t - p}\right) \log (n s), (179) \\ \end{array}
$$
where in the second line we have used Proposition B.6.
Now, note that if regression works, this yields an upper bound of  $O(n^{r - t - p}) \log (ns)$ . But since  $p > 1$ , this is  $o(n^{r - t - 1})$ , which means this contribution is dominated by the denominator.
On the other hand, if regression fails, then the upper bound is now  $\mu^{-1} O(n^{r - t - p}) \log (ns)$ , which we claim is  $o(\mu^{-2} n^{r - t - 1})$ . Indeed, from the definition of the bi-level ensemble Definition 1, we have  $p > q + r$ , so
$$
\begin{array}{l} \min  \left\{1, \mu^ {- 1} \right\} n ^ {r - t - p} \leq \mu^ {- 1} n ^ {r - t - 1} \cdot n ^ {1 - p} \\ \leq \mu^ {- 1} o \left(n ^ {r - t - 1} \cdot n ^ {1 - q - r}\right) \\ = \mu^ {- 2} o (n ^ {r - t - 1}), \\ \end{array}
$$
Let us now go back to Eq. (174) and combine our two bounds. Since  $\lambda_U = O(1)$ , we have just shown that
$$
\operatorname {c o r} _ {\alpha , \beta , U} \leq \min  \left\{1, \mu^ {- 2} \right\} o \left(n ^ {r - t - 1}\right) + o \left(n ^ {1 - t - p}\right), \tag {180}
$$
G A new variant of the Hanson-Wright inequality with soft sparsity
In this section, we prove Theorem 4.1. First, we outline a high level idea of the proof. The starting point of the proof is to explicitly decompose the quadratic form into diagonal and off-diagonal terms
$$
\begin{array}{l} \boldsymbol {x} ^ {\top} \boldsymbol {M} \boldsymbol {y} - \mathbb {E} [ \boldsymbol {x} ^ {\top} \boldsymbol {M} \boldsymbol {y} ] = \sum_ {i, j} m _ {i j} X _ {i} Y _ {j} - \sum_ {i} m _ {i i} \mathbb {E} [ X _ {i} Y _ {i} ] (181) \\ = \underbrace {\sum_ {i} m _ {i i} \left(X _ {i} Y _ {i} - \mathbb {E} \left[ X _ {i} Y _ {i} \right]\right)} _ {\triangleq S _ {\mathrm {d i a g}}} + \underbrace {\sum_ {i \neq j} m _ {i j} X _ {i} Y _ {j}} _ {\triangleq S _ {\mathrm {o f f d i a g}}} (182) \\ \end{array}
$$
where in the first line we have used the fact that for  $i \neq j$ ,  $X_{i}$  and  $Y_{j}$  are independent and mean zero to conclude that  $\mathbb{E}[X_iY_j] = 0$ .
We can start with the upper tail inequality  $\mathbb{P}[x^{\top}M\pmb {y} - \mathbb{E}[x^{\top}M\pmb {y}] > t]$  and conclude the lower tail inequality by replacing  $M$  with  $-M$ . To bound  $S_{\mathrm{diag}}$  and  $S_{\mathrm{offdiag}}$ , we will proceed by explicitly bounding the MGF and applying Chernoff's inequality.
G.1 Diagonal terms
For the diagonal terms, we want to bound the MGF of  $S_{\mathrm{diag}} = \sum_{i}m_{ii}\bigl (X_iY_i - \mathbb{E}[X_iY_i]\bigr)$ . For  $\lambda^2 < \frac{1}{2C_1K^2\max_i m_{ii}^2}$ , we obtain
$$
\begin{array}{l} \mathbb {E} \exp \left(\lambda S _ {\mathrm {d i a g}}\right) = \prod_ {i = 1} ^ {n} \mathbb {E} _ {X _ {i}, Y _ {i}} \exp \left(\lambda m _ {i i} \left(X _ {i} Y _ {i} - \mathbb {E} \left[ X _ {i} Y _ {i} \right]\right)\right) (183) \\ \leq \prod_ {i = 1} ^ {n} \mathbb {E} _ {Y _ {i}} \mathbb {E} _ {X _ {i}} [ \exp (\lambda m _ {i i} Y _ {i} (X _ {i} - \mathbb {E} [ X _ {i} | Y _ {i} ])) | Y _ {i} ] (184) \\ \leq \prod_ {i = 1} ^ {n} \mathbb {E} _ {Y _ {i}} \exp \left(C _ {1} \lambda^ {2} m _ {i i} ^ {2} K ^ {2} Y _ {i} ^ {2}\right) (185) \\ \end{array}
$$
where we have applied Jensen's inequality in the second line and the subgaussian assumption on  $X_{i}$  conditioned on  $Y_{i}$  in the last line. Here,  $C_1$  is a universal positive constant relating the equivalent formulations of subgaussianity Vershynin (2018). Continuing with our calculation, we have
$$
\begin{array}{l} \mathbb {E} \exp \left(\lambda S _ {\mathrm {d i a g}}\right) \leq \prod_ {i = 1} ^ {n} \mathbb {E} _ {Y _ {i}} \left[ 1 + 2 C _ {1} \lambda^ {2} m _ {i i} ^ {2} K ^ {2} Y _ {i} ^ {2} \right] (186) \\ \leq \prod_ {i = 1} ^ {n} \left(1 + 2 C _ {1} \pi \lambda^ {2} K ^ {2} m _ {i i} ^ {2}\right) (187) \\ \leq \exp \left(2 C _ {1} \pi \lambda^ {2} K ^ {2} \sum_ {i = 1} ^ {n} m _ {i i} ^ {2}\right). (188) \\ \end{array}
$$
where in the first line we have used the inequality  $\exp(x) \leq 1 + 2x$  valid for  $x \leq \frac{1}{2}$ , in the second line we have used the soft sparsity assumption on  $Y_{i}$ , and in the last line we have used the inequality  $1 + x \leq \exp(x)$ , valid for all  $x$ .
Now Markov's inequality yields for  $\epsilon > 0$  that
$$
\begin{array}{l} \Pr \left[ S _ {\text {d i a g}} > \epsilon \right] \leq \frac {\mathbb {E} \exp \left(\lambda S _ {\text {d i a g}}\right)}{\exp (\lambda \epsilon)} (189) \\ \leq \exp \left(- \lambda \epsilon + 2 \pi C _ {1} K ^ {2} \lambda^ {2} \sum_ {i = 1} ^ {n} m _ {i i} ^ {2}\right), (190) \\ \end{array}
$$
and optimizing  $\lambda$  in the region  $\lambda^2 \leq \frac{1}{2C_1K^2\max_i m_{ii}^2}$  yields
$$
\lambda = \min  \left\{\frac {\epsilon}{2 C _ {1} K ^ {2} \pi \sum_ {i = 1} ^ {n} m _ {i i} ^ {2}}, \frac {1}{2 C _ {1} K \max  _ {i} | m _ {i i} |} \right\}. \tag {191}
$$
Plugging in this value of  $\lambda$  into the Markov calculation yields the desired upper tail bound. We can repeat the argument with  $-M$  to get the lower tail bound. A union bound completes the proof.
G.2 Offdiagonal terms
Following Rudelson and Vershynin (2013), for the offdiagonal terms we can decouple the terms in the sum. More precisely, the terms in  $S_{\mathrm{offdiag}}$  involving indices  $i$  and  $j$  are precisely  $m_{ij}X_iY_j + m_{ji}Y_iX_j$ . The issue is that  $Y_{i}$  can be correlated with  $X_{i}$ , which complicates the behavior of this random variable. Decoupling ensures that for any  $j \in [n]$  we will have exactly one term which involves either  $X_{j}$  or  $Y_{j}$ , so in particular we will regain independence of the terms, allowing us to bound the MGF more easily.
Let  $\{\delta_i\}_{i\in [n]}$  denote iid Bernoulli's with parameter  $1 / 2$  , which are independent of all other random variables.
$$
S _ {\delta} \triangleq \sum_ {i \neq j} m _ {i j} \delta_ {i} (1 - \delta_ {j}) X _ {i} Y _ {j}.
$$
Since  $\mathbb{E}[\delta_i(1 - \delta_j)] = \frac{1}{4}$ , we have
$$
S _ {\text {o f f d i a g}} = 4 \mathbb {E} _ {\delta} [ S _ {\delta} ],
$$
Hence, Jensen's inequality yields
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \exp (\lambda S _ {\text {o f f d i a g}}) \leq \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}, \delta} \exp (4 \lambda S _ {\delta}),
$$
where we have used the independence of  $\delta$  and all other random variables. It follows that it suffices to upper bound the MGF of  $S_{\delta}$ .
Define the random set  $\Lambda_{\delta} = \{i\in [n]:\delta_i = 1\}$  to denote the indices selected by  $\delta$ . For a vector  $\pmb {u}\in \mathbb{R}^n$  we also introduce the shorthand  $\pmb{u}_{\Lambda_{\delta}}$  to denote the subvector of  $\pmb{u}$  where  $\delta_{i} = 1$  and  $\pmb{u}_{\Lambda_{\delta}^{-}}$  to denote the subvector of  $\pmb{u}$  where  $\delta_{i} = 0$ .
Hence, we can rewrite  $S_{\delta} \triangleq \sum_{i \in \Lambda_{\delta}, j \in \Lambda_{\delta}^{c}} m_{ij} X_{i} Y_{j}$ . For  $|\lambda| \leq \frac{1}{2C_{1} K \|M\|_{2}}$ , we have
$$
\begin{array}{l} \mathbb {E} \exp \left(\lambda S _ {\text {o f f d i a g}}\right) \leq \mathbb {E} \exp \left(4 \lambda S _ {\delta}\right) (192) \\ \leq \mathbb {E} _ {\delta} \prod_ {i \in \Lambda_ {\delta}, j \in \Lambda_ {\delta} ^ {c}} \mathbb {E} _ {\boldsymbol {x} _ {\Lambda_ {\delta}}, \boldsymbol {y} _ {\Lambda_ {\delta} ^ {c}}} [ \exp (\lambda m _ {i j} X _ {i} Y _ {j}) ] (193) \\ \end{array}
$$
Now we can use the fact that the  $X_{i}$  and  $Y_{j}$  are mean zero and independent because  $i\in \Lambda_{\delta}$  and  $j\in \Lambda_{\delta}^{c}$ , to show that
$$
\begin{array}{l} \prod_ {i \in \Lambda_ {\delta}, j \in \Lambda_ {\delta} ^ {c}} \mathbb {E} _ {\boldsymbol {x} _ {\Lambda_ {\delta}}, \boldsymbol {y} _ {\Lambda_ {\delta} ^ {c}}} [ \exp (\lambda m _ {i j} X _ {i} Y _ {j}) ] \leq \prod_ {i \in \Lambda_ {\delta}, j \in \Lambda_ {\delta} ^ {c}} \mathbb {E} _ {\boldsymbol {y} _ {\Lambda_ {\delta} ^ {c}}} [ \exp (C _ {1} \lambda^ {2} K ^ {2} m _ {i j} ^ {2} Y _ {j} ^ {2}) ] (194) \\ \leq \prod_ {i \in \Lambda_ {\delta}, j \in \Lambda_ {\delta} ^ {c}} \mathbb {E} _ {\boldsymbol {y} _ {\Lambda_ {\delta} ^ {c}}} \left[ 1 + 2 C _ {1} \lambda^ {2} K ^ {2} m _ {i j} ^ {2} Y _ {j} ^ {2}) \right] (195) \\ \leq \prod_ {i \in \Lambda_ {\delta}, j \in \Lambda_ {\delta} ^ {c}} \left(1 + 2 \pi C _ {1} \lambda^ {2} K ^ {2} m _ {i j} ^ {2} Y _ {j} ^ {2}\right) (196) \\ \leq \prod_ {i \in \Lambda_ {\delta}, j \in \Lambda_ {\delta} ^ {c}} \exp \left(2 \pi C _ {1} \lambda^ {2} K ^ {2} m _ {i j} ^ {2} Y _ {j} ^ {2}\right) (197) \\ \leq \exp \left(2 \pi C _ {1} \lambda^ {2} K ^ {2} \| M \| _ {F} ^ {2}\right). (198) \\ \end{array}
$$
In the first line, we have used the subgaussianity of  $X_{i}$ ; in the second line, we have used the assumption on  $\lambda$ , in the third line, we have used the variance bound on  $Y_{j}$ .
Again, we can apply Markov's inequality and to see that for  $\epsilon > 0$ ,
$$
\begin{array}{l} \Pr \left[ S _ {\text {d i a g}} > \epsilon \right] \leq \frac {\mathbb {E} \exp \left(\lambda S _ {\text {d i a g}}\right)}{\exp (\lambda \epsilon)} (199) \\ \leq \exp \left(- \lambda \epsilon + 2 \pi C _ {1} K ^ {2} \lambda^ {2} \| \boldsymbol {M} \| _ {F} ^ {2}\right), (200) \\ \end{array}
$$
$$
\lambda = \min  \left\{\frac {\epsilon}{2 C _ {1} K ^ {2} \pi \| \boldsymbol {M} \| _ {F} ^ {2}}, \frac {1}{2 C _ {1} K \| \boldsymbol {M} \| _ {2}} \right\} \tag {201}
$$
yields the desired result.
H Proofs of main lemmas for concentration of spectrum
The goal of this section is ultimately to prove Proposition B.6, which asserts that for valid  $(T,S)$ , the hat matrix  $H_{T,S}$  is a flat matrix whose spectrum is min  $\{\mu,1\} (1 + o(1))$  with extremely high probability. First, let us recall some notation. For any  $\varnothing \neq T \subseteq S \subseteq [s]$ , we can define the  $(T,S)$  hat matrix as  $H_{T,S} \triangleq W_T^\top A_{-S}^{-1}W_T$ . Here,  $W_T$  is the  $n \times |T|$  matrix of weighted features in  $T$ , and  $A_{-T} = A - W_T W_T^\top$  is the leave- $T$ -out Gram matrix.
First, Wishart concentration applied to  $\mathbf{W}_T^\top \mathbf{W}_T$  yields the following result.
Lemma H.1. Recall that  $\mu \triangleq n^{q + r - 1}$  and  $\pmb{W}_{T}^{\top}\pmb{W}_{T}\in \mathbb{R}^{|T|\times |T|}$ . For any nonempty  $T\subseteq [s]$ , with probability at least  $1 - 2e^{-\sqrt{n}}$  we have that for all  $i\in [|T|]$
$$
\mu_ {i} \left(\boldsymbol {W} _ {T} ^ {\top} \boldsymbol {W} _ {T}\right) = \left(1 \pm c _ {T} \sqrt {\frac {| T |}{n}}\right) \mu^ {- 1} n ^ {p} \tag {202}
$$
Proof. We can apply Lemma B.3 with  $M = Z_{T} \in \mathbb{R}^{n \times |T|}$ , with  $M = n$ ,  $m = |T| = o(n)$ , and  $\epsilon = n^{\frac{1}{4}} = o(\sqrt{n|T|})$ . Hence we have
$$
n - 2 \sqrt {n | T |} + o (\sqrt {n | T |}) \leq \mu_ {| T |} (\pmb {Z} _ {T} ^ {\top} \pmb {Z} _ {T}) \leq \mu_ {1} (\pmb {Z} _ {T} ^ {\top} \pmb {Z} _ {T}) \leq n + 2 \sqrt {n | T |} + o (\sqrt {n | T |}).
$$
Plugging in the scaling  $\lambda_F = n^{p - q - r}$  and dividing through by  $n$  yields the desired result. Here, we define  $c_{T}$  to be an appropriately defined positive constant which only depends on  $|T|$  (as the favored features are identically distributed).
Next, we can use Wishart concentration to bound the spectrum of  $A_U$ .
Lemma H.2 (Concentration of spectrum for unfavored Gram matrix). Throughout this theorem, assume we are in the bi-level model (Definition 1). Define  $\mu \triangleq n^{q + r - 1}$ . Recall  $A_U = \lambda_U\sum_{j > s}z_jz_j^\top \in \mathbb{R}^{n\times n}$ . With probability at least  $1 - 2e^{-n}$ , for  $i\in [n]$  we have
$$
\mu_ {i} \left(\boldsymbol {A} _ {U}\right) = \left(1 \pm c _ {1 8} n ^ {\kappa_ {7}}\right) n ^ {p}, \tag {203}
$$
where  $c_{18}$  and  $\kappa_7$  are positive constants. In other words, the spectrum of the unfavored Gram matrix  $\mathbf{A}_U$  is flat.
Proof. Note that  $A_U = \lambda_U \sum_{j > s} z_j z_j^\top$ . Under the bi-level model,  $\lambda_U = 1 + o(1)$ . Now we can apply Lemma B.3 with  $M = \sum_{j > s} z_j z_j^\top$ ,  $M = d - s = n^p - n^r$ ,  $m = n = o(d)$ , and  $\epsilon = \sqrt{2n}$  to conclude that with probability at least  $1 - 2e^{-n}$ , we have
$$
d - 2 \sqrt {d n} + n - \sqrt {2 n} \leq \mu_ {n} \left(\sum_ {j > s} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}\right) \leq \mu_ {1} \left(\sum_ {j > s} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}\right) \leq d + 2 \sqrt {d n} + n + \sqrt {2 n}. \tag {204}
$$
We can obtain the spectrum of  $A_U$  by multiplying through by
$$
\begin{array}{l} \lambda_ {U} = \frac {(1 - a) d}{d - s} (205) \\ = 1 + n ^ {\max  \{- q, r - p \}} + o \left(n ^ {\max  \{- q, r - p \}}\right), (206) \\ \end{array}
$$
where in the last line we have used the power series expansion for  $\frac{1}{1 - x} = 1 + x + o(x)$ . Preserving only first order terms for  $\lambda_U$  and the spectrum of  $\sum_{j > s} z_j z_j^\top$  in Eq. (204) yields
$$
\mu_ {i} \left(\boldsymbol {A} _ {U}\right) = \left(1 \pm c _ {1 8} n ^ {\max  \left\{\frac {1 - p}{2}, r - p, - q \right\}}\right) n ^ {p}. \tag {207}
$$
In fact, we know  $\frac{1 - p}{2} > 1 - p > r - p$ , since  $r < 1$  and  $p > 1$ . This means we can neglect the  $r - p$  term in the max, define  $\kappa_7 = \min \left\{\frac{p - 1}{2}, -q\right\} > 0$  and  $c_{18}$  to be an appropriately defined positive constant.
Since  $\mathbf{A}_{-T} = \mathbf{A}_U + \mathbf{W}_{[s]\backslash T}\mathbf{W}_{[s]\backslash T}^\top$ , we can apply Lemmas H.1 and H.2 to control the spectrum of  $\mathbf{A}_{-T}$ . We show that there is a (potentially) spiked portion of the spectrum corresponding to the  $s - |T|$  favored features which were not taken out, whereas the rest of the  $n - s + |T|$  eigenvalues are flat.
Lemma H.3. Recall that  $A_{-T} \in \mathbb{R}^{n \times n}$ . For any nonempty  $T \subseteq [s]$ , with probability at least  $1 - 2e^{-\sqrt{n}} - 2e^{-n}$ , we have that for all  $i \in [s - |T|]$
$$
\mu_ {i} (\boldsymbol {A} _ {- T}) = \left(1 \pm c _ {T} \sqrt {\frac {| T |}{n}}\right) \mu^ {- 1} n ^ {p} + \left(1 \pm c _ {1 8} n ^ {- \kappa_ {7}}\right) n ^ {p}. \tag {208}
$$
For all  $i \in [n] \setminus [s - |T|]$ , we have
$$
\mu_ {i} \left(\boldsymbol {A} _ {- T}\right) = \left(1 \pm c _ {1 8} n ^ {- \kappa_ {7}}\right) n ^ {p}. \tag {209}
$$
$$
\boldsymbol {A} _ {- T} = \boldsymbol {W} _ {[ s ] \backslash T} \boldsymbol {W} _ {[ s ] \backslash T} ^ {\top} + \boldsymbol {A} _ {U}. \tag {210}
$$
Weyl's inequality (Horn and Johnson, 2012, Corollary 4.3.15) implies that for any  $i \in [n]$ , we have
$$
\mu_ {i} \left(\boldsymbol {W} _ {[ s ] \backslash T} \boldsymbol {W} _ {[ s ] \backslash T} ^ {\top}\right) + \mu_ {n} (\boldsymbol {A} _ {U}) \leq \mu_ {i} (\boldsymbol {A} _ {- T}) \leq \mu_ {i} \left(\boldsymbol {W} _ {[ s ] \backslash T} \boldsymbol {W} _ {[ s ] \backslash T} ^ {\top}\right) + \mu_ {1} (\boldsymbol {A} _ {U}). \tag {211}
$$