SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {\Delta} y ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {\Delta} y = c _ {1 5} ^ {2} (1 \pm o (1)) n ^ {1 - t - p}, \tag {136}
$$
$$
\begin{array}{l} \boldsymbol {M} _ {s} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {M} _ {s} = \boldsymbol {W} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {W} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \\ = \boldsymbol {W} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {H} _ {s} (\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \\ = \boldsymbol {W} _ {s} \left(\boldsymbol {I} _ {s} - \left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 1}\right) \left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top} \\ = \boldsymbol {W} _ {s} \left(\left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 1} - \left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 2}\right) \boldsymbol {W} _ {s} ^ {\top} \\ \end{array}
$$
Due to Proposition B.6,  $\mu_i((\boldsymbol{I}_s + \boldsymbol{H}_s)^{-1}) = \min \{\mu, 1\} (1 \pm o(1))$  for all  $i$  with very high probability. Hence to handle the error terms that are not  $\Delta y^\top A_U^{-1}\Delta y$ , it suffices to asymptotically bound  $\Delta y^\top A_U^{-1}M_sA_U^{-1}\Delta y$ . In turn, we can couple this to the quadratic form  $\min \{\mu, 1\} (1 \pm o(1))\Delta y^\top A_U^{-1}W_sW_sA_U^{-1}\Delta y$ . By Proposition C.4, we have with probability at least  $1 - O(1 / nk)$
$$
\begin{array}{l} \min  \left\{\mu , 1 \right\} (1 \pm o (1)) \boldsymbol {\Delta} y ^ {\top} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {W} _ {s} \boldsymbol {W} _ {s} \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {\Delta} y (137) \\ \leq c _ {8} ^ {2} \lambda_ {F} \min  \left\{\mu , 1 \right\} (1 \pm o (1)) \left(n ^ {2 - 2 t - 2 p} + n ^ {r + 1 - t - 2 p}\right) \log (n s k) (138) \\ \leq c _ {8} ^ {2} \min  \left\{1, \mu^ {- 1} \right\} (1 \pm o (1)) \left(n ^ {1 - 2 t - p} + n ^ {r - t - p}\right) \log (n s k) (139) \\ \end{array}
$$
For  $t > 0$ , we claim that the term in Eq. (139) is  $o(n^{1 - t - p})$ , because  $1 - 2t - p < 1 - t - p$  and  $r - t - p < 1 - t - p$ . Hence if  $t > 0$  then by union bound we have with probability at least  $1 - O(1 / nk)$  that
$$
\mathsf {C N} _ {\alpha , \beta , U} ^ {2} = c _ {1 5} ^ {2} (1 \pm o (1)) n ^ {1 - t - p}, \tag {140}
$$
On the other hand, if  $t = 0$ , we only have an issue if  $q + r < 1$ , so that  $\min \left\{1, \mu^{-1}\right\} = 1$ . In this case, the deviation term  $n^{1 - 2t - p} = n^{1 - t - p}$ . However, this won't affect the fact that the upper bound on contamination will still be  $\tilde{O}(n^{1 - t - p})$ . More precisely, this bound concludes by arguing that
$$
\mathbb {C N} _ {\alpha , \beta , U} ^ {2} \leq c _ {1 6} ^ {2} \min  \left\{1, \mu^ {- 1} \right\} n ^ {1 - t - p} \log (n s k), \tag {141}
$$
where  $c_{16}$  is an appropriately defined positive constant.
It turns out we don't have to worry about this edge case at all for the lower bound on  $\mathsf{CN}_{\alpha, \beta, U}$ , because the stated conditions for misclassification imply that  $t > 0$  anyway. This completes the proof of the lemma.
F Obtaining tight misclassification rate
In this section, we will prove Proposition A.5. Let us restate the main proposition and sketch out its proof more formally.
Proof sketch. Note that the  $Z^{(\beta)}$ 's that must outcompete the decaying survival to contamination ratio are jointly Gaussian, as they are projections of a standard Gaussian vector  $\pmb{x}_{\mathrm{test}} \in \mathbb{R}^d$ . Hence if we want to study the probability that  $\max_{\beta} Z^{(\beta)}$  outcompetes  $n^{-u}$ , we have to understand the correlation structure of the  $Z^{(\beta)}$ 's.
We will argue that for  $\beta, \gamma \in [k]$  with  $\alpha, \beta, \gamma$  pairwise distinct, the correlation between  $Z^{(\beta)}$  and  $Z^{(\gamma)}$  is  $\frac{1}{2} \pm o(1)$  with high probability. To that end, we want to look at the correlation (inner product) between the vectors  $\left\{\lambda_j \widehat{\pmb{h}}_{\alpha,\beta}[j]\right\}$  for  $j \notin \{\alpha,\beta\}$  and  $\left\{\lambda_j \widehat{\pmb{h}}_{\alpha,\gamma}[j]\right\}$  for  $j \notin \{\alpha,\gamma\}$ . However, note that by independence of the components of  $\pmb{x}_{\mathrm{test}}$  from every other random variable and the fact that they are mean zero, we have
$$
\mathbb {E} [ \widehat {\pmb {h}} _ {\alpha , \beta} [ \gamma ] \pmb {x} _ {\mathrm {t e s t}} [ \gamma ] \widehat {\pmb {h}} _ {\alpha , \gamma} [ \beta ] \pmb {x} _ {\mathrm {t e s t}} [ \beta ] ] = 0.
$$
Hence it suffices to look at the correlation for  $j \notin \{\alpha, \beta, \gamma\}$ .
We assume WLOG that  $\alpha = 1, \beta = 2, \gamma = 3$ . Let
$$
\Lambda_ {\alpha , \beta} \triangleq \operatorname {d i a g} (1 - \mathbf {1} _ {j = \alpha} - \mathbf {1} _ {j = \beta}) _ {j \in [ d ]} \circ \operatorname {d i a g} (\lambda_ {j}) _ {j \in [ d ]} \in \mathbb {R} ^ {d \times d}
$$
represent the diagonal matrices containing the squared feature weights with indices  $\alpha, \beta$  zeroed out. Next, let  $\pmb{v}_{\alpha,\beta} \in \mathbb{R}^d$  denote the vector with  $\pmb{v}_{\alpha,\beta}[\alpha] = \pmb{v}_{\alpha,\beta}[\beta] = 0$  and  $\pmb{v}_{\alpha,\beta}[j] = \lambda_j \widehat{\pmb{h}}_{\alpha,\beta}[j]$  for  $j \in [d], j \notin \{\alpha,\beta\}$ . Hence  $\pmb{v}_{\alpha,\beta} = \Lambda_{\alpha,\beta}^{1/2} (\widehat{\pmb{f}}_\alpha - \widehat{\pmb{f}}_\beta)$ . Since  $Z^{(\beta)} = \langle \pmb{v}_{\alpha,\beta}, \pmb{x}_{\mathrm{test}} \rangle$ , in order to analyze the correlations between  $Z^{(\beta)}$  and  $Z^{(\gamma)}$ , it suffices to analyze  $\pmb{v}_{\alpha,\beta}$ . Indeed, we will show that the weighted halfspaces  $\Lambda_{\alpha,\beta}^{1/2} \widehat{\pmb{f}}_\alpha \in \mathbb{R}^d$  and  $\Lambda_{\alpha,\beta}^{1/2} \widehat{\pmb{f}}_\beta \in \mathbb{R}^d$  are asymptotically orthogonal.
In other words, we need to show that
$$
\frac {\left\langle \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\alpha} , \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\beta} \right\rangle}{\left\| \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\alpha} \right\| _ {2} \left\| \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\beta} \right\| _ {2}} = o (1)
$$
with probability at least  $1 - O(1 / nk)$ ; we can then union bound against all choices of  $\beta$ . This is the most technically involved part of the proof, and is the content of Proposition F.1.
This in turn will imply (see Lemma F.2) that the maximum (and minimum) correlation between the  $\pmb{v}_{\alpha,\beta}$  for different  $\beta$  is  $\frac{1}{2} \pm o(1)$ . Let  $(\overline{Z}_{\beta})_{\beta \in [k], \beta \neq \alpha}$  be equicorrelated Gaussians with correlation  $\bar{\rho} = \frac{1}{2} + o(1)$ , and  $(\underline{Z}_{\beta})_{\beta \in [k], \beta \neq \alpha}$  be equicorrelated Gaussians with correlation  $\underline{\rho} = \frac{1}{2} - o(1)$ . By Slepian's lemma, for any  $u > 0$ , the probability of  $\max_{\beta} Z^{(\beta)}$  losing to  $n^{-u}$  is sandwiched as
$$
\Pr \left[ \max  _ {\beta} \underline {{Z}} _ {\beta} \leq n ^ {- u} \right] \leq \Pr \left[ \max  _ {\beta} Z ^ {(\beta)} \leq n ^ {- u} \right] \leq \Pr \left[ \max  _ {\beta} \overline {{Z}} _ {\beta} \leq n ^ {- u} \right],
$$
where we have adopted the shorthand  $\max_{\beta}$  to denote  $\max_{\beta \in [k],\beta \neq \alpha}$ .
Theorem 2.1 of Pinasco et al. (2021) shows that jointly gaussian vectors in  $\mathbb{R}^k$  with equicorrelation  $\rho$  lie in the positive orthant with probability  $\Theta(k^{1 - 1 / \rho})$ . In particular, applied to  $\overline{Z}_{\beta}$ , with correlation  $\overline{\rho} = \frac{1}{2} + o(1)$ , we find that
$$
\Pr \left[ \max  _ {\beta} \overline {{Z}} _ {\beta} \leq 0 \right] = \Theta (k ^ {- 1 + o (1)}),
$$
and similarly for  $\underline{Z}_{\beta}$ . Anticoncentration for Gaussian maxima (Chernozhukov et al., 2015, Corollary 1) implies that we can transfer over the bound on  $\operatorname*{Pr}\left[\max_{\beta}\overline{Z}_{\beta}\leq 0\right]$  to a bound on  $\operatorname*{Pr}\left[\max_{\beta}\overline{Z}_{\beta}\leq n^{-u}\right]$  to show that for every  $\epsilon >0$ , we have
$$
\Theta \left(k ^ {- 1 - o (1)}\right) - \epsilon \leq \Pr \left[ \max  _ {\beta} Z ^ {(\beta)} \leq n ^ {- u} \right] \leq \Theta \left(k ^ {- 1 + o (1)}\right) + \epsilon \tag {142}
$$
for sufficiently large  $n$ . Taking the complement of the above event concludes the proof.
F.1 Main results for tight misclassification rates
The main result in this section is the following proposition, which states that the halfspace predictions are asymptotically orthogonal. Its proof is deferred to the subsequent sections.
Proposition F.1. Assume we are in the bi-level ensemble model (Definition 1), the true data generating process is 1-sparse (Assumption 1), and the number of classes scales with  $n$  (i.e.  $t > 0$ ).
For any distinct  $\alpha, \beta \in [k]$ , with probability at least  $1 - O(1 / nk)$ , we have
$$
\frac {\left\langle \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\alpha} , \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\beta} \right\rangle}{\left\| \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\alpha} \right\| _ {2} \left\| \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\beta} \right\| _ {2}} = o (1).
$$
Given Proposition F.1, we can show that the  $Z^{(\beta)}$  have correlations that approach  $\frac{1}{2}$ . The intuitive reason that this correlation approaches  $\frac{1}{2}$  is that the contribution from  $\alpha$  is common. The following lemma formalizes this intuition.
Lemma F.2 (Correlation of relative differences of almost orthogonal vectors). Suppose that we have  $n$  unit vectors  $\pmb{x}_1, \dots, \pmb{x}_n \in \mathbb{R}^d$  such that  $|\langle \pmb{x}_i, \pmb{x}_j \rangle| \leq \gamma$  for  $\gamma > 0$ . Then for any distinct  $i, j, k \in [n]$ , we have
$$
\left| \frac {\langle \pmb {x} _ {j} - \pmb {x} _ {i} , \pmb {x} _ {i} - \pmb {x} _ {k} \rangle}{\| \pmb {x} _ {j} - \pmb {x} _ {i} \| \| \pmb {x} _ {i} - \pmb {x} _ {k} \|} - \frac {1}{2} \right| \leq \frac {2 \gamma}{1 - \gamma}.
$$
Proof. For any  $i \neq j$ , we have  $\| \pmb{x}_i - \pmb{x}_j \|^2 = 2 - 2\langle \pmb{x}_i, \pmb{x}_j \rangle$ . Hence we have
$$
2 - 2 \gamma \leq \left\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right\| ^ {2} \leq 2 + 2 \gamma .
$$
$$
\begin{array}{l} 2 - 2 \gamma \leq \left\| \boldsymbol {x} _ {j} - \boldsymbol {x} _ {k} \right\| ^ {2} \\ = \left\| \pmb {x} _ {i} - \pmb {x} _ {j} \right\| ^ {2} + \left\| \pmb {x} _ {i} - \pmb {x} _ {k} \right\| ^ {2} - 2 \left\langle \pmb {x} _ {j} - \pmb {x} _ {i}, \pmb {x} _ {i} - \pmb {x} _ {k} \right\rangle \\ \leq 4 + 4 \gamma - 2 \left\langle \boldsymbol {x} _ {j} - \boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} - \boldsymbol {x} _ {k} \right\rangle . \\ \end{array}
$$
Since  $\| \pmb{x}_i - \pmb{x}_j\| \geq \sqrt{2 - 2\gamma}$ , we can rearrange and obtain that
$$
\frac {\left\langle \boldsymbol {x} _ {j} - \boldsymbol {x} _ {i} , \boldsymbol {x} _ {i} - \boldsymbol {x} _ {k} \right\rangle}{\| \boldsymbol {x} _ {j} - \boldsymbol {x} _ {i} \| \| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {k} \|} \leq \frac {1 + 3 \gamma}{2 - 2 \gamma}.
$$
Similarly we can reverse the inequalities and get
$$
2 + 2 \gamma \geq 4 - 4 \gamma - 2 \left\langle \boldsymbol {x} _ {j} - \boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} - \boldsymbol {x} _ {k} \right\rangle ,
$$
$$
\frac {\left\langle \boldsymbol {x} _ {j} - \boldsymbol {x} _ {i} , \boldsymbol {x} _ {i} - \boldsymbol {x} _ {k} \right\rangle}{\| \boldsymbol {x} _ {j} - \boldsymbol {x} _ {i} \| \| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {k} \|} \geq \frac {1 - 3 \gamma}{2 + 2 \gamma}.
$$
Combining Proposition F.1 with Lemma F.2 yields the following formal statement about the correlations between the  $Z^{(\beta)}$ .
Lemma F.3 (Asymptotic correlation of relative survivals). For any distinct  $\alpha, \beta, \beta' \in [k]$ , under the same assumptions as Proposition F.1, as  $n \to \infty$ , with probability at least  $1 - O(1/n)$ , we have
$$
\left| \mathbb {E} [ Z ^ {(\beta)} Z ^ {(\beta^ {\prime})} ] - \frac {1}{2} \right| \leq o (1).
$$
As a consequence, the asymptotic correlation between the relative survivals approaches  $\frac{1}{2}$  at a polynomial rate.
Proof. Plugging in the result of Proposition F.1 into Lemma F.2, we obtain the stated result.
F.2 Lower bounding the denominator
Let us now begin to prove Proposition F.1. The first step is to bound the denominator of the normalized correlation. Writing out the definitions, we have
$$
\begin{array}{l} \left\| \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\alpha} \right\| ^ {2} = \sum_ {j \notin \{\alpha , \beta \}} \lambda_ {j} ^ {2} \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\alpha} \\ = \lambda_ {F} ^ {2} \sum_ {j \notin \{\alpha , \beta \}, j \in [ s ]} \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\alpha} + \lambda_ {U} ^ {2} \sum_ {j > s} \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\alpha} \\ \end{array}
$$
Note that these two terms are respectively analogous to  $\mathsf{CN}_{\alpha,\beta,L}^{2} + \mathsf{CN}_{\alpha,\beta,F}^{2}$  and  $\mathsf{CN}_{\alpha,\beta,U}^{2}$ . In fact, the proofs of the lower bounds for contamination essentially transfer over verbatim to the lower bounds on the denominator, because Hanson-Wright implies that we can show that  $\left\|A_{-s}^{-1}\pmb{y}_{\alpha}\right\|_2$  concentrates the same way that  $\left\|A_{-s}^{-1}\Delta y\right\|_2$  does. In essence, we are able to show the following proposition.
Proposition F.4 (Lower bound on norm of scaled halfspaces). Under the same assumptions as Proposition F.1, for any  $\alpha, \beta \in [k]$ , with  $\alpha \neq \beta$ , with probability at least  $1 - O(1 / nk)$ , we have
$$
\left\| \Lambda_ {\alpha , \beta} ^ {1 / 2} \widehat {\boldsymbol {f}} _ {\alpha} \right\| ^ {2} \geq \min  \left\{1, \mu^ {- 2} \right\} \Omega \left(n ^ {r - t - 1}\right) + \Omega \left(n ^ {1 - t - p}\right).
$$
F.3 Upper bounding the numerator: the unnormalized correlation
We now turn to the more involved part of the bound: proving an upper bound on the numerator. As before, we can bound the split up the numerator into favored and unfavored terms. For each term, we will show that it is dominated by the denominator, in the precise sense that each term is
$$
o (\min \left\{1, \mu^ {- 2} \right\} n ^ {r - t - 1} + n ^ {1 - t - p}).
$$
Now, let's look at the numerator, which is the bilinear form
$$
\begin{array}{l} \lambda_ {F} ^ {2} \sum_ {j \notin \{\alpha , \beta \}, j \in [ s ]} \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\beta} + \lambda_ {U} ^ {2} \sum_ {j > s} \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\beta}. (143) \\ = \underbrace {\lambda_ {F} ^ {2} \left\langle \mathbf {Z} _ {L} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {y} _ {\alpha} , \mathbf {Z} _ {L} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {y} _ {\beta} \right\rangle} _ {\operatorname {c o r} _ {\alpha , \beta , L}} + \underbrace {\lambda_ {F} ^ {2} \left\langle \mathbf {Z} _ {F} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {y} _ {\alpha} , \mathbf {Z} _ {F} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {y} _ {\beta} \right\rangle} _ {\operatorname {c o r} _ {\alpha , \beta , F}} + \underbrace {\lambda_ {U} ^ {2} \sum_ {j > s} \mathbf {y} _ {\alpha} ^ {\top} \mathbf {A} ^ {- 1} z _ {j} z _ {j} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {y} _ {\beta}} _ {\operatorname {c o r} _ {\alpha , \beta , U}} (144) \\ \end{array}
$$
We refer to the first term as the label defining correlation  $\mathsf{cor}_{\alpha,\beta,L}$ , the second term as the favored correlation  $\mathsf{cor}_{\alpha,\beta,F}$ , and the last term as the unfavored correlation  $\mathsf{cor}_{\alpha,\beta,U}$ . Here, we abuse terminology slightly and refer to these inner products as correlations, even though strictly speaking, they are unnormalized.
F.3.1 Bounding the favored correlation
We now bound the correlation coming from the favored features; we will ultimately show that its contribution is min  $\{1,\mu^{-2}\} o(n^{r - t - 1})$ . Recall that  $W_{R}\in \mathbb{R}^{n\times (s - 2)}$  is the weighted feature matrix
for the  $s - 2$  favored features aside from  $\alpha$  and  $\beta$ . Then the label-defining+favored correlation  $\mathsf{cor}_{\alpha,\beta,L} + \mathsf{cor}_{\alpha,\beta,F}$  can be written succinctly as
$$
\lambda_ {F} ^ {2} \left\langle \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle . \tag {145}
$$
Why should we be able to bound this better than Cauchy-Schwarz? Intuitively, although there is a mild dependence between  $\pmb{y}_{\alpha}$  and  $\pmb{y}_{\beta}$ , it is not strong enough to cause  $Z_{R}^{\top}A^{-1}\pmb{y}_{\alpha}$  and  $Z_{R}^{\top}A^{-1}\pmb{y}_{\beta}$  to point in the same direction.
To formalize this argument, we will first follow the strategy to bound the favored contamination. In particular, using the push-through form of Woodbury (Lemma B.2) we see that
$$
\left\langle \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle = \boldsymbol {y} _ {\alpha} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {Z} _ {R} \left(\boldsymbol {I} _ {s - 2} + \boldsymbol {H} _ {R}\right) ^ {- 2} \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\beta}. \tag {146}
$$
Now, we can apply Proposition B.6 to replace  $(\mathbf{I}_{s-2} + \mathbf{H}_R)^{-2}$  with  $\min \left\{\mu^2, 1\right\} (\mathbf{I}_{s-2} + \mathbf{E})$ , where  $\| \mathbf{E}\|_2 = O(n^{-\kappa_{11}})$  with extremely high probability. Cauchy-Schwarz yields that
$$
\begin{array}{l} \left\langle \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle (147) \\ \leq \min  \left\{\mu^ {2}, 1 \right\} \left\langle \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\alpha}, \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\beta} \right\rangle (148) \\ + \min  \left\{\mu^ {2}, 1 \right\} \| \boldsymbol {E} \| _ {2} \left\| \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\beta} \right\| _ {2} \left\| \boldsymbol {Z} _ {R} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\alpha} \right\| _ {2}. (149) \\ \end{array}
$$
The term in Eq. (149) can be bounded in the same way that we bounded the favored contamination. Indeed, since we can swap in  $\pmb{y}_{\alpha}$  and  $\pmb{y}_{\beta}$  with  $\Delta y$ , the argument that proved the bounds on  $\| Z_R^\top A_{-R}^{-1}\Delta y\|_2$  port over immediately. After using the scaling for  $\lambda_F$  and the fact that  $\| E\|_2 = O(n^{-\kappa_{11}})$ , we conclude that this Cauchy-Schwarz error term is at most min  $\{1,\mu^{-2}\} o(n^{r - t - 1})$  with probability at least  $1 - O(1 / nk)$ .
Let us now turn to the term in Eq. (148). As in the proof for the lower bound for favored contamination Lemma E.2, to get better concentration than Cauchy-Schwarz, we want to use Bernstein. We can rewrite it suggestively as
$$
\sum_ {j \in [ s ] \backslash \{\alpha , \beta \}} \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\alpha}\right) \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\beta}\right) \tag {150}
$$
We cannot immediately power through with the calculation, because  $\mathbf{A}_{-R}^{-1}$  is not independent of  $\mathbf{y}_{\alpha}$  or  $\mathbf{y}_{\beta}$ . The main idea is to again use Woodbury and show that the dependent portions contribute negligibly to  $\mathbf{z}_j^\top \mathbf{A}_{-R}^{-1} \mathbf{y}_{\alpha}$ . Therefore the dependent contributions get dominated by the lower bound on the correlation.
As in the proof for bounding the favored contamination, we can further define  $\mathbf{W}_{\alpha,\beta} = [\mathbf{w}_{\alpha} \quad \mathbf{w}_{\beta}]$  and  $\mathbf{H}_{\alpha,\beta}^{(s)} = \mathbf{W}_{\alpha,\beta}^{\top} \mathbf{A}_{-s}^{-1} \mathbf{W}_{\alpha,\beta}$ . Then we can deduce from another application of Woodbury that
$$
\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\alpha} = \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha} - \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} \left(\boldsymbol {I} _ {2} + \boldsymbol {H} _ {\alpha , \beta}\right) ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}. \tag {151}
$$
Again, we can argue that the second term is bounded in magnitude by
$$
\min  \left\{1, \mu^ {- 1} \right\} n ^ {\frac {1}{2} - t - p} \log (n s) = \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p} \cdot n ^ {- \kappa_ {6}}\right), \tag {152}
$$
because  $t > 0$ . Since Hanson-Wright (Corollary C.3) implies that  $\pmb{z}_j^\top \pmb{A}_{-s}^{-1}\pmb{y}_\alpha = \tilde{O}(n^{\frac{1 - t}{2} - p})$ , this implies that  $\pmb{z}_j^\top \pmb{A}_{-R}^{-1}\pmb{y}_\alpha = \tilde{O}(n^{\frac{1 - t}{2} - p})$ , and similarly for  $\beta$ . Hence we have
$$
\begin{array}{l} \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\alpha}\right) \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {y} _ {\beta}\right) (153) \\ \leq \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha} + \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p - \kappa_ {6}}\right)\right) \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta} + \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p - \kappa_ {6}}\right)\right) (154) \\ \leq \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}\right) \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta}\right) + \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p - \kappa_ {6}}\right) \tilde {O} \left(n ^ {\frac {1 - t}{2} - p}\right) (155) \\ \leq \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}\right) \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta}\right) + \min  \left\{1, \mu^ {- 1} \right\} o \left(n ^ {1 - t - 2 p}\right). (156) \\ \end{array}
$$
This implies that we can rewrite Eq. (150) as
$$
\left(\sum_ {j \in [ s ] \backslash \{\alpha , \beta \}} \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\alpha}\right) \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {y} _ {\beta}\right)\right) \pm \min  \{1, \mu^ {- 1} \} o \left(n ^ {r + 1 - t - 2 p}\right). \tag {157}
$$
Let us argue that the second term in Eq. (157) will be negligible compared to the denominator, which is  $\min \left\{1,\mu^{-2}\right\} \Omega (n^{r - t - 1})$ . Tracing back up the stack, we see that its contribution to the favored correlation will be at most