SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $\pmb{A}_{-k} \triangleq \sum_{i \notin [k]} \pmb{w}_i \pmb{w}_i^\top$  denote the leave-  $k$ -out Gram matrix which removes the  $k$  label-defining features. Similarly, let  $\pmb{A}_{-F} \triangleq \sum_{i \notin [s] \setminus [k]} \pmb{w}_i \pmb{w}_i^\top \in \mathbb{R}^{n \times n}$  to denote leave-  $(s - k)$ -out Gram matrix which removes the favored but not label-defining features. Finally, let  $\pmb{A}_U \triangleq \sum_{i \notin [s]} \pmb{w}_i \pmb{w}_i^\top \in \mathbb{R}^{n \times n}$  denote the leave-  $s$ -out matrix which only retains the unfavored features. We will also sometimes write  $A_{-s}$  instead of  $A_U$  to emphasize that the  $s$  favored features have all been removed.
Define the so-called hat matrices by
$$
\boldsymbol {H} _ {k} \triangleq \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {W} _ {k} \in \mathbb {R} ^ {k \times k} \tag {53}
$$
$$
\boldsymbol {H} _ {F} \triangleq \boldsymbol {W} _ {F} ^ {\top} \boldsymbol {A} _ {- F} ^ {- 1} \boldsymbol {W} _ {F} \in \mathbb {R} ^ {(s - k) \times (s - k)}. \tag {54}
$$
These hat matrices appear in the Woodbury inversion formula. For the sake of notational compactness, define
$$
\boldsymbol {M} _ {k} \triangleq \boldsymbol {W} _ {k} \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1} \boldsymbol {W} _ {k} ^ {\top} \in \mathbb {R} ^ {n \times n} \tag {55}
$$
$$
\boldsymbol {M} _ {F} \triangleq \boldsymbol {W} _ {F} \left(\boldsymbol {I} _ {s - k} + \boldsymbol {H} _ {F}\right) ^ {- 1} \boldsymbol {W} _ {F} ^ {\top} \in \mathbb {R} ^ {n \times n}. \tag {56}
$$
The Woodbury inversion formula yields
$$
\boldsymbol {A} ^ {- 1} = \left(\boldsymbol {W} _ {k} \boldsymbol {W} _ {k} ^ {\top} + \boldsymbol {A} _ {- k}\right) ^ {- 1} \tag {57}
$$
$$
= \boldsymbol {A} _ {- k} ^ {- 1} - \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {W} _ {k} \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1} \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \tag {58}
$$
$$
= \boldsymbol {A} _ {- k} ^ {- 1} - \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {M} _ {k} \boldsymbol {A} _ {- k} ^ {- 1}. \tag {59}
$$
Left multiplying (58) by  $\mathbf{W}_k^\top$  yields
$$
\boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} ^ {- 1} = \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} - \boldsymbol {H} _ {k} \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1} \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \tag {60}
$$
$$
= \left(\boldsymbol {I} _ {k} - \boldsymbol {H} _ {k} \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1}\right) \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \tag {61}
$$
$$
= \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1} \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1}. \tag {62}
$$
We can derive completely analogous identities using  $A_{-F}^{-1}$  instead of  $A_{-k}^{-1}$ . The above exposition is summarized by the following lemma.
Lemma B.2. We have
$$
\boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y = \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1} \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \tag {63}
$$
$$
\boldsymbol {W} _ {F} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y = \left(\boldsymbol {I} _ {s - k} + \boldsymbol {H} _ {F}\right) ^ {- 1} \boldsymbol {W} _ {F} ^ {\top} \boldsymbol {A} _ {- F} ^ {- 1} \boldsymbol {\Delta} y. \tag {64}
$$
Lemma B.2 is quite powerful. Indeed, consider the action of the linear operator  $W_{k}^{\top}A^{-1}:\mathbb{R}^{n}\to \mathbb{R}^{k}$  on  $\Delta y$ . The action is identical to that of the linear operator  $W_{k}^{\top}A_{-k}^{-1}:\mathbb{R}^{n}\to \mathbb{R}^{k}$ , up to some invertible transformation. This new linear operator is nice because  $A_{-k}^{-1}$  is independent of  $W_{k}$  and  $\Delta y$ , as it removes all of the label-defining features. Reclaiming independence sets the stage for using our variant of Hanson-Wright.
How does the invertible operator  $(\mathbf{I}_k + \mathbf{H}_k)^{-1}$  act? Our general strategy is to show that  $\mathbf{H}_k$  is itself close to a scaled identity matrix, i.e.  $\mathbf{H}_k \approx \nu \mathbf{I}_k$  for an appropriately defined  $\nu$ . Then for any  $i \in [k]$ , we have that
$$
\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y \approx (1 + \nu) ^ {- 1} \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y.
$$
Of course, there will be some error in this approximation, as  $H_{k}$  is not exactly equal to  $\nu I_{k}$ . Nevertheless, we can bound away the error that arises from this approximation.
B.3 Concentration of spectrum
As foreshadowed in the previous section, we will leverage the fact that the hat matrices such as  $H_{k}$  are close to a scaled identity. To formalize this, we appeal to random matrix theory and show that the spectra of various random matrices are very close to being flat (i.e. all eigenvalues are within  $1 + o(1)$  of each other). To that end, we present the following standard characterization of the spectrum of a standard Wishart matrix, which is Equation 2.3 in Rudelson and Vershynin (2010).
Lemma B.3 (Concentration of spectrum for Wishart matrices). Let  $M \in \mathbb{R}^{M \times m}$  with  $M > m$  be a real matrix with iid  $N(0,1)$  entries. Then for any  $\epsilon \geq 0$ , we have with probability at least  $1 - 2e^{-\epsilon^2 / 2}$  that
$$
\sqrt {M} - \sqrt {m} - \epsilon \leq \sigma_ {\min } (M) \leq \sigma_ {\max } (M) \leq \sqrt {M} + \sqrt {m} + \epsilon . \tag {65}
$$
In other words, the singular values of  $M$  satisfy subgaussian concentration.
Since  $\mu_{m}(M^{\top}M) = \sigma_{\min}(M)^{2}$  and  $\mu_{1}(M^{\top}M) = \sigma_{\max}(M)^{2}$ , we can conclude that if  $m = o(M)$ , then for any  $\epsilon > 0$  we have
$$
M - 2 \sqrt {M m} - \epsilon + o (\sqrt {M m}) \leq \mu_ {m} \left(\boldsymbol {M} ^ {\top} \boldsymbol {M}\right) \leq \mu_ {1} \left(\boldsymbol {M} ^ {\top} \boldsymbol {M}\right) \leq M + 2 \sqrt {M m} + \epsilon + o (\sqrt {M m}), \tag {66}
$$
with probability at least  $1 - 2e^{-\epsilon^2 /2}$
On the other hand, consider  $MM^{\top} \in \mathbb{R}^{M \times M}$ . Its spectrum is just that of  $M^{\top}M \in \mathbb{R}^{m \times m}$  with an additional  $M - m$  zeros corresponding to the fact that  $m < M$ .
We can use Lemma B.3 to prove concentration of the spectrum of the various matrices introduced in Appendix B.2. Let us summarize some convenient forms of these results; their proofs are deferred to Appendix H.
Proposition B.4 (Gram matrices have a flat spectrum). Recall that  $\mathbf{A}_U = \mathbf{A}_{-s} = \sum_{j > s}\lambda_j\mathbf{z}_j\mathbf{z}_j^\top \in \mathbb{R}^{n\times n}$  is the unfavored Gram matrix and  $\mathbf{A}_{-k} = \sum_{j > k}\lambda_j\mathbf{z}_j\mathbf{z}_j^\top \in \mathbb{R}^{n\times n}$  is the leave-k-out Gram matrix.
Then the following hold with probability at least  $1 - 2e^{-n} - 2e^{-\sqrt{n}}$ ,
(a) For all  $i\in [n]$ , we have  $\mu_{i}(\mathbf{A}_{U}) = n^{p}(1\pm O(n^{-\kappa_{7}}))$
(b) For all  $i \in [s - k]$ , we have
$$
\mu_ {i} \left(\boldsymbol {A} _ {- k}\right) = \left(1 + \mu^ {- 1}\right) n ^ {p} \left(1 \pm O \left(n ^ {- \kappa_ {9}}\right)\right), \tag {67}
$$
where  $\kappa_{9}$  is a positive constant. Moreover, for all  $i\in [n]\setminus [s - k]$ , we have
$$
\mu_ {i} \left(\boldsymbol {A} _ {- k}\right) = n ^ {p} \left(1 \pm O \left(n ^ {- \kappa_ {7}}\right)\right), \tag {68}
$$
where  $\kappa_7$  is a positive constant.
As a simple corollary, we can obtain the following cruder bounds on the trace and spectral norm of  $A_{-k}^{-1}$  and  $A_{-s}^{-1}$ .
Corollary B.5 (Trace and spectral norm of  $A_{-k}^{-1}$ ). In the bi-level model, with probability at least  $1 - 2e^{-n}$ , we have
$$
\operatorname {T r} \left(\boldsymbol {A} _ {U} ^ {- 1}\right) = n ^ {1 - p} \left(1 \pm O \left(n ^ {- \kappa_ {7}}\right)\right) \sqrt {\log k} \tag {69}
$$
$$
\operatorname {T r} \left(\boldsymbol {A} _ {- k} ^ {- 1}\right) = n ^ {1 - p} \left(1 \pm O \left(n ^ {- \kappa_ {3}}\right)\right) \sqrt {\log k} \tag {70}
$$
$$
\left. \max  \left\{\left\| \boldsymbol {A} _ {- k} ^ {- 1} \right\| _ {2}, \left\| \boldsymbol {A} _ {U} ^ {- 1} \right\| _ {2} \right\} \leq c _ {2} n ^ {- p}, \right. \tag {71}
$$
where  $c_{2},\kappa_{7}$  ,and  $\kappa_{3}$  are all positive constants.
Proof. We prove the claim for  $A_{-k}^{-1}$ ; the proof for  $A_U^{-1}$  is similar or easier because  $A_U^{-1}$  has a flat spectrum (Proposition B.4).
If  $q + r < 1$ , the upper bound for the spectral norm similarly follows. For the trace bounds, we can apply Proposition B.4, we have
$$
\begin{array}{l} \operatorname {T r} \left(\boldsymbol {A} _ {- k} ^ {- 1}\right) = \left(n - n ^ {r} + n ^ {t}\right) n ^ {- p} \left(1 \pm O \left(n ^ {- \kappa_ {7}}\right)\right) + \left(n ^ {r} - n ^ {t}\right) \cdot \left(1 + \mu^ {- 1}\right) n ^ {- p} \left(1 \pm O \left(n ^ {- \kappa_ {9}}\right)\right) (72) \\ = n ^ {1 - p} \left(1 \pm O \left(n ^ {- \kappa_ {1}}\right)\right) (73) \\ \end{array}
$$
$$
\kappa_ {1} = \min  \left\{r - 1, 2 - q - 2 r \right\} > 0,
$$
as  $q + 2r <   2(q + r) <   2$  by assumption.
On the other hand, the claim is obviously true when  $q + r > 1$ , as the entire spectrum of  $\mathbf{A}_{-k}^{-1}$  is  $(1 \pm O(n^{-\kappa_2}))n^{-p}$  with an appropriately defined positive constant  $\kappa_2$ . The spectral norm bound follows by defining  $c_2$  to be any positive constant greater than 1 which absorbs the  $o(1)$  deviation terms in the spectrum.
The proof concludes by setting  $\kappa_{3} = \min \left\{\kappa_{1},\kappa_{2}\right\}$
Finally, we have the following proposition which controls the spectrum of hat matrices such as  $\pmb{H}_k \triangleq \pmb{W}_k^\top \pmb{A}_{-k}^{-1} \pmb{W}_k \in \mathbb{R}^{k \times k}$ . The intuition is that even though the spectrum of  $\pmb{A}_{-k}^{-1}$  may be spiked, the spectrum of  $\pmb{W}_k^\top \pmb{A}_{-k}^{-1} \pmb{W}_k$  is ultimately flat because we are taking an extremely low dimensional projection which is unlikely to see significant contribution from the spiked portion of  $\pmb{A}_{-k}^{-1}$ .
In fact, we can prove a more general statement, which will be useful for us in the proof. Let  $\varnothing \neq T \subseteq S \subseteq [s]$ ; here  $T$  and  $S$  index nonempty subsets of the  $s$  favored features. Then we can define  $W_{T}$  to be the matrix of weighted features in  $T$  and the leave-  $T$ -out Gram matrix
$$
\boldsymbol {A} _ {- T} \triangleq \sum_ {j \notin T} \lambda_ {j} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}. \tag {74}
$$
Now define the  $(T, S)$  hat matrix as  $\mathbf{H}_{T,S} \triangleq \mathbf{W}_T^\top \mathbf{A}_{-S}^{-1} \mathbf{W}_T$ . Evidently we have  $\mathbf{H}_k = \mathbf{H}_{[k],[k]}$ , so our notion is more general. The full proof is deferred to Appendix H.
Proposition B.6 (Generalized hat matrices are flat). Assume we are in the bi-level ensemble Definition 1. For any nonempty  $T \subseteq S \subseteq [s]$ , with probability at least  $1 - 2e^{-\sqrt{n}} - 2e^{-n}$ , we have all the eigenvalues tightly controlled:
$$
\mu_ {i} \left(\left(\boldsymbol {I} _ {| T |} + \boldsymbol {H} _ {T, S}\right) ^ {- 1}\right) = \min  \left\{\mu , 1 \right\} \left(1 \pm c _ {T, S} n ^ {- \kappa_ {1 1}}\right). \tag {75}
$$
where  $c_{T,S}$  and  $\kappa_{11}$  are positive constants that depend on  $|T|$  and  $|S|$ .
C Utility bounds: applying the tools
Wishart concentration allows us to tightly bound the hat matrix and pass to studying bilinear forms of the form  $\boldsymbol{w}_i^\top \boldsymbol{A}_{-k}^{-1}\Delta y$  rather than  $\boldsymbol{w}_i^\top \boldsymbol{A}^{-1}\Delta y$ . Since  $\boldsymbol{A}_{-k}^{-1}$  is independent of  $\boldsymbol{W}_k$  and  $\Delta y$ , we can condition on  $\boldsymbol{A}_{-k}^{-1}$  and then apply Hanson-Wright (Theorem 4.1) to these bilinear forms for every realization of  $\boldsymbol{A}_{-k}^{-1}$ . In this section, we will explicitly calculate the scaling of the typical value of these bilinear forms using the bi-level ensemble scaling; these will prove to be useful throughout the rest of the paper.
We first state the following proposition which bounds the correlation between the relevant label-defining features and the label vectors; it is a combination of Propositions D.5 and D.6 in (Subramanian et al., 2022).
Proposition C.1. For any distinct  $\alpha, \beta \in [k]$ , we have
$$
\frac {1}{\sqrt {\pi \ln 2}} \cdot \frac {n}{k} \cdot \sqrt {\ln k} \leq \mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {y} _ {\alpha} \right] \leq \sqrt {2} \cdot \frac {n}{k} \cdot \sqrt {\ln k} \tag {76}
$$
$$
- \sqrt {2} \cdot \frac {n}{k} \cdot \frac {1}{k - 1} \cdot \sqrt {\ln k} \leq \mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {y} _ {\beta} \right] \leq - \frac {1}{\sqrt {\pi \ln 2}} \cdot \frac {n}{k} \cdot \frac {1}{k - 1} \cdot \sqrt {\ln k} \tag {77}
$$
With the above proposition in hand, we can prove the following lemma which gives concentration of the bilinear forms that we study.
Lemma C.2. Let  $i \in [d]$  and  $\Delta y = y_{\alpha} - y_{\beta}$  where  $\alpha, \beta \in [k]$  and  $\beta \neq \alpha$ . Let  $M \in \mathbb{R}^{n \times n}$  be a (random) matrix which is independent of  $z_i$  and  $\Delta y$ . Then conditioned on  $M$ , with probability at least  $1 - 1 / nk$ ,
$$
\left| \boldsymbol {z} _ {i} ^ {\top} M \boldsymbol {\Delta} y - \mathbb {E} [ \boldsymbol {z} _ {i} ^ {\top} M \boldsymbol {\Delta} y | M ] \right| \leq c _ {3} \sqrt {\frac {n}{k}} \| M \| _ {2} \sqrt {\log (n k)},
$$
and the same holds with  $\Delta y$  replaced with  $\mathbf{y}_{\alpha}$ . Here,  $c_{3}$  is an appropriately chosen universal positive constant.
(1) For any distinct  $\alpha, \beta \in [k]$ , we have
$$
\mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {M} \Delta y | \boldsymbol {M} \right] = c _ {7} \frac {\sqrt {\log k}}{k} \operatorname {t r} (\boldsymbol {M}) = - \mathbb {E} \left[ \boldsymbol {z} _ {\beta} ^ {\top} \boldsymbol {M} \Delta y \right] \tag {78}
$$
$$
\mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {M} \boldsymbol {y} _ {\alpha} | \boldsymbol {M} \right] = c _ {4} \frac {\sqrt {\log k}}{k} \operatorname {t r} (\boldsymbol {M}), \tag {79}
$$
where  $c_{7}$  and  $c_{4}$  are positive constants.
(2) For  $i\in [d]\setminus \{\alpha ,\beta \}$ , we have
$$
\mathbb {E} \left[ \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {M} \Delta y | \boldsymbol {M} \right] = 0. \tag {80}
$$
(3) For  $i\in [d]\setminus \{\alpha \}$ , we have
$$
\mathbb {E} \left[ \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {M} \boldsymbol {y} _ {\alpha} \mid \boldsymbol {M} \right] = - c _ {5} \frac {\sqrt {\log k}}{k (k - 1)}, \tag {81}
$$
where  $c_{5}$  is a positive constant.
Proof. Let us check the conditions for our new variant of Hanson-Wright with soft sparsity (Theorem 4.1). We want to apply it to the random vectors  $(\boldsymbol{z}_i, \Delta y) = (\boldsymbol{z}_i[j], \Delta y[j])_{j=1}^n$ . Some of the hypotheses are immediate by definition. Evidently,  $(\boldsymbol{z}_i[j], \Delta y[j])$  are independent across  $j$ , and are mean zero. Since  $\boldsymbol{z}_i[j] \sim N(0,1)$ , it is subgaussian with parameter at most  $K = 2$ . For the bounded and soft sparsity assumption, we clearly have  $|\Delta y[j]| \leq 1$  and  $\boldsymbol{y}_{\alpha}[j] \leq 1$  almost surely. Also, since  $\Delta y[j]^2 \sim \mathsf{Ber}\left(\frac{2}{k}\right)$ , we have  $\mathbb{E}[\Delta y[j]^2] = \frac{2}{k}$ . Similarly,  $\mathbb{E}[\boldsymbol{y}_{\alpha}[j]^2] = \frac{1}{k} (1 - \frac{1}{k})^2 + (1 - \frac{1}{k}) \frac{1}{k^2} \leq \frac{2}{k}$ .
The more complicated condition is the subgaussianity of  $\pmb{z}_i[j]$  conditioned on the value of  $\Delta y[j]$  or  $\pmb{y}_{\alpha}[j]$ . Regardless of whether we're conditioning on  $\Delta y$  or  $\pmb{y}_{\alpha}$ , it suffices to instead prove that  $\pmb{z}_i[j]$  is subgaussian conditioned on whether feature  $i$  won the competition for datapoint  $j$ . First, suppose  $i$  won, i.e.  $\pmb{y}_i^{\mathrm{oh}}[j] = 1$ . Then the Borell-TIS inequality (Adler et al., 2007, Theorem 2.1.1) implies that  $\pmb{z}_i[j]$  satisfies a subgaussian tail inequality. By the equivalent conditions for subgaussianity Vershynin (2018, Proposition 2.5.2), it follows that  $\pmb{z}_i[j] - \mathbb{E}[\pmb{z}_i[j]|\pmb{y}_i^{\mathrm{oh}}[j] = 1]$  conditionally has subgaussian norm bounded by some absolute constant  $K$ . If  $i$  doesn't win (or doesn't participate in the competition), then Proposition D.2 in Subramanian et al. (2022) implies that  $\pmb{z}_i[j] - \mathbb{E}[\pmb{z}_i[j]|\pmb{y}_i^{\mathrm{oh}}[j] = 0]$  conditionally has subgaussian norm bounded by 6.
Finally, since  $M$  is independent of  $z_{i}$  and  $\Delta y$ , we can condition on  $M$  and apply Theorem 4.1 to the bilinear form for every realization of  $M$ .
Hence, we conclude that with probability at least  $1 - 1 / nk$  we have
$$
\left| \boldsymbol {z} _ {i} ^ {\top} M \Delta y - \mathbb {E} \left[ \boldsymbol {z} _ {i} ^ {\top} M \Delta y \mid M \right] \right| \leq c _ {3} \sqrt {\frac {n}{k}} \| M \| _ {2} \sqrt {\log (n k)}, \tag {82}
$$
where  $c_{3}$  is an appropriately chosen absolute constant based on  $K$  and the constant  $c$  defined in Theorem 4.1.
Now we can compute  $\mathbb{E}[z_i^\top M\Delta y|M]$  to prove the rest of the theorem. If  $i = \alpha$  , we have
$$
\mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} M \Delta y | M \right] = \operatorname {t r} \left(M \mathbb {E} \left[ \Delta y \boldsymbol {z} _ {\alpha} ^ {\top} \right]\right).
$$
Let us now compute  $\mathbb{E}[\Delta y\pmb{z}_{\alpha}^{\top}]$ . From Eq. (76) in Proposition C.1, we have  $\mathbb{E}[y_{\alpha}\pmb{z}_{\alpha}^{\top}] = c_4\frac{\sqrt{\log k}}{k}\pmb{I}_n$  where  $\frac{1}{\sqrt{\pi\log 2}}\leq c_4\leq \sqrt{2}$ . Similarly, we have  $\mathbb{E}[y_{\beta}\pmb{z}_{\alpha}^{\top}] = -c_5\frac{\sqrt{\log k}}{k(k - 1)}\pmb{I}_n$  where  $\frac{1}{\sqrt{\pi\log 2}}\leq c_5\leq \sqrt{2}$ . It follows that  $\mathbb{E}[\Delta y\pmb{z}_{\alpha}^{\top}] = \Theta \left(\frac{\sqrt{\log k}}{k}\right)\pmb{I}_n$ .
For  $i\in [d]\setminus \{\alpha ,\beta \}$ , by symmetry we obtain  $\mathbb{E}[\pmb {y}_\alpha \pmb {z}_i^\top ] = \mathbb{E}[\pmb {y}_\beta \pmb {z}_i^\top ]$ . This implies  $\mathbb{E}[\Delta y\pmb {z}_i^\top ] = \mathbb{E}[\pmb {y}_\alpha \pmb {z}_i^\top ] - \mathbb{E}[\pmb {y}_\beta \pmb {z}_i^\top ] = 0$ , so we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \boldsymbol {z} _ {i} ^ {\top} M \Delta y | M \right] = \operatorname {t r} \left(M \mathbb {E} \left[ \Delta y \boldsymbol {z} _ {i} ^ {\top} \right]\right) (83) \\ = 0. (84) \\ \end{array}
$$
Plugging in the bi-level scaling, we obtain the following corollary.
Corollary C.3 (Asymptotic concentration of bilinear forms). In the bi-level model, for any  $i \in [k]$ , we have with probability at least  $1 - O(1 / nk)$  that
$$
\left| \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y - \mathbb {E} \left[ \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right] \right| \leq c _ {6} n ^ {\frac {1 - t}{2} - p} \sqrt {\log (n k)}.
$$
(1) For any distinct  $\alpha, \beta \in [k]$ ,
$$
\mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right] = c _ {7} n ^ {1 - t - p} \left(1 \pm O \left(n ^ {- \kappa_ {3}}\right)\right) \sqrt {\log k} = - \mathbb {E} \left[ \boldsymbol {z} _ {\beta} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right] \tag {85}
$$
The same statements hold (with different constants) if we replace  $A_{-k}^{-1}$  with  $A_{-s}^{-1}$ .
Proof. From Corollary B.5, we have  $\left\| A_{-k}^{-1}\right\|_2 \leq c_6 n^{-p}$ , where  $c_6$  is an appropriately chosen universal positive constant based on  $c_3$ . Recall that  $A_{-k}$  is obtained by removing the  $k$  label-defining features, so in particular  $A_{-k}^{-1}$  is independent of  $(z_i, \Delta y)$  for  $i \in [k]$ . Hence, the conditions for Lemma C.2 are satisfied. Then applying the union bound for the spectral norm bound on  $A_{-k}^{-1}$ , we see that with probability at least  $1 - O(1 / nk)$ , the deviation term from Hanson-Wright is at most  $c_6 n^{\frac{1}{2} - p} \sqrt{\log(nk)}$ .
We now turn to calculating the asymptotic scalings for the expectations. From Lemma C.2, we know that  $\mathbb{E}[z_{\alpha}^{\top}A_{-k}^{-1}\Delta y|A_{-k}^{-1}] = c_7\frac{\sqrt{\log k}}{k}\operatorname{tr}(A_{-k}^{-1})$ . Applying the high probability bound on  $\operatorname{tr}(A_{-k}^{-1})$  from Corollary B.5, we obtain that with probability at least  $1 - O(1 / nk)$  that
$$
\begin{array}{l} - \mathbb {E} \left[ \boldsymbol {z} _ {\beta} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right] = \mathbb {E} \left[ \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right] = c _ {7} n ^ {- t} n ^ {1 - p} \left(1 \pm O \left(n ^ {- \kappa_ {3}}\right)\right) \sqrt {\log k} (86) \\ = c _ {7} n ^ {1 - t - p} \left(1 \pm O \left(n ^ {- \kappa_ {3}}\right)\right) \sqrt {\log k} (87) \\ \end{array}
$$
where in the second line we have applied Corollary B.5 and  $c_7$  is an appropriately chosen positive constant. This proves (85).
With Corollary C.3 in hand, we are now in a position to do some straightforward calculations and bound some quantities which will pop up in the survival and contamination analysis.
Proposition C.4 (Worst-case bound based on Hanson-Wright). Let  $T \subseteq [s]$  be a subset of favored features such that  $\{\alpha, \beta\} \subseteq T$ . Assume that  $|T| = n^{\tau}$  for some  $\tau \leq r$ . Then with probability at least  $1 - O(1 / nk)$ , we have
$$
\left\| \boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {A} _ {- T} ^ {- 1} \boldsymbol {\Delta} y \right\| _ {2} \leq c _ {8} \left(n ^ {1 - t - p} + n ^ {\frac {1 + \tau - t}{2} - p}\right) \sqrt {\log (n k | T |)}. \tag {88}
$$
Proof. WLOG, suppose  $\alpha = 1$  and  $\beta = 2$ . By Corollary C.3 we have with probability at least  $1 - 1/n$  that
$$
\left| \boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {A} _ {- T} ^ {- 1} \boldsymbol {\Delta} y \right| \leq \left[ \begin{array}{c} c _ {7} n ^ {1 - t - p} \sqrt {\log k} \\ c _ {7} n ^ {1 - t - p} \sqrt {\log k} \\ c _ {6} n ^ {\frac {1 - t}{2} - p} \sqrt {\log (n k)} \\ \vdots \\ c _ {6} n ^ {\frac {1 - t}{2} - p} \sqrt {\log (n k)} \end{array} \right]. \tag {89}
$$
Hence, the norm of this vector is at most
$$
\begin{array}{l} \left\| \boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {A} _ {- T} ^ {- 1} \boldsymbol {\Delta} y \right\| _ {2} \leq 2 c _ {7} n ^ {1 - t - p} \sqrt {\log k} + n ^ {\frac {\tau}{2}} c _ {6} n ^ {\frac {1 - t}{2} - p} \sqrt {\log (n k)} (90) \\ \leq c _ {8} \left(n ^ {1 - t - p} + n ^ {\frac {1 + \tau - t}{2} - p}\right) \sqrt {\log (n k)}, (91) \\ \end{array}
$$
where  $c_{8}$  is a positive constant.
D Bounding the survival
Recall that the relative survival was defined to be  $\lambda_F\widehat{\pmb{h}}_{\alpha ,\beta}[\alpha ] = \lambda_F\pmb{z}_\alpha^\top \pmb{A}^{-1}\pmb {\Delta}\pmb {y}$ . The strategy is to apply our variant of Hanson-Wright to  $z_{\alpha}^{\top}\pmb{A}^{-1}\pmb {\Delta}\pmb{y}$ . Unfortunately,  $\pmb{A}^{-1}$  is not independent of  $z_{\alpha}$  or  $\Delta y$ , so we need to use Woodbury to extract out the independent portions and bound away the dependent portion. As we'll see shortly, the error from the dependent portions can also be controlled using Hanson-Wright. Let us now recall Proposition 4.2 for reference.
Proof. Recall that  $\widehat{h}_{\alpha,\beta}[\alpha] = z_{\alpha}^{\top}A^{-1}\Delta y$ . We first observe that for  $i \in [k]$ , the dependence between  $A^{-1}$  and  $z_{i}$  as well as  $A^{-1}$  and  $\Delta y$  only comes through the  $k$  label defining features. Hence, we can use the Woodbury identity to extract out the independent portions of  $A^{-1}$ .