SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Indeed, our "push through" lemma for Woodbury (Lemma B.2) and concentration of the hat matrix (Proposition B.6) implies that with extremely high probability
$$
\begin{array}{l} \boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {A} ^ {- 1} \Delta y = \left(\boldsymbol {I} _ {k} + \boldsymbol {H} _ {k}\right) ^ {- 1} \boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \Delta y (92) \\ = \min  \left\{\mu , 1 \right\} \left(\boldsymbol {I} _ {k} + \boldsymbol {E}\right) \boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \Delta y, (93) \\ \end{array}
$$
where  $\| \pmb {E}\| _2 = O(n^{-\kappa_{11}})$
Let  $\pmb{u}_{\alpha} \in \mathbb{R}^{k}$  denote the  $\alpha$ th row vector in  $\pmb{E}$ , and let  $\pmb{u}_{\alpha}^{-} \in \mathbb{R}^{k-1}$  denote the subvector of  $\pmb{u}_{\alpha}$  without index  $\alpha$ . By reading off the  $\alpha$ th row of Eq. (93), we see that
$$
\boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y = \min  \left\{\mu , 1 \right\} \left(\boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y + \left\langle \boldsymbol {u} _ {\alpha}, \boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right\rangle\right) \tag {94}
$$
Since  $\| \pmb{u}_{\alpha} \|_2 \leq \| \pmb{E} \|_2 = O(n^{-\kappa_{11}})$ , it follows from Cauchy-Schwarz that
$$
\left| \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y - \min  \left\{\mu , 1 \right\} \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right| \leq \min  \left\{\mu , 1 \right\} O \left(n ^ {- \kappa_ {1 1}}\right) \| \boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \| _ {2}. \tag {95}
$$
Let us pause for a moment and interpret Eq. (95). The term  $\min \{\mu, 1\}$  is merely capturing the difference in behavior when regression works and fails; if regression works ( $q + r < 1$ ) then it becomes  $\mu$ , and if regression fails ( $q + r > 1$ ), then it becomes 1. This behavior should be expected: in the regression works case, we expect the effect of interpolation to be a regularizing one: the signals are attenuated by a factor of  $\mu$ . The RHS of Eq. (95) is an error term, capturing how differently  $z_{\alpha}^{\top} A^{-1} \Delta y$  behaves from the expected behavior  $\min \{\mu, 1\} z_{\alpha}^{\top} A_{-k}^{-1} \Delta y$ .
Let us now bound the error term. From Proposition C.4 we have with probability at least  $1 - O(1 / nk)$  that
$$
\left\| \boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y \right\| _ {2} \leq c _ {8} \left(n ^ {1 - t - p} + n ^ {\frac {1}{2} - p}\right) \sqrt {\log \left(n k ^ {2}\right)}. \tag {96}
$$
Let us do casework on  $t$ . For  $t < \frac{1}{2}$ , we have  $\frac{1}{2} - p < 1 - t - p$ , so we conclude that the error term is min  $\{\mu, 1\} O(n^{1 - t - p} \cdot n^{-\kappa_4}) \sqrt{\log(nk^2)}$ , where  $\kappa_4$  is a positive constant.
On the other hand, our Hanson-Wright calculations imply (Corollary C.3) that with probability at least  $1 - O(1 / nk)$  that
$$
\left| \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} _ {- k} ^ {- 1} \boldsymbol {\Delta} y - c _ {7} n ^ {1 - t - p} (1 \pm O (n ^ {- \kappa_ {3}})) \sqrt {\log k} \right| \leq c _ {6} n ^ {\frac {1}{2} - p} \sqrt {\log (n k)}.
$$
Again, since  $t < \frac{1}{2}$ , the deviation term is  $o(n^{1 - t - p})\sqrt{\log k}$ .
Hence we conclude that with probability  $1 - O(1 / nk)$  we have
$$
\boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \Delta y = c _ {7} \min  \left\{\mu , 1 \right\} n ^ {1 - t - p} \left(1 \pm O \left(n ^ {- \kappa_ {5}}\right)\right) \sqrt {\log k},
$$
where  $\kappa_{5}$  is a positive constant.
Completely analogous logic handles the bounds for  $\pmb{z}_{\beta}^{\top}\pmb{A}^{-1}\pmb{\Delta}y$ . Let us now return back to the quantity of interest,  $\lambda_F\widehat{\pmb{h}}_{\alpha ,\beta}[\alpha ]$ . We can compute
$$
\begin{array}{l} \lambda_ {F} \pmb {z} _ {\alpha} ^ {\top} \pmb {A} ^ {- 1} \pmb {\Delta} y = n ^ {p - q - r} \cdot c _ {7} \min \{\mu , 1 \} n ^ {1 - t - p} (1 \pm O (n ^ {- \kappa_ {5}})) \sqrt {\log k} \\ = c _ {7} \mu^ {- 1} \min  \left\{\mu , 1 \right\} n ^ {- t} \left(1 \pm O \left(n ^ {- \kappa_ {5}}\right)\right) \sqrt {\log k} \\ = c _ {7} \min \left\{1, \mu^ {- 1} \right\} n ^ {- t} (1 \pm O (n ^ {- \kappa_ {5}})) \sqrt {\log k}. \\ \end{array}
$$
On the other hand, if  $t \geq \frac{1}{2}$  the error terms all dominate, and we replace  $n^{1 - t - p}$  with  $n^{\frac{1}{2} - p}$  everywhere. We conclude that with probability at least  $1 - O(1 / nk)$ ,
$$
\left| \boldsymbol {z} _ {\alpha} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y \right| \leq c _ {9} \min  \left\{\mu , 1 \right\} n ^ {\frac {1}{2} - p} \sqrt {\log (n k)}, \tag {97}
$$
where  $c_{9}$  is a positive constant. Plugging in the scaling for  $\lambda_F$  yields the desired result.
E Bounding the contamination
In this section we give a tight analysis of the contamination term. First, we rewrite the squared contamination term and separate it out into the contamination from the  $k - 2$  label-defining features which are not  $\alpha$  or  $\beta$ , the rest of the  $s - k$  favored features, and the remaining  $d - s$  unfavored features. From Eq. (30), we have
$$
\begin{array}{l} \mathsf {C N} _ {\alpha , \beta} ^ {2} = \sum_ {j \in [ d ] \backslash \{\alpha , \beta \}} \lambda_ {j} ^ {2} \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y\right) ^ {2} (98) \\ = \boldsymbol {\Delta} y ^ {\top} \boldsymbol {A} ^ {- 1} \left(\sum_ {j \in [ d ] \backslash \{\alpha , \beta \}} \lambda_ {j} ^ {2} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}\right) \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y (99) \\ = \underbrace {\Delta y ^ {\top} \boldsymbol {A} ^ {- 1} \left(\sum_ {j \in [ k ] \backslash \{\alpha , \beta \}} \lambda_ {j} ^ {2} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}\right) \boldsymbol {A} ^ {- 1} \Delta y} _ {\triangleq C N _ {\alpha , \beta , L} ^ {2}} + \underbrace {\Delta y ^ {\top} \boldsymbol {A} ^ {- 1} \left(\sum_ {j \in [ s ] \backslash [ k ]} \lambda_ {j} ^ {2} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}\right) \boldsymbol {A} ^ {- 1} \Delta y} _ {\triangleq C N _ {\alpha , \beta , F} ^ {2}} (100) \\ + \underbrace {\boldsymbol {\Delta} y ^ {\top} \boldsymbol {A} ^ {- 1} \left(\sum_ {j > s} \lambda_ {j} ^ {2} \boldsymbol {z} _ {j} \boldsymbol {z} _ {j} ^ {\top}\right) \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y} _ {\triangleq C N _ {\alpha , \beta , U} ^ {2}}. (101) \\ \end{array}
$$
Here,  $\mathsf{CN}_{\alpha,\beta,L}$  corresponds to contamination from label defining features,  $\mathsf{CN}_{\alpha,\beta,F}$  corresponds to contamination from favored features, and  $\mathsf{CN}_{\alpha,\beta,U}$  corresponds to contamination from unfavored features. The reason for separating out the contamination into these three subterms is that we will need slightly different arguments to bound each of them, although Hanson-Wright and Woodbury are central to all of the arguments. In Appendix E.1 we prove the upper bound on  $\mathsf{CN}_{\alpha,\beta,L} + \mathsf{CN}_{\alpha,\beta,F}$ ; in Appendix E.2 we prove the lower bound. Finally, in Appendix E.3 we bound  $\mathsf{CN}_{\alpha,\beta,U}$ . After putting these bounds together, we will obtain the main bounds on the contamination, which we restate here for reference.
E.1 Upper bounding the contamination from label-defining+favored features
In this section, we upper bound the contamination coming from the  $s - 2$  favored features which are not  $\alpha$  or  $\beta$ . This culminates in the following lemma.
Lemma E.1. In the same setting as Proposition 4.3, we have with probability  $1 - O(1 / nk)$  that
$$
\mathsf {C N} _ {\alpha , \beta , L} ^ {2} + \mathsf {C N} _ {\alpha , \beta , F} ^ {2} \leq c _ {1 2} ^ {2} \min \left\{1, \mu^ {- 2} \right\} n ^ {r - t - 1} \log (n s k) ^ {2},
$$
where  $c_{12}$  is a positive constant.
Proof. Let  $\mathbf{W}_R \in \mathbb{R}^{n \times (s - 2)}$  as the weighted feature matrix which includes all of the  $s - 2$  favored features aside from  $\alpha, \beta$ . We can then define  $\mathbf{A}_{-R} \triangleq \mathbf{A} - \mathbf{W}_R \mathbf{W}_R^\top$  and  $\mathbf{H}_R = \mathbf{W}_R^\top \mathbf{A}_{-R}^{-1} \mathbf{W}_R$ . Using Woodbury, an analogous computation to Lemma B.2 implies that
$$
\boldsymbol {W} _ {R} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y = \left(\boldsymbol {I} _ {s - 2} + \boldsymbol {H} _ {R}\right) ^ {- 1} \boldsymbol {W} _ {R} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y. \tag {102}
$$
The contamination from all of the  $s - 2$  favored features that are not  $\alpha$  or  $\beta$  satisfies
$$
\begin{array}{l} \mathsf {C N} _ {\alpha , \beta , L} ^ {2} + \mathsf {C N} _ {\alpha , \beta , F} ^ {2} = \lambda_ {F} \boldsymbol {\Delta y} ^ {\top} \boldsymbol {A} ^ {- 1} \sum_ {j \in [ s ] \backslash \{\alpha , \beta \}} \boldsymbol {w} _ {j} \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta y} \\ = \lambda_ {F} \boldsymbol {\Delta y} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {W} _ {R} \boldsymbol {W} _ {R} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta y} \\ = \lambda_ {F} \pmb {\Delta} y ^ {\top} \pmb {A} _ {- R} ^ {- 1} \pmb {W} _ {R} (\pmb {I} _ {s - 2} + \pmb {H} _ {R}) ^ {- 2} \pmb {W} _ {R} ^ {\top} \pmb {A} _ {- R} ^ {- 1} \pmb {\Delta} y. \\ \end{array}
$$
Since Proposition B.6 implies that  $\mu_1((\boldsymbol{I}_{s-2} + \boldsymbol{H}_R)^{-2}) \leq c_{10} \min \{\mu^2, 1\}$  with extremely high probability where  $c_{10}$  is a positive constant, we know the contamination is with extremely high probability upper bounded by the following quadratic form:
$$
\begin{array}{l} c _ {1 0} \min  \left\{\mu^ {2}, 1 \right\} \lambda_ {F} \boldsymbol {\Delta} y ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {W} _ {R} \boldsymbol {W} _ {R} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y (103) \\ = c _ {1 0} \min  \left\{\mu^ {2}, 1 \right\} \lambda_ {F} ^ {2} \sum_ {j \in [ s ] \backslash \{\alpha , \beta \}} \left\langle z _ {j}, A _ {- R} ^ {- 1} \Delta y \right\rangle^ {2}. (104) \\ \end{array}
$$
We still cannot apply Hanson-Wright, because  $A_{-R}^{-1}$  is not independent of  $\Delta y$ . However, we can use Woodbury again to take out  $z_{\alpha}, z_{\beta}$  from  $A_{-R}^{-1}$ .
Define  $\mathbf{W}_{\alpha, \beta} = [\mathbf{w}_{\alpha} \quad \mathbf{w}_{\beta}]$  and  $\mathbf{H}_{\alpha, \beta}^{(s)} = \mathbf{W}_{\alpha, \beta}^{\top} \mathbf{A}_{-s}^{-1} \mathbf{W}_{\alpha, \beta}$ . Then Woodbury implies that
$$
\boldsymbol {A} _ {- R} ^ {- 1} = \boldsymbol {A} _ {- s} ^ {- 1} - \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} \left(\boldsymbol {I} _ {2} + \boldsymbol {H} _ {\alpha , \beta} ^ {(s)}\right) ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1}. \tag {105}
$$
$$
\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \Delta y = \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \Delta y - \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} \left(\boldsymbol {I} _ {2} + \boldsymbol {H} _ {\alpha , \beta} ^ {(s)}\right) ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \Delta y. \tag {106}
$$
We will use Hanson-Wright and Cauchy-Schwarz to argue that the second term in Eq. (106) above will be dominated by the first term. Indeed, Corollary C.3 implies that  $z_{j}^{\top}A_{-s}^{-1}\Delta y \leq c_{6}n^{\frac{1 - t}{2} -p}\sqrt{\log(nsk)}$  with probability at least  $1 - O(1 / nsk)$ , so it suffices to show that the other term is dominated by  $n^{\frac{1 - t}{2} -p}$ . We will show that its contribution for each  $j$  is min  $\{1,\mu^{-1}\} \tilde{O} (n^{\frac{1}{2} -t - p})$ . By Cauchy-Schwarz, the magnitude of the second term of Eq. (106) is at most
$$
\begin{array}{l} \left\| \left(\boldsymbol {I} _ {2} + \boldsymbol {H} _ {\alpha , \beta}\right) ^ {- 1} \right\| _ {2} \left\| \boldsymbol {W} _ {\alpha , \beta} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {z} _ {j} \right\| _ {2} \left\| \boldsymbol {W} _ {\alpha , \beta} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \Delta y \right\| _ {2} (107) \\ \leq c _ {1 1} \lambda_ {F} \min  \left\{\mu , 1 \right\} n ^ {\frac {1}{2} - p} \sqrt {\log (n s k)} \cdot n ^ {1 - t - p} \sqrt {\log k} (108) \\ \leq c _ {1 1} \min  \left\{1, \mu^ {- 1} \right\} n ^ {\frac {1}{2} - t - p} \log (n s k), (109) \\ \end{array}
$$
where  $c_{11}$  is a positive constant. In the second line, we have used Proposition B.6 to upper bound  $\left\| (I_2 + H_{\alpha ,\beta}^{(s)})^{-1}\right\| _2\leq O(\min \{\mu ,1\})$  and we have used Theorem B.1 and Proposition B.4 to deduce that that  $\left|\pmb {z}_j^\top \pmb{A}_{-s}^{-1}\pmb {z}_a\right|\leq O(n^{\frac{1}{2} -p})\sqrt{\log(nsk)}$  with probability at least  $1 - O(1 / nsk)$ . Similarly, we used the scaling from Corollary C.3 to deduce that  $\left|\pmb {z}_\alpha^\top \pmb {A}_{-s}^{-1}\pmb {\Delta}\pmb {y}\right|\leq O(n^{1 - t - p})\sqrt{\log k}$ , and similarly for  $\beta$ .
Hence  $\pmb{z}_j^\top \pmb{A}_{-R}^{-1}\pmb{\Delta}y$  is  $O(n^{\frac{1 - t}{2} - p})\log (nsk)$  with probability  $1 - O(1 / nsk)$ . By union bounding over  $j$  and plugging our upper bound back into Eq. (104), we conclude that with probability at least  $1 - O(1 / nk)$
$$
\begin{array}{l} \mathsf {C N} _ {\alpha , \beta , L} ^ {2} + \mathsf {C N} _ {\alpha , \beta , F} ^ {2} \leq c _ {1 0} \min  \left\{\mu^ {2}, 1 \right\} \lambda_ {F} ^ {2} n ^ {r} \cdot O \left(n ^ {1 - t - 2 p}\right) \log (n s k) ^ {2} (110) \\ = \mu^ {- 2} \min  \left\{\mu^ {2}, 1 \right\} O \left(n ^ {r - t - 1}\right) \log (n s k) ^ {2} (111) \\ \leq c _ {1 2} ^ {2} \min  \left\{1, \mu^ {- 2} \right\} n ^ {r - t - 1} \log (n s k) ^ {2}, (112) \\ \end{array}
$$
where  $c_{12}$  is a positive constant, concluding the proof.
E.2 Lower bounding the contamination from label-defining+favored features
Lemma E.2. In the same setting as Proposition 4.3, if  $t > 0$ , with probability at least  $1 - O(1 / nk)$ , we have
$$
\mathsf {C N} _ {\alpha , \beta , L} ^ {2} + \mathsf {C N} _ {\alpha , \beta , F} ^ {2} \geq c _ {1 4} ^ {2} \min  \left\{1, \mu^ {- 2} \right\} n ^ {r - t - 1},
$$
where  $c_{14}$  is a positive constant.
Proof. Following the beginning of the proof of Lemma E.1 and what we know about the flatness of the spectra of hat matrices from Proposition B.6, we can deduce that there is some positive constant  $c_{13}$  such that with extremely high probability
$$
\mathsf {C N} _ {\alpha , \beta , L} ^ {2} + \mathsf {C N} _ {\alpha , \beta , F} ^ {2} \geq c _ {1 3} \min  \left\{\mu^ {2}, 1 \right\} \lambda_ {F} ^ {2} \sum_ {j \in [ s ] \backslash \{\alpha , \beta \}} \left\langle z _ {j}, A _ {- R} ^ {- 1} \Delta y \right\rangle^ {2}. \tag {113}
$$
We will further lower bound this by throwing out all label-defining  $j$ . In other words, the goal now is to lower bound
$$
\sum_ {j \in [ s ] \backslash [ k ]} \left\langle \boldsymbol {z} _ {j}, \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y \right\rangle^ {2} = \left\langle \boldsymbol {Z} _ {F} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y, \boldsymbol {Z} _ {F} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y \right\rangle . \tag {114}
$$
The main idea is to use Bernstein's inequality, but unfortunately  $\mathbf{A}_{-R}^{-1}$  is not independent of  $\Delta y$ , so we will again resort to Woodbury to take out  $\mathbf{z}_{\alpha}$  and  $\mathbf{z}_{\beta}$ . As in the proof for upper bounding the favored contamination, we have  $\mathbf{W}_{\alpha,\beta} = [\mathbf{w}_{\alpha} \quad \mathbf{w}_{\beta}]$  and  $\mathbf{H}_{\alpha,\beta}^{(s)} = \mathbf{W}_{\alpha,\beta}^{\top} \mathbf{A}_{-s}^{-1} \mathbf{W}_{\alpha,\beta}$ . Then we can deduce from another application of Woodbury that
$$
\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y = \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y - \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} \left(\boldsymbol {I} _ {2} + \boldsymbol {H} _ {\alpha , \beta}\right) ^ {- 1} \boldsymbol {W} _ {\alpha , \beta} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y. \tag {115}
$$
Again, we can argue that with probability  $1 - O(1 / nsk)$ , the second term is upper bounded in magnitude by
$$
\min  \left\{1, \mu^ {- 1} \right\} n ^ {\frac {1}{2} - t - p} \log (n s k) = \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p} \cdot n ^ {- \kappa_ {6}}\right), \tag {116}
$$
where  $\kappa_{6}$  is a positive constant because  $t > 0$ . Since Hanson-Wright (Corollary C.3) implies that  $\pmb{z}_j^\top \pmb{A}_{-s}^{-1}\pmb{\Delta}y = \tilde{O}(n^{\frac{1 - t}{2} - p})$ , this implies that  $\pmb{z}_j^\top \pmb{A}_{-R}^{-1}\pmb{\Delta}y = \tilde{O}(n^{\frac{1 - t}{2} - p})$ , and similarly for  $\beta$ . Hence we have
$$
\begin{array}{l} \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- R} ^ {- 1} \boldsymbol {\Delta} y\right) ^ {2} = \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y + \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p - \kappa_ {6}}\right)\right) ^ {2} (117) \\ = \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y\right) ^ {2} + \min  \left\{1, \mu^ {- 1} \right\} O \left(n ^ {\frac {1 - t}{2} - p - \kappa_ {6}}\right) \tilde {O} \left(n ^ {\frac {1 - t}{2} - p}\right) (118) \\ = \left(\boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y\right) ^ {2} + \min  \left\{1, \mu^ {- 1} \right\} o \left(n ^ {1 - t - 2 p}\right). (119) \\ \end{array}
$$
We are now in a position to analyze the contribution from the first term of Eq. (119) to Eq. (114): its contribution is  $\langle Z_F^\top A_{-s}^{-1}\Delta y, Z_F^\top A_{-s}^{-1}\Delta y\rangle$ . This does have all the independence required to apply Bernstein, because  $(A_{-s}^{-1}, \Delta y)$  are independent of  $Z_F$ . Hence conditioned on  $A_{-s}^{-1}$  and  $\Delta y$ ,  $\langle Z_F^\top A_{-s}^{-1}\Delta y, Z_F^\top A_{-s}^{-1}\Delta y\rangle$  is a sum of  $s - k$  subexponential variables, and by Lemma 2.7.7 of Vershynin (2018) each of these random variables conditionally has subexponential norm at most  $\left\| A_{-s}^{-1}\Delta y\right\|_2^2$  and conditional mean  $\langle A_{-s}^{-1}\Delta y, A_{-s}^{-1}\Delta y\rangle$ .
We can use Hanson-Wright (Theorem 4.1) to bound both of these quantities. Indeed, it implies that with probability at least  $1 - O(1 / nk)$ ,
$$
\left\| \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y \right\| _ {2} ^ {2} \leq O \left(n ^ {1 - t - 2 p}\right). \tag {120}
$$
Let us now compute the Hanson-Wright bound for  $\langle A_{-s}^{-1}\Delta y, A_{-s}^{-1}\Delta y \rangle$ . Note that  $A_{-s}^{-1}$  is independent of  $\Delta y$ , so we can condition on  $A_{-s}^{-1}$  and conclude that with probability at least  $1 - O(1 / nk)$
$$
\begin{array}{l} \left\langle \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y, \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y \right\rangle \geq \mathbb {E} \left[ \left\langle \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y, \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y \right\rangle \mid \boldsymbol {A} _ {- s} ^ {- 1} \right] - O \left(n ^ {\frac {1 - t}{2}}\right) \left\| \boldsymbol {A} _ {- s} ^ {- 2} \right\| _ {2} \sqrt {\log (n k)} (121) \\ = \operatorname {T r} \left(\boldsymbol {A} _ {- s} ^ {- 2} \mathbb {E} \left[ \boldsymbol {\Delta} y \boldsymbol {\Delta} y ^ {\top} \right]\right) - O \left(n ^ {\frac {1 - t}{2}}\right) \left\| \boldsymbol {A} _ {- s} ^ {- 2} \right\| _ {2} \sqrt {\log (n k)} (122) \\ = \frac {2}{k} \operatorname {T r} \left(\boldsymbol {A} _ {- s} ^ {- 2}\right) - O \left(n ^ {\frac {1 - t}{2}}\right) \| \boldsymbol {A} _ {- s} ^ {- 2} \| _ {2} \sqrt {\log (n k)}, (123) \\ \end{array}
$$
where we have used the fact that  $\Delta y$  is mean zero and  $\Delta y[i]^2\sim \mathrm{Ber}\left(\frac{2}{k}\right)$
From Proposition B.4, we obtain the scaling for  $\mathrm{Tr}\bigl (A_{-s}^{-2}\bigr)$  and  $\left\| A_{-s}^{-2}\right\| _2$  . This implies that with probability at least  $1 - O(1 / nk)$
$$
\begin{array}{l} \left\langle \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y, \boldsymbol {A} _ {- s} ^ {- 1} \boldsymbol {\Delta} y \right\rangle \geq \Omega \left(n ^ {1 - t - 2 p}\right) - O \left(n ^ {\frac {1 - t}{2} - 2 p}\right) \sqrt {\log (n k)} (124) \\ \geq \Omega \left(n ^ {1 - t - 2 p}\right). (125) \\ \end{array}
$$
as  $t < 1$
Bernstein and the union bound implies that with probability at least  $1 - O(1 / nk)$
$$
\begin{array}{l} \left\langle \boldsymbol {Z} _ {F} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \Delta y, \boldsymbol {Z} _ {F} ^ {\top} \boldsymbol {A} _ {- s} ^ {- 1} \Delta y \right\rangle \geq \left(\sum_ {j \in [ s ] \backslash [ k ]} \Omega \left(n ^ {1 - t - 2 p}\right)\right) - O \left(n ^ {\frac {r}{2} + 1 - t - 2 p}\right) (126) \\ \geq \Omega \left(n ^ {r + 1 - t - 2 p}\right), (127) \\ \end{array}
$$
as  $r > 0$
To wrap up, we will need to upper bound the contribution of the error term in Eq. (119). Its contribution from summing over  $j \in [s] \setminus [k]$  is min  $\{1, \mu^{-2}\} o(n^{r+1-t-2p})$ , which is negligible compared to the Bernstein term, which as we just proved is  $\Omega(n^{r+1-t-2p})$ . Hence  $\langle Z_F^\top A_{-R}^{-1}\Delta y, Z_F^\top A_{-R}^{-1}\Delta y \rangle \geq \Omega(n^{r+1-t-2p})$  with high probability, and inserting this back into our lower bound Eq. (113), we see that
$$
\begin{array}{l} \mathbb {C N} _ {\alpha , \beta , L} ^ {2} + \mathbb {C N} _ {\alpha , \beta , F} ^ {2} \geq c _ {1 3} \min  \left\{\mu^ {2}, 1 \right\} \lambda_ {F} ^ {2} \Omega \left(n ^ {r + 1 - t - 2 p}\right) (128) \\ = c _ {1 3} \mu^ {- 2} \min  \left\{\mu^ {2}, 1 \right\} \Omega \left(n ^ {r - t - 1}\right) (129) \\ \geq c _ {1 4} ^ {2} \min  \left\{1, \mu^ {- 2} \right\} n ^ {r - t - 1}, (130) \\ \end{array}
$$
E.3 Bounding the unfavored contamination
Finally, we wrap up the section by proving matching upper and lower bounds for the unfavored contamination  $\mathsf{CN}_{\alpha,\beta,U}$ .
Lemma E.3 (Bounding unfavored contamination). In the same setting as Proposition 4.3, if  $t > 0$  with probability  $1 - O(1 / nk)$ , the contamination from the unfavored features satisfies
$$
\mathsf {C N} _ {\alpha , \beta , U} ^ {2} = c _ {1 5} ^ {2} (1 \pm o (1)) n ^ {1 - t - p},
$$
where  $c_{15}$  is a positive constant.
On the other hand, if  $t = 0$ , then with probability  $1 - O(1 / nk)$ , the unfavored contamination satisfies
$$
\mathsf {C N} _ {\alpha , \beta , U} ^ {2} \leq c _ {1 6} ^ {2} \min  \left\{1, \mu^ {- 1} \right\} n ^ {1 - t - p} \log (n s k).
$$
Proof. By Woodbury, we have
$$
\boldsymbol {A} ^ {- 1} = \boldsymbol {A} _ {U} ^ {- 1} - \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {M} _ {s} \boldsymbol {A} _ {U} ^ {- 1}, \tag {131}
$$
$$
\boldsymbol {M} _ {s} \triangleq \boldsymbol {W} _ {s} \left(\boldsymbol {I} _ {s} + \boldsymbol {H} _ {s}\right) ^ {- 1} \boldsymbol {W} _ {s} ^ {\top}, \tag {132}
$$
and  $H_{s}\triangleq W_{s}^{\top}A_{-s}^{-1}W_{s}$
$$
\begin{array}{l} \mathrm {C N} _ {\alpha , \beta , U} ^ {2} = \boldsymbol {\Delta} y ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {A} _ {U} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y (133) \\ = \boldsymbol {\Delta} y ^ {\top} \left(\boldsymbol {A} _ {U} ^ {- 1} - \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {M} _ {s} \boldsymbol {A} _ {U} ^ {- 1}\right) \boldsymbol {A} _ {U} \left(\boldsymbol {A} _ {U} ^ {- 1} - \boldsymbol {A} _ {U} ^ {- 1} \boldsymbol {M} _ {s} \boldsymbol {A} _ {U} ^ {- 1}\right) \boldsymbol {\Delta} y (134) \\ = \Delta y ^ {\top} \left(A _ {U} ^ {- 1} - 2 A _ {U} ^ {- 1} M _ {s} A _ {U} ^ {- 1} + A _ {U} ^ {- 1} M _ {s} A _ {U} ^ {- 1} M _ {s} A _ {U} ^ {- 1}\right) \Delta y. (135) \\ \end{array}
$$
By Theorem 4.1, we have with probability at least  $1 - O(1 / nk)$