SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\widehat{\pmb{\Sigma}}_{1\cap 2} = \pmb{X}^{\top}\pmb{L}_{I_1\cap I_2}\pmb {X} / |I_1\cap I_2|$  and  $M_{j} = (X^{\top}L_{I_{1}}X / k + \lambda I_{p})^{-1}$  for  $j = 1,2$  . We analyze the deterministic equivalents of the latter for the two cases.
(1)  $\underline{\boldsymbol{A}}\perp \perp (\boldsymbol {X},\boldsymbol {y}).$
From Lemma D.1 (2), we know that when  $\mathbf{A}$  is independent to  $(\mathbf{X},\mathbf{y})$ , it follows that
$$
\begin{array}{l} \lambda^ {2} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} [ \boldsymbol {M} _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} _ {2} ] \\ \simeq \left(v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}\right) ^ {- 1} \left(\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \pmb {A} ^ {\top} \pmb {A}) \pmb {\Sigma} + \pmb {A} ^ {\top} \pmb {A}\right) \left(v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}\right) ^ {- 1}. \\ \end{array}
$$
Then, by the trace property of deterministic equivalents in Lemma E.3 (4), we have
$$
\begin{array}{l} \beta_ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \beta_ {0} \\ \stackrel {\text {a . s .}} {=} \boldsymbol {\beta} _ {0} ^ {\top} (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1} (\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) \boldsymbol {\Sigma} + \boldsymbol {A} ^ {\top} \boldsymbol {A}) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1} \boldsymbol {\beta} _ {0} \\ \stackrel {\text {a . s .}} {=} \widetilde {c} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}). \\ \end{array}
$$
(2)  $\underline{A} = n^{-1 / 2}\underline{X}$
From Lemma D.1 (4), it follows that
$$
\begin{array}{l} \lambda^ {2} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} [ M _ {1} \widehat {\boldsymbol {\Sigma}} M _ {2} ] \simeq \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} d H _ {p} (r)\right) \\ \cdot \left(1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)\right) \left(v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 2} \boldsymbol {\Sigma}. \\ \end{array}
$$
$$
\begin{array}{l} \beta_ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \hat {\boldsymbol {\Sigma}} \boldsymbol {B} _ {k} ^ {\lambda} \beta_ {0} \stackrel {{\text {a . s .}}} {{=}} \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} d H _ {p} (r)\right) (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)) \cdot \\ \boldsymbol {\beta} _ {0} ^ {\top} \left(v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {\Sigma} \left(v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {\beta} _ {0} \\ \stackrel {\mathrm {a . s .}} {=} \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} \widetilde {c} _ {p} (- \lambda ; \phi , \psi , \Sigma). \\ \end{array}
$$
Lemma A.2 (Variance term of generalized risk). Suppose the same assumptions in Theorem 1 hold with  $\lambda > 0$ . Let  $A_k^\lambda = \mathbb{E}_{I \sim \mathcal{I}_k}[(X^\top L_I X / k + \lambda I_p)^{-1} X^\top L_I / k]$  and  $B_k^\lambda = I_p - A_k^\lambda X = \mathbb{E}_{I \sim \mathcal{I}_k}[\lambda (X^\top L_I X / k + \lambda I_p)^{-1}]$ . For any  $A$  with  $\lim \sup \| A \|_{\mathrm{op}}$  bounded almost surely, the following statements hold:
$$
\boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} _ {\mathrm {N L}} - \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \left(1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \Sigma)\right) \xrightarrow {\text {a . s .}} 0.
$$
$$
\begin{array}{l} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} _ {\mathrm {N L}} \\ \xrightarrow {\mathrm {a . s .}} \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} \cdot \left(1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {\Sigma})\right) \\ + \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \left(2 \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r) - 1\right). \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{n} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}\right) ^ {\top} \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}\right) \boldsymbol {f} _ {\mathrm {N L}} \\ \xrightarrow {\mathrm {a . s .}} \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} \cdot \sigma^ {2} (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \Sigma)), \\ \end{array}
$$
Here the nonnegative constants  $v_{p}(-\lambda ;\psi)$  and  $\widetilde{v}_p(-\lambda ;\phi ,\psi ,A^\top A)$  are defined through the following equations:
$$
\begin{array}{l} \frac {1}{v _ {p} (- \lambda ; \psi)} = \lambda + \psi \int \frac {r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r), \\ \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) = \frac {\phi \operatorname {t r} [ \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {\Sigma} (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} ] / p}{v _ {p} (- \lambda ; \psi) ^ {- 2} - \phi \int \frac {r ^ {2}}{(1 + v _ {p} (- \lambda ; \psi) r) ^ {2}} \mathrm {d} H _ {p} (r)}. \\ \end{array}
$$
Proof. The two cases are treated separately below.
(1)  $\underline{A} \perp (\mathbf{X}, \mathbf{y})$ .
We split the proof into three different parts.
Part (1) Intersection concentration. Let  $\pmb{f}_{\mathrm{NL}} = \pmb{L}_{I_1 \cap I_2} \pmb{f}_{\mathrm{NL}} + \pmb{L}_{I_1 \setminus I_2} \pmb{f}_{\mathrm{NL}} + \pmb{L}_{I_2 \setminus I_1} \pmb{f}_{\mathrm{NL}} =: f_0 + f_1 + f_2$ . Note that
$$
\boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \boldsymbol {A} _ {k} ^ {\lambda_ {1} \top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda_ {1}} \boldsymbol {f} _ {\mathrm {N L}} = \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} \left[ (\boldsymbol {f} _ {0} + \boldsymbol {f} _ {1}) ^ {\top} \frac {\boldsymbol {X}}{k} \boldsymbol {M} _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} _ {2} \frac {\boldsymbol {X} ^ {\top}}{k} (\boldsymbol {f} _ {0} + \boldsymbol {f} _ {2}) \right].
$$
where  $M_{j} = (X^{\top}L_{I_{2}}X / k + \lambda I_{p})^{-1}$ . By conditional independence and Lemma S.8.5 of [51], the cross terms vanish
$$
\boldsymbol {f} _ {1} ^ {\top} \frac {\boldsymbol {X}}{k} M _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} M _ {2} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {2}}{k} \boldsymbol {f} _ {\mathrm {N L}} \xrightarrow {\text {a . s .}} 0, \quad \text {a n d} \quad \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \frac {\boldsymbol {L} _ {1} \boldsymbol {X}}{k} M _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} M _ {2} \frac {\boldsymbol {X} ^ {\top}}{k} \boldsymbol {f} _ {2} \xrightarrow {\text {a . s .}} 0. \tag {12}
$$
It remains to analyze the quadratic term of  $\pmb{f}_0$ :
$$
\boldsymbol {f} _ {0} ^ {\top} \frac {\boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} M _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} M _ {2} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1} \cap I _ {2}}}{k} \boldsymbol {f} _ {0}.
$$
By conditioning on  $L_{I_1 \cap I_2} X$ , from Lemma S.7.10 (1) of [12], we have
$$
\boldsymbol {M} _ {j} \simeq \boldsymbol {M} ^ {\det } := \frac {k}{i _ {0}} \left(\widehat {\boldsymbol {\Sigma}} _ {0} + \lambda \boldsymbol {I} _ {p} + \lambda \boldsymbol {C}\right) ^ {- 1},
$$
where  $\widehat{\pmb{\Sigma}}_0 = \pmb{X}_0^\top \pmb{X}_0 / i_0$ ,  $\pmb{X}_0 = \pmb{L}_{I_1 \cap I_2} \pmb{X}$ ,  $\pmb{C} = (k - i_0) / i_0 \cdot (v(-\lambda; \gamma_1, \pmb{\Sigma}_{\pmb{C}_1}) \pmb{\Sigma} + \pmb{I}_p)$ ,  $\pmb{C}_1 = i_0 (\lambda (k - i_0))^{-1} (\widehat{\pmb{\Sigma}}_0 + \lambda \pmb{I}_p)$  and  $i_0 = |I_1 \cap I_2|$ . Then we have
$$
\begin{array}{l} \frac {\boldsymbol {X} _ {0}}{k} \boldsymbol {M} _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} _ {2} \frac {\boldsymbol {X} _ {0} ^ {\top}}{k} \simeq \frac {k ^ {2}}{i _ {0} ^ {2}} \frac {\boldsymbol {X} _ {0}}{k} \boldsymbol {M} ^ {\det } \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} ^ {\det } \frac {\boldsymbol {X} _ {0} ^ {\top}}{k} \\ = \frac {k ^ {2}}{i _ {0} ^ {2}} \frac {\mathbf {X} _ {0} ^ {\prime}}{k} M ^ {\prime} \mathbf {A} ^ {\prime} M ^ {\prime} \frac {\mathbf {X} _ {0} ^ {\prime \top}}{k} \tag {13} \\ \end{array}
$$
where  $\pmb{X}_0' = \pmb{X}_0(\pmb{I}_p + \pmb{C})^{-1/2}$ ,  $\pmb{A}' = (\pmb{I}_p + \pmb{C})^{-1/2}\pmb{A}^\top\pmb{A}(\pmb{I}_p + \pmb{C})^{-1/2}$ , and  $\pmb{M}' = ((\pmb{I}_p + \pmb{C})^{-1/2}\widehat{\pmb{\Sigma}}_0(\pmb{I}_p + \pmb{C})^{-1/2} + \lambda \pmb{I}_p)^{-1}$ .
Part (2) Diagonal concentration. We next use a similar strategy as in the proof of Lemma A.16 in [14] by showing that the off-diagonal summation vanishes. Let  $\pmb{\Sigma}^{\prime} = (\pmb{I}_{p} + \pmb{C})^{-1 / 2}\pmb{\Sigma}(\pmb{I}_{p} + \pmb{C})^{-1 / 2}$ . Since  $\pmb{X}_0 = \pmb{Z}_0\pmb{\Sigma}^{1 / 2}$ , we have  $\pmb{X}_0^{\prime} = \pmb{Z}_0\pmb{\Sigma}^{\prime 1 / 2}$ . Then, the quadratic form becomes:
$$
\begin{array}{l} \frac {1}{k ^ {2}} \boldsymbol {f} _ {0} ^ {\top} \boldsymbol {X} _ {0} ^ {\prime} \boldsymbol {M} ^ {\prime} \boldsymbol {A} ^ {\prime} \boldsymbol {M} ^ {\prime} \boldsymbol {X} _ {0} ^ {\prime \top} \boldsymbol {f} _ {0} \\ = \frac {i _ {0}}{k ^ {2}} \boldsymbol {f} _ {0} ^ {\top} \left(\frac {\boldsymbol {Z} _ {0} \boldsymbol {\Sigma} ^ {\prime} \boldsymbol {Z} _ {0} ^ {\top}}{i _ {0}} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \frac {\boldsymbol {Z} _ {0}}{\sqrt {i _ {0}}} \boldsymbol {\Sigma} ^ {\prime \frac {1}{2}} \boldsymbol {A} ^ {\prime} \boldsymbol {\Sigma} ^ {\prime \frac {1}{2}} \frac {\boldsymbol {Z} _ {0} ^ {\top}}{\sqrt {i _ {0}}} \left(\frac {\boldsymbol {Z} _ {0} \boldsymbol {\Sigma} ^ {\prime} \boldsymbol {Z} _ {0} ^ {\top}}{i _ {0}} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {f} _ {0} \\ =: \frac {i _ {0}}{k ^ {2}} \boldsymbol {f} _ {0} ^ {\top} \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {f} _ {0}. \tag {14} \\ \end{array}
$$
Note that from Lemma D.4, we have for any  $t > 0$
$$
\boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1} = \frac {1}{t} (\boldsymbol {B} _ {1} ^ {- 1} - (\boldsymbol {B} _ {1} + t \boldsymbol {B} _ {2}) ^ {- 1}) + t \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} (\boldsymbol {B} _ {1} + t \boldsymbol {B} _ {2}) ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1}.
$$
Let  $\pmb{U} \in \mathbb{R}^{n \times n}$  with  $U_{ij} = [f_0]_i[f_0]_j \mathbb{1}\{i \neq j\}$ . We then have
$$
\begin{array}{l} \left| \sum_ {1 \leq i \neq j \leq n} \left[ B _ {1} ^ {- 1} B _ {2} B _ {1} ^ {- 1} \right] _ {i j} [ f _ {0} ] _ {i} [ f _ {0} ] _ {j} \right| \\ = | \langle \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1}, \boldsymbol {U} \rangle | \\ \leq \frac {1}{t} | \langle \boldsymbol {B} _ {1} ^ {- 1}, \boldsymbol {U} \rangle | - \frac {1}{t} | \langle (\boldsymbol {B} _ {1} + t \boldsymbol {B} _ {2}) ^ {- 1}, \boldsymbol {U} \rangle | + t \| \boldsymbol {B} _ {1} ^ {- 1} \| _ {\mathrm {o p}} ^ {2} \| \boldsymbol {B} _ {2} \| _ {\mathrm {o p}} ^ {2} \| (\boldsymbol {B} _ {1} + t \boldsymbol {B} _ {2}) ^ {- 1} \| _ {\mathrm {o p}} \| \boldsymbol {U} \| _ {\mathrm {t r}}. \tag {15} \\ \end{array}
$$
For the first two terms, Lemma D.3 implies that
$$
\frac {1}{k} | \langle \boldsymbol {B} _ {1} ^ {- 1}, \boldsymbol {U} \rangle | \xrightarrow {\text {a . s .}} 0, \quad \text {a n d} \quad \frac {1}{k} | \langle (\boldsymbol {B} _ {1} + t \boldsymbol {B} _ {2}) ^ {- 1}, \boldsymbol {U} \rangle | \xrightarrow {\text {a . s .}} 0.
$$
For the last term, note that  $\| B_2\|_{\mathrm{op}}\leq \| A\|_{\mathrm{op}}^2\| \widehat{\Sigma}_0\|_{\mathrm{op}}$  , where  $\| A\|_{\mathrm{op}}$  is almost surely bounded as assumed, and  $\| \widehat{\Sigma}_0\|_{\mathrm{op}}\leq r_{\max}(1 + \sqrt{\psi^2 / \phi})^2$  almost surely as  $k,n,p\to \infty$  and  $p / n\rightarrow \phi \in (0,\infty)$ $p / k\to \psi \in [\phi ,\infty ]$  (see, e.g., [45]). Also,  $\| U\|_{*}\leq 2\| f_{\mathrm{NL}}\|_{2}^{2}\xrightarrow{\mathrm{a.s.}}2\| f_{\mathrm{NL}}\|_{L_2}^2 <  \infty$  from the strong law of large numbers, Lemma D.5, and  $\| B_1\|_{\mathrm{op}}\leq \lambda^{-1}$  . Thus, the last term is almost surely bounded. It then follows that  $k^{-1}|\langle B_1^{-1}B_2B_1^{-1},U\rangle |\xrightarrow{\mathrm{a.s.}}0.$  Therefore,
$$
\left| \frac {1}{k} \boldsymbol {f} _ {0} ^ {\top} \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {f} _ {0} - \frac {1}{k} \sum_ {i = 1} ^ {n} [ \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1} ] _ {i i} [ \boldsymbol {f} _ {0} ] _ {i} ^ {2} \right| \xrightarrow {\text {a . s .}} 0.
$$
Part (3) Trace concentration. From the results in [56], it holds that
$$
\max  _ {1 \leq i \leq n} \left| [ B _ {1} ^ {- 1} B _ {2} B _ {1} ^ {- 1} ] _ {i i} - \frac {1}{n} \operatorname {t r} [ B _ {1} ^ {- 1} B _ {2} B _ {1} ^ {- 1} ] \right| \xrightarrow {\text {a . s .}} 0.
$$
Further,  $n^{-1}\sum_{i = 1}^{n}\pmb{f}_{i}^{2}\xrightarrow{\mathrm{a.s.}}\| f_{\mathrm{NL}}\|_{L^{2}}^{2}$  by strong law of large number and Lemma D.5. Therefore, we have
$$
\frac {1}{k} \left| \boldsymbol {f} ^ {\top} \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {f} - \operatorname {t r} \left[ \boldsymbol {B} _ {1} ^ {- 1} \boldsymbol {B} _ {2} \boldsymbol {B} _ {1} ^ {- 1} \right] \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \right| \xrightarrow {\text {a . s .}} 0. \tag {16}
$$
Combining (12), (13), and (16) yields that
$$
\left| \boldsymbol {f} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} - \operatorname {t r} \left[ \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \right] \cdot \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \right| \xrightarrow {\text {a . s .}} 0.
$$
By the trace property  $\operatorname{tr}[(\pmb{A}_k^\lambda)^\top \pmb{A}^\top \pmb{A}\pmb{A}_k^\lambda] = \operatorname{tr}[\pmb{A}_k^\lambda (\pmb{A}_k^\lambda)^\top \pmb{A}^\top \pmb{A}]$ , it remains to derive the deterministic equivalents of  $\pmb{A}_k^\lambda (\pmb{A}_k^\lambda)^\top \pmb{A}^\top \pmb{A}$ . Note that
$$
\boldsymbol {A} _ {k} ^ {\lambda} (\boldsymbol {A} _ {k} ^ {\lambda}) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} = \frac {| I _ {1} \cap I _ {2} |}{k ^ {2}} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} [ \boldsymbol {M} _ {1} \widehat {\boldsymbol {\Sigma}} _ {1 \cap 2} \boldsymbol {M} _ {2} ] \boldsymbol {A} ^ {\top} \boldsymbol {A}
$$
where  $\widehat{\Sigma}_{1\cap 2} = \pmb{X}^{\top}\pmb{L}_{I_1\cap I_2}\pmb {X} / |I_1\cap I_2|$  and  $M_{j} = (X^{\top}L_{I_{1}}X / k + \lambda I_{p})^{-1}$  for  $j = 1,2$ . The above quantity is well-defined almost surely because  $|I_{1}\cap I_{2}|$  converges to some positive quantity almost surely. Next, we analyze the trace term for the two cases.
From Lemma D.1 (3) we know that when  $\mathbf{A}$  is independent to  $(\mathbf{X},\mathbf{y})$ , it follows that
$$
\mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} \left[ M _ {1} \widehat {\boldsymbol {\Sigma}} _ {1 \cap 2} M _ {2} \right] \boldsymbol {A} ^ {\top} \boldsymbol {A} \simeq \phi^ {- 1} \widetilde {v} _ {v} (- \lambda ; \phi , \psi) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} \boldsymbol {\Sigma} \boldsymbol {A} ^ {\top} \boldsymbol {A}.
$$
We now have
$$
\begin{array}{l} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} _ {\mathrm {N L}} \stackrel {{\text {a . s .}}} {{=}} \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \frac {p}{k} \cdot \frac {| I _ {1} \cap I _ {2} |}{k} \cdot \frac {1}{p} \operatorname {t r} \left[ \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} \left[ M _ {1} \widehat {\Sigma} _ {1 \cap 2} M _ {2} \right] \boldsymbol {A} ^ {\top} \boldsymbol {A} \right] \\ \stackrel {\text {a . s .}} {=} \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \psi \cdot \frac {\phi}{\psi} \cdot \frac {1}{\phi} \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) \\ = \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}), \\ \end{array}
$$
where the second convergence is from Lemma S.8.3 of [12] and the trace property in Lemma E.3 (4).
(2)  $\underline{\boldsymbol{A}} = n^{-1 / 2}\underline{\boldsymbol{X}}$
Instead of working on (13), we use the following decomposition:
$$
\begin{array}{l} M _ {1} \hat {\Sigma} M _ {2} = M _ {1} \hat {\Sigma} _ {1} M _ {2} + \hat {\Sigma} _ {2} M _ {2} - M _ {1} \hat {\Sigma} _ {1 \cap 2} M _ {2} + M _ {1} \hat {\Sigma} _ {(1 \cup 2) ^ {c}} M _ {2} \\ = \sum_ {j = 1} ^ {2} M _ {j} - 2 \lambda M _ {1} M _ {2} - M _ {1} \widehat {\Sigma} _ {1 \cap 2} M _ {2} + M _ {1} \widehat {\Sigma} _ {(1 \cup 2) ^ {c}} M _ {2} \\ \end{array}
$$
Then, repeating Part (2) and (3) as above, it suffices to derive the deterministic equivalents of  $(A_k^\lambda)^\top \widehat{\Sigma} A_k^\lambda$ . We decompose this term into:
$$
\left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} = \frac {1}{n} \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}\right) ^ {\top} \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}\right) + \frac {1}{n} \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} + \boldsymbol {X} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top}\right) - \frac {1}{n} \boldsymbol {I} _ {n}. \tag {17}
$$
For the last two terms of the right hand side, following the similar argument as in Part (1), it holds that
$$
\frac {2}{n} \operatorname {t r} \left[ \boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} \right] - 2 \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r) \xrightarrow {\text {a . s .}} 0, \quad \text {a n d} \quad \frac {1}{n} \operatorname {t r} [ \boldsymbol {I} _ {n} ] = 1. \tag {18}
$$
For the first term of the right hand side, notice that it is the variance term of the mean squared error computed on all samples, which is the variance term of the numerator of the generalized cross-validation (GCV) estimator in [13]. It has been shown in Proposition 3.6 of [13] that in the full ensemble, the GCV estimator is consistent with the prediction risk, which is also true for the variance term:
$$
\frac {1}{n} \operatorname {t r} \left[ \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}\right) ^ {\top} \left(\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}\right) \right] \tag {19}
$$
$$
\xrightarrow {\text {a . s .}} \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} \cdot \sigma^ {2} \left(1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {\Sigma})\right). \tag {20}
$$
Finally, combining (14)-(16) with the above finishes the proof for the first convergence result of  $\mathbf{A} = n^{-1/2}\mathbf{X}$ .
Analogously, from (20), we also have
$$
\begin{array}{l} \frac {1}{n} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} (\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}) ^ {\top} (\boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} - \boldsymbol {I} _ {n}) \boldsymbol {f} _ {\mathrm {N L}} \\ \xrightarrow {\mathrm {a . s .}} \| f _ {\mathrm {N L}} \| _ {L ^ {2}} ^ {2} \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} \cdot \sigma^ {2} (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \Sigma)), \\ \end{array}
$$
which finishes the proof for the second convergence result of  $\mathbf{A} = n^{-1/2}\mathbf{X}$ .
Lemma A.3 (Cross terms of generalized risk). Suppose the same assumptions in Theorem 1 hold with  $\lambda > 0$ . Let  $A_k^\lambda = \mathbb{E}_{I \sim \mathcal{I}_k}[(X^\top L_I X / k + \lambda I_p)^{-1} X^\top L_I / k]$  and  $B_k^\lambda = I_p - A_k^\lambda X = \mathbb{E}_{I \sim \mathcal{I}_k}[\lambda (X^\top L_I X / k + \lambda I_p)^{-1}]$ . For any  $A$  with  $\lim \sup \| A \|_{\mathrm{op}}$  bounded almost surely, it holds that
$$
\boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \beta_ {0} \xrightarrow {\text {a . s .}} 0.
$$
$$
\begin{array}{l} n \| \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \beta_ {0} \| _ {2} ^ {2} \leq \frac {n}{k} \| \boldsymbol {A} _ {k} ^ {\lambda} \| _ {\mathrm {o p}} ^ {2} \| \boldsymbol {A} \| _ {\mathrm {o p}} ^ {4} \| \boldsymbol {B} _ {k} ^ {\lambda} \| _ {\mathrm {o p}} ^ {2} \| \beta_ {0} \| _ {2} ^ {2} \\ \leq \frac {n}{k} \mathbb {E} _ {I \sim \mathcal {I} _ {k}} [ \| \widehat {\boldsymbol {\Sigma}} _ {I} \| _ {\mathrm {o p}} ^ {\frac {1}{2}} \| \lambda (\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {X} / k + \lambda \boldsymbol {I} _ {p}) ^ {- 1} \| _ {\mathrm {o p}} ] ^ {2} \cdot \| \boldsymbol {B} _ {k} ^ {\lambda} \| _ {\mathrm {o p}} ^ {2} \| \boldsymbol {A} \| _ {\mathrm {o p}} ^ {4} \| \beta_ {0} \| _ {2} ^ {2} \\ \leq \frac {n}{k} \mathbb {E} _ {I \sim \mathcal {I} _ {k}} [ \| \widehat {\boldsymbol {\Sigma}} _ {I} \| _ {\mathrm {o p}} ^ {\frac {1}{2}} ] ^ {2} \cdot \| \boldsymbol {A} \| _ {\mathrm {o p}} ^ {4} \| \boldsymbol {\beta} _ {0} \| _ {2} ^ {2}. \\ \end{array}
$$
Let  $s_j^2$  be the singular value of  $\widehat{\Sigma}_j$ . From the results in [45], we have  $\lim \sup \| \widehat{\Sigma}_I \|_{\mathrm{op}} \leq \lim \sup \max_{1 \leq i \leq p} s_i^2 \leq r_{\max}(1 + \sqrt{\psi})^2$  almost surely as  $k, p \to \infty$  and  $p / k \to \psi \in (0, \infty]$ . On the other hand,  $n / k \xrightarrow{\text{a.s.}} \psi / \phi$  and  $\| A \|_2$  is uniformly bounded as assumed. Thus, we have  $\lim \sup n \| (A_k^\lambda)^\top A^\top AB_k^\lambda \beta_0 \|_2^2$  is bounded almost surely.
Similar to the proof of Theorem 1, we decompose  $\pmb{f}_{\mathrm{NL}}$  as  $\pmb{f}_{\mathrm{NL}} = \pmb{L}_{I_1 \cap I_2} \pmb{f}_{\mathrm{NL}} + \pmb{L}_{I_1 \setminus I_2} \pmb{f}_{\mathrm{NL}} + \pmb{L}_{I_2 \setminus I_1} \pmb{f}_{\mathrm{NL}} =: \pmb{f}_0 + \pmb{f}_1 + \pmb{f}_2$ . We then have