SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \frac {\boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} \boldsymbol {M} _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} _ {2} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} \beta_ {0} \\ \xrightarrow {\text {a . s .}} f _ {0} ^ {\top} \frac {\boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} M _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} M _ {2} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} \beta_ {0} \\ \xrightarrow {\text {a . s .}} \frac {i _ {0}}{k} \boldsymbol {f} _ {0} ^ {\top} L _ {I _ {1} \cap I _ {2}} \frac {k ^ {2}}{i _ {0} ^ {2}} \frac {\boldsymbol {X} _ {0} ^ {\prime}}{k} M ^ {\prime} A ^ {\prime} M ^ {\prime} \hat {\Sigma} _ {0} ^ {\prime} \beta_ {0} \\ = \frac {i _ {0}}{k} \boldsymbol {f} _ {0} ^ {\top} \boldsymbol {L} _ {I _ {1} \cap I _ {2}} \frac {k ^ {2}}{i _ {0} ^ {2}} \frac {\boldsymbol {X} _ {0} ^ {\prime}}{k} \boldsymbol {M} ^ {\prime} \boldsymbol {A} ^ {\prime} \boldsymbol {\beta} _ {0} - \lambda \frac {i _ {0}}{k} \boldsymbol {f} _ {0} ^ {\top} \boldsymbol {L} _ {I _ {1} \cap I _ {2}} \frac {k ^ {2}}{i _ {0} ^ {2}} \frac {\boldsymbol {X} _ {0} ^ {\prime}}{k} \boldsymbol {M} ^ {\prime} \boldsymbol {A} ^ {^ {\prime}} \boldsymbol {M} ^ {\prime} \boldsymbol {\beta} _ {0}, \\ \end{array}
$$
where the first convergence is from Lemma S.8.5 of [51] and the second convergence is from (13), with  $X_0' = X_0(I_p + C)^{-1/2}$ ,  $A' = (I_p + C)^{-1/2}A^\top A(I_p + C)^{-1/2}$ , and  $M' = (\widehat{\Sigma}_0' + \lambda I_p)^{-1}$  and  $\widehat{\Sigma}_0' = (I_p + C)^{-1/2}\widehat{\Sigma}_0(I_p + C)^{-1/2}$ . From Lemma D.2, the first term in the above display vanishes. Analogous to (15), the second term also vanishes by splitting and applying Lemma D.2. Thus, we have for  $I_1, I_2 \sim \mathcal{I}_k$ ,
$$
\boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \frac {\boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} \boldsymbol {M} _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} _ {2} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1} \cap I _ {2}} \boldsymbol {X}}{k} \beta_ {0} \xrightarrow {\text {a . s .}} 0.
$$
Finally, by Lemma G.5 (2) of [13], the conclusion follows.
B Proofs of results in Section 4
B.1 Proof of Theorem 3
We consider two cases.
(1)  $\lambda_1, \lambda_2 > 0$ .
We begin to prove for the full ensemble when  $M = \infty$ . For  $j = 1,2$  and  $I_{j} \sim \mathcal{I}_{k_{j}}$ , define  $\widehat{\pmb{\Sigma}}_j = \pmb{X}^\top \pmb{L}_{I_j}\pmb{X} / n$  and  $M_{j} = (\pmb{X}^{\top}\pmb{L}_{I_{j}}\pmb{X} / k + \lambda_{j}\pmb{I}_{p})^{+}$ . Recall that  $\pmb{y} = \pmb{X}\beta_{0} + \pmb{f}_{\mathrm{NL}}$  and
$$
\begin{array}{l} \widehat {\boldsymbol {\beta}} _ {k _ {j}, \infty} ^ {\lambda_ {j}} = \mathbb {E} _ {I _ {j} \sim \mathcal {I} _ {k _ {j}}} \left[ \left(\frac {1}{k _ {j}} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {j}} \boldsymbol {X} + \lambda_ {j} \boldsymbol {I} _ {p}\right) ^ {- 1} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {j}} \boldsymbol {y}}{k _ {j}} \right] \\ = \mathbb {E} _ {I _ {j} \sim \mathcal {I} _ {k _ {j}}} [ M _ {j} \widehat {\boldsymbol {\Sigma}} _ {j} ] \beta_ {0} + \mathbb {E} _ {I _ {j} \sim \mathcal {I} _ {k _ {j}}} [ M _ {j} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {j}} / k _ {j} ] \boldsymbol {f} _ {\mathrm {N L}}. \\ \end{array}
$$
Then, for  $\pmb{a} \in \mathbb{R}^p$  with bounded  $l_2$  norm, we have
$$
\begin{array}{l} \boldsymbol {a} ^ {\top} (\widehat {\boldsymbol {\beta}} _ {k _ {1}, \infty} ^ {\lambda_ {1}} - \widehat {\boldsymbol {\beta}} _ {k _ {2}, \infty} ^ {\lambda_ {2}}) = \underbrace {\boldsymbol {a} ^ {\top} (\mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} [ \boldsymbol {M} _ {1} \widehat {\boldsymbol {\Sigma}} _ {1} ] - \mathbb {E} _ {I _ {2} \sim \mathcal {I} _ {k _ {2}}} [ \boldsymbol {M} _ {2} \widehat {\boldsymbol {\Sigma}} _ {2} ]) \boldsymbol {\beta} _ {0}} _ {T _ {1}} \\ + \underbrace {\boldsymbol {a} ^ {\top} \left(\mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} \left[ M _ {1} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1}} \right] / k _ {1} - \mathbb {E} _ {I _ {2} \sim \mathcal {I} _ {k _ {2}}} \left[ M _ {2} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \right] / k _ {2}\right) \boldsymbol {f} _ {\mathrm {N L}}\left. \right)} _ {T _ {2}}. \tag {21} \\ \end{array}
$$
Next, we analyze the three terms separately.
For the first term, from Lemma D.1 (1) we have for  $\lambda_j > 0$
$$
\mathbb {E} _ {I _ {j} \sim \mathcal {I} _ {k _ {j}}} [ \boldsymbol {M} _ {j} \widehat {\boldsymbol {\Sigma}} _ {j} ] = \boldsymbol {I} _ {p} - \mathbb {E} _ {I _ {j} \sim \mathcal {I} _ {k _ {j}}} [ \lambda_ {j} \boldsymbol {M} _ {j} ] \simeq \boldsymbol {I} _ {p} - (v (- \lambda_ {j}; \psi_ {j}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1},
$$
where  $v(-\lambda_j; \psi_j)$  is as defined in (40). By Lemma C.1 we have that for  $\overline{\psi} \in [\phi, \infty]$  there exists a segment  $\mathcal{P}$  such that for all  $(\lambda_1, \psi_1), (\lambda_2, \psi_2) \in \mathcal{P}$ , it holds that  $v(-\lambda_1; \psi_1) = v(-\lambda_1; \psi_2)$  as  $p / k_1 \to \psi_1$  and  $p / k_2 \to \psi_2$ . By the definition of deterministic equivalents (Definition E.1), it follows that
$$
\begin{array}{l} T _ {1} = \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} \left[ M _ {1} \widehat {\boldsymbol {\Sigma}} _ {1} \right] \right] - \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \mathbb {E} _ {I _ {2} \sim \mathcal {I} _ {k _ {2}}} \left[ M _ {2} \widehat {\boldsymbol {\Sigma}} _ {2} \right] \right] \\ \xrightarrow {\mathrm {a . s .}} \lim  _ {p \rightarrow \infty} \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \left(\boldsymbol {I} _ {p} - (v (- \lambda_ {1}; \psi_ {1}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1}\right)\right] - \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \left(\boldsymbol {I} _ {p} - (v (- \lambda_ {2}; \psi_ {2}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1}\right)\right] \\ = 0. \\ \end{array}
$$
For the second term, notice that by Lemma D.2,
$$
\frac {1}{k _ {j}} \boldsymbol {a} ^ {\top} \boldsymbol {M} _ {j} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {k _ {j}}} \boldsymbol {f} _ {\mathrm {N L}} = \boldsymbol {a} ^ {\top} \frac {\boldsymbol {Z} \boldsymbol {\Sigma} ^ {\frac {1}{2}}}{k _ {j}} \left(\frac {\boldsymbol {Z} \boldsymbol {\Sigma} \boldsymbol {Z} ^ {\top}}{k _ {j}} + \lambda_ {j} \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {f} _ {\mathrm {N L}} \xrightarrow {\text {a . s .}} 0.
$$
From Lemma G.5 (2) of [13], it follows that
$$
\boldsymbol {a} ^ {\top} \mathbb {E} _ {I _ {j} \sim \mathcal {I} _ {k _ {j}}} [ \boldsymbol {M} _ {j} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {j}} ] / k _ {j} \boldsymbol {f} _ {\mathrm {N L}} \xrightarrow {\text {a . s .}} 0, \quad j = 1, 2,
$$
and  $T_{2}\xrightarrow{\mathrm{a.s.}}0$
Combining the above results and applying triangle inequality on (21) yields that
$$
\left| \boldsymbol {a} ^ {\top} \left(\widehat {\boldsymbol {\beta}} _ {k _ {1}, \infty} ^ {\lambda_ {1}} - \widehat {\boldsymbol {\beta}} _ {k _ {2}, \infty} ^ {\lambda_ {2}}\right) \right| \leq \left| T _ {1} \right| + \left| T _ {2} \right| \xrightarrow {\text {a . s .}} 0,
$$
which completes the proof when  $M = \infty$ . When  $M \in \mathbb{N}$ , replacing the above expectation by the average over  $M$  simple random samples  $I_1, I_2 \sim \mathcal{I}_k$  completes the proof.
(2)  $\lambda_1\lambda_2 = 0$
When  $\lambda_1 = \lambda_2 = 0$ , it is trivially true as  $k_{1} = k_{2}$ . Otherwise, without loss of generality, we assume  $\lambda_{1} = 0$  and  $\lambda_{2} > 0$ . We use the same decomposition (21) and first analyze  $T_{1}$ . From part one we have that for  $\lambda > 0$ ,  $P_{n,\lambda} - Q_{n,\lambda} \stackrel{\text{a.s.}}{\longrightarrow} 0$  where
$$
\begin{array}{l} P _ {n, \lambda}: = \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} \left[ \left(\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1}} \boldsymbol {X} / k _ {1} + \lambda \boldsymbol {I} _ {p}\right) ^ {+} \widehat {\boldsymbol {\Sigma}} _ {1} \right] \right], \\ Q _ {n, \lambda}: = \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \left(\boldsymbol {I} _ {p} - (v (- \lambda ; \psi_ {1}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1}\right) \right]. \\ \end{array}
$$
We next show that
$$
\lim  _ {n \rightarrow \infty} \lim  _ {\lambda \rightarrow 0 +} P _ {n, \lambda} = \lim  _ {\lambda \rightarrow 0 +} \lim  _ {n \rightarrow \infty} P _ {n, \lambda} = 0, \tag {22}
$$
by proving that the function  $P_{n,\lambda}$  is equicontinuous family in  $\lambda$  over  $\Lambda = [0, \lambda_{\max}]$  for any  $\lambda_{\max} \in (0, \infty)$  fixed. Note that for all  $\lambda \in \Lambda$ ,
$$
| P _ {n, \lambda} | \leq \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} [ \| (\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1}} \boldsymbol {X} / k _ {1} + \lambda \boldsymbol {I} _ {p}) ^ {+} \widehat {\boldsymbol {\Sigma}} _ {1} \| _ {\mathrm {o p}} ] \| \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \| _ {\mathrm {t r}} \leq \| \boldsymbol {\beta} _ {0} \| _ {2} ^ {2} \| \boldsymbol {a} \| _ {2} ^ {2}
$$
and its derivative
$$
\begin{array}{l} \left| \frac {\partial}{\partial \lambda} P _ {n, \lambda} \right| = \left| \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} [ \operatorname {t r} [ (\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1}} \boldsymbol {X} / k _ {1} + \lambda \boldsymbol {I} _ {p}) ^ {- 2} \widehat {\boldsymbol {\Sigma}} _ {1} \beta_ {0} \boldsymbol {a} ^ {\top} ] ] \right| \\ \leq \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k _ {1}}} \left[ \left\| \left(\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {1}} \boldsymbol {X} / k _ {1} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 2} \widehat {\boldsymbol {\Sigma}} _ {1} \right\| _ {\mathrm {o p}} \right] \| \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \| _ {\mathrm {t r}} \\ \leq \left\| \boldsymbol {\beta} _ {0} \right\| _ {2} ^ {2} \left\| \boldsymbol {a} \right\| _ {2} ^ {2} \\ \end{array}
$$
are uniformly bounded in  $\lambda$  almost surely. In the above inequalities, the operator norm is bounded because  $\| (\pmb{X}^{\top}\pmb{L}_{I_1}\pmb {X} / k_1 + \lambda \pmb {I}_p)^{-1}\widehat{\pmb{\Sigma}}_1\|_{\mathrm{op}}\leq s_i / (s_i + \lambda)\leq 1$  and  $\| (\pmb{X}^{\top}\pmb{L}_{I_1}\pmb {X} / k_1+$ $\lambda I_p)^{-2}\widehat{\pmb{\Sigma}}_1\|_{\mathrm{op}}\leq s_i / (s_i + \lambda)^2\leq 1 / (s_i + \lambda)\leq 1$ , by noting that  $\lim \sup \| \widehat{\pmb{\Sigma}}_j\|_{\mathrm{op}}\leq$ $\lim \sup \max_{1\leq i\leq p}s_i^2\leq r_{\max}(1 + \sqrt{\psi_1})^2$  and  $\lim \inf \| \widehat{\pmb{\Sigma}}_j\|_{\mathrm{op}}\geq \lim \inf \min_{1\leq i\leq p}s_i^2\geq$ $r_{\min}(1 - \sqrt{\psi_1})^2$  almost surely as  $k_{1},p\to \infty$  and  $p / k_{1}\rightarrow \psi_{1}\in (0,\infty)\setminus \{1\}$  [45]. On the other hand, since  $Q_{n,\lambda}$  is a continuous function of  $v(-\lambda ;\psi_1)$ , we have
$$
\begin{array}{l} \left| Q _ {n, \lambda} \right| \leq \left\| v (- \lambda_ {1}; \psi_ {1}) \boldsymbol {\Sigma} \left(v (- \lambda_ {1}; \psi_ {1}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 1} \right\| _ {\mathrm {o p}} \| \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \| _ {\mathrm {t r}} \leq \left\| \boldsymbol {\beta} _ {0} \right\| _ {2} ^ {2} \| \boldsymbol {a} \| _ {2} ^ {2} \\ \left| \frac {\partial}{\partial \lambda} Q _ {n, \lambda} \right| = \left| \operatorname {t r} \left[ \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} (v (- \lambda ; \psi_ {1}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} \boldsymbol {\Sigma} \right] \frac {\partial v (- \lambda ; \psi_ {1})}{\partial \lambda} \right| \\ \leq \| \boldsymbol {\beta} _ {0} \boldsymbol {a} ^ {\top} \| _ {\mathrm {o p}} \left| \frac {\partial v (- \lambda ; \psi_ {1})}{\partial \lambda} \right| \int \frac {r}{(1 + v (- \lambda ; \psi_ {1}) r) ^ {2}} \mathrm {d} H _ {p} (r). \\ \end{array}
$$
When  $\psi_{1} > 1$ ,  $\partial v(-\lambda ;\psi_1) / \partial \lambda$  is bounded over  $\Lambda$  [13, Lemma E.10 and Lemma E.11] and thus  $|\partial Q_{n,\lambda} / \partial \lambda |$  is also bounded almost surely. When  $\psi_{1} < 1$ , from Lemma E.12 of [13] we have
$$
\begin{array}{l} \left| \frac {\partial v (- \lambda ; \psi_ {1})}{\partial \lambda} \right| \int \frac {r}{(1 + v (- \lambda ; \psi_ {1}) r) ^ {2}} d H _ {p} (r) = - \frac {\partial v (- \lambda ; \psi_ {1})}{\partial \lambda} \int \frac {r}{(1 + v (- \lambda ; \psi_ {1}) r) ^ {2}} d H _ {p} (r) \\ = \frac {\int \frac {r}{(1 + v (- \lambda ; \psi_ {1}) r) ^ {2}} \mathrm {d} H _ {p} (r)}{\frac {1}{v (- \lambda ; \psi_ {1}) ^ {2}} - \psi_ {1} \int \frac {r ^ {2}}{(1 + v (- \lambda ; \psi_ {1}) r) ^ {2}} \mathrm {d} H _ {p} (r)} \\ \leq \frac {1}{1 - \psi_ {1}}, \\ \end{array}
$$
since  $v(-\lambda ;\psi_1)\leq v(0;\psi_1) = +\infty$ . Therefore,  $|\partial Q_{n,\lambda} / \partial \lambda |$  is uniformly bounded over  $\Lambda$  for  $\psi_{1}\in (0,\infty)\setminus \{1\}$ . Thus, by the Moore-Osgood theorem, the convergence is uniform in  $\lambda$  and (22) follows. Finally, since  $T_{2}$  is bounded analogously as in Part (1), the equivalence holds when  $\lambda_{1} = 0$ .
B.2 Proof of Proposition 4
From Lemma B.2, it follows that
$$
v (- \lambda ; \phi_ {n}) \simeq \frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} + \lambda I _ {p}\right) ^ {- 1} \right]. \tag {23}
$$
By the continuity of function  $\phi \mapsto v(-\lambda ;\phi)$  from Lemma F.10 and F.11 of [13], we have  $v(-\lambda ;\phi_n)\simeq v(-\lambda ;\phi)$  as  $\phi_{n} = p / n\to \phi$  . From Lemma C.1, there exists  $\bar{\lambda}_n$  such that
$$
v (0; \psi_ {n}) = v (- \bar {\lambda} _ {n}; \phi_ {n}),
$$
as  $\psi_n \to \bar{\psi}$  and  $\phi_n \to \phi$ . Involving (23) on the both sides yields that
$$
\frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} + \bar {\lambda} _ {n} I _ {p}\right) ^ {- 1} \right] \simeq \frac {1}{k} \operatorname {t r} \left[ \left(\frac {1}{k} \boldsymbol {X} \boldsymbol {L} _ {I _ {1}} \boldsymbol {X} ^ {\top}\right) ^ {+} \right] \simeq \frac {1}{M k} \sum_ {\ell = 1} ^ {M} \operatorname {t r} \left[ \left(\frac {1}{k} \boldsymbol {X} \boldsymbol {L} _ {I _ {\ell}} \boldsymbol {X} ^ {\top}\right) ^ {+} \right]
$$
where  $\{I_1,\dots ,I_M\}$  is a simple random sample from  $\mathcal{I}_k$
B.3 Proof of Corollary 5
We will use a structural equivalence (much more direct than the first-order equivalence considered in the paper) between generalized ridge predictor and the isotropic ridge predictor to prove the result.
The generalized ridge predictor (9), trained on the subsampled dataset  $\mathcal{D}_I$ , can be expressed as:
$$
\widehat {\beta} _ {k} ^ {\lambda , \boldsymbol {G}} (\mathcal {D} _ {I}) = \left(\frac {1}{k} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {X} + \lambda \boldsymbol {G}\right) ^ {- 1} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {y}}{k}.
$$
Observe that we can equivalently manipulate the generalized ridge estimator into:
$$
\widehat {\beta} _ {k} ^ {\lambda , \boldsymbol {G}} (\mathcal {D} _ {I}) = \boldsymbol {G} ^ {- 1 / 2} \left(\frac {1}{k} \boldsymbol {G} ^ {- 1 / 2} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {X} \boldsymbol {G} ^ {- 1 / 2} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {G} ^ {- 1 / 2} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {y}}{k}.
$$
Recalling from Assumption 2 that  $\mathbf{X} = \mathbf{Z}\mathbf{\Sigma}^{1/2}$ , where  $\mathbf{Z} \in \mathbb{R}^{n \times p}$  contains  $\mathbf{z}_i^\top$  in the  $i$ -th row for  $i \in [n]$ , we obtain
$$
\widehat {\beta} _ {k} ^ {\lambda , \boldsymbol {G}} (\mathcal {D} _ {I}) = \boldsymbol {G} ^ {- 1 / 2} \left(\frac {1}{k} \boldsymbol {G} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {Z} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {Z} \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {G} ^ {- 1 / 2} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \frac {\boldsymbol {G} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {Z} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {y}}{k}. \tag {24}
$$
We now define  $\Sigma_G = G^{-1/2}\Sigma G^{-1/2}$  and consider a transformed feature matrix  $X_G = Z\Sigma_G^{1/2}$ . Denote the transformed dataset corresponding to the new feature matrix by  $\mathcal{D}^G$  (where we keep the response vector as is). Using (24), we can write relate the generalized ridge estimator fitted on the dataset  $\mathcal{D}$  to the standard ridge estimator fitted on the data  $\mathcal{D}^G$  (both at the same scalar regularization level  $\lambda$ ) by:
$$
\widehat {\boldsymbol {\beta}} _ {k} ^ {\lambda , \boldsymbol {G}} (\mathcal {D} _ {I}) = \boldsymbol {G} ^ {- 1 / 2} \widehat {\boldsymbol {\beta}} _ {k} ^ {\lambda} (\mathcal {D} _ {I} ^ {\boldsymbol {G}}).
$$
Observe that the transformed dataset  $\mathcal{D}^G$  satisfies Assumption 2 as the eigenvalues of  $G$  are bounded away from 0 and  $\infty$ . Further, note that both Theorem 1 and Theorem 3 are invariant to linear transformations of the estimator. Thus, we conclude that the results of both the theorems continue to hold. The equivalence path now uses the modified  $\Sigma_G$ , which in turn changes the spectral distribution  $H_{p}$  to that of  $\Sigma_G$ , in defining the end points via (4), given by:
$$
\widetilde {H} _ {p} = \frac {1}{p} \sum_ {i = 1} ^ {p} \mathbb {1} _ {\{\tilde {r} _ {i} \leq r \}},
$$
where  $\widetilde{r}_i$  for  $i\in [p]$  are eigenvalues of  $\Sigma_G$ . This completes the proof.
B.4 Technical lemmas
Lemma B.1 (Relationship between  $\widehat{m}$  and  $\widehat{v}$ ). For  $X\in \mathbb{R}^{n\times p}$  and  $\phi_n = p / n$ , define
$$
\widehat {m} (z; \phi_ {n}) = \frac {1}{p} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} - z I _ {p}\right) ^ {- 1} \right], \quad a n d \quad \widehat {v} (z; \phi_ {n}) = \frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} - z I _ {p}\right) ^ {- 1} \right].
$$
$$
\phi_ {n} z \widehat {m} (z; \phi_ {n}) + \phi_ {n} - 1 = z \widehat {v} (z; \phi_ {n}). \tag {25}
$$
Proof. Let  $r$  be the rank of the matrix  $X$ . Denote by  $s_i$ ,  $i = 1, \dots, r$ , the non-zero eigenvalues of the matrix  $X^\top X$ . Note that these are the same non-zero eigenvalues of the matrix  $XX^\top$ . Define function  $S$  such that for  $z \neq 0$ ,
$$
S (z) = \sum_ {i = 1} ^ {r} \frac {1}{s _ {i} - z}.
$$
For  $z \neq 0$ , write out  $\widehat{v}$  and  $\widehat{m}$  in terms of  $S(z)$  as:
$$
\widehat {v} (z; \phi_ {n}) = \frac {1}{n} \sum_ {i = 1} ^ {r} \frac {1}{s _ {i} - z} - \frac {1}{n} \frac {n - r}{z} = \frac {1}{n} S (z) - \frac {1}{n} \frac {n - r}{z}
$$
$$
\widehat {m} (z; \phi_ {n}) = \frac {1}{p} \sum_ {i = 1} ^ {r} \frac {1}{s _ {i} - z} - \frac {1}{p} \frac {p - r}{z} = \frac {1}{p} S (z) - \frac {1}{p} \frac {p - r}{z}.
$$
We now expand the left-hand side of (25):
$$
\begin{array}{l} \phi_ {n} z \widehat {m} (z; \phi_ {n}) + \phi_ {n} - 1 = \frac {p}{n} z \left(\frac {1}{p} S (z) - \frac {1}{p} \frac {p - r}{z}\right) + \frac {p}{n} - 1 \\ = \frac {1}{n} z S (z) - \frac {p - r}{n} + \frac {p}{n} - 1 \\ = \frac {1}{n} z S (z) - \frac {n - r}{n} + \frac {n - r}{n} - \frac {p - r}{n} + \frac {p}{n} - 1 \\ = z \widehat {v} (z; \phi_ {n}) + \frac {n - p}{n} + \frac {p}{n} - 1 \\ = z \widehat {v} (z; \phi_ {n}), \\ \end{array}
$$
Lemma B.2 (Equivalence of  $\widehat{v}$  and  $v$ ). Suppose Assumptions 1-2 hold. Then it holds that
$$
v (- \lambda ; \phi_ {n}) \simeq \frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} + \lambda I _ {p}\right) ^ {- 1} \right].
$$
Proof. From Lemma B.1, we have
$$
z \widehat {v} (z; \phi_ {n}) + (1 - \phi_ {n}) = \phi_ {n} z \widehat {m} (z; \phi_ {n}).
$$
Substituting  $z = -\lambda$  yields that
$$
\lambda \frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} + \lambda I _ {p}\right) ^ {- 1} \right] + (\phi_ {n} - 1) = \phi_ {n} \lambda \frac {1}{p} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} + \lambda I _ {p}\right) ^ {- 1} \right]. \tag {26}
$$
From Corollary E.4, we have
$$
\frac {1}{v (- \lambda ; \phi_ {n})} = \lambda + \phi \mathrm {t r} [ \Sigma (v (- \lambda ; \phi_ {n}) \Sigma + I _ {p}) ^ {- 1} ] / p,
$$
$$
\frac {1}{p} \mathrm {t r} [ (v (- \lambda ; \phi_ {n}) \Sigma + I _ {p}) ^ {- 1} ] \simeq \frac {1}{p} \lambda \mathrm {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} + \lambda I _ {p}\right) ^ {- 1} \right].
$$
$$
\begin{array}{l} 1 = \lambda v (- \lambda ; \phi_ {n}) + \phi_ {n} \operatorname {t r} [ v (- \lambda ; \phi_ {n}) \Sigma (v (- \lambda ; \phi_ {n}) \Sigma + I _ {p}) ^ {- 1} ] / p \\ = \lambda v (- \lambda ; \phi_ {n}) + \phi_ {n} - \phi_ {n} \operatorname {t r} \left[ \left(v (- \lambda ; \phi_ {n}) \Sigma + I _ {p}\right) ^ {- 1} \right]) / p \\ \simeq \lambda v (- \lambda ; \phi_ {n}) + \phi_ {n} - \phi_ {n} \lambda \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} + \lambda I _ {p}\right) ^ {- 1} \right] / p \\ = 1 + \lambda v (- \lambda ; \phi_ {n}) - \frac {1}{n} \lambda \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} + \lambda I _ {p}\right) ^ {- 1} \right], \\ \end{array}
$$
where the last equality follows from (26). This concludes the proof.
C Proof of results in Section 5
C.1 Proof of Theorem 6
By Assumptions 3-4, for  $\mathbf{A} = \mathbf{\Sigma}^{1/2}$  and  $\mathbf{b} = \mathbf{0}$ ,  $R_p(\lambda; \phi, \psi)$ , as defined in (11), converges to
$$
R _ {p} (\lambda ; \phi , \psi) \xrightarrow {\text {a . s .}} \rho^ {2} \widetilde {c} (- \lambda ; \phi , \psi) + \sigma^ {2} \widetilde {v} (- \lambda ; \phi , \psi) =: \mathscr {R} (\lambda ; \phi , \psi) \tag {27}
$$
where the nonnegative constants  $\widetilde{c}(-\lambda; \phi, \psi)$  and  $\widetilde{v}(-\lambda; \phi, \psi)$  are defined through the following equations:
$$
\widetilde {v} (- \lambda ; \phi , \psi) = \frac {\phi \int \frac {r ^ {2}}{(1 + v (- \lambda ; \psi) r) ^ {2}} \mathrm {d} H (r)}{v (- \lambda ; \psi) ^ {- 2} - \phi \int \frac {r ^ {2}}{(1 + v (- \lambda ; \psi) r) ^ {2}} \mathrm {d} H (r)},
$$
$$
\widetilde {c} (- \lambda ; \phi , \psi) = (\widetilde {v} (- \lambda ; \phi , \psi) + 1) \int \frac {r}{1 + v (- \lambda ; \psi) r} \mathrm {d} G (r),
$$
$$
\frac {1}{v (- \lambda ; \psi)} = \lambda + \psi \int \frac {r}{1 + v (- \lambda ; \psi) r} \mathrm {d} H (r).
$$
From the proof of Theorem 1, we also have that  $\mathcal{R}(\lambda ;\phi ,\psi)\simeq R_p(\lambda ;\phi ,\psi)\simeq R(\widehat{\beta}_{k,\infty}^{\lambda};\Sigma ,0,\beta_0)$
Since  $\mathcal{R}(0;\phi ,\psi)$  is a continuous function of  $\phi$  and  $v(0;\psi)$  and is increasing in  $\phi$  for any fixed  $\psi$ , it follows that for  $0 < \phi_1\leq \phi_2 < \infty$ ,
$$
\min  _ {\psi \geq \phi_ {1}} \mathcal {R} (0; \phi_ {1}, \psi) \leq \min  _ {\psi \geq \phi_ {2}} \mathcal {R} (0; \phi_ {1}, \psi) \leq \min  _ {\psi \geq \phi_ {2}} \mathcal {R} (0; \phi_ {2}, \psi),
$$
where the first inequality follows because  $\{\psi : \psi \geq \phi_1\} \supseteq \{\psi : \psi \geq \phi_2\}$ , and the second inequality follows because  $\mathcal{R}(0; \phi, \psi)$  is increasing in  $\phi$  for a fixed  $\psi$ . Thus,  $\min_{\psi \geq \phi} \mathcal{R}(0; \phi, \psi)$  is a continuous and monotonically increasing function in  $\phi$ .