SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since  $\mathbf{z}_i, \mathbf{z}_j$ , and  $\mathbf{R}_{-(ij)}$  are mutually independent, by Lemma S.8.5 of [51], we have  $\langle \mathbf{z}_i, \mathbf{R}_{-(ij)} \mathbf{z}_j \rangle / n \xrightarrow{\text{a.s.}} 0$  and the denominator concentrates on  $\lambda^2$ . By Lemma G.5 (1) of [13], it follows that  $\max_{1 \leq i \neq j \leq n} d_{ij} \xrightarrow{\text{a.s.}} \lambda^2$ . Thus, there exists  $N_0 \in \mathbb{N}$  such that for all  $n \geq N_0$ ,  $\max_{1 \leq i \neq j \leq n} d_{ij} \geq \lambda^2 / 2$  almost surely. Then, it follows that for  $n \geq N_0$ ,
$$
\begin{array}{l} \left| \frac {1}{p} \sum_ {1 \leq i \neq j \leq n} Y _ {i j} g \left(\boldsymbol {z} _ {i}\right) g \left(\boldsymbol {z} _ {j}\right) \right| \leq \frac {2}{\lambda^ {2}} \frac {1}{n ^ {2}} \sum_ {1 \leq i \neq j \leq n} | \langle \boldsymbol {z} _ {i}, \boldsymbol {R} _ {- (i j)} \boldsymbol {z} _ {j} \rangle g (\boldsymbol {z} _ {i}) g (\boldsymbol {z} _ {j}) | \\ = \frac {2}{\lambda^ {2}} \frac {1}{n ^ {2}} \sum_ {1 \leq i \neq j \leq n} \left(\boldsymbol {z} _ {i} g (\boldsymbol {z} _ {i})\right) ^ {\top} \boldsymbol {R} _ {- (i j)} \left(\boldsymbol {z} _ {j} g (\boldsymbol {z} _ {j})\right) \\ \xrightarrow {\mathrm {a . s .}} 0, \\ \end{array}
$$
where the last convergence is from Lemma S.8.5 of [51] and Lemma G.5 (2) of [13].
Lemma D.4. For any two conforming matrices  $A$  and  $B$  and any  $t \neq 0$ , we have
$$
\boldsymbol {A} ^ {- 1} \boldsymbol {B} \boldsymbol {A} ^ {- 1} = (\boldsymbol {A} ^ {- 1} - (\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1}) / t + t \boldsymbol {A} ^ {- 1} \boldsymbol {B} (\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1} \boldsymbol {B} \boldsymbol {A} ^ {- 1}.
$$
Proof. We recall the Woodbury matrix identity:
$$
\left(\boldsymbol {A} + \boldsymbol {U C V}\right) ^ {- 1} = \boldsymbol {A} ^ {- 1} - \boldsymbol {A} ^ {- 1} \boldsymbol {U} \left(\boldsymbol {C} ^ {- 1} + \boldsymbol {V A} ^ {- 1} \boldsymbol {U}\right) ^ {- 1} \boldsymbol {V A} ^ {- 1}.
$$
This holds for any conforming matrices  $A, U, C$ , and  $V$ . We will need to apply the Woodbury matrix identity twice below.
1. Applying the Woodbury identity for the first time with  $A = A$ ,  $U = t$ ,  $C = B$ , and  $V = 1$ , we get
$$
(\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1} = \boldsymbol {A} ^ {- 1} - t \boldsymbol {A} ^ {- 1} (\boldsymbol {B} ^ {- 1} + t \boldsymbol {A} ^ {- 1}) ^ {- 1} \boldsymbol {A} ^ {- 1}.
$$
Rearranging, this yields
$$
\left(\boldsymbol {A} ^ {- 1} - \left(\boldsymbol {A} + t \boldsymbol {B}\right) ^ {- 1}\right) / t = \boldsymbol {A} ^ {- 1} \left(\boldsymbol {B} ^ {- 1} + t \boldsymbol {A} ^ {- 1}\right) ^ {- 1} \boldsymbol {A} ^ {- 1} = \boldsymbol {A} ^ {- 1} \left(t \boldsymbol {A} ^ {- 1} + \boldsymbol {B} ^ {- 1}\right) ^ {- 1} \boldsymbol {A} ^ {- 1}. \tag {38}
$$
2. Applying the Woodbury identity the second time with  $A = tB$ ,  $U = 1$ ,  $C = A$ , and  $V = 1$  we get
$$
(\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1} = t ^ {- 1} \boldsymbol {B} ^ {- 1} - t ^ {- 1} \boldsymbol {B} ^ {- 1} (\boldsymbol {A} ^ {- 1} + t ^ {- 1} \boldsymbol {B} ^ {- 1}) ^ {- 1} t ^ {- 1} \boldsymbol {B} ^ {- 1}.
$$
Multiplying by  $tB$  from the left yields
$$
t \boldsymbol {B} (A + t \boldsymbol {B}) ^ {- 1} = I - \left(\boldsymbol {A} ^ {- 1} + t ^ {- 1} \boldsymbol {B} ^ {- 1}\right) ^ {- 1} t ^ {- 1} \boldsymbol {B} ^ {- 1} = I - \left(t \boldsymbol {A} ^ {- 1} + \boldsymbol {B} ^ {- 1}\right) ^ {- 1} \boldsymbol {B} ^ {- 1}.
$$
Now, multiplying by  $B$  on the right, notice that
$$
t \boldsymbol {B} (A + t \boldsymbol {B}) ^ {- 1} \boldsymbol {B} = \boldsymbol {B} - \left(t \boldsymbol {A} ^ {- 1} + \boldsymbol {B} ^ {- 1}\right) ^ {- 1}. \tag {39}
$$
From (38), observe that
$$
\begin{array}{l} \boldsymbol {A} ^ {- 1} \boldsymbol {B} \boldsymbol {A} ^ {- 1} - \left(\boldsymbol {A} ^ {- 1} - (\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1}\right) / t = \boldsymbol {A} ^ {- 1} \boldsymbol {B} \boldsymbol {A} ^ {- 1} - \boldsymbol {A} ^ {- 1} (t \boldsymbol {A} ^ {- 1} + \boldsymbol {B} ^ {- 1}) ^ {- 1} \boldsymbol {A} ^ {- 1} \\ = \boldsymbol {A} ^ {- 1} (\boldsymbol {B} - (t \boldsymbol {A} ^ {- 1} + \boldsymbol {B} ^ {- 1}) ^ {- 1}) \boldsymbol {A} ^ {- 1}. \\ \end{array}
$$
Using (39), we then have
$$
\boldsymbol {A} ^ {- 1} \boldsymbol {B} \boldsymbol {A} ^ {- 1} - (\boldsymbol {A} ^ {- 1} - (\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1}) / t = t \boldsymbol {A} ^ {- 1} \boldsymbol {B} (\boldsymbol {A} + t \boldsymbol {B}) ^ {- 1} \boldsymbol {B} \boldsymbol {A} ^ {- 1}.
$$
Rearranging, we arrive at the desired equality:
D.3 Concentration of energy of linear and nonlinear components
Lemma D.5. Under Assumptions 1-2, the best linear estimator  $\beta_0$  and the nonlinear component  $f_{\mathrm{NL}}$  as defined in (10) satisfies that  $\| \beta_0\| _2$  and  $\| f_{\mathrm{NL}}\|_{L_{4 + \delta}}$  are bounded almost surely.
Proof. By Jensen's inequality and Assumption 1, we have  $\mathbb{E}[y^2] \leq \mathbb{E}[y^{4 + \delta}]^{2 / (4 + \delta)} \leq C_0^{2 / (4 + \delta)}$  for some constant  $C_0 > 0$ .
By the orthogonality, we have that  $\mathbb{E}[y^2] = \mathbb{E}[(\boldsymbol{x}^\top \boldsymbol{\beta}_0)^2] + \| f_{\mathrm{NL}}\|_{L^2}^2$ , which implies that  $\mathbb{E}[(\boldsymbol{x}^\top \boldsymbol{\beta}_0)^2]$  and  $\| f_{\mathrm{NL}}\|_{L^2}^2$  is bounded. Since by Assumption 2, the eigenvalues of  $\pmb{\Sigma}$  is lower bounded by  $r_{\min} > 0$ , we have that  $\mathbb{E}[(\boldsymbol{x}^\top \boldsymbol{\beta}_0)^2] = \beta_0\Sigma \beta_0 \geq r_{\min}\|\beta_0\|_2^2$  and thus  $\|\beta_0\|_2$  is bounded.
Since  $f_{\mathrm{NL}} = y - \pmb{x}^{\top}\beta_{0}$ , by triangle inequality we have that
$$
\begin{array}{l} \left\| f _ {\mathrm {N L}} \right\| _ {L _ {4 + \delta}} \leq \left\| y \right\| _ {L _ {4 + \delta}} + \left\| \boldsymbol {x} ^ {\top} \boldsymbol {\beta} _ {0} \right\| _ {L _ {4 + \delta}} \\ \leq \| y \| _ {L _ {4 + \delta}} + C \| \beta_ {0} \| _ {2}) \\ \end{array}
$$
for some constant  $C > 0$ . The second inequality of the above display is from Lemma 7.8 of [57]. Since  $\| y\|_{L_{4 + \delta}}$  and  $\| \beta_0\| _2$  are bounded, it follows that  $\| f_{\mathrm{NL}}\|_{L_{4 + \delta}}$  is also bounded.
E Asymptotic equivalents: background and known results
Preliminary background. In several proofs, we employ the concept of asymptotic equivalence of sequences of random matrices; see [12, 44, 49, 51]. This section provides a brief overview of the associated terminology and calculus principles.
Definition E.1 (Asymptotic equivalence). Consider sequences  $\{A_p\}_{p \geq 1}$  and  $\{B_p\}_{p \geq 1}$  of (random or deterministic) matrices of growing dimensions. We say that  $A_p$  and  $B_p$  are equivalent and write  $A_p \simeq B_p$  if  $\lim_{p \to \infty} |\operatorname{tr}[C_p(A_p - B_p)]| = 0$  almost surely for every sequence of random matrices  $C_p$  independent to  $A_p$  and  $B_p$ , with bounded trace norm such that  $\lim \sup_{p \to \infty} \|C_p\|_{\mathrm{tr}} < \infty$  almost surely.
The notion of asymptotic equivalence of two sequences of random matrices above can be further extended to incorporate conditioning on another sequence of random matrices; see [12] for more details.
Definition E.2 (Conditional asymptotic equivalence). Consider sequences  $\{\pmb{A}_p\}_{p\geq 1}$ ,  $\{\pmb{B}_p\}_{p\geq 1}$  and  $\{\pmb{D}_p\}_{p\geq 1}$  of (random or deterministic) matrices of growing dimensions. We say that  $\pmb{A}_p$  and  $\widehat{\pmb{B}}_p$  are equivalent given  $\pmb{D}_p$  and write  $\pmb{A}_p\simeq \pmb{B}_p\mid \pmb{D}_p$  if  $\lim_{p\to \infty}|\operatorname {tr}[C_p(\pmb {A}_p - \pmb {B}_p)]| = 0$  almost surely conditional on  $\{\pmb {D}_p\}_{p\geq 1}$ , i.e.,
$$
\mathbb {P} \left(\lim  _ {p \rightarrow \infty} | \operatorname {t r} [ \boldsymbol {C} _ {p} (\boldsymbol {A} _ {p} - \boldsymbol {B} _ {p}) ] | = 0 \Bigg | \{\boldsymbol {D} _ {p} \} _ {p \geq 1}\right) = 1,
$$
for any sequence of random matrices  $C_p$  independent to  $\mathbf{A}_p$  and  $B_p$  conditional on  $\mathbf{D}_p$ , with bounded trace norm such that  $\lim \sup \| C_p\|_{\mathrm{tr}} < \infty$  as  $p\to \infty$ .
Below we summarize the calculus rules for conditional asymptotic equivalence in Definition E.2. These are adapted from Lemma S.7.4 and S.7.6 of [12].
Lemma E.3 (Calculus of asymptotic equivalents). Let  $A_{p}, B_{p}, C_{p}$  and  $D_{p}$  be sequences of random matrices. The calculus of asymptotic equivalents satisfies the following properties:
(1) Equivalence: The relation  $\simeq$  is an equivalence relation.
(2) Sum: If  $\mathbf{A}_p \simeq \mathbf{B}_p \mid \mathbf{E}_p$  and  $\mathbf{C}_p \simeq \mathbf{D}_p \mid \mathbf{E}_p$ , then  $\mathbf{A}_p + \mathbf{C}_p \simeq \mathbf{B}_p + \mathbf{D}_p \mid \mathbf{E}_p$ .
(3) Product: If  $\mathbf{A}_p$  has bounded operator norms such that  $\lim \sup_{p\to \infty}\| \mathbf{A}_p\|_{\mathrm{op}} < \infty$ ,  $\mathbf{A}_p$  is conditional independent to  $\mathbf{B}_p$  and  $\mathbf{C}_p$  given  $\mathbf{E}_p$  for  $p\geq 1$ , and  $\mathbf{B}_p\simeq \mathbf{C}_p\mid \mathbf{E}_p$ , then  $\mathbf{A}_p\mathbf{B}_p\simeq \mathbf{A}_p\mathbf{C}_p\mid \mathbf{E}_p$ .
(4) Trace: If  $\mathbf{A}_p \simeq \mathbf{B}_p \mid \mathbf{E}_p$ , then  $\mathrm{tr}[\mathbf{A}_p] / p - \mathrm{tr}[\mathbf{B}_p] / p \to 0$  almost surely when conditioning on  $\mathbf{E}_p$ .
(5) Differentiation: Suppose  $f(z, \mathbf{A}_p) \simeq g(z, \mathbf{B}_p) \mid \mathbf{E}_p$  where the entries of  $f$  and  $g$  are analytic functions in  $z \in S$  and  $S$  is an open connected subset of  $\mathbb{C}$ . Suppose for any sequence  $\mathbf{C}_p$  of deterministic matrices with bounded trace norm we have  $|\operatorname{tr}[C_p(f(z, \mathbf{A}_p) - g(z, \mathbf{B}_p))]| \leq M$  for every  $p$  and  $z \in S$ . Then, we have  $f'(z, \mathbf{A}_p) \simeq g'(z, \mathbf{B}_p) \mid \mathbf{E}_p$  for every  $z \in S$ , where the derivatives are taken entrywise with respect to  $z$ .
(6) Unconditioning: If  $\mathbf{A}_p \simeq \mathbf{B}_p \mid \mathbf{E}_p$ , then  $\mathbf{A}_p \simeq \mathbf{B}_p$ .
(7) Substitution: Let  $v: \mathbb{R}^{p \times p} \to \mathbb{R}$  and  $f(v(\boldsymbol{C}), \boldsymbol{C}): \mathbb{R}^{p \times p} \to \mathbb{R}^{p \times p}$  be a matrix function for matrix  $\boldsymbol{C} \in \mathbb{R}^{p \times p}$  and  $p \in \mathbb{N}$ , that is continuous in the first augment with respect to operator norm. If  $v(\boldsymbol{C}) \stackrel{\text{a.s.}}{=} v(\boldsymbol{D})$  such that  $\boldsymbol{C}$  is independent to  $\boldsymbol{D}$ , then  $f(v(\boldsymbol{C}), \boldsymbol{C}) \simeq f(v(\boldsymbol{D}), \boldsymbol{C}) \mid \boldsymbol{C}$ .
Standard ridge resolvents and various extensions. In this section, we collect various asymptotic equivalents. The following corollary is a simple consequence of Theorem 1 in [58]. It provides deterministic equivalent for the scaled ridge resolvent.
Corollary E.4 (Deterministic equivalent for scaled ridge resolvent). Suppose  $\boldsymbol{x}_i \in \mathbb{R}^p$ ,  $1 \leq i \leq n$ , are i.i.d. random vectors such that each  $\boldsymbol{x}_i = \boldsymbol{z}_i \boldsymbol{\Sigma}^{1/2}$ , where  $\boldsymbol{z}_i$  is a random vector consisting of i.i.d. entries  $z_{ij}$ ,  $1 \leq j \leq p$ , satisfying  $\mathbb{E}[z_{ij}] = 0$ ,  $\mathbb{E}[z_{ij}^2] = 1$ , and  $\mathbb{E}[|z_{ij}|^{8 + \alpha}] \leq M_\alpha$  for some constants  $\alpha > 0$  and  $M_\alpha < \infty$ , and  $\boldsymbol{\Sigma} \in \mathbb{R}^{p \times p}$  is a positive semidefinite matrix satisfying  $0 \preceq \boldsymbol{\Sigma} \preceq r_{\max} I_p$  for some constant  $r_{\max} < \infty$  (independent of  $p$ ). Let  $X \in \mathbb{R}^{n \times p}$  the concatenated matrix with  $x_i^\top$ ,  $1 \leq i \leq n$ , as rows, and let  $\widehat{\boldsymbol{\Sigma}} \in \mathbb{R}^{p \times p}$  denote the random matrix  $X^\top X / n$ . Let  $\gamma = p / n$ . Then, for  $z \in \mathbb{C}^+$ , as  $n, p \to \infty$  such that  $0 < \lim \inf \gamma \leq \lim \sup \gamma < \infty$ , and  $\lambda > 0$ , we have
$$
\lambda (\widehat {\boldsymbol {\Sigma}} + \lambda \boldsymbol {I} _ {p}) ^ {- 1} \simeq (v (- \lambda ; \gamma) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1},
$$
where  $v(-\lambda ;\gamma) > 0$  is the unique solution to the fixed-point equation
$$
\frac {1}{v (- \lambda ; \gamma)} = \lambda + \gamma \int \frac {r}{1 + v (- \lambda ; \gamma) r} d H _ {p} (r). \tag {40}
$$
Here  $H_{n}$  is the empirical distribution (supported on  $\mathbb{R}_{\geq 0}$ ) of the eigenvalues of  $\pmb{\Sigma}$ .
Side remark: The parameter  $v(-\lambda ;\gamma)$  in Corollary E.4 is also the companion Stieltjes transform of the spectral distribution of the sample covariance matrix  $\widehat{\Sigma}$ . It is also the Stieltjes transform of the spectral distribution of the gram matrix  $XX^{\top} / n$ . This is essentially the characterization we use in proving Proposition 4.
The following lemma uses Corollary E.4 along with calculus of deterministic equivalents (from Lemma E.3), and provides deterministic equivalents for resolvents needed to obtain asymptotic bias and variance of standard ridge regression. It is adapted from Lemma S.6.10 of [51].
Lemma E.5 (Deterministic equivalents for ridge resolvents associated with generalized bias and variance). Suppose  $\boldsymbol{x}_i \in \mathbb{R}^p$ ,  $1 \leq i \leq n$ , are i.i.d. random vectors with each  $\boldsymbol{x}_i = \boldsymbol{z}_i \boldsymbol{\Sigma}^{1/2}$ , where  $\boldsymbol{z}_i \in \mathbb{R}^p$  is a random vector that contains i.i.d. random variables  $z_{ij}$ ,  $1 \leq j \leq p$ , each with  $\mathbb{E}[z_{ij}] = 0$ ,  $\mathbb{E}[z_{ij}^2] = 1$ , and  $\mathbb{E}[|z_{ij}|^{4 + \alpha}] \leq M_\alpha$  for some constants  $\alpha > 0$  and  $M_\alpha < \infty$ , and  $\boldsymbol{\Sigma} \in \mathbb{R}^{p \times p}$  is a positive semidefinite matrix with  $r_{\min} \boldsymbol{I}_p \preceq \boldsymbol{\Sigma} \preceq r_{\max} \boldsymbol{I}_p$  for some constants  $r_{\min} > 0$  and  $r_{\max} < \infty$  (independent of  $p$ ). Let  $X \in \mathbb{R}^{n \times p}$  be the concatenated random matrix with  $x_i$ ,  $1 \leq i \leq n$ , as its rows, and define  $\widehat{\boldsymbol{\Sigma}} = X^\top X / n \in \mathbb{R}^{p \times p}$ . Let  $\gamma = p / n$ . Then, for  $\lambda > 0$ , as  $n, p \to \infty$  with  $0 < \lim \inf \gamma \leq \lim \sup \gamma < \infty$ , the following statements hold:
(1) Bias of ridge regression:
$$
\begin{array}{l} \lambda^ {2} \left(\widehat {\boldsymbol {\Sigma}} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {A} \left(\widehat {\boldsymbol {\Sigma}} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \\ \simeq (v (- \lambda ; \gamma , \pmb {\Sigma}) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1} (\widetilde {v} _ {b} (- \lambda ; \gamma , \pmb {\Sigma}, \pmb {A}) \pmb {\Sigma} + \pmb {A}) (v (- \lambda ; \gamma , \pmb {\Sigma}) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1}. \\ \end{array}
$$
(2) Variance of ridge regression:
$$
(\widehat {\pmb {\Sigma}} + \lambda \pmb {I} _ {p}) ^ {- 2} \widehat {\pmb {\Sigma}} \pmb {A} \simeq \widetilde {v} _ {v} (- \lambda ; \gamma , \pmb {\Sigma}) (v (- \lambda ; \gamma , \pmb {\Sigma}) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 2} \pmb {\Sigma} \pmb {A}.
$$
Here  $v(-\lambda; \gamma, \pmb{\Sigma}) > 0$  is the unique solution to the fixed-point equation
$$
\frac {1}{v (- \lambda ; \gamma , \boldsymbol {\Sigma})} = \lambda + \int \frac {\gamma r}{1 + v (- \lambda ; \gamma , \boldsymbol {\Sigma}) r} d H _ {n} (r; \boldsymbol {\Sigma}), \tag {41}
$$
and  $\widetilde{v}_b(-\lambda ;\gamma ,\Sigma)$  and  $\widetilde{v}_v(-\lambda ;\gamma ,\Sigma)$  are defined through  $v(-\lambda ;\gamma ,\Sigma)$  by the following equations:
$$
\widetilde {v} _ {b} (- \lambda ; \gamma , \boldsymbol {\Sigma}, \boldsymbol {A}) = \frac {\gamma \operatorname {t r} [ \boldsymbol {A} \boldsymbol {\Sigma} (v (- \lambda ; \gamma , \boldsymbol {\Sigma}) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} ] / p}{v (- \lambda ; \gamma , \boldsymbol {\Sigma}) ^ {- 2} - \int \gamma r ^ {2} (1 + v (- \lambda ; \gamma , \boldsymbol {\Sigma}) r) ^ {- 2} \mathrm {d} H _ {n} (r ; \boldsymbol {\Sigma})}, \tag {42}
$$
$$
\widetilde {v} _ {v} (- \lambda ; \gamma , \boldsymbol {\Sigma}) ^ {- 1} = v (- \lambda ; \gamma , \boldsymbol {\Sigma}) ^ {- 2} - \int \gamma r ^ {2} (1 + v (- \lambda ; \gamma , \boldsymbol {\Sigma}) r) ^ {- 2} d H _ {n} (r; \boldsymbol {\Sigma}), \tag {43}
$$
where  $H_{n}(\cdot ;\Sigma)$  is the empirical distribution (supported on  $[r_{\mathrm{min}},r_{\mathrm{max}}])$  of the eigenvalues of  $\pmb{\Sigma}$ .
Though Lemma E.5 states the dependency explicitly, we will simply write  $H_{p}(r)$ ,  $v(-\lambda ;\gamma)$ ,  $\widetilde{v}_b(-\lambda ;\gamma ,A)$ , and  $\widetilde{v}_v(-\lambda ;\gamma)$  to denote  $H_{n}(r;\Sigma)$ ,  $v(-\lambda ;\gamma ,\Sigma)$ ,  $\widetilde{v}_b(-\lambda ;\gamma ,\Sigma ,A)$ , and  $\widetilde{v}_v(-\lambda ;\gamma ,\Sigma)$ , respectively, for simplifying various notations when it is clear from the context. When  $\mathbf{A} = \mathbf{\Sigma}$ , we simply write  $\widetilde{v}_b(-\lambda ;\gamma) = \widetilde{v}_b(-\lambda ;\gamma ,\mathbf{A})$
F.1 Reproducibility and compute details
The source code for generating all of our figures is included with the supplementary material. The source code also includes details about the computational resources used to run the code and other timing details.
F.2 Simulation details
The covariance matrix of an auto-regressive process of order 1 (AR(1)) is given by  $\Sigma_{\mathrm{ar1}}$ , where  $(\Sigma_{\mathrm{ar1}})_{ij} = \rho_{\mathrm{ar1}}^{|i - j|}$  for some parameter  $\rho_{\mathrm{ar1}} \in (0,1)$ . Define  $\beta_0 := \frac{1}{5} \sum_{j=1}^{5} w_{(j)}$  where  $w_{(j)}$  is the eigenvector of  $\Sigma_{\mathrm{ar1}}$  associated with the top  $j$ th eigenvalue  $r_{(j)}$ . We generated data  $(x_i, y_i)$  for  $i = 1, \ldots, n$  from a nonlinear model:
$$
y _ {i} = \pmb {x} _ {i} ^ {\top} \beta_ {0} + \frac {1}{p} (\| \pmb {x} _ {i} \| _ {2} ^ {2} - \mathrm {t r} [ \pmb {\Sigma} _ {\mathrm {a r 1}} ]) + \varepsilon_ {i}, \quad \pmb {x} _ {i} = \pmb {\Sigma} _ {\mathrm {a r 1}} ^ {\frac {1}{2}} \pmb {z} _ {i}, \quad z _ {i j} \stackrel {i i d} {\sim} \frac {t _ {5}}{\sigma_ {5}}, \quad \varepsilon_ {i} \sim \frac {t _ {5}}{\sigma_ {5}}, \quad (\mathbf {M - A R 1})
$$
where  $\sigma_{5} = \sqrt{5 / 3}$  is the standard deviation of  $t_5$  distribution. The benefit of using the above nonlinear model is that we can clearly separate the linear and the nonlinear components and compute the quantities of interest because  $\beta_0$  happens to be the best linear projection.
For the simulations, we set  $\rho_{\mathrm{ar1}} = 0.5$ . For finite ensembles, the risks are averaged across 50 simulations.
F.3 Risk comparisons along equivalence paths for finite ensembles
Figure F5 verifies the following linear (in  $M$ ) relationship along the path in Theorem 3 for finite ensembles:  $R(\widehat{\beta}_{[p / \psi_1\rfloor ,M};A,b,\beta_0) - R(\widehat{\beta}_{[p / \psi_2\rfloor ,M};A,b,\beta_0)\simeq \Delta /M$ , for some  $\Delta$  (independent of  $M$ ), which is eventually almost surely bounded.
Figure F5: The range (the difference between the maximum and the maximum along the path) of the finite-sample generalized risk of  $M$ -ensemble ridge estimators on the path through  $(\lambda, \psi) = (0,2)$ , for varying ensemble size  $M$  under the same setting as in Figure 1. The generalized risks include the estimation risk, the training error, the prediction risk, and the out-of-distribution (OOD) prediction risk.
Ensemble size  $M$
F.4 Illustration of risk monotonicity of optimal ridge
Figure F6: Illustration of risk monotonicity of optimal subsampled ridgeless regression versus optimal regularized ridge regression. The underlying model has a non-isotropic covariance as described in Appendix F.2 with a linearized signal-to-noise ratio of 0.6 and  $\| f_{\mathrm{NL}}\|_{L^2} = 1$ . This results in a null risk of 1.6, which is kept the same across all regression problems. The left panel shows the limiting risk of the full-ensemble ridgeless regression at various data and subsample aspect ratios  $(\phi ,\psi)$ . The right panel shows the limiting risk of the ridge predictor (on the full data) at various data aspect ratios and regularization penalties  $(\phi ,\lambda)$ . Optimal risks for each data aspect ratio are highlighted using slender red lines in both panels. Observe that the optimal risk in both cases is increasing as a function of  $\phi$ . In the left panel, in addition, for every subsample aspect ratio  $\psi$ , the risk of subsampled ridgeless is increasing in the data aspect ratio  $\phi$ . This in turn implies the risk monotonicity claim for the optimal ridge regression in Theorem 6 through a slick triangle inequality argument (see the proof in Appendix C for more details).
F.5 Real-world datasets
We conduct experiments on real-world datasets to examine the equivalence in a more general setting. We utilized three image datasets for our experimental analysis: CIFAR 10, MNIST, and USPS [59]. For the CIFAR 10 dataset, we subset the images labeled as "dog" and "cat". For other datasets, we subset the images labeled "3" and "8". Then we treat them as binary labels  $y \in \{0,1\}$  and use the flattened image as our feature vector  $x$ . The training sample sizes, the feature dimensions, and the test sample sizes  $(n,p,n_{\mathrm{te}})$  are (10000, 3072, 2000), (12873, 784, 2145), and (22358, 3072, 7981) for the three datasets, respectively.
For the first experiment in Figure 3, we fix  $\phi = p / n$  and  $\overline{\psi} = 4\phi$  using the training set of CIFAR-10. For  $\lambda_{\overline{\psi}} \in \{0.01, 0.05, 0.1, 1\}$ , we compute the data-dependent value of  $\overline{\lambda}$  based on Proposition 4. Each value of  $\lambda_{\overline{\psi}}$  gives a path between  $(\lambda_{\overline{\psi}}, \overline{\psi})$  and  $(\overline{\lambda}, \phi)$ .
For the second experiment in Figure F7, by varying the values of subsample aspect ratios  $\overline{\psi}$ , we compare the random Gaussian linear projection of the estimators and the prediction risk at the two endpoints  $(\bar{\lambda}, \phi)$  and  $(0, \overline{\psi})$ , on CIFAR-10, MNIST, and USPS datasets.
Figure F7: The functional equivalences of subsampling and ridge regularization on different datasets. For each dataset with an aspect ratio  $\phi = p / n$  and each value of  $\overline{\psi}$ , the corresponding ridge penalty  $\bar{\lambda}$  is estimated by Proposition 4. The two models with ridge penalty and subsample aspect ratio  $(\bar{\lambda},\phi)$  and  $(0,\overline{\psi})$  are compared. Note that the model corresponding to  $(\bar{\lambda},\phi)$  is ridge regression at ridge penalty  $\bar{\lambda}$  without subsampling, and the model corresponding to  $(0,\overline{\psi})$  is full-ensemble ridgeless at a subsample aspect ratio  $\overline{\psi}$ .
F.6 Random features model
The data  $(x_{i},y_{i})$  is generate from the nonlinear model
$$
y _ {i} = \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta} _ {0} + \frac {1}{p} (\| \boldsymbol {x} _ {i} \| _ {2} ^ {2} - p) + \varepsilon_ {i},
$$
where  $x_{ij} \stackrel{iid}{\sim} \mathcal{N}(0,1)$ , and  $\varepsilon_i \sim \mathcal{N}(0,1)$ . The prediction risk of the ridge ensemble ( $M = 100$ ) is computed based on the random feature  $\varphi(\boldsymbol{F}\boldsymbol{x}_i)$  and the response  $y_i$ , where  $\boldsymbol{F} \in \mathbb{R}^{d \times p}$  is the random weight matrix with  $F_{ij} \stackrel{iid}{\sim} \mathcal{N}(0,p^{-1})$ . Here,  $\varphi$  is a nonlinear activation function (sigmoid, ReLU, or tanh) that is applied entry-wise to  $\boldsymbol{F}\boldsymbol{x}_i$ . For the experiment, we set  $p = 250$ ,  $d = 500$ , and  $\phi = d / n = 0.1$ .
F.7 Kernel ridge regression
For a given feature map  $\Phi : \mathbb{R}^p \to \mathbb{R}^d$ , the kernel ridge estimator is defined as:
By using the kernel trick, the above optimization problem is equivalent to solving: