SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{M} \sum_ {\ell = 1} ^ {M} \mathrm {t r} \left[ \pmb {K} _ {I _ {\ell}} ^ {+} \right] = \mathrm {t r} \left[ \left(\pmb {K} _ {[ n ]} + \frac {n}{p} \bar {\lambda} _ {n} \pmb {I} _ {n}\right) ^ {- 1} \right].
$$
Define the data-dependent path  $\mathcal{P}_n = \mathcal{P}(\overline{\lambda}_n;\phi_n,\overline{\psi}_n)$ . Then, the conclusions in Theorem 1, Proposition 2, and Theorem 3 continue to hold for kernel ridge ensembles with  $\mathcal{P}$  replaced by  $\mathcal{P}_n$ .
The empirical evidence here strongly suggests that the relationship between subsampling and ridge regression, which we have proved for ridge regression, holds true at least when the gram matrix "linearizes" in the sense of asymptotic equivalence [14, 54, 55]. This intriguing observation opens up a whole host of avenues towards fully understanding the universality of this relationship and establishing precise connections for a broader range of models [17, 18, 20, 21]. We hope to share more on these compelling directions in the near future.
Tying other implicit regularizations. Finally, as noted in related work, the equivalences between ridge regularization and subsampling established in this paper naturally offer opportunities to interpret and understand the other implicit regularization effects such as dropout regularization [27, 28], early stopping in gradient descent variants [36, 38, 39]. Furthermore, these equivalences provide avenues to understand the combined effects of these forms of implicit regularization, such as the effects of both subsample aggregating and gradient descent in mini-batch gradient descent, and contrast them to explicit regularization. Whether the explicit regularization can always be cleanly expressed as ridge-like regularization or takes a more generic form is an intriguing question. Exploring this is exciting, and we hope our readers also share this excitement!
We are indebted to Ryan Tibshirani, Alessandro Rinaldo, Arun Kuchibhotla, Yuting Wei, Matey Neykov, Edgar Dobriban, Daniel LeJeune, Shamindra Shrotriya, Yuchen Wu for many insightful conversations. We are also grateful to anonymous reviewers of the precursor [13] for encouraging us to work on several follow-up directions that make up parts of this successor. On a more personal note: The lack of proof of risk monotonicity of optimally-tuned ridge in general had been "monotone bugging" (in a good way) the first author, ever since plotting Figure 8 (and other similar plots) in [51]. The resolution in Theorem 6 is of great personal significance to him and he is heartily appreciative of all the collaborators on related threads, especially the second author, for being a dedicated climbing partner on this technical "ascent". At the risk of over analogizing, the metaphor of letting a nut soak in water instead of forcibly cracking it open with a hammer, from Grothendieck<sup>2</sup>, may perhaps describe the proof of Theorem 6. Establishing subsampling and ridge equivalences first and using it to demonstrate ridge monotonicity, in retrospect, seems a much less arduous route than a direct brute-force attempt (with calculus, say). But we will let the readers decide!
This work used the Bridges2 system at the Pittsburgh Supercomputing Center (PSC) through allocations MTH230020 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program. The code for reproducing the results of this paper can be found at https://jaydu1.github.io/overparameterized-ensembling/equiv.
[1] Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. In International Conference on Learning Representations, 2021.
[2] Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.
[3] Peter Buhlmann and Bin Yu. Analyzing bagging. The Annals of Statistics, 30(4):927-961, 2002.
[4] Abraham J. Wyner, Matthew Olson, Justin Bleich, and David Mease. Explaining the success of adaboost and random forests as interpolating classifiers. The Journal of Machine Learning Research, 18(1):1558-1590, 2017.
[5] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.
[6] Lucas Mentch and Siyu Zhou. Randomization as regularization: A degrees of freedom explanation for random forest success. The Journal of Machine Learning Research, 21(1):6918-6953, 2020.
[7] Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
[8] Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Applications to nonorthogonal problems. Technometrics, 12(1):69-82, 1970.
[9] Trevor Hastie. Ridge regularization: An essential concept in data science. Technometrics, 62 (4):426-433, 2020.
[10] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-986, 2022.
[11] Daniel LeJeune, Hamid Javadi, and Richard Baraniuk. The implicit regularization of ordinary least squares ensembles. In International Conference on Artificial Intelligence and Statistics, 2020.
$^{2}$ http://math.stanford.edu/~vakil/216blog/FOAGmay2523public.pdf, page 9.
[50] Chen Cheng and Andrea Montanari. Dimension free ridge regression. arXiv preprint arXiv:2210.08571, 2022.
[51] Pratik Patil, Arun Kumar Kuchibhotla, Yuting Wei, and Alessandro Rinaldo. Mitigating multiple descents: A model-agnostic framework for risk monotonization. arXiv preprint arXiv:2205.12937, 2022.
[52] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
[53] Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932-1964, 2023.
[54] Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "ridgeless" regression can generalize. The Annals of Statistics, 2020.
[55] Mojtaba Sahraee-Ardakan, Melikasadat Emami, Parthe Pandit, Sundeep Rangan, and Alyson K Fletcher. Kernel methods and multi-layer perceptrons learn linear models in high dimensions. arXiv preprint arXiv:2201.08082, 2022.
[56] Antti Knowles and Jun Yin. Anisotropic local laws for random matrices. Probability Theory and Related Fields, 169:257-352, 2017.
[57] László Erdős and Horng-Tzer Yau. A Dynamical Approach to Random Matrix Theory. American Mathematical Society, 2017.
[58] Francisco Rubio and Xavier Mestre. Spectral convergence for a general class of random matrices. Statistics & probability letters, 81(5):592-602, 2011.
[59] TorchVision. Torchvision: PyTorch's computer vision library. https://github.com/pytorch/vision, 2016.
[60] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
Supplementary material for "Generalized equivalences between subsampling and ridge regularization"
This document serves as a supplement to the paper "Generalized equivalences between subsampling and ridge regularization." The initial (unnumbered) section of the supplement provides a summary of the notation used in both the paper and supplement, followed by an organization for the supplement.
Notation. Below we provide an overview of the notation used throughout.
General notation:
Notation	Description
Non-bold	Denotes univariates (calvars, functions, distributions, etc.) (e.g., λ, f, H).
Bold lower case	Denotes vectors (e.g., x, β).
Bold upper case	Denotes matrices (e.g., X).
Calligraphic font	Denotes sets (e.g., D).
Script font	Denotes certain limiting functions (e.g., R in (27)).
N	Set of positive integers.
R, R+	Set of real numbers and positive real numbers.
C, C+	Set of complex numbers and complex numbers with positive imaginary part.
[n]	Set {1, ..., n} for natural number n.
(x)+	The positive part of real number x.
[x], [x]	The floor and ceiling of real number x.
||f||L2	The L2 norm of function f, ||f||L2 = Ex[f2(x)].
||β||2	The l2 norm of vector β.
{v, w}	The inner product of vectors v and w.
XT, X+	The transpose and Moore-Penrose inverse of matrix X ∈ Rn×p.
tr[A], A-1	The trace and inverse (if invertible) of a square matrix A ∈ Rp×p.
Σ1/2	The principal square root of positive semi-definite matrix Σ.
Ip or I	The p × p identity matrix.
||X||op	The operator norm (spectral norm) of real matrix X.
||X||tr	The trace norm (nuclear norm) tr[(X^T X)^1/2] of real matrix X.
f(A)	The matrix Vf(R)V-1 where A = VRV-1 is eigenvalue decomposition, and f(R) is f applied (elementwise) to the diagonals of R.
A ≈ B	The Loewner ordering for symmetric matrices A and B.
p, a.s., d	Almost sure convergence, convergence in probability, and weak convergence.
Specific notation:
Notation	Description
(x,y)	The population random vector in Rp × R.
Dn	A dataset containing i.i.d. samples of (x,y): Dn = {(xi,yi): i ∈ [n]}.
X,y	The feature matrix in Rn×p and the response vector in Rn
D_I	A subsampled dataset D_I = {(xi,yi): i ∈ I}.
βkλ(D_I)	A ridge estimator fitted on D_I with |I| = k observations and ridge penalty λ.
βk,M(D_n; {I_l}M_{l=1})	An M-ensemble ridge estimator fitted on {D_{I_l}}_l^M with subsample size k and ridge penalty λ.
β0	The best (population) linear projection coefficient of y onto x: E[xx^T]^-1E[xy].
f_LI(x)	The best (population) linear projection of y onto x: x^Tβ0.
f_NL(x)	The component of y that is not explained by x: y - x^Tβ0.
f_LI, f_NL	f_LI = Xβ0, f_NL = [f_NL(xi)]_{i∈[n]} .
A note on indexing of sequences:
In the subsequent sections, we will prove the results for  $n, k, p$  being a sequence of integers  $\{n_m\}_{m=1}^{\infty}$ ,  $\{k_m\}_{m=1}^{\infty}$ ,  $\{p_m\}_{m=1}^{\infty}$ . Alternatively, one can also view  $k$  and  $p$  as sequences  $k_n$  and  $p_n$  that are indexed by  $n$ . For notational brevity, we drop the subscript when it is clear from the context.
Organization. Below we outline the structure of the rest of the supplement. The technical lemmas refer to the main ingredients that we use to prove results in the corresponding sections.
Table 3: Outline of the supplement.
Section	Description	Technical lemmas
Appendix A	Proof of Theorem 1, Proposition 2 (from Section 3)	Lemma A.1, Lemma A.2, Lemma A.3
Appendix B	Proof of Theorem 3, Proposition 4, Corollary 5 (from Section 4)	Lemma B.1, Lemma B.2
Appendix C	Proof of Theorem 6 (from Section 5)	Lemma C.1
Appendix D	New asymptotic equivalents, concentration results and other useful lemmas (used in Appendices A–C)	Lemma D.1, Lemma D.2, Lemma D.3, Lemma D.5
Appendix E	Asymptotic equivalents: background and known results (used in Appendices A–D)
Appendix F	Additional details for all the experiments
A Proofs of results in Section 3
Given an observation  $(\pmb{x},y)$ , recall the decomposition  $y = f_{\mathrm{LI}}(\pmb{x}) + f_{\mathrm{NL}}(\pmb{x})$  explained in Section 2. For  $n$  i.i.d. samples from the same distribution as  $(x,y)$ , we define analogously the vector decomposition:
$$
\boldsymbol {y} = \boldsymbol {f} _ {\mathrm {L I}} + \boldsymbol {f} _ {\mathrm {N L}}, \tag {10}
$$
where  $\pmb{f}_{\mathrm{LI}} = \pmb{X}\pmb{\beta}_0$  and  $\pmb{f}_{\mathrm{NL}} = [f_{\mathrm{NL}}(\pmb{x}_i)]_{i\in [n]}$ . Let  $n_A = \operatorname{nrow}(\mathbf{A})$ . Note that
$$
R (\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda}; \pmb {A}, \pmb {b}, \boldsymbol {\beta} _ {0}) = R (\widehat {\pmb {\beta}} _ {k, \infty} ^ {\lambda}; \pmb {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}) + 2 n _ {A} ^ {- 1} \pmb {b} ^ {\top} \pmb {A} (\widehat {\pmb {\beta}} _ {k, \infty} ^ {\lambda} - \boldsymbol {\beta} _ {0}) + n _ {A} ^ {- 1} \| \pmb {b} \| _ {2} ^ {2}.
$$
By Theorem 3, the cross term vanishes, i.e.,  $n_A^{-1}\pmb{b}^\top \pmb {A}(\widehat{\beta}_{k,\infty}^{\lambda} - \beta_0)\xrightarrow{\mathrm{a.s.}}0$ . We then have
$$
\left| R \left(\widehat {\boldsymbol {\beta}} _ {k _ {1}, \infty} ^ {\lambda_ {1}}; \mathbf {A}, \mathbf {b}, \boldsymbol {\beta} _ {0}\right) - R \left(\widehat {\boldsymbol {\beta}} _ {k _ {2}, \infty} ^ {\lambda_ {2}}; \mathbf {A}, \mathbf {b}, \boldsymbol {\beta} _ {0}\right) \right| \xrightarrow {\text {a . s .}} \left| R \left(\widehat {\boldsymbol {\beta}} _ {k _ {1}, \infty} ^ {\lambda_ {1}}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}\right) - R \left(\widehat {\boldsymbol {\beta}} _ {k _ {2}, \infty} ^ {\lambda_ {2}}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}\right) \right|.
$$
It suffices to analyze  $R(\widehat{\beta}_{k,\infty}^{\lambda};A,0,\beta_{0})$
For simplicity, we treat  $\mathbf{A}$  as the normalized matrix  $n_{\mathbf{A}}^{-1/2} \mathbf{A}$  to avoid the notation of  $\mathrm{nrow}(\mathbf{A})$ . Note that  $\mathbf{y} = \mathbf{X} \boldsymbol{\beta}_0 + \mathbf{f}_{\mathrm{NL}} + \varepsilon$ , where  $\boldsymbol{\beta}_0$  is the best linear projection of  $\mathbf{y}$  on  $\mathbf{X}$ ,  $\mathbf{f}_{\mathrm{NL}}$  is the nonlinear residual and  $\varepsilon$  is the independent noise. Let  $\mathbf{A}_k^\lambda = \mathbb{E}_{I \sim \mathcal{I}_k}[(\mathbf{X}^\top \mathbf{L}_I \mathbf{X} / k + \lambda \mathbf{I}_p)^{-1} \mathbf{X}^\top \mathbf{L}_I / k]$  and  $\mathbf{B}_k^\lambda = \mathbf{I}_p - \mathbf{A}_k^\lambda \mathbf{X} = \mathbb{E}_{I \sim \mathcal{I}_k}[\lambda (\mathbf{X}^\top \mathbf{L}_I \mathbf{X} / k + \lambda \mathbf{I}_p)^{-1}]$ . We begin by decomposing the generalized risk for arbitrary  $(\lambda, k)$  into different terms:
$$
\begin{array}{l} R \left(\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}\right) = \left\| \mathbf {A} \left(\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda} - \boldsymbol {\beta} _ {0}\right) \right\| _ {2} ^ {2} \\ = \| \boldsymbol {A} (\boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {y} - \beta_ {0}) \| _ {2} ^ {2} \\ = \left\| \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \left(\boldsymbol {X} \beta_ {0} + \boldsymbol {f} _ {\mathrm {N L}}\right) - \boldsymbol {A} \beta_ {0} \right\| _ {2} ^ {2} \\ = \beta_ {0} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {X} - \boldsymbol {I} _ {p}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \left(\boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {X} - \boldsymbol {I} _ {p}\right) \beta_ {0} + \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} _ {\mathrm {N L}} \\ + 2 \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \left(\boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {X} - \boldsymbol {I} _ {p}\right) \beta_ {0} \\ = \boldsymbol {\beta} _ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {\beta} _ {0} + \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} (\boldsymbol {A} _ {k} ^ {\lambda}) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} _ {\mathrm {N L}} - 2 \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} (\boldsymbol {A} _ {k} ^ {\lambda}) ^ {\top} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {\beta} _ {0}. \\ \end{array}
$$
When  $\lambda > 0$ , from Lemma A.3, we know that the cross terms  $2f_{\mathrm{NL}}^{\top}(\pmb{A}_k^{\lambda})^{\top}\pmb{A}^{\top}\pmb{A}\pmb{B}_k^{\lambda}\beta_0$  vanishes as  $p$  tends to infinity. Further, by Lemma A.1 and Lemma A.2, it follows that when  $\pmb{A} \perp (\pmb{X}, \pmb{y})$ , as  $p / n \to \phi$  and  $p / k \to \psi$ ,
$$
\left| R \left(\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}\right) - R _ {p} (\lambda ; \phi , \psi) \right| \xrightarrow {\text {a . s .}} 0,
$$
where the function  $R_{p}$  is defined as
$$
R _ {p} (\lambda ; \phi , \psi) = \widetilde {c} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) + \| f _ {\mathrm {N L}} \| _ {L _ {2}} ^ {2} \widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}), \tag {11}
$$
and the nonnegative constants  $\widetilde{c}_p(-\lambda ;\phi ,\psi ,\mathbf{A}^\top \mathbf{A})$  and  $\widetilde{v}_p(-\lambda ;\phi ,\psi ,\mathbf{A}^\top \mathbf{A})$  are as defined in Lemma A.1.
When  $\lambda = 0$ , as in the proof of Theorem 3, we again show that  $P_{n,\lambda} - Q_{n,\lambda}$  is equivcontinuous over  $\Lambda = [0,\lambda_{\max}]$  for any  $\lambda_{\max} \in (0,\infty)$  fixed, where we define
$$
P _ {n, \lambda} = R (\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda}; \boldsymbol {A}, \boldsymbol {0}, \boldsymbol {\beta} _ {0}), \quad \text {a n d} \quad Q _ {n, \lambda} = R _ {p} (\lambda ; \phi , \psi).
$$
When  $\psi \neq 1$ , it can be verified that  $|P_{n,\lambda}|, |\partial P_{n,\lambda} / \partial \lambda|, |Q_{n,\lambda}|, |\partial Q_{n,\lambda} / \partial \lambda|$  are bounded almost surely. Thus, by the Moore-Osgood theorem, it follows that, with probability one,
$$
\lim _ {n \to \infty} | R (\widehat {\beta} _ {k, \infty} ^ {0}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}) - R _ {p} (0; \phi , \psi) | = \lim _ {\lambda \to 0 ^ {+}} \lim _ {n \to \infty} | R (\widehat {\beta} _ {k, \infty} ^ {\lambda}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}) - R _ {p} (\lambda ; \phi , \psi) | = 0.
$$
Note that  $\widetilde{c}_p(-\lambda ;\phi ,\psi ,A^\top A)$  and  $\widetilde{v}_p(-\lambda ;\phi ,\psi ,A^\top A)$  are functions of the fixed-point solution  $v_{p}(-\lambda ;\psi)$ . By Lemma C.1 we have that for  $\overline{\psi}\in [\phi ,\infty ]$  there exists a segment  $\mathcal{P}$  such that for all  $(\lambda_1,\psi_1),(\lambda_2,\psi_2)\in \mathcal{P}$ , it holds that  $v(-\lambda_1;\psi_1) = v(-\lambda_2;\psi_2)$  as  $p / k_1\to \psi_1$  and  $p / k_2\rightarrow \psi_2$ . This implies that
$$
R _ {p} (\lambda_ {1}; \phi , \psi_ {1}) = R _ {p} (\lambda_ {2}; \phi , \psi_ {2}).
$$
Thus, by triangle inequality, we have
$$
\left| R \left(\widehat {\boldsymbol {\beta}} _ {k _ {1}, \infty} ^ {\lambda_ {1}}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}\right) - R \left(\widehat {\boldsymbol {\beta}} _ {k _ {2}, \infty} ^ {\lambda_ {2}}; \mathbf {A}, \mathbf {0}, \boldsymbol {\beta} _ {0}\right) \right| \xrightarrow {\text {a . s .}} 0,
$$
When  $A = X$  and  $\pmb {b} = \pmb{f}_{\mathrm{NL}}$  , the generalized risk reduces to
$$
\begin{array}{l} R \left(\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda}; \mathbf {A}, \mathbf {b}, \boldsymbol {\beta} _ {0}\right) = \frac {1}{n} \| \mathbf {X} \left(\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda} - \boldsymbol {\beta} _ {0}\right) - \mathbf {f} _ {\mathrm {N L}} \| _ {2} ^ {2} \\ = \frac {1}{n} \| \boldsymbol {X} \widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda} - (\boldsymbol {X} \boldsymbol {\beta} _ {0} + \boldsymbol {f} _ {\mathrm {N L}}) \| _ {2} ^ {2} \\ = \frac {1}{n} \| \boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda} (\boldsymbol {X} \boldsymbol {\beta} _ {0} + \boldsymbol {f} _ {\mathrm {N L}}) - (\boldsymbol {X} \boldsymbol {\beta} _ {0} + \boldsymbol {f} _ {\mathrm {N L}}) \| _ {2} ^ {2} \\ = \beta_ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \widehat {\Sigma} \boldsymbol {B} _ {k} ^ {\lambda} \beta_ {0} + \frac {1}{n} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} (\boldsymbol {I} _ {p} - \boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda}) ^ {\top} (\boldsymbol {I} _ {p} - \boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda}) \boldsymbol {f} _ {\mathrm {N L}} \\ + \frac {2}{n} \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {I} _ {p} - \boldsymbol {X} \boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \boldsymbol {X} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {\beta} _ {0}. \\ \end{array}
$$
The proof then follows analogously as in Theorem 1 by involving Lemma A.1 and Lemma A.2 with  $\mathbf{A} = n^{-1/2}\mathbf{X}$ .
When  $A = X$  and  $b = 0$ , we have
$$
R \left(\widehat {\boldsymbol {\beta}} _ {k, \infty} ^ {\lambda}; \boldsymbol {A}, \boldsymbol {b}, \boldsymbol {\beta} _ {0}\right) = \boldsymbol {\beta} _ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \widehat {\boldsymbol {\Sigma}} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {\beta} _ {0} + \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \widehat {\boldsymbol {\Sigma}} \boldsymbol {A} _ {k} ^ {\lambda} \boldsymbol {f} _ {\mathrm {N L}} - 2 \boldsymbol {f} _ {\mathrm {N L}} ^ {\top} \left(\boldsymbol {A} _ {k} ^ {\lambda}\right) ^ {\top} \widehat {\boldsymbol {\Sigma}} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {\beta} _ {0}.
$$
Invoking Lemma A.2 for  $\pmb{f}_{\mathrm{NL}}^{\top}(\pmb{A}_k^{\lambda})^{\top}\widehat{\pmb{\Sigma}}\pmb{A}_k^{\lambda}\pmb{f}_{\mathrm{NL}}$  instead of  $\pmb{f}_{\mathrm{NL}}^{\top}(\pmb{I}_p - \pmb{X}\pmb{A}_k^{\lambda})^{\top}(\pmb{I}_p - \pmb{X}\pmb{A}_k^{\lambda})\pmb{f}_{\mathrm{NL}}$ , the proof follows analogously.
A.3 Technical lemmas
Lemma A.1 (Bias of generalized risk). Suppose the same assumptions in Theorem 1 hold with  $\lambda >0$ . Let  $B_{k}^{\lambda} = I_{p} - A_{k}^{\lambda}\mathbf{X} = \mathbb{E}_{I\sim \mathcal{I}_{k}}[\lambda (\mathbf{X}^{\top}\mathbf{L}_{I}\mathbf{X} / k + \lambda I_{p})^{-1}]$ . For any  $\mathbf{A}$  with  $\lim \sup \| \mathbf{A}\|_{\mathrm{op}}$  bounded almost surely, the following statements hold:
(1) If  $\mathbf{A} \perp (\mathbf{X}, \mathbf{y})$ , then
$$
\beta_ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \beta_ {0} - \widetilde {c} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) \xrightarrow {\text {a . s .}} 0.
$$
(2) If  $\mathbf{A} = n^{-1/2}\mathbf{X}$ , then
$$
\boldsymbol {\beta} _ {0} ^ {\top} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {\beta} _ {0} - \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} \widetilde {c} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {\Sigma}) \xrightarrow {\text {a . s .}} 0.
$$
Here the nonnegative constants  $v_{p}(-\lambda ;\psi), \widetilde{v}_{p}(-\lambda ;\phi ,\psi ,A^{\top}A)$  and  $\widetilde{c}_p(-\lambda ;\phi ,\psi ,A^\top A)$  are defined through the following equations:
$$
\frac {1}{v _ {p} (- \lambda ; \psi)} = \lambda + \psi \int \frac {r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r),
$$
$$
\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) = \frac {\phi \operatorname {t r} [ \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {\Sigma} (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} ] / p}{v _ {p} (- \lambda ; \psi) ^ {- 2} - \phi \int \frac {r ^ {2}}{(1 + v _ {p} (- \lambda ; \psi) r) ^ {2}} \mathrm {d} H _ {p} (r)},
$$
$$
\widetilde {c} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) = \beta_ {0} ^ {\top} (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1} (\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {A} ^ {\top} \boldsymbol {A}) \boldsymbol {\Sigma} + \boldsymbol {A} ^ {\top} \boldsymbol {A}) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 1} \beta_ {0}.
$$
Proof. Note that  $\beta_0$  is independent of
$$
\boldsymbol {B} _ {k} ^ {\lambda} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {B} _ {k} ^ {\lambda} = \lambda^ {2} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} [ \boldsymbol {M} _ {1} \boldsymbol {A} ^ {\top} \boldsymbol {A} \boldsymbol {M} _ {2} ],
$$