SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
R \left(\widehat {\beta} _ {\lfloor p / \psi_ {1} \rfloor , \infty} ^ {\lambda_ {1}}; \boldsymbol {A}, \boldsymbol {b}, \beta_ {0}\right) \simeq R \left(\widehat {\beta} _ {\lfloor p / \psi_ {2} \rfloor , \infty} ^ {\lambda_ {2}}; \boldsymbol {A}, \boldsymbol {b}, \beta_ {0}\right). \tag {6}
$$
In other words, Theorem 1 establishes the equivalences of subsampling and ridge regularization in terms of generalized risk for many statistical learning problems, encompassing coefficient estimation,
coefficient confidence interval, and test error estimation. All of these problems fall in the category when  $(A, b)$  are independent of  $X$ . However, there are other statistical learning problems not covered in Theorem 1, such as the in-sample prediction risk and the training error, which correspond to the case when  $A = X$ . Fortunately, the equivalences also apply to them, as summarized in Proposition 2.
Proposition 2 (Risk equivalences when  $\mathbf{A} = \mathbf{X}$ ). Under Assumptions 1-2, when  $\mathbf{A} = \mathbf{X}$ , the conclusion in Theorem 1 continues to hold for the cases of in-sample prediction risk ( $\mathbf{b} = \mathbf{0}$ ) and training error ( $\mathbf{b} = -\mathbf{f}_{\mathrm{NL}}$ ).
Both Theorem 1 and Proposition 2 provide specific second-order equivalences for the full-ensemble ridge estimators, in the sense that the quadratic functionals of the estimators associated with  $(A,b)$  are asymptotically the same. See Figure 1 for visual illustrations of both equivalences. These statements are presented separately because their proofs differ, with the proof for the dependent case being more intricate. We note that by combining the risk functionals in Table 2, it is possible to further extend the equivalences to other types of functionals not directly covered by statements above, using composition and continuous mapping. For example, it is not difficult to show that similar equivalences hold for the generalized cross-validation in the full ensembles [13]. This follows by combining the result for training error from Proposition 2 and for denominator concentration proved in Lemma 3.4 of [13].
Theorem 1 generalizes the existing equivalence results for the prediction risk [11-13] to include general risk functionals under milder assumptions. As summarized in Table 1, we allow for a general response model and feature covariance without assuming convergence of the spectral distributions  $H_{p}$  to a fixed distribution  $H$ . This flexibility is achieved by showing equivalences of the underlying resolvents in the estimators rather than deriving the limiting functionals as done in previous works. To extend results allowing for a general response, we generalize certain concentration results by leveraging tools from [14] to handle the dependency between the linear and non-linear components.
As alluded to earlier, it is worth noting that the generalized risk equivalences only hold in the full ensemble. However, by using the risk decomposition obtained from Lemma S.1.1 of [12] for finite ensembles, one can obtain the following relationship along the path in Theorem 3:  $R\big(\widehat{\beta}_{[p / \psi_1],M}^{\lambda_1};A,b,\beta_0\big) - R\big(\widehat{\beta}_{[p / \psi_2],M}^{\lambda_2};A,b,\beta_0\big)\simeq \Delta /M$ , for some  $\Delta$  (independent of  $M$ ) that is eventually almost surely bounded. In other words, the difference between any two estimators on the same path scales as  $1 / M$  when  $n$  is sufficiently large, as numerically verified in Figure F5.
4 Structural equivalences of ensemble estimators
The equivalences established in the previous section only hold for the full-ensemble estimators in a second-order sense. We can go a step further and ask if there exist any equivalences for the finite ensemble and if there are any equivalences at the estimator coordinate level between the estimators. Specifically, we aim to inspect whether each coordinate of the  $p$ -dimensional estimated coefficients asymptotically equals. This section establishes structural relationships between the estimators in a first-order sense that any bounded linear functionals of them are asymptotically the same.
Theorem 3 (Structural equivalences). Suppose Assumptions 1-2 hold. Let  $n, p \to \infty$  with  $p / n \to \phi \in (0, \infty)$ . For any  $\overline{\psi} \in [\phi, +\infty]$ , let  $\bar{\lambda}$  be as in (4). Then, for any  $M \in \mathbb{N} \cup \{\infty\}$  and any pair of  $(\lambda_1, \psi_1)$  and  $(\lambda_2, \psi_2)$  on the path (5), the  $M$ -ensemble estimators are asymptotically equivalent:
$$
\widehat {\boldsymbol {\beta}} _ {\lfloor p / \psi_ {1} \rfloor , M} ^ {\lambda_ {1}} \simeq \widehat {\boldsymbol {\beta}} _ {\lfloor p / \psi_ {2} \rfloor , M} ^ {\lambda_ {2}}. \tag {7}
$$
Put another way, Theorem 3 implies that for any fixed ensemble size  $M$ , the  $M$ -ensemble ridgeless estimator at the limiting aspect ratio  $\overline{\psi}$  is equivalent to ridge regression at the regularization level  $\bar{\lambda}$ . This equivalence is visually illustrated in Figure 2, where the data is simulated from an anisotropic and nonlinear model (see Appendix F.2 for more details). Furthermore, the contour path  $\mathcal{P}(\bar{\lambda}; \phi, \overline{\psi})$  connecting the endpoints  $(0, \overline{\psi})$  and  $(\bar{\lambda}, \phi)$  is a straight line, although it may have varying slopes. Along any path, all  $M$ -ensemble estimators at the limiting aspect ratio  $\psi$  and ridge penalty  $\lambda$  are equivalent for all  $(\lambda, \psi) \in \mathcal{P}(\bar{\lambda}; \phi, \overline{\psi})$ . The equivalences here are for any arbitrary linear combinations of the estimators. In particular, this implies that the predicted values (or even any continuous function applied to them due to the continuous mapping theorem) of any test point will eventually be the same, almost surely, with respect to the training data. Finally, we remark that in the statement of Theorem 3, the equivalence is defined on extended reals in order to incorporate the ridgeless case when  $\overline{\psi} = 1$ .
Figure 2: Heat map of the values of various linear functionals of ensemble ridge estimators  $\pmb{a}^{\top}\widehat{\beta}_{k,M}^{\lambda}$ , for varying ridge penalties  $\lambda$ , subsample aspect ratios  $\psi = p / k$ , and ensemble size  $M$  on the log-log scale. The data model is described in Appendix F.2 with  $p = 1000$ ,  $n = 10000$ , and  $\phi = p / n = 0.1$ . (a) The values of the uniformly weighted projection for varying ensemble sizes ( $M = 1, 5$ , and 50). (b) The values of  $M = 100$  for varying projection vectors (uniformly weighted, random Gaussian, and random student's  $t$ ). The black dashed line is estimated based on Proposition 4 with  $\bar{\psi} = 2$ . Observe that the values along the data-dependent paths are indeed very similar, in line with Proposition 4.
The paths (5) in Theorem 3 are defined via the spectral distribution  $H_{p}$ , which requires knowledge of  $\Sigma$ . This is often difficult to obtain in practice from the observed data. Fortunately, we can provide an alternative characterization for the path (5) solely through the data, as summarized in Proposition 4.
Proposition 4 (Data-dependent equivalence path characterization). Suppose Assumptions 1-2 hold. Define  $\phi_n = p / n$ . Let  $k \leq n$  be the subsample size and denote by  $\bar{\psi}_n = p / k$ . For any  $M \in \mathbb{N} \cup \{\infty\}$ , let  $\bar{\lambda}_n$  be the value that satisfies the following equation in ensemble ridgeless and ridge gram matrices:
$$
\frac {1}{M} \sum_ {\ell = 1} ^ {M} \frac {1}{k} \operatorname {t r} \left[ \left(\frac {1}{k} \boldsymbol {L} _ {I _ {\ell}} \boldsymbol {X} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I _ {\ell}}\right) ^ {+} \right] = \frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top} + \bar {\lambda} _ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \right]. \tag {8}
$$
Define the data-dependent path  $\mathcal{P}_n = \mathcal{P}(\bar{\lambda}_n; \phi_n, \bar{\psi}_n)$ . Then, the conclusion in Theorem 3 continues to hold if we replace the (population) path  $\mathcal{P}$  by the (data-dependent) path  $\mathcal{P}_n$ .
The term "data-dependent path" signifies that we can estimate the level of implicit regularization induced by subsampling solely based on the observed data by solving (8). We remark that (8) always has at least one solution for a given triplet of  $(n, k, p)$ . This is because the right-hand side of (8) is monotonically decreasing in  $\lambda$ , and the left-hand side always lies within the range of the right-hand side. The powerful implication of the characterization in Proposition 4 is that it enables practical computation of the path using real-world data. In Section 6, we will demonstrate this approach on various real-world datasets, allowing us to predict an equivalent amount of explicit regularization matching the implicit regularization due to subsampling.
One natural extension of the results is to consider the generalized ridge as a base estimator (1):
$$
\widehat {\boldsymbol {\beta}} _ {k} ^ {\lambda , \boldsymbol {G}} (\mathcal {D} _ {I}) = \underset {\boldsymbol {\beta} \in \mathbb {R} ^ {p}} {\operatorname {a r g m i n}} \frac {1}{k} \sum_ {i \in I} \left(y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta}\right) ^ {2} + \lambda \| \boldsymbol {G} ^ {\frac {1}{2}} \boldsymbol {\beta} \| _ {2} ^ {2}, \tag {9}
$$
where  $G \in \mathbb{R}^{p \times p}$  is a positive definite matrix. The equivalences still hold, albeit on different paths:
Corollary 5 (Equivalences for generalized ridge regression). Suppose Assumptions 1-2 hold. Let  $\pmb{G} \in \mathbb{R}^{p \times p}$  be a deterministic and symmetric matrix with eigenvalues uniformly bounded between constants  $g_{\mathrm{min}} > 0$  and  $g_{\mathrm{max}} < \infty$ . Let  $n, p \to \infty$  such that  $p / n \to \phi \in (0, \infty)$ . For any  $\bar{\psi} \in [\phi, +\infty]$ , let  $\bar{\lambda}$  be as defined in (4) with  $H_p$  replaced  $\tilde{H}_p$ , the empirical spectral distribution of  $\pmb{G}^{-1/2} \pmb{\Sigma} \pmb{G}^{-1/2}$ . Then, the conclusions in Theorems 1 and 3 continue to hold for the generalized ridge ensemble predictors.
Another extension of our results is to incorporate subquadratic risk functionals in Theorem 1 to derive equivalences of the cumulative distribution functions of the predictive error distributions associated
with the estimators along the equivalence paths. One can use such equivalences for downstream inference questions. Finally, while our statements are asymptotic in nature, we expect a similar analysis as done in [50] can yield finite-sample statements. We leave these extensions for the future.
5 Implications of equivalences
The generalized risk equivalences establish a connection between the risks of different ridge ensembles, including the ridge predictors on full data and the full-ensemble ridgeless predictors. These connections enable us to transfer properties from one estimator to the other. For example, by examining the impact of subsampling on the estimator's performance, we can better understand how to optimize the ridge penalty. We will next provide an example of this. Many common methods, such as ridgeless or lassoless predictors, have been recently shown to exhibit non-monotonic behavior in the sample size or the limiting aspect ratio. An open problem raised by Nakkiran et al. [1, Conjecture 1] asks whether the prediction risk of the ridge regression with optimal ridge penalty  $\lambda^{*}$  is monotonically increasing in the data aspect ratio  $\phi = p / n$ . The non-monotonicity risk behavior implies that more data can hurt performance, and it's important to investigate whether optimal ridge regression also suffers from this issue. Our equivalence results provide a surprising (even to us) indirect way to answer this question affirmatively under proportional asymptotics.
To analyze the monotonicity of the risk, we will need to impose two additional regularity assumptions to guarantee the convergence of the prediction risk. Let  $\pmb{\Sigma} = \sum_{j=1}^{p} r_j \pmb{w}_j \pmb{w}_j^\top$  denote the eigenvalue decomposition, where  $(r_j, \pmb{w}_j)$ 's are pairs of associated eigenvalue and normalized eigenvector. The following assumptions ensure that the hardness of the underlying regression problems across different limiting data aspect ratios is comparable.
Assumption 3 (Spectral convergence). We assume there exists a deterministic distribution  $H$  such that the empirical spectral distribution of  $\Sigma$ ,  $H_{p}(r) = p^{-1}\sum_{i=1}^{p}\mathbb{1}_{\{r_{i} \leq r\}}$ , weakly converges to  $H$ , almost surely (with respect to  $\boldsymbol{x}$ ).
Assumption 4 (Limiting signal and noise energies). We assume there exists a deterministic distribution  $G$  such that the empirical distribution of  $\beta_0$ 's (squared) projection onto  $\Sigma$ 's eigenspace,  $G_{p}(r) = \|\beta_{0}\|_{2}^{-2}\sum_{i=1}^{p}(\beta_{0}^{\top}\boldsymbol{w}_{i})^{2}\mathbb{1}_{\{r_{j}\leq r\}}$ , weakly converges to  $G$ . As  $n,p\to \infty$  and  $p / n\to \phi \in (0,\infty)$ , the limiting linearized energy  $\rho^{2} = \lim \|\beta_{0}\|_{2}^{2}$  and the limiting nonlinearized energy  $\sigma^{2} = \lim \|f_{\mathrm{NL}}\|_{L_{2}}^{2}$  are finite.
Assumption 3 is commonly used in random matrix theory and overparameterized learning [10, 12, 13] and ensures that the limiting spectral distribution of the sample covariance matrix converges to a fixed distribution. Under these assumptions, we show in Appendix C that there exists a deterministic function  $\mathcal{R}(\lambda; \phi, \psi)$ , such that  $R(\widehat{\beta}_{k,\infty}^{\lambda}; \boldsymbol{A}, \boldsymbol{b}, \boldsymbol{\beta}_0) \simeq \mathcal{R}(\lambda; \phi, \psi)$ . Notably, the deterministic profile on the right-hand side is a monotonically increasing and a continuous function in the first parameter  $\phi$  when limiting values of  $\| \beta_0 \|_2^2$  and  $\| f_{\mathrm{NL}} \|_{L^2}^2$  are fixed (i.e., when Assumption 4 holds). Moreover, the deterministic risk equivalent of the optimal ridge predictor matches that of the optimal full-ensemble ridgeless predictor; in other words,  $\min_{\lambda \geq 0} \mathcal{R}(\lambda; \phi, \phi) = \min_{\psi \geq \phi} \mathcal{R}(0; \phi, \psi)$ . The same monotonicity property of the two optimized functions leads to the following result.
Theorem 6 (Monotonicity of prediction risk with optimal ridge regularization). Suppose the conditions of Theorem 1 and Assumptions 3-4 hold. Let  $k, n, p \to \infty$  such that  $p / n \to \phi \in (0, \infty)$  and  $p / k \to \psi \in [\phi, \infty]$ . Then, for  $A = \Sigma^{1/2}$  and  $b = 0$ , the optimal risk of the ridgeless ensemble,  $\min_{\psi \geq \phi} \mathcal{R}(0; \phi, \psi)$ , is monotonically increasing in  $\phi$ . Consequently, the optimal risk of the ridge predictor,  $\min_{\lambda \geq 0} \mathcal{R}(\lambda; \phi, \phi)$ , is also monotonically increasing in  $\phi$ .
Under Assumptions 3 and 4, the linearized signal-to-noise ratio (SNR) is maintained at the same level across varying distributions as  $\phi$  changes. The key message of Theorem 6 is then that, for a sequence of problems with the same SNR (indicating the same level of regression hardness), the asymptotic prediction risk of optimized ridge risk gets monotonically worse as the aspect ratio of the problem increases. This is intuitive because a smaller  $\phi$  corresponds to more samples than features. In this sense, Theorem 6 certifies that optimal ridge uses the available data effectively, avoiding sample-wise non-monotonicity. Moreover, as the null risk is finite, this also shows that the optimal ridge estimator mitigates the "double or multiple descents" behaviors in the generalization error [1, 5, 51].
Attempting to prove Theorem 6 directly is challenging due to the lack of a closed-form expression for the optimal risk of ridge regression in general. Moreover, the risk profile of ridge regression,
Figure 3: Comparison of various linear functionals (Theorem 3) and generalized risk functionals (Theorem 1) on different paths evaluated on CIFAR-10. The aspect ratios  $\phi = p / n$  and  $\overline{\psi} = 4\phi$  are estimated from the dataset and fixed. For different values of  $\lambda_{\overline{\psi}}$  (different colors), we estimate  $\overline{\lambda}$  from Proposition 4, which gives a path between  $(\lambda_{\overline{\psi}},\overline{\psi})$  and  $(\overline{\lambda},\phi)$ . For each path, we uniformly sample 5 points and compute the functionals of the ridge ensemble using  $M = 100$ . The values of different paths are then normalized by subtracting the mean value in the first path and dividing by the mean difference of values on the last and first paths.
$\mathcal{R}(\lambda ;\phi ,\phi)$ , does not exhibit any particular structure as a function of  $\lambda$ . On the other hand, the risk profile of the full-ensemble ridgeless,  $\mathcal{R}(0;\phi ,\psi)$ , has a very nice structure in terms of  $\psi$ . This, coupled with our equivalence result, allows one to prove quite non-trivial behavior of optimal ridge. See Figure F6 for illustrations.
Motivated by the recent subsampling and ridge equivalences for the prediction risk [11-13], this paper establishes generalized risk equivalences (Section 3) and structural equivalences (Section 4) within the family of ensemble ridge estimators. Our results precisely link the implicit regularization of subsampling to explicit ridge penalization via a path  $\mathcal{P}$  defined in (5), which connects two endpoints  $(0, \bar{\psi})$  and  $(\bar{\lambda}, \phi)$  regardless of the risk functionals. Furthermore, we provide a data-dependent method (Proposition 4) to estimate this path. Our results do not assume any specific relationship between the response variable  $y$  and the feature vector  $x$ . We next explore some extensions of these equivalences.
Real data. While our theoretical results assume RMT features (Assumption 2), we anticipate that the equivalences will very likely hold under more general features [50]. To verify this, we examine the equivalences in Proposition 4 with  $M = 100$  on real-world datasets. Figure 3 depicts both the linear functionals in Theorem 3 and the generalized quadratic functionals in Theorem 1 computed along four paths, shown in different colors. We observe that the variation within each path is small, while different paths have different functional values. This suggests that the theoretical finding in the previous sections also hold quite well on real-world datasets. We investigate this further by varying  $\bar{\psi}$  on the three image datasets, CIFAR-10, MNIST, and USPS. Figure F7 shows similar values and trends at the two points. These experiments demonstrate that the amount of explicit regularization induced by subsampling can indeed be accurately predicted based on the observed data using Proposition 4.
Random features. Closely related to two-layer neural networks [52], we can consider the random feature model,  $f(\pmb{x}; \pmb{\beta}, \pmb{F}) = \pmb{\beta}^{\top} \varphi(\pmb{F}\pmb{x})$ , where  $\pmb{F} \in \mathbb{R}^{d \times p}$  is some randomly initialized weight matrix, and  $\varphi: \mathbb{R} \to \mathbb{R}$  is a nonlinear activation function applied element-wise to  $\pmb{F}\pmb{x}$ . As from "universality/invariance", certain random feature models are asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix [53], we expect the theoretical results in Theorems 3-1 to likely hold, although the relationship (4) will now depend on the non-linear activation functions. Empirically, we indeed observe similar equivalence phenomena of the prediction risks with random features ridge regression using sigmoid, ReLU, and tanh activation functions, as shown in Figure 4. Analogous to Proposition 4, we conjecture a similar data-driven relationship between  $(\overline{\lambda}_n, \phi_n)$  and  $(0, \overline{\psi}_n)$  for random features  $\widetilde{\pmb{X}} = \varphi(\pmb{X}\pmb{F}^{\top})$ :
Conjecture 7 (Data-dependent equivalences for random features (informal)). Suppose Assumptions 1-2 hold. Define  $\phi_n = p / n$ . Let  $k \leq n$  be the subsample size and denote by  $\overline{\psi}_n = p / k$ . Suppose  $\varphi$  satisfies certain regularity conditions. For any  $M \in \mathbb{N} \cup \{\infty\}$ , let  $\bar{\lambda}_n$  be the value that satisfies
$$
\frac {1}{M} \sum_ {\ell = 1} ^ {M} \frac {1}{k} \operatorname {t r} \left[ \left(\frac {1}{k} \varphi (\boldsymbol {L} _ {I _ {\ell}} \boldsymbol {X} \boldsymbol {F} ^ {\top}) \varphi (\boldsymbol {L} _ {I _ {\ell}} \boldsymbol {X} \boldsymbol {F} ^ {\top}) ^ {\top}\right) ^ {+} \right] = \frac {1}{n} \operatorname {t r} \left[ \left(\frac {1}{n} \varphi (\boldsymbol {X} \boldsymbol {F} ^ {\top}) \varphi (\boldsymbol {X} \boldsymbol {F} ^ {\top}) ^ {\top} + \bar {\lambda} _ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \right].
$$
Figure 4: Heat map of prediction risks of full-ensemble ridge estimators (approximated with  $M = 100$ ) using random features, for varying ridge penalties  $\lambda$ , subsample aspect ratios  $\psi = p / d$  on the log-log scale. We consider random features  $\varphi(\boldsymbol{F}\boldsymbol{x}_i) \in \mathbb{R}^d$ , where  $\varphi$  is an activation function (sigmoid, ReLU, or tanh). The data model is described in Appendix F.6 with  $p = 250$ ,  $d = 500$ ,  $n = 5000$ , and  $\phi = d / n = 0.1$ . As in Theorem 1, we see clear risk equivalence paths across activations.
Define the data-dependent path  $\mathcal{P}_n = \mathcal{P}(\overline{\lambda}_n;\phi_n,\overline{\psi}_n)$ . Then, the conclusions in Theorem 1, Proposition 2, and Theorem 3 continue to hold on  $(\varphi (\pmb {X}\pmb{F}^{\top}),\pmb {y})$  with  $\mathcal{P}$  replaced by  $\mathcal{P}_n$ .
Kernel features. We can also consider kernel ridge regression. For a given feature map  $\Phi : \mathbb{R}^p \to \mathbb{R}^d$ , the kernel ridge estimator (in the primal form) is defined as:
$$
\widehat {\beta} _ {k} ^ {\lambda} (\mathcal {D} _ {I}) = \underset {\boldsymbol {\beta} \in \mathbb {R} ^ {p}} {\operatorname {a r g m i n}} \sum_ {i \in I} (k ^ {- 1 / 2} y _ {i} - k ^ {- 1 / 2} \Phi (\boldsymbol {x} _ {i}) ^ {\top} \boldsymbol {\beta}) ^ {2} + \lambda \| \boldsymbol {\beta} \| _ {2} ^ {2} \sum_ {i \in I} (y _ {i} - \Phi (\boldsymbol {x} _ {i}) ^ {\top} \boldsymbol {\beta}) ^ {2} + \frac {k}{p} \lambda \| \boldsymbol {\beta} \| _ {2} ^ {2}.
$$
Leveraging the kernel trick, the preceding optimization problem translates to solving the following problem (in the dual domain):  $\widehat{\pmb{\alpha}}_k^\lambda (\mathcal{D}_I) = \mathrm{argmin}_{\pmb {\alpha}\in \mathbb{R}^k}\pmb{\alpha}^\top (\pmb {K}_I + k\lambda \pmb {I}_k)\pmb {\alpha} + \pmb{\alpha}^\top \pmb {y}_I$ , where  $\pmb {K}_I = \pmb {\Phi}_I\pmb {\Phi}_I^\top \in \mathbb{R}^{k\times k}$  is the kernel matrix and  $\Phi_I = (\Phi (\pmb {x}_i))_{i\in I}\in \mathbb{R}^{n\times d}$  is the feature matrix. The correspondence between the dual and primal solutions is simply given by:  $\widehat{\beta}_k^\lambda (\mathcal{D}_I) = \Phi_I^\top \widehat{\alpha}_k^\lambda (\mathcal{D}_I)$ .
Figure F8 illustrate results for kernel ridge regression using the same data-generating process as in the previous subsection. The figure shows the prediction risk of kernel ridge ensembles for polynomial, Gaussian, and Laplace kernels exhibit similar equivalence patterns, leading us to formulate an analogous conjecture for kernel ridge regression (which when  $\Phi(x) = x$  gives back Proposition 4 with appropriate rescaling of features):
Conjecture 8 (Data-dependent equivalences for kernel features (informal)). Suppose Assumptions 1-2 hold. Define  $\phi_n = p / n$ . Suppose the kernel  $K$  satisfies certain regularity conditions. Let  $k \leq n$  be the subsample size and denote by  $\overline{\psi}_n = p / k$ . For any  $M \in \mathbb{N} \cup \{\infty\}$ , let  $\bar{\lambda}_n$  be a solution to