Implicit Regularization Paths of Weighted Neural Representations

Authors: jin-hong  du
pratik  patil
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:04:14

Abstract: We study the implicit regularization effects induced by (observation) weighting of pretrained features.
For weight and feature matrices of bounded operator norms that are infinitesimally free with respect to (normalized) trace functionals, we derive equivalence paths connecting different weighting matrices and ridge regularization levels.
Specifically, we show that ridge estimators trained on weighted features along the same path are asymptotically equivalent when evaluated against test vectors of bounded norms.
These paths can be interpreted as matching the effective degrees of freedom of ridge estimators fitted with weighted features.
For the special case of subsampling without replacement, our results apply to independently sampled random features and kernel features and confirm recent conjectures (Conjectures 7 and 8) of the authors on the existence of such paths in Patil and Du (2023).
We also present an additive risk decomposition for ensembles of weighted estimators and show that the risks are equivalent along the paths when the ensemble size goes to infinity.
As a practical consequence of the path equivalences, we develop an efficient cross-validation method for tuning and apply it to subsampled pretrained representations across several models (e.g., ResNet-50) and datasets (e.g., CIFAR-100).

Main paper: 
 introduction: 
 In recent years, neural networks have become state-of-the-art models for tasks in computer vision and natural language processing by learning rich representations from large datasets. Pretrained neural networks, such as ResNet, which are trained on massive datasets like ImageNet, serve as valuable resources for new, smaller datasets [32] . These pretrained models reduce computational burden and generalize well in tasks such as image classification and object detection due to their rich feature space [32, 69] . Furthermore, pretrained features or neural embeddings, such as the neural tangent kernel, extracted from these models, serve as valuable representations of diverse data [33, 66] .However, despite their usefulness, fitting models based on pretrained features on large datasets can be challenging due to computational and memory constraints. When dealing with high-dimensional pretrained features and large sample sizes, direct application of even simple linear regression may be computationally infeasible or memory-prohibitive [23, 44] . To address this issue, subsampling has emerged as a practical solution that reduces the dataset size, thereby alleviating the computational and memory burden. Subsampling involves creating smaller datasets by randomly selecting a subset of the original data points. Beyond these computational and memory advantages, subagging can also greatly improve predictive performance in overparameterized regimes, especially near model interpolation thresholds [53] . Moreover, through distributed learning, models fitted on multiple subsampled datasets can be aggregated as an ensemble to provide more stable predictions [20, 21, 51] .There has been growing interest in understanding the effects of subsampling (without replacement) [16, 25, 37, 50, 51] . These works relate subsampling to explicit ridge regularization, assuming either 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Gaussian features ϕ ∼ N (0 p , Σ) or linearly decomposable features (referred to as linear features in this paper) ϕ = Σ 1/2 z, where Σ ∈ R p×p is the covariance matrix and z ∈ R p contains i.i.d. entries with zero means and bounded 4 + δ moments for some δ > 0. Specifically, [50] establish a connection between implicit regularization induced by subsampling and explicit ridge regularization through a path defined by the tuple (k/n, λ), where k and n are the subsample size and the full sample size, respectively, and λ is the ridge regularization level. Along this path, any subsample estimator with the corresponding ridge regularization exhibits the same first-order (or estimator equivalence) and second-order (or risk equivalence) asymptotic limits. Moreover, the endpoints of all such paths along the two axes of k = n (no subsampling) and λ = 0 (no regularization) span the same range. Although these results have been demonstrated for linear features, [50] also numerically observe similar equivalence behavior in more realistic contexts and propose conjectures for random features and kernel features based on heuristic "universality" justifications. However, extending these results to encompass more general feature structures and other sampling schemes remains an open question.Towards answering this question, in this paper, we view subsampling as a weighted regression problem [67] . This perspective allows us to study the equivalence in its most general form, considering arbitrary feature structures and weight structures. The general weight matrix approach used in this study encompasses various applications, including subsampling, bootstrapping, variance-adaptive weighting, survey, and importance weighting, among others. By interpreting subsampling as a weighted regression problem, we leverage recent tools from free probability theory, which have been developed to analyze feature sketching [39, 42, 54] . Building on these theoretical tools, we establish implicit regularization paths for general weighting and feature structures. We summarize our main results below and provide an overview of our results in the context of recent related work in Table 1 .We summarize our main results and provide an outline for the paper below.• Paths of weighted representations. In Section 3, we demonstrate that general weighted models exhibit first-order equivalence along a path (Theorem 1) when the weight matrices are asymptotically independent of the data matrices. This path of equivalence can be computed directly from the data using the formula provided in Equation (2) . Furthermore, we provide a novel interpretation of this path in terms of matching effective degrees of freedom of models along the path for general feature structures when the weights correspond to those arising from subsampling (Theorem 2).• Paths of subsampled representations. We further specialize our general result in Theorem 2 for the weights induced by subsampling without replacement to structured features in Section 3.2. These include results for linear random features, nonlinear random features, and kernel features, as shown in Propositions 3-5, respectively. The latter two results also resolve Conjectures 7 and 8 raised by [50] regarding subsampling regularization paths for random and kernel features, respectively.• Risk equivalences and tuning. In Section 4, we demonstrate that an ensemble of weighted models has general quadratic risk equivalence on the path, with an error term that decreases inversely as 1/M as the number of ensemble size M increases (Theorem 6). The risk equivalence holds for both in-distribution and out-of-distribution settings. For subsampling general features, we derive an upper bound for the optimal subsample size (Proposition 7) and propose a cross-validation method to tune the subsample and ensemble sizes (Algorithm 1), validated on real datasets in Section 4.3.This level of generality is achievable because we do not analyze the risk of either the full model or the weighted models in isolation. Instead, we relate these two sets of models, allowing us to maintain weak assumptions about the features. The key assumption underlying our results is the asymptotic freeness of weight matrices with respect to the data matrices. While directly testing this assumption is generally challenging, we verify its validity through its consequences on real datasets in Section 4.3.We provide a brief account of other related work below to place our work in a better context.Linear features. Despite being overparameterized, neural networks generalize well in practice [70, 71] .Recent work has used high-dimensional "linearized" networks to investigate the various phenomena that arise in deep learning, such as double descent [12, 46, 48] , benign overfitting [10, 35, 45] , and scaling laws [7, 19, 66] . This literature analyzes linear regression using statistical physics [14, 60] and random matrix theory [22, 30] . Risk approximations hold under random matrix theory assumptions [6, 30, 66] in theory and apply empirically on a variety of natural data distributions [43, 60, 66] .Random and kernel features. Random feature regression, initially introduced in [56] as a way to scale kernel methods, has recently been used for theoretical analysis of neural networks and trends of double descent in deep networks [1, 46] . The generalization of kernel ridge regression has been studied in [11, 40, 57] . The risks of kernel ridge regression are also analyzed in [9, 19, 29] . The neural representations we study are motivated by the neural tangent kernel (NTK) and related theoretical work on ultra-wide neural networks and their relationships to NTKs [34, 68] .Resampling analysis. Resampling and weighted models are popular in distributed learning to provide more stable predictions and handle large datasets [20, 21, 51] . Historically, for ridge ensembles, [36, 61] derived risk asymptotics under Gaussian features. Recently, there has been growing interest in analyzing the effect of subsampling in high-dimensional settings. [37] considered least squares ensembles obtained by subsampling, where the final subsampled dataset has more observations than the number of features. For linear models in the underparameterized regime, [59] also provide certain equivalences between subsampling and iterative least squares approaches. The asymptotic risk characterization for general data models has been derived by [51] . [25, 50] extended the scope of these results by characterizing risk equivalences for both optimal and suboptimal risks and for arbitrary feature covariance and signal structures. Very recently, different resampling strategies for high-dimensional supervised regression tasks have been analyzed by [17] under isotropic Gaussian features. Cross-validation methods for tuning the ensemble of ridge estimators and other penalized estimators are discussed in [13, 25, 26] . Our work adds to this literature by considering ensembles of models with general weighting and feature structures. 
preliminaries: 
 In this section, we formally define our weighted estimator and state the main assumption on the weight matrix. Let f nn : R d → R p be a pretrained model. Let {(x i , y i ) : i = 1, . . . , n} in R d × R be the given dataset. Applying f nn to the raw dataset, we obtain the pretrained features ϕ i = f nn (x i ) for i = 1, . . . , n as the resulting neural representations or neural embeddings. In matrix notation, we denote the pretrained feature matrix by Φ = [ϕ 1 , . . . , ϕ n ] ⊤ ∈ R n×p . Let W ∈ R n×n be a general weight matrix used for weighting the observations. The weight matrix W is allowed to be asymmetric, in general.We consider fitting ridge regression on the weighted dataset (W Φ, W y). Given a ridge penalty λ, the ridge estimator fitted on the weighted dataset is given by:β W ,λ := argmin β∈R p ∥W y -W Φβ∥ 2 2 n + λ∥β∥ 2 2 = (Φ ⊤ W ⊤ W Φ + nλI p ) † Φ ⊤ W ⊤ W y. (1)In the definition above, we allow for λ = 0, in which case the corresponding ridgeless estimator is defined as the limit λ → 0 + . For λ < 0, we use the Moore-Penrose pseudoinverse. An important special case is where W is a diagonal matrix, in which case the above estimator reduces to weighted ridge regression. This type of weight matrix encompasses various applications, such as resampling, bootstrapping, and variance weighting. Our main application in this paper will be subsampling.For our theoretical results, we assume that the weight matrix W preserves some spectral structure of the feature matrix Φ. This assumption is captured by the condition of asymptotic freeness between W ⊤ W and the feature Gram matrix ΦΦ ⊤ . Asymptotic freeness is a concept from free probability theory [64] .Assumption A (Weight structure). Let W ⊤ W and ΦΦ ⊤ /n converge almost surely to bounded operators that are infinitesimally free with respect to (tr[•], tr[C(•)]) for any C independent of W with ∥C∥ tr uniformly bounded. Additionally, let W ⊤ W have a limiting S-transform that is analytic on the lower half of the complex plane.At a high level, Assumption A captures the notion of independence but is adapted for non-commutative random variables of matrices. We provide background on free probability theory and asymptotic freeness in Appendix A.3. Here, we briefly list a series of invertible transformations from free probability to help define the S-transform [47] . The Cauchy transform is given byG A (z) = tr[(zI -A) -1 ].The moment generating series is given byM A (z) = z -1 G A (z -1 ) -1. The S-transform is given by S A (w) = (1 + w -1 )M ⟨-1⟩A (w). These are the Cauchy transform (negative of the Stieltjes transform), moment generating series, and S-transform of A, respectively. Here, M ⟨-1⟩ A denotes the inverse under the composition of M A . The notation tr[A] denotes the average trace tr[A]/p of A ∈ R p×p .The freeness of a pair of matrices A and B means that the eigenvectors of one are completely unaligned or incoherent with those of the other. For example, if A = U RU ⊤ for a uniformly random unitary matrix U drawn independently of the positive semidefinite B and R, then A and B are almost surely asymptotically infinitesimally free [15] . Other well-known examples include Wigner matrices, which are asymptotically free with respect to deterministic matrices [4, Theorem 5.4.5] . Gaussian matrices, where the Gram matrix G = ΦΦ ⊤ /n = U (V V ⊤ /n)U ⊤ and any deterministic S, are almost surely asymptotically free [47, Chapter 4, Theorem 9] . Although not proven in full generality, it is expected that diagonal matrices are asymptotically free from data Gram matrices constructed using i.i.d. data. In Section 3.2, we will provide additional examples of feature matrices, such as random and kernel features from machine learning, for which our results apply.Our results involve the notion of degrees of freedom from statistical optimism theory [27, 28] . Degrees of freedom in statistics count the number of dimensions in which a statistical model may vary, which is simply the number of variables for ordinary linear regression. To account for regularization, this notion has been extended to effective degrees of freedom (Chapter 3 of [31] ). Under some regularity conditions, from Stein's relation [63] , the degrees of freedom of a predictor f are measured by the trace of the operators y → (∂/∂y) f (Φ). For the ridge estimator β I,µ fitted on (Φ, y) with penalty µ, the degrees of freedom is consequently the trace of its prediction operator y → Φ(Φ ⊤ Φ + µI p ) † Φ ⊤ y, which is also referred to as the ridge smoother matrix. That is, df( β I,µ ) = tr[Φ ⊤ Φ(Φ ⊤ Φ + µI p ) † ]. We denote the normalized degrees of freedom by df = df/n. Note that df( β I,µ ) ≤ min{n, p}/n ≤ 1.Finally, we express our asymptotic results using the asymptotic equivalence relation. Consider sequences {A n } n≥1 and {B n } n≥1 of (random or deterministic) matrices (which includes vectors and scalars). We say that A n and B n are equivalent and writeA n ≃ B n if lim p→∞ | tr[C n (A n - B n )]| = 0 almostsurely for any sequence C n of matrices with bounded trace norm such that lim sup ∥C n ∥ tr < ∞ as n → ∞. Our forthcoming results apply to a sequence of problems indexed by n. For notational simplicity, we omit the explicit dependence on n in our statements.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The assumptions are discussed and explained after their first mention (either in Section 2 or right after the theoretical result that uses them). The main limitations of the paper are discussed in the last section (Section 5).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
implicit regularization paths: 
 We begin by characterizing the implicit regularization induced by weighted pretrained features. We will show that the degrees of freedom of the unweighted estimator β I,µ on the full data (Φ, y) with regularization parameter µ are equal to the degrees of freedom of the weighted estimator β W ,λ for some regularization parameter λ. For estimator equivalence, our data-dependent set of weighted ridge estimators (W , λ) that connect to the unweighted ridge estimator (I, µ) is defined in terms of "matching" effective degrees of freedom of component estimators in the set.To state the upcoming result, denote the Gram matrix of the weighted data as G W = W ΦΦ ⊤ W ⊤ /n and the Gram matrix of the unweighted data as G I = ΦΦ ⊤ /n. Furthermore, let λ + min (A) denote the minimum positive eigenvalue of a symmetric matrix A. Theorem 1 (Implicit regularization of weighted representations). For G I ∈ R n×n , suppose that the weight matrix W ∈ R n×n satisfies Assumption A and lim sup ∥y∥ 2 2 /n < ∞ as n → ∞. For any µ > -lim inf n→∞ λ + min (G I ), let λ > -λ + min (G W ) be given by the following equation:EQUATIONwhere S W ⊤ W is the S-transform of the operator W ⊤ W . Then, as n → ∞, it holds that:EQUATIONIn other words, to achieve a target regularization of µ on the unweighted data, Theorem 1 provides a method to compute the regularization penalty λ with given weights W from the available data using (2) . The weighted estimator then has asymptotically the same degrees of freedom as the unweighted estimator. This means that the level of effective regularization of the two estimators is the same. Moreover, the estimators themselves are structurally equivalent; that is, c ⊤ ( β W ,λ -β I,µ )a.s.--→ 0 for every constant vector c with bounded norm. The estimator equivalence in Theorem 1 is a "first-order" result, while we will also characterize the "second-order" effects in Section 4.The notable aspect of Theorem 1 is its generality. The equivalence results hold for a wide range of weight matrices and allow for negative values for the regularization levels. Furthermore, we have not made any direct assumptions about the feature matrix Φ, the weight matrix W , and the response vector y (other than mild bounded norms). The main underlying ingredient is the asymptotic freeness between W and Φ, which we then exploit using tools developed in [39] in the context of feature sketching. We discuss special cases of interest for W and Φ in the upcoming Sections 3.1 and 3.2.There are two classes of weighting matrices that are of practical interest:• Non-diagonal weighting matrices. One can consider observation sketching, which involves some random linear combinations of the rows of the data matrix. Such observation sketching is beneficial for privacy, as it scrambles the rows of the data matrix, which may contain identifiable information about individuals. It also helps in reducing the effect of non-i.i.d. data that arise in time series or spatial data, where one wants to smooth away the impact of irregularities or non-stationarity. • Diagonal weighting matrices. When observations are individually weighted, W is a diagonal matrix, which includes scenarios such as resampling, bootstrapping, and subsampling. Note that even with subsampling, one can have a non-binary diagonal weighting matrix. For example, one can consider sampling with replacement or sampling with a particular distribution, which yields non-binary diagonal weighting matrices. Other examples of non-binary diagonal weighting matrices include inverse-variance weighting sampling to mitigate the effects of heterogeneous variations if the responses have different variances for different units.In general, the set of equivalent weighted estimators depends on the corresponding S-transform as in (2) , and it can be numerically evaluated. When focusing on subsampling without replacement, the data-dependent path for equivalent estimators with associated subsampling and regularization levels can be explicitly characterized in the following result by analyzing the S-transform of subsampling operators. Theorem 2 (Regularization paths due to subsampling). For a subsampling matrix W (k) consisting of k unit diagonal entries, the path (2) in terms of (k, λ) simplifies to:EQUATIONwhere we denote by df = df( β I,µ ) = df( β W ,λ ) for notational simplicity.The relation ( 4) is remarkably simple, yet quite general! It provides an interplay between the normalized target complexity df/n, regularization inflation λ/µ, and subsample fraction k/n:EQUATION)Since the normalized target complexity and subsample fraction are no greater than one, (5) also implies that the regularization level λ for the subsample estimator is always lower than the regularization level µ for the full estimator. In other words, subsampling induces (positive) implicit regularization, reducing the need for explicit ridge regularization. This is verified numerically in Figure 1 .For a fixed target regularization amount µ, the degrees of freedom df( β I,µ ) of the ridge estimator on full data is fixed. Thus, we can observe that the path in the (k/n, λ)-plane is a line. There are two extreme cases: (1) when the subsample size k is close to n, we have µ ≈ λ; and (2) when the subsample size is near 0, we have µ ≈ ∞. When λ = 0, the effective regularization level λ is such that df( β W (k) ,λ ) = df( β I,µ ) = k, which we find to be a neat relation! Beyond subsampling without replacement, one can also consider other subsample matrixs. For example, for bootstrapping k entries, we observe a similar equivalent path in Figure 5 . Additionally, for random sample reweighting, as shown in Figure 6 , we also observe certain equivalence behaviors of degrees of freedom. This indicates that Theorem 1 also applies to more general weighting schemes.As mentioned in Section 2, when the feature matrix Φ consists of i.i.d. Gaussian features, any deterministic matrix W satisfies the condition stated in Assumption A. However, our results are not limited to Gaussian features. In this section, we will consider more general families of features commonly analyzed in machine learning and demonstrate the applicability of our results to them.(1) Linear features. As a first example, we consider linear features composed of (multiplicatively) transformed i.i.d. entries with sufficiently bounded moments by a deterministic covariance matrix. Proposition 3 (Regularization paths with linear features). Suppose the feature ϕ can be decomposed as ϕ = Σ 1/2 z, where z ∈ R p contains i.i.d. entries z i for i = 1, . . . , p with mean 0, variance 1, and satisfies E[|z i | 4+µ ] ≤ M µ < ∞ for some µ > 0 and a constant M µ , and Σ ∈ R p×p is a deterministic symmetric matrix with eigenvalues uniformly bounded between constants r min > 0 and r max < ∞.Then, as n, p → ∞ such that p/n → γ > 0, the equivalences in (3) hold along the path (4).Features of this type are common in random matrix theory [8] and in a wide range of applications, including statistical physics [14, 60] , high-dimensional statistics [22, 55, 58] , machine learning [18] , among others. The generalized path (2) in Theorem 2 recovers the path in Proposition 4 of [50] .Although the technique in this paper is quite different and more general than that of [50] .(2) Kernel features. As the second example, Theorem 2 also applies to kernel features. Kernel features are a generalization of linear features and lift the input feature space to a high-or infinitedimensional feature space by applying a feature map x → ϕ(x). Kernel methods use the kernel function K(x i , x j ) = ⟨ϕ(x i ), ϕ(x j )⟩ to compute the inner product in the lifted space.Proposition 4 (Regularization paths with kernel features). Suppose the same conditions as in Proposition 3 and the kernel function is of the form K(x i , x j ) = g(∥x i ∥ 2 2 /p, ⟨x i , x j ⟩/p, ∥x j ∥ 2 2 /p), where g is C 1 around (τ, τ, τ ) and C 3 around (τ, 0, τ ) and τ := lim p→∞ tr[Σ]/d. Then, as n → ∞, the equivalences in (3) hold in probability along the path (4).The assumption in Proposition 4 is commonly used in the risk analysis of kernel ridge regression [9, 19, 29, 57] , among others. Here, C k denotes the class of functions that are k-times continuously differentiable. It includes neural tangent kernels (NTKs) as a special case. Proposition 4 confirms Conjecture 8 of [50] for these types of kernel functions.(3) Random features. Finally, we consider random features that were introduced by [56] as a way to scale kernel methods to large datasets. Linked closely to two-layer neural networks [46] , the random feature model has f nn (x) = σ(F x), where F ∈ R d×p is some randomly initialized weight matrix, and σ : R → R is a nonlinear activation function applied element-wise to F x.Proposition 5 (Regularization paths with random features). Suppose x i ∼ N (0, Σ) and the activation function σ : R → R is differentiable almost everywhere and there are constants c 0 and c 1 such that |σ(x)|, |σ ′ (x)| ≤ c 0 e c1x , whenever σ ′ (x) exists. Then, as n, p, d → ∞ such that p/n → γ > 0 and d/n → ξ > 0, the equivalences in (3) hold in probability along the path (4).As mentioned in the related work, random feature models have recently been used as a standard model to study various generalization phenomena observed in neural networks theoretically [1, 46] . Proposition 5 resolves Conjecture 7 of [50] under mild regularity conditions on the activation function.It is worth noting that the prior works mentioned above, including [50] , have focused on first characterizing the risk asymptotics in terms of various population quantities for each of the cases above. In contrast, our work in this paper deviates from these approaches by not expressing the risk in population quantities but rather by directly relating the estimators at different regularization levels. In the next section, we will explore the relationship between their squared prediction risks.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
prediction risk asymptotics and risk estimation: 
 The results in the previous section provide first-order equivalences of the estimators, which are related to the bias of the estimators. In practice, we are also interested in the predictive performance of the estimators. In this section, we investigate the second-order equivalence of weighting and ridge regularization through ensembling. Specifically, we show that aggregating estimators fitted on different weighted datasets also reduces the additional variance. Furthermore, the prediction risks of the full-ensemble weighted estimator and the unweighted estimator also match along the path.Before presenting our risk equivalence result, we first introduce some additional notation. Assume there are M i.i.d. weight matrices W 1 , . . . , W M ∈ R n×n . The M -ensemble estimator is defined as:EQUATIONand its performance is quantified by the conditional squared prediction risk, given by:EQUATIONwhere (x 0 , y 0 ) is a test point sampled independently from some distribution P x0,y0 that may be different from the training distribution P x,y , and ϕ 0 = f nn (x 0 ) is the pretrained feature at the test point. The covariance matrix of the test features ϕ 0 is denoted by Σ 0 . When P x0,y0 = P x,y , we refer to it as the in-distribution risk. On the other hand, when P x0,y0 differs from P x,y , we refer to it as the out-of-distribution risk. Note that the conditional risk R M is a scalar random variable that depends on both the dataset (Φ, y) and the weight matrix W m for m ∈ [M ]. Our goal in this section is to analyze the prediction risk of the ensemble estimator (6) for any ensemble size M . Theorem 6 (Risk equivalence along the path). Under the setting of Theorem 1, assume that the operator norm of Σ 0 is uniformly bounded in p and that each response variable y i for i = 1, . . . , n has mean 0 and satisfies E[|y i | 4+µ ] ≤ M µ < ∞ for some µ, M µ > 0.Then, along the path (4),EQUATIONwhere the constant C is given by:EQUATIONAt a high level, Theorem 6 provides a bias-variance-like risk decomposition for both the squared risks of weighted ensembles. The risk of the weighted predictor is equal to the risk of the unweighted equivalent implicit ridge regressor (bias) plus a term due to the randomness due to weighting (variance). The inflation factor C controls the magnitude of this term, and it decreases at a rate of 1/M as the ensemble size M increases (see Figure 7 for a numerical verification of this rate). Therefore, by using a resample ensemble with a sufficiently large size M , we can retain the statistical properties of the full ridge regression while reducing memory usage and increasing parallelization.Theorem 6 extends the risk equivalence results in [50, 52] . Compared to previous results, Theorem 6 provides a broader risk equivalence that holds for general weight and feature matrices, as well as an arbitrary ensemble size M . It is important to note that Theorem 6 holds even when the test distribution differs from the training data, making it applicable to out-of-distribution risks. Furthermore, our results do not rely on any specific distributional assumptions for the response vector, making them applicable in a model-free setting. The key idea behind this result is to exploit asymptotic freeness between the subsample and data matrices. Next, we will address the question of optimal tuning.As in Theorem 2, we next analyze various properties related to optimal subsampling weights and their implications for the risk of optimal ridge regression. Recall that the subsampling matrix W (k) is a diagonal matrix with k ∈ {1, . . . , n} nonzero diagonal entries, which is parameterized by the subsample size k. Note that the optimal regularization parameter µ * for the full data (W (k) = I or k = n) is a function of the distribution of pretrained data and the test point. Based on the risk equivalence in Theorem 6, there exists an optimal path of (k, λ) with the corresponding full-ensemble estimatorβ W (k) 1:∞ ,λ := lim M →∞ β W (k)1:M ,λ that achieves the optimal predictive performance at (n, µ * ). In particular, the ridgeless ensemble with λ * = 0 happens to be on the path. From previous work [25, 50] , the optimal subsample size k * for λ * = 0 has the property that k * ≤ p under linear features. We show in the following that this property can be extended to include general features. Proposition 7 (Optimal subsample ratio). Assume the subsampling matrix W as defined in Theorem 2. Let µ * = argmin µ≥0 R( β W (k) 1:∞ ,µ ). Then the corresponding subsample size satisfies:EQUATIONThe optimal subsample size k * obtained from Proposition 7 is asymptotically optimal. For linear features, in the underparameterized regime where n > p, [25, 50] show that the optimal subsample size k * is asymptotically no larger than p. This result is covered by Proposition 7 by noting that rank(G I ) ≤ p under linear features. It is interesting and somewhat surprising to note that in the underparameterized regime (when p ≤ n), we do not need more than p observations to achieve the optimal risk. In this sense, the optimal subsampled dataset is always overparameterized.When the limiting risk profiles R(γ, ψ, µ) := lim p/n→γ,p/k→ψ R( β W (k) 1:∞ ,µ ) exist for subsample ensembles, the limiting risk of the optimal ridge predictor inf µ≥0 R(γ, γ, µ) is monotonically decreasing in the limiting sample aspect ratio γ [50] . This also (provably) confirms the sample-wise monotonicity of optimally-tuned risk for general features in an asymptotic sense [48] . Due to the risk equivalence in Theorem 6, for any µ > 0, there exists ψ such that R(γ, γ, µ) = R(γ, ψ, 0). This implies that inf µ≥0 R(γ, γ, µ) = inf ψ≥γ R(γ, ψ, 0). In other words, tuning over subsample sizes with sufficiently large ensembles is equivalent to tuning over the ridge penalty on the full data.As suggested by Proposition 7, the optimal subsample size is smaller than the rank of the Gram matrix. This result has important implications for real-world datasets where the number of observations (n) is much larger than the number of features (p). In such cases, instead of using the entire dataset, we can efficiently build small ensembles with a subsample size k ≤ p. This approach is particularly beneficial when n is significantly higher than p, for example, when n = 1000p. By fitting ensembles with only M = 100 base predictors, we can potentially reduce the computational burden while still achieving optimal predictive performance. Furthermore, this technique can be especially valuable in scenarios where computational resources are limited or when dealing with massive datasets that cannot be easily processed in their entirety.In the following, we propose a method to determine the optimal values of the regularization parameter µ * for the full ridge regression, as well as the corresponding subsample size k * and the optimal ensemble size M * . According to Theorem 6, the optimal value of M * is theoretically infinite. However, in practice, the prediction risk of the M -ensemble predictor decreases at a rate of 1/M as M increases. Therefore, it is important to select a suitable value of M that achieves the desired level of performance while considering computational constraints and the specified error budget. By carefully choosing an appropriate M , we can strike a balance between model accuracy and efficiency, ensuring that the subsampled neural representations are effectively used in downstream tasks.Consider a grid of subsample size K n ⊆ {1, . . . , n}; for instance, K n = {0, k 0 , 2k 0 , . . . , n} where k 0 is a subsample size unit. For a prespecified subsample size k ∈ K n and ensemble size M 0 ∈ Algorithm 1 Meta-algorithm for tuning of ensemble sizes and subsample matrices. ,λ with R m,k by CV methods such as CGCV [13] , for k ∈ K n and m = 1, . . . , M 0 . 3: Extrapolate the risk estimations R m,k for m > M 0 using (11) and (12) . 4: Select a subsample size k ∈ argmin k∈Kn R ∞,k . that minimizes the extrapolated estimates. 5: Select an ensemble sizeInput: A dataset D n = {(x i , y i ) ∈ R p × R : 1 ≤ i ≤ n},M ∈ argmin m∈N 1{ R m, k > R ∞, k + δ} for the δ-optimal risk. 6: If M > M 0 , fit a M -ensemble estimator β W ( k) 1: M ,λ.Output: Return the tuned estimator βW ( k) 1: M ,λ, and the risk estimators R M,k for all M, k.N, suppose we have multiple risk estimates R m of R m for m = 1, . . . , M 0 . The squared risk decomposition [51, Eq (7) ] along with the equivalence path (8) EQUATIONThen, the extrapolated risk estimates R m (with m > M 0 ) are defined as:EQUATIONThe meta-algorithm that implements the above cross-validation procedure is provided in Algorithm 1.To efficiently tune the parameters of ridge ensembles, we use and combine the corrected generalized cross-validation (CGCV) method [13] and the extrapolated cross-validation (ECV) method [26] . The improved CV method is implemented in the Python library [24] .In this section, we present numerical experiments to validate our theoretical results on real-world datasets. Figure 3 provides evidence supporting Assumption A on pretrained features extracted from commonly used neural networks applied to real-world datasets. The first panel of the figure demonstrates the equivalence of degrees of freedom for these pretrained features. Furthermore, we also observe consistent behavior across different neural network architectures and different datasets (see Figures 8 and 9 ). Remarkably, the path of equivalence can be accurately predicted, offering valuable insight into the underlying dynamics of these models. This observation suggests that the pretrained features from widely used neural networks exhibit similar properties when applied to realworld data, regardless of the specific architecture employed. The ability to predict the equivalence path opens up new possibilities for optimizing the performance of these models in practical applications.One implication of the equivalence results explored in Theorems 1 and 6 is that instead of tuning for the full ridge penalty µ on the large datasets, we can fix a small value of the ridge penalty λ, fit subsample ridge ensembles, and tune for an optimal subsample size k. To illustrate the validity of the tuning procedure described in Algorithm 1, we present both the actual prediction errors and their estimates by Algorithm 1 in Figure 4 . We observe that the risk estimates closely match the prediction risks at different ensemble sizes across different datasets. Even with a subsampling ratio k/n of 0.01 and a sufficiently large M , the risk estimate is close to the optimal risk. A smaller subsample size could also yield even smaller prediction risk in certain datasets. M = 1 M = 1 M = 2 M = 2 M = 5 M = 5 M = 10 M = 10 M = 100 M = 100 
limitations and outlook: 
 While our results are quite general in terms of applying to a wide variety of pretrained features, they are limited in that they only apply to ridge regression fitted on the pretrained features. The key challenge for extending the analysis based on Assumption A to general estimators beyond ridge regression is the characterization of the effect of subsampling general resolvents as additional ridge regularization. To extend to generalized linear models, one approach is to view the optimization as iteratively reweighted least squares [38] in combination with the current results. Another approach is to combine our results with the techniques in [41] to obtain deterministic equivalents for the Hessian, enabling an understanding of implicit regularization due to subsampling beyond linear models.Beyond implicit regularization due to subsampling, there are other forms of implicit regularization, such as algorithmic regularization due to early stopping in gradient descent [2, 3, 49] , dropout regularization [62, 65] , among others. In some applications, multiple forms of implicit regularization are present simultaneously. For instance, during a mini-batch gradient step, implicit regularization arises from both iterative methods and mini-batch subsampling. The results presented in this paper may help to make explicit the combined effect of various forms of implicit regularization.This serves as an appendix to the paper "Implicit Regularization Paths of Weighted Neural Representations." The beginning (unlabeled) section of the appendix provides an organization for the appendix, followed by a summary of the general notation used in both the paper and the appendix. Any other specific notation is explained inline where it is first used.• In Appendix A, we provide a brief technical background on free probability theory and various transforms that we need and collect known asymptotic ridge equivalents that we use in our proofs. • In Appendix B, we present proofs of the theoretical results in Section 3 (Theorems 1 and 2 and Propositions 3-5). • In Appendix C, we present proofs of the theoretical results in Section 4 (Theorem 6 and Proposition 7). • In Appendix D, we provide additional illustrations for the results in Section 3 (Figures 5 and 6 ).• In Appendix E, we provide additional illustrations for the results in Section 4 (Figures 7 8 9 ), including our meta-algorithm for tuning (Algorithm 1) that is not included in the main text due to space constraints. • In Appendix F, we provide additional details on the experiments in both Section 3 and Section 4.We use blackboard letters to denote some special sets: N denotes the set of natural numbers, R denotes the set of real numbers, R + denotes the set of positive real numbers, C denotes the set of complex numbers, C + denotes the set of complex numbers with positive imaginary part, and C - denotes the set of complex numbers with negative imaginary part. We use [n] to denote the index set {1, 2, . . . , n}.We denote scalars and vectors using lower-case letters and matrices using upper-case letters. For a vector β, β ⊤ denotes its transpose, and ∥β∥ 2 denotes its ℓ 2 norm. For a pair of vectors u and v, ⟨u, v⟩ denotes their inner product. For a matrix X ∈ R n×p , X ⊤ ∈ R p×n denotes its transpose, and X † ∈ R p×n denotes its Moore-Penrose inverse. For a square matrix A ∈ R p×p , tr[A] denotes its trace, tr[A] denotes its average trace tr[A]/p, and A -1 denotes its inverse, provided that A is invertible. For a symmetric matrix A, λ + min (A) denotes its minimum nonzero eigenvalue. For a positive semidefinite matrix G, G 1/2 denotes its principal square root. For a matrix X, we denote by ∥X∥ op its operator norm with respect to the ℓ 2 vector norm. It is also the spectral norm of X. For a matrix X, we denote by ∥X∥ tr its trace norm. It is given by tr[(X ⊤ X) 1/2 ], and is also the nuclear norm X. We denote p × p identity matrix by I p , or simply by I when it is clear from the context. For symmetric matrices A and B, we use A ⪯ B to denote the Loewner ordering to mean that A -B is a positive semidefinite matrix. For two sequences of matrices A p and B p , we use A p ≃ B p to denote a certain asymptotic equivalence; see Appendix A.3 for a precise definition.In this section, we briefly review definitions from free probability theory and its applications to random matrices. This review will help set the stage by introducing the various mathematical structures and spaces we are working with. It will also introduce some of the notation used throughout the text.Free probability is a mathematical framework that deals with non-commutative random variables [19] . The use of free probability theory has appeared in various recent works in statistical machine learning, including [1, 2, 11, 12, 17] . Good references on free probability theory include [3, 13] , from which we borrow some basic definitions in the following. All the material in this section is standard in free probability theory and mainly serves to keep the definitions self-contained. Definition 8 (Non-commutative algebra). A set A is called a (complex) algebra (over the field of complex numbers C) if it is a vector space (over C with addition +), equipped with a bilinear multiplication •, such that for all x, y, z ∈ A and α ∈ C,(1) x • (y • z) = (x • y) • z, (2) (x + y) • z = x • z + y • z, (3) x • (y + z) = x • y + x • z, (4) α(x • y) = (αx) • y = x • (αy).In addition, an algebra is called unital if a multiplicative identity element exists. We will use 1 A to denote this identity element. We will drop the "•" symbol to denote multiplication over the algebra. Definition 9 (Non-commutative probability space). Let A over C be a unital algebra with identity 1 A . Let φ : A → C be a linear functional which is unital (that is, φ(1 A ) = 1). Then (A, φ) is called a non-commutative probability space, and φ is called a state. A state φ is said to be tracial if φ(xy) = φ(yx) for all x, y ∈ A. Definition 10 (Moments). Let (A, φ) be a non-commutative probability space. The numbers {φ(x k )} ∞ k=1 are called the moments of the variable x ∈ A. Definition 11 ( * -algebra). An algebra A is called a * -algebra if there exists a mapping x → x * from A → A such that, for all x, y ∈ A and α ∈ C,(1) (x + y) * = x * + y * , (2) (αx) * = αx * , (3) (xy) * = y * x * , (4) (x * ) * = x.A variable x of a * -algebra is called self-adjoint if x = x * . A unital linear functional φ on a * -algebra is said to be positive if φ(x * x) ≥ 0 for all x ∈ A. Definition 12 ( * -probability space). Let A be a unital * -algebra with a positive state φ. Then (A, φ) is called a * -probability space. Example 1. Denote by M p (C) the collection of all p × p matrices with complex entries. Let the multiplication and addition operations be defined in the usual way. The * -operation is the same as taking the conjugate transpose. Let tr : M p (C) → C be the normalized trace defined by:tr(A) = 1 p tr[A].The state tr is tracial and positive. Definition 13 (Free independence). Suppose (A, φ) is a * -probability space. Then, the * -subalgebras {A i } i∈I of A are said to be * -freely independent (or simply * -free) if, for all n ≥ 2 and allx 1 , x 2 , • • • , x n from {A i } i∈I , κ n (x 1 , x 2 , • • • , x n ) = 0 whenever at least two of the x i are from different A i .In particular, any collection of variables is said to be * -free if the sub-algebras generated by these variables are * -free. Lemma 14. Suppose (A, φ) is a * -probability space. If x and y are free in (A, φ), then for all non-negative integers n and m, φ(x n y m ) = φ(x n )φ(y m ) = φ(y m x n ).In other words, elements of the algebra are considered free if any alternating product of centered polynomials is also centered.In this work, we will consider φ to be the normalized trace. The normalized trace is the generalization of 1 p tr[A] for A ∈ C p×p to elements of a C * -algebra A. Specifically, for any self-adjoint a ∈ A and any polynomial p, we haveφ(p(a)) = p(z) dµ a (z),where µ a is the probability measure that characterizes the spectral distribution of a. Definition 15 (Convergence in spectral distribution). Let (A, φ) be a C * -probability space. We say that A 1 , . . . , A m ∈ C p×p converge in spectral distribution to elements a 1 , . . . , a m ∈ A if, for all 1 ≤ ℓ < ∞ and 1 ≤ i j ≤ m for 1 ≤ j ≤ ℓ, we have1 p tr[A i1 • • • A i ℓ ] → φ(a i1 • • • a i ℓ ).Then, with slight abuse of notation, two matrices A, B ∈ R p×p are said to be free if1 p tr L ℓ=1 poly A ℓ (A)poly B ℓ (B) = 0,for all L ≥ 1 and all centered polynomials, that is, tr[poly A ℓ (A)] = 0. This notation is an abuse of notation because finite matrices cannot satisfy this condition. However, they can satisfy it asymptotically as p → ∞, and in this case, we say that A and B are asymptotically free.Note: With some abuse of notation, we will let matrices in boldface denote both the finite matrix and the limiting element in the free probability space. The limiting element can be understood, for example, as a bounded linear operator on a Hilbert space. We also remark that all notions we need are well-defined in this limit as well, as long as they are appropriately normalized.In this section, we review the key transforms used in free probability theory and their interrelationships. Definition 16 (Cauchy transform). Let a be an element of a * -probability space (A, φ). Suppose there exists some C > 0 such that |φ(a n )| ≤ C n for all n ∈ N. Then the Cauchy transform of a is defined as:G a (z) = ∞ n=0 φ(a n ) z n+1for all z ∈ C with |z| > C.Note that the Cauchy transform is the negative of the Stieltjes transform. In this paper, we will focus only on the Cauchy transform. Recall that for a probability measure ν on R and for z / ∈ R, the Cauchy transform of ν is defined as:G(z) = R 1 z -x dν(x).The definition above is motivated by the following property of the Cauchy transform of a measure. Suppose ν is a probability measure whose support is contained in [-C, C] for some C > 0 and which has moments {m k (ν)} ∞ k=0 . Then the Cauchy transform of ν is defined for z ∈ C with |z| > C as:G ν (z) = ∞ k=0 m k (ν) z k+1 .Definition 17 (Moment generating function). Let a be an element of a * -probability space (A, φ).The moment generating function of a is defined as:M a (z) = 1 + ∞ k=1 φ(a k )z k for z ∈ C such that |z| < r a .Here, r a is the radius of convergence of the series.For a probability measure ν, the moment generating function is defined analogously. (Note: The definition above is not to be confused with the moment generating function of a random variable in probability theory.) The Cauchy transform is related to the moment series via:EQUATION)In the other direction, we have:EQUATION)Definition 18 (S-transform). ForM a (z) = ∞ m=0 φ(a m )z m ,we define the S-transform of a by:EQUATIONwhere M ⟨-1⟩ denotes the inverse under composition of M.Finally, in terms of operator A, we summarize the series of invertible transformations between the various transforms introduced in this section.• Cauchy transform:G A (z) = tr[(zI -A) -1 ].• Moment generating series:M A (z) = 1 z G A 1 z -1.• S-transform:S A (w) = 1 + w w M ⟨-1⟩A (w).Here:• M A (z) = ∞ k=1 tr[A k ]z k is the moment generating series. • M ⟨-1⟩ Adenotes the inverse under composition of M A .• tr[A] denotes the average trace tr[A]/p of a matrix A ∈ R p×p .In this section, we provide a brief background on the language of asymptotic equivalents used in the proofs throughout the paper. We will state the definition of asymptotic equivalents and point to useful calculus rules. For more details, see [16, Appendix S.7] .To concisely present our results, we will use the framework of asymptotic equivalence [5, 6, 16] , defined as follows. Let A p and B p be sequences of matrices of arbitrary dimensions (including vectors and scalars). We say that A p and B p are asymptotically equivalent, denoted asA p ≃ B p , if lim p→∞ | tr[C p (A p -B p )]| = 0 almostsurely for any sequence of random matrices C p with bounded trace norm that are independent of A p and B p . Note that for sequences of scalar random variables, the definition simply reduces to the typical almost sure convergence of sequences of random variables involved.The notion of deterministic equivalents obeys various calculus rules such as sum, product, differentiation, conditioning, and substitution. We refer the reader to [16] for a comprehensive list of these calculus rules, their proofs, and other related details.Next, we collect first-and second-order asymptotic equivalents for sketched ridge resolvents from [11, 17] , which will be useful for our extensions to weighted ridge resolvents. Assumption B (Sketch structure). Let S ∈ R p×q be the feature sketching matrix and X ∈ R n×p be the data matrix. Let SS ⊤ and 1 n X ⊤ X converge almost surely to bounded operators that are infinitesimally free with respect to ( 1 p tr[•], tr[Θ(•)]) for any Θ independent of S with ∥Θ∥ tr uniformly bounded. Additionally, let SS ⊤ have a limiting S-transform that is analytic on the lower half of the complex plane.For the statement to follow, let us define Σ := 1 n X ⊤ X. Let λ 0 := -lim inf p→∞ λ + min (S ⊤ ΣS). Here, recall that λ + min (A) represents the minimum nonzero eigenvalue of a symmetric matrix A. Theorem 19 (Free sketching equivalence; [11] , Theorem 7.2). Under Assumption B, for all λ > λ 0 ,EQUATIONwhere ν > -λ + min ( Σ) is increasing in λ > λ 0 and satisfies:EQUATIONLemma 20 (Second-order equivalence for sketched ridge resolvents; [17] , Lemma 15) . Under the settings of Lemma 21, for any positive semidefinite Ψ with uniformly bounded operator norm, for all λ > λ 0 ,EQUATIONwhere ν ′ Ψ ≥ 0 is given by:EQUATIONB Proofs in Section 3Our main ingredient in the proof is Lemma 21. We will first show estimator equivalence and then show degrees of freedom equivalence.Estimator equivalence. Recall from (1) the ridge estimator on the weighted data is:β W ,λ = (Φ ⊤ W ⊤ W Φ/n + λI p ) † Φ ⊤ W ⊤ W y/n.This is the "primal" form of the ridge estimator. Using the Woodbury matrix identity, we first write the estimator into its "dual" form.β W ,λ = Φ ⊤ W ⊤ (W ΦΦ ⊤ W ⊤ /n + λI n ) † W y/n = Φ ⊤ W ⊤ (G W + λI n ) † W y/n.Now, we can apply the first part of Lemma 21 to the matrix W ⊤ (G W + λI n ) † W . From (31), we then have the following equivalence:β W ,λ ≃ Φ ⊤ (G I + µI n ) † y/n = Φ ⊤ (ΦΦ ⊤ + µI n ) † y/n = (Φ ⊤ Φ/n + µI n ) † Φ ⊤ y/n = β I,µ ,where µ satisfies the following equation:µ = λS W ⊤ W - tr[G I (G I + µI n ) † ] n = λS W ⊤ W (-df( β I,µ )).Note that in the simplification, we used the Woodbury identity again to go back from the dual form into the primal form for the ridge estimator based on the full data. Rearranging, we obtain the desired estimator equivalence. We next move on to showing the degrees of freedom equivalence.Degrees of freedom equivalence. For the subsampled estimator β W ,λ , the effective degrees of freedom is given by:df( β W ,λ ) = tr[Φ ⊤ Φ/n(Φ ⊤ Φ/n + λI p ) † ] = tr[Φ(Φ ⊤ Φ/n + λI p ) † Φ ⊤ /n] = tr[(ΦΦ ⊤ /n + λI n ) † ΦΦ ⊤ /n].The second equality above follows from the push-through identity Φ(Φ ⊤ Φ/n + λI p ) † Φ ⊤ = (ΦΦ ⊤ + λI n ) † ΦΦ ⊤ . Recognizing the quantity inside the trace as the degrees of freedom of the full ridge estimator, we have µ = λS W ⊤ W (-df( β I,µ )).We can equivalently write the equation above as-S -1 W ⊤ W µ λ = df( β I,µ ) or µ λ = S W ⊤ W (-df( β I,µ )).Rearranging the display above provides the desired degrees of freedom equivalence and finishes the proof.We will apply Theorem 1 to the subsampling weight matrix W . The main ingredient that we need is the S-transform of the spectrum of the matrix W ⊤ W . As summarized in Appendix A.2, one approach to compute the S-transform is to go through the following chain of transforms. First, we apply the Cauchy transform, then the moment-generating series, and finally, take the inverse to obtain the S-transform. We will do this in the following steps.Cauchy transform. Recall that the Cauchy transform from Definition 16 can be computed as:G W ⊤ W (z) = tr[(zI n -W ⊤ W ) -1 ].Moment generating series. We can then compute the moment series from Definition 17 using ( 14) as follows:M W ⊤ W (z) = 1 z tr 1 z I n -W ⊤ W -1 -1 = tr[(I n -zW ⊤ W ) -1 ] -tr[I n ] = -tr[I n ] + tr[(I n -zW ⊤ W ) -1 ] = tr[(zW ⊤ W -I n + I n )(I n -zW ⊤ W ) -1 ] = tr[zW ⊤ W (I n -zW ⊤ W ) -1 ].We now note that the matrix W ⊤ W has k eigenvalues of 1 and n -k eigenvalues of 0. Therefore, we haveEQUATIONS-transform. The inverse of the moment generating series map z → M W ⊤ W (z) from ( 20) is:EQUATIONTherefore, from Definition 18 and using (21), we haveEQUATIONNow, we are ready to apply Theorem 1 to the subsampling matrix W .Substituting ( 22) into (2), we getµ λ = S(-df( β I,µ )) = 1 -df( β I,µ ) -df( β I,µ ) + k/n .Here, K(x, X) ∈ R n with i-th entry K(x, x i ), and K(X, X) ∈ R n×n with the ij-th entry K(x i , x j ).The predicted values on the training data X are given byf W ,λ (X) = K(X, X) ⊤ W ⊤ (W K(X, X)W ⊤ + λI n ) † W y.Here, the matrix K(X, X) ⊤ W ⊤ (W K(X, X)W ⊤ + λI n ) † W is the smoothing matrix.Define G I = K(X, X) and G W = W K(X, X)W ⊤ . Leveraging the kernel trick, the preceding optimization problem translates into solving the following problem (in the dual domain):α W ,λ = argmin α∈R n α ⊤ (G W + λI n ) α + 2α ⊤ W y,where the dual solution is given by α W ,λ = (G W + λI n ) † W y. The correspondence between the dual and primal solutions is simply given by:β W ,λ = Φ ⊤ W ⊤ α W ,λ where Φ = [ϕ(x 1 ), . . . , ϕ(x n )]⊤ is the feature matrix and ϕ : R d → H is the feature map of the Hilbert space H with kernel K.Thus, f W ,λ (X) = W Φ β W ,λ = W ΦΦ ⊤ W ⊤ α W ,λ = G W (G W + λI n ) † W yand the degrees of freedom is given by df(β I,µ ) = tr[G W (G W + λI n ) † ].Next, we show that (3) holds. Alternatively, one can also show that α W ,λ ≃ α I,µ , and f W ,λ (x 0 ) ≃ f I,µ (x 0 ), which we omit due to similarity. Our proof strategy consists of two steps. We first show that it suffices to establish the desired result for the linearized version. We then show that we can suitably adapt our result for the linearized version.Linearization of kernels. In the below, we will show that for µ ≥ 0,W ⊤ (G W + λI n ) † W ≃ (G I + µI n ) † , tr[λ(G W + λI n ) † ] ≃ tr[µ(G I + µI n ) † ],where W and λ ≥ 0 satisfy thatEQUATIONUsing assumptions of Proposition 3 and the assumption in Proposition 4, by [18, Proposition 5.1],2 ∥G I -G lin I ∥ op p - → 0,where G lin I = c 0 I n + c 1 1 n 1 ⊤ n + c 2 XX ⊤ and (c 0 , c 1 , c 2 )EQUATIONEQUATIONEQUATIONAssume C n is a sequence of random matrices with bounded trace norm. Note thattr[C p ((G I + µI n ) † -(G lin I + µI n ) † )] ≤ tr[C p ]∥(G I + µI n ) † -(G lin I + µI n ) † ∥ op ≤ tr[C p ]∥(G I + µI n ) † ∥ op ∥(G lin I + µI n ) † ∥ op ∥G I -G lin I ∥ op a.s.--→ 0.where in the last inequality, we use a matrix identity A -1 -B -1 = A -1 (B -A)B -1 for two invertible matrices A and B. Thus, we haveEQUATIONHence, combining (29) and the transition property of asymptotic equivalence [15, Lemma S.7.4 (1) ], it suffices to showW ⊤ (G lin W + λI n ) † W ≃ (G lin I + µI n ) †, where G lin W = W G lin I W ⊤ , and λ and µ satisfy (25) . Similarly, we can also show that the path ( 25) is asymptotically equivalent toEQUATIONEquivalence for linearized kernels. We next show that the resolvent equivalence result holds for K lin . This follows from additional manipulations building on Lemma 21.In the below, we will show that for µ ≥ 0,W ⊤ (W ΦΦ ⊤ W ⊤ /n + λI n ) † W ≃ (ΦΦ ⊤ /n + µI n ) † , tr[λ(W ΦΦ ⊤ W ⊤ /n + λI n ) † ] ≃ tr[µ(ΦΦ ⊤ /n + µI n ) † ].Under assumptions in Proposition 5, the linearized features take the formΦ lin = ρ s d F X + √ ρ s ω s U ,where the constants ρ s and ω s are given in Proposition 5 and U ∈ R n×p has i.i.d. standard normal entries. From Claim A.13 of [10] , the linear functionals of the estimators β D,λ and β I,µ with random features Φ and Φ lin are asymptotically equivalent. Now, following the proof of Proposition 4, we apply Lemma 21 on Φ lin to yield the desired result.In preparation for the forthcoming statement, define λ 0 = -lim inf n→∞ λ + min (G W ). Recall the Gram matrices G = ΦΦ ⊤ /n and G W = W ΦΦ ⊤ W ⊤ /n. Lemma 21 (General first-order equivalence for freely subsampled ridge resolvents). For W ∈ R n×n , suppose Assumption A holds for W W ⊤ . Then, for all λ > λ 0 ,EQUATIONEQUATIONwhere µ > -λ + min (G) solves the equation:EQUATION)Proof of Lemma 21. The first result follows from using Theorem 19 by suitably changing the roles of X and Φ. In particular, we set Φ to be X ⊤ and W to be S and apply Theorem 19 to obtainEQUATION)Writing in terms of G and G W , this proves the first part (31) .For the second part, we use the result (34) in the first part and multiply both sides by ΦΦ ⊤ /n to get(ΦΦ ⊤ /n) • W ⊤ (W ΦΦ ⊤ W ⊤ /n + λI n ) † W ≃ (ΦΦ ⊤ /n) • (ΦΦ ⊤ /n + µI n ) † .Using the trace property of asymptotic equivalence [15, Lemma S.7.4 (4)], we havetr[(ΦΦ ⊤ /n) • W ⊤ (W ΦΦ ⊤ W ⊤ /n + λI n ) † W ] ≃ tr[(ΦΦ ⊤ /n) • (ΦΦ ⊤ /n + µI n ) † ].Using the cyclic property of the trace operator yieldstr[(W ΦΦ ⊤ /n) • W ⊤ (W ΦΦ ⊤ W ⊤ /n + λI n ) † ] ≃ tr[(ΦΦ ⊤ /n) • (ΦΦ ⊤ /n + µI n ) † ].In terms of G and G W , this is the same astr[G W (G W + λI n ) † ] ≃ tr[G(G + µI n ) † ].Adding and subtracting λI n and µI n on the left-and right-hand resolvents, we arrive at the second part (32) . This completes the proof.C.1 Proof of Theorem 6The main ingredients of the proof are Lemmas 21 and 22. We begin by decomposing the unknown response y 0 into its linear predictor and residual. Specifically, let β 0 be the optimal projection parameter given by β 0 = Σ -1 0 E[ϕ 0 y 0 ]. Then, we can express the response as the sum of its best linear predictor, ϕ ⊤ 0 β 0 , and the residual, y 0 -ϕ ⊤ 0 β 0 . Denote the variance of this residual byσ 2 0 = E[(y 0 -ϕ ⊤ 0 β 0 ) 2 ].It is easy to see that the risk decomposes as follows: R(β W 1:M ,λ ) = E (y 0 -ϕ ⊤ 0 β W 1:M ,λ ) 2 | Φ, y, {W m } M m=1 = ( β W 1:M ,λ -β 0 ) ⊤ Σ( β W 1:M ,λ -β 0 ) + σ 2 0 .Here, we used the fact that (y 0 -ϕ ⊤ 0 β 0 ) is uncorrelated with ϕ 0 , that is, E[ϕ 0 (y 0 -ϕ ⊤ 0 β 0 )] = 0 p . We note that ∥β 0 ∥ 2 < ∞ and Σ 0 has uniformly bounded operator norm.Observe thatR( β W 1:M ,λ ) = ( β W 1:M ,λ -β 0 ) ⊤ Σ 0 ( β W 1:M ,λ -β 0 ) + σ 2 0 = 1 M M m=1 β Wm,λ -β 0 ⊤ Σ 0 1 M M m=1 β Wm,λ -β 0 + σ 2 0 = 1 M 2 M k,ℓ=1 β ⊤ W k ,λ Σ 0 β W ℓ ,λ - 2 M M m=1 β ⊤ 0 Σ 0 β Wm,λ + β ⊤ 0 Σ 0 β 0 + σ 2 0 = 1 M 2 M k,ℓ=1 ( β ⊤ W k ,λ Σ 0 β W ℓ ,λ -β ⊤ I,µ Σ 0 β I,µ ) + β ⊤ I,µ Σ 0 β I,µ - 2 M M m=1 β ⊤ 0 Σ 0 β Wm,λ + β ⊤ 0 Σ 0 β 0 + σ 2 0 .By Lemma 21, note that1 M M k=1 β Wm,λ ≃ β I,µ .Thus, we haveR( β W 1:M ,λ ) ≃ 1 M 2 M k,ℓ=1 β ⊤ W k ,λ Σ 0 β W ℓ ,λ -β ⊤ I,µ Σ 0 β I,µ + β ⊤ I,µ Σ 0 β I,µ - 2 M M m=1 β ⊤ 0 Σ 0 β I,µ + β ⊤ 0 Σ 0 β 0 + σ 2 0 .Now, by two applications of Lemma 21, we know thatβ ⊤ W k ,λ Σ 0 β W ℓ ,λ -β I,µ Σ 0 β I,µ a.s.--→ 0 when k ̸ = ℓ since W k and W ℓ are independent. Hence, we haveEQUATIONwhere we used the fact that the M terms where k = ℓ converge identically in the second to last line and a risk decomposition similar to that for β W 1:M ,λ in the last line. Thus, it suffices to evaluate the difference β ⊤ λ Σ 0 β λ -β ⊤ µ Σ 0 β µ to finish the proof. We haveEQUATIONwhere in the third line, we used the second-order equivalence for freely weighted ridge resolvents from Lemma 22; in the fourth line, we employed the push-through identity multiple times. Substituting for µ ′ Σ0 from Lemma 22 in (36) and substituting this back into (35) , we arrive at the desired decomposition. This completes the proof.We use the path (4) with k * and µ * , and setting λ * = 0:1 - df( β I,µ * ) n = 1 - k * n .This suggests thatk * = df( β I,µ * ).Note that r := rank(X ⊤ X) = rank(G I ). By the definition of degrees of freedom, it follows thatdf( β I,µ * ) = tr[X ⊤ X(X ⊤ X + µ * I p ) † ] = r i=1 s i s i + µ * ≤ r = rank(G I ),where s 1 , . . . , s r are non-zero eigenvalues of X ⊤ X. This finishes the proof.Recall from Appendix B.6 that we define λ 0 = -lim inf n→∞ λ + min (W ΦΦ ⊤ W ⊤ /n). Lemma 22 (General second-order equivalence for freely weighted ridge resolvents). Under the settings of Lemma 21, for any positive semidefinite Σ 0 with uniformly bounded operator norm, for all λ > λ 0 ,1 n W ⊤ W Φ( 1 n Φ ⊤ W ⊤ W Φ + λI p ) † Σ 0 ( 1 n Φ ⊤ W ⊤ W Φ + λI p ) † Φ ⊤ W ⊤ W • Signal model. Define β 0 = 1 5 5j=1 w (j) where w (j) is the eigenvector of Σ ar1 associated with the top jth eigenvalue r (j) .• Response model. We generated data (x i , y i ) ∈ R d × R for i = 1, . . . , n from a nonlinear model:y i = x ⊤ i β 0 + 1 p (∥x i ∥ 2 2 -tr[Σ ar1 ]) + ε i , x i = Σ 1 2 ar1 z i , z ij iid ∼ t 5 σ 5 , ε i ∼ t 5 σ 5 , (M-AR1)where σ 5 = 5/3 is the standard deviation of t 5 distribution.The benefit of using the above nonlinear model is that we can clearly separate the linear and the nonlinear components and compute the quantities of interest because β 0 happens to be the best linear projection.The linear, random, and kernel features are generated as follows.• Linear features. For a given feature dimension p, we use d = p raw features from (M-AR1) as linear features. • Random features. For generating random features, we use d = 2p raw features from (M-AR1) and sample a randomly initialized weight matrix F ∈ R p×d whose entries are i.i.d. samples from N (0, d -1/2 ). Then the transform feature is given by x i = φ(F x i ) ∈ R p , where φ is a nonlinear transformation and set to be ReLU function in our experiment. • Kernel features. For kernel features, we use d = p raw features from (M-AR1) to construct the kernel matrix.In the simulations, the estimates are averaged across 20 simulations with different random seeds.Following the similar experimental setup in [20] , we use residual networks to extract features on several computer vision datasets, both at random initialization and after pretraining. More specifically, we consider ResNet-{18, 34, 50, 101} applied to the CIFAR-{10,100} [9] , Fashion-MNIST [21] , Flowers-102 [14] , and Food-101 [4] datasets. All random initialization was done following [8] ; pretrained networks (obtained from PyTorch) were pretrained on ImageNet, and the outputs of the last pretrained layer on each dataset mentioned above were used as the embedding feature Φ.After obtaining the embedding features from the last layer of the neural network model, we further normalize each row of the pretrained feature to have a norm of p, and center the one-hot labels to have zero means. To reduce the computational burden, we only consider the first 10 one-hot labels of all datasets. For datasets with different data aspect ratios, we stratify 10% of the training samples as the training set for the CIFAR-100 dataset. The training and predicting errors are the mean square errors on the training and test sets, respectively, aggregated over all the labels. Justification: All the claims made the abstract are justified by both theoretical and experimental results in Sections 3 and 4 and the appendix.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The compute resources are described in the README file of the submitted code in the supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Assumption A1 of[18] requires finite 5 + δ-moments, which can be relaxed to only finite 4 + δ-moments as in the assumption of Proposition3, by a truncation argument as in the proof of Theorem 6 of[7, Appendix A.4]. 
 
 Appendix:  


Decision: After the rebuttal, all reviewers gave positive scores. The AC recommends acceptance based on the following merits:

* The submission presents strong theoretical results based on relaxed assumptions on both the weighting matrix and the feature matrix. It demonstrates the precise relationship between the regularization of weighted ridge regression and simple ridge regression.

* The proposed theory introduces a novel approach to tuning regularization strengths through subsampling. The experiments in Figure 4 confirm the theoretical relationship between risks and validate the effectiveness of the tuning procedure.

The authors are encouraged to improve the organization of the paper following the advice of reviewer cp5Y in the final version.

Reviews: review: paper_summary: paper_summary: This paper investigates the implicit regularization effects of weighted pretrained features. It establishes a path of equivalence between different weighting matrices and ridge regularization with matching effective degrees of freedom. The study extends results to structured features and ensembles, providing theoretical validation for random and kernel features, and proposes an efficient cross-validation method to finetune subsampled pretrained representations in practice.
strength_weakness: strength_weakness: - While Table 1 and the summary of results provide a relatively clear view of the theoretical contributions, the abstract and the motivation part of the introduction read somewhat confusing. For example, the "path of equivalence" is repeatedly mentioned in the abstract and introduction as one of the main contributions, but without a clear explanation/intuition of what it means until later.
- The problem setup and notations are not clearly stated before being used. For example, in terms of setup, in Assumption A, it's unclear what the convergence refers to in "$W^\top W$ and $\Phi \Phi^\top/n$ converge almost surely to bounded operators". In terms of notations, $\|\cdot\|_{tr}$ and $\overline{tr}(\cdot)$ are used without clear definitions for reference.
- Overall, the layout of the results is a little bit hard to follow, partially due to the lack of clear problem setup and notations mentioned before, and partially due to presuming prior knowledge of specific technical tools like free probability theory.
questions: questions: - In Definition A, what's the precise definition of $\overline{tr}(\cdot)$? What does it mean by saying "$W^\top W$ and $\Phi \Phi^\top/n$ converge almost surely to bounded operators"? If it refers to the convergence as $n \to \infty$, then does it mean the analysis is conducted in the asymptotic regime?
- In line 196, if (5) implies $\lambda < \mu$, isn't the regularization level $\lambda$ of the subsampled estimator lower (instead of higher) than that of the full estimator $\mu$?
- It's somehow counter-intuitive that the equivalence path results depend only on the subsample fraction $k/n$ under the "general" data assumption. Common analyses in the asymptotic regime reduce data properties to the subsample ratio $k/n$ usually based on strong assumptions on the data distribution like Gaussian, while data-agnostic subsampling on adversarial data should bring inevitable compromise compared to full data (e.g., with a direction that is only learnable through a single data point). It seems that the "infinitesimally freeness" in Assumption A implicitly circumvents this issue, but it's unclear in the current discussion. For example:
    - Whether Assumption A enforces a "nice" data distribution, a "good" subsampling strategy, or both?
    - If Assumption A implies a "good" subsampling strategy, can such subsampling matrix $W$ be efficiently constructed in practice?
limitations: limitations: Limitations are well-discussed in the Conclusion. Some further limitations are mentioned in Weaknesses and Questions.
paper_summary: paper_summary: This paper studies the weighted linear regression problem where the feature matrix is left-multiplied with a random matrix $\mathbf{W}$ denoting the sample weighting. Under the assumption of the asymptotic freeness between this weighting matrix and the feature matrix, the paper shows that the weighted ridge regression estimator is equivalent to a simple ridge regression estimator in the limit of infinite samples by establishing a connection between the regularization strengths of the weighted and the simple ridge regression. Based on this regularization path, the paper gives illustrative examples of concrete weighting matrix and feature matrix. Moreover, the paper also shows a equivalence between the risk of ensemble weighted and simple ridge regression under the same regularization path. Based on this result, the paper derives an optimal sub-sample size for the ensemble training. Theoretical results of this paper is accompanied by experimental verification.
strength_weakness: strength_weakness: 1. The results derived in the paper has limited applicability. In particular, although it would be interesting to know the equivalence between the weighted ridge regression and the simple ridge regression, the regularization path does not tell much about how should we choose a weighting matrix, even assuming the optimal ridge regression regularization is known. Moreover, the optimal sub-sample size in Proposition 7 involves computing $\mu^*$ which cannot be done easily. Although the paper discussed a way to measure $\mu^*$, it is still computationally heavy, and even in the case where $\mu^*$ is known, one might not be able to measure the degree of freedom easily.
2. The paper considers an asymptotic regime where the sample size goes to infinity. The results in the paper will have better implications if a dependency on $n$ is given.
3. The paper's result is derived under the ridge regression setting. Although one can always choose $\Phi$ to be the output of some pre-trained neural network, the results barely connect to any property of neural network training.
questions: questions: 1. Are there any example in real-world applications where the weighting matrix is not a diagonal matrix?
2. Based on the theoretical results in the paper, what would be the benefit of using a sub-sampling matrix with non-binary diagonal entries?
limitations: limitations: The paper has a good discussion of its limitations.
paper_summary: paper_summary: Neural networks are a powerful tool for extracting features from data, but these features can be very high-dimensional. This high dimensionality can be a bottleneck for training machine learning models, requiring computational power and memory. This manuscript investigates implicit regularization through subsampling and draws a connection to weighted regression problems. The authors provide theoretical results for the claimed connection between implicit regularization via subsampling and regression. The authors provide numerical evidence of their investigations.
strength_weakness: strength_weakness: Some works have provided between subsampling and iterative least squares approaches, see Slagel et al., Sampled Tikhonov Regularization for Large Linear Inverse Problems.
questions: questions: see weaknesses.
limitations: limitations: yes.

Comments: title: Response to the Author's Rebuttal
comment: Thanks, we are happy that our response addressed your concerns! Thanks again for the reference pointer! We will definitely mention this and the comparison in our revision.
Great, thanks! We are happy that you found our response helpful and convincing! Thanks again for all the feedback, which we will definitely use in our revision.
Thanks, we appreciate it! And yes, we will definitely improve on the presentation as indicated in our response above.
Thanks a lot for the detailed response. Assuming a better presentation, I will increase my score to 5.
Thank you for your response. The authors have addressed some of my concerns and I have slightly increased my rating.
Thank you for your response. The authors response to [W1] and [Q1] further convinced me about the applicability of the results. Moreover, the author also discussed how to extend the result to finite $n$ in the response to [W2]. Therefore, I have raised my score accordingly.