Title: Universality in Transfer Learning for Linear Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian Universality

Transfer Learning

Linear Regression

Binary Classification
Abstract: We study the problem of transfer learning and fine-tuning in linear models for both regression and binary classification. In particular, we consider the use of stochastic gradient descent (SGD) on a linear model initialized with pretrained weights and using a small training data set from the target distribution. In the asymptotic regime of large models, we provide an exact and rigorous analysis and relate the generalization errors (in regression) and classification errors (in binary classification) for the pretrained and fine-tuned models. In particular, we give conditions under which the fine-tuned model outperforms the pretrained one. An important aspect of our work is that all the results are "universal", in the sense that they depend only on the first and second order statistics of the target distribution. They thus extend well beyond the standard Gaussian assumptions commonly made in the literature. Furthermore, our universality results extend beyond standard SGD training to the test error of a classification task trained using ridge regression.

Universality in Transfer Learning for Linear Models

Abstract

We study the problem of transfer learning and fine-tuning in linear models for both regression and binary classification. In particular, we consider the use of stochastic gradient descent (SGD) on a linear model initialized with pretrained weights and using a small training data set from the target distribution. In the asymptotic regime of large models, we provide an exact and rigorous analysis and relate the generalization errors (in regression) and classification errors (in binary classification) for the pretrained and fine-tuned models. In particular, we give conditions under which the fine-tuned model outperforms the pretrained one. An important aspect of our work is that all the results are "universal", in the sense that they depend only on the first and second order statistics of the target distribution. They thus extend well beyond the standard Gaussian assumptions commonly made in the literature. Furthermore, our universality results extend beyond standard SGD training to the test error of a classification task trained using ridge regression.

1 Introduction

Deep neural networks have revolutionized the way data processing and statistical inference are conducted. Despite their ground-breaking performance, these models often require a plethora of training samples, which can make the process of data acquisition expensive. Moreover, with the advent of increasingly complex deep networks and especially Large Language Models (LLMs), the process of training from scratch has also become prohibitively expensive. To alleviate the scarcity of prepared data and the high training costs, various strategies have been proposed. One such method is fine-tuning in which a network previously trained on a source dataset/task different from the target dataset/task is leveraged as the initialization point for training on the target dataset/task. In many practical applications, especially for networks containing billions of parameters, only a subset of weights are updated to adapt the model to the new task. A particularly attractive method involves fine-tuning only the last layer of the network. A foundational question would be: How effective is this procedure? Are there fundamental limits to how much one can achieve by utilization of a model pretrained on a different distribution?

In this work, we would like to investigate this problem rigorously through the lens of linear regression and binary classification in the overparametrized regime, where the number of weights/parameters exceeds the number of data points. To do so, we analyze the performance of regressors/classifiers obtained through performing SGD initialized at a weight w 0 ∈ R 𝑑 acquired through training on the source domain. It was shown in Gunasekar et al. [2018] and Azizan and Hassibi [2018] that gradient descent (GD) and stochastic gradient descent(SGD) iterations converge to the optimal solution w ∈ R 𝑑 of the following optimization problem:

where w 0 ∈ R 𝑑 is the initialization point for SGD, X ∈ R 𝑛 × 𝑑 is the design/data matrix which is comprised of independent rows reflecting the fact that datapoints ( x 𝑖 , 𝑦 𝑖 ) are sampled independently, and y ∈ R 𝑛 is the vector of labels. This property of SGD is known as "Implicit Regularization" and will serve as the basis for our analysis as the pretraining step is captured by the vector w 0 . Understanding this "interpolating regime" is key to the theoretical analysis of machine learning models as most of the deep neural networks, on account of their highly overparametrized nature, operate in a setting where they are able to attain negligible training error. It is noteworthy that, even for linear models, characterizing the exact performance of fine-tuning approach has been somewhat limited and is inhibited furthermore by the Gaussianity assumption on X . One of our main contributions is to overcome this limitation. In fact, to showcase the ubiquity of our results, we establish a general universality theorem that applies to a large class of data distributions and extends beyond the context of Transfer Learning. To go into more detail, we say that Gaussian universality holds for a data distribution P and a training algorithm 𝑇 if the test error obtained by training on data sampled from P using 𝑇 is the same as the test error obtained by training on data sampled from the Gaussian distribution N( µ P , 𝚺 P ) using 𝑇 , where µ P and 𝚺 P stand for the mean and the covariance matrix of P respectively. In the context of binary classification, we establish universality of the classification error with respect to the distribution of each class. In light of the the latter result, the problem reduces to analyzing the case of the Gaussian design matrices. We allow classes with different non-scalar covariance matrices 𝚺 1 and 𝚺 2 and build on the results of Akhtiamov et al. [2024] to address the problem in this specific scenario.

1.1 Related Works

Transfer Learning has been an active topic of research at least since the 1970's Bozinovski [2020]. It is mainly applied in the situations where obtaining data from the true distribution, which we will refer as the target distribution , is costly but there is a cheap way to access data from a source distribution , which bears resemblance to the target distribution. The two most popular approaches to transfer learning consist of instance-based transfer learning and fine-tuning.

Instance-based transfer learning incorporates the source dataset, along with the target dataset, and trains the model on this amalgamated dataset. The key insight is that, provided that the source distribution is close enough to the target distibution and a suitable training scheme is chosen, the final performance enjoys an improvement. We refer the reader to the landmark workDai et al. [2007] as well as to the comprehensive overviews covering the empirical advances in this area Tan et al. [2018], Zhuang et al. [2020]. For theoretical analysis, we would like to emphasize a recent work Jain et al. [2024] that characterized the scaling laws for ridge regression over the union of the source and target data in the high dimensional regime.

The present manuscript focuses on fine-tuning in which a model is first pretrained on the source distribution and then fine-tuned using the target data. Dar et al. [2021] analyzes transfer learning for linear regression assuming Gaussianity of the data. The results obtained in this paper, in particular, imply that the generalization error obtained in their work is universal, in the sense that they can be immediately extended to a much broader set of target distributions, which we elaborate on in much greater detail in the main body of the paper.

Gerace et al. [2022] consider the problem of binary classification via a two-layer neural network in a synthetic setting. The source data and the source labels are sampled according to x = 𝑅𝑒𝑙𝑢 ( Ga ) and 𝑦 = 𝑆𝑖𝑔𝑛 ( a 𝑇 z ) respectively where G , a and z are i.i.d. Gaussian. The target data and labels are generated by perturbing G and a and then applying the same rules as for the source data. Gerace et al. [2022] trained the network on the source dataset and then proceeded by only fine-tuning the

last layer on the target domain using the cross-entropy loss with an ℓ 2 regularizer. The analysis was conducted using the Replica method. The authors of Gerace et al. [2022] compared their results with an equivalent random feature model and observed empirically a certain kind of Gaussian universality for real-world datasets, such as MNIST.

Extension of results obtained under Gaussianity assumptions to non-Gaussian distributions remains a pertinent research direction in which the idea of Gaussian universality plays a central role. Montanari and Nguyen [2017] proved the universality of the generalization error for the elastic net, assuming the design matrix A is i.i.d subgaussian. Panahi and Hassibi [2017] generalized the result to the problem of regularized regression with a quadratic loss and a convex separable regularizer which is either 𝑓 (·) = ∥ · ∥ 1 (LASSO) or strongly convex. Han and Shen [2023] generalized Panahi and Hassibi [2017]'s results to non-separable Lipschitz test functions and provided non-asymptotic bounds for the concentration of solutions. In the random geometry literature, Oymak and Tropp [2018] showed universality of the embedding dimension of randomized dimension reduction linear maps with i.i.d entries satisfying certain moment conditions. Abbasi et al. [2019] proved universality of the recovery threshold for minimizing strongly convex functions under linear measurements constraint, assuming the rows are iid and the norm of the mean is asymptotically negligible compared to the noise. Lahiry and Sur [2023], proved the universality of generalization error for ridge regression and LASSO when the rows are distributed iid with a specific block structure dependence per row, AD where A has zero mean iid random subgaussian vectors per row and D being diagonal.

Leveraging universality is not limited to the signal recovery literature. As an example, Hu and Lu [2022], Bosch et al. [2023], Schröder et al. [2023], have analyzed the performance of the random feature models by replacing nonlinear functions of Gaussians with the corresponding Gaussian distribution having the same mean and covariance in single-layer and multiple-layer scenarios, respectively, through a universality argument, referred to as the Gaussian equivalence principle. In a more general setting, under subgaussianity assumption on the data, Montanari and Saeed [2022] proved the universality of the training error for general loss and regularizer and test error when the regularizer is strongly convex and the loss is convex. Hastie et al. [2022] proved universality for the the minimal ℓ 2 -norm linear interpolators for the data generated via a very specific rule x = 𝜎 ( W 𝚺 1 2 z ) , where z is i.i.d zero mean and satisfies several other technical properties and 𝚺 is a deterministic PSD matrix. Dandi et al. [2024] considered the universality of mixture distributions in a similar setting to Montanari and Saeed [2022] and proved the universality of the free energy of the test error.

1.2 Outline of the Paper

In Section 2.1, we formally define the optimization problem for which we aim to establish universality. Section 2.2 outlines our primary contributions. In Section 3, we present our main universality result, accompanied by several insightful remarks. Section 4 focuses on our findings related to fine-tuning in the contexts of linear regression and binary classification. In Section 5, we validate our theoretical results through empirical experiments. Finally, we conclude with a summary and discussion in Section 6.

2 Problem Formulation

2.1 The Setting

We use bold letters for vectors and matrices. We denote the 𝑖 'th largest singular value of matrix A by 𝑠 𝑖 ( A ) . We consider the proportional regime where 𝑑 = Θ ( 𝑛 ) and 𝑑 𝑛 → 𝜅 > 1 . We refer to Section A.1 for the other notations and definitions necessary for the rest of this exposition.

Given a training dataset {( x 𝑖 , 𝑦 𝑖 )} 𝑛 𝑖 = 1 and a model with a fixed architecture, the conventional approach of learning the weights for the model consists of choosing a loss function L and finding 𝑤 minimizing 1 𝑛 ˝ 𝑛 𝑖 = 1 L( x 𝑖 , w 𝑖 , 𝑦 𝑖 ) . In this exposition we focus on linear models and loss functions of the form L( x 𝑖 , w 𝑖 , 𝑦 𝑖 ) = ℓ ( 𝑦 𝑖 -x 𝑇 𝑖 w 𝑖 ) , where ℓ is differentiable and ℓ ( 0 ) = 0 . Azizan and Hassibi [2018] characterized behavior of a broad family of optimizers, called stochastic mirror descent (SMD) algorithms, for linear models in the over-parametrized regime. For a strictly convex function 𝑔 : R 𝑑 → R , the update rule of the SMD with a mirror 𝑔 and a learning rate 𝜂 > 0 is defined as

where 𝜂 > 0 is the learning rate and L 𝑡 is the loss function L evaluated at a point chosen at random in the dataset corresponding to the 𝑡 'th iteration. Due to strict convexity, ∇ 𝑔 (·) defines an invertible transformation, which is why (2) is indeed a well-defined update rule. Note also that this includes SGD as a special case, which corresponds to taking 𝑔 (·) = ∥ · ∥ 2 2 . Azizan and Hassibi [2018] show that applying SMD initialized at w 0 to minimize 1 𝑛 ˝ 𝑛 𝑖 = 1 ℓ ( 𝑦 𝑖 -x 𝑇 𝑖 w ) yields a weight vector defined by the following optimization problem:

We construct the random matrix A ∈ R 𝑛 × 𝑑 by setting its rows equal to x 𝑖 . Hence, stating in a more general fashion, we are interested in the analysis of the following optimization problem, with the main case of interest being when 𝑓 is a quadratic:

As the objective is comprised of minimizing a strongly convex function 𝑓 over a closed convex set, it has a unique minimizer. By using a Lagrange multiplier 𝜆 ∈ R , this convex optimization problem can be written in the unconstrained form:

Considering the objective without the sup 𝜆> 0 yields the following regularized linear regression problem for which we will also establish a universality theorem:

Note that (4) captures the case of explicit regularization, which is of highest importance whenever there is a high amount of noise or label corruption present. Furthermore, by deliberately choosing the regularizer 𝑓 , it is possible to obtain solutions that exhibit certain behavior, viz being sparse or compressed. In order to tackle optimizations such as 3, 4, we prove their equivalence to a problem with a suitable Gaussian design G which is described in Definition 4 in Section A.1. Our approach for proving universality will be the Lindeberg approach Lindeberg [1922] Chatterjee [2006].

2.2 Our Contributions

2.2.1 Transfer Learning

Linear Regression We extend the results from Dar et al. [2021] by providing precise expressions for the generalization error in linear regression tasks. In addition, we demonstrate that the test error is always lower-bounded by a quantity that is attained only when the covariance matrix of the data is scalar. Furthermore, we identify specific conditions under which fine-tuning is successful.

Binary Classification We present the first precise characterization of the classification error for linear models trained using stochastic gradient descent (SGD) on data drawn from a general mixture distribution with arbitrary covariance matrices. Moreover, we delineate the regimes in which finetuning proves effective.

2.2.2 Universality

Proof of universality for implicit regularization. In the context of SGD and, more generally, SMD (see 2) with a mirror 𝑓 satisfying the assumptions of Theorem 1, we prove universality of the generalization error as well as of the value of the corresponding implicit regularization objective for a wide range of data distributions characterized by Assumptions 1. Note that this cannot be reduced to any known results on universality of constrained objectives as the latter assume that the constraints are deterministic, i.e the optimization variables belong to some determinitic set C chatacterized by the constraints. While to analyze SMD one has to deal with constraints of the form Aw = y which represents a random polytope. To the best of our knowledge, the only other paper dealing with universality in the context of implicit regularization is Hastie et al. [2022]. They study minimal

ℓ 2 -norm linear interpolators, but they work with a very specific (random feature) model for data distributions defined by x = 𝜎 ( W 𝚺 1 2 z ) , where z is i.i.d zero mean and W is i.i.d. Gaussian. Our paper generalizes theirs in two non-trivial ways, as it allows for arbitrary smooth convex strongly convex mirrors 𝑓 ( w ) as well as more general data distributions.

Relaxing assumptions for explicit regularization. To the best of the authors' knowledge, all previous results on universality assumed either that the data points x ∈ R 𝑑 have i.i.d. entries or that the rows are independent and x is subgaussian. We relax these assumptions. That is, we show that, for the quadratic loss function and any strongly convex not necessarily differentiable regularizer universality holds as long as the rows are i.i.d., all moments of the x up to the 6th satisfy

Extending universality to mixtures of distributions. Since the main motivation behind this work is the study of generalization in classification tasks, we focus on data matrices sampled from mixture distributions. Note that most previous papers on universality, such as Montanari and Saeed [2022], do not apply to this setting directly. As an illustration to why results of Montanari and Saeed [2022] cannot just be applied to the mixture distributions directly, consider P = 1 2 N( µ , I ) + 1 2 N(µ , I ) corresponding to a mixture of two classes with antipodal means and the resulting classification error depends on ∥ µ ∥ , while P has mean 0 and covariance I meaning that the matching gaussian distribution N( 0 , I ) does not contain any information about the classes.

However, there is one prior paper Dandi et al. [2024] that studies universality specifically in the context of mixture distributions. It is worth mentioning that their definition of universality is different. Namely, they prove universality of the expectation with respect to the Gibbs distribution, which suffices to show universality of the train but not of the test error.

Allowing for non-vanishing means . In Definition 2, we assume only that E a ∥ a -µ ∥ 2 2 = 𝑂 ( 1 ) , whereas in Abbasi et al. [2019], Montanari and Saeed [2022] and Dandi et al. [2024] they have

∥ µ ∥ 2 2 E a ∥ a -µ ∥ 2 2 → 0 , implying that the norms of the means are negligible compared to the norms of the data points on average. To our knowledge, our work is the first to tackle it.

3 Main Result

In this section, we present our main universality theorem and before doing so, we list the required technical assumptions. Consider the following convex optimization problem

Denote the solution to the optimization problem 5 above by w Φ 𝜆 ( A ) . Then we assume the following:

Assumptions 1. 1. A is a regular block matrix and G is its matching Gaussian matrix as in Section A.1.

The regularizer 𝑓 (·) is regular also as Section A.1 or there exists a sequence of regular 𝑓 𝑚 's converging uniformly to 𝑓 .

𝑓 (·) is 𝑀 -strongly convex.

E y ∥ y ∥ 2 2 = 𝑂 ( 𝑑 ) with E | 𝑦 𝑗 | 𝑞 = 𝑂 ( 1 ) for 𝑗 ∈ [ 𝑛 ] and 𝑞 ∈ [ 6 ] and is independent of A .

𝑠 min ( AA 𝑇 ) = Ω ( 1 ) with high probability,

𝑉𝑎𝑟 (∥ y ∥ 2 2 ) → 0

∥∇ 𝑓 ( w Φ 𝜆 ( A ) )∥ 2 ≤ 𝐾 𝑓 √ 𝑛 for a constant 𝐾 𝑓 independent of 𝜆 .

Theorem 1. If Ψ = 𝚽 = sup 𝜆> 0 Φ 𝜆 , suppose parts (1) - (7) of Assumptions 1 hold and for Ψ = Φ 𝜆 , only assume parts (1) - (4) of Assumptions 1. Then

(Universality of the training error) For any L-Lipschitz 𝑔 : R 𝑑 → R , and every 𝑡 1 > 0 , 𝑡 2 > 𝑡 1 , 𝑐 ∈ R the following holds:

If P (| 𝑔 ( Ψ ( G )) -𝑐 | > 𝑡 1 ) → 0 then P (| 𝑔 ( Ψ ( A )) -𝑐 | > 𝑡 2 ) → 0 .

Furthermore, if 𝑔 has bounded second derivative, then

(Universality of the solution) For a function 𝜓 with ∇ 2 𝜓 ⪯ 𝑞 I if either 𝜓 is regular (Definition 1) or there is a sequence of regular functions converging uniformly to 𝜓 ,

If for every 𝑡 > 0 , P (| Ψ ( G ) -𝑐 | > 𝑡 ) → 0 then for B ∈ { A , G } , there exists 𝑐 1 ∈ R such that for any 𝑡 1 > 0 we have P (| 𝜓 ( w Ψ ( B ) ) -𝑐 1 | > 𝑡 1 ) → 0 .

Furthermore, if 𝜓 is bounded

To sum up, Theorem 1 says that the vector of weights wA trained on data sampled from a non-gaussian distribution A either via SMD with a mirror 𝑓 or by minimizing the least squares objective with regularizer 𝑓 , shares many similar characteristics with the vector of weights wG trained on data sampled from the matching GMM G via the same optimization procedure under certain technical assumptions on A and 𝑓 . By "similar characteristics", we mean that for any 𝜓 : R 𝑑 → R with bounded Hessian, 𝜓 ( wA ) = 𝜓 ( wG ) holds in the limit.

We would like to point out a few remarks.

Remark 1. Note that 𝜓 need not be convex, in fact any linear combination of regular 𝜓 is also regular. An immediate result is the universality of the empirical distribution of the coordinates of solutions. √

Remark 2. The assumption ∥∇ 𝑓 ( w Φ 𝜆 ( A ) )∥ 2 = 𝑂 ( 𝑛 ) in Theorem 1 is satisfied for any function 𝑓 with a locally lipschitz gradient, i.e ∥∇ 𝑓 ( u )∥ 2 ≤ 𝐾 ( 1 + ∥ u ∥ 2 ) . In particular it applies to 𝑓 (·) = ∥ · ∥ 2 2 .

Remark 3. In both parts of Theorem 1, the regularizer is allowed to be an 𝑀 -strongly convex uniform limit of differentiable convex functions, which implies that 𝑓 ( w ) = 𝑡 ∥ w ∥ 1 + 𝑀 ∥ w ∥ 2 2 , 𝑡 > 0 , 𝑀 > 0 , known as the elastic net, is also included in our results.

Remark 4. In Theorem 1, the optimal value of Φ 𝜆 is only achieved when 𝜆 →∞ and is not attained at any finite 𝜆 . This means that proving the results for 𝚿 = Φ in Theorem 1 requires additional ideas apart from the case Ψ = Φ 𝜆 in Theorem 1, as the latter makes use of the boundedness of 𝜆 extensively. To tackle this issue, we present a uniform convergence result in 𝜆 which might be of independent interest.

Remark 5. The condition 𝑠 min ( AA 𝑇 ) = Ω ( 1 ) can be satisfied for a variety of random matrix models. If for each block of A , A 𝑖 = ˜ A 𝑖 𝚺 1 / 2 𝑖 where ˜ A 𝑖 has iid entries and 𝑠 min ( 𝚺 𝑖 ) = Ω ( 1 ) , then by the Bai-Yin's law Bai and Yin [2008] the condition is satisfied. The second family is comprised of blocks with independent and identical rows where the norm of rows have exponentrial concentration. We prove this in the Appendix. One particular instance will be the distributions satsifying LCP from Definition 3.

Remark 6. The second assumption in Assumptions 1 is not too restrictive and is in particular satisfied for 𝜓 ( w ) = P ( a 𝑇 w > 𝑐 ) . We have with high probability for Ψ ∈ { Φ 𝜆 , Φ }

For a independent of A , G but sampled from the same distribution as the rows of A , respectively. Note that by this argument we have reduced the problem of verifying CLT for a 𝑇 w Ψ ( A ) to its Gaussian counterpart, a 𝑇 w Ψ ( G ) . Then, specifically for the applications presented in this paper, we provide a proof that a 𝑇 w Ψ ( G ) satisfies CLT w.r.t. to randomness in a with high probability in G . However, in general the latter CLT condition has to be verified on a case by case basis.

4 Applications: Transfer Learning

4.1 Regression

We consider the classical problem of recovering the best linear regressor for the following linear model

Here, w ∗ is the ground truth, the rows of the data matrix X are i.i.d with E x 𝑖 = 0 and E x 𝑖 x 𝑇 𝑖 = : R 𝑥 , z is centered and is independent of X and satisfies E z ∥ z ∥ 2 2 = 𝜎 2 𝑛 . For a given ˆ w , its generalization error is defined by:

Now in order to recover w ∗ given observations ( y , X ) , we choose to optimize the least squares objective, min w ∥ y -Xw ∥ 2 2 , and to do so, we leverage SGD. We would like to investigate how useful having a pretrained classifier w 0 can be for the recovery of w ∗ . As pointed out earlier, SGD initialized from w = w 0 , by its implicit regularization property, converges to the solution ˆ w of (1). We denote 𝑒 𝑎 : = 𝑒 𝑔𝑒𝑛 ( w 0 ) ; hence the name "a priori error", 𝑒 𝑎 . We provide the assumptions necessary for our results on linear regression in Section A.2. In what follows we give a precise characterization of the posterior error, 𝑒 𝑝 , of the model after training with SGD, in terms of 𝑒 𝑎 and the other parameters of the problem. In order to leverage Theorem 1 to establish the universality of the generalization error, it is sufficient to apply a change of variable w ′ : = R -1 / 2 𝑥 w and use 𝜓 ( w ) = ∥ w ∥ 2 2 as the test function.

Theorem 2. Under Assumptions 3 in Section A.2, the generalization error of the SGD solution initialized from w 0 converges in probability to

With 𝑡 = ∫ 4 𝑝 ( 𝑟 ) ( 2 + 𝑟 𝜃 ) 2 𝑑𝑟 where 𝜃 is found through 𝜅 -1 𝜅 = 1 𝜃 𝑆 𝑝 (-1 𝜃 ) (12). And 𝜅 > 1 is the proportional constant, i.e for X ∈ R 𝑛 × 𝑑 , 𝑑 𝑛 → 𝜅 . Moreover, for any distribution 𝑝 ( 𝑟 ) we have

The lower bound is attained if and only if 𝑝 ( 𝑟 ) = 𝛿 ( 𝑟 -𝑟 0 ) for some 𝑟 0 > 0

The following remark is immediate:

Remark 7. Theorem 2 entails that, depending on the noise level present in the data, training could be even potentially harmful. Indeed, if 𝜎 2 ≥ 𝑒 𝑎 , then 𝑒 𝑝 ≥ 1 𝜅 -1 𝜎 2 + 𝜅 -1 𝜅 𝑒 𝑎 ≥ 𝜎 ( 2 √ 𝑒 𝑎 -𝜎 ) ≥ 𝑒 𝑎 for any 𝜅 > 1 and it is therefore more appropriate to use vector w 0 instead of performing fine-tuning. Moreover, if the covariance Rx is scalar, then the converse is true. Namely, if 𝑒 𝑎 ≥ 𝜎 2 , then the best achievable error corresponds to 𝜅 ∗ = √ 𝑒 𝑎 √ 𝑒 𝑎 -𝜎 and equals 𝜎 ( 2 √ 𝑒 𝑎 -𝜎 ) , which is less than 𝑒 𝑎 . Therefore, transfer learning contributes to improving the test performance when the variance of the noise is not too high, but the model has to be fine-tuned on the correct amount of target data.

4.2 Classification

4.2.1 Problem setting

Consider a binary classification task and let X stand for the data matrix and y denote the vector of labels where each 𝑦 𝑖 = ± 1 depending on what class the 𝑖 -th point x 𝑖 falls into. We denote µ ℓ : = E x 𝑖 and 𝚺 ℓ : = E x 𝑖 x 𝑇 𝑖 -µ ℓ µ 𝑇 ℓ , for the mean and covariance of each datapoint x 𝑖 , respectively and ℓ ∈ { 1 , 2 } depending on the class of x 𝑖 . After learning a linear classifier w , we assign labels to new previously unseen points according to y 𝑛𝑒𝑤 = 𝑠𝑖𝑔𝑛 ( w 𝑇 x 𝑛𝑒𝑤 ) . Without loss of generality we will assume that the first 𝑛 2 rows of X are sampled from the first class and the remaining rows are sampled from the second as we can permute the rows otherwise. Modulo such permutation, it is straightforward to see that X satisfies parts (1) - (3) of Definition 2. Since the topic of the present paper is fine-tuning, we assume that the following steps are performed:

Obtain a pre-trained classifier w 0

Renormalize w 0 obtained during the previous step using the target data via

This yields:

After finding 𝛼 take w ′ 0 : = 𝛼 w 0 . This transform preserves the direction of w 0 , while setting its squared magnitude to 𝛼 2 ∥ w 0 ∥ 2 = ( y 𝑇 Xw 0 ) 2 ∥ w 0 ∥ 2 ∥ Xw 0 ∥ 4 , which depends only on the direction of w 0 but not on its magnitude anymore. We find applying this transform very meaningful, as it does not change the classification error for the source data but simplifies learning for the regression problem defined by the target data.

Learn the final classifier from the target data X ∈ R 𝑛 × 𝑑 and labels y ∈ R 𝑛 using SGD initialized at w ′ 0 , obtained from the previous step.

We will use the assumptions in Section A.3 to define further details of the classification task we consider. Note that in practice the main difficulty of the classification task in the over-parametrized regime ( 𝑛 < 𝑑 ) arises due to the fact that µ 1 , µ 2 , 𝚺 1 and 𝚺 2 are not known and cannot be estimated reliably. Nevertheless, we would like to start with characterizing the optimal performance in the scenario where these are provided to us by an oracle. Lemma 1 in Section A.3 provides such a characterization under certain symmetry assumptions. In view of Lemma 1 it is natural to introduce the following assumption:

Assumptions 2. The initialization point w 0 satisfies

where w ∗ = ( 𝚺 1 + 𝚺 2 ) -1 ( µ 1 -µ 2 ) is the optimal classifier defined as in Lemma 1. η is a noise vector with ∥ η ∥ 2 = 1 -𝑟 and for any deterministic matrix C of bounded operator norm it holds that η 𝑇 C η converges in probability to Tr ( C ) ( 1 -𝑟 ) 𝑑 . Note that the smaller the ratio 𝑡 𝜂 𝑡 ∗ is, the better performance w 0 has.

Indeed, 10 captures the closeness of the initilization point w 0 to the optimal classifier w ∗ . For example, in the isotropic case 𝚺 1 = 𝚺 2 = 𝜎 2 I , if w 0 has the classification error 𝑒 𝑎 , this means that we should take 𝑡 𝜂 𝑡 ∗ = 𝑑 2 𝜎 2 √︃ 𝑑 ( 1 -𝑟 ) 𝜎 2 𝑄 -1 ( 𝑒 𝑎 ) 2 -2 . Finally, we would like to remark that, under notation and assumptions from Theorem 1, the following corollary is implied by Theorem 1 modulo Theorem 4 presented in the Appendix along with the explanation of the implication:

Corollary 1 (Universality of the classification error for SGD and ridge regression objectives) . If 𝑓 ( w ) = ∥ w -w 0 ∥ 2 2 , when ∥ w 0 ∥ 2 2 = 𝑂 ( 𝑑 ) then for a , g independent of A , G but sampled from the same distribution as their rows, respectively, we have with high probability for Ψ ∈ { Φ 𝜆 , Φ }

4.2.2 Analysis of the Classification Error

Theorem 3. Let w 0 be an initialization point that satisfies Assumption 2 and X be a data matrix satisfying Assumptions 4 in Section A.3. Then the classification error of the SGD solution initialized at 𝛼 w 0 for 𝛼 defined by (9) and trained on X , y is given by 𝑒 𝑝 = 𝑄 -𝛾 + 2 2 √︃ 𝜏 2 -𝛾 2 4 ! where 𝛾 and 𝜏 are √ √

determined in terms of a 𝜃 solving 2 2 𝜃 √ 𝑛 𝑆 𝚺 1 + 𝚺 2 (-2 2 𝜃 √ 𝑛 ) = 1 -1 𝜅 whose expressions are provided in the Appendix (cf. Section I, equation 40) with 𝑑 𝑛 → 𝜅 > 1 .

The expressions from Theorem 3 can be simplified drastically in the case of scalar covariance matrices.

Corollary 2. Under the notation from Theorem 3, assume Σ 1 = Σ 2 = 𝜎 2 I 𝑑 𝑑 and define 𝜌 : = 𝑑 ( 1 -𝑟 ) 𝜎 2 . Let 𝑒 𝑎 be the classification error of w 0 initialized according to 10. Then

We arrive at the following conclusion summarizing the derivations above.

tuning always succeeds. It is observed that for 𝑒 𝑎 > 𝑄 ( 1 ) , the worse 𝑒 𝑎 is, the better 𝑒 𝑝

If 𝜌 = Θ ( 1 ) , then the fine-tuning step may or may not succeed depending on whether 𝑒 𝑝 ( 𝜅, 𝜌 ) < 𝑒 𝑎 or not. See Section 5.2 for further (empirical) explorations on the usefulness of the fine-tuning of the pre-trained solution for this regime.

If 𝜌 = 𝑜 ( 1 ) , then the classification error of any linear classifier goes to 1 2 as 𝑛 →∞ since it is lower bounded by 𝑄 ( √︃ 𝜌 2 ) according to Lemma 1. Thus, any kind of learning will fail in this regime.

An interesting regime is where the number of samples is much lower than the number of parameters, which naturally rises in the context of fine-tuning large models and corresponds to letting 𝜅 ≫ 1 . For this regime, we have

We observe that if 𝜌 < 1 , transfer learning always fails independent of value of 𝑒 𝑎 for 𝜅 ≫ 1 and training would not help with improving performance. On the other hand, if 𝑒 𝑎 > 𝑄 ( √︃ 𝜌 -1 3 ) > 𝑄 ( √︃ 𝜌 2 ) , for large enough 𝜅 , transfer learning always achieves an error less than 𝑒 𝑎 .

5 Numerical experiments

5.1 Regression

To corroborate our findings, we plotted the generalization error of the weight obtained through running SGD according to the Assumptions 3 in Section 4.1 with respect to 𝜅 = 𝑑 𝑛 . To do so, we fixed 𝑑 = 1000 and varied 𝑛 across different values. We used CVXPY (Grant and Boyd [2014], Agrawal et al. [2018]) to solve (1) efficiently on a Laptop CPU. To verify the universality of our results, we initially constructed a centered random matrix X ′ with i.i.d components according to the distributions N( 0 , 1 ) , 𝐵𝑒𝑟 ( 0 . 5 ) , and 𝜒 2 ( 1 ) and using a correlation matrix R 𝑥 we defined X : = X ′ R 1 / 2 𝑥 . On the other hand, we generated R 𝑥 according to the following three distributions: single level 𝑝 ( 𝑟 ) : = 𝛿 ( 𝑟 -1 ) , bilevel 𝑝 ( 𝑟 ) : = 0 . 3 𝛿 ( 𝑟 -1 ) + 0 . 7 𝛿 ( 𝑟 -5 ) and uniform on the interval [ 1 , 5 ] . We specifically consider these cases as they are common in the literature and we use the parameter 𝜎 2 for the component-wise variance of z in 6. Additionally, w 0 is chosen according to Assumptions 3 in such a manner that 𝑒 𝑎 = 1 . In both Figures 1, 2 the blue line represents the prediction 7 made by Theorem 2, the red line depicts the lower bound 8. The markers showcase the performance of weights obtained under different distributions as described earlier. It can be seen that from Figures 1, 2, for the bilevel and uniform distributions, depending on the value of 𝜎 , transfer learning might not be beneficial as discussed in Remark 7. In particular, in Figures 1c and 2c, the generalization error is always lower bounded by 𝑒 𝑎 = 1 and only in 𝜅 →∞ can get close to 1. Finally, the single-level distribution on R 𝑥 is always a lower bound for the generalization error of various distributions on R 𝑥 .

5.2 Classification

Similar to the preceding subsection, we experimented with sampling the entries of X independently from three different centered distributions: normal, Bernoulli, and 𝜒 2 . We also sampled the means µ 1 and µ 2 from N( 0 , 1 𝑑 I 𝑑 ) with a cross-correlation 𝑟 = E [ µ 1 𝑖 µ 2 𝑖 ] = 0 . 9 and added them to the corresponding rows of X . For Figures 3, we fixed 𝜌 = 0 . 8 , 2 , 5 respectively and plotted the classification error predicted by Corollary 2 as a solid red line, empirically observed classification errors for the normal, Bernoulli and 𝜒 2 entries as black squares, green circles and red triangles respectively. The blue lines depict the classification error at the initialization. It can be seen that there is a close match between the empirical errors between points from different distributions as well as with the theoretical prediction, thus validating both Theorem 1 and Theorem 3. Note that fixing 𝜌

in this setting corresponds to fixing 𝜎 2 as 𝜌 = 𝑑 ( 1 -𝑟 ) 𝜎 2 . It is also worth mentioning that fine-tuning improves performance in the setting of Figure 3c but hurts it for Figure 3a. Moreover, note that in Figure 3b, although for smaller 𝜅 transfer learning hurts, past a certain 𝜅 , training improves the performance. Also we observe that by increasing 𝜌 across the three plots, the classification task becomes easier and fine-tuning improves performance as supported by Remark 8.

2

4

6

8

10

0

0

.

2

0

.

4

0

.

6

0

.

8

1

𝜅

Generalization Error

N

𝐵

𝜒

2

𝑒

𝑝

Lower Bound

(a)

𝜎

=

0

.

01

2

4

6

8

10

0

.

4

0

.

6

0

.

8

1

𝜅

Generalization Error

N

𝐵

𝜒

2

𝑒

𝑝

Lower Bound

(b)

𝜎

=

0

.

15

2

4

6

8

10

0

50

100

150

𝜅

Generalization Error

N

𝐵

𝜒

2

(

1

)

𝑒

𝑝

Lower Bound

(c)

𝜎

=

2

2

4

6

8

10

0

0

.

2

0

.

4

0

.

6

0

.

8

1

𝜅

Generalization Error

N

𝐵

𝜒

2

𝑒

𝑝

Lower Bound

(a)

𝜎

=

0

.

01

2

4

6

8

10

0

.

4

0

.

6

0

.

8

𝜅

Generalization Error

N

𝐵

𝜒

2

𝑒

𝑝

Lower Bound

(b)

𝜎

=

0

.

15

2

4

6

8

10

0

50

100

𝜅

Generalization Error

N

𝐵

𝜒

2

𝑒

𝑝

Lower Bound

(c)

𝜎

=

2

0

5

10

15

20

0

.

3

0

.

35

0

.

4

0

.

45

𝜅

Classification Error

N

𝐵

𝜒

2

𝑒

𝑝

(a)

𝜌

=

0

.

8

, 𝑟

=

0

.

9

, 𝑒

𝑎

=

0

.

3

0

5

10

15

20

0

.

3

0

.

32

0

.

34

0

.

36

0

.

38

𝜅

Classification Error

𝑒

𝑎

N

𝐵

𝜒

2

𝑒

𝑝

(b)

𝜌

=

2

, 𝑟

=

0

.

9

, 𝑒

𝑎

=

0

.

3

0

5

10

15

20

0

.

2

0

.

22

0

.

24

0

.

26

0

.

28

0

.

3

𝜅

Classification Error

N

𝐵

𝜒

2

𝑒

𝑝

(c)

𝜌

=

5

, 𝑟

=

0

.

9

, 𝑒

𝑎

=

0

.

3

6 Conclusion and future work

We presented a novel Gaussian universality result and used it to study the problem of transfer learning in linear models, for both regression and binary classification. In particular, we were able to precisely relate the performance of the pretrained model to that of the fine-tuned model trained via SGD and, as a result, identified situations where transfer learning helps and where it or does not. Possible future directions include investigating other problems where the universality result may be useful, extending the results to potential functions that are not necessarily convex nor separable, as well as exploring the implications of universality for objectives with explicit regularization.

References

Abbasi, F. Salehi, and B. Hassibi. Universality in learning from linear measurements. Advances in Neural Information Processing Systems , 32, 2019.

Agrawal, R. Verschueren, S. Diamond, and S. Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Akhtiamov, D. Bosch, R. Ghane, K. N. Varma, and B. Hassibi. A novel gaussian min-max theorem and its applications. arXiv preprint arXiv:2402.07356 , 2024.

Azizan and B. Hassibi. Stochastic gradient/mirror descent: Minimax optimality and implicit regularization. In International Conference on Learning Representations , 2018.

Z.-D. Bai and Y.-Q. Yin. Limit of the smallest eigenvalue of a large dimensional sample covariance matrix. In Advances In Statistics , pages 108-127. World Scientific, 2008.

G. Bobkov. On concentration of distributions of random weighted sums. Annals of probability , pages 195-215, 2003.

Bosch, A. Panahi, and B. Hassibi. Precise asymptotic analysis of deep random feature models. In The Thirty Sixth Annual Conference on Learning Theory , pages 4132-4179. PMLR, 2023.

Bozinovski. Reminder of the first paper on transfer learning in neural networks, 1976. Informatica , 44(3), 2020.

Chatterjee. A generalization of the lindeberg principle. 2006.

Dai, Q. Yang, G.-R. Xue, and Y. Yu. Boosting for transfer learning. In Proceedings of the 24th international conference on Machine learning , pages 193-200, 2007.

Dandi, L. Stephan, F. Krzakala, B. Loureiro, and L. Zdeborová. Universality laws for gaussian mixtures in generalized linear models. Advances in Neural Information Processing Systems , 36, 2024.

Dar, D. LeJeune, and R. G. Baraniuk. The common intuition to transfer learning can win or lose: Case studies for linear regression. arXiv preprint arXiv:2103.05621 , 2021.

Gerace, L. Saglietti, S. S. Mannelli, A. Saxe, and L. Zdeborová. Probing transfer learning with a model of synthetic correlated datasets. Machine Learning: Science and Technology , 3(1):015030, 2022.

Grant and S. Boyd. CVX: Matlab software for disciplined convex programming, version 2.1, Mar. 2014.

Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1832-1841. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/ v80/gunasekar18a.html .

Han and Y. Shen. Universality of regularized regression estimators in high dimensions. The Annals of Statistics , 51(4):1799-1823, 2023.

Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics , 50(2):949, 2022.

Hu and Y. M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69(3):1932-1964, 2022.

Jain, A. Montanari, and E. Sasoglu. Scaling laws for learning with real and surrogate data. arXiv preprint arXiv:2402.04376 , 2024.

Lahiry and P. Sur. Universality in block dependent linear models with applications to nonparametric regression. arXiv preprint arXiv:2401.00344 , 2023.

Liese and K.-J. Miescke. Statistical decision theory. In Statistical Decision Theory: Estimation, Testing, and Selection , pages 1-52. Springer, 2008.

W. Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung. Mathematische Zeitschrift , 15(1):211-225, 1922.

Montanari and P.-M. Nguyen. Universality of the elastic net error. In 2017 IEEE International Symposium on Information Theory (ISIT) , pages 2338-2342. IEEE, 2017.

Montanari and B. N. Saeed. Universality of empirical risk minimization. In Conference on Learning Theory , pages 4310-4312. PMLR, 2022.

Oymak and J. A. Tropp. Universality laws for randomized dimension reduction, with applications. Information and Inference: A Journal of the IMA , 7(3):337-446, 2018.

Panahi and B. Hassibi. A universal analysis of large-scale regularized least squares solutions. Advances in Neural Information Processing Systems , 30, 2017.

Schröder, H. Cui, D. Dmitriev, and B. Loureiro. Deterministic equivalent and error universality of deep random features learning. In International Conference on Machine Learning , pages 30285-30320. PMLR, 2023.

E. A. Seddik, C. Louart, M. Tamaazousti, and R. Couillet. Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning , pages 8573-8582. PMLR, 2020.

Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu. A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning-ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27 , pages 270-279. Springer, 2018.

Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. A comprehensive survey on transfer learning. Proceedings of the IEEE , 109(1):43-76, 2020.

A Notations, Definitions and Assumptions

A.1 Notations and Definitions

We call a convex function 𝑓 : R 𝑑 → R separable if 𝑓 ( w ) = ˝ 𝑑 𝑖 = 1 𝑓 𝑖 ( 𝑤 𝑖 ) , where 𝑓 𝑖 : R → R are convex. Examples of such functions include ∥ · ∥ 𝑝 𝑝 for every 𝑝 ≥ 1 . A function 𝑓 : R 𝑑 → R is 𝑀 -strongly convex for 𝑀 > 0 if for every w ∈ R 𝑑 , 𝑓 ( w ) -𝑀 ∥ w ∥ 2 2 is convex.

Functions satisfying the following definition will be instrumental in the statements of our results: Definition 1. We call a function 𝑓 : R 𝑑 → R regular if it satisfies the following conditions

𝑓 is convex with 𝑓 ( 0 ) = 𝑂 ( 𝑑 ) .

𝑓 is three times differentiable.

𝑓 satisfies the following third-order condition for some 𝐶 𝑓 > 0 :

Note that separable functions with bounded third order derivatives satisfy the third assumption of Definition 1. For the description of the main results on regression and classification we recall several probability theory concepts.

Given a real-valued measure 𝜇 , its Stieltjes transform is defined as

With H + = { 𝑧 : 𝐼𝑚 ( 𝑧 ) > 0 } . Moreover, we denote convergence in probability and weak convergence for measures by P - → and ⇝ respectively. For a random variable 𝑋 , we sometimes use 𝑉𝑎𝑟 ( 𝑋 ) for the variance of 𝑋 . For a convex differentiable function 𝑔 , the Bregman divergence is defined by

Next, we will provide a description of the design matrices investigated in this paper. It will be clear soon that these assumptions are often satisfied in practice.

Definition 2. 2 We call a random matrix A ∈ R 𝑛 × 𝑑 block-regular if it satisfies the following properties:

A 𝑇 = GLYPH<0> A 𝑇 1 A 𝑇 2 . . . A 𝑇 𝑘 GLYPH<1> with 𝑘 being finite and for each A 𝑖 ∈ R 𝑛 𝑖 × 𝑑 , 𝑛 𝑖 = Θ ( 𝑛 )

For 1 ≤ 𝑖 ≤ 𝑘 , the rows of A 𝑖 are distributed independently and identically.

E A 𝑖 = : 1 µ 𝑇 𝑖

Let a be any row of A and µ be its mean. Then for any deterministic vector v ∈ R 𝑑 , and 𝑞 ∈ N , 𝑞 ≤ 6 , there exists a constant 𝐾 > 0 such that E a | ( a -µ ) 𝑇 v | 𝑞 ≤ 𝐾 ∥ v ∥ 𝑞 2 𝑑 𝑞 / 2

For any deterministic matrix C ∈ R 𝑑 × 𝑑 of bounded operator norm we have 𝑉𝑎𝑟 ( a 𝑇 Ca ) → 0 as 𝑑 →∞ .

Each ∥ µ 𝑖 ∥ 2 2 = 𝑂 ( 1 )

Note that assumptions 1-3 from Definition 2 are satisfied for the design matrix both in the classification and regression scenarios, and the assumption ∥ µ 𝑖 ∥ 2 2 = 𝑂 ( 1 ) is just a matter of normalization. While assumptions 4 and 5 might appear obscure at first, any a ∈ R 𝑑 satisfying the Lipschitz Concentration Property (LCP) also satisfies the aforementioned assumptions. Recall that the LCP is defined as follows.

Definition 3. We say that a distribution a ∈ R 𝑑 satisfies LCP if the following inequality holds for any Lipschitz function 𝜙 : R 𝑑 → R

.

It is also worth mentioning that, as shown in Seddik et al. [2020], any data distribution from which one can sample using a GAN satisfies Definition 3. Since it is known that many natural datasets can be approximated well by GAN-generated data in practice, we thus find this assumption realistic. These assumptions also bear significance for the instance-based approach to transfer learning as utilizing GANs can remedy the dearth of data. What is more, these assumptions are not limited to the subgaussians and include many other subexponential distributions, including 𝜒 2 which appears naturally in many signal processing applications such as Phase Retrieval. In order to tackle optimizations such as 3, 4, we prove their equivalence to a problem with a suitable Gaussian design G . For this purpose we will need the following definition:

Definition 4. We call a Gaussian matrix G matching a block-regular A if G is block-regular as well, E G 𝑖 = E A 𝑖 = 1 µ 𝑇 𝑖 and for any row g of G 𝑖 and any row a of A 𝑖 , it holds that E gg 𝑇 = E aa 𝑇 .

A.2 Assumptions for Theorem 2

Assumptions 3. 1. R 𝑥 ≻ 0 is diagonal without loss of generality.

Let ˆ 𝑝 𝑁 be the empirical spectral density of R 𝑥 , then ˆ 𝑝 𝑁 ⇝ 𝑝 for some probability distribution 𝑝 .

w 0 is a white perturbation of true w ∗ in the R 𝑥 basis. That is, w 0 = w ∗ + R -1 / 2 𝑥 ξ where E ξ = 0 , E ξξ 𝑇 = 𝑒 𝑎 𝑑 I for a fixed 𝑒 𝑎 > 0 .

The third assumption from Assumptions 3 represents the fact that w 0 is a perturbation of the ground truth w ∗ in the eigenbasis of 𝑅 𝑥 .

A.3 Assumptions for Theorem 3

Assumptions 4. 1. The data matrix X satisfies parts (4)-(6) of Definition 2

The means µ 1 , µ 2 are of norm 1 and for any deterministic matrix C of bounded operator norm it holds that µ 𝑇 1 C µ 1 , µ 𝑇 2 C µ 2 and µ 𝑇 1 C µ 2 converge in probability to Tr ( C ) 𝑑 , Tr ( C ) 𝑑 and 𝑟 Tr ( C ) 𝑑 respectively.

𝚺 1 , 𝚺 2 are diagonal.

Let ˆ 𝑝 𝑁 be the joint empirical spectral density of 𝚺 1 , 𝚺 2 , then ˆ 𝑝 𝑁 ⇝ 𝑝 , where the distribution 𝑝 is such that 𝑝 ( 𝑠 1 , 𝑠 2 ) = 𝑝 ( 𝑠 2 , 𝑠 1 ) holds for all 𝑠 1 , 𝑠 2 .

For a discussion of the first assumption from Assumptions 4, see the paragraph after Definition 2. The second assumption from Assumptions 4 is satisfied, for example, for any random vector of the form 𝚺 1 / 2 z , where 𝚺 is an arbitrary 𝑃𝑆𝐷 matrix and z is i.i.d. The third assumption postulates that the means are normalized generic 𝑑 -dimensional vectors with an angle arccos 𝑟 between them. The fourth assumption says that 𝚺 1 and 𝚺 2 are simultaneously diagonalizable and, finally, the fifth assumption simply introduces notation for the joint density function of the eigenvalues of 𝚺 1 and 𝚺 2 .

Lemma 1. Assume that the feature vectors are equally likely to be drawn from class 1 or class 2 and Assumptions 4 hold. Then the optimal classifier is given by w ∗ = ( 𝚺 1 + 𝚺 2 ) -1 ( µ 1 -µ 2 ) and its classification error is equal to the following, where 𝑄 (·) is the integral of the tail of the standard normal distribution.

B Proof of Theorem 1

In this section we provide the complete proof of Theorem 1.

B.1 Proof of part 1 of Theorem 1 when Ψ = Φ 𝜆

We prove the universality of the objective value and then use it to prove the universality of a 𝜓 applied to the optimal solutions. To do so, we construct the perturbed objective. Let

Where | 𝜖 | is small enough that 𝜖𝜓 ( w ) + 𝑓 ( w ) is 𝜌 -strongly convex. For such 𝜖 's, we will show that Φ 𝜆,𝜖 ( A ) and Φ 𝜆,𝜖 ( B ) converge in probability to the same value and use that to deduce a similar result involving 𝜓 ( w Φ 𝜆, 0 ( A ) ) , 𝜓 ( w Φ 𝜆, 0 ( B ) ) . A key step is to reduce the proof to upper bounding the difference of expectations, to do so, the following proposition will be instrumental whose proof is given in Appendix D.

Proposition 1. If for any Lipschitz function 𝑔 : R → R and 𝑡 1 > 0 , P GLYPH<16> | 𝑔 ( Φ 𝜆,𝜖 ( B )) -𝑐 | > 𝑡 1 GLYPH<17> → 0 and for every twice differentiable ˜ 𝑔 : R → R with bounded second derivative, we have that lim 𝑛 →∞ GLYPH<12> GLYPH<12> GLYPH<12> E A , y ˜ 𝑔 ( Φ 𝜆,𝜖 ( B )) -E B , y ˜ 𝑔 ( Φ 𝜆,𝜖 ( A )) GLYPH<12> GLYPH<12> GLYPH<12> → 0 then for any 𝑡 2 > 𝑡 1 .

Thus it suffices to prove lim 𝑛 →∞ GLYPH<12> GLYPH<12> GLYPH<12> E A , y 𝑔 ( Φ 𝜆 ( B )) -E B , y 𝑔 ( Φ 𝜆 ( A )) GLYPH<12> GLYPH<12> GLYPH<12> = 0 for every twice differentiable 𝑔 with bounded second derivative. As stated earlier, we proceed by Lindeberg's approach. Fixing 𝜆 and 𝜖 , to simplify notation we use Φ ( A ) for Φ 𝜆,𝜖 ( A ) . For 0 ≤ 𝑗 ≤ 𝑛 , let 𝑞 ∈ N be the largest such that 𝑗 ≥ ˝ 𝑞 𝑙 = 1 𝑛 𝑙 , and let 𝑟 = 𝑗 -˝ 𝑞 𝑙 = 1 𝑛 𝑙 if , then

It follwos that A = ˆ A 0 and B = ˆ A 𝑛 . Then we have by a telescopic sum

Now we define a new matrix by dropping the 𝑗 'th row.

Note that

And we define M( a ) : = min w ∈S 𝑤 𝓂 ( a , w ) . So, we need to bound GLYPH<12> GLYPH<12> GLYPH<12> E ˆ A 𝑗 , y 𝑔 ( Φ ( ˆ A 𝑗 )) -E ˆ A 𝑗 -1 , y 𝑔 ( Φ ( ˆ A 𝑗 -1 )) GLYPH<12> GLYPH<12> GLYPH<12> , to do so, we condition on ˜ A 𝑗 :

Let us define the optimization whose terms are shared by both of M( a 𝑗 ) and M( b 𝑗 )

Now note that since the second derivative of 𝑔 is bounded, we have

Adding and subtracting 𝑔 ( Θ ) and taking expectation and moving inside, we have

Using 13, and by the independence of Θ and a 𝑗 , b 𝑗 , 𝑦 𝑗 , we have

Thus we focus on M( a 𝑗 ) , M( 𝑏 𝑗 ) conditioned on ˜ A 𝑗 . From now on, we drop the index 𝑗 from ˜ A 𝑗 , a 𝑗 , b 𝑗 , 𝑦 𝑗 . Intuitively, we show that by dropping ( a 𝑇 w -𝑦 ) 2 and ( b 𝑇 w -𝑦 ) 2 and approximating 𝑓 , 𝜓 by their second-order Taylor expansion, the objective values Φ ( ˆ A 𝑗 ) , Φ ( ˆ A 𝑗 -1 ) would not change much. Which implies Φ ( ˆ A 𝑗 ) and Φ ( ˆ A 𝑗 ) are also close in value.

Let 𝑢 : = arg min 𝜆 2 ∥ ˜ Aw -˜ y ∥ 2 + 𝑓 ( w ) + 𝜖𝜓 ( w ) and it is unique because of strong convexity. Now we use the second order Taylor expansion of 𝑓 + 𝜖𝜓 :

Where lim w → u 𝑅 ( w -u ) ∥ w -u ∥ 2 = 0 . Let g : = ∇( 𝑓 ( u ) + 𝜖𝜓 ( u )) , H : = ∇ 2 ( 𝑓 ( u ) + 𝜖𝜓 ( u )) . Moreover, let

Note that a and b match in distribution up to the second moment, which implies that

By triangle inequality we obtain

First we will provide an upper bound for the third and fourth terms in 15. As the function 𝑥 ↦→ 1 1 + 𝑥 is 1-Lipschitz and using Cauchy Schwartz we arrive at

By Lemma 7, there exists a constant 𝐶 1 < ∞ such that E ˜ A , ˜ y , a , 𝑦 ( a 𝑇 u -𝑦 ) 4 < 𝐶 1 for any 𝑛 . Furthermore, by the assumptions, lim 𝑛 →∞ E ˜ A , ˜ y , a , 𝑦 ( a 𝑇 𝛀 -1 a -E aa 𝑇 𝛀 -1 a ) 2 = 0 .

Now for the first and second terms in (15), using Lemma 5, we obtain with probability 1,

Which implies

To deal with the terms involving P , we leverage Markov's inequality

By Lemma 7, there exists a 𝐶 2 < ∞ such that E ˜ A , ˜ y , a , 𝑦 ∥ u -w L∥ 3 3 ≤ 𝐶 2 𝑑 , and recall that E ˜ A , ˜ y , a , 𝑦 ( a 𝑇 u -𝑦 ) 4 < 𝐶 1 for any 𝑛 . Thus

For some constant ˜ 𝐶 = 𝜆 √ 𝐶 1 2 GLYPH<0> 18 𝐶 𝑓 + 𝜖 𝜓 𝜌 GLYPH<1> 3 / 2 + 8 𝐶 2 𝐶 𝑓 + 𝜖 𝜓 . Plugging (16) and (18) in (15), yields E ˜ A , ˜ y GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> E a , b , 𝑦 GLYPH<20> (M( a ) - M( b )) GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> ˜ A , ˜ y GLYPH<21> GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> ≤ ˜ 𝐶 + ˜ 𝐶 ′ 𝑛 √ 𝑑 + 𝜆 2 ˜ 𝐶 2 𝑛 √︃ E ˜ A , ˜ y 𝑉𝑎𝑟 a ( a 𝑇 𝛀 -1 a ) + 𝜆 2 ˜ 𝐶 ′ 2 𝑛 √︃ E ˜ A , ˜ y 𝑉𝑎𝑟 b ( b 𝑇 𝛀 -1 b )

For the other terms in (14), we have by Lemmas 4, 7:

Plugging in (14), we obtain

Therefore,

Now note that since ∥ 𝛀 ∥ 𝑜𝑝 ≤ 1 𝜌 with probability 1, lim 𝑛 →∞ 𝑉𝑎𝑟 a 𝑗 ( a 𝑇 𝑗 𝛀 -1 a 𝑗 ) = 0 from assumptions, thus for every class [ 𝑗 ] , there exists a function 𝜁 [ 𝑗 ] ( 𝑛 ) that 𝑉𝑎𝑟 a 𝑗 ( a 𝑇 𝑗 𝛀 -1 a 𝑗 ) ≤ 𝜁 2 [ 𝑗 ] ( 𝑛 ) for every 𝑛 ∈ N and lim 𝑛 →∞ 𝜁 [ 𝑗 ] ( 𝑛 ) = 0 . Thus

Now since 𝑘 is finite and 𝑛 𝑖 = Θ ( 𝑛 ) for every 1 ≤ 𝑖 ≤ 𝑘 , we have that lim 𝑛 →∞ max1 ≤ 𝑖 ≤ 𝑘 𝑛 𝑖 2 𝑛 𝜁 𝑖 ( 𝑛 ) = 0 . Thus for every 𝜆, 𝜖 > 0

and the proof of part 1 of Theorem 1 is concluded.

B.2 Proof of part 1 of Theorem 1 when Ψ = Φ

Now to prove the part 1 of Theorem 1, we will combine the following lemma with the previous section. For the ease of notation, we drop the dependency on 𝜖 . Recall that since 𝑓 is 𝜌 -strongly convex, we can write

For some convex ℎ : R 𝑑 → R .

Lemma 2. If for every 𝜆 > 0 , we have Φ 𝑛 𝜆 ( 𝐴 ) P - → 𝑐 𝜆 , then for every Lipschitz function 𝑔 : R → R , we have 𝑔 ( sup 𝜆> 0 Φ 𝑛 𝜆 ( 𝐴 )) P - → 𝑔 ( sup 𝜆> 0 𝑐 𝜆 ) and 𝑔 ( sup 𝜆> 0 Φ 𝑛 𝜆 ( 𝐵 )) P - → 𝑔 ( sup 𝜆> 0 𝑐 𝜆 )

Proof. First note that for a given 𝑡 > 0 , we have for every Lipschitz function 𝑔

Letting 𝛿 : = 𝑡 𝐿 , ˜ 𝛿 1 > 0 , we know that for every 𝜆 ≥ 0 , there exists 𝑁 1 ∈ N such that for every 𝑛 ≥ 𝑁 1

We will show that an 𝑅 1 > 0 can be chosen in such a way that there exists 𝑁 2 ∈ N such that for every 𝑛 ≥ 𝑁 2

To see this, by the monotonicity of Φ 𝑛 𝜆 in 𝜆 , we have by the fundamental theorem of calculus for a given 𝑅 1

Now for the gradient of the main optimization we have, by 𝑔 : = ∇ ℎ ( w )

Which implies w = ( 𝜆 A 𝑇 A + 𝜌 I ) -1 ( 𝜆 A 𝑇 y -g ) . Hence by matrix inversion lemma,

By triangle inequality and definition of the operator norm we have

Whence plugging back in 19 yields

For the norm of g , by assumption, there exists a 𝐶 𝑔 independent of 𝜆 , such that ∥ g ( 𝑠 ) ∥ 2 2 𝑛 ≤ 𝐶 𝑔 with high probability.

Consider the following the events

We further have

Thus we need to focus on the operator norm of each term in 20. After diagonalizing and using the fact that for any two rectangular matrices X , Y with matching dimensions, 𝑠𝑝𝑒𝑐 ( XY ) = 𝑠𝑝𝑒𝑐 ( YX )

Now by the event T 𝐴 , 𝑠 min ( AA 𝑇 ) ≥ 𝐶 𝐴 , then for a large enough 𝑅 1 such that 𝜌 𝑅 1 ≤ 𝐶 𝐴 , then for every 𝜆 ≥ 𝑅 1

Bounding the integrals implies

Summarizing,

Now we choose 𝑅 1 to be large enough such that the first two terms vanish. Therefore there exists an 𝑁 2 ∈ N such that for every 𝑛 ≥ 𝑁 2 , P (T 𝑐 𝑦 ) + P (T 𝑐 𝑔 ) + 2 P (T 𝑐 𝐴 ) ≤ ˜ 𝛿 2 . Thus for such 𝑅 1 and 𝑁 2 , we have

Now we can let 𝑅 2 > 0 be chosen in such a way that:

Take the maximum of 𝑅 1 and 𝑅 2 as 𝑅 . By triangle inequality we observe

First we prove 𝑐 𝜆 is concave in 𝜆 . Then by appealing to the Convexity Lemma (lemma 7.75 in Liese and Miescke [2008]), the result follows. Note that Φ 𝑛 𝜆 is concave in 𝜆 as it is the pointwiseminimum of a concave function in 𝜆 everywhere. Now to prove the concavity of 𝑐 𝜆 , for a given pair of 𝜆 1 , 𝜆 2 ∈ [ 0 , 𝑅 ] and 0 < 𝜃 < 1 , we prove the deterministic event { 𝑐 𝜃𝜆 1 +( 1 -𝜃 ) 𝜆 2 -𝜃𝑐 𝜆 1 -( 1 -𝜃 ) 𝑐 𝜆 2 < 0 } has probability zero. First note that

Thus by union bound for 𝜖 > 0

Now by taking 𝑛 to be large enough, the RHS can be made arbitrarily small for every 𝜖 and 𝜃 , thus 𝑐 𝜆 is concave. Furthermore, since [ 0 , 𝑅 ] is a compact set, we can appeal to the convexity lemma 7.75 in Liese and Miescke [2008] and the first claim follows. For the second claim, note that so far we have proved

Then since | Φ 𝜆 ( 𝐵 ) -Φ 𝜆 ( 𝐴 )| P - → 0 thus Φ 𝜆 ( 𝐵 ) P - → 𝑐 𝜆 , repeating the same argument this time for Φ 𝜆 ( 𝐵 ) yields

B.2.1 Sequence of Regularizers

We prove that by constraining ourselves to a large enough compact set S w , if 𝑓 𝑚 → 𝑓 uniformly, then the universality results also hold. Note for the Φ 𝜆 case, we have that by setting w = 0 in the optimization, we have ∥ w Φ 𝜆 ( A ) ∥ 2 ≤ 𝐶 𝑤 √ 𝑛 for some 𝐶 𝑤 > 0 with high probability, thus we can instead set S w : = 𝐶 𝑤 √ 𝑛 and consider the following equivalent constrained optimization problem

Now consider a sequence of regular functions 𝑓 𝑚 , converging uniformly to 𝑓 on S w . Take 𝑚 large enough such that ∥ 𝑓 -𝑓 𝑚 ∥∞ ≤ 𝑛𝛿 for any 𝑛 which implies | ˜ Φ 𝑚 𝜆,𝜖 ( A ) -˜ Φ 𝜆,𝜖 ( A )| ≤ 𝛿 . Thus for any 𝑡 > 0

If ˜ Φ 𝑚 𝜆,𝜖 ( A ) P - → 𝑐 𝑚 𝜆,𝜖 , then by (22), we have ˜ Φ 𝜆,𝜖 ( A ) P - → 𝑐 𝜆,𝜖 . A similar argument holds for

Furthermore, subsequent results also hold for 𝜓 ( w Φ 𝑚 𝜆 ( A ) ) and 𝜓 ( w Φ 𝜆 ( A ) ) , 𝜓 ( w Φ 𝑚 ( A ) ) and 𝜓 ( w Φ ( A ) )

B.3 Proof of Part 2 of Theorem 1

B.3.1 Proof of Part 2 of Theorem 1 for a regular test function when Ψ = Φ 𝜆

Now we prove the second part of Theorem 1. Since we assumed Φ 𝜆, 0 ( A ) converges to some 𝑐 𝜆, 0 in probability, then for a small enough 𝜖 , we also have Φ 𝜆,𝜖 ( A ) P - → 𝑐 𝜆,𝜖 . Therefore we may write by an application of triangle inequality and the union bound

For every 𝜖, 𝑡 > 0 , the RHS of (23) goes to zero. Similarly we have P GLYPH<18> GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> Φ 𝜆, 0 ( A ) -Φ 𝜆, -𝜖 ( A ) 𝜖 -𝑐 𝜆, 0 -𝑐 𝜆, -𝜖 𝜖 GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> > 𝑡 GLYPH<19> → 0 .

Now we observe that by triangle inequality

We take ˆ 𝜖 to be small enough such that

This implies

And by Danskin's theorem, Φ 𝜆,𝜖 ( A ) is minimum of a concave function 𝜖 , therefore it is also concave and it is differentiable with respect to 𝜖 , we observe that 𝜓 ( 𝑤 Φ 𝜆 ( A ) ) = 𝑑 Φ 𝜆,𝜖 ( A ) 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 . Moreover, by concavity over ˆ 𝜖 :

Whence

By (23), the RHS goes to zero, thus 𝜓 ( w Φ 𝜆 ( A ) ) P - → 𝑑𝑐 𝜆,𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 . Furthermore, we have

By the result of the first part, we know that the RHS (24) goes to zero as 𝑛 →∞ . Hence, 𝜓 ( w Φ 𝜆 ( B ) ) P - → 𝑑𝑐 𝜆,𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 .

Moreover, if 𝜓 is bounded, by definition of convergence in distribution, | E A 𝜓 ( w Φ 𝜆 ( A ) ) -E B 𝜓 ( w Φ 𝜆 ( B ))|→ 0 .

B.3.2 Proof of Part 2 of Theorem 1 for a regular test function for Ψ = Φ

Defining

Since 𝑠𝑢𝑝 𝜆> 0 ˜ Φ 𝜆,𝜖 ( A ) P - → 𝑠𝑢𝑝 𝜆> 0 𝑐 𝜆,𝜖 , or equivalently ˜ Φ 𝜖 ( A ) P - → 𝑐 𝜖 , we can use the same argument from previous section and Danskin's theorem, to observe that 𝜓 ( w ˜ Φ 𝜆 ( A ) ) P - → 𝑑𝑐 𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 and 𝜓 ( w ˜ Φ 𝜆 ( B ) ) P - → 𝑑𝑐 𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 .

B.3.3 Sequences of regular test functions

By a similar approach, it follows that for a sequence of regular functions 𝜓 𝑚 converging uniformly to 𝜓 , one also has 𝜓 ( w Φ 𝜆 ) P - → 𝑑𝑐 𝜆,𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 : = lim 𝑚 →∞ 𝑑𝑐 𝜖 𝑚 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 . Indeed,

Since we have uniform convergence, for 𝑡 2 , there exists an 𝑚 0 such that for all 𝑚 ≥ 𝑚 0 and all 𝑥 ∈ R 𝑑 , we have | 𝜓 𝑚 ( 𝑥 ) -𝜓 ( 𝑥 )| < 𝑡 2 . Thus for such 𝑚 's, P GLYPH<18> GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> 𝜓 ( w Φ 𝜆 ( A ) ) -𝜓 𝑚 ( w Φ 𝜆 ( A ) ) GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> > 𝑡 2 GLYPH<19> = 0 . Then by triangle inequality

By assumption P GLYPH<18> GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> 𝜓 𝑚 ( w Φ 𝜆 ( A ) ) -𝑑𝑐 𝑚 𝜆,𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> > 𝑡 4 GLYPH<19> → 0 . And by the definition of 𝑑𝑐 𝜆,𝜖 𝑑𝜖 GLYPH<12> GLYPH<12> GLYPH<12> 𝜖 = 0 the claim follows.

C Proof of Corollary 1

We will utilize the result of the following Theorem.

Theorem 4. (Bobkov [2003]) For an isotropic random vector X ∈ R 𝑑 , and an isotropic gaussian g ∈ R 𝑑 if P (| ∥ X ∥ 2 √ 2 -1 | ≥ 𝜖 𝑑 ) ≤ 𝜖 𝑑 , then for all 𝛿 > 0

Where θ ∼ 𝜈 the uniform measure on the unit sphere S 𝑑 -1

Essentially, through CGMT, we observe that when we use a quadratic regularizer or run SGD, the solution, w Ψ ( G ) converges weakly in distribution to a Gaussian vector, g 𝐴𝑂 . Whence we conclude that w Ψ ( G ) behaves the same as a generic vector. Therefore, by choosing X : = a -E a , a row of our data matrix A , Theorem 4 enables us to prove the universality of the classification error. Note that in general E w Ψ ( G ) ≠ 0 , but by Assumptions 4, the means of the classes are generic which allows us to employ Theorem 4 to conclude the proof.

D Lemmata for the proof of Theorem 1

Proposition 2. If for any Lipschitz function 𝑔 : R → R and 𝑡 1 > 0 , P GLYPH<16> | 𝑔 ( Φ 𝜆,𝜖 ( B )) -𝑐 | > 𝑡 1 GLYPH<17> → 0 and for every twice differentiable ˜ 𝑔 : R → R with bounded second derivative, we have that lim 𝑛 →∞ GLYPH<12> GLYPH<12> GLYPH<12> E A , y ˜ 𝑔 ( Φ 𝜆,𝜖 ( B )) -E B , y ˜ 𝑔 ( Φ 𝜆,𝜖 ( A )) GLYPH<12> GLYPH<12> GLYPH<12> → 0 then for any 𝑡 2 > 𝑡 1 .

Proof. First note that P (| Φ 𝜆,𝜖 ( B ) -𝑐 | > 𝑡 ) → 0 , implies P (| 𝑔 ( Φ 𝜆,𝜖 ( B )) -𝑔 ( 𝑐 )| > 𝐿𝑡 ) → 0 . Thus it suffices to verify P (| Φ 𝜆,𝜖 ( B ) -𝑐 | > 𝑡 ) → 0 . This follows similar to Panahi and Hassibi [2017]. Let 𝜉 ( 𝑥 ) : = 2 1 (-∞ , -𝑡 2 ]∪[ 𝑡 2 , ∞) ( 𝑥 ) + GLYPH<0> 𝑡 1 + 𝑡 2 2 - (| 𝑥 | -𝑡 2 ) 2 GLYPH<1> 1 (-𝑡 2 , -𝑡 1 + 𝑡 2 2 ]∪[ 𝑡 1 + 𝑡 2 2 ,𝑡 2 ) ( 𝑥 ) + (| 𝑥 | -𝑡 1 ) 2 1 (-𝑡 1 + 𝑡 2 2 , -𝑡 1 ]∪[ 𝑡 1 , 𝑡 1 + 𝑡 2 2 ) ( 𝑥 ) . Then we have

Then by Markov's and triangle inequality

Note that 𝜉 ( 𝑥 ) ≤ 2 1 (-∞ , -𝑡 1 ]∪[ 𝑡 1 , ∞) . Thus 1 2 E B , y 𝜉 GLYPH<0> Φ 𝜆,𝜖 ( B ) -𝑐 GLYPH<1> ≤ P GLYPH<16> | Φ 𝜆,𝜖 ( B ) -𝑐 | > 𝑡 1 GLYPH<17> . Moreover, we know that 𝜉 ( . ) has bounded second derivative derivatives almost everywhere and by assumption on 𝑔 ′ , we would have

Lemma 3. We have the following:

Proof. We know for Θ , taking derivative yields

This implies -𝜆 ˜ A 𝑇 ˜ y + g = -𝜆 ˜ A 𝑇 ˜ Au .

Note that for L( a ) , since the objective is quadratic in w , we solve the optimization in closed form, we have

Let 𝛀 : = 𝜆 ˜ A 𝑇 ˜ A + H . hence

Using Sherman-Morrison as 𝛀 ≻ 0 :

From here we obtain for the solution Thus for the first claim:

Hence we have for the first claim

Now we continue with the calculation of the objective value

= u 𝑇 𝛀 u

Thus

Since 𝑔 = 𝜆 ˜ A 𝑇 ˜ y -𝜆 ˜ A 𝑇 ˜ A 𝑢 , then g 𝑇 u = 𝜆 ˜ y 𝑇 ˜ Au -𝜆 u 𝑇 ˜ A ˜ A 𝑢 , therefore

Lemma 4. We have the following:

Proof. 1. By definition,

2. The idea of the proof is by strong convexity. By assumption, 𝑓 ( w ) + 𝜖𝜓 ( w ) for small enough 𝜖 is 𝜌 -strongly convex. This implies from defintion:

By Taylor remainder theorem for some 0 < 𝑡 < 1 :

Since 𝑓 and 𝜓 are separable, 𝜕 𝛼 ( 𝑓 + 𝜖𝜓 ) = 0 unless 𝛼 = 3 𝑒 𝛼 ( 𝑒 𝛼 is the standard basis vector in R 𝑑 . Then by assumption on the third derivative of 𝑓 + 𝜖𝜓 ,

Now we are ready to prove the following key lemma.

Proof. Let w M = arg min 𝓂 ( a , w ) . Note that since Θ ≤ M( a ) and by Lemma 4 we have

On the other hand, note that 𝓂 ( a , u ) = Θ + 𝜆 2 𝑛 ( a 𝑇 u -𝑦 ) 2 and Θ ≤ L( a )

Thus as 𝜆 2 𝑛 ( a 𝑇 u -𝑦 ) 2 ≥ 𝜆 2 𝑛 ( a 𝑇 u -𝑦 ) 2 1 + 𝜆 a 𝑇 𝛀 -1 a

But as we will see, ( a 𝑇 u -𝑦 ) 2 is of order 𝑂 ( 1 ) , which will not suffice. Thus we break down M( a ) - L( a ) depending on the distance of 𝑢 and w L

For the other direction

Let 𝑅 : = ∥ w L u ∥ 3 and consider the ℓ 3 ball centered at w L . Now if ∥ w M w L∥ 3 ≤ 𝑅 then ∥ w M u ∥ 3 ≤ 2 𝑅 by a straightforward application of triangle inequality. Which implies L( a ) - M( a ) ≤ 8 𝐶 𝑓 + 𝜖 𝜓 𝑛 𝑅 3 . Now when does ∥ w M w L∥ 3 ≤ 𝑅 hold? We show that for every point on the boundary, w , 𝓂 ( a , w ) ≥ 𝓂 ( a , w L) , this implies that w M is indeed inside the ball. We have

Now if 𝑅 ≤ 𝜌 18 𝐶 𝑓 + 𝜖 𝜓 , then 𝓂 ( a , w ) -𝓂 ( a , w L) ≥ 0 . Thus all in all

□

Lemma 6. We have the following upper bounds

∥ u ∥ 2 2 = 𝑂 ( 𝑛 ) with high probabilty.

Proof. 1. Since 𝑓 + 𝜖𝜓 is 𝜌 -strongly convex, one can decompose 𝑓 ( w ) + 𝜖𝜓 ( w ) = 𝜌 2 ∥ w ∥ 2 2 + ℎ ( w ) for some convex function ℎ (·) . Then

Thus this implies ∥ u ∥ 2 2 = 𝑂 ( 𝑛 ) .

2. In a similar fashion, we have for E ˜ A , ˜ y µ 𝑇 u

Let ˜ A 𝑇 = GLYPH<16> ˆ ˜ A 𝑇 1 ˆ ˜ A 𝑇 2 . . . ˆ ˜ A 𝑇 𝑘 GLYPH<17> . Now we use the definition of ˜ A to obtain a lower bound:

Thus plugging in w = 0 in the optimization yields an upper bound, which is a quadratic inequality in µ 𝑇 𝑖 u :

Now note that for a quadratic optimization with 𝑏 ≥ 0

Whence

We apply this result to 25,

Now taking expectation and using a combination Cauchy-Schwarz and Jensen inequalities yields

Now note that E ˆ ˜ A 𝑖 = 0 , and by the independence of the rows:

By assumption, for each row ˆ a 𝑖 of ˆ ˜ A 𝑖 , we have that E ˆ a 𝑖 ∥ ˆ a 𝑖 ∥ 2 2 = 𝑂 ( 1 ) , which implies E ˜ A , ˜ y ∥ 1 𝑇 ˆ ˜ A 𝑖 ∥ 2 2 = 𝑂 ( 𝑑 ) . Therefore, by the previous part, since ∥ u ∥ 2 2 ≤ 𝜆 𝜌 ∥ ˜ y ∥ 2 2 + 2 𝜌 ℎ ( 0 ) , then E ˜ y ∥ u ∥ 2 2 ≤ 2 𝜌 ℎ ( 0 ) + 𝜆 𝜌 E ˜ y ∥ ˜ y ∥ 2 2 = 𝑂 ( 𝑑 ) by assumption on the ˜ y . Hence it can be seen that E ˜ A , ˜ y µ 𝑇 u = 𝑂 ( 1 ) □

Now we are ready to conclude the proof with the following lemma.

Lemma 7. We have the following

Proof. 1. First note that

Moreover,

By assumption E | 𝑦 | 2 𝑞 -𝑦 ≤ 𝐶 for some constant 𝐶 > 0 that is dimension-independent. Moreover, by our assumption, E | ( a -µ ) 𝑇 u | 𝑗 ≤ ˜ 𝐶 ∥ u ∥ 𝑗 2 𝑑 𝑗 / 2 . Also, from the previous lemma, E | µ 𝑇 u | 2 𝑗 = 𝑂 ( 1 ) . The claim follows.

2. First note that ∥ u -w L∥ 3 3 = 𝜆 3 GLYPH<12> GLYPH<12> GLYPH<12> 𝑦 -a 𝑇 u 1 + 𝜆 a 𝑇 𝛀 -1 a GLYPH<12> GLYPH<12> GLYPH<12> 3 ∥ 𝛀 -1 a ∥ 3 3 and since 𝛀 ≻ 0 , 1 + 𝜆 a 𝑇 𝛀 -1 a > 0 hence by Minkowski's inequality

Now since we assumed that the objective is 𝜌 -strongly convex, or there exists 𝜌 > 0 that ∥ 𝛀 -1 ∥ 𝑜𝑝 ≤ 1 𝜌 , denoting ω 𝑗 as the 𝑗 th row of 𝛀 -1 , we have by the inequality between Frobenius and Operator norm, ∥ ω 𝑘 ∥ 2 ≤ 1 𝜌 . Now by the assumptions we have

We also have for the other term:

Note that ∥ 𝛀 -1 µ ∥ 2 ≤ ∥ 𝛀 -1 / 2 ∥ 𝑜𝑝 ∥ 𝛀 -1 / 2 µ ∥ 2 . Thus it is sufficient to analyze µ 𝑇 𝛀 -1 µ 𝑇 . Let 𝑡 ∈ R be an upper bound for µ 𝑇 𝛀 -1 µ 𝑇 , that is µ 𝑇 𝛀 -1 µ 𝑇 ≤ 𝑡 . Note that by the Schur complement property, we need to find the smallest 𝑡 > 0 such that 𝛀 -1 𝑡 µµ 𝑇 ⪰ 0 with holds with high probability. Recall that 𝛀 : = 𝜆 ˜ A 𝑇 ˜ A + H and by assumption H ⪰ 𝜌 I , thus it suffices to find the largest 𝑡 > 0 that 𝜆 ˜ A 𝑇 ˜ A + 𝜌 I -1 𝑡 µµ 𝑇 ⪰ 0 . Let ˜ A ′ : = ˜ A -˜ M be centered. This problem is equivalent to having ∥( ˜ A ′ + ˜ M ) v ∥ 2 2 + 𝜌 ∥ v ∥ 2 2 -1 𝑡 ( µ 𝑇 v ) 2 ≥ 0 for every v ∈ R 𝑑 .

Thus we should have

Let 𝑡 ∗ : = sup v ( µ 𝑇 v ) 2 ∥ ( ˜ A ′ + ˜ M ) v ∥ 2 2 + 𝜌 ∥ v ∥ 2 2 . We show 𝑡 ∗ = 𝑂 ( 𝑛 -𝑠 ) for some 𝑠 > 0 with high probability. Let v ∗ : = arg max ( µ 𝑇 v ) 2 ∥ ( ˜ A ′ + ˜ M ) v ∥ 2 2 + 𝜌 ∥ v ∥ 2 2 . This implies

On the other hand, by triangle inequality we have

. Now we have

We obtain a quadratic inequality in 𝑡 ∗

Which implies the following upper bound

According to Lemma (8) ∥ ˜ A ′ ∥ 𝑜𝑝 ≤ 𝑛 1 3 + 𝑠 with probability 1 -𝐶 log ( 𝑑 + 𝑛 ) 𝑛 𝑠 .Thus using the law of total expectation, we also observe that

6 2

Lemma 8. Let ˜ A ′ be a centered random matrix following the assumptions. We have ∥ ˜ A ′ ∥ 𝑜𝑝 ≤ 𝐶𝑛 1 3 + 𝑠 with probability at least 1 -𝐶 ′ log ( 𝑑 + 𝑛 ) 𝑛 𝑠 .

Proof. First, we provide an upper bound for the expectation E ˜ A ′ ∥ ˜ A ′ ∥ 𝑜𝑝 using the matrix Bernstein inequality and the symmetrization technique. First we construct a corresponding hermitian matrix:

With X 𝑖 : = GLYPH<18> 0 E 𝑖𝑖 ˜ A ′ ˜ A ′ 𝑇 E 𝑖𝑖 0 GLYPH<19> where E 𝑖𝑖 is the all-zero matrix except ( 𝑖, 𝑖 ) -entry where it is equal to 1. Note that I = ˝ 𝑛 𝑖 = 1 E 𝑖𝑖 and E 𝑖𝑖 ˜ A ′ essentially picks the i'th row of ˜ A ′ , which is ˜ a ′ 𝑇 𝑖 . Let { 𝜖 𝑖 } 𝑛 𝑖 = 1 be an iid sequence of symmetric Bernoulli random variables supported on {± 1 } , independent of X 𝑖 's. Then by the symmetrization lemma Vershynin [2018], we have

Now conditioning on ˜ A ′ or equivalently X 𝑖 's, since 𝜖 X 𝑖 's are independent zero-mean symmetric matrices of size ( 𝑑 + 𝑛 ) × ( 𝑑 + 𝑛 ) with bounded operator norm, we can leverage Matrix Bernstein inequality Vershynin [2018] to obtain ( ≲ means up to some constant)

Note that for GLYPH<13> GLYPH<13> X 𝑖 GLYPH<13> GLYPH<13> 𝑜𝑝 , we have

Moreover, using triangle inequality

Combining these results along with Jensen's inequality yields

Now to analyze E ˜ A ′ GLYPH<13> GLYPH<13> GLYPH<13> GLYPH<13> ˝ 𝑛 𝑖 = 1 ˜ a ′ 𝑖 ˜ a ′ 𝑇 𝑖 GLYPH<13> GLYPH<13> GLYPH<13> GLYPH<13> 𝑜𝑝 , we use the symmetrization trick again with iid Bernoulli { 𝜖 ′ 𝑖 } 𝑛 𝑖 = 1 and Bernstein's similar to Vershynin's

Now for the first term, applying Cauchy Schwartz yields

Moreover, for the last term we have from our assumptions

Thus we have

GLYPH<13>

GLYPH<13>

𝑜𝑝

Thus

Summarising, we have

E ˜ A ′ ∥ ˜ A ′ ∥ 𝑜𝑝 ≲ √︁ log ( 𝑑 + 𝑛 ) GLYPH<18> E ˜ A ′ max 1 ≤ 𝑖 ≤ 𝑛 ∥ ˜ a ′ 𝑖 ∥ 2 2 + log 𝑛 E ˜ A ′ max 1 ≤ 𝑖 ≤ 𝑛 ∥ ˜ a ′ 𝑖 ∥ 2 2 + 𝐶 GLYPH<19> 1 / 2 + log ( 𝑑 + 𝑛 ) E ˜ A ′ max 1 ≤ 𝑖 ≤ 𝑛 ∥ ˜ a ′ 𝑖 ∥ 2 2 Thus it remains to provide an upper bound for E ˜ A ′ max1 ≤ 𝑖 ≤ 𝑛 ∥ ˜ a ′ 𝑖 ∥ 2 2 , we have by Jensen's

Now by Definition 2 and Holder's we have

Thus by Markov's inequality for 𝑠 > 0

Lemma 9. For any function 𝜓 : R → R with bounded third derivative, ˜ 𝜓 ( w ) : = E a 𝜓 ( a 𝑇 w ) is regular

Proof. We only need to verify the third order Taylor remainder bound,

By boundedness of the third derivative of 𝜓 , and assumptions

𝑖

𝑖

Solving for

All in all

˜

A

′

GLYPH<13>

𝑖

=

1

˜

a

˜

a

GLYPH<13>

yields

E Generalized CGMT

After reducing the analysis to the equivalent gaussian model, one can leverage a new generalization of the CGMT Akhtiamov et al. [2024] to analyze Φ ( G ) which is stated as follows and recover the precise asymptotic properties of the solutions.

Theorem 5 (Generalized CGMT) . Let S 𝑤 ⊂ R 𝑑 , S 𝑣 1 ⊂ R 𝑛 1 , . . . , S 𝑣 𝑘 ⊂ R 𝑛 𝑘 be compact convex sets. Denote S 𝑣 : = S 𝑣 1 × · · · × S 𝑣 𝑘 , let v ∈ S 𝑣 stand for ( v 1 , . . . , v 𝑘 ) ∈ S 𝑣 1 × · · · × S 𝑣 𝑘 and 𝜓 ( w , v ) : S 𝑤 × S 𝑣 → R be convex on S 𝑤 and concave on S 𝑣 . Also let 𝚺 1 , . . . , 𝚺 𝑘 ∈ R 𝑑 × 𝑑 be arbitrary PSD matrices. Furthermore, let G 1 ∈ R 𝑛 1 × 𝑑 , . . . , G 𝑘 ∈ R 𝑛 𝑘 × 𝑑 , g 1 , . . . , g 𝑘 ∈ R 𝑑 , h 1 ∈ R 𝑛 1 , . . . , h 𝑘 ∈ R 𝑛 𝑘 all have i.i.d N( 0 , 1 ) components and G = ( G 1 , . . . , G 𝑘 ) , g = ( g 1 , . . . , g 𝑘 ) , h = ( h 1 , . . . , h 𝑘 ) be the corresponding 𝑘 -tuples. Define

Let S be an arbitrary open subset of S 𝑤 and S 𝑐 = S 𝑤 \ S . Let 𝜙 S be the optimal value of the optimization 27 when S 𝑤 is restricted to S . Assume also that there exist 𝜖, 𝛿 > 0 , ¯ 𝜙, ¯ 𝜙 S 𝑐 such that

¯ 𝜙 S 𝑐 ≥ ¯ 𝜙 + 3 𝛿

𝜙 ( 𝑔, ℎ ) < ¯ 𝜙 + 𝛿 with probability at least 1 -𝜖

𝜙 S 𝑐 > ¯ 𝜙 S 𝑐 -𝛿 with probability at least 1 -𝜖

Then for the optimal solution w of the optimization 26, we have P ( w Ψ ( G ) ∈ S) ≥ 1 -2 𝑘 + 1 𝜖

F Proof of Theorem 2

By leveraging our universality theorem 1 we are ready to analyze the linear regression problem descrided in 2 in full details

Proof. First note that for the ground truth we have 𝑤 ∗ = 𝑎𝑟𝑔𝑚𝑖𝑛 E 𝑃 ( 𝑦 -x 𝑇 w ) 2 = R -1 𝑥 R 𝑥𝑦 . By a simple transformation, the constraint could also be written as

We would like to write for a G with i.i.d entries:

Nowwecandenote S = R 1 / 2 𝑥 R -𝑇 / 2 𝑥 R 𝑥𝑦 0 𝑅 1 / 2 Δ ! with 𝑅 Δ : = 𝑅 𝑦 -R 𝑦𝑥 R -1 𝑥 R 𝑥𝑦 . And R -𝑇 / 2 𝑥 R 𝑥𝑦 = R 1 / 2 𝑥 w ∗ .

Thus we can write the optimization as

Indeed

we can write the optimization as

Let u : = R 1 / 2 𝑥 ( w -w ∗ ) and x = w ∗ -w 0 , then

We Lagrange multiplier to bring in the constraint:

Now we can appeal to CGMT to obtain the AO:

Dropping the 𝑔 ′ 𝑅 1 / 2 Δ term and doing the optimization over the direction of v , yields

Using the identity √ 𝑥 = min 𝜏 ≥ 0 𝜏 2 + 𝑥 2 2 𝜏

Now we obtain a quadratic optimization over 𝑢 which could be rewritten as

Doing the optimization over u , we are left with

Expanding the third term yields

Let z : = R 1 / 2 𝑥 x , we assume E zz 𝑇 = ∥ z ∥ 2 𝑑 𝐼 , and 𝑒 𝑎 = x 𝑇 R 𝑥 x = ∥ z ∥ 2 using concentration:

Therefore

Dropping the constant term ∥ x ∥ 2 , we arrive at the following scalarized optimization

Let 𝑒 ′ 𝑎 : = 𝑒 𝑎 𝑑 . To analyze the optimization further, noting that the objective is differentiable w.r.t 𝛽, 𝜏 , by taking derivatives we have at the optimal point:

Rearranging

Summing and subtracting:

Thus

If one can solve 𝑑 -𝑛 𝑑 = 2 𝜏 𝛽 √ 𝑛 𝑆 𝑃 (-2 𝜏 𝛽 √ 𝑛 + 𝑖 0 ) to find 𝜃 : = 𝛽 𝜏 , then plugging into the second equation yields

Which yields the following equation for 𝜏 2

Now note that

Let 𝛼 : = 4 ∫ 𝑝 ( 𝑟 ) ( 2 + 𝑟 𝜃 √ 𝑛 ) 2 𝑑𝑟 , 𝛼 ′ : = 2 𝜃 √ 𝑛 ∫ 𝑝 ( 𝑟 ) 𝑟 ( 2 + 𝑟 𝜃 √ 𝑛 ) 2 𝑑𝑟 and . Then the expression for 𝜏 could be written as

As 𝛼 + 𝛼 ′ = 𝑑 -𝑛 𝑑 , one can write

Now the generalization error is

With 𝛼 = 4 ∫ 𝑝 ( 𝑟 ) ( 2 + 𝑟 𝜃 √ 𝑛 ) 2 𝑑𝑟 . Now note that using Cauchy Schwarz applied to √︁ 𝑝 ( 𝑟 ) and 2 √ 𝑝 ( 𝑟 ) 2 + 𝑟 𝜃 √ 𝑛

Thus 𝛼 ≥ ( 𝑑 -𝑛 𝑑 ) 2 . Now for the scalar distribution 𝑝 ( 𝑟 ) = 𝛿 ( 𝑟 -𝑟 0 ) , since 2 + 𝑟 0 𝜃 √ 𝑛 = 2 + 2 𝑛 𝑑 -𝑛 = 2 𝑑 𝑑 -𝑛 , then 𝛼 = ( 𝑑 -𝑛 𝑑 ) 2 . Thus 𝛼 achieves its lower bound (uniquely) for the scalar distribution. Now note that assuming 𝑛, 𝑑, 𝑒 𝑎 , 𝑅 Δ are fixed, the generalization error is increasing in 𝛼 , thus the error expression for the generalization error in the scalar distribution case is a lower bound. □

G Useful Results

Lemma 10. Given a weight vector w , and assuming the feature vectors are equally likely to be drawn from class 1 or class 2 and satisfy part 2 of Assumptions 4, the corresponding classification error is given by

Lemma 11. Assume that the feature vectors are equally likely to be drawn from class 1 or class 2 , Assumptions 4 hold and w 0 is defined as in Assumption 2. Then the classification error corresponding to w 0 is equal to

«

H Proof of Lemma 1

Let us proceed to prove Lemma 1. We will take the gradient of (28) w.r.t. 𝑤 and set it to 0 :

Proof.

Hence, the following equality holds at any optimal w :

Therefore, w = ( ˜ 𝑠 1 𝚺 1 + ˜ 𝑠 2 𝚺 2 ) -1 ( ˜ 𝑡 1 µ 1 -˜ 𝑡 2 µ 2 ) , where we have

‹

«

‹

Now we can consider the following optimization over 4 scalar variables:

Note that since the optimization (29) is low-dimensional, by the Convexity Lemma Liese and Miescke [2008], we can interchange the limit in 𝑛 and min , implying that we can analyze the concentrated optimization instead. We then arrive at the following fixed point equations using Assumptions 4:

Note that, since we assume 𝑝 is exchangeable in its arguments, the system of equations (30) is invariant under the transformation ( 𝑠 1 , 𝑡 1 , 𝑠 2 , 𝑡 2 ) → (-𝑠 2 , 𝑡 2 , -𝑠 1 , 𝑡 1 ) . As such, 𝑠 1 = -𝑠 2 and 𝑡 1 = 𝑡 2 at the optimal point, implying that w ∗ = ( 𝚺 1 + 𝚺 2 ) -1 ( µ 1 -µ 2 ) is the optimal classifier, as for the chosen linear classification procedure the performance in invariant to the rescalings of the classifier vector and therefore we can completely drop 𝑠 1 , 𝑠 2 , 𝑡 1 , 𝑡 2 from the equations.

□

I Proof of Theorem 3

By Gaussian universality proved in Theorem 1, we analyze the binary classification problem as follows

Proof. Define

Note that the eigenvalue density of R 0 is captured by 𝜙 ( 𝑠 1 , 𝑠 2 ) = 𝑡 2 𝜂 ( 1 -𝑟 ) 𝑑 + 2 𝑡 2 ∗ ( 1 -𝑟 ) 𝑑 ( 𝑠 1 + 𝑠 2 ) -2 , assuming ( 𝑠 1 , 𝑠 2 ) ∼ 𝑝 , where 𝑝 is the joint EPD function of 𝚺 1 and 𝚺 2 .

We would like to start with finding 𝛼 . Note that the data matrix X can be decomposed as follows where G 1 , G 2 are i.i.d. standard normal:

We then have:

It is immediate to see that the first term of (34) is a centered normally distributed random vector with covariance diag ( w 𝑇 0 𝚺 1 w 0 , . . . , w 𝑇 0 𝚺 1 w 0 , w 𝑇 0 𝚺 2 w 0 , . . . , w 𝑇 0 𝚺 2 w 0 ) , and therefore its norm concentrates to √︃ 𝑛 2 w 𝑇 0 ( 𝚺 1 + 𝚺 2 ) w 0 = Θ ( √︁ Tr ( 𝚺 1 + 𝚺 2 )∥ w 0 ∥) according to our assumptions. The second term of (34) is a deterministic vector of norm Θ ( √ 𝑛 ∥ w 0 ∥ √ 1 -𝑟 ) . Similarly:

Note that we can drop the first term of (35) compared to the second term. Indeed, the first term is a centered normal random variable with variance 𝑛 2 w 𝑇 0 ( 𝚺 1 + 𝚺 2 ) w 0 = Θ (∥ w 0 ∥ 2 Tr ( 𝚺 1 + 𝚺 2 )) and the second term is a deterministic quantity of order Θ ( 𝑛 ∥ w 0 ∥ √ 1 -𝑟 ) . Thus

Moreover, by independence of η from µ 𝑖 we can drop the first term in ( µ 1 -µ 2 ) 𝑇 w 0 = 𝑡 𝜂 ( µ 1 -µ 2 ) 𝑇 η + 𝑡 ∗ ( µ 1 -µ 2 ) 𝑇 w ∗ and have

On the other hand:

Which implies that we can drop the cross term in the expansion of ( µ 𝑇 1 w 0 ) 2 :

We can drop the first without loss of generality as 𝑡 𝜂 usually does not grow with 𝑑 . This implies that we have for 𝛼

Then SGD converges to

Introducing a Lagrange multiplier v ∈ R 𝑛 :

Using Theorem 4 from Akhtiamov et al. [2024],

Denoting 𝛽 𝑖 : = ∥ v 𝑖 ∥ and performing optimization over the direction of 𝑣 𝑖 :

We have

Applying the square root trick to the norm inside the objective we arrive at:

Introducing 𝛾 𝑖 's as the Fenchel duals for ( µ 𝑇 𝑖 w + (-1 ) 𝑖 ) 2 we have:

Performing optimization over w yields:

We drop the 𝛼 2 ∥ w 0 ∥ 2 from the objective because it does not contain any of the variables the optimization is performed over and therefore dropping it will not change the optimal values of 𝛽 1 , 𝛽 2 , 𝛾 1 , 𝛾 2 , 𝜏 1 , 𝜏 2 . We obtain:

The objective above can be rewritten in the integral form in the following way:

Note that

Moreover

And

The latter term can be derived in the integral form as:

Since 𝑝 ( 𝑠 1 , 𝑠 2 ) is exchangeable in its arguments, we see that the latter objective remains invariant under the transformation that swaps ( 𝛽 1 , 𝜏 1 , 𝛾 1 ) with ( 𝛽 2 , 𝜏 2 , -𝛾 2 ) . Therefore, 𝛽 : = 𝛽 1 = 𝛽 2 , 𝜏 : = 𝜏 1 = 𝜏 2 and 𝛾 : = 𝛾 2 = -𝛾 1 holds at the optimal point. The objective then reduces to the following:

Note that 𝛾 can be found explicitly in terms of 𝛽 and 𝜏 as the optimization over 𝛾 is quadratic. Denote 𝑒 0 = ∥ 𝑤 0 ∥ 2 𝑑 . To facilitate derivations, let us introduce a few notations. Let

Thus the objective is

Now taking derivatives w.r.t 𝜏 yields

Now for 𝛽 :

Setting the derivative of the objective w.r.t 𝜏 to 0 :

Multiplying by 𝜏 2 and dropping 𝛽 yields:

Derivative w.r.t. 𝛽

Multiplying by 𝜏 ,

Therefore the set of equations is:

Summing the equations yields

Multiplying both sides by √ 𝑛 2 𝑑𝜏 and introducing 𝜃 = 𝛽 𝜏 we obtain:

Which implies

This can be rewritten as:

We arrive at the following equation for 𝜃 :

The last equation above can be reformulated in terms of the Stiltjes transform as follows:

After finding 𝜃 from the equation above, we proceed to identify the optimal value of 𝛾 . We will do so by taking the derivative of (36) by 𝛾 and setting it to 0 :

Dividing both sides by 𝜃 , factoring 𝑛 out and using (39) we obtain:

Hence, we can find 𝛾 via

Finally, to find 𝜏 , recall (40)

Using 𝐹 2 is quadratic in its second argument we have:

We have

Moreover

Then

Thus, all in all:

The generalization error can be found through 𝑄 GLYPH<18> 1 -𝛾 2 √ 𝜏 2 -( 𝛾 2 ) 2 GLYPH<19> .

I.1 Single direction case.

In the special case when Σ 1 = Σ 2 = 𝜎 2 I 𝑑 𝑑 we obtain as R 0 = 𝑡 2 𝜂 ( 1 -𝑟 ) 𝑑 𝐼 𝑑 + 2 𝑡 2 ∗ 𝑑 ( 1 -𝑟 )( 𝚺 1 + 𝚺 2 ) -2 :

We also have

Multiplying both sides by 𝑡 ∗ ( 1 -𝑟 ) 𝑑 2 𝜎 2 we arrive at:

Note that plugging in w 0 = 𝑡 ∗ ( 𝚺 1 + 𝚺 2 ) -1 ( µ 1 -µ 2 ) + 𝑡 𝜂 η for 𝚺 1 = 𝚺 2 = 𝜎 2 𝑑 I 𝑑 into the expression for the generalization error yields

Taking 𝑄 -1 of both sides and then squaring them we arrive at:

This yields that by taking 𝜌 : = 𝑑 ( 1 -𝑟 ) 𝜎 2

We thus obtain from (41):

We proceed to find 𝜃 next. From the general case we had

Now plugging for the distribution of 𝚺 1 , 𝚺 2 yields

Using (41) we can simplify the last expression above:

Thus we can write for 𝛾

Furthermore, we have for 𝜏 2

Simplifying furthermore yields

Further manipulation leads to

Recall that Γ = 1 -1 𝑄 -1 ( 𝑒 𝑎 ) 2 + 1 and Γ 𝑄 -1 ( 𝑒 𝑎 ) 2 = 1 -Γ . Hence, we obtain:

On the other hand, 𝛾 = 1 -( 1 -1 𝜅 ) Γ 1 2 + 𝜌 4 𝜅 and Γ = 1 -1 𝑄 -1 ( 𝑒 𝑎 ) 2 + 1 combining these fact with 43 yields

□

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main results are stated in Sections 3, 4 along with numerical evidence in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our approach and assumptions are explained in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical results, assumptions, and definitions are provided in sections 2, 3, 4. Moreover, the complete proof of arguments in the main body is presented in the Appendices B, F, H, I.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the necessary details are given in Section 5. Moreover, the code is attached to the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is attached to the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the necessary details are given in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: As the main focus of the paper is the theoretical contribution, the experiments are conducted using specific data distributions whose statistics are exactly known. ( 𝜒 2 , Bernoulli, Gaussian)

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the necessary details are given in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms with Neurips Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Since this paper is concerned with the mathematical foundations of AI.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Since this paper is concerned with the mathematical foundations of AI.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the necessary details are given in Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Since this paper is concerned with the mathematical foundations of AI.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Since this paper is concerned with the mathematical foundations of AI.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Since this paper is concerned with the mathematical foundations of AI.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper derives a new universality result, which can be leveraged to solve a mirror descent optimization problem for a large family of data distributions by relating the solution to a Gaussian distribution with matching first and second-order statistics. This result is then applied to analyze transfer learning for linear models in the setting of regression and binary classification.

STRENGTHS:
The paper derives a new universality result by adapting an existing method. The universality is used to extend results about transfer learning in linear regression models to a larger class of data distributions, and further results are developed in the setting of binary classification. The paper is overall well-written and easy to follow.

WEAKNESSES:
Parts of Assumptions 2 and 3 seem unintuitive to me, see the questions below.

QUESTIONS:
In assumption 2.2, why should the covariance depend on the mean $\mu$?

In assumption 2.3 and 3, how can the condition hold for any matrix of bounded operator norm? It seems possible to construct a matrix for which the condition does not hold.

LIMITATIONS:
Limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the application of transfer learning within the framework of linear models. The authors focus on a model-based approach, where a model pre-trained on a source distribution is fine-tuned on a few samples from a target distribution. Authors extend the concept of universality, traditionally used in random matrix theory and high-dimensional statistics, to the context of transfer learning. They demonstrate that certain properties of linear models trained on large datasets can be transferred to new tasks with minimal data. They also provide a thorough theoretical analysis, establishing conditions under which the transfer learning approach achieves performance close to that of a model trained directly on the target task with abundant data.\n\nSTRENGTHS:\nUniversality has traditionally been applied in random matrix theory and high-dimensional statistics, but not in the context of transfer learning. The authors show properties of linear models trained on large datasets can effectively transfer to new tasks with limited data. The authors explore the conditions under which their transfer learning approach is effective. Combination of theoretical analysis and empirical validation provides a comprehensive exploration of this concept.\n\nWEAKNESSES:\n1. The empirical validation, while convincing, is somewhat limited in scope. The experiments are conducted on a few specific tasks and datasets, which may not fully capture the diversity of real-world applications. Datasets with different characteristics, such as varying levels of noise, feature distributions, and dimensionalities, could provide a more comprehensive assessment of the proposed methods. \n2. The comparison with existing transfer learning methods is limited. The paper could benefit from a more comprehensive comparison to highlight the advantages and limitations of the proposed approach relative to current state-of-the-art methods.\n3. The paper provides certain properties of linear models trained on large datasets can be transferred to new tasks with minimal data, but in real applications, is there any indication, when transfer methods are useful?\n\nQUESTIONS:\n1. Can you provide more details on the boundary conditions under which the universality principle might fail? Are there specific cases or distributions where this principle does not hold?\n2. Can you elaborate on the choice of datasets and tasks used for empirical validation? Are there plans to test the proposed method on more diverse and complex datasets?\n\nLIMITATIONS:\nWhile the authors provide thorough theoretical proofs and some experimental validation, the paper lacks comparative experiments with existing methods. This makes it difficult to fully evaluate the practical advantages and limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

comments: SUMMARY:
The paper studies model-based transfer learning. In this setting, a model is pre-trained on the source data, and the learner aims to fine-tune it on the target data by running SGD initialized at the pre-trained model. The paper focuses on linear regression and classification, aiming to generalize the assumption of Gaussianity to general distributions and investigate the scenarios where fine-tuning on the target data can or cannot be beneficial.

STRENGTHS:
The paper derives a universality result that allows replacing Gaussian data with more general distributions. It then analyzes the generalization error in linear regression and classification for SGD initialized at a model pre-trained on the source data. The study shows that when the noise in the data is high, fine-tuning can be harmful, and just using the model pre-trained on the source data results in a lower error. Conversely, when the noise is less than the error of the model pre-trained on the source data, transfer learning improves test performance.

WEAKNESSES:
The paper attempts to generalize the assumption of Gaussianity but is limited to the linear model, which is inherently restrictive. Furthermore, it focuses exclusively on a specific algorithm—SGD initialized at the model pre-trained on the source data—without demonstrating the optimality of these results or determining whether better results are achievable with alternative algorithms.

QUESTIONS:
It appears that there are some parameters in the paper that have not been defined.

In line 194, "p" is used without definition.

In equation (7), \kappa is referenced without definition.

In line 206, it's unclear what "p(r)" represents.

LIMITATIONS:
The paper does not have any potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers transfer learning (fine-tuning) for linear models in the over-parameterized regime in the proportional regime $k / n \\rightarrow \\infty$.  In this setting, gradient descent converges to the solution a convex optimization problem with linear constraints. This work builds on this result to show that for a distribution $P$, the test error converges to the test error of a Gaussian distribution with mean and covariance equal to that of $P$---in other words, universality holds.\n\nSTRENGTHS:\nThis work brings a fairly atypical and and relatively unexplored perspective to transfer learning. The results hold for a large family of distributions (theorem 1), although limited to linear models in the infinite width regime. Finally, the results indicate that Transfer learning depends on the noise levels of the problem, i.e., one should choose to fine-tune the model if the noise level is very high (remark 1). I unfortunately did not check all the proofs carefully but appreciate the authors attempt at distilling the results into the main paper despite the depth of results.\n\nWEAKNESSES:\nMy main concern is that the setting is not conducive to studying transfer learning. Transfer learning is a fundamentally small-data phenemenon, i.e., the target dataset usually contains a limited number of samples. However, the results (in the proportional regime) do not include: (1) The number of samples in each task or (2) the distance between the two tasks. The only quantity that features in the equations is the noise level. \n\nI also believe the presentation could be better. For example, some text explaining theorem 1 (in english) or explaining what $\\theta$ and and $t$ represent in theorem 2 could be helpful to a reader. I also wonder why the authors chose to call in model-based transfer. I have not heard of this specific term before and it would help if the authors cited previous references to model-based transfer learning. Isn't fine-tuning a more appropriate term. \n\nFinally, the looming question is the connection to deep networks. While perhaps out of the scope of this paper, there are questions about how this affects or relates to the practice of deep learning. Are linear models in the proportional regime informative for how we should do Transfer learning in practice?\n\nQUESTIONS:\n1. Why is renormalization needed in in the classification problem setting? \n2. In definition 2, the authors claim that the assumptions are satisfied in practice. Could the authors expand on line 156-160 and explain the results of Seddik et al. ?\n3. How do I interpret quantities like $t$ and $\\theta$ in theorem 2?\n\nLIMITATIONS:\nLimitations are addressed but the authors could comment on the assumptions and their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}