Features are fate: a theory of transfer learning in high-dimensional regression
Authors: Javan Tahir
Surya Ganguli
Grant M. Rotskoff
Venue: ICML 2025 poster (2025)
Keywords: transfer learning
feature learning
random matrix theory
Primary area: deep_learning->theory

Abstract: With the emergence of large-scale pre-trained neural networks, methods to adapt such "foundation" models to data-limited downstream tasks have become a necessity.
Fine-tuning, preference optimization, and transfer learning have all been successfully employed for these purposes when the target task closely resembles the source task, but a precise theoretical understanding of ``task similarity'' is still lacking. 
We adopt a \emph{feature-centric} viewpoint on transfer learning and establish a number of theoretical results that demonstrate that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch.
We study deep linear networks as a minimal model of transfer learning in which we can analytically characterize the transferability phase diagram as a function of the target dataset size and the feature space overlap.
For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance, especially in the low data limit. 
These results build on an emerging understanding of feature learning dynamics in deep linear networks, and we demonstrate numerically that the rigorous results we derive for the linear case also apply to nonlinear networks.

Features are fate: a theory of transfer learning in high-dimensional regression
Javan Tahir
Surya Ganguli Grant Rotskoff?
Abstract
datasets of the scale used for natural language processing or computer vision difficult, if not impossible. to generate. To alleviate the problem of inadequate dataset scale, the representations of foundation model seem to provide a useful inductive bias for adaptation to a target task. While they are now ubiquitous, transfer learning methods lack solid theoretical foundation or algorithmic design princi- ples. As such, it remains difficult to predict when- and with which paproach--taasfer learning will outperform training the target task alone. By studying analytically solv- able model, we find that the feature space learned during pretraining is the relevant object for predicting transfer per- formance. Of course, adopting a feature-centrrg viewpoint creates model-specific challenges because unambiguously identifying learned features remains an outstanding and diffi- cult characterization problem for deep neural networks. For this reason, in this work we focus on deep linear networks trained with gradient flow, as feature learning dynamics are well-understood in this setting'. We develop an intuitive understanding of linear transfer and full fine-tuning in this model. In contrast to other recent work, we quantify transfer performance relative to training on the target task alone and precisely identify when transfer learning leads improved performance, effectively building phase diagram for trans- fer efficiency. Finally, in numerical experiments, we show that this picture holds qualitatively for nonlinear networks as well.
With the emergence of large-scald pre-trained neu- ral networks, methods to adapt such "foundation" models to become necessity. Fine-tuning, preference opti- mization, and transfer learning have all been suc- cessfully employed for these purposes when the target task closely resembles the source task, but precise theoretical understanding of "task simi- larity" is still lacking. We adopt feature-centre viewpoint transfer learning and establish number of theoretical results that demonstrate that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch. We study deep linear networks as a minimal model of transfer learning in which we can analytically characterize the transferability phase diagram as a function of the target dataset size and the feature space overlap. For this model, we establish rigor- ously that when the feature space overlap between the source and target tasks sufficiently strong. both linear transfer and fine-tuning improve per- formance, especially the low data limit. These results build emerging understanding of fea- ture learning dynamics in deep linear networks, and we demonstrate numerically that the rigorous results we derive for the linear case also apply to nonlinear networks.
downstream tasks have
Related Work
Theoretical aspects of transfer learning A number of recent works have studied theoretical aspects of transfer learning, focusing the risk associated with various trans- fer algorithms. Wu et al. (2020) use information theory to derive bounds on the risk of transfer learning using a mixture of source and target data. Shilton et al. (2017) ana- lyze transfer in the context of Gaussian process regression. Tripuraneni al. (2020) work in a fairly general setting, and derive bounds the generalization error of transferred models through complexity argument, highlighting the importance of feature diversity among tasks. Aminian et al. (2024) study transfer learning in highly overparameterized
1. Introduction
State-of-the-ard neural network models have billions tril- lions of parameters and trained on datasets of similar scale. The benefits of dataset scale are manifest in the as- tounding generalization capability of these foundation mod- els (Bahri et al., 2024). For many applications, however,
'Department of Applied Physics, Stanford University. Stan- ford CA. USA DDepartment of Chemistry, Stanford Univer- sity, Stanford CA, USA. Correspondence to: Javan Tahir <ja- van @ edd...
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
All code to reproduce the results in this paper can be found https: // thub. com/ javantahi ittiaaiirrreeeeeeee www.hirrreeateeeeeeeeeee fate

---

Features are fate: a theory of transfer learning in high-dimensional regression
models, including one hidden layer neural networks, and derive bounds on the excess risk. Bu et al. (2021) study the excess risk of transferred models optimized with the Gibbs algorithm and highlight a bias- varianar interpretation of the generalization performance. Liu et al. (2019); Neyshabur et al. (2020) study transfer learning from the perspective of the loss landscape and find that transferred models often find flatter minima than those trained from scratch. Consistent with our feature-centric viewpoint, Kumar et al. (2022) show that fine-tuning can distort the pretrained features, leading to poor out of distribution behavior. Integral probability metrics on the dataset and target have been used to pre- dict transfer learning performance. For example, Wu et al. (2020); Nguyen et al. (2021); Alvarez-Melis Fusi (2020) suggest that these metries correlate with generalization er- ror on the target task. However, we prove in Appendix A,
regime. Allen-Zhu & Li (2020a:b) give a fairly complete treatment of feature learning for synthetic datasets whose features are readily identifiable. Chen et al. (2023) study synthetic dataset closely related to that in Allen-Zhu & Li (2020a) and describe how generalizable features are impor- tant to out of distribution generalization, a conclusion we also draw in the context of transfer learning.
Our contributions
Since transfer performance depends on the learned representation of source task, we study deep linear net- works trained on a linear target function, a setting in which training dynamics, implicit bias, and generaliza- tion error can be described rigorously.
that integral probability metrics such as the Wasserstein-1 Distance, Dudley Metric and <-divergences such as the KL-divergence need not correlate with transfer efficiency, emphasizing that transferrability is a property of the learned feature space and not the source and target datasets alone.
Within this model, we analytically compute a phase diagram that illustrates how transfer learning performs relative to training from scratch on a given task.
We prove that simple diagnosties, such as distributional measures of source-target distance are insufficient for predicting the success of transfer learning and advance the idea that task similarity should be measured in the space of features instead.
Transfer learning in solvable models Similar to our ap- proach, several theory works have worked with analytically tractable models to more precisely characterize transfer per- formance. Lampinen & Ganguli (2018); Atanasov et al. (2021): Shachaf et al. (2021) also study transfer learning in deep linear networks, but focus on the generalization error alone, not the transferability relative to scratch trained baseline, which obfuscates the conditions for transfer learn- ing to be beneficial. Gerace et al. (2022) studies transfer learning with small nonlinear networks with data generated from a "hidden manifold" (Goldt et al., 2020) and find it to be effective when tasks are very similar, and data is scarce, but do not theoretically describe regions of negative transfer. Saglietti& & Zdehorova (2022) studies knowledge distillation in solvable model, which can be viewed of transfer learning. Ingrosso et al. (2024) study transfer learning in model similar to ours using the replica method and similarly conclude that a feature- based metric for task similarity is predictive of transfer performance.
We also compute the transfer phase diagram for non- linear neural networks and show that the same picture applies to the reproducing kernel Hilbert space (RKHS) associated with the nonlinear features of the pre-trained network.
2. General theoretical setting
We begin by introducing the general theoretical frame- work under which we study transfer learning. We con- sider source and target regression tasks defined by probabil- ity distributions pa(ar, y) and (r( y) over inputs æ e Rd and labels y E R. We focus on concept drift, in which Ps (2. = p( x) and (c y) p( =) for the same input distribution p(x)). The labels are gen- erated from noisy samples of source and target functions Is = f (2)+ and = (x) +ez where f. (x), f: (x) e (g((()) and 6g. 4 ~ (O o2). During pretraining, we train a model with parameters = (c, 0) of the form
special case
Feature learning The feature space of a neural network evolves in the feature learning regime, which should be con- trasted with the static neural tangent kernel limit, in which networks regress functions onto a set of basis functions which are fixed at initialization (Jacot et al., 2018). Recently, there has been an explosion of interest in understanding dy- namics in these two regimes of neural network optimization. Woodworth et al. (2020); Atanasov et al. (2021): Yang & Hu (2021); Kunin et al. (2024); Yun et al. (2021); Chizat (2020) focus on the role of initialization, learning rate, and implicit bias in feature learning. Petrini et al. (2022) highlights the potential for overfitting when training in the feature learning
[(:; +
Σ
co 0)
(1)
the source task using mean squared loss. Note that the features (1x 0) are left general and could for example represent final hidden activations of deep neural network. After pretraining, the model is transferred by training a subset of the parameters O' C 2 on the target task, while leaving θ e' at their pretrained values. To model the
2

---

Features are fate: a theory of transfer learning in high-dimensional regression
modern setting for transfer learning, in which the number of data points in the source task far exceeds those in the target task, we train the source task on the population distribution and the target task on finite dataset D of n independent samples.
distortion under various transfer learning schemes. Let ! denote a deep linear network with I. layers
f(x) - xx W: W2 2............................. WL
(7)
where Wi E R 1xd for E 21......J. .. 1] and WL e Rdi-1x1 x1 For notational convenience we have renamed c in (1) WT and for simplicity we set do d1
(0)
E F (f((,, 0)- y)
(2)
2
1 E Bm (f 0) y)
d - d, the dimension of the data. The parameter matri- ces are initialized as Wi(0) = aWi where a E R. The ma- trices (i) additionally satisfy (19), which is a technical assumption that generalizes common initialization schemes such as He initialization (Yun et al., 2021; He et al., 2015). Since transfer learning relies on learning features in the source task, initialize the network in the feature learning regime - 0. In the following, we assume:
(( 2
where (h(x,, 8)) 1 over the empirical distribution of D. We focus on two widely employed transfer methods, linear transfer and fine- tuning. In linear transfer, the pretrained features <xx; 0) are frozen and only the output weights e are trained on the target task. In fine-tuning, the entire set of parameters O are trained from the pretrained initialization on the target task. In Appendix C, we explore the case when the source dataset is also finite, and show that for fine-tuning, the qualitative picture discussed below is unchanged. To optimize the loss functions (2) and (3), we use gradient flow,
Mi) is the expectation
Assumption 3.1. Assume that the input data E Rd is normally distributed and that each dataset D consists of pairs {<x; b sampled iid from pt with Gaussian label noise of variance o².
Assumption 3.2. We assume that the source and target func- tions are each linear functions in L2 (Rd p); equivalently, (=) = 3 x. f: [x) = S x with 3| = 1|3. = 1.
do: di - Vo,
To control the level of source-target task similarity, we fix the angle 0 between the ground truth source and target func- tions so that 3 3, = cos The source and target loss functions are given by (2) and (3). When training over the empirical loss, it is convenient to work in vector nota- tion ((( X E Roxd and y E R". We study this model in the high dimensional limit in which = n/ remains constant as n, d + 0o.
where we have set the learning rate equal to unity for the purpose of analysis. To assess the performance of transfer learning we compare the generalization of the transferred model a scratch-trained model with the same architecture (1) trained on the target same data from a random initial- ization. We introduce the transferability to quantify this relationship:
ب.m Ily
WLlB where
T= ER - R(x))
(5)
where Ep is the expectation over iid draws of the training set and Rxx and Rue are the generalization errors of the transferred model and scratch trained model, respectively, where the generalization error (or population risk) is given by.
Linear networks have the advantage of analytic tractability, but we note that the representation capacity of these mod- els is limited to affine transformations. Furthermore, the expressiveness of the model is independent of the number of layers. As result, this model may fail to capture aspects of transfer learning that depend strongly on depth separa- tion (Telgarsky, 2016; Daniely, 2017) or other nonlinear phenomena. However, overparameterized linear models, recapitulate many phenomena observed in deep learning, including double descent (Nakkiran al., 2021; Belkin et al., 2019), scaling laws (Bahri et al., 2024), feature learn- ing (Vyas et al., 2024; Atanasov et al., 2021) and, as we show, the impact of feature learning on transfer efficiency.
R = P ((C -)- - (2)))
(6)
We consider transfer learning successful when T > 0, i.e., when the expected generalization of transfer learning out- performs training from scratch on the target task. We refer to the situation T < 0 as negative transfer, since pretraining leads to degradation of the generalization error.
3. Deep linear networks: an exactly solvable model
3.1. Pretrained models represent source features
Because insight into feature learning is required to under- stand the dynamics of transfer learning, we first consider an analytically solvable model of transfer learning using deep linear networks. This model gives us direct access to the evolution of the time-dependent feature space and its
To describe transfer efficiency in this setup, we need to understand the function that the model implements af- ter training on the source task. We can describe the network in function space by tracking the evolution of ((t) = W, Wr under gradient flow, such that
3

---

Features are fate: theory of transfer learning in high-dimensional regression
the network function at any point in the optimization is f(:; = (t) x. The following Lemma establishes that pretraining perfectly learns the source task in the large source data limit.
final generalization error over training data
(1-7)
7 < 1
EpR- =
(8)
2 >> 1>1
Lemma 3.3. Under gradient flow (4) on the popula- tion risk objective (2) with initialization satisfying (19),
Theorem (3.6) is known result for linear regression (Hastie et al., 2022: Canatar et al., 2021; Belkin et al., 2019; Ad- vani & Ganguli, 2016; Mel & Ganguli, 2021; Bartlett et al., 2020), but we provide proof based on random projections and random matrix theory in Section D.5. This expression exhibits double descent behavior: in the overparameterized regime, the generalization error first decreases, then be- comes infinite as y - 1, while in the underparameterizee regime, the generalization error monotonically decreases with increasing 7. As we will see in Section 3.3, this dou- ble descent behavior leads to two distinct regions in the transferability phase diagram. The fact that scratch-traine performance can be arbitrarily bad is result of the implicit regularization of gradient flow this model. This effect can be eliminated by appropriately regularizing the scratch- trained model with weight decay. In the interest of analytic tractability we do not include regularization when training from scratch, but we explore its effects in simulation in Appendix Fig. 5
lim B(t) = Ba
We prove Lemma 3.3 in Appendix D.2. While this result establishes recovery of the ground truth on the source task, it does not describe the feature space of the pretrained model, which is relevant for transferability. To this end, following Yun et al. (2021), we show that in the feature learning regime a + O, the hidden features of the model sparsify to those present in the source task.
Theorem 3.4 (Yun et al). Let the columns of & W1 W.... WL-1 denote the hidden features of the model.
After pretraining
lim lim 4- = =B,07-1 8-000000
for some vector V1-1-
We prove Theorem 3.4 in Appendix D.3. Theorem 3.4 demonstrates that after pretraining in the feature learning regime, the d-dimensional feature space of the model par- simoniously represents the ground truth function in a sin- gle, rank-one component. We refer to this phenomenon as feature sparsification, which a hallmark of the fea- ture learning regime, and has important consequences for transferability, particularly in the linear setting Section 3.3.
3.3. Linear transfer
The simplest transfer learning method is known as lin- ear transfer, in which only the final layer weights of the pretrained network are trained on the target task. In par- ticular, (Wi)ss-1
fixed after pretraining and Wz solves the linear regression problem with features o XW1 WL-1.
3.2. Scratch trained models represent minimum norm solutions
1
W, = arg min
WW,, - yr 2n
For the empirical training objective (3), there are multiple zero training error solutions when the model is overparame- terized y < 1. As noted in Yun et al. (2021) and Atanasov et al. (2021), there is an implicit bias of gradient flow to the minimum norm solution when a + 0
W.ER
When there are multiple solutions to the optimization prob- lem (9), we choose the solution with minimum norm. We characterize the generalization error of linear transfer in the following theorem.
Theorem 3.5 ( (Yun et al., 2021)). Under gradient flow on the empirical risk minimization objective (3) with initial- ization satisfying (19), lima- oi B(t) B, where G the minimum norm solution to the linear least squares problem
Theorem 3.7. Under Assumptions 3.1 and 3.2, and assum- ing the source-target overlap is 0, the expected generaliza- tion error of the linearly transferred model is an explicit function 0f0, 0, the label noise o, and the dataset size
σ² sin?
3 =arg min
1 B 2n
x
X yt
EdRa = =in?
n 2
(10)
We prove Theorem 3.7 in Appendix D.6. The structure of the result in Theorem 3.7 merits some discussion. After pretraining in the feature learning regime a + 0, the fea- ture space of the network has sparsified so that it can only express functions along B, (Theorem 3.4). Since the fea- tures of the network cannot change in linear transfer, the
We prove Theorem 3.5 in Appendix D.4. Knowing the final predictor of the empirical training also allows us to compute the generalization eитоr of the scratch trained model Theorem 3.6. Under gradient flow on the empirical objec- tive (3), the high dimensional limit the expectation of the
4

---

Features are fate: theory of transfer learning in high-dimensional regression
(a)
(b)
(c)
1.00
0.75
=3m
0.25
T
1.0
0.00
0.25
0.5
0.50
0.75
9/3
-1.00
T/6 7/3
x/2
0.5
7
500 to produce labels from linear source function 3 x using the population loss (2). We then retrain the final layer weights sample of -d points (ar,, M/ = 3 a; + e.) where 3 3, = cos and E; ~ NN(0, a = 0.2). and compare its generalization enror that of model trained from scratch the target dataset. (a) The theoretical transferability surface (11) function of target dataset size "y and task overlap Light blue lines indicate the boundary between positive and negative transfer. (h) Top-down view of Fig. 1(a) shaded by sign of transferability. Red regions indicate negative transfer T < O, blue region indicates positive transfer T > 0. (c) Slices of the transfcrability surface (11) for constant 0. Solid lines represent theoretical values, circles are points from experiments. Error bars represent the standard deviation over 20 draws of the target
Figure Linear transferability phase diagram. pretrain a linear network (7) with
main contribution the generalization error sin 2 which can be viewed the norm of the projection of the target function into the space orthogonal to the features spanned by the pretrained network. This is an irreducible error that is the best case risk given that the features cannot change. The second term comes from the finiteness of the training set. Since linear transfer learns from finite sample of training points, minimizing the training error can overfit the noise and the learned function distorts away from the ground truth. I Luckily, since the pretrained feature space has sparsified, the effect of finite sampling and additive label noise decays as 1/n, effectively filtering out the d-dimensional noise by projecting it onto a single vector. Compare this to the generalization the scratch trained network (8). There, features of the equivalent linear regression problem, X. have support over all d-dimensions, so there is irre- ducible error term. The expressivity, however, comes cost. Each dimension of the regression vector is vulnerable to noise in the training data, and the projection of the target function onto the feature space is strongly distorted due to finite sampling (i.e. ~ 7). We can precisely analyze this trade off by comparing (8) and (10). In the limit n, d -> co, the transferability (5)
which is plotted Fig. 1(a). From (11) we can identify the regions of negative transfer for this model, which shaded in red in Fig. 1(b). In the underparameterizeed regime (y > 1), there is negative transfer for all y - sin' In words, fixed y and a, i.e., fixing the number of data points and label noise, transfer efficiency degrades as the norm of the out-of out-of-subspace component increases.
In the overparameterized regime (7 < 1), negative trans- fer only occurs when < 1.This can be viewed as a condition on the signal-to-noise ratio of the target data: SNR ||3, 12//22 1/o². When SNR training never recover the underlying vector Br and pretraining is always beneficial. When SNR 1, neg- ative transfer occurs when 0 (arccos(1 - a), o), T/2) and E (7+, where ? = [[11 + cos2 0 - o²) +
scratch
(1 cos 0 02ح² 4 cos 0J. In the noiseless case 0, this expression simplifies to 0 (0, T/2), cos? 0 < 7 < 1 (see Appendix G Fig. 7). We can view sionless measure of the amount of target data, and cos? o as the amount of power the target function has in the sub- space of the pretraining task. The condition for negative transfer is satisfied when there is more target data than there is power the pretrained subspace. As increases, the region of negative transfer shrinks, since the noise corrupts the scratch trained accuracy. Finally we mention that the two regions of negative transfer in Fig. are separated by
dimen-
(1-7) 2702 sin A 1-
7 <<1
Tu
(11)
sin2
7 >
7-1
5

---

Features are fate: theory of transfer learning in high-dimensional regression
positive transfer that persists even when 0 = T 7/2. We dub this effect anomalous positive transfer, since the pretrained features are completely orthogonal to those in the target, yet transferability is still positive. In this regime, transfer is pos- itive soley because of the disproportionately large amount of data in the source task, not because pretraining learned useful features for the downstream task. By comparing the transferred model to regularized scratch-trained model, can eliminate this effect, which show in simulation in Appendix G Fig. 5. In Appendix G Fig. 4 we demon- strate that distributional discrepency measures are indeed misleading: neither DKL, nor W1 are negatively correlated with increased transferability.
Theorem 3.9. Under Assumptions 3.1 and 3.2, and assum- ing the source-target overlap is 9, the expected generaliza- tion error of the fine-tuned model over the training data is:
RD + +11 (1-7)(1 cos 0) EDRac
7 (14)
Ep Ra =
? >>
where EDRe is the expected generalization error of the scratch trained model
Theorem 3,9 is proven in Appendix D.8. Theorem 3.9 yields expression for the fine-tuning transferability, which is plotted in Fig. 2(a):
3.3.1. RIDGE REGULARIZATION CANNOT FIX NEGATIVE TRANSFER
(7 - 1)(1 2211-2cos00 0) 0
S
Ta
(15)
In the previous section, the network sparsified features that incompletely described the target function, leading to negative transfer given sufficient target data. A common approach to mitigate this kind of multicollinearity in linear regression is to add an lg penalty to the regression objective (9) so that
When the model is underparamettrrized y > 1, there is a unique global minimum in the space of 3 W; ... W.. Since gradient flow converges to a global minimum, (Theorem 3.5), fine tuning loses the memory of the pretrained initialization leading to zero transferability (white region in Fig. 2(b)). When the network is overpa- rameterized, however, there is subspace of global minima. We show in the Section D.8 that the pretrained initialization induces an implicit bias of gradient flow away from the mini- mum norm solution. When the target vector has the majority of its norm along the pretraining subspace, i.e. cos 8 > 1/2, the pretrained features are beneficial, leading to positive transfer. For cos < 1 1/2, however, the pretrained features bias the network too strongly toward the source task, leading to negative transfer.
WL arg min
|&W WEERd 2n
32112
W. 112
(12)
In the following theorem, which we prove in Appendix D.7, we show that the generalization error for regularized linear transfer is strictly increasing function of the ridge parameter X, leading to a larger region of negative transfer for any À > 0 (Fig. 6).
Theorem 3.8. Under Assumptions 3.1 and 3.2, and assum- ing the source-target overlap is 0, the expected generaliza- tion error of the ridge linear transfer model over the training data is
lim R = - 21) 150
4. Student-teacher ReLU networks
cos2 0
(13)
(1 - X)2
In the following. we demonstrate that many of the re- sults from our analytically solvable model also hold, qual- itatively, in the more complicated setting of linear trans- fer with nonlinear networks. In particular, we choose model of the form f(x) 1 Σ coa x) where (y) max(0, y} is the ReLU activation. We scale the model by 1/m to place the network in the mean field, fea- ture learning regime (Chizat et al., 2019; Mei et al., 2018 8; Rotskoff & Vanden Eijndenn, 2022). As in the deep linear model, we choose source and target functions that are repre- sentable by the network. That is, we study this model in the student teacher setting. To vary the level of feature space overlap between the source and target functions, we define a network of m, neurons for the target task, and generate the source network by ablating a fraction μ of the hidden neurons form the target. More precisely, let A be a uni- formly random subset of the index set (1, ... m.) with
Ridge regression attenuates the power of the predictor in all directions of the data, including the direction parallel to the signal. Due to sparisification of Theorem 3.4, & regu- larization is non-optimal and hence regularization impairs generalization by attenuating useful features, i.e., those with 0 < 7/2.
3.4. Fine-tuning
Another common transfer learning strategy is fine-tuning, in which all model parameters are trained on the target task from the pre-trained initialization. For general nonlinear models, analyzing the limit points of gradient flow from arbitrary initialization is a notoriously difficult task. For the deep linear model however, we can solve for the expected generalization error of fine-tuning exactly.
6

---

Features are fate: theory of transfer learning in high-dimensional regression
2.0
(a)
(b)
(c)
0.8
0 - /8
0
-
1.5
0.6
8 33
25
T
0.4
1.0
0.2
1/2 0.5
0.0
05 15
0.2
10
0
0 +/6
0 3/2
0.5
1
1.5
2
7
Figure Fine-tuning transferability surface Using same transfer setup as in Fig. we fine tune all the weights on the target dataset starting from the pretrained weight initialization. (a) The theoretical transferability surface (15) as function of target dataset size y and task overlap 0. The light blue line parallel to the "y axis indicates the boundary between positive and negative transfer, while the one parallel the 0 axis indicates the boundary for transferability. (b) Top-down view of Fig. 2(a) shaded by sign of transferability. Red region indicates negative transfer T < 0, blue region indicates positive transfer T > 0. The white region indicates no transfer benefit T (c) Slices of the transferability surface (15) for constant O. Solid lines represent theoretical values, circles points from experiments. Error bars represent the standard deviation over 20 draws of the target set.
|A| = pm. Then
the teacher, since any superfluous weight directions can be eliminated by setting the corresponding output weight to (16) zero. However, we demonstrate empirically in Fig. 3(a)-(b) that this relationship is preserved at the level of the model's kernel, so that k(x, x²) (T-u)m_ [w (ww;""
1
(xx)
(1 p.)m.
TEAt
Sm a (wn o (w) x²) s (w x²). This observation is
1 mx Ccolw
f f((x))
(17)
analogous to Theorem 3.4: training the feature learn- ing regime causes the model's features to sparsify those present in the target function.
Thus the source has pm" fewer hidden features than the target task, and so the fraction μ controls the degree of discrepancy between source and target feature spaces. In essence when j 0 the source and target spaces are iden- tical. However, as f increases, an increasing fraction of new target features, that were not present in pre-training must be learned. We constrain the hidden features in the model, source, and target the d-dimensional unit sphere wi, ,. E gd-1, As in the deep linear model, we choose ~ (0, Ia), train the source task on the population loss, which can be computed exactly for this model, and the target task on a finite sample of n data points.
Now, linear transfer in this model can be formulated as kerne interpolation problem with this kemel. The general- ization error of kernel interpolation can be separated into an ne-eepeneent component, and an irreducible error term which corresponds the norm of the projection of the target function into the subspace of Le(p) orthogonal the RKHS defined by the kernel:
ED C(n) + +1 ((
(18)
As expected, the norm of this projection increases mono- tonically with μ as shown Fig. 3(d). We show how to compute this projection in Appendix E. In the deep linear setting, || (x))), sin 8, and C(n) 1/n. While the asymptotic, typical generalization error of kernel re- gression has been studied in (Canatar et al., 2021), for the purposes estimating the generalization error of the trans- ferred model, we assume here that this generalization error is dominated by this irreducible term for the large n target
Previous work (Rotskoff & Vanden- Eijnden, 2022; Mei al., 2018: Chizat, 2020) has shown that in the over- parameterized setting >> m., gradient flow will con- verge to global minimizer of the population loss, so that limm-+oo lim f(x) = (x), which establishes that the trained network builds a representation of f. (z) in the
mean field limit. This does not necessarily mean that all of the hidden neurons of the model converge to those of

---

Features are fate: theory of transfer learning in high-dimensional regression
(a)
(c)
220
sepoimon mB
(c)
1.5
1000
Rx
1.0
800
0.5
1.0
10
71.
600
0.0
(d)
2/6
0.00
400
0.5
1P
0.08
TRIP
200
-1.0
0.02
01:03
-1.5
0.2
0.4
0.6
0.8
IUC
/
0.25
0.50
0.70
Figure 3. I. Linear transfer in two-layer ReLU networks We train a two layer ReLU network with m 1000 neurons a teacher with m. = 100 neurons and d = 100 dimensional gaussian data, according to the ablated transfer setup (16), (17). For these experiments, we set the label noise 0. (a) Gram matrix from the kernel of the pretrained model (b) Gram matrix from the kernel of the ground truth source function f (az). The two gram matrices are nearly indistinguishable suggesting that the kemeel sparsifies to the represent features in the source task. Generalization error of the scratch trained model function of dataset size n. fit to power law (d) Norm of aut-of-RKHS component of target function ||P f. (x)]1,,, normalized by target function norm [| (x)]12, as function of excess target function of excess target features μ and dataset size n. We normalize the transferability by variance in the target data. Gray circles represent the point of negative transfer predicted by our theory. Results are averaged over 100
features μ. (e) Heat map transferability
realizations of data and 10 realizations random draws of the teacher.
dataset sizes we consider, just as showed for the deep linear model.
Overall, this demonstrates that we are able to predict the phase boundary between positive and negative transfer in the ReLU case, using our conceptual understanding in the deep linear case.
However, an expression for the generalization error of the scratch-trained model also needed to derive the trans- ferability. We are not aware of a theory of generalization error for infinite width nonlinear networks trained on a finite data in the mean field regime. Intriguingly, however, we demonstrate empirically (Fig. 3(c)) that the generalization error obeys a power law Rx ~ An-" with v = 1.18. By setting our theoretically predicted generalization error of our transferred model ||P" ft (x)]12, equal to the empiri- cally observed scaling law An- for our scratch-trained model, we can approximately identify the point of negative transfer in for any given y (gray circles in Fig. 3(e)). It is clear from Fig. 3(c) that this heuristic for finding the boundary between positive and negative transfer becomes more accurate as the number of target points becomes large, since the n-dependent component of the kernel regression generalization error goes to zero in this limit. The phase di- agram in Fig. for noiseless ReLU networks resembles the phase diagram for linear transfer with deep linear networks in the noiscless setting with 0 (Appendix G Fig.7. 8).
5. Conclusion
In this paper, we highlight the importance of thinking about transfer learning in the context of the feature space of the pretrained model. We rigorously identify the number of data points necessary for transfer learning to outperform scratch training as function of feature space overlap in deep linear networks. We also demonstrate that our understanding of linear transfer carries over to shallow nonlinear networks well. One of primary findings is that transferabil- ity is inherited from the learned features of the pretraining task. In the rich training regime, this can lead to an inabil- ity for the pretrained model to transfer to tasks outside the source feature space. On the other hand, a model trained in the lazy regime is unlikely to outperfrom scratch training, since features are not updated in this limit. This suggests that models trained somewhere along the lazy-to-rich hier-
8

---

Features are fate: theory of transfer learning in high-dimensional regression
archy may be more flexible in their transfer capabilities. In Appendix G Fig. 9 generate sweep of nonlinear mod- els trained with varying degrees of feature learning the source task and show that we can eliminate negative transfer the pretrained model lies optimally between the lazy and rich regimes. These experiments demonstrate that regulariz- ing pretrained models to avoid feature sparsification in the source task is promising direction for improving transfer learning capabilities. With this work, we hope advance the idea that transfer learning performance should be under- stood through the learned feature space of the pretrained model and not as a property of the dataset alone.
Bartlett, P. L.,, Long, P. M., Lugosi, G., and Tsigler, A. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30033-3007)) 2020.
Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical bias- variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849 -58544 2019.
Bu, Y., Aminian, G., Toni, I.,, Rodrigues, M., and Wornell, G. Characterizing and understanding the generalization eгтor of transfer learning with gibbs algorithm, 2021. URL https: http://xxvvvv.//// www.:iii.oooooooooooo..o abs //1//111.11.1 01635.
Impact Statement
Canatar, A., Bordelon, B., and Pehlevan, C. Spectral bias and task mosel alignment explain generalization in kernel regression and infinitely wide neural networks. Nature commanications, 12(1):2914, 2021. 2021.
This work aims to understand the theoretical aspects of transfer learning, and advances the idea that the feature space of the pretrained model determines the capability of the model to transfer to downstream tasks. This theoretical understanding could be used to create data and compute ef- ficient pretraining procedures that maximize transfcrability to downstream tasks of interest.
Chen, Y., Huang, W., Zhou, K., Bian, Y., Han, B., and Cheng, J. Understanding and improving feature learn- ing for out-of-distributie generalization, 2023. URL https: htt///axxvv.oor/bbeee/a /2304. 11327.
References
Chizat, L. Sparse optimization on measures with over- parameterized gradient descent, 2020.
Advani, M. and Ganguli, Statistical mechanics optimal convex inference in high dimensions. Physical Review X, 6(3):031034, 2016.
Chizat, L., Oyallon, E., and Bach, On lazy training in differentiable programming. Advances in neural informa- tion processing systems, 32, 2019.
Allen-Zhu, Z. and Li, Y. Towards understanding en- semble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.098 16, 2020a. URL https: htapp::///rxivvv /tt.vviivvvv.////e//eee//...avv. org/ abs 2tbs/2012.. 2012.09816...
Cho, Y. and Saul, L.. Kernel Methods for Deep Learning. In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
Allen-Zhu, Z. and Li, Feature purification: How ad- versarial training performs robust deep learning. CoRR, abs/2005.1013 2020b. Btthttsss//ariivv fbos2205511199.
Daniely, A. Depth separation for neural networks. In Con- ference on Learning Theory, pp. 690 690-696. PMLR, 2017.
Gerace, F, Saglietti, L., Mannelli, S. S., Saxe, A., and Zdeborová, L. Probing transfer learning with model of synthetic correlated datasets. Machine Learning: Science and Technology, 3(1:0015330, 2022.
Alvarez-Melis, and Fusi, N. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428-21439, 2020.
Goldt, S., Mézard, M., Krzakala, F, and Zdeborová, L. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Physical Review X, 10(4):041044, 2020.
Aminian, G., Lukasz Szpruch, and Cohen, S. N. Under- standing transfer learning via mean-field analysis, 2024. URL Btthtt:::/// /ttp://aaxvvvv/.. org/ abs/
Atanasov, A., Bordelon, B., and Pehlevan, C. Neural networks as kernel learners: The silent alignment ef- fect, 2021. URL httpstps: //t::://///////// http///wwwi..og/b////////o 00034.
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, J. Surprises in high-dimensional ridgeless least squares in- terpolation. Annals of statistics, 50(2):949, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR. abs/1502.01852. 2015. URL
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Ex- plaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27)e2311187111, 2024.

---

Features are fate: theory of transfer learning in high-dimensional regression
Ingrosso, A., Pacelli, R., Rotondo, P., and Gerace, F. Sta- tistical mechanics of transfer learning in fully-connected networks in the proportional limit, 2024. URL https: / //aaxiv..rrg//////.......o.o abs/ /20///240... 07168.
Potters, M. and Bouchaud, J.-P. A First Course in Ran- dom Matrix Theory: for Physicists, Engineers and Data Scientists. Cambridge University Press, 2020.
Rotskoff, G. and Vanden- E. Trainability and accu- racy of artificial neural networks: An interacting partiele system approach. Communications on Pure and Applied Mathematics, 15(9):1889-199)) 2022.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Saglietti, L. and Zdeborova, L. Solvable Model for Inherit- ing the Regularization through Knowledge Distillation. In Proceedings of the 2nd Mathematical and Scientific Ma- chine Learning Conference, pp. 809-848 846. PMLR, April 2022.
Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine- Tuning can Distort Pretrained Features and Un- derperform nittof-Distribution, February 2022.
Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. arXiv preprint arXiv:2406.06115. 2024.
Shachaf, G., Brutzkus, A., and Globerson, A. A theoretical analysis of fine-tuning with linear teachers. Advances in Neural Information Processing Systems, 34:15382- 15394, 2021.
Lampinen, A. K. and Ganguli, S. An analytic theory of generalization dynamies and transfer learning in deep linear networks. arXiv preprint arXiv: 1809. 10374, 2018.
Shilton, A., Gupta, S., Rana, S., and Venkatesh, S. Regret bounds for transfer learning in bayesian optimisation. In Artificial Intelligence and Statistics, pp. 307-315. PMLR, 2017.
Liu, H., Long, M., Wang, J., and Jordan, M. I. Towards understanding the transferability of deep representations. arXiv preprint arXiv: 1909. 1199..20000 12031, 2019.
Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Lanck- rict, G. R. G., and Schölkopf, B. A note on inte- gral probability metrics and $ A.phis-divergencer CoRR, abs/0901 2698, 2009. URL http:://axxi...t. abs /bbs0090.2699.
Mei, S., Montanari, A., and Nguyen, P.-M. A mean field view of the landscape of two-layer neural networks. Pro- ceedings of the National Academy of Sciences, 1 115(33): E7665-E7671, 2018.
Telgarsky, M. Benefits of depth in neural networks. In Conference learning theory, pp. 1517-1539. PMLR, 2016. Tripuraneni, N., Jordan, M., and Jin, C. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33: 7852-7862, 2020.
Mel, G. and Ganguli, S. A theory of high dimensional regres- sion with arbitrary correlations between input features and target functions: sample complexity, multiple descent curves and hierarchy of phase transitions. Interna- fional Conference on Machine Learning, pp. 7578-7587 PMLR, 2021.
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechan- ics: Theory and Experiment, 2021 (02))(1))12400)) 2021.
Vershynin, R. High- Diigh-Dimesiooal Probability: An Introduc- tion with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018
Neyshabur, B., Sedghi, H., and Zhang, C. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Vyas, N., Atanasov, A., Bordelon, B., Morwani, D., Sainathan, S., and Pehlevan, C. Feature-learning net- works consistent across widths at realistic scales. Ad- vances in Neural Information Processing Systems, 36. 2024.
Nguyen, C., Do, T.-T., and Carneiro, G. Similarity of classi- fication tasks. arXiv preprint arXiv:2 11201, 2021.
Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N. Ker- nel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Petrini, L., Cagnetta, F., Vanden-Eijnden E., and Wyart, M. Learning sparse features can lead to overfitting in neural networks. Advances in Neural Information Processing Systems, 1119403-9416 December 2022.
10

---

Features are fate: a theory of transfer learning in high-dimensional regression
Wu, L.., Wang, Q. and Ma, C. Global convergence of gradi- ent descent for deep linear residual networks. Advances in Neural Information Processing Systems, 32, 2019.
Wu, X., Manton, J. H., Aickelin, U., and Zhu, Information- theoretic analysis for transfer learning. In 2020 IEEE International Symposium on Information Theory (ISIT), pp. 2819-2824, 2020. doi: 10.1009/SIT444442000. 9173989.
Yang, G. and Hu, J. Tensor programs iv: Feature learning in wnfihite-width neural networks. In International Con- ference Machine Learning, pp. 11727-11737. PMLR, 2021.
Yun, C., Krishnan, S., and Mobahi, H. unifying view on implicit bias in training linear neural networks, 2021. URL https: // /arxivv.. org/ abs/2010 02501.

---

Features are fate: theory of transfer learning in high-dimensional regression
A. Dataset similarity is not predictive of transfer efficiency
The common wisdom in transfer learning is that related tasks should transfer effectively to one another. While it may seem that closeness of task distributions should correlate with transfer performance, we show that this is not necessarily the case. In particular, we select a member of each family and prove that, within our model, one can achieve positive transfer (7 > 0) with distributions that arbitrarily far apart. Two functions representable with the same features can be "far apart". We formalize this notion with the following theorem.
Assumption A.I. We assume f E T22 (R. .)) and for each at E Rid we define the random variable y : R IR through the relation y f()) + e with e o2) Let (r y) denote the joint probability density of æ and y. We assume + C (p) is a linear subspace with orthonormal basis { M and M may be infinite.
Theorem A.2. Assume A.I. Then for any f E ④, and any there exists g E 4 such thut
7s(ps- Pg) 2
where ya(p,p") is the Dudley Metric. Similarly, for any I E 0, and any 5 > 0 there exists E O such that
DRL(Pyllp)) 20
where DKL(Pyllp,) is the Kullback Leibler divergence.
We prove this theorem in Appendix D.1. We note that this theorem also holds for any IPM over function class that is larger than the class of Bounded Lipschitz functions. In particular, the theorem holds for the Longe-Kantorovich (W;) metric, since any function that satisfies || llB. S 1 also satisfies [[ S 1.
Theorem A.2 demonstrates that for a given source distribution, one can always find a target distribution generated from the same feature space that is arbitrarily distant with respect to these metries, perhaps creating the illusion that transfer likely to fail. However, even when the distance is large, if the source and target functions lie in the same feature space and pretraining creates basis for this space, transfer the target task will be positive, since only the output weights need be relearned in the target task. We show this is indeed the case for deep linear networks in the following section.
B. Initialization assumption
Following (Yun et al., 2021) we place the following constraint on the initialization for some À >
w W - W141
W (41 & Al
(19)
To our knowledge, this is the most general assumption on weight initializations in the literature that Ieads to the implicit biases that are crucial for our analysis. This initialization scheme generalizes that in Wu al. (2019); Atanasov et al. (2021).
C. Fine tuning transferability with finite source data
In the main text, we consider training the source task on the population loss, which can be understood as having infinite data points in the source task. Although the number of source data points typically dominates the number of target data points in real-world applications, is interesting to consider the scenario in which the dataset sizes are comparable. To this end, we consider a source task with ni, = 7 data points and a target task with n, = 7pd data points, generalizing the setup 3. Since the source dataset is now finite, we also add label noise the source task so that the labels are generated as
Us = B.. +
(20)
y1 3 +
(21)
where 6, ~ ~N(0 oz3) and 4 ~ NO, of). Aside from these changes, the setup in the following is the same as that in 3. We derive the following expression for the expected generalization error using full fine-tuning, which we prove in Appendix D.9.
12

---

Features are fate: theory of transfer learning in high-dimensional regression
Theorem C.1. Under assumptions above, and assuming the source-target overlap is 0, the expected transferability of the fine-tuned model over the training data is:
2- o 7o( - (1(111)111 220s-2cs000
Tas 7t. S
I-
Tr. ED, Ryc ED,,D, D. Rr
( 1)(1 cos 0) ----00
is 1 > 7t
(22)
0
7/
Note that this expression agrees with (15) in the limit % determined by Ys. since Yt only enters the expression via global multiplicative factor. Secondly, without label noise in the source task, the phase diagram is the same as in Fig 2. That is to say, fine tuning does not depend on the amount of source data if there is no label noise. In the general case, for fixed Ya the boundary between positive and negative transferability occurs at fixed value of
and that the negative transfer boundary is completely
-+1-2
cos
I-7 %
7/s
<
(23)
+7
Therefore, the fine tuning transferability phase diagram with finite source dataset is qualitatively the same as that with infinite source data (Fig. 2), but the position of the phase boundary is modulated by the size of the source dataset.
D. Proofs
D.1. Proof of Theorem A.2
We begin by recalling the definition of the Dudley Metric
(p( 4)
sup || | =111|
Exhh
Eigh
(24)
(25)
By conditioning (2, y) and (r, on x, we write
- -(x-1(0)2
-xx o(=))
7g(Ps.Pg)
sup
h(2, u)e
20
h(2,, y)e 20
p(x)dxdyy
(26)
2
(cos(y)
S(v)) 20
cos(y)
- 11-111)12
20
p(x)dxdyy
(27)
2TTO²
2
002122
/ (cos(f(x)))
(28)
e 000212
/ [5(0)) + g(x)² p(r)dx
(29)
(30)
The first inequality follows from the fact that || ([ss(t)) ||L 1, and the second follows from the identity cos(x) x² 2 cos(2) 2) - =² for any x, E IR. We can expand f in the orthonormal basis { as f = ΣM a,0,, so that
f [(x))p())dr Easa, / ((0)) Σap a?
(31)
2.j
Since, E (p),, the sum on right hand side of (31) converges to some < 00. We can choose
26602/2_a Σ M
Dis agd; which completes the first half of the proof. To prove the result about the KL divergence,
13

---

Features are fate: theory of transfer learning in high-dimensional regression
directly calculate PRL(P)llpg)
/ p(x)e
(u- f(ri)
(y) g(x))²
(y - (x))2
DRI(psllpg)
20
drdy
(32)
V2TO2
20²
20²
V2TO2 / p(x)e
20
[o(z)) =(x))2 - 2yg(x) + 2yf(2) drdy
(33)
20 [11f1 I191it 2(f.g)]
(34)
|ls -gll (z(e))
(35)
2
For any 8 > 0 we can choose g -af with
084/2 M
which completes the proof.
D.2. Proof of Lemma 3.3
We proceed by bounding the dynamics of the loss by an exponentially decaying dynamics, proving convergence to a global minimum. Then we show that the value of 33 at global minimum is unique. To begin, note that the matrix
D. = W/ Wi W4+1 W
(36)
an invariance of the gradient flow dynamics, that D(I) = D(0) a² a2(, W W for all time (Atanasov et al., 2021; Kunin al., 2024; Yun et al., 2021). Let r (Wi W WL 3a) and note that
=
VIL. Wi)
(37)
E=1
Σ VICI2
(38)
(39)
W W r1
(40)
< 22 (WT-1...Wo ... WT)
(41)
(42)
where min (WT-1 ... W is the smallest singular value of WT_ ... WT. To proceed we bound a (WT_
w()
away from zero by showing that WI-1... W, WTWT" W 1 is positive definite
W W W W W W WWW? + Di) WI-1
(43)
> WT_ W W ... W
(44)
(WI-, W
(45)
(WLWH + D2)
(46)
> (a²2)--1
(47)
where we have used the conservation law (36) and the initialization assumption (19). We now have
c < -2(
(48)
C(t) < ((0))-2(03))) -200
(49)
lim L(t) 0 (+0
(50)
Since the loss converges to zero, lim W,W, W1 limy-soe 3 = Ba. which is unique. Note that while this solution
unique in function space, degenerate in parameter space.
14

---

Features are fate: a theory of transfer learning in high-dimensional regression
D.3. Proof of Theorem 3.4
To prove the feature space sparsification, rely on the following Lemma, which is proven in (Yun al., 2021) (see Section H.2). So that this work is self-contained, we include the proof here.
Lemma D.I. Under gradient flow on the population objective (2) or the empirical objective (3),
Wi = ((())))(())n((++(((())))
(51)
for all time. Furthermore
lim lim (1)+1(1)² (1)) =1 a-0
(52)
Proof. To prove Lemma D.1 we bound the difference |
Willep which is equal the norm of the subleading
singular vectors of W2 and show that this bound is proportional a² The argument here follows that in ((Yun al., 2021)).
Taking the trace of both sides in (36) have
Will
|| = a² (1 W(+1
(53)
W 112 = a² W 1|? (
(54)
Will - |WL13 = a² WI (2
(55)
Let uz, be the top left and right singular vectors of W. To bound the maximum singular value of Wi we have
Will's lllp oo W/ Win 2
4 T
W.WWu
(56)
T (D + W W41)uu41
(57)
W1H 1Pg + az (W. W, W.+ W 1
(58)
W1+1 11g a² (I] (+1 Iep Willpp)
(59)
Summing this inequality from to L - 1 we have
Will 2 1 + +a² a
(60)
Combining (54) and (60) have
will 1 S ( | Wilpe 1
(61)
This shows all of the parameter matrices are approximately rank one with corrections upper bounded by O(a²), proving the first claim. To show the alignment of adjacent singular vectors we again take advantage of the invariant quantity (36)
oa Wh+ WT = o/ W Win of Dio
(62)
ot a² Wt W: Wen1 W W4lppp
(63)
we also derive the following upper bound on (63)
of W W = of ( W WTT, 14 (1:111141111
(64)
8 (of 2 | |1+1
(65)
combining these two bounds
of s (o u+11 + a² W/" W, - W. W + +|| [| 11: -
| W1+1 11
(66)
< 22 (of u + o² W/ W. W 11 + a² W/ ||22|1 W - ||
(67)
15

---

Features are fate: theory of transfer learning in high-dimensional regression
where have used the result derived in the previous proof for the second inequality. Finally, derive an upper bound on this quantity
ot 2 u14 W W0
(68)
2 s a W/ Wi Wi+1 W 1
(69)
We combine the upper and lower bounds and divide by s to conclude
(vf 11+1)² 2 1 a² Ci
(70)
$441 G 2 W? W WT WT + W W&ll W lepp
(71)
This proves that adjacent singular vectors align as long as the singular values are bounded away from zero. To show that this requirement is satisfied at the end of training, note that in the proofs Lemma 3.3 and Theorem 3.5 we show that gradient flow converges to global minimizer of the loss. Let ÿ lime X WiWW W.... WL denote the final network predictions. Then
| X
- lim WWW.2... Wella - lim 2 to
(72)
lop
If d 2 n. y is just equal the vector of target outputs which is larger than by construction. If d <, n, is the projection of the targets into the space spanned by the rows of X, which is almost surely non-zero vector. This implies that
L s 0
lim
(73)
which implies that the individual singular values are bounded away from zero at the end of training. In the population training case, the proof is nearly same, replacing ý = lim,-+00 W1W, WL = Bg
By Lemma D.I, we have
Wi W.... WL-1 of
(74)
after pretraining, for some e R. However, from Theorem 3.5 we know that after pretraining
W Wz = 3
(75)
cui (VL, Wx)
(76)
(77)
where we have used Lemma D.1 in the third equality to eliminate the inner product between the adjacent singular vectors. The possible factor of -1 can be absorbed into the definition of u, This implies
W.W... WI-1 = =3,0f_1
(78)
D.4. Proof of Theorem 3.5
This proof follows (Yun et al., 2021) closely but extends their result to the case >> d. We first show that gradient flow converges to global minimum of the empirical loss (3). We then show that as a + 0, this minimum corresponds to the minimum norm least squares solution.
Part Gradient flow converges to a global minimum
This proof follows the same logic as the proof for Lemma 3.3. First, define the residual vector XW, W.... Wt W. Then write the empirical loss as
L =
||r =
2n (( ||r
(79)
2n
16

---

Features are fate: theory of transfer learning in high-dimensional regression
where rj is the component of in im(X) and T1 is the component of in ker(X*), ). Since X X W2 ... WL E im( the global minimum of (79) is equal to ||r l12- Therefore, to show that gradient flow converges global minimum it
is sufficient to show that lim (f) (t) 12 = O. Let Pi and P1 be the orthogonal projectors onto im and ker(X
respectively, so that Li == || = ( WL y and L1 == ||r = (XW, WL y))12
Then we have
i = Σ
VIL Wi)
(80)
Σ (V ViC)
(81)
1=1
Σ (IVVC 12 + (V,,,,
(82)
Taking the gradient of L1 we have
W W'x'P
WT, =0
(83)
So
L 1
41 =
(84)
1=1
< -11V
(85)
-I|WI-1....
WT x P1
x
(86)
S 22 min (WE.... W X Pi
(87)
where miin W ... WF) is the smallest singular value of WT L-11 WT. From (43)-(47) we can bound this quantity away from zero Then we have
4 - X Pir C
(88)
S 2(a Aminn
(89)
where Amiu is the smallest nonzero eigenvalue of XXT. (0)e ())())e=(a")) X which proves lim;~+ool a,n,d, and we take the limit a + 0 after i + xc.
The solution to the dynamics (89) L <
= 0. Note that this part of the theorem holds for any
Part 2: as + O. gradient flow finds the minimum norm interpolator
In the case n > d, the least squares problem () is overdetermined so the solution is unique. That is, the unique solution is trivially the minimum norm solution. In the case n I d, there are multiple ((t) that yield zero training error. Lemma D.1 shows that the parameter matrices are approximately rank one at all times and u441 and on align the end of training as a + 0, which means that
lim lim 3(1) = lim lim W.W, Wr. cut 000 +50
(90)
0-40 +oc
where e 0. Next we show that u, E row row(X), (X). We can break W1 into two components W!!l and W, where the columns of w are in row(X) and the columns of W are in ()) The left hand side of (83) also shows that the gradient of W is zero, which means that this component remains unchanged under gradient flow dynamics. Therefore we have
W | (0)l F a Will
(91)
which vanishes in the limit a + O. This implies that u, E row(X) at all times. The only global minimizer with this property is the minimum norm solution. As a final comment, we note that this thcorem is also proven in Atanasov et al. (2021) using different techniques.
17

---

Features are fate: theory of transfer learning in high-dimensional regression
D.5. Proof of Theorem 3.6
Let B 4o0 W W ... Wt. From Theorem error at the end of training be written
X Xte, Then the average generalization
R = Ex 3 - 212
(92)
= -1+Ex 1/112 2Ex 2 (1)
(93)
=1+Ex B (x X )d X 3) + Ex - (x+) x+e 2 (x=x)
(94)
-f (x + X 1,3, + Ex,,,, x
(95)
=1-Ex Ex Prow(X 3e 11 Ex
(96)
where we have used the independence of and X, well the fact that the operator X X is the projector subspace spanned by the rows of X, (x) Since the entries of the data matrix X are independent Gaussians, the n-dimensional
subspace row(2) (X) ) is uniformly random in the Grassmanian manifold Gn.a (Vershynin, 2018), so )
Gr is random
projection of B.. Then
E row (X) 3.11 =
(97)
which is classic result the theory of random projections (c.f. (Vershynin, 2018) Lemma 5.3.2). We now turn to the final term in (96). Let (oi)ismin(n.d) be the nonzero singular values of the data matrix X. Then
min(n,d)
Ext(((Xt) Ex
Σ
(98)
of
First take the case y < 1. Then there are n nonzero singular values of X, which are the eigenvalues of the Wishart matrix C 1 and
(x
2 E Extr(C n
(99)
--7 lim - [[(r(([(-))11) 2
(100)
-- lim sc())
(101)
In the second line we have introduced the complex variable z, which casts the quantity of interest as the 2 - limit of the normalized expected trace of the resolvent of C. In the limit of large n, this quantity tends to the Stieltjes transform of the Wishart matrix (z)),, which has closed form expression (see (Potters & Bouchaud, 2020) Ch.4 for a proof).
7) /= (1 vJ
v7)²
lim (z) lim 200
(102)
8-0
27z
1
(103)
so (xxr((x+)XXXX 12g for 7 < 1. In the case 7 > 1, there will be terms in the sum (98), which are proportional
to the eigenvalues the covariance matrix 1
n _ X. If we define n' d, d' = n, n'/d' and X' x E R='xd' xd'
equations (99) (101) hold under the substitution 7 T. So Ex X x 2" --1 for 7 1. Putting everything together we have
(1 2172 7 < 1
Ex, R =
(104)
9>1
1-1
D.6. Proof of Theorem 3.7
Theorem 3,4 implies that the pretrained feature matrix is + = (X(3))TT. easy to compute
Since & is a rank one matrix its pseudoinverse
X8,12 (
(105)
18

---

Features are fate: theory of transfer learning in high-dimensional regression
The coefficent vector 3 after linear transfer is
3 = WT WL-1 WL
(106)
= -W... WL-1 2
(107)
113
(108)
where
B X yr
b
(109)
B X 3
3 X X 3r
3 X
(110)
B
X
B
X
As in the proof of Theorem 3.6, we can write the typical generalization error as
Ex Ru =133 -.112 =1+ Ex, 2 cos 0Ex.eb
(111)
(112)
To proceed, we can write 3: cos 03, + sin dv for some vector 1 Bs, and introduce the independent n nn-dimensional Gaussian vectors 2 = x3 ~ (0, I,) and w Xv N(O In). With this change of variables we have
Ex b = e
(113)
coss A
(114)
Ex Ex, b² Ez, w. eebb
(115)
= cos2 0 + (sin 0 + +
(116)
|
The integral E2 ||
can be solved exactly
e Σi- 1//2
Ex
(117)
= 13
(2 nn/2
30 Σ j=1 zj
Sn-1
p 2 dr
(118)
(2n)m/2
Sn-1
'47 -2 -2
(119)
4n²/2
Sn-1
4mm12 1
2 -
(120)
(121)
11 2
which completes the proof.
D.7. Proof of Theorem 3.8
We begin by writing down the solution to the optimization problem (12)
Wr ( + -1 31
(122)
As in the proof of Theorem 3.7, we have
$ = x33))o711
(123)
W W ... WT-1 = G,v
(124)
19

---

Features are fate: theory of transfer learning in high-dimensional regression
Combining these expressions solve for the linear function the network implements after transfer learning with ridge regression
8 W2W2 ... W
(125)
B.vL 1 (1,11 mAIa) ULL
(126)
(x,3.)²
Bx
(127)
1x n2
As in the proof of Theorem 3.7, we write B. cos 0,3g + sin dv for some vector 1 Gg, and introduce the independent
n- n-dimeen dimmensioonl Gaussian vectors X,3, (0, In) and expression for the generalization error of ridge linear transfer:
Xv NN In). Then get the following
Rx || 3.11 =1 + (cos² 0) (n 2, (sin 0 o c2sss² (occos0)) 0))2(n, A)
(128)
(129)
where we have used spherical coordinates to define the following integrals
||=1| Th n++ 1 (1|=113 + nA)? ||=1|=
Sn-1
(m, A) = E=
dr
(130)
(2m)m//2
Jo
(r² + n2)²
Sn-1
p 22 nn nd
12 A) Ez
dr
(131)
|| + n2
(2m)n/2
We evaluate (n, A), 1 + 2) and 12 A) for large n. To avoid cluttering the notation, we ignore the coefficient Sa-1 while solving the integral and restore it at the end of the calculation. Then
(27)4/7
wn/2e- (2u + nd)² du
[n, )) x 2n/2
(132)
t (2nt + nAya dt g(t)e dt
= mn2n)mm/
(133)
= n((n)m/2
(134)
2m n (tto) ((to)e
2 n(2n)n/2
(135)
We have introduced the change of variables / the first line, u/n in the second line, and finally evaluated the integral for large using the saddle point method. In the last line, tu is critical point of f(t) 1 log t and ((()= = (2nl + mx)-2, Differentiating (() and setting equal to zero we find to 1/2. So for large n,
Vinn
I (n, A)
(136)
(n n2)2
We can now restore the angular coefficient to the integral
(nn,)) = Sn-1
1/nn"/2e--0
(137)
(2T)m122
(n + nd)²
n/2
12
-n/2
n 2 / 1/2 --n21 n/2
(138)
v m
2
(n + nd)2
(139)
n11 x)2
where we have used Stirling's approximation in the second line. Therefore, lim (n, X) = We stress that although the integral was approximated the saddle point, the limit n Do is exact since corrections the saddle point value are
20

---

Features are fate: theory of transfer learning in high-dimensional regression
subleading in n. Similar calculations yield
1 1 (1 X)2
(140)
1
T2( = 1+
(141)
for large n. Plugging this into (128), we have
(1 + 2X) cos2 0
lim Ex R2 n-00
(142)
(1 + X)2
This is a strictly increasing function 2 for any E =0, x/2), which implies that the optimal regularization value is x =
D.8. Proof of Theorem 3.9
The proof involves slightly tweaking the proof of Theorem 3.5. Since the source trained model obeyed the initialization assumption (19), the invariant matrix (36) is equal to its value at initialization before pretraining throughout fine tuning as well. This implies that the first half of the proof of Theorem (3.5) holds in the fine tuning case and the model will converge to global minimizer of the training loss. The invariance throughout fine tuning also implies that (90) holds and that W,- does not change during fine tuning, and remains fixed its initial value from pretraining. Therefore, by the proof of Theorem 3.7, at the beginning of fine tuning, Prow (X) j will evolve to the minimum norm solution. Combining these results, after fine tuning,
Bg and (I P Prow x) x)3 is the component of u, that does not evolve. Meanwhile,
lim lim Bt((t) = BBc + (I 1-Proc row x ))3 a-
(143)
where ABc is the minimum norm solution. We can now write the expected generalization error
Ex, Ex, Brll
Exx.RRs + Ex |1(T [ Prow! )1a11 21E (Bi, (I (ow[x)))9..
Exc + max (0, 7) 2Ex (Br, (I Prow(x))3..)
Rec + max(0, 7) cos 0E (Bs, P Prow( x )
2 sin 8Ex (V, (I P Prow(x)) 13)
R.x + (0, (1 cos 0) 2E sin 0ip, (I Prom(x) (
where we have used the fact that P Pr, is random projection in the proof of Theorem 3.6 and set B. cos 0/3g + sin dy for some v 1 B.. The final term equal to zero for the following reason. The operator I - Pjow(X) (X) is a random projector onto the d - n dimensional subspace orthogonal to row Since the uniform distribution of random subspaces is rotationally invariant, we can instead fix a particular subspace and average over Bg ~ Uniform Un) Using rotation invariance again, we can fix the projection be along the first d coordinates of B.. Then we have
n-d
E(v, (I P P = Σ v BB.)
(144)
= 0
(145)
This completes the proof
D.9. Proof of Theorem C.1
To begin note that by Theorem 3.6, the network will implement the least squares solution to the source task when it is trained with gradient flow finite size data set. Denote this vector 3rmm X y.- By the same reasoning as in Appendix D.8, after fine tuning
lim lim Bn (1) Brc + (I P.)
(146)
0-40 1-300
21

---

Features are fate: theory of transfer learning in high-dimensional regression
where P. is the orthogonal projector onto the space spanned by the rows of X, With this expression we write down the expected generalization error of the fine-tuned model
ex X., -c -
(147)
= , 3,, Bac 11 Ex,,x,,5, F.. |1 P.) - 2 ,,,, ( (I- P.)
(148)
Ep, Rec + max (0, - -t X),x,,,.. 113....1111... -2x,, Bf - P:),3
(149)
where we have used the result on random projections employed in the proof of Theorem 3.6 along with the independence of the matrices X, and X, Let P. denote the orthogonal projector onto the rowspace of X. Then we can write
Ex,,e, 1,3.,,,11..1. Ex.,e. .
Pg3 + 2
(150)
-x,,c 1 +
(151)
diza
Ys' + To 7/8 <1
(152)
1 7.-I 7/ >
where the final line follows from the same reasoning as in the proof of Theorem 3.6 (Appendix D.5). The final term 149 can be computed using similar techniques those Appendix D.8. Let B. av + bB3 for some 3, in the subspace spanned by Br and Bz. Then
Ex,,x,,,, B/ -I-
P.) = ex aBx v² P.) + Ex,,, 3 (I- P.)..
(153)
Ex,,,, 3r 4I 3.)
(154)
Ex,,, bE, |1(1 Px2
(155)
max((0, 1 111-)-EE--- xxx bl1Bal12
(156)
3 cos (0, 7+) min(1.74
(157)
(158)
In the second line we have used the same reasoning as the end of Appendix D.8, in the third line we have used the properties of orthogonal projectors, and the remaining lines we have used properties of random projections that are discussed in detail in Appendix D.5. Putting the results above together completes the proof.
E. ReLU networks
In this section, we describe how to compute projections into (and out of) the RKHS defined by one hidden layer ReLU network. Consider a network (xx and target function f. (xr).
m c x)
f(x)
(159)
f=(x)
co(w
(160)
m.
The feature space of the model is span(o(w? xi) in (2(p) To form projectors into this space and its orthogonal complement, introduce the Mereer decomposition. For any positive definite, symmetrie kernel k X x X - R we can define features through partial evaluation of the kernel, i.e., Φ(x) k(-, x). This kernel also induces a reproducing kernel Hilbert space (RKHS) via the Arone-Aronszaj theorem, which is defined as the set of all functions that are linear combinations of these features,
H4. { - Σ agkl,, zi) for some M E N, a, E R, EX
M
(161)
22

---

Features are fate: theory of transfer learning in high-dimensional regression
The associated norm of a function f E H2 is given by
M
Il
a,k a,k(z;, 22)
(162)
We can also define the operator T3 L2(D) + L2(p) with action
T&f = f dx' (a(a,))())))
f
(163)
The spectral decomposition of this operator, {AZ, 31 is known as the Mercer decomposition and the eigenfunctions form a basis for (p)). The eigenfunctions (x)) satisfy
Tesh-
Artf2
(164)
where Å the associated eigenvalue. The eigenfunctions with non-zero eigenvalue form basis for the RKHS Hu- Given a function f = E C///2 one can show by direct computation that
M = Σ G/ 1=1
(165)
which also demonstrates that functions with support eigenmodes with zero eigenvalue are not in the RKHS. If we can construct the Mercer eigenfunctions we can build orthogonal projection operators into the RKHS and its orthogonal complement. To begin note that for Gaussian data, p(x))= = (0, Ia). we can exactly compute the expected overlap between two ReLU functions in terms of their weight vectors (Cho & Saul, 2009):
(o ()) = (1) ())
(166)
1 2x
(
(V1-u;,
u. u( arccos
(167)
where usj
wf w; |willa wylla
With this in hand, we can define the following matrices:
Kij (aw ar)o (w) a)) is
(168)
m. (aw x)) (w x)) La
(169)
K
(aww) x) /z x))
(170)
mm.
The Mercer eigenfunctions can be constructed by diagonalizing the matrix K. 2j is an eigenvector K with eigenvalue
X7, then
(x)
(wf x)
(171)
mn
a Mercer eigenfuction with eigenvalue X7. which can be verified by plugging the expression into the eigenvalue equation (164). Since the feature space is m-dimensional, know that these eigenfunctions span the RKHS. We now write down expressions for the projections of (x) into this space and its orthogonal complement
1 T K 'Kc. m,
(P)
(172)
1 TTKTK" K -
[[),,( 111f-12, (
C.
(173)
m,
14
23

---

Features are fate: theory of transfer learning in high-dimensional regression
F. Experimental details
F.. Deep linear models
For the experiments in deep linear models, we train a two layer linear network with dimension d 500. We initialize the weight matrices with random normal weights and scale parameter a 10-5 To approximate gradient flow, we use full batch gradient descent with small learning rate 7/ 10 We train each model for 10 steps until the training loss reaches 10-6 We perform target training for 20 instances of the training data and grid of dataset sizes and values of
F.2. ReLU networks
For the experiments in shallow ReLU networks, weight matrices randomly the sphere and the output weights initialized at 10-7, We approximate gradient flow with full batch gradient descent and learning rate 0.0lm and train for 10x iterations until the loss reaches 10-6 For training with finite dataset we use 100 realizations of the training data, and average over 10 random initialization seeds.
the parameters d 100, m 1000, m. 100. We initialize the
G. Additional Figures
(a)
(b)
0.4
0.4
0.2
0.2
T
T
0.0
0.0
0.2
0.2
0.4
-0.4
20
40
526
527
528
529
DkI(Pallet)
(1 Pt)
Figure 4. Transferability is not predicted by g-divergences integral probability metrics We generate source and target distributions Ps. Pt according to setup Section and plot the transferability (5) function of (a) KL divergence DKL (p. lp:) and (b) the Wasserstein 1-metric. The KL. divergence can be computed exactly in this setting (see Section D.1). 1). Wi is computed from finite samples using the algorithm Sriperumbudur et al. (2009).
24

---

Features are fate: theory of transfer learning in high-dimensional regression
(a)
(b)
15-
7
1000
00.75
6/10 9.5
472
Figure 5. Regularizing scratch training eliminates anomalous positive transfer. Simulated linear transfer phase diagram for I, = 2, 0.2, = 500 (a) with optimal weight decay in the scratch training and (b) without. To the weight decay hyperparameter, sweep over grid of Awd E 0, 10 10-3 10 10= and choose the model that has the lowest generalization error. The transfer regularized plot (a), the spike positive transfer along is eliminated, as the regularized scratch trained model does not undergo double descent.
learning procedure identical to Fig. only scratch training altered.
theory
experiment
0.24
0.22
RA
0.20
0.18
0.16
0.0
0.2
0.4
0.6
X
Figure Ridge regularization leads to worse generalization linear transfer. Linear transfer generalization error for r 0.5 as function of regularization parameter X. The generalization error is strietly increasing function of X, which implies that the optimal regularizer is A. = O. Solid line is theory (3.8), points are experiments. Error bars represent the standard deviation over 20 realizations of the target dataset.
25

---

Features are fate: theory of transfer learning in high-dimensional regression
(b) 2.0
(a)
(c)
8 =
8 = 4/
0.5
0 =3 /8
T
1.0
0.0
0.5
2/2
x/3
a/2
0.5
1.5
2
7
We pretrain linear network (7) with L 2 and 500 produce un-noised labels from linear source function y = 3. x using the population loss (2). We then retrain the final layer weights on a sample of n. = -d points (ær,, Mr = 3.T x.) where 88. 3.
Figure 7. Linear transfcrability,
and compare its generalization error to that of model trained from scratch (a) Thcoretical transfcrability surface (5) as function of the number of data points
target dataset. n//d and task overlap 6. (b) Top-down view of and blue indicates positive transferability that transfer is always negative when > 1, since the scratch trained model can perfectly learn the target task as there is no label noise.
(a), shaded by sign of transferability. Red indicates negative transferability
0. Note
(c) Slices (a) for constant 0. Solid lines are theory, are from numerical experiments. Error bars represent standard deviation over 20 draws of the training data.
(a)
(b)
0.75
1.5
0.50
10-
1.0
0.25
0.5
0.5
0.50
00.75
0
01
-1.00
x/6
n/3
4/2
7/0
7/3
R/2
8
Figure Linear transfer theory vs. experiment Identical Fig. 7(b), but shaded according the value the transferability. (b) Results of numerical simulations with = 2, d = 500
26

---

Features are fate: theory of transfer learning in high-dimensional regression
regularized
2
unregularized
1
T
-1
200
400
600
800
1000
n
Figure Regularizing pretrained models toward the lazy regime eliminates negative transfer: We train two layer ReLU network on the transfer learning task defined by (16) and (17) with 0.9, m 1000, m. 100, d 100. During pretraining, we include a regularization term A >Em!l || uD; w where w/ is the random initial value weight vector w; This regularization prevents the weights of the network from straying far from their intital values. When À + 0o, features not updated and model operates in lazy regime. We generate a sweep of pretrained models for À E [0, 10 10-3 10-2 10- We then lincarly transfer cach of these pretrained models the target task and choose the model with the best generalization error (blue). The transferability degrades with target set size as expected, but the optimally regularized pretrained model avoids negative transfer. while the fully rich model (pink) transfers poorly nearly all dataset sizes considered.
27


Reviews:
[{"review_id": "so4qTp4eVt", "reviewer": "Reviewer_65EM", "summary": "The authors develop a a feature-centric theory of transfer learning, based on their insight that transferability is a property of the learned\nfeature space and not only of the source and target datasets.\nThe theory is developed for a deep linear networks and analytically characterizes the transferability phase diagram as a\nfunction of the target dataset size and the feature space overlap. \nThe authors report a few insights based on their theoretical analysis, and demonstrate potential applicability in two-layer ReLU networks", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 65EM,\n\nThank you for your thoughtful review and feedback. You are correct that the rigorous proofs are limited to deep linear networks. This architecture exhibits certain symmetries which generate a conserved quantity in gradient flow that we exploit heavily to prove convergence. However, we believe the main takeaway of the work is applicable to other architectures, as demonstrated by the experiments in shallow nonlinear networks. Namely, during gradient descent in the feature learning regime, the model will learn features present in the source task. This defines a subspace of functions that the model can represent. When transferring to a target task with linear transfer, the model can only fit the portion of the target function that lives in the space spanned by the learned features of the target task. Since we can show this effect precisely in the deep linear setting we focus on this as the primary running example in the manuscript. The main practical take-away is that feature space overlap is the relevant object for predicting transferability. The focus of this paper is to demonstrate this phenomenon precisely in a solvable model, but we believe that interesting directions for future work include designing feature-level metrics that are predictive of transfer performance, or designing optimal sampling protocols for the source task when source sampling is readily available, but target data is scarce. Per your request of practical implications, we will add these ideas to our discussion section. While the space constraints keep us from including a simple example in the main body of the paper, we plan to make our code publicly available, including a Jupyter notebook with a simple example to help readers explore the concepts of our work. Additionally, we will include an additional figure with a cartoon depiction of the feature space overlap concept in the main body of the paper. We also thank you for raising the question of finite training time and learning rate. We show in Appendix C2 that in deep linear networks that convergence to a global minimum is subexponential in time, so at long times the results of our paper hold with very small error. However, optimal early stopping is an interesting and likely useful regularization technique that would prevent the model from sparsifying to the source features. We predict that the results of source task early stopping would be similar to those for weight decay, which we demonstrate in the appendix (Figure 5). That is, keeping the model from completely learning the source task may actually help transfer if the source and target features are very different. As for the learning rate, we believe that this framework would also describe models trained with other optimization algorithms such as finite size gradient descent or SGD. The challenge lies in describing their dynamics precisely. For this reason, we focused on the analytically tractable setting of gradient flow. However, similar theorems on the global convergence of gradient descent for finite step size are proven in “Global Convergence of Gradient Descent for Deep Linear Residual Networks” (Wu 2019). The main takeaway is that with a sufficiently small learning rate the network converges to a global optimum, in which case our results would hold exactly as written in the paper. Finally, we respond to your specific questions below:\n\nWe define the feature space overlap in the following way: during the source training the model will learn a function $f = \\sum_{i} c_i \\phi_i{x}$. We call span($\\{\\phi_i\\}$) the feature space of the source task. The feature space overlap is the norm of the orthogonal projection (in the $L_2$ sense) of the target function $f_t$ into this space. In the case of our model, the learned feature space is $\\beta_{s}$ (see Theorem 3.4), the projection is simply $\\beta_{s}^T \\beta{t}$. Since both vectors have unit norm, this can be viewed as the cosine angle between the two tasks, so we plot the overlap in radians. In the case of two layer ReLU networks, computing the overlap is also analytically tractable. We describe how this is done in Appendix D. \nThat is the correct intuition. We measure the success of transfer learning by its generalization relative to training from scratch on the target dataset. If the target dataset size is very large, there is little benefit to pretraining.\nYes we believe our results extend beyond two layer ReLU networks. Equation 18 holds generally for any Reproducing Kernel Hilbert Space. We choose two layer ReLU networks with Gaussian data since the projection in Equation 18 can be computed exactly.", "claims_and_evidence": "Paritally.  I am unsure about the claim about lazy vs rich training mentioned in Section 5, and in generally,  claims related to training dynamics.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Partially", "experimental_designs_or_analyses": "N/A. The analysis is mainly theoretical.", "relation_to_broader_scientific_literature": "Sufficient novelty", "essential_references_not_discussed": "Please cite the following reference for lazy vs rich training:\n\nAndrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR 2014", "other_strengths_and_weaknesses": "The authors provide many profound insights, however:\n- Taken strictly, the results are limited to deep linear networks.\n- The parctical implications are not clear.\n- The manuscript is rather difficult to follow. I understand that this is a theory paper, but I would appreciate including an example that guides the reader through the theorems and what they mean for that example. \n- A visualization of the feature space would be very helpful, especially to explain the overlap.\n- I miss  the number of epochs and learning rate as influential parameters that impacts transferability in the rich regime.", "other_comments_or_suggestions": "A few typos:\n- intital => initial\n- discrepency => discrepancy\n- and model operates => the model\n- Gram matrix => always capitalize", "questions_for_authors": "- How do you measure feature space overlap ($\\theta$)? And why you use radian as its unit?\n\n- I found it surprising to see a goldilock effect w.r.t. target dataset size ($\\lambda$), as illustrated in Figure 1. Is this because a large target set negates any benefits of transfer learning and tip the scale towards training from scratch?\n\n- Do the results extend beyond two-layers in ReLU networks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KGlTsTUbYs", "reviewer": "Reviewer_UEjP", "summary": "The paper analyzes the transferability capability of deep linear networks. Specifically, it theoretically analyzes the generalization error of deep linear networks when they are trained from scratch versus linear transfer and fine-tuning in a regression problem. The paper also extends this study to the use of the ReLU activation function. In particular, it focuses on a theoretical analysis of performance based on a feature space learned by the pre-trained model. The aim is to evaluate when a model can be beneficial, for which the theoretical assumptions consider a mathematically tractable model to study transfer learning, introducing deep linear networks, which, although a simplification, capture how neural networks learn features.\nThe findings of this article demonstrate that transfer improves performance if the overlap of the feature space between the source task and the target task is sufficiently strong. Furthermore, approaches such as linear transfer and fine-tuning are compared, showing that the efficiency of transfer depends on the structure of the features learned during pretraining.\n\n## update after rebuttal\nAfter reading other reviews and responses, I prefer to keep my score. There are no critical points to decrease/increase my evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UEjP,\n\nThank you for taking the time to review our paper. We are happy to hear that you found our results interesting and the presentation clear. We agree that we could be more detailed in our presentation of the simulations. In the final draft of the manuscript we plan to make the code publicly available, including a jupyter notebook that readers can use to explore the concepts of the paper for themselves. In addition, we will mention Appendix E in the main text, per your recommendation.", "claims_and_evidence": "Yes, they are.", "methods_and_evaluation_criteria": "Yes, they make sense. The paper is theoretical, and some empirical examples are used to corroborate the theory.", "theoretical_claims": "No, I did not have time to analyze the proofs corresponding to Appendix C.", "experimental_designs_or_analyses": "Yes, the analysis seems fine.", "relation_to_broader_scientific_literature": "It's quite important. Linear transfer and fine tuning are some of the most used methods in the literature. Unfortunately, people use them without any knowledge and do not understand that sometimes, the model can do even worse. Even though the paper makes several assumptions, this can be a very important theoretical contribution.", "essential_references_not_discussed": "No, I can not think of any paper that must be included.", "other_strengths_and_weaknesses": "Strengths:\n-The writing is clear, and the theoretical formulations are well explained, allowing the article to be understood even by readers without deep knowledge of the topic. The logical organization of the sections also facilitates comprehension, and the article is presented in an accessible manner, maintaining an appropriate balance between formality and clarity. \n-The choice of a linear network-based approach simplifies the mathematical formulation and facilitates the demonstration of phenomena related to model transfer. This theoretical approach provides clarity in the concepts and makes the article accessible to a broader audience.\n-The article provides a detailed analysis of the conditions for successful transfer, especially by considering the similarity between the source and target tasks. It also clearly discusses whether the representations learned in both domains are relevant, which is essential for transfer success. This analysis is a strong point, as it helps to connect ideas with practical situations in Transfer Learning.\n\nWeaknesses:\n-Equation (6), a parenthesis is missing in f(x)^2?", "other_comments_or_suggestions": "-While the article is based on a robust theoretical analysis, it would be helpful to include a more detailed section on experiments or simulations that support the theoretical conclusions, providing more concrete validation of the approach.", "questions_for_authors": "No comments or questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ubWPcPuXLK", "reviewer": "Reviewer_AyJw", "summary": "This paper theoretically analyzes transfer learning under a multi-layer neural network model. The exact setting considered is a label shift setting with Gaussian noise and linear targets. The paper analyzes the features learned by the penultimate layer of the linear network and studies how the learned features relate to transferability. Simulations on two-layer ReLU networks trained in a mean-field regime show that the theoretical results on linear networks also transfer to non-linear ReLU networks.\n\n**Post rebuttal update:** The authors' rebuttal addresses most of my concerns. While I still feel that the technical contribution of the paper is a bit limited, I agree that analyzing transfer learning with the inductive biases in feature learning considered, even in a simplified linear setting, holds some significance. I thus keep my score as is.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer AyJw, \n\nThank you for your review of our work. We are glad you found the paper well written and we thank you for your feedback on how to improve the manuscript. First, we appreciate your pointing out the additional references. We will discuss their relevance to our work in the Related Works section of our updated draft. Thank you for pointing out moments that felt unclear or overstated. In regards to Lines 104-109 we primarily want to highlight that training dynamics, implicit bias, and generalization error are all relevant to a theory of transfer learning and that deep linear models are an ideal test bed to explore the interplay of these phenomena since their manifestations are analytically solvable. We agree that many works have explored deep linear networks, and we will change these lines to highlight that our results build on prior work to tackle these multifaceted aspects of the transfer learning problem. Thank you for also pointing out the lack of clarity in Lines 271-273. We will change this sentence to: “We can view $\\gamma$ as a dimensionless measure of the amount of target data, and $\\cos^2 \\theta$ as the amount of power that the target function has in the subspace of the pretraining task. The condition for negative transfer is satisfied when there is more target data than there is power in the pretrained subspace”. In response to your final question, there indeed is a regime where linear probing will outperform fine tuning. We can solve for this condition by subtracting Equation 15 from Equation 11 and looking for points in the $(\\theta, \\gamma)$ plane where this function is positive. In the noiseless case, the expression simplifies a bit. In the overparameterized regime ($\\gamma < 1$) linear probing out performs fine tuning as long as $\\gamma < \\sin^2(\\theta/2)$. The intersection of this condition with that for positive transfer ($\\gamma < \\cos^2{\\theta}$) is satisfied when there is limited target data. We will include a plot of this region in the appendix of the final draft. For the underparameterized case ($\\gamma > 1$) linear probing always induces negative transfer whereas fine tuning has zero transferability. So, although fine tuning has better transferability than linear probing, it’s not worth doing any pretraining in the underparameterized regime if there is no label noise.", "claims_and_evidence": "All claims made in the paper are supported by clear evidence.", "methods_and_evaluation_criteria": "The paper does not propose a new method. The experiments mainly serve as proof-of-concept demonstrations of theoretical results, which make sense to me.", "theoretical_claims": "I did not carefully check the proofs. Yet, many results (e.g., Theorem 3.4 and 3.5) are built on prior work (Yun et al., 2021), and based on my understanding of the problem, they seem correct.", "experimental_designs_or_analyses": "Experimental designs are OK.", "relation_to_broader_scientific_literature": "Most of the theoretical results of this work are built upon Yun et al. (2021): since the trained model can be fully characterized in the considered multi-layer linear network setting, analyzing the generalization error/transferability seems rather straightforward based on the trained model. That being said, I think the transferability results presented are a decent addition to the existing literature. I also found the results showing the insufficiency of distributional source-target measures interesting.", "essential_references_not_discussed": "To my knowledge, most of the related works are properly cited and discussed.", "other_strengths_and_weaknesses": "**Strengths:**\n- The theoretical parts are well-written.\n- Simulations are comprehensive and some results are interesting to me.\n\n**Weaknesses:**\n- The technical novelty is limited. Multi-layer linear networks are a rather well-understood setting with much prior work, and the paper mainly uses the results in prior work to further analyze transfer learning.", "other_comments_or_suggestions": "- I think the first contribution listed in Lines 104-109 is a bit of an overclaim: as I have mentioned above, multi-layer linear networks are not a novel setting itself, so the claim \"We develop an analytically solvable model of transfer learning that captures training dynamics, implicit bias, and generalization error in deep linear networks\" feels too big for me.\n\n- In the \"Feature learning\" paragraph of related work, I think several prior works should also be discussed (yet they do not belong to \"essential references\" so I did not list them above): [1] and [2] are among the first works theoretically analyzing the feature learning process beyond the neural tangent kernel regime. More related to transfer learning, [3] and [4] analyze the feature learning process of neural networks and its impact on generalization to new test distributions.\n\n- Lines 271-273: \"This condition requires that there is more data than the there is target function power in the direction learned during pretraining.\" (non-comprehensible sentence)\n\n---\n\n[1] Allen-Zhu et al. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. arXiv, 2020.\n\n[2] Allen-Zhu et al. Feature Purification: How Adversarial Training Performs Robust Deep Learning. FOCS, 2021.\n\n[3] Chen et al. Understanding and improving feature learning for out-of-distribution generalization. NeurIPS, 2023.\n\n[4] Zhang et al. Feature contamination: Neural networks learn uncorrelated features and fail to generalize. ICML, 2024.", "questions_for_authors": "Based on the transferability analysis, can you comment on the benefits of linear probing/fine-tuning? For example, when should we choose fine-tuning/linear probing for transfer learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "3yBAcmz8XM", "reviewer": "Reviewer_jfTK", "summary": "The manuscript theoretically analyzes the transfer learning from a feature-centric viewpoint. Specifically, the authors consider the deep linear model and two transfer schemes, i.e. linear transfer and fine-tuning. Multiple theoretical results, such as phase diagrams, are established to uncover when transfer learning will outperform the training from scratch. Some insights are then extended to nonlinear networks slightly.\n\n## update after rebuttal\nThe authors' additional results for finite-sample pre-training are interesting. Although I can't see the revised version, and the problem setting under the linear form is limited, I think this work will be a fair addition to the transfer learning community. Please make sure the finite-sample pre-training results will be added to the final version. As my original score is already positive, I will keep it as is.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jfTK,\n\nThank you for your thoughtful review of our work. We are grateful you found the manuscript easy to follow and we appreciate your feedback on areas that can be improved. We respond to your smaller comments below and leave our response to finite source datasets to the end, as this is more involved. \n\nFirst, thank you for pointing out these two references we have not cited. They will be included in the Related Works section of the introduction\nWhile we agree that deep linear networks are particularly simple, we chose to prioritize analytic tractability in this work. However, as we mention in our response to Reviewer 65EM, a lot of the take aways are the same in the case of linear transfer with nonlinear networks. In this case we can describe the feature space using Reproducing Kernel Hilbert Space theory as in Appendix D, where we carry out a calculation for two layer ReLu networks. \nYou are correct that Theorem A.2 is proven only for the Dudley Metric and KL divergence. However, we note that there is a hierarchy of integral probability metrics. For example, the theorem also holds for The Wasserstein metric since it is lower bounded by the Dudley Metric. Although we expect it to be the case, we were not able to prove that the relation holds for any IPM or $\\phi-$divergence. \nYou are correct that we have called our model label shift, but it appears that the term “concept drift” is more common in the literature. We will adjust our terminology accordingly in the revised manuscript\nAs for the question of finite source datasets, we agree that this is a richer setting to study transfer learning. To this end, we have carried out a calculation for the **full fine tuning** transferability surface in the case of a finite source dataset, the results of which are described in the following. Let $\\gamma_s = n_s/d$ and $\\gamma_t = \\n_t/d$ where $n_s$ and $n_t$ are the number of source and target data points respectively. Let $\\sigma_s$ be the standard deviation of the Gaussian label noise in the source task (defined analogously to Assumption 3.1). Then the transferability is \n\n$$\n\\begin{cases}\n\\frac{(\\gamma_t - 1)}{1-\\gamma_s}\\sigma_s^2 \\gamma_s + \\gamma_s (\\gamma_t -1) (1- 2\\cos\n\\theta) & \\gamma_s, \\gamma_t < 1,  \\\\\n (\\gamma_t -1)(1-2\\cos\\theta) -  \\frac{(\\gamma_t - 1)}{1-\\gamma_s}\\sigma_s^2 & \\gamma_s > 1>\\gamma_t , \\\\\n0 & \\gamma_t > 1\n\\end{cases}\n$$\n\nThe outline of the proof, which we will include in the final version of the paper, is similar in spirit to that of Theorem 3.9. In particular we know that gradient flow will converge to the minimum norm least square solution during source training, and we reason about the norm of its projection in the space orthogonal to the row space of the target data. There are a few interesting aspects of this expression. First, note that as long as the target task is overparameterized ($\\gamma_t < 1$), the negative transfer boundary is completely determined by $\\gamma_s$. Second, in the case of no label noise in the source task, the phase diagram is the same as in Fig 2. That is to say that fine tuning does not depend on the amount of source data if there is no label noise. The case of general noise level and source dataset size is more interesting: for some values of $\\sigma_s$ there are disconnected regions of positive transfer. We will include the analysis of these equations, as well as plots of the phase diagram in the final version of the manuscript. We thank the reviewer for this suggestion.", "claims_and_evidence": "The authors claimed those distributional measures between source and target data distributions are not enough to predict the success of transfer learning over training from scratch. This is also listed as one of the contributions of this manuscript. However, the authors showed that positive transfer can happen even if source and target distributions are far apart only in Appendix A. Since this is the motivation to consider the feature-centric viewpoint (and even the claimed contribution), the authors should not put it in the appendix.", "methods_and_evaluation_criteria": "The evaluation criteria are reasonable.\n\nAs for the training procedure of this paper, I’m not convinced. Specifically, the authors considered the pre-training over the source domain was conducted on the population distribution rather than empirical one, as the authors would like to mimic the practical setting where the sample size over the source domain greatly exceeds the sample size over the target domain. Therefore, the authors can show the pre-trained coefficient $\\beta_{s}$ indeed converges to ground truth as $t\\rightarrow\\infty$.\n\nHowever, I think this setting is problematic. It is unreasonable to assume one can train on the population distribution (or equivalent $n_{s} \\rightarrow \\infty$) and one can do the pre-training infinitely long. It is more reasonable to consider pre-training in which the source sample size is much larger than the target one ($n_{S}\\gg n_{t}$), and the pre-trained model can not fully recover the source ground truth $t\\neq \\infty$. A lot of transfer learning theory works consider such cases where they can more precisely evaluate how sample sizes and model similarity can produce a positive transferability; see [1,2] and more references therein. Besides, since the pre-trained model can not perfectly recover the ground truth, there will be some extra error induced by adapting these features in linear transfer, which could change the phase diagrams greatly. \n\n[1] Du, Simon S., et al. \"Hypothesis transfer learning via transformation functions.\" *Advances in neural information processing systems* 30 (2017).\n\n[2] Li, Sai, T. Tony Cai, and Hongzhe Li. \"Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality.\" *Journal of the Royal Statistical Society Series B: Statistical Methodology* 84.1 (2022): 149-173.", "theoretical_claims": "I only went through the proofs roughly (which seem correct), but I did not check them in great detail.", "experimental_designs_or_analyses": "Experimental designs seem correct, and the results seem to align with the theoretical results.", "relation_to_broader_scientific_literature": "The result of data similarity (distributional measure) might not be enough to predict the success of transfer learning and motivate the feature-centric viewpoint can produce a broader impact in transfer learning research.", "essential_references_not_discussed": "I do not heavily work in transfer learning within over-parametrizing and feature learning regimes, but it seems like the authors have cited some reasonable reference in this field.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The manuscript is well-written and very easy to follow.\n2. It is interesting to see how feature/model similarity can be used to predict the success of transfer learning, which is now gaining attention and investigation in the statistics community.\n3. Results in Theorems 3.7, 3.8, and 3.9 seem novel in the field (transfer learning with deep linear network).\n\nWeakness:\n\n1. The pre-training process. Please refer to the “Methods And Evaluation Criteria\" part.\n2. The deep linear model is a very simple setting. Although I understand this may be due to the lack of available theoretical tools or techniques in the community to investigate the nonlinear network, this can limit how the theoretical analyses in this manuscript can provide insights into practical settings.\n3. The argument for “distributional measure is not predictive of the success of transfer learning” is confined to only using KL divergence and Dudley Metric. It does not seem fully convincing, as there may be other measures/metrics.", "other_comments_or_suggestions": "The authors can probably be clearer about some of the notations. For example, explain what is $\\bar{W}\\_{l}$ and how to set it instead of just citing other papers, and state the signal strength $\\||\\beta\\||_{2}$ equals 1.", "questions_for_authors": "Questions:\n\n1. Is it possible to make Theorem A.2 to be measure-free or for a broader class of distributional measure? I know this can be challenging, but it can make the motivation for a centric viewpoint convincing.\n2. If the pre-training is not on population distribution, and the setting is $n_{s} \\gg n_{t}$ and $t\\neq \\infty$, which part of your analyses will no longer hold?\n3. Isn't the label shift referred to the case where the marginal distribution of $Y$ is unchanged across domains? I believe the case you studied is usually referred to as concept or model shifts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission focuses on the relevance of feature representation as key element to determine the success of transfer learning. It has been overall appreciated for its novelty in content and its contribution in shedding some light on the role of features in such a learning strategy. The analysis provides in fact a theoretical framework to a number of practices (such as fine tuning), although it is focused a minimal theoretical model (a possible weakness of the submission) that might be far from real-world setup. Nevertheless, the shared positive impression leads to me to suggest to *Accept* the manuscript in ICML.
Decision: Accept (poster)