Features are fate: a theory of transfer learning in high-dimensional regression

## Abstract

Abstract With the emergence of large-scale pre-trained neural networks, methods to adapt such “foundation” models to data-limited downstream tasks have become a necessity.
Fine-tuning, preference optimization, and transfer learning have all been successfully employed for these purposes when the target task closely resembles the source task, but a precise theoretical understanding of “task similarity” is still lacking.
While conventional wisdom suggests that simple measures of similarity between source and target distributions, such as ϕ italic-ϕ \phi italic_ϕ -divergences or integral probability metrics, can directly predict the success of transfer, we prove the surprising fact that, in general, this is not the case.
We adopt, instead, a feature-centric viewpoint on transfer learning and establish a number of theoretical results that demonstrate that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch.
We study deep linear networks as a minimal model of transfer learning in which we can analytically characterize the transferability phase diagram as a function of the target dataset size and the feature space overlap.
For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance, especially in the low data limit.
These results build on an emerging understanding of feature learning dynamics in deep linear networks, and we demonstrate numerically that the rigorous results we derive for the linear case also apply to nonlinear networks.

## 1 Introduction

State of the art neural network models have billions to trillions of parameters and are trained on datasets of a similar scale. The benefits of dataset scale are manifest in the astounding generalization capability of these foundation models (Bahri et al., ). For many applications, however, datasets of the scale used for natural language processing or computer vision are hard, if not impossible, to generate.

To alleviate the problem of inadequate dataset scale, the representations of a foundation model seem to provide a useful inductive bias for adaptation to a target task.

While they are now ubiquitous, transfer learning methods lack a solid theoretical foundation or algorithmic design principles.
As such, it remains difficult to predict when—and with which approach—transfer learning will outperform training on the target task alone.
Intuitively, if the source task resembles the target task, transfer learning should be beneficial. The important question of how to quantify task relatedness is one that remains unanswered. In this work, we address this question and prove the surprising fact that discrepancies between source and target data distributions can be misleading when it comes to transferability. We instead find that the feature space learned during pretraining is the relevant object for predicting transfer performance, which means that model-agnostic metrics between tasks are unlikely to successfully predict task overlap.
Of course, adopting a feature-centric viewpoint creates model-specific challenges because unambiguously identifying learned features remains an outstanding and difficult characterization problem for deep neural networks .
For this reason, in this work we focus on deep linear networks trained with gradient flow, as feature learning dynamics are well-understood in this setting. We develop an intuitive understanding of linear transfer and full fine-tuning in this model. In contrast to other recent work, we quantify transfer performance relative to training on the target task alone and precisely identify when transfer learning leads to improved performance, effectively building a phase diagram for transfer efficiency. Finally, we show in numerical experiments that this picture holds qualitatively for nonlinear networks, as well.

### 1.1 Related work

- •
Theoretical aspects of transfer learning A number of recent works have studied theoretical aspects of transfer learning, focusing on the risk associated with various transfer algorithms. Wu et al. () use information theory to derive bounds on the risk of transfer learning using a mixture of source and target data. Shilton et al. () analyze transfer in the context of gaussian process regression. Tripuraneni et al. () work in a fairly general setting, and derive bounds on the generalization error of transferred models through a complexity argument, highlighting the importance of feature diversity among tasks. Liu et al. (); Neyshabur et al. () study transfer learning from the perspective of the loss landscape and find that transferred models often find flatter minima than those trained from scratch. Consistent with our feature-centric viewpoint, Kumar et al. () show that fine-tuning can distort the pretrained features, leading to poor out of distribution behavior.
- •
Transfer learning in solvable models Similar to our approach, several theory works have worked with analytically tractable models to more precisely characterize transfer performance. Lampinen & Ganguli (); Atanasov et al. (); Shachaf et al. () also study transfer learning in deep linear networks, but focus on the generalization error alone, not the transferability relative to a scratch trained baseline, which obfuscates the conditions for transfer learning to be beneficial. Gerace et al. () studies transfer learning with small nonlinear networks with data generated from a “hidden manifold“ (Goldt et al., ) and find transfer learning to be effective when tasks are very similar, and data is scarce, but do not theoretically describe regions of negative transfer. Saglietti & Zdeborova () studies knowledge distillation in a solvable model, which can be viewed as a special case of transfer learning.
- •
Metrics for transferability In an effort to define task similarity, there has been recent interest (Gao & Chaudhari, ; Nguyen et al., ; Alvarez-Melis & Fusi, ) in designing metrics to predict transfer performance, an approach whose validity we assess in Section 2.1.
- •
Feature learning The notion of feature learning is central to our results. While the rich, feature learning regime is often heuristically defined as the opposite of the neural tangent regime (Jacot et al., ), a precise definition is still lacking. Nevertheless, there has been an explosion of interest in understanding dynamics in these two regimes of neural network optimization Woodworth et al. (); Atanasov et al. (); Yang & Hu (); Kunin et al. (); Yun et al. (); Chizat () focus on the role of initialization, learning rate, and implicit bias in feature learning. Petrini et al. () highlights the potential for overfitting when training in the feature learning regime.

## 2 General theoretical setting

We begin by introducing the general theoretical framework under which we study transfer learning. We consider source and target regression tasks defined by probability distributions $p_{\mathrm{s}}(\bm{x},y)$ and $p_{\mathrm{t}}(\bm{x},y)$ over inputs $\bm{x}\in\mathbb{R}^{d}$ and labels $y\in\mathbb{R}$. We focus on label shift, in which $p_{\mathrm{s}}(\bm{x},y)=p(\bm{x})p_{\mathrm{s}}(y\mid\bm{x})$ and $p_{\mathrm{t}}(\bm{x},y)=p(\bm{x})p_{\mathrm{t}}(y\mid\bm{x})$ for the same input distribution $p(\bm{x})$. The labels are generated from noisy samples of source and target functions

$$ $\displaystyle y_{\textrm{s}}$ $\displaystyle=f_{\textrm{s}}^{*}(\bm{x})+\epsilon$ (1) $\displaystyle y_{\textrm{t}}$ $\displaystyle=f_{\textrm{t}}^{*}(\bm{x})+\epsilon$ (2) $$

where $f_{\textrm{s}}^{*}(\bm{x}),f_{\textrm{t}}^{*}(\bm{x})\in L_{2}(p(\bm{x}))$ and $\epsilon\sim\mathcal{N}(0,\sigma^{2})$. During pretraining, we train a model with parameters $\bm{\Theta}=(\bm{c},\bm{\theta})$ of the form

$$ $f(\bm{x};\bm{\Theta})=\sum_{i=1}^{m}c_{i}\phi_{i}(\bm{x};\bm{\theta})$ (3) $$

on the source task using a mean squared loss. Note that the features $\phi(\bm{x},\bm{\theta})$ are left general and could for example represent final hidden activations of a deep neural network. After pretraining, the model is transferred by training a subset of the parameters $\bm{\Theta}^{\prime}\subset\bm{\Theta}$ on the target task, while leaving $\bm{\Theta}-\bm{\Theta}^{\prime}$ at their pretrained values. To model the modern setting for transfer learning, in which the number of data points in the source task far exceeds those in the target task, we train the source task on the population distribution and the target task on a finite dataset $\mathcal{D}$ of $n$ independent samples.

$$ $\displaystyle\mathcal{L}_{\textrm{s}}(\bm{\Theta})$ $\displaystyle=\frac{1}{2}\mathbb{E}_{p_{\mathrm{s}}(\bm{x},y)}[(f(\bm{x},\bm{ \Theta})-y)^{2}]$ (4) $\displaystyle\mathcal{L}_{\textrm{t}}(\bm{\Theta}^{\prime})$ $\displaystyle=\frac{1}{2}\hat{\mathbb{E}}_{p_{\mathrm{t}}(\bm{x},y)}[(f(\bm{x} ,\bm{\Theta})-y)^{2}]$ (5) $$

where $\hat{\mathbb{E}}_{p}(h(\bm{x},y))=\frac{1}{n}\sum_{i=1}^{n}h(\bm{x}_{i},y_{i})$ is the expectation over the empirical distribution of $\mathcal{D}$. We focus on two widely employed transfer methods, linear transfer and fine-tuning. In linear transfer, the pretrained features $\phi(\bm{x},\bm{\theta})$ are frozen and only the output weights $\bm{c}$ are trained on the target task. In fine-tuning, the entire set of parameters $\bm{\Theta}$ are trained from the pretrained initialization on the target task. To optimize the loss functions (4) and (5), we use gradient flow,

$$ $\frac{d\bm{\Theta}_{i}}{dt}=-\nabla_{\bm{\Theta}_{i}}\mathcal{L}(\bm{\Theta}),$ (6) $$

where we have set the learning rate equal to unity for the purpose of analysis. To assess the performance of transfer learning we compare the performance of the transferred model to a scratch-trained model with the same architecture (3) trained only on the target task from a random initialization. We introduce the transferability to quantify this relationship:

$$ $\mathcal{T}=\mathbb{E}_{\mathcal{D}}(\mathcal{R}_{\textrm{sc}}-\mathcal{R}_{ \textrm{tx}})$ (7) $$

where $\mathbb{E}_{\mathcal{D}}$ is the expectation over iid draws of the training set and $\mathcal{R}_{\textrm{tx}}$ and $\mathcal{R}_{\textrm{sc}}$ are the generalization errors of the transferred model and scratch trained model, respectively, where the generalization error (or population risk) is given by,

$$ $\mathcal{R}=\mathbb{E}_{p(\bm{x})}[(f(\bm{x},\bm{\Theta})-f^{*}(\bm{x})^{2}].$ (8) $$

We consider transfer learning successful when $\mathcal{T}>0$, i.e., when the expected generalization of transfer learning outperforms training from scratch on the target task. We refer to the situation $\mathcal{T}<0$ as negative transfer, since pretraining leads to degradation of the generalization error.

### 2.1 Dataset similarity is not predictive of transfer efficiency

The common wisdom in transfer learning is that related tasks should transfer effectively to one another. However, a standard and mathematically precise definition of task relatedness is currently lacking. One reasonable notion of task relatedness is to compare the source and target data distributions $p_{\mathrm{s}}$ and $p_{\mathrm{t}}$ using a discrepancy measure between probability distributions, for example an integral probability metric (IPM) or a $\phi$-divergence Sriperumbudur et al. (). While $\phi$-divergences like the Kullback-Leibler divergence are well-known, they are often hard to compute for high-dimensional distributions. Wu et al. (); Nguyen et al. () suggest that these kinds of measures will correlate with transfer performance, as measured by generalization error on the target task. However, we argue that a meaningful measure of transfer performance must compare to a scratch trained baseline, not target task performance alone. IPMs, such as the Wasserstein-1 Distance, Dudley Metric, and Kernel Mean Discrepency, are bona fide metrics on the space of probability distributions and are of theoretical importance in optimal transport, statistics, and probability theory. Using ideas from optimal transport, (Alvarez-Melis & Fusi, ) attempt to correlate transfer performance with the Wasserstein distance. While it may seem that closeness of task distributions should correlate with transfer performance, we show that this is not necessarily the case. In particular, we select a member of each family and prove that, within our model, one can achieve positive transfer ($\mathcal{T}>0$) with distributions that are arbitrarily far apart.

Two functions representable with the same features can be “far apart”.
We formalize this notion with the following theorem.

###### Assumption 2.1 .

We assume $f\in L_{2}(\mathbb{R}^{d},p)$ and for each $\bm{x}\in\mathbb{R}^{d}$ we define the random variable $y:\mathbb{R}^{d}\to\mathbb{R}$ through the relation $y=f(\bm{x})+\epsilon$ with $\epsilon\sim\mathcal{N}(0,\sigma^{2})$. Let $p_{f}(\bm{x},y)$ denote the joint probability density of $\bm{x}$ and $y$. We assume $\Phi\subset L_{2}(p)$ is a linear subspace with orthonormal basis $\{\phi_{i}\}_{i=1}^{M}$ and $M$ may be infinite.

###### Theorem 2.2 .

Assume 2.1. Then for any $f\in\Phi$, and any $\delta>0$ there exists $g\in\Phi$ such that

$$ $\gamma_{\beta}(p_{f},p_{g})\geq\delta$ $$

where $\gamma_{\beta}(p,p^{\prime})$ is the Dudley Metric. Similarly, for any $f\in\Phi$, and any $\delta>0$ there exists $g\in\Phi$ such that

$$ $D_{\mathrm{KL}}(p_{f}\|p_{g})\geq\delta$ $$

where $D_{\mathrm{KL}}(p_{f}\|p_{g})$ is the Kullback Leibler divergence.

We prove this theorem in Appendix B.1.
We note that this relationship theorem also holds for any IPM over a function class that is larger than the class of Bounded Lipschitz functions. In particular, the theorem holds for the Monge-Kantorovich ($W_{1}$) metric, since any function that satisfies $\lVert f\rVert_{\mathrm{BL}}\leq 1$ also satisfies $\lVert f\rVert\leq_{L}1$. Because it is simpler to compute $W_{1}$ numerically, in Fig. 5 we plot the Wasserstein distance, not the Dudley metric.

Theorem 2.2 demonstrates that for a given source distribution, one can always find a target distribution generated from the same feature space that is arbitrarily distant with respect to these metrics. However, despite such distance, intuitively, if the source and target functions lie in the same feature space and pretraining creates a basis for this space, transfer to the target task should likely be positive. We show this is indeed the case for deep linear networks in the following section.

###### Assumption 2.1 .

###### Theorem 2.2 .

## 3 Deep linear networks: an exactly solvable model

As an analytically solvable model of transfer learning we consider a deep linear network with $L$ layers

$$ $f(\bm{x})=\bm{x}^{T}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}$ (9) $$

where $\bm{W}_{l}\in\mathbb{R}^{d_{l-1}\times d_{l}}$ for $l\in[1,2,\dots L-1]$ and $\bm{W}_{L}\in\mathbb{R}^{d_{L-1}\times 1}$. For notational convenience we have renamed $\bm{c}$ in (3) as $\bm{W}_{L}$ and for simplicity we set $d_{0}=d_{1}=\dots=d_{L-1}=d$, the dimension of the data. The parameter matrices are initialized as $\bm{W}_{l}(0)=\alpha\bar{\bm{W}}_{l}$ where $\alpha\in\mathbb{R}$. Since transfer learning relies on learning features in the source task, we initialize the network in the feature learning regime $\alpha\to 0$.
In the following, we assume:

###### Assumption 3.1 .

Assume that the input data $\bm{x}\in\mathbb{R}^{d}$ is normally distributed and that each dataset $\mathcal{D}$ consists of $n$ pairs $\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$ sampled iid from $p_{\mathrm{t}}$ with Gaussian label noise of variance $\sigma^{2}.$

###### Assumption 3.2 .

We assume that the source and target functions are each linear functions in $L_{2}(\mathbb{R}^{d},p)$; equivalently, $f_{\mathrm{s}}^{*}(\bm{x})=\bm{\beta}_{\mathrm{s}}^{T}\bm{x}$, $f_{\mathrm{t}}^{*}(\bm{x})=\bm{\beta}_{\mathrm{t}}^{T}\bm{x}$ with $\lVert\bm{\beta}_{\mathrm{s}}\rVert_{2}^{2}=\lVert\bm{\beta}_{\mathrm{t}}
\rVert_{2}^{2}$.

To control the level of source-target task similarity, we fix the angle $\theta$ between the ground truth source and target functions so that $\bm{\beta}_{\mathrm{s}}^{T}\bm{\beta}_{\mathrm{t}}=\cos\theta$.
The source and target loss functions are given by (4) and (5). When training over the empirical loss, it is convenient to work in vector notation
$\mathcal{L}_{\textrm{t}}(\{\bm{W}_{l\leq L}\})=\frac{1}{2n}\lVert\bm{y}_{
\textrm{t}}-\bm{X}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}\rVert^{2}_{2}$
where $\bm{X}\in\mathbb{R}^{n\times d}$ and $\bm{y}\in\mathbb{R}^{n}$.
We study this model in the high dimensional limit in which $\gamma=n/d$ remains constant as $n,d\to\infty$.

###### Assumption 3.1 .

###### Assumption 3.2 .

### 3.1 Pretrained models represent source features

To describe transfer efficiency in this setup, we need to understand the function that the model implements after training on the source task. We can describe the network in function space by tracking the evolution of $\bm{\beta}(t)=\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}$ under gradient flow, such that the network function at any point in the optimization is $f(\bm{x};t)=\bm{\beta}(t)^{T}\bm{x}$. The following Lemma establishes that pretraining perfectly learns the source task in the large source data limit.

###### Lemma 3.3 .

Under gradient flow (6) on the population risk objective (4) with initialization satisfying (22), $\lim_{t\to\infty}\bm{\beta}(t)=\bm{\beta}_{\mathrm{s}}$

We prove Lemma 3.3 in Appendix B.2.
While this result establishes recovery of the ground truth on the source task, it does not describe the feature space of the pretrained model, which is relevant for transferability. To this end, following Yun et al. (), we show that in the feature learning regime $\alpha\to 0$, the hidden features of the model sparsify to those present in the source task.

###### Theorem 3.4 (Yun et al) .

Let the columns of $\bm{\Phi}=\bm{W}_{1}\bm{W}_{2}\cdots\bm{W}_{L-1}$ denote the hidden features of the model. After pretraining

$$ $\lim_{\alpha\to 0}\lim_{t\to\infty}\bm{\Phi}=\bm{\beta}_{\mathrm{s}}\bm{v}_{L- 1}^{T}$ $$

for some vector $\bm{v}_{L-1}$.

We prove Theorem 3.4 in Appendix B.3. Theorem 3.4 demonstrates that after pretraining in the feature learning regime, the $d$-dimensional feature space of the model parsimoniously represents the ground truth function in a single, rank-one component. We refer to this phenomenon as feature sparsification, which is a hallmark of the feature learning regime, and has important consequences for transferability, particularly in the linear setting Section 3.3.

###### Lemma 3.3 .

###### Theorem 3.4 (Yun et al) .

### 3.2 Scratch trained models represent minimum norm solutions

For the empirical training objective 5, there are multiple zero training error solutions when the model is overparameterized $\gamma<1$. As noted in Yun et al. () and Atanasov et al. (), there is an implicit bias of gradient flow to the minimum norm solution when $\alpha\to 0$

###### Theorem 3.5 ( (Yun et al., 2021 ) ) .

Under gradient flow on the empirical risk minimization objective (5) with initialization satisfying (22), $\lim_{\alpha\to 0}\lim_{t\to\infty}\bm{\beta}(t)=\hat{\bm{\beta}}$, where $\hat{\bm{\beta}}$ is the minimum norm solution to the linear least squares problem

$$ $\hat{\bm{\beta}}=\operatorname*{arg\,min}_{\bm{\beta}}\frac{1}{2n}\lVert\bm{y} _{\mathrm{t}}-\bm{X}\bm{\beta}\rVert^{2}_{2}=\bm{X}^{+}\bm{y}_{\mathrm{t}}$ $$

We prove Theorem 3.5 in Appendix B.4. Knowing the final predictor of the empirical training also allows us to compute the generalization error of the scratch trained model

###### Theorem 3.6 .

Under gradient flow on the empirical objective (5), in the high dimensional limit the expectation of the final generalization error over training data is

$$ $\mathbb{E}_{\mathcal{D}}\mathcal{R}=\begin{cases}\frac{(1-\gamma)^{2}+\gamma \sigma^{2}}{1-\gamma}&\gamma<1\\ \frac{\sigma^{2}}{\gamma-1}&\gamma>1\end{cases}$ (10) $$

Theorem (3.6) is a known result for linear regression (Hastie et al., ; Canatar et al., ; Belkin et al., ; Advani & Ganguli, ; Mel & Ganguli, ; Bartlett et al., ), but we provide a proof based on random projections and random matrix theory in Section B.5. This expression exhibits double descent behavior: in the overparameterized regime, the generalization error first decreases, then becomes infinite as $\gamma\to 1$, while in the underparameterized regime, the generalization error monotonically decreases with increasing $\gamma$. As we will see in Section 3.3, this double descent behavior leads to two distinct regions in the transferability phase diagram.

###### Theorem 3.5 ( (Yun et al., 2021 ) ) .

###### Theorem 3.6 .

### 3.3 Linear transfer

The simplest transfer learning method is known as linear transfer, in which only the final layer weights of the pretrained network are trained on the target task. In particular, $\{\bm{W}_{l}\}_{l\leq L-1}$ are fixed after pretraining and $\hat{\bm{W}}_{L}$ solves the linear regression problem with features $\bm{\Phi}=\bm{X}\bm{W}_{1}\dots\bm{W}_{L-1}$.

$$ $\hat{\bm{W}}_{L}=\operatorname*{arg\,min}_{\hat{\bm{W}}_{L}\in\mathbb{R}^{d}} \frac{1}{2n}\lVert\bm{\Phi}\bm{W}_{L}-\bm{y}_{\mathrm{t}}\rVert_{2}^{2}$ (11) $$

When there are multiple solutions to the optimization problem (11), we choose the solution with minimum norm. We characterize the generalization error of linear transfer in the following theorem.

Figure: Figure 1: Linear transferability phase diagram. We pretrain a linear network (9) with $L=2$ and $d=500$ to produce labels from linear source function $y=\bm{\beta}_{\mathrm{s}}^{T}\bm{x}+\epsilon$ using the population loss (4). We then retrain the final layer weights on a sample of $n=\gamma d$ points $(\bm{x}_{i},y_{i}=\bm{\beta}_{\mathrm{t}}^{T}\bm{x}_{i}+\epsilon_{i})$ where $\bm{\beta}_{\mathrm{s}}^{T}\bm{\beta}_{\mathrm{t}}=\cos\theta$ and $\epsilon_{i}\sim\mathcal{N}(0,\sigma=0.2)$, and compare its generalization error to that of a model trained from scratch on the target dataset. (a) The theoretical transferability surface (13) as a function of target dataset size $\gamma$ and task overlap $\theta$. Light blue lines indicate the boundary between positive and negative transfer. (b) Top-down view of Fig. 1(a) shaded by sign of transferability. Red regions indicate negative transfer $\mathcal{T}<0$, blue region indicates positive transfer $\mathcal{T}>0$. (c) Slices of the transferability surface (13) for constant $\theta$. Solid lines represent theoretical values, circles are points from experiments. Error bars represent the standard deviation over $20$ draws of the target set.
Refer to caption: x1.png

###### Theorem 3.7 .

Under Assumptions 3.1 and 3.2, and assuming the source-target overlap is $\theta,$ the expected generalization error of the linearly transferred model is an explicit function of $\theta$, the label noise $\sigma$, and the dataset size $n$:

$$ $\mathbb{E}_{\mathcal{D}}\mathcal{R}_{\mathrm{lt}}=\sin^{2}\theta+\frac{\sigma^ {2}+\sin^{2}\theta}{n-2}.$ (12) $$

We prove Theorem 3.7 in Appendix B.6. The structure of the result in Theorem 3.7 merits some discussion. After pretraining in the feature learning regime $\alpha\to 0$, the feature space of the network has sparsified so that it can only express functions along $\bm{\beta}_{\mathrm{s}}$ (Theorem 3.4). Since the features of the network cannot change in linear transfer, the main contribution to the generalization error is $\sin^{2}\theta$, which can be viewed as the norm of the projection of the target function into the space orthogonal to the features spanned by the pretrained network. This is an irreducible error that is the best case risk given that the features cannot change. The second term comes from the finiteness of the training set. Since linear transfer learns from a finite sample of training points,
minimizing the training error can effectively “overfit the noise” and the learned function distorts away from the ground truth.

Luckily, since the pretrained feature space has sparsified, the effect of finite sampling and additive label noise decays as $\sim 1/n$, effectively filtering out the $d$-dimensional noise by projecting it onto a single vector. Compare this to the generalization of the scratch trained network (10). There, the features of the equivalent linear regression problem, $\bm{X}$, have support over all $d$-dimensions, so there is no irreducible error term. The expressivity, however, comes at a cost. Each dimension of the regression vector is vulnerable to noise in the training data, and the projection of the target function onto the feature space is strongly distorted due to finite sampling (i.e. $\sim\gamma$). We can precisely analyze this trade off by comparing (10) and (12). In the limit $n,d\to\infty$, the transferability (7) is

$$ $\mathcal{T}_{\mathrm{lt}}=\begin{cases}\frac{(1-\gamma)^{2}+\gamma\sigma^{2}}{ 1-\gamma}-\sin^{2}\theta&\gamma<1\\ \frac{\sigma^{2}}{\gamma-1}-\sin^{2}\theta&\gamma>1\end{cases}$ (13) $$

which is plotted in Fig. 1(a). From (13) we can identify the regions of negative transfer for this model, which are shaded in red in Fig. 1(b). In the underparameterized regime ($\gamma>1$), there is negative transfer for all $\gamma-1>\frac{\sigma^{2}}{\sin^{2}\theta}$.
In words, at fixed $\gamma$ and $\sigma$, i.e., fixing the number of data points and label noise, as the norm of the out-of-subspace component increases, transfer efficiency degrades.

In the overparameterized regime ($\gamma<1$), negative transfer only occurs when $\sigma<1$.This can be viewed as a condition on the signal-to-noise ratio of the target data: $\textrm{SNR}=\lVert\bm{\beta}_{\mathrm{t}}\rVert_{2}^{2}/\sigma^{2}=1/\sigma^{2}$. When $\textrm{SNR}<1$, scratch training can never recover the underlying vector $\bm{\beta}_{\mathrm{t}}$ and pretraining is always beneficial. When $\textrm{SNR}>1$, negative transfer occurs when $\theta\in(\arccos(1-\sigma),\pi/2)$ and $\gamma\in(\gamma_{+},\gamma_{-})$ where $\gamma_{\pm}=\frac{1}{2}[(1+\cos^{2}\theta-\sigma^{2})\pm\sqrt{(1+\cos^{2}
\theta-\sigma^{2})^{2}-4\cos^{2}\theta}]$. In the noiseless case $\sigma\to 0$, this expression simplifies to $\theta\in(0,\pi/2)$, $\cos^{2}\theta<\gamma<1$ (see Appendix E Fig. 6). This condition requires that there is more data than the there is target function power in the direction learned during pretraining. As $\sigma$ increases, the region of negative transfer shrinks, since the noise corrupts the scratch trained accuracy. Finally we mention that the two regions of negative transfer in Fig. 1 are separated by positive transfer that persists even when $\theta=\pi/2$. We dub this effect anomalous positive transfer, since the pretrained features are completely orthogonal to those in the target, yet transferability is still positive. This effect stems from the double descent phenomenon in scratch training— even though linear transfer performance is as bad as possible, transferability is still positive, since the scratch trained model has infinite generalization error. In Appendix E Fig. 5 we demonstrate that the dataset based discrepency measures of Section 2.1 are indeed misleading: neither $D_{\mathrm{KL}}$, nor $W_{1}$ are negatively correlated with increased transferability.

###### Theorem 3.7 .

#### 3.3.1 Ridge regularization cannot fix negative transfer

In the previous section, the network sparsified to features that incompletely described the target function, leading to negative transfer given sufficient target data. A common approach to mitigate this kind of multicollinearity in linear regression is to add an $\ell_{2}$ penalty to the regression objective (11) so that

$$ $\hat{\bm{W}}_{L}=\operatorname*{arg\,min}_{\hat{\bm{W}}_{L}\in\mathbb{R}^{d}} \frac{1}{2n}\lVert\bm{\Phi}\bm{W}_{L}-\bm{y}_{\mathrm{t}}\rVert_{2}^{2}+\frac{ \lambda}{2}\lVert\bm{W}_{L}\rVert_{2}^{2}.$ (14) $$

In the following theorem, which we prove in Appendix B.7, we show that the generalization error for regularized linear transfer is a strictly increasing function of the ridge parameter $\lambda$, leading to a larger region of negative transfer for any $\lambda>0$ (Fig. 2).

Figure: Figure 2: Ridge regularization leads to worse generalization in linear transfer. Linear transfer generalization error for $\gamma=0.5$ as a function of regularization parameter $\lambda$. The generalization error is a strictly increasing function of $\lambda$, which implies that the optimal regularizer is $\lambda_{*}=0$. Solid line is theory (3.8), points are experiments. Error bars represent the standard deviation over $20$ realizations of the target dataset.
Refer to caption: x2.png

###### Theorem 3.8 .

Under Assumptions 3.1 and 3.2, and assuming the source-target overlap is $\theta,$ the expected generalization error of the ridge linear transfer model over the training data is

$$ $\lim_{n\to\infty}\mathbb{E}_{\mathcal{D}}\mathcal{R}_{\mathrm{lt}}^{\lambda}=1 -\frac{(1+2\lambda)}{(1+\lambda)^{2}}\cos^{2}\theta$ (15) $$

Ridge regression attenuates the power of the predictor in all directions of the data, including the direction parallel to the signal.
Due to sparisification of Theorem 3.4, $\ell_{2}$ regularization is non-optimal and hence regularization impairs generalization by attenuating useful features, i.e., those with $\theta<\pi/2$.

###### Theorem 3.8 .

### 3.4 Fine-tuning

Another common transfer learning strategy is fine-tuning, in which all model parameters are trained on the target task from the pre-trained initialization. For general nonlinear models, analyzing the limit points of gradient flow from arbitrary initialization is a notoriously difficult task. For the deep linear model however, we can solve for the expected generalization error of fine-tuning exactly.

###### Theorem 3.9 .

Under Assumptions 3.1 and 3.2, and assuming the source-target overlap is $\theta$, the expected generalization error of the fine-tuned model over the training data is:

$$ $\mathbb{E}_{\mathcal{D}}\mathcal{R}_{\mathrm{ft}}=\begin{cases}\mathbb{E}_{ \mathcal{D}}\mathcal{R}_{\mathrm{sc}}+(1-\gamma)(1-2\cos\theta)&\gamma\leq 1\\ \mathbb{E}_{\mathcal{D}}\mathcal{R}_{\mathrm{sc}}&\gamma>1\end{cases}$ (16) $$

where $\mathbb{E}_{\mathcal{D}}\mathcal{R}_{\mathrm{sc}}$ is the expected generalization error of the scratch trained model

Theorem 3.9 is proven in Appendix B.8. Theorem 3.9 yields an expression for the fine-tuning transferability, which is plotted in Fig. 3(a):

$$ $\mathcal{T}_{\mathrm{ft}}=\begin{cases}(\gamma-1)(1-2\cos\theta)&\gamma\leq 1 \\ 0&\gamma>1\end{cases}$ (17) $$

Figure: Figure 3: Fine-tuning transferability surface Using the same transfer setup as in Fig. 1 we fine tune all of the weights on the target dataset starting from the pretrained weight initialization. (a) The theoretical transferability surface (17) as a function of target dataset size $\gamma$ and task overlap $\theta$. The light blue line parallel to the $\gamma$ axis indicates the boundary between positive and negative transfer, while the one parallel to the $\theta$ axis indicates the boundary for zero transferability. (b) Top-down view of Fig. 3(a) shaded by sign of transferability. Red region indicates negative transfer $\mathcal{T}<0$, blue region indicates positive transfer $\mathcal{T}>0$. The white region indicates no transfer benefit $\mathcal{T}=0$. (c) Slices of the transferability surface (17) for constant $\theta$. Solid lines represent theoretical values, circles are points from experiments. Error bars represent the standard deviation over $20$ draws of the target set.
Refer to caption: x3.png

When the model is underparameterized $\gamma>1$, there is a unique global minimum in the space of $\bm{\beta}=\bm{W}_{1}\bm{W}_{2}\cdots\bm{W}_{L}$. Since gradient flow converges to a global minimum, (Theorem 3.5), fine tuning loses the memory of the pretrained initialization leading to zero transferability (white region in Fig. 3(b)). When the network is overparameterized, however, there is a subspace of global minima. We show in the Section B.8 that the pretrained initialization induces an implicit bias of gradient flow away from the minimum norm solution. For $\theta<\pi/3$, the pretrained features are beneficial, leading to positive transfer. For $\theta>\pi/3$, however, the pretrained features bias the network too strongly toward the source task, leading to negative transfer.

###### Theorem 3.9 .

## 4 Student-teacher ReLU networks

Figure: Figure 4: Linear transfer in two-layer ReLU networks We train a two layer ReLU network (18) with $m=1000$ neurons on a teacher with $m_{*}=100$ neurons and $d=100$ dimensional gaussian data, according to the ablated transfer setup (19), (20). For these experiments, we set the label noise $\sigma=0$. (a) Gram matrix from the kernel of the pretrained model (b) Gram matrix from the kernel of the ground truth source function $f^{*}_{\textrm{s}}(\bm{x})$. The two gram matrices are nearly indistinguishable suggesting that the kernel sparsifies to the represent features in the source task. (c) Generalization error of the scratch trained model as a function of dataset size $n$, fit to a power law (d) Norm of out-of-RKHS component of target function $\lVert P^{\perp}f^{*}_{\textrm{t}}(\bm{x})\rVert_{L_{2}}^{2}$, normalized by target function norm $\lVert f^{*}_{\textrm{t}}(\bm{x})\rVert_{L_{2}}^{2}$ as a function of excess target features $\mu$. (e) Heat map of transferability as a function of excess target features $\mu$ and dataset size $n$. We normalize the transferability by variance in the target data. Gray circles represent the point of negative transfer predicted by our theory. Results are averaged over $100$ realizations of the data and $10$ realizations of random draws of the teacher.
Refer to caption: x4.png

In the following, we demonstrate that many of the results from our analytically solvable model also hold, qualitatively, in the more complicated setting of linear transfer with nonlinear networks. In particular, we choose a model of the form

$$ $f(\bm{x})=\frac{1}{m}\sum_{i=1}^{m}c_{i}\sigma(\bm{w}_{i}^{T}\bm{x})$ (18) $$

where $\sigma(y)=\max\{0,y\}$ is the ReLU activation. We scale the model by $1/m$ to place the network in the mean field, feature learning regime (Chizat et al., ; Mei et al., ; Rotskoff & Vanden-Eijnden, ). As in the deep linear model, we choose source and target functions that are representable by the network. That is, we study this model in the student teacher setting. To vary the level of feature space overlap between the source and target functions, we define a network of $m_{*}$ neurons for the target task, and generate the source network by ablating a fraction $\mu$ of the hidden neurons form the target. More precisely, let $\mathcal{A}$ be a uniformly random subset of the index set $\{1,2,\cdots m_{*}\}$ with $|\mathcal{A}|=\mu m_{*}$. Then

$$ $\displaystyle f_{\textrm{s}}^{*}(\bm{x})$ $\displaystyle=\frac{1}{(1-\mu)m_{*}}\sum_{i\in\mathcal{A}^{c}}c^{*}_{i}\sigma( \bm{w}_{i}^{*T}\bm{x})$ (19) $\displaystyle f_{\textrm{t}}^{*}(\bm{x})$ $\displaystyle=\frac{1}{m_{*}}\sum_{i=1}^{m_{*}}c^{*}_{i}\sigma(\bm{w}_{i}^{*T} \bm{x})$ (20) $$

Thus the source has $\mu m^{*}$ fewer hidden features than the target task, and so the fraction $\mu$ controls the degree of discrepancy between source and target feature spaces. In essence when $\mu=0$ the source and target spaces are identical. However, as $\mu$ increases, an increasing fraction of new target features, that were not present in pre-training, must be learned. We constrain the hidden features in the model, source, and target to the $d$-dimensional unit sphere $\bm{w}_{i},\bm{w}_{i}^{*}\in\mathbb{S}^{d-1}$. As in the deep linear model, we choose $\bm{x}\sim\mathcal{N}(0,\bm{I}_{d})$, train the source task on the population loss, which can be computed exactly for this model, and the target task on a finite sample of $n$ data points.

Previous work (Rotskoff & Vanden-Eijnden, ; Mei et al., ; Chizat, ) has shown that in the overparameterized setting $m\gg m_{*}$, gradient flow will converge to a global minimizer of the population loss, so that $\lim_{m\to\infty}\lim_{t\to\infty}f(\bm{x})=f_{\textrm{s}}^{*}(\bm{x})$, which establishes that the trained network builds a representation of $f_{\textrm{s}}^{*}(\bm{x})$ in the mean field limit. This does not necessarily mean that all of the hidden neurons of the model converge to those of the teacher, since any superfluous weight directions can be eliminated by setting the corresponding output weight to zero. However, we demonstrate empirically in Fig.  4(a)-(b) that this relationship is preserved at the level of the model’s kernel, so that $k(\bm{x},\bm{x}^{\prime})=\frac{1}{m}\sum_{i=1}^{m}\sigma(\bm{w}^{T}_{i}\bm{x}
)\sigma(\bm{w}^{T}_{i}\bm{x}^{\prime})\approx\frac{1}{(1-\mu)m_{*}}\sum_{i\in
\mathcal{A}^{c}}\sigma(\bm{w}^{*T}_{i}\bm{x})\sigma(\bm{w}^{*T}_{i}\bm{x}^{
\prime})$. This observation is analogous to Theorem 3.4: training in the feature learning regime causes the model’s features to sparsify to those present in the target function.

Now, linear transfer in this model can be formulated as a kernel interpolation problem with this kernel. The generalization error of kernel interpolation can be separated into an $n$-dependent component, and an irreducible error term which corresponds to the norm of the projection of the target function into the subspace of $L_{2}(p)$ orthogonal to the RKHS defined by the kernel:

$$ $\mathbb{E}_{D}\mathcal{R}_{\textrm{lt}}=C(n)+\lVert P^{\perp}f^{*}_{\textrm{t} }(\bm{x})\rVert_{L_{2}}^{2}.$ (21) $$

As expected, the norm of this projection increases monotonically with $\mu$ as shown in Fig. 4(d). We show how to compute this projection in Appendix C. In the deep linear setting, $\lVert P^{\perp}f^{*}_{\textrm{t}}(\bm{x})\rVert_{L_{2}}^{2}=\sin^{2}\theta$, and $C(n)\sim 1/n$. While the asymptotic, typical generalization error of kernel regression has been studied in (Canatar et al., ), for the purposes of estimating the generalization error of the transferred model, we assume here that this generalization error is dominated by this irreducible term for the large $n$ target dataset sizes we consider, just as we showed for the deep linear model.

However, an expression for the generalization error of the scratch-trained model is also needed to derive the transferability. We are not aware of a theory of generalization error for infinite width nonlinear networks trained on a finite data in the mean field regime. Intriguingly, however, we demonstrate empirically (Fig. 4(c)) that the generalization error obeys a power law $\mathcal{R}_{\textrm{sc}}\sim An^{-\nu}$ with $\nu=1.18$. By setting our theoretically predicted generalization error of our transferred model $\lVert P^{\perp}f^{*}_{\textrm{t}}(\bm{x})\rVert_{L_{2}}^{2}$ equal to the empirically observed scaling law $An^{-\nu}$ for our scratch-trained model, we can approximately identify the point of negative transfer in $n$ for any given $\mu$ (gray circles in Fig. 4(e)). It is clear from Fig. 4(e) that this heuristic for finding the boundary between positive and negative transfer becomes more accurate as the number of target points becomes large, since the $n$-dependent component of the kernel regression generalization error goes to zero in this limit. The phase diagram in Fig. 4 for noiseless ReLU networks resembles the phase diagram for linear transfer with deep linear networks in the noiseless setting with $\sigma=0$ (Appendix E Fig.6. 7). Overall, this demonstrates that we are able to predict the phase boundary between positive and negative transfer in the ReLU case, using our conceptual understanding in the deep linear case.

## 5 Conclusion

In this paper, we highlight the importance of thinking about transfer learning in the context of the feature space of the pretrained model. In particular, we show that certain Integral Probability Metrics and $\phi$-divergence can be misleading when it comes to predicting transfer learning performance using the datasets alone. We then rigorously identify the number of data points necessary for transfer learning to outperform scratch training as a function of feature space overlap in deep linear networks. Finally, we demonstrate that our understanding of linear transfer carries over to shallow nonlinear networks as well.

## Appendix A Initialization assumption

Following
Yun et al. () we place the following constraint on the initialization for some $\lambda>0$.

$$ $\bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{l}-\bar{\bm{W}}_{l+1}\bar{\bm{W}}_{l+1}^{T} \succcurlyeq\lambda I$ (22) $$

To our knowledge, this is the most general assumption on the weight initializations in the literature that leads to the implicit biases that are crucial for our analysis. This initialization scheme generalizes that in Wu et al. (); Atanasov et al. ().

## Appendix B Proofs

### B.1 Proof of Theorem 2.2

We begin by recalling the definition of the Dudley Metric

$$ $\displaystyle\gamma_{\beta}(p,q)$ $\displaystyle=\sup_{\lVert h\rVert_{\mathrm{BL}}\leq 1}\left|\mathbb{E}_{p}h- \mathbb{E}_{q}h\right|$ (23) $\displaystyle\lVert h\rVert_{\mathrm{BL}}$ $\displaystyle=\rVert h\lVert_{L}+\rVert h\lVert_{\infty}$ (24) $$

By conditioning $p_{f}(x,y)$ and $p_{g}(x,y)$ on $x$, we can write

$$ $\displaystyle\gamma_{\beta}(p_{f},p_{g})$ $\displaystyle=\sup_{\lVert h\rVert_{BL}\leq 1}\left|\frac{1}{\sqrt{2\pi\sigma^ {2}}}\int\left[h(x,y)e^{\frac{-(y-f(x))^{2}}{2\sigma^{2}}}-h(x,y)e^{\frac{-(y- g(x))^{2}}{2\sigma^{2}}}\right]p(x)\mathrm{d}x\mathrm{d}y\right|$ (25) $\displaystyle\geq\left|\frac{1}{\sqrt{2\pi\sigma^{2}}}\int\left[\frac{\cos(y)} {2}e^{\frac{-(y-f(x))^{2}}{2\sigma^{2}}}-\frac{\cos(y)}{2}e^{\frac{-(y-g(x))^{ 2}}{2\sigma^{2}}}\right]p(x)\mathrm{d}x\mathrm{d}y\right|$ (26) $\displaystyle=\left|\frac{e^{-\sigma^{2}/2}}{2}\int\left[\cos(f(x))-\cos(g(x)) \right]p(x)\mathrm{d}x\right|$ (27) $\displaystyle\geq\frac{e^{-\sigma^{2}/2}}{2}\int\left[f(x)^{2}+g(x)^{2}\right] p(x)\mathrm{d}x$ (28) $$

The first inequality follows from the fact that $\lVert\frac{\cos(y)}{2}\rVert_{\mathrm{BL}}=1$, and the second follows from the identity $\cos(x)+x^{2}\geq\cos(z)-z^{2}$ for any $x,z\in\mathbb{R}$. We can expand $f$ in the orthonormal basis $\{\phi_{i}\}_{i=1}^{M}$ as $f=\sum_{i=1}^{M}\alpha_{i}\phi_{i}$, so that

$$ $\int f(x)^{2}p(x)\mathrm{d}x=\sum_{i,j}\alpha_{i}\alpha_{j}\int p(x)\phi_{i}(x )\phi_{j}(x)\mathrm{d}x=\sum_{i}\alpha_{i}^{2}$ (30) $$

Since, $f\in L_{2}(p)$, the sum on right hand side of Eq. (30) converges to some $a<\infty$. We can choose $g=\sqrt{\left|\frac{2\delta e^{\sigma^{2}/2}-a}{a}\right|}\sum_{i=1}^{M}\alpha
_{i}\phi_{i}$ which completes the first half of the proof. To prove the result about the KL divergence, can directly calculate $\mathcal{D}_{KL}(p_{f}\|p_{g})$

$$ $\displaystyle\mathcal{D}_{KL}(p_{f}\|p_{g})$ $\displaystyle=\frac{1}{\sqrt{2\pi\sigma^{2}}}\int p(x)e^{-\frac{(y-f(x))^{2}}{ 2\sigma^{2}}}\left[\frac{(y-g(x))^{2}}{2\sigma^{2}}-\frac{(y-f(x))^{2}}{2 \sigma^{2}}\right]\mathrm{d}x\mathrm{d}y$ (31) $\displaystyle=\frac{1}{\sqrt{2\pi\sigma^{2}}}\int p(x)e^{-\frac{(y-f(x))^{2}}{ 2\sigma^{2}}}\left[g(x)^{2}-f(x)^{2}-2yg(x)+2yf(x)\right]\mathrm{d}x\mathrm{d}y$ (32) $\displaystyle=\frac{1}{2\sigma^{2}}\left[\lVert f\rVert^{2}_{L_{2}}+\lVert g \rVert^{2}_{L_{2}}-2\langle f,g\rangle\right]$ (33) $\displaystyle=\frac{1}{2\sigma^{2}}\lVert f-g\rVert_{L_{2}(p)}^{2}$ (34) $$

For any $\delta>0$ we can choose $g=-\alpha f$ with $\alpha>\frac{\sigma\delta^{1/2}}{\lVert f\rVert_{L_{2}(p)}}$ which completes the proof.

### B.2 Proof of Lemma 3.3

We proceed by bounding the dynamics of the loss by an exponentially decaying dynamics, proving convergence to a global minimum. Then we show that the value of $\bm{\beta}$ at a global minimum is unique. To begin, note that the matrix

$$ $\bm{D}_{l}=\bm{W}_{l}^{T}\bm{W}_{l}-\bm{W}_{l+1}\bm{W}_{l+1}^{T}$ (35) $$

is an invariance of the gradient flow dynamics, so that $\bm{D}(t)=\bm{D}(0)=\alpha^{2}(\bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{l}-\bar{\bm{W
}}_{l+1}\bar{\bm{W}}_{l+1}^{T})$ for all time (Atanasov et al., ; Kunin et al., ; Yun et al., ). Let $\bm{r}=(\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}-\bm{\beta}_{\mathrm{s}})$ and note that

$$ $\displaystyle\dot{\mathcal{L}}$ $\displaystyle=\sum_{l=1}^{L}\langle\nabla_{l}\mathcal{L},\dot{\bm{W}}_{l}\rangle$ (36) $\displaystyle=-\sum_{l=1}^{L}\lVert\nabla_{l}\mathcal{L}\rVert_{F}^{2}$ (37) $\displaystyle\leq\lVert\nabla_{L}\mathcal{L}\rVert_{F}^{2}$ (38) $\displaystyle=-\lVert\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T}\bm{r}\rVert_{2}^{2}$ (39) $\displaystyle\leq-2\sigma_{\min}^{2}(\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T}) \mathcal{L}$ (40) $$

where $\sigma_{\min}(\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T})$ is the smallest singular value of $\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T}$. To proceed we bound $\sigma_{\min}(\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T})$ away from zero by showing that $\bm{W}_{L-1}\dots\bm{W}_{1}\bm{W}_{1}^{T}\dots\bm{W}_{L-1}^{T}$ is positive definite

$$ $\displaystyle\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T}\bm{W}_{1}\dots\bm{W}_{L-1}$ $\displaystyle=\bm{W}_{L-1}^{T}\dots\bm{W}_{2}^{T}(\bm{W}_{2}\bm{W}_{2}^{T}+\bm {D}_{1})\bm{W}_{2}\dots\bm{W}_{L-1}$ (42) $\displaystyle\succcurlyeq\bm{W}_{L-1}^{T}\dots\bm{W}_{3}^{T}(\bm{W}_{2}^{T}\bm {W}_{2})^{2}\bm{W}_{3}\dots\bm{W}_{L-1}$ (43) $\displaystyle\mathmakebox[\widthof{{}={}}][c]{\vdots}$ $\displaystyle\succcurlyeq(\bm{W}_{L-1}^{T}\bm{W}_{L-1})^{L-1}$ (44) $\displaystyle=(\bm{W}_{L}\bm{W}_{L}^{T}+\bm{D}_{L})^{L-1}$ (45) $\displaystyle\succcurlyeq(\alpha^{2}\lambda)^{L-1}$ (46) $$

where we have used the conservation law (35) and the initialization assumption (22). We now have

$$ $\displaystyle\dot{\mathcal{L}}$ $\displaystyle\leq-2(\alpha^{2}\lambda)^{L-1}\mathcal{L}$ (47) $\displaystyle\implies\mathcal{L}(t)=\mathcal{L}(0)e^{-2(\alpha^{2}\lambda)^{L- 1}t}$ (48) $\displaystyle\implies\lim_{t\to\infty}\mathcal{L}(t)=0$ (49) $$

Since the loss converges to zero, $\lim_{t\to\infty}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}=\lim_{t\to\infty}\bm{
\beta}=\bm{\beta}_{\mathrm{s}}$, which is unique. Note that while this solution is unique in function space, it is degenerate in parameter space.

### B.3 Proof of Theorem 3.4

To prove the feature space sparsification, we rely on the following Lemma, which is proven in (Yun et al., ) (see Section H.2). So that this work is self-contained, we include the proof here.

###### Lemma B.1 .

Under gradient flow on the population objective (4) or the empirical objective (5),

$$ $\bm{W}_{l}=\sigma_{l}(t)\bm{u}_{l}(t)\bm{v}_{l}(t)+\mathcal{O}(\alpha^{2})$ (50) $$

for all time. Furthermore

$$ $\lim_{\alpha\to 0}\lim_{t\to\infty}(\bm{u}_{l+1}(t)^{T}\bm{v}_{l}(t))^{2}=1$ (51) $$

###### Proof.

To prove Lemma B.1 we bound the difference $\lVert\bm{W}_{l}\rVert^{2}_{F}-\lVert\bm{W}_{l}\rVert^{2}_{op}$ which is equal to the norm of the subleading singular vectors of $\bm{W}_{l}$ and show that this bound is proportional to $\alpha^{2}$. The argument here follows that in (Yun et al. ()). Taking the trace of both sides in (35) we have

$$ $\lVert\bm{W}_{l}\rVert_{F}^{2}-\lVert\bm{W}_{l+1}\rVert_{F}^{2}=\alpha^{2}( \lVert\bar{\bm{W}}_{l}\rVert_{F}^{2}-\lVert\bar{\bm{W}}_{l+1}\rVert_{F}^{2})$ (52) $$

$$ $\displaystyle\sum_{k=l}^{L-1}\lVert\bm{W}_{k}\rVert_{F}^{2}-\lVert\bm{W}_{k+1} \rVert_{F}^{2}$ $\displaystyle=\alpha^{2}\sum_{k=l}^{L-1}(\lVert\bar{\bm{W}}_{k}\rVert_{F}^{2}- \lVert\bar{\bm{W}}_{k+1}\rVert_{F}^{2})$ (53) $\displaystyle\lVert\bm{W}_{l}\rVert_{F}^{2}-\lVert\bm{W}_{L}\rVert_{F}^{2}$ $\displaystyle=\alpha^{2}(\lVert\bar{\bm{W}}_{l}\rVert_{F}^{2}-\lVert\bar{\bm{W }}_{L}\rVert_{F}^{2})$ (54) $$

Let $\bm{u}_{l},\bm{v}_{l}$ be the top left and right singular vectors of $\bm{W}_{l}$. To bound the maximum singular value of $\bm{W}_{l}$ we have

$$ $\displaystyle\lVert\bm{W}_{l}\rVert_{\mathrm{op}}^{2}=\bm{v}_{l}^{T}\bm{W}_{l} ^{T}\bm{W}_{l}\bm{v}_{l}$ $\displaystyle\geq\bm{u}_{l+1}^{T}\bm{W}_{l}^{T}\bm{W}_{l}\bm{u}_{l+1}$ (55) $\displaystyle=\bm{u}_{l+1}^{T}(\bm{D}_{l}+\bm{W}_{l+1}^{T}\bm{W}_{l+1})\bm{u}_ {l+1}$ (56) $\displaystyle=\lVert\bm{W}_{l+1}\rVert_{\mathrm{op}}^{2}+\alpha^{2}\bm{u}_{l+1 }^{T}(\bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{l}-\bar{\bm{W}}_{l+1}\bar{\bm{W}}_{l+1 }^{T})\bm{u}_{l+1}$ (57) $\displaystyle\geq\lVert\bm{W}_{l+1}\rVert_{\mathrm{op}}^{2}+\alpha^{2}(\lVert \bar{\bm{W}}_{l+1}\rVert_{\mathrm{op}}^{2}-\lVert\bar{\bm{W}}_{l}\rVert_{ \mathrm{op}}^{2})$ (58) $$

Summing this inequality from $l$ to $L-1$ we have

$$ $\lVert\bm{W}_{l}\rVert_{\mathrm{op}}^{2}\geq\lVert\bar{\bm{W}}_{L}\rVert_{ \mathrm{op}}^{2}+\alpha^{2}(\lVert\bar{\bm{W}}_{L}\rVert_{\mathrm{op}}^{2}- \lVert\bar{\bm{W}}_{l}\rVert_{\mathrm{op}}^{2})$ (59) $$

Combining (53) and (59) we have

$$ $\lVert\bm{W}_{l}\rVert^{2}_{F}-\lVert\bm{W}_{l}\rVert^{2}_{\mathrm{op}}\leq \alpha^{2}(\lVert\bar{\bm{W}}_{l}\rVert_{F}^{2}-\lVert\bar{\bm{W}}_{L}\rVert_{ F}^{2}+\lVert\bar{\bm{W}}_{l}\rVert_{\mathrm{op}}^{2}-\lVert\bar{\bm{W}}_{L} \rVert_{\mathrm{op}}^{2})$ (60) $$

This shows all of the parameter matrices are approximately rank one with corrections upper bounded by $\mathcal{O}(\alpha^{2})$, proving the first claim. To show the alignment of adjacent singular vectors we again take advantage of the invariant quantity (35)

$$ $\displaystyle\bm{v}_{l}^{T}\bm{W}_{l+1}\bm{W}_{l+1}^{T}\bm{v}_{l}$ $\displaystyle=\bm{v}_{l}^{T}\bm{W}_{l}^{T}\bm{W}_{l}\bm{v}_{l}-\bm{v}_{l}^{T} \bm{D}_{l}\bm{v}_{l}$ (61) $\displaystyle\geq s_{l}^{2}-\alpha^{2}\lVert\bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{ l}-\bar{\bm{W}}_{l+1}\bar{\bm{W}}_{l+1}^{T}\rVert_{\mathrm{op}}^{2}$ (62) $$

we also derive the following upper bound on (62)

$$ $\displaystyle\bm{v}_{l}^{T}\bm{W}_{l+1}\bm{W}_{l+1}^{T}\bm{v}_{l}$ $\displaystyle=\bm{v}_{l}^{T}(s_{l+1}^{2}\bm{u}_{l+1}\bm{u}_{l+1^{T}}\bm{W}_{l+ 1}\bm{W}_{l+1}^{T}-s_{l+1}^{2}\bm{u}_{l+1}\bm{u}_{l+1^{T}})\bm{v}_{l}$ (63) $\displaystyle\leq s_{l+1}^{2}(\bm{v}_{l}^{T}\bm{u}_{l+1})^{2}+\lVert\bm{W}_{l+ 1}\rVert_{F}^{2}-\lVert\bm{W}_{l+1}\rVert_{F}^{2}$ (64) $$

combining these two bounds

$$ $\displaystyle s_{l}^{2}$ $\displaystyle\leq s_{l+1}^{2}(\bm{v}_{l}^{T}\bm{u}_{l+1})^{2}+\alpha^{2}\lVert \bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{l}-\bar{\bm{W}}_{l+1}\bar{\bm{W}}_{l+1}^{T} \rVert_{\mathrm{op}}^{2}+\lVert\bm{W}_{l+1}\rVert_{F}^{2}-\lVert\bm{W}_{l+1} \rVert_{F}^{2}$ (65) $\displaystyle\leq s_{l+1}^{2}(\bm{v}_{l}^{T}\bm{u}_{l+1})^{2}+\alpha^{2}\lVert \bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{l}-\bar{\bm{W}}_{l+1}\bar{\bm{W}}_{l+1}^{T} \rVert_{\mathrm{op}}^{2}+\alpha^{2}(\lVert\bar{\bm{W}}_{l}\rVert_{F}^{2}- \lVert\bar{\bm{W}}_{L}\rVert_{F}^{2}+\lVert\bar{\bm{W}}_{l}\rVert_{\mathrm{op} }^{2}-\lVert\bar{\bm{W}}_{L}\rVert_{\mathrm{op}}^{2})$ (66) $$

where we have used the result derived in the previous proof for the second inequality. Finally, we derive an upper bound on this quantity

$$ $\displaystyle s_{l}^{2}$ $\displaystyle\geq\bm{u}_{l+1}^{T}\bm{W}_{l}^{T}\bm{W}_{l}\bm{u}_{l+1}$ (67) $\displaystyle\geq s_{l+1}^{2}-\alpha^{2}\lVert\bar{\bm{W}}_{l}^{T}\bar{\bm{W}} _{l}-\bar{\bm{W}}_{l+1}\bar{\bm{W}}_{l+1}^{T}\rVert_{\mathrm{op}}^{2}$ (68) $$

We can combine the upper and lower bounds and divide by $s_{l+1}^{2}$ to conclude

$$ $\displaystyle(\bm{v}_{l}^{T}\bm{u}_{l+1})^{2}$ $\displaystyle\geq 1-\alpha^{2}\frac{C_{l}}{s_{l+1}^{2}}$ (69) $\displaystyle C_{l}$ $\displaystyle=2\lVert\bar{\bm{W}}_{l}^{T}\bar{\bm{W}}_{l}-\bar{\bm{W}}_{l+1} \bar{\bm{W}}_{l+1}^{T}\rVert_{\mathrm{op}}^{2}+\lVert\bar{\bm{W}}_{l}\rVert_{F }^{2}-\lVert\bar{\bm{W}}_{L}\rVert_{F}^{2}+\lVert\bar{\bm{W}}_{l}\rVert_{ \mathrm{op}}^{2}-\lVert\bar{\bm{W}}_{L}\rVert_{\mathrm{op}}^{2}$ (70) $$

This proves that adjacent singular vectors align as long as the singular values are bounded away from zero. To show that this requirement is satisfied at the end of training, note that in the proofs of Lemma 3.3 and Theorem 3.5 we show that gradient flow converges to a global minimizer of the loss. Let $\hat{\bm{y}}=\lim_{t\to\infty}\bm{X}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}$ denote the final network predictions. Then

$$ $\displaystyle\frac{\lVert\hat{\bm{y}}\rVert_{2}}{\lVert\bm{X}\rVert_{\mathrm{ op}}}$ $\displaystyle\leq\lim_{t\to\infty}\lVert\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L} \rVert_{2}\leq\lim_{t\to\infty}\prod_{l=1}^{L}s_{l}^{2}$ (71) $$

If $d\geq n$, $\hat{\bm{y}}$ is just equal to the vector of target outputs which is larger than zero by construction. If $d<n$, $\hat{\bm{y}}$ is the projection of the targets into the space spanned by the rows of $\bm{X}$, which is almost surely a non-zero vector. This implies that

$$ $\lim_{t\to\infty}\prod_{l=1}^{L}s_{l}^{2}>0$ (72) $$

which implies that the individual singular values are bounded away from zero at the end of training. In the population training case, the proof is nearly same, replacing $\hat{\bm{y}}=\lim_{t\to\infty}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}=\bm{\beta}_{
\mathrm{s}}$

∎

By Lemma B.1, we have

$$ $\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L-1}=c\bm{u}_{1}\bm{v}_{L-1}^{T}$ (73) $$

after pretraining, for some $c\in\mathbb{R}$. However, from Theorem 3.5 we know that after pretraining

$$ $\displaystyle\bm{W}_{1}\dots\bm{W}_{L-1}\bm{W}_{L}$ $\displaystyle=\bm{\beta}_{\mathrm{s}}$ (74) $\displaystyle=c\bm{u}_{1}(\bm{v}_{L_{1}}^{T}\bm{W}_{L})$ (75) $\displaystyle=c\bm{u}_{1}$ (76) $$

where we have used Lemma B.1 in the third equality to eliminate the inner product between the adjacent singular vectors. The possible factor of $-1$ can be absorbed into the definition of $\bm{u}_{1}$. This implies

$$ $\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L-1}=\bm{\beta}_{\mathrm{s}}\bm{v}_{L-1}^{T}$ (77) $$

###### Lemma B.1 .

###### Proof.

### B.4 Proof of Theorem 3.5

This proof follows Yun et al. () closely but extends their result to the case $n>d$. We first show that gradient flow converges to a global minimum of the empirical loss (5). We then show that as $\alpha\to 0$, this minimum corresponds to the minimum norm least squares solution.
Part 1: Gradient flow converges to a global minimum
This proof follows the same logic as the proof for Lemma 3.3. First, we define the residual vector $\bm{r}=\bm{X}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}-\bm{y}_{\textrm{t}}$. Then we can write the empirical loss as

$$ $\mathcal{L}=\frac{1}{2n}\lVert\bm{r}\rVert_{2}^{2}=\frac{1}{2n}(\lVert\bm{r}_{ \parallel}\rVert_{2}^{2}+\lVert\bm{r}_{\perp}\rVert_{2}^{2})$ (78) $$

where $\bm{r}_{\parallel}$ is the component of $\bm{r}$ in $\text{im}(\bm{X})$ and $\bm{r}_{\perp}$ is the component of $\bm{r}$ in $\ker(\bm{X}^{T})$. Since $\bm{X}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}\in\text{im}(\bm{X})$, the global minimum of (78) is equal to $\lVert\bm{r}_{\perp}\rVert_{2}^{2}$. Therefore, to show that gradient flow converges to a global minimum it is sufficient to show that $\lim_{t\to\infty}\lVert\bm{r}_{\parallel}(t)\rVert_{2}^{2}=0$. Let $\bm{P}_{\parallel}$ and $\bm{P}_{\perp}$ be the orthogonal projectors onto $\text{im}(\bm{X})$ and $\ker(\bm{X}^{T})$ respectively, so that $\mathcal{L}_{\parallel}\vcentcolon=\lVert\bm{r}_{\parallel}\rVert_{2}^{2}=
\lVert\bm{P}_{\parallel}(\bm{X}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}-\bm{y}_{
\textrm{t}})\rVert_{2}^{2}$ and $\mathcal{L}_{\perp}\vcentcolon=\lVert\bm{r}_{\perp}\rVert_{2}^{2}=\lVert\bm{P}
_{\perp}(\bm{X}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}-\bm{y}_{\textrm{t}})\rVert_
{2}^{2}$. Then we have

$$ $\displaystyle\dot{\mathcal{L}_{\parallel}}$ $\displaystyle=\sum_{l=1}^{L}\langle\nabla_{l}\mathcal{L}_{\parallel},\dot{\bm{ W}}_{l}\rangle$ (79) $\displaystyle=-\sum_{l=1}^{L}\langle\nabla_{l}\mathcal{L}_{\parallel},\nabla_{ l}\mathcal{L}\rangle$ (80) $\displaystyle=-\sum_{l=1}^{L}(\lVert\nabla_{l}\mathcal{L}_{\parallel}\rVert_{F }^{2}+\langle\nabla_{l}\mathcal{L}_{\parallel},\nabla_{l}\mathcal{L}_{\perp}\rangle)$ (81) $$

Taking the gradient of $\mathcal{L}_{\perp}$ we have

$$ $\nabla_{l}\mathcal{L}_{\perp}=\bm{W}_{l-1}^{T}\dots\bm{W}_{1}^{T}\bm{X}^{T}\bm {P}_{\perp}\bm{r}\bm{W}_{L}^{T}\dots\bm{W}_{l+1}^{T}=0$ (82) $$

so

$$ $\displaystyle\dot{\mathcal{L}_{\parallel}}$ $\displaystyle=-\sum_{l=1}^{L}\lVert\nabla_{l}\mathcal{L}_{\parallel}\rVert_{F} ^{2}$ (83) $\displaystyle\leq-\lVert\nabla_{L}\mathcal{L}_{\parallel}\rVert_{F}^{2}$ (84) $\displaystyle=-\lVert\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T}\bm{X}^{T}\bm{P}_{ \parallel}\bm{r}\rVert_{2}^{2}$ (85) $\displaystyle\leq-\sigma_{\min}^{2}(\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T})\lVert \bm{X}^{T}\bm{P}_{\parallel}\bm{r}\rVert_{2}^{2}$ (86) $$

where $\sigma_{\min}(\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T})$ is the smallest singular value of $\bm{W}_{L-1}^{T}\dots\bm{W}_{1}^{T}$. From Eq. (42) - (46) we can bound this quantity away from zero. Then we have

$$ $\displaystyle\dot{\mathcal{L}_{\parallel}}$ $\displaystyle\leq-(\alpha^{2}\lambda)^{L-1}\lVert\bm{X}^{T}\bm{P}_{\parallel} \bm{r}\rVert_{2}^{2}$ (87) $\displaystyle\leq-2(\alpha^{2}\lambda)^{L-1}\lambda_{\min}\mathcal{L}_{\parallel}$ (88) $$

where $\lambda_{\min}$ is the smallest nonzero eigenvalue of $\bm{X}\bm{X}^{T}$. The solution to the dynamics (88) is $\mathcal{L}_{\parallel}(t)\leq\mathcal{L}_{\parallel}(0)e^{-2(\alpha^{2}
\lambda)^{L-1}\lambda_{\min}t}$, which proves $\lim_{t\to\infty}\lVert\bm{r}_{\parallel}(t)\rVert_{2}^{2}=0$. Note that this part of the theorem holds for any $\alpha$,$n$,$d$, and we take the limit $\alpha\to 0$ after $t\to\infty$.
Part 2: as $\alpha\to 0$, gradient flow finds the minimum norm interpolator
In the case $n>d$, the least squares problem () is overdetermined so the solution is unique. That is, the unique solution is trivially the minimum norm solution. In the case $n\leq d$, there are multiple $\bm{\beta}(t)$ that yield zero training error. Lemma B.1 shows that the parameter matrices are approximately rank one at all times and $\bm{u}_{l+1}$ and $\bm{v}_{l}$ align at the end of training as $\alpha\to 0$, which means that

$$ $\lim_{\alpha\to 0}\lim_{t\to\infty}\bm{\beta}(t)=\lim_{\alpha\to 0}\lim_{t\to \infty}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}=c\bm{u}_{1}$ (89) $$

where $c>0$. Next we show that $\bm{u}_{l}\in\text{row}(\bm{X})$. We can break $\bm{W}_{1}$ into two components $\bm{W}_{1}^{\parallel}$ and $\bm{W}_{1}^{\perp}$ where the columns of $\bm{W}_{1}^{\parallel}$ are in $\text{row}({\bm{X}})$ and the columns of $\bm{W}_{1}^{\perp}$ are in $\ker({\bm{X}^{T}})$. The left hand side of (82) also shows that the gradient of $\bm{W}_{1}^{\perp}$ is zero, which means that this component remains unchanged under gradient flow dynamics. Therefore we have

$$ $\lVert\bm{W}_{1}^{\perp}(t)\rVert_{F}=\lVert\bm{W}_{1}^{\perp}(0)\rVert_{F} \leq\alpha\lVert\bar{\bm{W}}_{1}\rVert_{F}$ (90) $$

which vanishes in the limit $\alpha\to 0$. This implies that $\bm{u}_{1}\in\text{row}(\bm{X})$ at all times. The only global minimizer with this property is the minimum norm solution. As a final comment, we note that this theorem is also proven in Atanasov et al. () using different techniques.

### B.5 Proof of Theorem 3.6

Let $\hat{\bm{\beta}}=\lim_{t\to\infty}\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L}$. From Theorem 3.5, $\hat{\bm{\beta}}=\bm{X}^{+}\bm{y}=\bm{X}^{+}\bm{X}\bm{\beta}_{\mathrm{t}}+\bm{
X}^{+}\bm{\epsilon}$. Then the average generalization error at the end of training can be written

$$ $\displaystyle\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}$ $\displaystyle=\mathbb{E}_{\bm{X},\bm{\epsilon}}\lVert\bm{\beta}_{\mathrm{t}}- \hat{\bm{\beta}}\rVert_{2}^{2}$ (91) $\displaystyle=1+\mathbb{E}_{\bm{X},\bm{\epsilon}}\lVert\hat{\bm{\beta}}\rVert_ {2}^{2}-2\mathbb{E}_{\bm{X},\bm{\epsilon}}\langle\hat{\bm{\beta}},\bm{\beta}_{ \mathrm{t}}\rangle$ (92) $\displaystyle=1+\mathbb{E}_{\bm{X}}\bm{\beta}_{\mathrm{t}}^{T}(\bm{X}^{+}\bm{X })^{T}(\bm{X}^{+}\bm{X})\bm{\beta}_{\mathrm{t}}+\mathbb{E}_{\bm{X},\bm{ \epsilon}}\bm{\epsilon}^{T}(\bm{X}^{+})^{T}\bm{X}^{+}\bm{\epsilon}+2\mathbb{E} _{\bm{X},\bm{\epsilon}}\bm{\epsilon}^{T}(\bm{X}^{+}\bm{X})\bm{\beta}_{\mathrm{ t}}$ (93) $\displaystyle-2(\mathbb{E}_{\bm{X}}\bm{\beta}_{\mathrm{t}}^{T}(\bm{X}^{+}\bm{X })\bm{\beta}_{\mathrm{t}}+\mathbb{E}_{\bm{X},\bm{\epsilon}}\bm{\beta}_{\mathrm {t}}^{T}\bm{X}^{+}\bm{\epsilon})$ (94) $\displaystyle=1-\mathbb{E}_{\bm{X}}\lVert\bm{P}_{\text{row}(\bm{X})}\bm{\beta} _{\mathrm{t}}\rVert_{2}^{2}+\sigma^{2}\mathbb{E}_{\bm{X}}\text{tr}((\bm{X}^{+} )^{T}\bm{X}^{+})$ (95) $$

where we have used the independence of $\bm{\epsilon}$ and $\bm{X}$, as well as the fact that the operator $\bm{X}^{+}\bm{X}$ is the projector onto subspace spanned by the rows of $\bm{X}$, $\bm{P}_{\text{row}(\bm{X})}$. Since the entries of the data matrix $\bm{X}$ are independent Gaussians, the n-dimensional subspace $\text{row}(\bm{X})$ is uniformly random in the Grassmanian manifold $\mathcal{G}_{n,d}$ Vershynin (), so $\bm{P}_{\text{row}(\bm{X})}\bm{\beta}_{\mathrm{t}}$ is a random projection of $\bm{\beta}_{\mathrm{t}}$. Then

$$ $\mathbb{E}_{\bm{X}}\lVert\bm{P}_{\text{row}(\bm{X})}\bm{\beta}_{\mathrm{t}} \rVert_{2}^{2}=\gamma$ (96) $$

which is a classic result in the theory of random projections (c.f. Vershynin () Lemma 5.3.2). We now turn to the final term in (95). Let $\{\sigma_{l}\}_{l\leq\min(n,d)}$ be the nonzero singular values of the data matrix $\bm{X}$. Then

$$ $\mathbb{E}_{\bm{X}}\text{tr}((\bm{X}^{+})^{T}\bm{X}^{+})=\mathbb{E}_{\bm{X}} \sum_{l=1}^{\min(n,d)}\frac{1}{\sigma_{l}^{2}}$ (97) $$

First take the case $\gamma<1$. Then there are $n$ nonzero singular values of $\bm{X}$, which are the eigenvalues of the Wishart matrix $\bm{C}=\frac{1}{d}\bm{X}\bm{X}^{T}$ and

$$ $\displaystyle\mathbb{E}_{\bm{X}}\text{tr}((\bm{X}^{+})^{T}\bm{X}^{+})$ $\displaystyle=\frac{\gamma}{n}\mathbb{E}_{\bm{X}}\text{tr}(\bm{C}^{-1})$ (98) $\displaystyle=-\gamma\lim_{z\to 0}\frac{1}{n}\mathbb{E}[\text{tr}((z\bm{I}-\bm {C})^{-1})]$ (99) $\displaystyle=-\gamma\lim_{z\to 0}\mathfrak{g}_{\bm{C}}(z)$ (100) $$

In the second line we have introduced the complex variable $z$, which casts the quantity of interest as the $z\to 0$ limit of the normalized expected trace of the resolvent of $\bm{C}$. In the limit of large $n$, this quantity tends to the Stieltjes transform of the Wishart matrix $\mathfrak{g}_{\bm{C}}(z)$, which has a closed form expression (see Potters & Bouchaud () Ch.4 for a proof).

$$ $\displaystyle\lim_{z\to 0}\mathfrak{g}_{\bm{C}}(z)$ $\displaystyle=\lim_{z\to 0}\frac{z-(1-\gamma)-\sqrt{z-(1+\sqrt{\gamma})^{2}} \sqrt{z-(1-\sqrt{\gamma})^{2}}}{2\gamma z}$ (101) $\displaystyle=-\frac{1}{1-\gamma}$ (102) $$

so $\mathbb{E}_{\bm{X}}\text{tr}((\bm{X}^{+})^{T}\bm{X}^{+})=\frac{\gamma}{1-\gamma}$ for $\gamma<1$. In the case $\gamma>1$, there will be $d$ terms in the sum (97), which are proportional to the eigenvalues of the covariance matrix $\frac{1}{n}\bm{X}^{T}\bm{X}$. If we define $n^{\prime}=d,d^{\prime}=n,\gamma^{\prime}=n^{\prime}/d^{\prime}$ and $\bm{X}^{\prime}=\bm{X}^{T}\in\mathbb{R}^{n^{\prime}\times d^{\prime}}$, equations (98) - (100) hold under the substitution $\gamma\to\gamma^{\prime}$. So $\mathbb{E}_{\bm{X}}\text{tr}((\bm{X}^{+})^{T}\bm{X}^{+})=\frac{\gamma^{\prime}
}{1-\gamma^{\prime}}=\frac{1}{\gamma-1}$ for $\gamma>1$. Putting everything together we have

$$ $\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}=\begin{cases}\frac{(1-\gamma)^{2} +\gamma\sigma^{2}}{1-\gamma}&\gamma<1\\ \frac{\sigma^{2}}{\gamma-1}&\gamma>1\end{cases}$ (103) $$

### B.6 Proof of Theorem 3.7

Theorem 3.4 implies that the pretrained feature matrix is $\bm{\Phi}=(\bm{X}\bm{\beta}_{\mathrm{s}})\bm{v}_{L-1}^{T}$. Since $\bm{\Phi}$ is a rank one matrix its pseudoinverse is easy to compute

$$ $\bm{\Phi}^{+}=\frac{1}{\lVert\bm{X}\bm{\beta}_{\mathrm{s}}\rVert_{2}^{2}}\bm{v }_{L-1}(\bm{X}\bm{\beta}_{\mathrm{s}})^{T}$ (104) $$

The coefficent vector $\hat{\bm{\beta}}$ after linear transfer is

$$ $\displaystyle\hat{\bm{\beta}}$ $\displaystyle=\bm{W}_{1}\dots\bm{W}_{L-1}\hat{\bm{W}}_{L}$ (105) $\displaystyle=\bm{W}_{1}\dots\bm{W}_{L-1}\bm{\Phi}^{+}\bm{y}_{\textrm{t}}$ (106) $\displaystyle=b\bm{\beta}_{\mathrm{s}}$ (107) $$

where

$$ $\displaystyle b$ $\displaystyle=\frac{\beta_{src}^{T}\bm{X}^{T}\bm{y}_{\textrm{t}}}{\bm{\beta}_{ \mathrm{s}}^{T}\bm{X}^{T}\bm{X}\bm{\beta}_{\mathrm{s}}}$ (108) $\displaystyle=\frac{\beta_{src}^{T}\bm{X}^{T}\bm{X}\bm{\beta}_{\mathrm{t}}}{ \bm{\beta}_{\mathrm{s}}^{T}\bm{X}^{T}\bm{X}\bm{\beta}_{\mathrm{s}}}+\frac{ \beta_{src}^{T}\bm{X}^{T}\bm{\epsilon}}{\bm{\beta}_{\mathrm{s}}^{T}\bm{X}^{T} \bm{X}\bm{\beta}_{\mathrm{s}}}$ (109) $$

As in the proof of Theorem 3.6, we can write the typical generalization error as

$$ $\displaystyle\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{lt}$ $\displaystyle=\lVert\hat{\bm{\beta}}-\bm{\beta}_{\mathrm{t}}\rVert_{2}^{2}$ (110) $\displaystyle=1+\mathbb{E}_{\bm{X},\bm{\epsilon}}b^{2}-2\cos\theta\mathbb{E}_{ \bm{X},\bm{\epsilon}}b$ (111) $$

To proceed, we can write $\bm{\beta}_{\mathrm{t}}=\cos\theta\bm{\beta}_{\mathrm{s}}+\sin\theta\bm{\nu}$ for some vector $\bm{\nu}\perp\bm{\beta}_{\mathrm{s}}$, and introduce the independent $n-$dimensional Gaussian vectors $\bm{z}=\bm{X}\bm{\beta}_{\mathrm{s}}\sim\mathcal{N}(0,\bm{I}_{n})$ and $\bm{w}=\bm{X}\bm{\nu}\sim\mathcal{N}(0,\bm{I}_{n})$. With this change of variables we have

$$ $\displaystyle\mathbb{E}_{\bm{X},\bm{\epsilon}}b$ $\displaystyle=\mathbb{E}_{\bm{z},\bm{w},\bm{\epsilon}}b$ (112) $\displaystyle=\cos\theta$ (113) $\displaystyle\mathbb{E}_{\bm{X},\bm{\epsilon}}b^{2}$ $\displaystyle=\mathbb{E}_{\bm{z},\bm{w},\bm{\epsilon}}b^{2}$ (114) $\displaystyle=\cos^{2}\theta+(\sin^{2}\theta+\sigma^{2})\mathbb{E}_{\bm{z}} \frac{1}{\lVert\bm{z}\rVert_{2}^{2}}$ (115) $$

The integral $\mathbb{E}_{\bm{z}}\frac{1}{\lVert\bm{z}\rVert_{2}^{2}}$ can be solved exactly

$$ $\displaystyle\mathbb{E}_{\bm{z}}\frac{1}{\lVert\bm{z}\rVert_{2}^{2}}$ $\displaystyle=\frac{1}{(2\pi)^{n/2}}\int_{-\infty}^{\infty}\frac{e^{-\sum_{i=1 }^{n}z_{i}^{2}/2}}{\sum_{j=1}z_{j}^{2}}d\bm{z}$ (116) $\displaystyle=\frac{S_{n-1}}{(2\pi)^{n/2}}\int_{0}^{\infty}r^{n-3}e^{r^{2}/2}dr$ (117) $\displaystyle=\frac{S_{n-1}}{4\pi^{n/2}}\int_{0}^{\infty}e^{-t}t^{\frac{n}{2}- 2}dt$ (118) $\displaystyle=\frac{S_{n-1}}{4\pi^{n/2}}\Gamma\left(\frac{n}{2}-1\right)$ (119) $\displaystyle=\frac{1}{n-2}$ (120) $$

which completes the proof.

### B.7 Proof of Theorem 3.8

We begin by writing down the solution to the optimization problem (14)

$$ $\hat{\bm{W}_{L}}=(\bm{\Phi}^{T}\bm{\Phi}+n\lambda\bm{I}_{d})^{-1}\bm{\Phi}^{T} \bm{y}_{\textrm{t}}$ (121) $$

As in the proof of Theorem 3.7, we have

$$ $\displaystyle\bm{\Phi}$ $\displaystyle=(\bm{X}\bm{\beta}_{\mathrm{s}})\bm{v}_{L-1}^{T}$ (122) $\displaystyle\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L-1}=\bm{\beta}_{\mathrm{s}}\bm{ v}_{L-1}^{T}$ (123) $$

Combining these expressions we can solve for the linear function the network implements after transfer learning with ridge regression

$$ $\displaystyle\hat{\bm{\beta}}$ $\displaystyle=\bm{W}_{1}\bm{W}_{2}\dots\bm{W}_{L-1}\hat{\bm{W}_{L}}$ (124) $\displaystyle=\bm{\beta}_{\mathrm{s}}\bm{v}_{L-1}^{T}(\lVert\bm{X}\bm{\beta}_{ \mathrm{s}}\rVert_{2}^{2}+n\lambda\bm{I}_{d})^{-1}\bm{v}_{L-1}(\bm{X}\bm{\beta }_{\mathrm{s}})^{T}\bm{y}_{\textrm{t}}$ (125) $\displaystyle=\left(\frac{(\bm{X}\bm{\beta}_{\mathrm{s}})^{T}\bm{y}_{\textrm{t }}}{\lVert\bm{X}\bm{\beta}_{\mathrm{s}}\rVert_{2}^{2}+n\lambda}\right)\bm{ \beta}_{\mathrm{s}}$ (126) $$

As in the proof of Theorem 3.7, we write $\bm{\beta}_{\mathrm{t}}=\cos\theta\bm{\beta}_{\mathrm{s}}+\sin\theta\bm{\nu}$ for some vector $\bm{\nu}\perp\bm{\beta}_{\mathrm{s}}$, and introduce the independent $n-$dimensional Gaussian vectors $\bm{z}=\bm{X}\bm{\beta}_{\mathrm{s}}\sim\mathcal{N}(0,\bm{I}_{n})$ and $\bm{w}=\bm{X}\bm{\nu}\sim\mathcal{N}(0,\bm{I}_{n})$. Then we can get the following expression for the generalization error of ridge linear transfer:

$$ $\displaystyle\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{lt}^{\lambda}$ $\displaystyle=\lVert\hat{\bm{\beta}}-\bm{\beta}_{\mathrm{t}}\rVert_{2}^{2}$ (127) $\displaystyle=1+(\cos^{2}\theta)I_{1}(n+2,\lambda)+(\sin^{2}\theta+\sigma^{2}) I_{1}(n,\lambda)-(2\cos^{2}\theta)I_{2}(n,\lambda)$ (128) $$

where we have used spherical coordinates to define the following integrals

$$ $\displaystyle I_{1}(m,\lambda)$ $\displaystyle=\mathbb{E}_{z}\left(\frac{\lVert z\rVert_{2}^{m-n+2}}{(\lVert z \rVert_{2}^{2}+n\lambda)^{2}}\right)=\frac{S_{n-1}}{(2\pi)^{n/2}}\int_{0}^{ \infty}\frac{r^{m+1}e^{-r^{2}/2}}{(r^{2}+n\lambda)^{2}}dr$ (129) $\displaystyle I_{2}(m,\lambda)$ $\displaystyle=\mathbb{E}_{z}\left(\frac{\lVert z\rVert_{2}^{m-n+2}}{\lVert z \rVert_{2}^{2}+n\lambda}\right)=\frac{S_{n-1}}{(2\pi)^{n/2}}\int_{0}^{\infty} \frac{r^{m+1}e^{-r^{2}/2}}{r^{2}+n\lambda}dr$ (130) $$

We evaluate $I_{1}(n,\lambda),I_{1}(n+2,\lambda)$ and $I_{2}(n,\lambda)$ for large $n$. To avoid cluttering the notation, we ignore the coefficient $\frac{S_{n-1}}{(2\pi)^{n/2}}$ while solving the integral and restore it at the end of the calculation. Then

$$ $\displaystyle I_{1}(n,\lambda)$ $\displaystyle\propto 2^{n/2}\int\frac{u^{n/2}e^{-u}}{(2u+n\lambda)^{2}}du$ (131) $\displaystyle=n(2n)^{n/2}\int\frac{t^{n/2}e^{-nt}}{(2nt+n\lambda)^{2}}dt$ (132) $\displaystyle=n(2n)^{n/2}\int g(t)e^{nf(t)}dt$ (133) $\displaystyle\approx n(2n)^{n/2}\sqrt{\frac{2\pi}{n\lvert f^{\prime\prime}(t_{ 0})\rvert}}g(t_{0})e^{nf({t_{0}})}$ (134) $$

We have introduced the change of variables $u=r^{2}/2$ in the first line, $t=u/n$ in the second line, and finally evaluated the integral for large $n$ using the saddle point method. In the last line, $t_{0}$ is a critical point of $f(t)=\frac{1}{2}\log{t}-t$ and $g(t)=(2nt+n\lambda)^{-2}$. Differentiating $f(t)$ and setting equal to zero we find $t_{0}=1/2$. So for large $n$,

$$ $\displaystyle I_{1}(n,\lambda)\propto\frac{\sqrt{\pi n}n^{n/2}e^{-n/2}}{(n+n \lambda)^{2}}$ (135) $$

We can now restore the angular coefficient to the integral

$$ $\displaystyle I_{1}(n,\lambda)$ $\displaystyle=\frac{S_{n-1}}{(2\pi)^{n/2}}\frac{\sqrt{\pi n}n^{n/2}e^{-n/2}}{( n+n\lambda)^{2}}$ (136) $\displaystyle\approx\frac{n\pi^{n/2}}{\sqrt{\pi n}}\left(\frac{n}{2}\right)^{- n/2}e^{n/2}\frac{\sqrt{\pi n}n^{n/2}e^{-n/2}}{(n+n\lambda)^{2}}$ (137) $\displaystyle=\frac{1}{n(1+\lambda)^{2}}$ (138) $$

where we have used Stirling’s approximation in the second line. Therefore, $\lim_{n\to\infty}I_{1}(n,\lambda)=0$. We stress that although the integral was approximated at the saddle point, the limit $n\to\infty$ is exact since corrections to the saddle point value are subleading in $n$. Similar calculations yield

$$ $\displaystyle I_{1}(n+2,\lambda)$ $\displaystyle=\frac{1}{(1+\lambda)^{2}}$ (139) $\displaystyle I_{2}(n,\lambda)$ $\displaystyle=\frac{1}{1+\lambda}$ (140) $$

for large $n$. Plugging this into (127), we have

$$ $\lim_{n\to\infty}\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{lt}^{\lambda}=1 -\frac{(1+2\lambda)}{(1+\lambda)^{2}}\cos^{2}\theta$ (141) $$

This is a strictly increasing function in $\lambda\geq 0$ for any $\theta\in[0,\pi/2]$, which implies that the optimal regularization value is $\lambda^{*}=0$.

### B.8 Proof of Theorem 3.9

The proof involves slightly tweaking the proof of Theorem 3.5. Since the source trained model obeyed the initialization assumption (22), the invariant matrix (35) is equal to its value at initialization before pretraining throughout fine tuning as well. This implies that the first half of the proof of Theorem (3.5) holds in the fine tuning case and the model will converge to a global minimizer of the training loss. The invariance throughout fine tuning also implies that (89) holds and that $\bm{W}_{1}^{\perp}$ does not change during fine tuning, and remains fixed at its initial value from pretraining. Therefore, by the proof of Theorem 3.7, at the beginning of fine tuning, $\bm{u}_{1}=\bm{\beta}_{\mathrm{s}}$ and $(\bm{I}-\bm{P}_{\text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}$ is the component of $\bm{u}_{1}$ that does not evolve. Meanwhile, $\bm{P}_{\text{row}}(\bm{X})\bm{u}_{1}$ will evolve to the minimum norm solution. Combining these results, after fine tuning,

$$ $\lim_{\alpha\to 0}\lim_{t\to\infty}\bm{\beta}_{ft}(t)=\bm{\beta}_{sc}+(\bm{I}- \bm{P}_{\text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}$ (142) $$

where $\bm{\beta}_{sc}$ is the minimum norm solution. We can now write the expected generalization error

$$ $\displaystyle\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{ft}$ $\displaystyle=\mathbb{E}_{\bm{X},\bm{\epsilon}}[\lVert\bm{\beta}_{\mathrm{t}}- \bm{\beta}_{\mathrm{ft}}\rVert_{2}^{2}]$ $\displaystyle=\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{sc}+\mathbb{E}_{ \bm{X}}\lVert(\bm{I}-\bm{P}_{\text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}\rVert _{2}^{2}-2\mathbb{E}_{\bm{X}}\langle\bm{\beta}_{\mathrm{t}},(\bm{I}-\bm{P}_{ \text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}\rangle$ $\displaystyle=\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{sc}+\max({0,1- \gamma})-2\mathbb{E}_{\bm{X}}\langle\bm{\beta}_{\mathrm{t}},(\bm{I}-\bm{P}_{ \text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}\rangle$ $\displaystyle=\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{sc}+\max({0,1- \gamma})-2\cos\theta\mathbb{E}_{\bm{X}}\langle\bm{\beta}_{\mathrm{s}},(\bm{I}- \bm{P}_{\text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}\rangle$ $\displaystyle-2\sin\theta\mathbb{E}_{\bm{X}}\langle\bm{\nu},(\bm{I}-\bm{P}_{ \text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}\rangle$ $\displaystyle=\mathbb{E}_{\bm{X},\bm{\epsilon}}\mathcal{R}_{sc}+\max({0,1- \gamma})(1-2\cos\theta)-2\mathbb{E}_{\bm{X}}\sin\theta\langle\bm{\nu},(\bm{I}- \bm{P}_{\text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}\rangle$ $$

where we have used the fact that $\bm{P}_{\text{row}(\bm{X})})\bm{\beta}_{\mathrm{s}}$ is a random projection as in the proof of Theorem 3.6 and set $\bm{\beta}_{\mathrm{t}}=\cos\theta\bm{\beta}_{\mathrm{s}}+\sin\theta\bm{\nu}$ for some $\bm{\nu}\perp\bm{\beta}_{\mathrm{s}}$. The final term is equal to zero for the following reason. The operator $\bm{I}-\bm{P}_{\text{row}(\bm{X})}$ is a random projector onto the $d-n$ dimensional subspace orthogonal to $\text{row}{X}$ Since the uniform distribution of random subspaces is rotationally invariant, we can instead fix a particular subspace and average over $\bm{\beta}_{\mathrm{s}}\sim\mathrm{Uniform}(S^{d-1})$. Using rotation invariance again, we can fix the projection to be along the first $d-n$ coordinates of $\bm{\beta}_{\mathrm{s}}$. Then we have

$$ $\displaystyle\mathbb{E}\langle\bm{\nu},(\bm{I}-\bm{P}_{\text{row}(\bm{X})})\bm {\beta}_{\mathrm{s}}\rangle$ $\displaystyle=\sum_{k=1}^{n-d}\bm{\nu}_{k}\mathbb{E}(\bm{\beta}_{\mathrm{s}})_ {k}$ (143) $\displaystyle=0$ (144) $$

This completes the proof

## Appendix C ReLU networks

In this section, we describe how to compute projections into (and out of) the RKHS defined by a one hidden layer ReLU network. Consider a network $f(\bm{x})$ and a target function $f_{*}(\bm{x})$.

$$ $\displaystyle f(\bm{x})$ $\displaystyle=\frac{1}{m}\sum_{i=1}^{m}c_{i}\sigma(\bm{w}_{i}^{T}\bm{x})$ (145) $\displaystyle f_{*}(\bm{x})$ $\displaystyle=\frac{1}{m_{*}}\sum_{i=1}^{m_{*}}c^{*}_{i}\sigma(\bm{w}_{i}^{*T} \bm{x})$ (146) $$

The feature space of the model is $\textrm{span}\{\sigma(\bm{w}_{i}^{T}\bm{x})\}_{i\leq m}$ in $L_{2}(p)$. To form projectors into this space and its orthogonal complement, we introduce the Mercer decomposition.
For any positive definite, symmetric kernel $k:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$ we can define features through partial evaluation of the kernel, i.e., $\mathcal{\phi}(\bm{x})=k(\cdot,\bm{x})$.
This kernel also induces a reproducing kernel Hilbert space (RKHS) via the Moore–Aronszajn theorem, which is defined as the set of all functions that are linear combinations of these features,

$$ $\mathcal{H}_{k}=\left\{f\Big{|}f=\sum_{i=1}^{M}\alpha_{i}k(\cdot,\bm{z}_{i}) \text{ for some $M\in\mathbb{N}$, $\alpha_{i}\in\mathbb{R}$, $\bm{z}_{i}\in \mathcal{X}$}\right\}$ (147) $$

The associated norm of a function $f\in\mathcal{H}_{k}$ is given by

$$ $||f||_{k}^{2}=\sum_{ij}^{M}\alpha_{i}k(\bm{z}_{i},\bm{z}_{j})\alpha_{j}$ (148) $$

We can also define the operator $T_{k}:L_{2}(p)\to L_{2}(p)$ with action

$$ $T_{k}f=\int d\bm{x}^{\prime}p(\bm{x}^{\prime})k(\bm{x},\bm{x}^{\prime})f(\bm{x }^{\prime})$ (149) $$

The spectral decomposition of this operator, $\{\lambda^{2}_{l},\psi_{l}\}_{l=1}^{\infty}$ is known as the Mercer decomposition and the eigenfunctions form a basis for $L_{2}(p)$. The eigenfunctions $\psi_{l}(\bm{x})$ satisfy

$$ $T_{k}\psi_{l}=\lambda_{l}\psi_{l}$ (150) $$

where $\lambda_{l}$ is the associated eigenvalue. The eigenfunctions with non-zero eigenvalue form a basis for the RKHS $\mathcal{H}_{k}$. Given a function $f=\sum_{l=1}^{\infty}c_{l}\psi_{l}$ one can show by direct computation that

$$ $||f||_{k}^{2}=\sum_{l=1}^{\infty}c_{l}^{2}/\lambda_{l}^{2}$ (151) $$

which also demonstrates that functions with support on eigenmodes with zero eigenvalue are not in the RKHS. If we can construct the Mercer eigenfunctions we can build orthogonal projection operators into the RKHS and its orthogonal complement. To begin note that for Gaussian data, $p(\bm{x})=\mathcal{N}(0,\bm{I}_{d})$, we can exactly compute the expected overlap between two ReLU functions in terms of their weight vectors (Cho & Saul, ):

$$ $\displaystyle\langle\sigma(\bm{w}_{i}^{T}\bm{x})\sigma(\bm{w}_{j}^{T}\bm{x}) \rangle_{L_{2}}$ $\displaystyle=\int p(\bm{x})\sigma(\bm{w}_{i}^{T}\bm{x})\sigma(\bm{w}_{j}^{T} \bm{x})$ (152) $\displaystyle=\frac{1}{2\pi}\left(\sqrt{1-u_{ij}^{2}}+u(\pi-\arccos{u_{ij}})\right)$ (153) $$

where $u_{ij}=\frac{\bm{w}_{i}^{T}\bm{w}_{j}}{\lVert\bm{w}_{i}\rVert_{2}\lVert\bm{w}_
{j}\rVert_{2}}$ With this in hand, we can define the following matrices:

$$ $\displaystyle\bm{K}_{ij}$ $\displaystyle=\frac{1}{m}\langle\sigma(\bm{w}_{i}^{T}\bm{x})\sigma(\bm{w}_{j}^ {T}\bm{x})\rangle_{L_{2}}$ (154) $\displaystyle\bm{K}^{*}_{ij}$ $\displaystyle=\frac{1}{m_{*}}\langle\sigma(\bm{w}_{i}^{*T}\bm{x})\sigma(\bm{w} _{j}^{*T}\bm{x})\rangle_{L_{2}}$ (155) $\displaystyle\tilde{\bm{K}}_{ij}$ $\displaystyle=\frac{1}{\sqrt{mm_{*}}}\langle\sigma(\bm{w}_{i}^{T}\bm{x})\sigma (\bm{w}_{j}^{*T}\bm{x})\rangle_{L_{2}}$ (156) $$

The Mercer eigenfunctions can be constructed by diagonalizing the matrix $K$. If $\bm{z}_{l}$ is an eigenvector of $K$ with eigenvalue $\lambda_{l}^{2}$, then

$$ $\psi_{l}(\bm{x})=\frac{1}{\sqrt{m\lambda_{l}^{2}}}\sum_{l=1}^{m}(z_{l})_{i} \sigma(\bm{w}_{i}^{T}\bm{x})$ (157) $$

is a Mercer eigenfuction with eigenvalue $\lambda_{l}^{2}$, which can be verified by plugging the expression into the eigenvalue equation (150). Since the feature space is $m$-dimensional, we know that these $m$ eigenfunctions span the RKHS. We can now write down expressions for the projections of $f_{*}(\bm{x})$ into this space and its orthogonal complement

$$ $\displaystyle\lVert P_{\parallel}f_{*}(\bm{x})\rVert_{L_{2}}^{2}$ $\displaystyle=\frac{1}{m_{*}}\bm{c}_{*}^{T}\tilde{\bm{K}}^{T}\bm{K}^{-1}\tilde {\bm{K}}\bm{c}_{*}$ (158) $\displaystyle\lVert P_{\perp}f_{*}(\bm{x})\rVert_{L_{2}}^{2}$ $\displaystyle=\lVert f_{*}\rVert_{L_{2}}^{2}-\lVert P_{\parallel}f_{*}(\bm{x}) \rVert_{L_{2}}^{2}=\frac{1}{m_{*}}\bm{c}_{*}^{T}\bm{K}_{*}\bm{c}_{*}-\frac{1}{ m_{*}}\bm{c}_{*}^{T}\tilde{\bm{K}}^{T}\bm{K}^{-1}\tilde{\bm{K}}\bm{c}_{*}$ (159) $$

## Appendix D Experimental details

### D.1 Deep linear models

For the experiments in deep linear models, we train a two layer linear network with dimension $d=500$. We initialize the weight matrices with random normal weights and scale parameter $\alpha=10^{-5}$. To approximate gradient flow, we use full batch gradient descent with small learning rate $\eta=10^{-3}$. We train each model for $10^{5}$ steps or until the training loss reaches $10^{-6}$. We perform target training for 20 instances of the training data and a grid of dataset sizes and values of $\theta$

### D.2 ReLU networks

For the experiments in shallow ReLU networks, we use the parameters $d=100$, $m=1000$, $m_{*}=100$. We initialize the weight matrices randomly on the sphere and the output weights are initialized at $10^{-7}$. We approximate gradient flow with full batch gradient descent and learning rate $0.01m$ and train for $10^{5}$ iterations or until the loss reaches $10^{-6}$. For training with a finite dataset we use $100$ realizations of the training data, and average over $10$ random initialization seeds.

## Appendix E Additional Figures

Figure: Figure 5: Transferability is not predicted by $\phi$-divergences or integral probability metrics We generate source and target distributions $p_{\mathrm{s}}$, $p_{\mathrm{t}}$ according to the setup in Section 3 and plot the transferability $\mathcal{T}$ (7) as a function of (a) the KL divergence $D_{\mathrm{KL}}(p_{\mathrm{s}}\|p_{\mathrm{t}})$ and (b) the Wasserstein $1$-metric. The KL divergence can be computed exactly in this setting (see Section B.1). $W_{1}$ is computed from finite samples using the algorithm in Sriperumbudur et al. ().
Refer to caption: x5.png

Figure: Figure 6: Linear transferability, $\bm{\sigma=0}$ We pretrain a linear network (9) with $L=2$ and $d=500$ to produce un-noised labels from linear source function $y=\bm{\beta}_{\mathrm{s}}^{T}\bm{x}$ using the population loss (4). We then retrain the final layer weights on a sample of $n=\gamma d$ points $(\bm{x}_{i},y_{i}=\bm{\beta}_{\mathrm{t}}^{T}\bm{x}_{i})$ where $\bm{\beta}_{\mathrm{s}}^{T}\bm{\beta}_{\mathrm{t}}=\cos\theta$ and compare its generalization error to that of a model trained from scratch on the target dataset. (a) Theoretical transferability surface (7) as a function of the number of data points $\gamma=n/d$ and task overlap $\theta$. (b) Top-down view of (a), shaded by sign of transferability. Red indicates negative transferability $\mathcal{T}<0$ and blue indicates positive transferability $\mathcal{T>0}$. Note that transfer is always negative when $\gamma>1$, since the scratch trained model can perfectly learn the target task as there is no label noise. (c) Slices of (a) for constant $\theta$. Solid lines are theory, dots are from numerical experiments. Error bars represent the standard deviation over $20$ draws of the training data.
Refer to caption: x6.png

Figure: Figure 7: Linear transfer $\sigma=0$: theory vs. experiment (a) Identical to Fig. 6(b), but shaded according to the value of the transferability. (b) Results of numerical simulations with $L=2$, $d=500$
Refer to caption: x7.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1- Does the proposed theory shed some light into why some approaches (e.g., contrastive learning with self-supervised learning ) learn better and more transferable representation compared to others? 

Q2- Is it possible to use the proposed theory to improve the transfer learning capabilities of the models? For example, during the representation learning phase with the source data, we can add a regularizer that tries to maximize the the overlap in the feature space using a small amount of the target data. I would like to hear the authors thoughts on this and other ways to leverage this theory in practice.
soundness: 3
strengths: 1- Even thought the structure of the paper is atypical (e.g., short introduction with related work being a subsection), the paper is well-written and easy to follow.

2- Transfer learning is very important topic in DL and understanding when/why it works is crucial for the overall understanding of deep learning.

3- I did not read all the proofs in extreme details. I only checked the skeleton of the proofs and they seem correct and the results are reasonable. However, I am not update with the current advancements in transfer learning theory so i can not judge the novelty of the proofs/theoretical results. Hence so the low confidence in the score.
summary: This paper addresses the gap in understanding how to adapt large pre-trained models to data-limited tasks by examining the theory behind task similarity in transfer learning. It challenges the conventional belief that similarity between source and target data distributions, such as through ϕ-divergences, predicts transfer success. Instead, it suggests a feature-centric approach where transfer learning is effective when the target task aligns well with the feature space of the pre-trained model. The authors demonstrate their theory using deep linear networks, providing insights into when transfer or fine-tuning outperforms training from scratch, especially with limited data.
weaknesses: 1- Lack of the empirical results: the authors only consider a small example of a two layer neural networks. So it is hard to see if the conclusion/insights of their theory  can really translate to deep state-of-the-art models. 

2- The proposed theory does not offer any insights on the design of the representation learning approaches and why some work better than other (e.g., contrastive learning with self-supervised learning) which in my opinion is a very interesting question crucial to understand transfer learning.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: If I have a misconception regarding Theorem 2.2, can you please clarify your explanation? The entire paper revolves around this result.

>> Thank you for your discussion around Theorem 2.2. While reading your rebuttal plus re-reading the paper, I have a few additional concerns that should be discussed.

1. You initially say that x and y are jointly distributed and that the labels are real numbers y in R (second sentence in Section 2). This is fine although a slight abuse of notation since random variables are not numbers. Then you define noisy versions of y and call them ys and yt. They are essentially noisy transformations of x. So far that is fine. Now in Assumption 2.1, you say that x is a data point and that y is a function on x that derives its statistical properties from f(x) + e. This is incorrect, because, if y is a function and not a random variable, then it cannot have statistical properties. Essentially, you are simultaneously saying that y=f(x)+e AND that y is a function of x (ie y(x) =f(x)+e), which doesn't make sense. Additionally, if x is a point, then y is just Gaussian since e is Gaussian. So where did the randomness in x go? Also are you only considering Gaussian data? Actually, it seems to be univariate Gaussian, and so the most basic case possible.

2. Your explanation is helpful but not complete. For example, let's say that we train the source model and get f which is essentially a wavelet transform. Then the basis functions phi's are just some wavelet basis in L^2. Then the subspace is the entire space. Then you are saying that you can find another function using the wavelet basis that gives poor performance. Obviously that is true -- just pick random wavelet coefficients. And if you want larger error you can just scale the coefficient weights.

3. Point 2 seems especially true, since you do not say that f and g are similar. You just say that you can find a g in the same subspace that f is in that is different.

4. If x has a fixed distribution for source and target tasks but fs and ft are different then the joint distributions p(x,ys) and p(x,yt) will be different. So I am not sure how you can claim in your rebuttal that "it is impossible to build two tasks with identical distribution for which transferring is poor". It must be that you are defining these functions, somehow, by learning them. But it is not clear how. Maybe you are assuming that the networks converge to a global minima?
soundness: 2
strengths: The authors are correct that a theory of fine-tuning foundation models is lacking, and so work in this space is welcome. Indeed an explanatory theory for transfer/fine-tuning of nonlinear, deep networks would be a significant advance in the field.

The feature-centric viewpoint appears novel.
summary: This paper aims to provide new theoretical insights into fine-tuning and transfer learning with large-scale foundation models. In particular, they aim to improve the theoretical basis of measuring the similarity between two machine learning tasks - the baseline task for which the foundation model was trainied and the target task for which we are fine-tuning/transfering. The authors argue that one should study this problem from the standpoint of data features. The authors claim to make progress and relate their results to feature learning dynamics.
weaknesses: I found it difficult to assess the contributions of the paper because of unclear explanations. In particular, Theorem 2.2 is crucial because it is claimed to motivate the work (“General Theoretical Setting”), but I cannot follow the argument. What I can best infer from the text is that the authors are saying the following:  If the source and target distributions are the same (or similar) but the features are different, then transferring will fail. This means that, even if the data is the same for both source and target, if a network f is the pretrained one, then we can find a "bad" network g that has worse performance. But this seems obvious, since one can pick a random network g that will with overwhelmingly high probability have worse performance than any trained network. My suspicion from this is that the authors’ results won't actually hold up in practice. But it is hard to tell because the set up is so unclear.

The claimed theory (that I could not exactly follow due to the unclarity above) is proved only for deep linear networks and shallow (e.g, 2 layer) nonlinear networks, which significantly reduces its potential explanatory potential in realistic deep learning settings.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: See weaknesses.
soundness: 2
strengths: 1. The results presented, especially Theorem 3.7, 3.8, and 3.9 are, to my knowledge, novel and relevant to the development of the theory of transfer learning in neural networks. Although in the linear setting, these results are non-trivial to obtain since the authors operate in the rich regime, which corresponds to a non-convex optimization problem from an optimization perspective (all the weight matrices move from initialization in the limit).

2. The paper is overall very well-written. The results are nicely presented, and the storyline is logically flawless. I especially appreciated how results from existing work are adequately presented, such that the novel theorems feel tied into the existing literature. Finally, the results are adequately discussed and put into context (e.g. Theorem 3.6).
summary: The paper theoretically studies transfer learning for deep linear networks in the feature-learning "rich" regime, in the joint scaling of large number of fine-tuning samples $n$ and dimension (including width) $d$. The pertaining is performed on the population loss to simulate the fact that the source task has to be larger than the fine-tuning one.

The authors first argue that the model-agnostic task-similarity metrics (based solely on the source and target distributions) are insufficient to capture how good the transfer is and find counter examples where it can be the case. Then, they propose a new metric based on the discrepancy between the performances of the fine-tuned model and the trained-from-scratch model.

Given this metric, they compute the network performances in the asymptotic limit described above, both in the case of linear fine-tuning and "full" fine-tuning of all the parameters.
weaknesses: The main fundamental weaknesses of this paper in my opinion lie in how certain results are portrayed, which can be quite misleading. More concretely:

1. First of all, I think studying linear networks in the specific context of transfer learning is an inherent limitation that should be more clearly discussed. This is because the function class is linear no matter how many hidden layers. In fact, one could consider a simple linear regression model (for which the asymptotic analysis is known, as the authors correctly cite), and then perform linear evaluation on fresh new weights. This is how I interpret Theorem 3.5 and 3.6, where there is no dependence on the depth.

2. I find it a bit misleading to consider as misleading the dataset-based discrepancy metrics based on the “anomalous positive transfer”. In this case, as the authors state, the positive transferability comes purely from the double descent phenomenon and not from anything inherent positive feature of fine-tuning. In a sense, the benefits of pretraining come purely from the additional datapoints that the model is pre-trained on and that makes you avoid double descent. In this sense, I feel that the message stated in the abstract that model-agnostic similarity metrics are insufficient, while the proposed feature-centric metric is portrayed as the solution, is misleading: the proposed metric has precisely this flaw that does not take into account the double descent phenomenon. To give an example, when the source and target distributions are exactly the same, the metric would still present this anomaly. Thus, comparing with the trained-from-scratch model is not enough to evaluate transferability.

3.  The results in Section 3.8 are also incomplete. How does the measure of transferability $\mathcal{T}$ change? I appreciate the result that in general, the transfer is worse (and this is a nice result per se), but it would be nice to tie it to the previous section, i.e. comparing it to the trained-from-scratch model. I would imagine that the picture in Figure 1 (b) would be different, as regularization makes training from scratch in general a bit better by avoiding double descent. Again, in this case, I would imagine that the dataset-based discrepancy metrics correlate with transferability. More generally, I wonder how much of these conclusions would hold excluding the double descent phenomenon, or if what we are observing is entirely due to it. If this is the case, then advocating for a feature-centric view of transfer learning is misleading.

4. Figure C requires better formatting. 

Minor issues/questions:

1. What is M, line 161?

2. Dudley Metric not introduced

3. Assumption (22) should be explained in the main text, or at least referenced because initialization is not really discussed there.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Can this framework also explain what happens in the case of gradient descent (rather than gradient flow)? As a follow up, what would be the effect of noise in stochastic gradient descent?

While I do understand and agree with Theorem 2.2, it seems to go against prior literature showing that these metrics correlate well with transfer. Would it be possible to empirically test the claim? More concretely, would it be possible to show an experiment where the source and target distributions are “far apart” with respect to the stated metrics, yet the transfer achieved is still positive?

Can these results also be extended to logistic regression in a classification setting?
soundness: 3
strengths: The paper is very well written in general and the result is novel and interesting. The model is able to capture transfer learning both just by fine tuning on the last layer, and over the whole parameters. The authors validate empirically their results on regression tasks.
summary: The paper studies an analytical model for transfer learning in deep linear networks. To begin, the authors prove that general wisdom surrounding feature learning, namely that being close in distribution between the source and the target domains, is not a necessary condition for transfer learning. The authors study deep linear networks trained using gradient flow and establish a measure of transfer, namely the transferability, which is the difference between the generalization error of the model trained from scratch and the one that has undergone transfer learning. The authors study the transferability phase plots as a function of source-target alignment and the data to parameter ratio for linear transfer, meaning just the readout layer is trained on the target task, and fine-tuning, where all the layers are trained on the target. Finally, the authors show empirically that their predictions hold in the case of a 2 layer relu network.
weaknesses: The paper is generally well written and understandable, and there aren’t really any major flaws that I could find. While one may consider the very simplified setting as a weakness, especially since in deep non-linear networks and Transformers the transfer learning behaviour might be different, I believe that such models would currently be too complicated to study. While I do not find any technical flaw with the paper, I do have several questions that I detailed below.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Equations (1) and (2) use the same $\epsilon$. Does this imply that the source and target outputs for a given input $x$ share the same noise component?

2. In Equation (6), could the authors clarify the interpretation of $\Theta_i$?

3. In line 174, what is the meaning of the symbol $\leq_L$?

4. Theorems 3.4 and 3.5 are referenced from Yun et al. What is the motivation for re-proving these theorems, and why are they prominently highlighted in the main text?
soundness: 2
strengths: This paper addresses the essential and meaningful topic of understanding transferability in transfer learning. The problem setup, including assumptions, is clearly and concisely presented.
summary: This paper develops a theoretical framework to analyze transferability in high-dimensional regression tasks within the context of transfer learning. The authors adopt a feature-based approach, proposing that the similarity between source and target tasks is best understood through feature space overlap rather than distributional similarity. The paper establishes phase diagrams that quantify the conditions under which transfer learning outperforms training from scratch, particularly when feature space alignment is high and target data is limited. The analysis primarily focuses on deep linear networks but extends some insights to nonlinear networks.
weaknesses: The main theoretical results depend on assumptions like normally distributed inputs and linear source and target functions, which limit applicability. Nonetheless, those simplifications are reasonable for an initial exploration of this topic.

Meta-review: This paper studies under a new feature-centric viewpoint the similarity of source and target tasks in transfer learning. The implications of such an analysis should be important because they can help understand the intricacies of knowledge transfer from a foundation model to a new situation, through e.g. fine-tuning. 

It is both interesting and novel how this paper advocates for a “feature view” (over “data view”) for transfer learning. Moreover, the area of understanding transferability and related concepts is significant. The paper is generally well written, however Theorem 2.2 has caused some confusion. 

While the theoretical contributions are interesting, the arguments put forward in the paper are not entirely supported by the research conducted. Firstly, the reviewers consider the theoretical analysis carried out on linear models to be limited. They also mention prior theoretical work on transfer learning which actually considers both the linear and non-linear case. Secondly, reviewers point out certain ambiguities in the discussion about data-centric vs feature-centric metrics, and overall remain somewhat confused about the connection between the “storyline” of the paper and the results/claims presented. 

Overall, this paper has benefited from five expert reviews who also engaged in discussions, however none of the reviewers felt that this paper in its current form clearly passes the acceptance threshold for ICLR. However, they did point out several directions for improvement, in particular regarding “tying” the motivation with the storyline and results/claims of the paper, as well as, ideally, extending the analysis on non-linear cases.

Decision: Reject