A Theoretical Analysis of Fine-tuning with Linear Teachers
Authors: Gal Shachaf
Alon Brutzkus
Amir Globerson
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning theory
learning theory
transfer learning
fine-tuning

Abstract: Fine-tuning is a common practice in deep learning, achieving excellent generalization results on downstream tasks using relatively little training data. Although widely used in practice, it is not well understood theoretically. Here we analyze the sample complexity of this scheme for regression with linear teachers in several settings. Intuitively, the success of fine-tuning depends on the similarity between the source tasks and the target task. But what is the right way of measuring this similarity? We show that the relevant measure has to do with the relation between the source task, the target task and the covariance structure of the target data. In the setting of linear regression, we show that under realistic settings there can be substantial sample complexity reduction when the above measure is low. For deep linear regression, we propose a novel result regarding the inductive bias of gradient-based training when the network is initialized with pretrained weights. Using this result we show that the similarity measure for this setting is also affected by the depth of the network. We conclude with results on shallow ReLU models, and analyze the dependence of sample complexity there on source and target tasks. We empirically demonstrate our results for both synthetic and realistic data.

A Theoretical Analysis of Fine-tuning with Linear Teachers
Gal Shachaf
Alon Brutzkus
Blavatnik School of Computer Science, Tel Aviv University, Israel
Blavatnik School of Computer Science, Tel Aviv University, Israel
Amir Globerson
Blavatnik School of Computer Science, Tel Aviv University, Israel and Google Research
Abstract
Fine-tuning is a common practice in deep learning, achieving excellent general- ization results on downstream tasks using relatively little training data. Although widely used in practice, it is lacking strong theoretical understanding. Here we analyze the sample complexity of this scheme for regression with linear teachers in several architectures. Intuitively, the success of fine-tuning depends on the similarity between the source tasks and the target task, however measuring this similarity is non trivial. We show that generalization is related to measure that considers the relation between the source task, target task and covariance structure of the target data. In the setting of linear regression, show that under realistic settings a substantial sample complexity reduction is plausible when the above present novel result regarding
measure is low. For deep linear regression, the inductive bias of gradient- baaded training when the network is initialized with pretrained weights. Using this result we show that the similarity measure for this setting is also affected by the depth of the network. We further present results shallow ReLU models, and analyze the dependence of sample complexity on source and target tasks in this setting.
Introduction
In recent years fine-tuning has emerged as an effective approach learning tasks with relatively little labeled data. In this setting, model is first trained on a source task where much data is available (e.g., masked language modeling for BERT), and then is further tuned using gradient descent methods Furthermore, it has been observed that fine-tuning can outperform the strategy of fixing the representation learned on the source task, mainly in natural language processing [1. Despite its empirical success, fine-tuning is poorly understood from theoretical perspective. One apparent conundrum is that fine-tuned models can be much larger than the number of target training points, resulting in heavily overparameterized model that is prone to overfitting and poor generalization. Thus, the answer must lie in the fact that fine-tuning performed with gradient descent and not an arbitrary algorithm that could potentially "ignore" the source task 6). Here we set out to formalize this problem and understand the factors that determine whether fine-tuning will succeed. We note that this question can be viewed part of the general quest to understand the implicit bias gradient based methods context of fine-tuning.
on labeled data of target task [ 3
but the particular
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We begin by highlighting the obvious link between fine-tuning and initialization. Namely, the only difference between "standard" training of target task and fine-tuning on it, is the initial value of the model weights before beginning the gradient updates. Our goal is to understand the interplay between the model parameters at initialization (namely the source task), the target distribution, and the accuracy of the fine-tuned model. A natural hypothesis is that the distance between the pretrained and fine-tuned model weights is what governs the success of fine-tuning. Indeed, some argue that this is both the key to bound the generalization error of model and the implicit regularization of gradient- methods 14, [3] 16. [7]. However, this approach has been discouraged both by empirical testing of the generalization bounds inspired by it [18] and by theoretical works showing this cannot be the inductive bias in deep neural networks [9]. Our results further establish the hypothesis that the success of fine-tuning is affected by other factors.
In this paper we focus on the case which both source and target regression tasks are linear functions of the input. We start by considering one layer linear networks, and derive novel sample complexity results for fine-tuning. We then proceed to the more complex case of deep linear networks, and prove novel result characterizing the fine-tuned model as a function of both the weights after pretraining and the depth of the network, and use to derive corresponding generalization results.
Our results provide several surprising insights. First, we show that the covariance structure of the target data has a significant effect on the success of fine-tuning. In particular, sample complexity is affected by the degree of alignment between the source- sarrett weight difference and the eigenvectors of the target covariance. Second, we find a strong connection between the depth of the network and the results of the fine-tuning process, since deeper networks will serve to cancel the effect of scale differences between source and target tasks. Our results are corroborated by empirical evaluations.
We conclude with results on ReLU networks, providing the first sample complexity result for fine- tuning. For the case of linear teachers, this asserts simple connection between the source and target models and the test error of fine-tuning.
Taken together, results demonstrate that fine-tume fine-tuning is affected not only by some notion of distance between the source and target tasks, but also by the target covariance and the architecture of the model. These results can potentially lead to improved accuracy in this setting via appropriate design of the tasks used for pretraining and the choice of the model architecture
2 Related work
Empirical work (20) has shown that two instances of models initialized from pre-trained weights are more similar in features space than those initialized randomly. Other works [21 22 23 have shown that fine-tuned models generalize well when the representation used by the target task is similar to the one used by the source tasks.
In linear regression, 24 showed that gradient descent finds the solution with minimal distance the initial weights. More recently, attention has turned towards the phenomenon of "benign overfitting" (25 26) in high dimensional linear regression, where despite fitting noise in training data, population risk may be low. Theoretical analysis of this setting (25) studied how it is affected by the data covariance structure. Benign overfitting was also recently analyzed the context of ridge-regression 27 and online stochastic gradient descent (28). Our work continues this line of work on high dimensional regression, but differs from the above papers as we start from a source task, then train on fixed training set from target task and consider the global optimum of the this training loss (unlike online SGD). Furthermore, we go beyond the linear regression framework, and obtain surprising characteristics of fine-tuning in deep linear networks.
For linear regression with deep linear models, layer network with deterministic initialization, and 30 have shown an implicit bias for network with arbitrary depth and near-zero random initialization. Our work generalizes the inductive bias found by 29 to a network of arbitrary depth, and analyses the generalization error of such networks for infinite depth. For linear regression with shallow linear networks (1 have shown a generalization bound that depends only the norm of the target task, which we use in Section 6
29 have recently shown an implicit bias for two-

---

3 Preliminaries and settings
Notations Let || be the 2 norm for vectors and the spectral norm for matrices. For a vector v we denote ú _ 12 |a For a matrix M E IRd and some 0 S m S d, we define Mgm E Rdxm to be the matrix containing the first m columns of M. Similarly, we let M denote the matrix containing the columns from m +
in M.
Let DD be a distribution over R2d Let Σ be the covariance matrix of DD and let VAV be its eigenvalue
decomposition such that À 2 ... 2 Aa- We define the projection matrices:
P
P V >k>
projecting onto the span of the top & eigenvectors of E, onto the span of the d- 1 bottom eigenvectors of Σ, respectively. We will refer to the former as the "top-k span" of Σ, and to the latter as the "bottom-k span" of E.
Let X Raxd be the row matrix of < d samples drawn from D, and denote the empirical covariance matrix ! by Σ. Deñne Pi be the projection matrix into the row space of X, and P_ to be the projection matrix into its orthogonal complement, i.e.:
P P
XX
X
P - P
Consider set of parameters O and let (t) denote the set of parameters at time t. We denote the output of model whose weights are (t) on a vector x by f (x; (()) E R. In the different sections of this work we will overload f with different architectures.
We consider the problem of fine-tum lne-tuning based transfer learning in regression tasks with linear teachers. Let 0r E R be the ground-trute truth parameters of the target task, i.e. the linear teacher which we wish learn, and y E R" be the target labels of X, s.t. y XØr.
We define (()) to be the empirical MSE loss on X, and define R(O) as the DD population loss:
L ( A 1
()( - y12 R(O) A ExOD [xx 0r S (x, @1)
We separate the training procedure into two parts. In the first "pretraining" part, we train model on ns pretraining samples Xş Rnsxd xd labeled by linear teacher 0s (i.e., ys X E RTs), resulting in the set of model weights O In the second part, which we call fine-tuning, we initialize a model with the pretrained weights ((0) = O₂ and learn the target task by optimizing L(O
Optimization is done by either gradient descent (GD) or gradient flow (GF). Let (() be some weight vector or weight matrix in (t). The dynamics for gradient descent optimization with some learning rate > are ((t+1) = O(t) - a and the dynamics for gradient flow are ((t) 00(()
(
(0(C)
Next we state several assumptions about our setup.
Assumption 3.1. XXT is non-singular. i.e. the rows of X are finearl-indeepender
This assumption holds with high probability for, e.g., a continuous distribution with support over a non-zero measure set. This assumption is only used for simplicity, the high probability can be incorporated into the analysis.
Assumption 3.2 (Perfect pretraining). The pretraining optimization process learns the linear teacher perfectly, egg. for linear regression assume that (x. Os) Os.for
D.
Notice that for linear and deep linear models, perfect pretraining be achieved when ns 2 d. Our results can be easily extended to the case where the equality f (x, θ = x 0g holds approximately and with high probability, but for simplicity assume equality.
Assumption 3.3 (Zero train loss). The fine-tuning converges, i.e. lim, L (e())) 0.
We note that when f is standard linear regression, arbitrarily small train loss can be obtained via gradient descent. For deep linear networks, can be shown (32] that under suitable initialization a global optimum can be reached, and thus Assumption 3.3 holds for this framework as well.

---

4 Analyzing fine-tuning in linear regression
In this section we analyze fine-tuning for the case of linear teachers for linear regression when using gradient descent for optimization. We define (t) w(t) E IR and overload / [xx, +))) _ x (().
In what follows we denote the parameter learned in the fine-tuning process by 7 _ lime-+oo (t).
4.1 Results
The following known results (e.g.. [4 25 10 show the inductive bias of gradient descent with non-zero initialization in under-determined linear regression and the corresponding population loss.
70J When fx, )) is a linear function, fine-tuning with GD under Assump- tion 3.7 Assumption 3.2 and Assumption 3.3 results in the following model:
Theorem 4.1.
224 25
= p1 Os + PIOTT
and
R(7) = > p (Or 0s) 0s)
(2)
Theorem 4.1
provides two interesting observations: the first is that ry consists of two parts, one which is the projection of the initial weights 0g into the null space of X, and the other which the projection of 0r into the span of X. The second observation is that the population risk depends solely on the difference o 0s that is projected to the null space of the data. For completeness, the proof of Theorem 4. is given in the supplementary.
Theorem 4.1 depends on the data matrix (via Pli: P1). However, to better understand the properties of fine-tuning, a high probability bound on R that does not depend on X is desirable. We provide such a bound, highlighting the dependence the population risk on the source and target tasks, and the target covariance E.
Theorem 4.2. Assume the conditions of Theorem 4.1 hold, and assume that the rows of X are i.i.d. subgaussian centered random vectors. Then, there exists a constant > 0, such that, for all 2 1, and for all 1 S. m S. d such that Am > 0, with probability at least 1 - e over X. the population risk R(y) is bounded by:
P Sm 00, -ssll
2g(A, 6, n)s
+ +g 8,
P> (0, 0s)l
(3)
X2
where g(A, 8, n) = cc max
E nA1
E.
14.4 } and Σ Σ S g(A d,
nA:
In the proof, we address the randomness of P (0, 0g) in by decomposing Or 0g into its top-k span and bottom-k span components, and then applying the Devis-Kahan sin theorem (33 to bound the norm of the projection of the former to the null space of the data, The full proof is given in the supp.
The bound in Theorem 4.2 has two key components. The first is the function g(X, 8, n) that captures how well the covariance Σ is estimated, and shows the dependence of the bound on the number of train samples used (as it depends on n -0.5). The second relates to the two matrix norms of 0, 0s with respect to different parts of the covariance Σ. Notice that the term relating to the top-k span decreases like n -1.3, while the term relating bottom-k span decreases like n. -0.5
This theorem highlights the conditions under which fine-tuning is expected to perform well. For small enough s.t. g(A, 5, n)) > 1, the bound mainly depends on || (Or 0s) I- In this case, the bound will be low if 0r and 0g are close in the span of the top eigenvectors of the target distribution. On the other hand, for large enough s.t. g\,, ó, n) < 1, the bound mainly depends on |P>m (0, 0g) I|- Thus, the bound will be low if 0x and 0g are close in the span of the bottom eigenvectors of the target distribution.
We conclude with a remark regarding the integer m appearing in the bound, in the case where g(A, n) < 1. While finding the exact m that minimizes the bound is not straightforward, the trf in selecting it suggests taking the largest m which holds Am- An- This will "cover"
more of P mm (Or 0s) without greatly increasing the left part of 33)

---

Table 1: Correlation coefficient R² between the accuracy on different transfer tasks MNIST and various population risk upper bounds. Each value is mean over 10 calculations of R with different initialization, and each R2 is calculated from 20 points, each one representing a mean accuracy value of 25 random samples.
Number of Samples
10
15
20
25
30
| - 0sll²
0.69 = 0.03
0.68 + 0.04
0.66 = 0,04
0.64
0.03
0,62 + 0.02
Bound from [25 Ours for m = 2
0.73 = 0.03
0.75 0.03
0.74 - 0.03
0.71 + 0.02
0.67 = 0.02
0.86 = 0.02
0.89 = 0.02
0.84 = 0.02
0.75 - 0.01
0.69 + 0.02
4.2 Experiments
In Figure we empirically verify the conclusions from the bound in 3 We set d = 1000 and design the target covariance E s.t. the first m 50 eigenvalues are significantly larger than the rest (1.5 vs. 0.3). We then consider two settings for Or Og. In the first, which call "Top Eigen Align", select Or and 0s such that P sm (r 0s) = O. In the second which we call "Bottom Eigen Align" we set P>m (0T 0s) O. In both settings we use the same norm | Os 121 to show that the bound is not affected by this norm.
As discussed above, our bound suggests better generalization performance of "Bottom Eigen Align" for large n and better performance of "Top Eigen Align" for small n. Indeed, we see that while for very few samples "Top Eigen Align" has a lower population loss than "Bottom Eigen Align", the population loss of "Bottom Eigen Align" drops significantly as n grows, and drops to zero well before = d.
We next evaluate the bound on fine-tuning tasks taken from the MNIST dataset [34], and compare it to alternative bounds. Specifically, since we do not expect bounds to be numerically accurate, we calculate the correlation between the actual risk in the experiment and the risk predicted by the bounds. The task we consider (both source and target) is binary classification, which we model as regression to outputs {-1, +1}. {1]. We generate K source-target task pairs (e.g., source task is label 2 vs label 3 and target tasks is label 5 vs label 6). For each such pair we perform source training followed by fine-tuning to target. We then record both the 0-1 error on an independent test set and the value predicted by the bounds. This way we obtain K pairs of points (i.e., actual error vs bound), and calculate the R² for these pairs, indicating the level to which the bound agrees with the actual error. In addition to our bound in (3), we consider the following: the norm of source-target difference |0r 2 and bound adapted from [25] to the case of fine-tuning ! The results in Table | show that there is a strong correlation between our bound and the actual error, and the correlation weaker for the other bounds.
5 Analyzing fine-tuning in deep linear networks
In this section we focus on the setting overparameterized deep linear networks. Although the resulting function is linear in its inputs, like in the previous section, we shall see that the effect of fine-tuning is markedly different. Previous works (e.g. [35] (66) have shown that lincar networks exhibit many interesting properties which make them good study case towards more complex non-linear networks.
We consider networks with L layers, given by the following matrices: (t) = ..., (t)
s.t. (1) E (dd-1idd do = d. dr and for 1 < < L 1 1: dj 2 d. We also define:
((t)= Wi(!) W2( ... W ((),
such that / (x: (()) (1) = x B(). From Assumption 3.2 we have that ((0) = 0s-
We recall the condition of perfect balancedness (or O-balancedness) (3)
Definition 5.1. The weights of a depth L deep linear network at time L called O-balanced if:
W W (t) = W Wt)+ for j E [ 15-1] -1.
4)
'The adaptation is straightforward: since the population loss for non-random initialization depends on 0r Os instead of Or, can replace the ground-trui expression 0* in Theorem 4 from 25 with Or 0s.
5

---

Top Eigen Align
Bottom Eigen Align
0.8
Loss
0.6
0.0
200
400
600
800
1000
# Samples
Figure 1: Comparison between different Qr
Os. "Top Eigen Align" is the linear predictor
initialized with Pm (0r 0g) = 0 and "Bottom Eigen Align" is the linear predictor initialized with Pam (r 0g) = 0, for m=50. The top eigenvalues have the value 1.5, compared the rest which have the value 0.3.
Our analysis requires the initial random initialization (prior to pretraining) to be O-balanced, which be achieved with near zero random initialization, as discussed in [32]. We provide three results on the effect of fine-tuning in this setting. The first result shows the inductive bias of fine- tinettuning depth L deep linear network (Theorem (.2h, which holds for arbitrary L and generalizes known results for L 1 (Theorem 4.1) and L = [9]. The second result analyzes the population risk of such predictor when L. for certain settings (Theorem 5.3 and Theorem (.4). The third result shows why fixing the first layer (or any set of layers containing the first layer) after pretraining can harm fine-tuning tne-tuning (Theorem (55)..
The next theorem characterizes the model learned by fine-tuning the above setting (it can thus be viewed as the deep-linear version of the L = 1 result in Theorem 4.1
Theorem 5.2. Assume that before pretraining, the weights of the model were O-balanced and that Assumption 3.7 Assumption 3.2 and Assumption 3.3 hold. Then:
4.2
|limm B(OII)
lim B(t) t1500
PiOs Pr
(5)
Asl
and:
= [P
lim lim 3(t)
PLOs + Pr
(6)
1-00 t10
0s
To prove this, we focus on W1, and notice that the gradients W are in the span of X, and hence P W (0) and its norm remain static during the GF optimization ((30)). We then analyze the norm of the fine-tuned model by using the O-balancedness property of the weights and the min-norm solution to the equivalent linear regression problem, and achieve (5). (6) is achieved by calculating the limit w.r.t. L. The proof of Theorem 5.2 is given in the supplementary.
Although the expression in (5) is not a closed form expression for lim B(t) (because [lim, [1()] appears on the RHS), taking to infinity (6) does result in closed form ex- pression and demonstrates the effect of increasing model depth. As in (I). we see that the end-to-end equivalent has two components: one which parallel to the data and one which is orthogonal to However, while in (I) the orthogonal component has the original norm of the orthogonal projection of 0g, the expression
(6) offers a re-scaling of the norm of this component by some ratio that also depends on 0r. Presenting this phenomenon for the infinity depth limit might look impractical, but the empirical results given in this section show that the effect of depth is apparent even for models of relatively small depth.

---

5.1 When Does Depth Help Fine-Tuning"
In this subsection we wish understand the effect of depth on the population risk the fine-tuned model. For simplicity we focus on the limit in (6. and denote 3 = lim lim ((t).
Since the linear network is linear function of of the network, similar to (2)
derive expression for the population risk
P
P 0rl
2
R(3) - P 0r
However, since P1 depends on the random matrix X, without further assumptions this expression by itself is not enough to understand the behaviour of R(3). Theorem 5.3 and Theorem 5.4 analyze
cases for which a bound on 7 can be achieved, showing that it depends on ||O||l (0, 0s ). i.e. the
product of the norm of 0r and the difference of the normalized 0r and 0s. compared to (2) which depends on the difference between the un-normalized vectors. This observation further highlights the fact that the distance between source and target vectors not good predictor of fine-tuning accuracy for some architectures, as fine-tuning can still succeed even if the source and target are very far as long as they are aligned.
We formalize this in the following result, where 0 is identical to g in direction, but not in norm.
Theorem 5.3. Assume that the conditions of Theorem 5.2 hold, and that θ 0s. Namely:
0 = a0g, for a > 0, then for + 3o the risk of the end-to-end solution / is R(33 0,
while for the L 1 solution y, the risk is:
a E P 1rrl1rr112 #0 #0
R(y)
for a + 1, > 0.
(8)
This setting highlights our conclusion on the role of alignment in deep linear models: if the tasks are aligned. the deep linear predictor achieves zero generalization even with single sample, while the population risk of the L = 1 predictor still depends on
Another example for this behaviour can be seen when X is i..d Gaussian (i.e., D =NN00, 1)4).
Theorem 5.4. Assume that the conditions of Theorem 5.2 hold, and let X ~ NN, Suppose n S then there exists constant > 0 such that for any > with probability least 1 - exxp(-ce2n) - cxx (-c22 [n)) the population risk for the L - oo end-to-end predictor
bounded us follows:
R(B)
d - (1
10 | +)²
0s||²=
05|
d -
d ((100=11)
d
for (([(0]) ss || For the linear regression solution y this risk is bounded by R(>) d- --= (1
- +)2² 0- 0ssl
(10)
The above result a direct analysis of 7 when
by using Lemma 5.3.2 from
analyze
the effects of P ,PP Comparing (9) and (TO), we see that while (T0) depends on the distance between the two un-normalized tasks, (9) depends on the norm of the target task and the alignment of
the tasks, but not at all on the norm of the source task. The proofs of Theorem 5.3 and Theorem 5.4 are given in the supp.
5.2 Deep linear fine-tuning with fixing the first layer(s)
common trick when performing fine-tuning is to fix, or "freeze" (i.e. not train), the first & layers of model during the optimization the target task. This method reduces the risk of over-fitting these layers to the small training set 2 The next theorem shows that for deep linear networks this method degenerates the training process.
This over- fitting is sometimes referred to as "catastrophic forgetting" of the source task.
7

---

a alpha alpha
source target
The key idea in the proof is to show that the product of the k first layers is equal 0, up scaling factor, which is result of (30). The result implies that after fine-tuning the model is still equal to the source task, independently of the target task. Thus, fine-tuning essentially fails completely, and its Test Lass
Test LOoS
of Layers
Scaling Factor
(b)
Figure 2: (a) The effect of depth on fine-tuning when 07 is a scaled, noised version of 0g with 1/10 samples. (b) The effect of changing the scale of either source weights or target weights 7-layers model.
Theorem 5.5. Assume the setting of Theorem 5.2 of first layers) during fine-tuning, the fine-tuned model will be given by (3(1), x) = c(x, Os) for some constant
Then, if we freeze the first layer (or any number
error cannot be reduced with additional target data.
1.2
Frozen
1.0
Finetune
Vanilla
0.8
0.6
Test
0.4
0.2
0.0
200
400
500
800
1000
# of Samples
Figure 3: network whose first layer fixed has constant generalization loss due to degeneration
effect in Theorem 5.5
This result is achieved under the assumption of O-balancedness prior to pretraining, which happens e.g. when initializing the weights with an infinitesimally small variance, as this property leads to degeneracy the output of the frozen k-layers. Though the proof of Theorem depends on this 0-balancedness property of the network, the experiments shown in Figure 3 were conducted with a small initialization scale, that is not guaranteed to result O-balancedness, but rather in 6-approximate balancedness [32] when 8 is small. These experiments show empirically that the phenomenon of learning failure is observed even when d > 0. Intuitively, this is because the effective rank of the weight matrices is close to one, and thus learning the second layer is an ill-conditioned problem, which leads to slower convergence and can prevent the model from fine-tuning on the target data with constant gradient step.
possible workaround to this failure of learning would be to initialize the weights prior to pretraining with larger scale of initialization (e.g. with Xavier (38) ) thus increasing the rank of each layer and
preventing degeneracy. Pre-training with multiple source tasks (as suggested in e.g. 22 may also help the fine-tuning optimization.

---

5.3 Experiments
We next describe experiments that support the results in this section. Theorem 5.3 nets will successfully learn a case where source and target vectors are aligned, but with different norms. This demonstrated in Figure[Zawhere where source and target tasks are related via Or = a0s + 6, where e is standard Gaussian vector whose norm is approximately 0.5 ||0 It can be seen that when a R3 1, there no difference between models of different depth. However, as a increases, adding depth has positive effect on fine-tuning accuracy. Theorem 5.4 predicts that the test loss for a deep linear model would depend only on the alignment of 0g and Or (i.e. |0r Qs and on the | but not on initialized s.t. 00 0g 0.1. In each experiment, either 0r a0f or 0s adg. where a is the "'Scaling Factor", and the other has norm of I. It can be seen that increasing the norm of the target vector harms generalization much more than increasing the norm of the source vector, as the theorem predicts, even for relatively shallow model.
predicts that deeper
1slll This is demonstrated in Figure 2b where source and target task are
Theorem 5.5 states that fixing the first layer in deep linear nets can result in failure to fine-tune. We illustrate this empirically in Figure 3] where we compare three two-layer linear models on the same target task: 1) A "Frozen" model that fixes the first layer after pretraining. 2) A "Vanilla" model that trains the network from scratch on the target, ignoring the source pre-training, 3) A "Finetune" model that first trains on source and fine-tunes to target. As predicted by theory, the "frozen" model's performance is poor, and fine-tuning has better sample complexity.
6 Analyzing fine-tuning in shallow ReLU networks
Analyzing optimization and generalization in non con-linear networks is challenging. However, analysis in the Neural Tangent Kemnl (NTK) regime is sometimes simpler [9] Thus, here we take a first step towards understanding fine-tuning in non-line -on-linear networks by analyzing this problem in the NTK regime. Specifically, we consider the setting of a two-layer ReLU network with m neurons in the hidden layer. Hence, we consider
(() = (t),, a and f (x; (()) = - Em a (w w() where a is the ReLU function, w| (4). Wm (1) E Rd, the rows of W(), are vectors in the first layer, and a E 1)m is the vector of weights in the second layer. We initialize uniformly and fix it during optimization as in (39]. Before pretraining, the first layer parameters are initialized from a standard Gaussian with variance x², We also assume that ||x for all samples from D. We let fX θ) E Rn be the vector of predictions of f on the data X.
For the next theorem we do not assume linear teachers, and instead assume an arbitrary labeling function Is such that ys = gx(Xs), for Xg E R xd yy E R the pretraining data and labels, respectively. We also assume that y gr gT(()) for some arbitrary function yr. For simplicity, we assume lyl: < for i E [n). We consider a setting where the pretraining phase is done using a two-layer network in the NTK regime, under the assumptions of Theorem 4.1 from (31] with respect to the variables m, K. r) and sufficiently many iterations 3 Next, in the fine-tuning phase, we train a network initialized with the weights given by the pretraining phase. We use the same value of for the fine-tuning phase. We rely on the analysis given in [39 31 and achieve an upper bound on the population risk of the fine-tuned model:
Theorem 6.1. Fix a failure probability 8 E (0, 1). We assume that Assumption 3./ holds. Suppose
n O Aaáe ). m 2 x= poly (n, (n,ns, 1, .0-1). Consider any loss function 8: R x R - [0, 1 that
1-Lipschitz in the first argument such that f(y. y) 0. Then with probability least 8 the 1 log 11/1 iterations has nAo
two-layer neural network f(: (()) fine-tuned by GI for 2 S
population loss:
ÿ (H>)) y
log And
((O())) S 21
0
(1))
71
for ý = y ((x
3 'See the supp for a bound on the number of iterations.
*Over the random initialization of the pretraining network.

---

The above result shows that the true risk of the fine-tuned model is related to the distance of learned outputs y from the outputs after pretraining f (X, (0)). The proof of Theorem 6.1 is given in the supp.
As in previous NTK regime analyses, this result holds when the weights of the fine-tuned model do not "move" too far away from the weights at random initialization. Thus, the proof approach is to bound the distance between the Gram matrix Ht) and the infinite-width gram matrix H with decreasing function in m. The main challenge is that the weights (()) are not initialized i.i.d as described above. To address this we provide a careful analysis of the dynamies and show that H(t) close to H at random initialization, even when considering the pretraining phase, which in turn is close to H
We next apply our results to the case of linear source and target tasks. We thus assume that gs. 9T linear functions with parameters 0s, Or. For simplicity of exposition we assume (x, (0))
x Og exactly (Assumption 3.2 Before bounding the risk of fine-tuning we bound the RHS of
in the linear case:
Corollary 6.2. Suppose that gs X) & x Os, 9r (X) X Or,
000,, and assume Assumption 3.2 holds.
Then, v (H*) 3 0r 0sll2.
This is a direct corollary of Theorem 6.1 from 31 on ý defined above. Theorem 6.1 and Corollary 6.2 result in the bound on the risk of the fine-tuner model:
Corollary 6.3. Under the conditions of Theorem 6. and Corollary 6.2 it holds that
R(O(t)) 5 6 01 0sll2 to
(1 log
72
We note that fine-tuning is improved as the distance between source and target decreases. In our analysis of linear networks (Theorem 4.2 and Theorem (544) obtained more fine-grained result depending on the covariance structure. We conjecture that the non-linear case will have similar results, which will likely involve the covariance structure in the NTK feature space.
7 Discussion
This paper gives a fine-grained analysis of the process of fine-tuning with linear teachers in several different architectures. It offers insights into the inductive bias of gradient-descent and the implied relation between the source task, the target task and the target covariance that is needed for this process to succeed. We believe our conclusions pave a way towards understanding why some pretrained models work better than others and what biases transferred from those models during fine-tuning.
limitation of our work is the simplicity of the models analyzed, and would certainly be interesting to extend these. Our setting deals only with linear teachers, and assumes the label noise to be zero. Furthermore, we only show upper bounds on the population risk, and not matching lower bounds. For deep linear networks we assume certain initialization which is less standard than normalized initializers such as Xavier. For non- non-linear models, we analyze the simple model of shallow ReLU network, and only in the NTK regime.
An interesting direction explore is formulating a bound similar Theorem 4.2 for regression in the RKHS space given by the NTK, where the covariance is now over the RKHS space and thus more challenging to analyze. Another interesting setting is classification with exponential losses. Since the classifier learned by GD in this case has diverging norm, it is not clear how fine-tuning is beneficial, although in practice it often is. We leave these questions for future work.
Acknowledgments and Disclosure of Funding
This work has been supported by the Israeli Science Foundation research grant 1186/18 8 and the Yandex Initiative for Machine Learning. AB is supported by the Google Doctoral Fellowship in Machine Learning.
10

---

References
[1] Jacob Devlin, Ming-Woiteea Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 11866, 2019.
[2] Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, 2019.
[3] Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 946 958, 2020.
[4] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, 2020.
[5] Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune not tune? adapting pre- trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (Rept.4NI.-2019)) pages 7-14, 2019.
[6] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[7] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. International Conference on Learning Representations, 2019.
[8] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635-3673, PMLR, 2020.
[9] Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory, pages 1305-1338. PMLR, 2020.
[10] Jingfeng Wu, Difan Zou, Vladimir Braverman, and Quanquan Gu. Direction matters: On the implicit bias of stochastic gradient descent with moderate learning rate. In International Conference on Learning Representations, 2021.
[11] Edward Moroshko, Blake E Woodworth, Suriya Gunasekar, Jason D Lee, Nati Srebro, and Daniel Soudry. Implicit bias in deep linear classification: Initialization scale vs training accuracy. Advances in Neural Information Processing Systems, 33, 2020.
[12] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[13] Roei Sarussi, Alon Brutzkus, and Amir Globerson. Towards understanding learning in neural networks with linear teachers. In International Conference Machine Learning, 2021.
[14] Vaishnavh Nagarajan and Zico Kolter. Generalization in deep networks: The role of distance from initialization. arXiv preprint NXXX:11900001622 2019.
[15] Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping provably robust to label noise for overparameterized neural networks. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2020.
[16] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization generalization of neural networks. In International Conference on Learning Representations (ICLR), 2019.
11

---

[17] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, arXiv preprini arXiv: 1703. 1008, 2017.
[18] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.02788 2019.
[19] Noam Razín and Nadav Cohen, Implicit regularization in deep learning may not explainable by norms. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[20] Behnam Neyshabur, Hanic Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008. 1667, 2020.
[21] Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning arXiv preprint arXiv:2105.022 02221,, 2021.
[22] Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021.
[23] Daniel McNamara and Maria-Florina Balean. Risk bounds for transferring representations with and without fine-tuning. In International Conference on Machine Learming, pages 2373-2381. PMLR, 2017.
[24] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias terms of optimization geometry. In International Conference on Machine Learning, pages 1 1841.1841. PMLR, 2018.
[25] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063-30077 2020.
[26] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560 2019.
[27] Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009.14285 2020.
[28] Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Benign overfitting of constant-stepsizz sgd for linear regression. arXiv preprint arXiv:2103.12699 2021.
[29] Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. arXiv preprint arXiv:2 102.09969, 2021.
[30] Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations, 2020.
[31] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
[32] Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2018.
[33] Chandler Davis and William Morton Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1-46, 1970.
[34] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
12

---

[35] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterizatton. International Conference on Machine Learning, pages 244-24 253. PMLR, 2018.
[36] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In 7th International Conference on Learning Representations. ICLR 2019, 2019.
[37] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[38] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249- 256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
[39] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2018.
[40]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[41] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Vir- tanen, David Coumapeau,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Hal- dane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre KEraaddMarrhhan Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362, September 2020.
[42] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero. Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
[43] John D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering. 933):90-55, 2007.
[44] Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli,
2017.
[45] jlewk projection. MathOverflow. URL.Ittpss/maathoorrtvvvwee///// (version 2021-05-25).
Difference between identity and a random
[46] Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009. 2020.
13


Reviews:
[{"review_id": "b6Ob0MuG6oT", "reviewer": "Reviewer_J96B", "summary": "Taking linear regression as a base model, this paper provides some interesting insights about model fine-tuning by theoretical analyses. The main findings are demonstrating more factors such as covariance structure of target data, and network depth, have impact to the process of fine-tuning.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Providing theoretical support to fine-tuning is sort of new in machine learning, although there are some pioneer works which are also mentioned in the paper. As I am not knowledgeable enough to check the correctness of theory details, I cannot justify its technical novelty.  \n\nQuality: This paper focuses on theory analysis to fine-tuning and the techniques are complete. Some conclusions are interesting and I only have some minor concerns.\nFirst, I feel confused when I try to read the theorems. For example, it seems theta_T, theta have the different meanings in Section 3 and 4 from line105 and Eq. 1.\nSecond, it is strange to put Experiments as Section 4.1 with two the main conclusion Theorem 4.2 before it. Is it clearer to make them as two sequential subsections?\nThird, it seems that Section 5.2 tries to say freezing the former k layers and updating the remaining ones fails. But any workaround?\n\nClarity: The submission might be clear but it is not easy to follow the details for me. But overall I feel it is good. \n\nSignificance: I felt the results are important and also interesting. This work tries to answer some challenging questions for fine-tuning, which may inspire the future research.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have stated their main limitation, that is, working on linear regression model. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "YXHeTP1yLq6", "reviewer": "Reviewer_e71p", "summary": "This paper presents a theoretical analysis of fine-tuning for linear regression problems. The main results in the paper are as follows.\n\n1. Theorem 4.2 develops an upper bound on the population risk on the target task in terms of the number of samples of n, the training time t, the data matrix of the target task and the difference between the true weights of the target task (the teacher) and the pertained weights on the source. There is technical novelty in how this bound is derived (matrix perturbation methods). The conclusion of this theorem is that if the teacher weights and pertained weights are close, as measured by the top few eigenvectors of the target data, then one gets a better population risk by training for fewer iterations.\n2. This analysis is repeated for deep linear networks in Theorem 5.1 with similar conclusions.\n3. Theorems 5.2-5.3 show an interesting phenomenon where if the student is a deep linear network, then the weights of the student need only be aligned with the teacher (as opposed to being close in ell-2 norm).\n4. Theorem 6.1 is an analogue of the above result in the NTK regime. The technical novelty in the proof here is that one has to analyze the NTK dynamics in two stages. \n\nSome experimental results are provided on synthetic data and MNIST.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper on a practically relevant problem, namely that of understanding when fine-tuning is effective for transfer learning. The technical development of the paper is however quite underwhelming because it heavily relies on linear regression setup.\n\n1. I have concerns about the intellectual novelty of this paper. The paper should address (a) how these results can be extended to more practically relevant setups beyond linear models, (b) why they do not follow from existing results on linear/two-layer networks [27-29] or the NTK regime of [39], and (c) how these are results practically relevant. In particular, it is not sufficient to measure the distance between the source and target tasks using the difference/alignment of their weights because it is unlikely to extend to nonlinear teachers. A more useful way to think about this problem seems to be assume a similarity measure on the tasks (and line 348 in the discussion section is not far from this) and study this setting. Distances between even simple tasks can be quite non-trivial, e.g., https://arxiv.org/abs/2002.04747.\n\n2. Theorem 5.4 which claims that freezing the bottom few layers of the student does not work for target tasks. This certainly does not hold for nonlinear networks where freezing weights is a commonly used strategy to reduce the sample complexity of learning the new task.\n\n3. The development in section 6 should come with the following caveat that the authors are encouraged to make prominent. The NTK regime holds when the pertained model does not move very far away from its initialization. Similarly fine-tuning in the NTK regime holds when the fine-tuned model does not travel very far away from the pertained one.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "RfPpNata2QI", "reviewer": "Reviewer_ynu3", "summary": "The authors perform an analysis of fine-tuning a parameter learned from a source task for a target task in the few-shot setting (i.e. $n \\ll d$). They provide:\n\n\n1. A fine-grained analysis of the linear setting, where both source and target tasks involve linear functions of the input $x$, with risk bounds that incorporate the data covariance of the target task.\n2. A similar analysis for the overparameterized linear setting, providing risk bounds that are not a function of $\\lVert \\theta_S - \\theta_T\\rVert$ in the infinite depth limit. Furthermore, in the same limit, they show that fixing the initial layers provably forces the predictor to remain in the span of $\\theta_S$.\n3. An analysis of fine-tuning for shallow ReLU networks in the NTK regime.\n\nThe theoretical results are corroborated by simulations.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**  \nThe authors provide a novel analysis of fine-tuning in the deep linear setting, as well as in the NTK setting. The NTK analysis in particular incorporates a new argument allowing for the analysis of gradient descent in this regime from a non-random initialization obtained from pre-training.\n\n**Quality:**  \nThe submission seems technically sound.  I thought that the experimental evaluation was very thorough and illustrated the predictions of the theory well. It was particularly interesting to see the tighter analysis incorporating the covariance in the linear setting, as well as the accompanying characterization of when fine-tuning helps.\n\nQuestions:\n1. How are the results in Section 5 impacted by weakening the $0$-balancedness assumption on the weights to a $\\delta$-approximate balancedness? According to [1], initializing with small variance only guarantees approximate balance, not $0$-balance. How does $\\delta$ enter into the risk bounds, and does approximate balance affect the negative result on freezing earlier layers in Theorem 5.4?\n2. Can Theorem 5.2 be experimentally verified more directly? For example, a simple experiment would be to fix $\\langle \\hat\\theta_S, \\hat\\theta_T\\rangle$ and verify that scaling does not affect the risk (and vice versa).\n\n**Clarity:**  \nI thought that the results were clearly presented for the most part; however, I did find the following issues:\n1. The deep linear setting was not particularly well-described in the main text. What are the dimensions on the matrices $W_1, \\dots, W_L$? What are the assumptions made on these dimensions?\n2. Line 168-171: I did not quite follow the discussion of the tradeoff for the choice of $m$. Is this assuming that $g(\\lambda, t, n)^{3/2} < 1$, and so it is advantageous to push more of the norm of $\\theta_T - \\theta_S$ into the first term since it decays more quickly?\n\nMinor Typos:\n1. Line 211: Missing parenthesis around “Theorem 4.1”\n2. $\\beta$ is not defined in Equation 5, which I assume is supposed to be $\\beta(\\infty)$.\n3. Line 222: “$\\lim_t{t\\to\\infty}\\beta(t)$” should be “$\\lim_{t \\to \\infty}\\beta(t)$”.\n4. Line 347: Fix theorem reference\n\n**Significance:**  \nThe analysis in the paper suggests future directions in obtaining tighter bounds in the few-shot setting, as well as better characterizations of when pretraining helps, by explicitly incorporating the data covariance in the analysis.\n\nI thought the NTK-regime analysis of fine-tuning was particularly interesting, especially since prior work (to my knowledge) relies heavily on the random initialization throughout the analysis, and thus it is unclear how one would analyze meta-learning/few-shot learning (where one starts from a fixed initialization) within the NTK regime.\n\n[1] Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of their work. I particularly appreciate the thoroughness with which the authors addressed the shortcomings of the settings they considered in their discussion in Section 7.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5W1fploEVRS", "reviewer": "Reviewer_PQB2", "summary": "Fine-tuning is a common technique in training large networks on relatively small datasets. It achieves excellent generalization results. However, it is not well understood theoretically. This paper attempts to analyze the sample complexity of fine-tuning for regression with linear teachers in several settings. Intutitvely, the success of fine-tuning depends on the similarity between the source tasks and the target tasks. This paper shows that the relevant measure has to do with the relation between the source task, the target task and the covariance structure of the target data. The analysis offers the insights into the inductive bias of gradient-descent optimization and the implied relation between the source task, the target task and the target covariance that is needed for this process to succeed.\n\nThis work focuses on the case where both source and target regression tasks are linear functions of the input. It first considers the case where the model architecture is one layer linear networks, and derive sample complexity results for fine-tuning. It shows that under certain assumptions there can be substantial sample complexity reduction when certain measure of source task and target task is low. \n\nFurther, it analyzes deep linear networks. There it derives a novel result providing the learned model as a function of initialization, and uses it to derive corresponding sample complexity bounds. The results provide several surprising insights. The first is that the covariance structure of the target data has a significant effect on the success of fine-tuning. In particular the relation between the vector of the source-target difference and the eigenvectors of the target covariance will affect how well fine-tuning will work. Second, it finds that the depth of the network can dramatically affect the results of the fine-tuning process, since deeper networks will serve to cancel the effect of scale differences between source and target tasks. Further, it corroborates the results by empirical evaluations.\n\nA limitation of the work is the simplicity of the models analysed. Their setting deals only with linear teachers, and assumes the label noise to be zero. Furthermore, it only shows upper bounds on the population loss, and not matching lower bounds. For deep linear networks it assumes a certain initialization which is less standard than normalized initializers like Xavier. For non-linear models, it analyses the simple model of a Shallow-ReLU network, and only in the NTK regime.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of the paper is in the theoretical results, the analysis of the fine-tuning for regression with linear teachers in several settings. It gives an insight into the right measure of similarity between the source tasks and the target tasks, for fine-tuning to succeed. Particularly, it shows that the relevant measure has to do with the relation between the source task, the target task, and teh covariance structure of the target data. This is an interesting find. The theoretical results are non-trivial and did require some effort. \n\nThe main weakness is in the fact that the theoretical results are for the restrictive setting of a linear teacher. Further, the theoretical results does not lead to any insight into improving fine-tuning methodology for any setting. It only gives some insights into the method of fine-tuning. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents new results for fine-tuning in the linear regime (linear parameterization and linearized NN). While there is still a gap between the linear regime and practice, the reviewers and the AC believe this paper gives an important initial step towards building a comprehensive theory for fine-tuning. 
Decision: Accept (Poster)