Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors perform an analysis of fine-tuning a parameter learned from a source task for a target task in the few-shot setting (i.e. $n \ll d$). They provide:


1. A fine-grained analysis of the linear setting, where both source and target tasks involve linear functions of the input $x$, with risk bounds that incorporate the data covariance of the target task.
2. A similar analysis for the overparameterized linear setting, providing risk bounds that are not a function of $\lVert \theta_S - \theta_T\rVert$ in the infinite depth limit. Furthermore, in the same limit, they show that fixing the initial layers provably forces the predictor to remain in the span of $\theta_S$.
3. An analysis of fine-tuning for shallow ReLU networks in the NTK regime.

The theoretical results are corroborated by simulations.

main_review: **Originality:**  
The authors provide a novel analysis of fine-tuning in the deep linear setting, as well as in the NTK setting. The NTK analysis in particular incorporates a new argument allowing for the analysis of gradient descent in this regime from a non-random initialization obtained from pre-training.

**Quality:**  
The submission seems technically sound.  I thought that the experimental evaluation was very thorough and illustrated the predictions of the theory well. It was particularly interesting to see the tighter analysis incorporating the covariance in the linear setting, as well as the accompanying characterization of when fine-tuning helps.

Questions:
1. How are the results in Section 5 impacted by weakening the $0$-balancedness assumption on the weights to a $\delta$-approximate balancedness? According to [1], initializing with small variance only guarantees approximate balance, not $0$-balance. How does $\delta$ enter into the risk bounds, and does approximate balance affect the negative result on freezing earlier layers in Theorem 5.4?
2. Can Theorem 5.2 be experimentally verified more directly? For example, a simple experiment would be to fix $\langle \hat\theta_S, \hat\theta_T\rangle$ and verify that scaling does not affect the risk (and vice versa).

**Clarity:**  
I thought that the results were clearly presented for the most part; however, I did find the following issues:
1. The deep linear setting was not particularly well-described in the main text. What are the dimensions on the matrices $W_1, \dots, W_L$? What are the assumptions made on these dimensions?
2. Line 168-171: I did not quite follow the discussion of the tradeoff for the choice of $m$. Is this assuming that $g(\lambda, t, n)^{3/2} < 1$, and so it is advantageous to push more of the norm of $\theta_T - \theta_S$ into the first term since it decays more quickly?

Minor Typos:
1. Line 211: Missing parenthesis around “Theorem 4.1”
2. $\beta$ is not defined in Equation 5, which I assume is supposed to be $\beta(\infty)$.
3. Line 222: “$\lim_t{t\to\infty}\beta(t)$” should be “$\lim_{t \to \infty}\beta(t)$”.
4. Line 347: Fix theorem reference

**Significance:**  
The analysis in the paper suggests future directions in obtaining tighter bounds in the few-shot setting, as well as better characterizations of when pretraining helps, by explicitly incorporating the data covariance in the analysis.

I thought the NTK-regime analysis of fine-tuning was particularly interesting, especially since prior work (to my knowledge) relies heavily on the random initialization throughout the analysis, and thus it is unclear how one would analyze meta-learning/few-shot learning (where one starts from a fixed initialization) within the NTK regime.

[1] Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2018.

limitations_and_societal_impact: The authors have adequately addressed the limitations of their work. I particularly appreciate the thoroughness with which the authors addressed the shortcomings of the settings they considered in their discussion in Section 7.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Fine-tuning is a common technique in training large networks on relatively small datasets. It achieves excellent generalization results. However, it is not well understood theoretically. This paper attempts to analyze the sample complexity of fine-tuning for regression with linear teachers in several settings. Intutitvely, the success of fine-tuning depends on the similarity between the source tasks and the target tasks. This paper shows that the relevant measure has to do with the relation between the source task, the target task and the covariance structure of the target data. The analysis offers the insights into the inductive bias of gradient-descent optimization and the implied relation between the source task, the target task and the target covariance that is needed for this process to succeed.

This work focuses on the case where both source and target regression tasks are linear functions of the input. It first considers the case where the model architecture is one layer linear networks, and derive sample complexity results for fine-tuning. It shows that under certain assumptions there can be substantial sample complexity reduction when certain measure of source task and target task is low. 

Further, it analyzes deep linear networks. There it derives a novel result providing the learned model as a function of initialization, and uses it to derive corresponding sample complexity bounds. The results provide several surprising insights. The first is that the covariance structure of the target data has a significant effect on the success of fine-tuning. In particular the relation between the vector of the source-target difference and the eigenvectors of the target covariance will affect how well fine-tuning will work. Second, it finds that the depth of the network can dramatically affect the results of the fine-tuning process, since deeper networks will serve to cancel the effect of scale differences between source and target tasks. Further, it corroborates the results by empirical evaluations.

A limitation of the work is the simplicity of the models analysed. Their setting deals only with linear teachers, and assumes the label noise to be zero. Furthermore, it only shows upper bounds on the population loss, and not matching lower bounds. For deep linear networks it assumes a certain initialization which is less standard than normalized initializers like Xavier. For non-linear models, it analyses the simple model of a Shallow-ReLU network, and only in the NTK regime.
main_review: The main strength of the paper is in the theoretical results, the analysis of the fine-tuning for regression with linear teachers in several settings. It gives an insight into the right measure of similarity between the source tasks and the target tasks, for fine-tuning to succeed. Particularly, it shows that the relevant measure has to do with the relation between the source task, the target task, and teh covariance structure of the target data. This is an interesting find. The theoretical results are non-trivial and did require some effort. 

The main weakness is in the fact that the theoretical results are for the restrictive setting of a linear teacher. Further, the theoretical results does not lead to any insight into improving fine-tuning methodology for any setting. It only gives some insights into the method of fine-tuning. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a theoretical analysis of fine-tuning for linear regression problems. The main results in the paper are as follows.

1. Theorem 4.2 develops an upper bound on the population risk on the target task in terms of the number of samples of n, the training time t, the data matrix of the target task and the difference between the true weights of the target task (the teacher) and the pertained weights on the source. There is technical novelty in how this bound is derived (matrix perturbation methods). The conclusion of this theorem is that if the teacher weights and pertained weights are close, as measured by the top few eigenvectors of the target data, then one gets a better population risk by training for fewer iterations.
2. This analysis is repeated for deep linear networks in Theorem 5.1 with similar conclusions.
3. Theorems 5.2-5.3 show an interesting phenomenon where if the student is a deep linear network, then the weights of the student need only be aligned with the teacher (as opposed to being close in ell-2 norm).
4. Theorem 6.1 is an analogue of the above result in the NTK regime. The technical novelty in the proof here is that one has to analyze the NTK dynamics in two stages. 

Some experimental results are provided on synthetic data and MNIST.
main_review: This is a well-written paper on a practically relevant problem, namely that of understanding when fine-tuning is effective for transfer learning. The technical development of the paper is however quite underwhelming because it heavily relies on linear regression setup.

1. I have concerns about the intellectual novelty of this paper. The paper should address (a) how these results can be extended to more practically relevant setups beyond linear models, (b) why they do not follow from existing results on linear/two-layer networks [27-29] or the NTK regime of [39], and (c) how these are results practically relevant. In particular, it is not sufficient to measure the distance between the source and target tasks using the difference/alignment of their weights because it is unlikely to extend to nonlinear teachers. A more useful way to think about this problem seems to be assume a similarity measure on the tasks (and line 348 in the discussion section is not far from this) and study this setting. Distances between even simple tasks can be quite non-trivial, e.g., https://arxiv.org/abs/2002.04747.

2. Theorem 5.4 which claims that freezing the bottom few layers of the student does not work for target tasks. This certainly does not hold for nonlinear networks where freezing weights is a commonly used strategy to reduce the sample complexity of learning the new task.

3. The development in section 6 should come with the following caveat that the authors are encouraged to make prominent. The NTK regime holds when the pertained model does not move very far away from its initialization. Similarly fine-tuning in the NTK regime holds when the fine-tuned model does not travel very far away from the pertained one.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Taking linear regression as a base model, this paper provides some interesting insights about model fine-tuning by theoretical analyses. The main findings are demonstrating more factors such as covariance structure of target data, and network depth, have impact to the process of fine-tuning.
main_review: Originality: Providing theoretical support to fine-tuning is sort of new in machine learning, although there are some pioneer works which are also mentioned in the paper. As I am not knowledgeable enough to check the correctness of theory details, I cannot justify its technical novelty.  

Quality: This paper focuses on theory analysis to fine-tuning and the techniques are complete. Some conclusions are interesting and I only have some minor concerns.
First, I feel confused when I try to read the theorems. For example, it seems theta_T, theta have the different meanings in Section 3 and 4 from line105 and Eq. 1.
Second, it is strange to put Experiments as Section 4.1 with two the main conclusion Theorem 4.2 before it. Is it clearer to make them as two sequential subsections?
Third, it seems that Section 5.2 tries to say freezing the former k layers and updating the remaining ones fails. But any workaround?

Clarity: The submission might be clear but it is not easy to follow the details for me. But overall I feel it is good. 

Significance: I felt the results are important and also interesting. This work tries to answer some challenging questions for fine-tuning, which may inspire the future research.
limitations_and_societal_impact: The authors have stated their main limitation, that is, working on linear regression model.
review_title: null

Meta-review: This paper presents new results for fine-tuning in the linear regime (linear parameterization and linearized NN). While there is still a gap between the linear regime and practice, the reviewers and the AC believe this paper gives an important initial step towards building a comprehensive theory for fine-tuning. 
Decision: Accept (Poster)