Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a theoretical analysis of network-based transfer learning using layer variational analysis. Their analysis is claimed to prove that the success of transfer learning is guaranteed with certain data conditions. Based on their analysis, they also propose an alternative method for network-based transfer learning, which demonstrates better efficiency and accuracy for domain adaptation.
strength_and_weaknesses: **Strength:**

1. Most technical analysis is correct and their empirical study is also sufficient.
2. The proposed approach which directly solves the linear functional on top of the pre-trained feature extraction show promising results on downstream tasks

**Weaknesses:**

1. Their analysis does not capture many important aspects of network-based transfer learning. Firstly, they only discuss the training loss but do not consider the generalization ability of models on test data. But achieving good training loss does not necessarily mean the learning is successful. Secondly, SGD optimization is normally used to train neural networks, their analysis does not consider the complexity of optimization. And their variational analysis in function space does not capture the optimization procedure in parameter space.
2. While the proposed method based on their analysis is interesting in the transfer learning setting, the idea is not entirely new. Similar ideas already exist in the meta-learning literature. ([Meta-Learning with Differentiable Closed-Form Solvers](https://arxiv.org/abs/1805.08136), [Meta-Learning with Differentiable Convex Optimization](https://openaccess.thecvf.com/content_CVPR_2019/papers/Lee_Meta-Learning_With_Differentiable_Convex_Optimization_CVPR_2019_paper.pdf), [Meta-Learning Priors for Efficient Online Bayesian Regression](https://arxiv.org/abs/1807.08912)). I am not sure if it is the first time the idea is applied to transfer learning. Moreover, although I can see the connection between the proposed method and their theoretical analysis, I don't think the theoretical part is absolutely necessary to come to their proposed approach. They could simply talk about their method without the layer variational analysis.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**

The paper is mostly well-written with minor issues:

1. In definition 3.2, the authors use g for the upper layers. However, in equation 3.2, they changed to f for both pre-trained and upper layers. I don't understand why this change is necessary.
2. The format of citations sometimes has issues. For example in paragraph 2 in the introduction, these citations should be in parathesis (using \citep)
3. The use of mathematical notation is sometimes inappropriate. For example, $q_i \cong 0$ on page 5. The notation $\cong$ normally means "is congruent to". I am not sure why it is used here.

**Quality:**

I think there are probably no mathematical mistakes in their theoretical analysis and their empirical study is also sufficient. However, I am not fully convinced by the following:
1. The definition of data deviation. According to their definition, if there is an extreme outlier in the new dataset, the data deviation can be very large even though other data points are exactly the same as those in the pre-training dataset. Because of that, it does not seem to be a good measure of data distance to me.
2. At the beginning of section 3.2, it is stated that neural networks are Lipschitz continuous. Do you mean all neural networks are Lipschitz continuous? It does not seem to be right to me.
3. Remark 3.7, only from equation 10, how can you tell that $\mathcal{L}(g)$ will become large if $\epsilon_{\text{data}}$ is large? Equation 10 is inequality so $\mathcal{L}(g)$ can be very small even if the bound on the right is large.

In addition to the above, I am also concerned by the high-level issues mentioned in the weaknesses section.

**Novelty:**

It may be the first time a similar analysis and the proposed method are used for transfer learning. However, the idea behind the proposed method is not entirely new (see the weaknesses section above), and their theoretical analysis does not capture the important aspects of transfer learning.

**Reproducibility:**

The code is provided and the experimental details are sufficient.
summary_of_the_review: I recommend rejection of this paper because of the following:
Their theoretical analysis does not capture several important aspects of transfer learning (see Weaknesses section) and their analysis has a few potential technical issues (see Quality section). The proposed method based on their analysis may not be seen as entirely new to me because similar ideas do exist in meta-learning literature (see Weaknesses section).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a theory framework to enable better finetuning by only adding a Lipschitz function.
strength_and_weaknesses: pros:

1. It is interesting to see we can only add a Lipschitz function $\delta_f$ to finetune on different datasets.


Cons:

1. The framework requires sample alignment first. To me, the sample alignment can be a very huge cost and may not be applicable in most finetuning cases. For example,  given a pre-trained ImageNet classifier and a new dataset such as Office31, we have to align samples first. It can be challenging to measure the difference and thus align successfully. If we can align them, we may just use the aligned image and labels to train the model. So we don't need to finetune.

2. The experiment results are not enough and I would like to see quantitative comparisons.  In particular, since the paper title suggests the domain adaptation, I would like to see how can we benefit from the proposed framework on popular domain adaptation datasets, e.g., DomainNet, Office31.
clarity,_quality,_novelty_and_reproducibility: The proposed framework allows us to finetune the model by only adding a residual but the residual can be different to compute and may not even be applicable for many cases.
summary_of_the_review: The proposed framework allows us to finetune the model by only adding a residual but the residual can be different to compute and may not even be applicable for many cases.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new aspect to look at transfer learning. The basic idea is to allow each layer $f_n$ to change a little bit ($\delta f_n$). This gives bounds on losses on the new dataset ($L(g)$) under a few assumptions. Based on this framework, $L(g)$ can be re-written as a least-square problem, and $\delta f_n$ can be optimized for the new dataset. The paper conducts experiments on a synthetic dataset, a speech enhancement task, and image de-blurring/super-resolution. Results show the proposed method outperforms GD. 
strength_and_weaknesses: **Strengths**

- The paper has a good overview of the literature.
- The method of layer-wise optimization (from the $\delta f_n$ point of view) is novel. Eq (17)-(18) gives new insights to transfer learning.
- Experiments are conducted on multiple domains with consistent improvement over the GD baseline.

**Weaknesses**

- The paper does not explicitly define norms such as $\\| \cdot \\|_{\\mathcal{Y}}$.
- The data deviation assumption might be strong in practice.
- Thm 3.6 is not clear and does not provide much insight. Do you mean $\exists \delta_f$ or $\forall \delta f$? Does it need to satisfy the constraint that $f+\delta f$ is a network layer? Also, simply by setting $\delta f==0$ should yield eq (10); in that case, why do you have this theorem. 
- Eq (16): it seems to be $\approx$ instead of $=$ because $f_n$ is an affine transformation followed by activation. This might also introduce further error terms in the following equations, so I would like to see analysis on that. E.g., a higher-order analysis.
- Around eq (21): it should be $\min$ eq (17) instead of (19). That $\delta f_n$ is linear seems infeasible due to non-linear activation, so I think mentioning the pseudo-inverse is unnecessary.
- Experiments on speech enhancement: 
  - Do you use mismatched speakers of clean speech between training and adaption sets? 
  - Why is the fmax=6k given that training data are 16kHz? 
  - It would be interesting to test on the no-reverb test set as it provides different noise types and part of them can be used as the adaptation set. 
  - The baseline is sort of weak. More recent denoisers can achieve test pesq > 2.8 or even 3.0 (see e.g. FullSubNet, Demucs, CleanUNet, ...). I think it's feasible to take the pretrained models (all of these models have) and conduct adaptation to new noise types and languages in DNS-2021/2022. If it comes out with good results then the paper can have more impact. 
- Experiments on image super-resolution/de-blurring: the major concern is that there is no quantitative results, so it is hard to evaluate the methods fairly. 
clarity,_quality,_novelty_and_reproducibility: Clarity: most part of the paper is clear except for those mentioned in questions above.

Quality: the overall quality is good, with some concerns on the theory and experiments as discussed above.

Novelty: the proposed method is novel to my knowledge.

Reproducibility: code is available.
summary_of_the_review: Based on the questions and concerns above, I think the paper has some problems in theory and space for improvement in experiments. Therefore I think this version cannot be accepted. I will make the final decision based on authors' response and revised version. 

----------

After rebuttal:

The authors improved the experiments and I found them good, so I'll increase my score to 6. I think the higher-order analysis can be further improved, especially for more standard neural networks with Lipschitz activations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The authors aim to tackle a very challenging issue, namely  establishing both formal derivations and heuristic analysis to formulate the theory of transfer learning in deep learning. To that end, the authors put forth a framework based on  layer variational analysis and show that  that the success of transfer learning can be guaranteed with corresponding data conditions. The paper conducts experiments on a synthetic dataset, a speech enhancement task, and image de-blurring/super-resolution.

The paper is nicely organized and well-written, the code is provided and the experimental details are sufficient. The major novel aspect is the method of layer-wise optimization, and  Eqs. (17)-(18) give new insights to transfer learning. In the first review round, the reviewers however rises several  concerns, specifically about some technical aspects of the proposed solution, and the experimental validation. The authors did a great job in responding to the reviewers’ comments, and most of the negative comments have been properly addressed and solved. One of the reviewers has also increased his/her initial score. A few comments are still pending, i.e., the authors have not provided a determined  response to the following issues: (i) Generalization ability of the model on the test data. Nonetheless, the authors have given a generalization error bound, which may partially address the reviewer's concern. (ii) The framework requires sample alignment, which can have a  huge cost and may not be applicable in most fine-tuning cases. Nonetheless, the authors have provided a reasonable was to (practically) address the problem, and  (iii) A few of the assumptions made by the authors may not always capture the essence of the network-based transfer learning

 In sum, the paper is nicely written with some elements of novelty, and it also  provides a good deal of experimental results and analyses to back-up the authors’ claim. A few concerns remain, but the paper could be considered a first step to establish a theoretical analysis of network-based transfer learning.
justification_for_why_not_higher_score: The are few issues with the proposed technique that have been  addressed only partially.
justification_for_why_not_lower_score: The paper discuss a challenging problem, and the proposed idea is interesting and sound. The paper is well written. The technical novelty is sufficient.
Decision: Accept: poster