Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a transferability measure that takes into account the task similarity as well as the sample size and model complexity. The theoretical formulation is then applied to develop an iterative algorithm to train deep neural networks in a supervised transfer learning setting. It is further shown that the algorithm can be applied to multi-source transfer.

main_review: **Originality**
The proposed mathematical formulation is novel as it takes into account the sample size and model complexity on top of task similarity which is lacking in previous methods. However, the relation between recently introduced transferability measures e.g. LEEP (Nguyen et al. ICML 2020), NCE (Tran et al. ICCV 2019), and their derivatives are neither discussed nor cited. The results are compared to domain adaptation methods making it unclear whether this work is more suitable as a domain adaptation paper or transferability.  

**Quality**
I appreciate the author's effort in deriving a mathematical formulation of transferability that takes into account sample size and model complexity. However, the experiments performed in the paper do not seem sufficient due to the following reasons:

1. The method introduces a transferability measure. Therefore ideally it should be compared to methods such as LEEP and more recent state-of-the-art transferability measures. In case, the proposed method is not suitable to be compared with these methods, this should be clearly explained in the paper.
2. In the current results, the datasets used to evaluate the performance are more suitable for domain adaptation. A more recent standard benchmark VTAB (Zhai et al. arxiv 2019) might be more suitable or the ones used in LEEP, NCE papers making the results of the current method comparable to state-of-the-art.
3. Even in the domain adaptation benchmark the methods compared are not current state-of-the-art e.g. in Office Caltech dataset Wang et al. AAAI 20 is current state of the art. IT is not necessary to beat state-of-the-art methods but at least these methods should be mentioned and the relation between these methods to the current method should be discussed.

**Clarity**
The mathematical formulation is described well and augmented with the algorithm. Although some parts of the description may be improved e.g. in line 109 it is not clear why that term accounts for model parameters.  

The related literature section lacks citations to multiple relevant papers as mentioned above making it difficult to assess the contribution of the current paper when put in context with recent papers on transferability measures. 

**Significance**
Theoretical estimates of transferability are crucial in order to assess the application of off-the-shelf models in real-world scenarios. In this aspect, the paper does a good job in obtaining an estimate that takes into account the sample size of source and target tasks which is a crucial factor.

However, the experiments performed in the current version do not provide sufficient evidence of the applicability of the proposed measure. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a mathematical analysis aiming at quantifying the transferability in Transfer Learning.  More specifically, they consider a transferability measure and they show that such a quantity is characterized  by the number of samples, the complexity or dimensionality of the machine learning model and the similarity (according to the \Chi-squared distance) between source and target tasks (see Thm. 2 and Thm. 3). For given source tasks, the authors learn the target task by a class of learning model which linearly combines the models learned from individual tasks by some designable coefficients. Then, they use the optimal combining coefficients achieving the minimum testing loss on the target task as the transferability measure. By doing this, they manage to illustrate the contribution of each model in learning the target task, and effectively quantify the knowledge transferable among different tasks. They start from discrete data setting and then, they extend and adapt the analysis to the continuous data setting by introducing a related transferability measure that can be computed by solving a non-negative quadratic programming problem. Exploiting this, the authors develop an alternating iterative algorithm for training deep neural networks in standard supervised transfer learning settings. The numerical experiments on real image classifications tasks show the effectiveness of the proposed method in comparison to many existing state-of-the-art algorithms in multi-source and few-shot settings.
main_review: The authors propose an interesting and novel analysis on a topic that could provide new insights on the design of new Transfer Learning algorithms. In particular, they manage to show the natural finding according to which the transferability of a particular source task is typically proportional to the number of samples and the measure of similarity to the target task and it is inversely proportional to the model complexity.

The paper is quite well written.

In lines 28-33, the authors say that “In theoretical analyses [4, 5, 6], sample sizes and model complexity are often included in deriving upper bounds for the transferability or the performance of transfer learning algorithms. However, it is pointed out that such bounds derived under general learning settings are often relatively loose [7], and hence the algorithms designed by directly applying theoretical results can hardly achieve satisfactory performance in practical applications.” Explain to the reader why such bounds are loose. This point is important for understanding the contribution of the paper.

Explain more the intuition of Definition 1. Which is the relation between the similarity measured the authors consider in such a definition and those usually used in standard multi-task and meta-learning framework, such as the variance among the true risk minimizers of the tasks or the complexity of the shared common representation? Is it possible to extend the analysis to more general types of similarity measures, maybe non-heterogeneous measures such as those considered in [1,2,3] below?

Which is the difference between multi-source transfer learning and online meta-learning (see e.g. [4,5,6,7] below) in which we interpret the source tasks as training tasks? Discuss this point and associated literature.

Add a sketch of the proof of the main statements could help the reader in understanding the main tools used during the proofs.

1.   A structured prediction approach for conditional meta-learning, Wang et al. 2020
2.   Conditional Meta-Learning of Linear Representations, Denevi et al. 2021
3.   The Advantage of Conditional Meta-Learning for Biased Regularization and Fine Tuning, Denevi et al. 2020.
4.   Provable Guarantees for Gradient-Based Meta-Learning, Khodak et al. 2019
5.   Online-Within-Online Meta-Learning, Denevi et al. 2019
6.   Learning-to-Learn Stochastic Gradient Descent with Biased Regularization, Denevi et al. 2019.
7.   Adaptive gradient-based meta-learning methods, Khodak et al. 2019
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors propose a mathematical framework for quantifying the transferability in multi-source transfer learning problem. Specifically, the target model is considered to be a linear combination of the source model, and the combination coefficients is used to denote the transferability. Analyses further show that the transferability is characterized by the sample size, model complexity, and source-target similarity. A parameterized model is then proposed to achieve the quantifiable transferability measure. Experiments on image classification task are done to verify the effectiveness of the proposed algorithms. 
main_review: Here are the comments:

Pros:

(1)	A mathematical framework that establishes a transferability measure quantified by sample size, model complexity, and source-target X^2-distance. 

(2)	A parameterized model based on the theoretical framework, and an iterative algorithm for training deep neural network.

(3)	The experiments show the effectiveness of the proposed algorithm in some extent. 

Cons:

(1)	The analyses are limited to a stacking-based target model and the proposed X^2-distance, which constrains its generality. The problem setting studied requires the target labelled data. It is more interesting to have discussions on the popular unsupervised domain adaptation setting where the target labels are not available. Moreover, [ref1] studies the similar problem setting (multi-source setting with limited target labelled data) and also proposes a stacking-based target model with the base model coefficients as the domain similarity. It is necessary to discuss with this work. 

[ref1] Source-target similarity modelings for multi-source transfer gaussian process regression. ICML. 2017.

(2)	The experiments are done on the classification tasks. Is the analysis applicable to the transfer regression problem? 

(3)	Regarding Eq.(12), is it only for the target task (as the P_Y^{(0)}(y) is used)? What is the dimensionality of \mathbf{g}_i? How to obtain P_{XY}^{(0)} and P_{X}^{(0)} in Eq. (15)?

(4)	The improvements of the proposed method compared with the other baselines are not significant as shown in Table 3 and Table 4. Note that some baselines are unsupervised domain adaptation method, which cannot leverage the target labels. 

(5)	The empirical evaluation will be more convincing if promising results are obtained on the more challenging dataset, e.g., office-home dataset.

limitations_and_societal_impact: An explicity limitation section is encrouged.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims to quantify transferability in multi-task in multi-source transfer learning in terms of model complexity, sample size, and similarity of the tasks. The analysis is done for the discrete case and later generalized for a specific model with a fixed embedding structure and a particular readout layer that resembles the factorization machines. The results are validated on a number of transfer learning datasets.
main_review: Developing a measure of transferability is an important problem for understanding the dynamics of transfer learning. However, the claims made in the paper are not consistent with the analysis. The paper claims quantifying a notion of transferability in terms of the model complexity and sample size. In statistics and ML, model complexity is generally shown in terms of standard quantities such as Rademacher complexity or VC dimension. Also, sample complexity generally imposes a lower-bound on the number of examples that a certain algorithm requires to learn a particular task. Instead, the provided analysis only shows the optimal loss in terms of the discretized probability densities of the tasks. This quantity is not related to the complexity of the model that is used for transfer learning nor the number of training samples. Therefore, the results do not provide any insights on transferability other than how much the densities of the two tasks are similar.

On the experimental side, it is hard to tell whether the small improvements are significant, given that no error margins are provided. 

In summary, the measure of transferability that is developed in the paper is not related to model complexity or sample size. It simply quantifies how close the discretized distributions of two given tasks are to each other. On the experimental side, it is unclear whether this quantity provides any significant improvements. 

Minor: the analysis for the binary case is redundant and included in the multi-task setup. This part can be removed without hurting the construction.

=========================================================

After rebuttal:
The authors have addressed most of my questions/concerns regarding the technical details of the paper. The construction in the paper is interesting and non-trivial, especially for the continuous case. Therefore, I would like to raise my score to 7 and vote for acceptance.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: In this work, a novel mathematical formulation is proposed for the problem of multi-source transfer learning. The theoretical analysis is interesting. Empirical evaluation is good but can be more convincing by conducting experiments on more challenging datasets.

In the rebuttal, most of the concerns raised by reviewers have been well addressed. Though one reviewer sticks to his originally negative rating of 5, I found that the authors have given clear explanations in their responses. Therefore, I recommend an acceptance for this work.

However, the authors are encouraged to make the descriptions clearer in the revision based on reviewers' comments as well as the authors' responses in the rebuttal.
Decision: Accept (Poster)