A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning

Abstract: Current transfer learning algorithm designs mainly focus on the similarities between source and target tasks, while the impacts of the sample sizes of these tasks are often not sufficiently addressed. This paper proposes a mathematical framework for quantifying the transferability in multi-source transfer learning problems, with both the task similarities and the sample complexity of learning models taken into account. In particular, we consider the setup where the models learned from different tasks are linearly combined for learning the target task, and use the optimal combining coefficients to measure the transferability. Then, we demonstrate the analytical expression of this transferability measure, characterized by the sample sizes, model complexity, and the similarities between source and target tasks, which provides fundamental insights of the knowledge transferring mechanism and the guidance for algorithm designs. Furthermore, we apply our analyses for practical learning tasks, and establish a quantifiable transferability measure by exploiting a parameterized model. In addition, we develop an alternating iterative algorithm to implement our theoretical results for training deep neural networks in multi-source transfer learning tasks. Finally, experiments on image classification tasks show that our approach outperforms existing transfer learning algorithms in multi-source and few-shot scenarios.


Reviews and author responses:

 We thank the reviewer so much for the comments.

Regarding the model complexity, we would like to clarify that the quantity $V^{(0)}$ is contained in the mathematical expression of the testing loss without forethoughts, where we interpret this quantity as the model complexity. We find that it corresponds to the number of parameters in the model and also corresponds to the intuition that, in practical problems, the task will be harder when more parameters need learning. Meanwhile, the least problem might be that the defined model complexity is different from other complexity measures, e.g., the VC dim. Both our defined model complexity and VC dim can reflect the complexity of the task under the corresponding mathematical frameworks.

The paper proposed a new objective function for transfer learning that accounts for the similarity between tasks and model complexity. An iterative algorithm is also proposed to solve this objective. The proposed method is validated on three real-world datasets empirically, comparing against various alternative methods for multi-source and few-shot transfer learning.  I think the paper aims to address an important machine learning problem in the field of transfer learning. The paper is clearly written, and the authors make an effort to factor in various important aspects of transfer learning in their solutions including sample complexity and task similarity. However, the proposed solution is relatively straightforward without rigorous, principled justifications. Experimental results are limited on three datasets.

Originality: the originality of the paper lies in the proposal of a new objective function for transfer learning that accounts for both task similarity and sample complexity. I find such a contribution somewhat limited.

Quality: while the proposed objective function and algorithm intuitively makes sense, the proposal is not validated in a rigorous, principled manner, ideally with certain theoretical justification. I find the claim that the objective provides “fundamental insights” somewhat unsubstantiated. Furthermore, the iterative algorithm makes sense because each sub-step is concerned with solving a different subproblem. It is unclear to me whether a simple first-order method to solve the parameters jointly will suffice, making the proposed algorithm unnecessary. Experimental results are also somewhat limited both in terms of datasets and in terms of competing methods. It is unclear to me why the datasets and methods compared are representative enough to be compared against.

Clarity: I think the presentation in this paper is clear. Mathematical notations are well setted-up. I can follow the major argument of the paper easily.

Significance: based on my concern in originality and quality, I find the paper is of limited significance.

======== After rebuttal ========

Thank you for the detailed response. My questions are partially answered by the authors, and I would like to keep my original rating. There is a lack of discussion of the limitations of the proposed method.

The paper proposes a framework for evaluating transferability, i.e., to quantify how much knowledge one can pass from a source task to a target task. Task transferability is studied under the assumption that the target task’s model is a convex combination of the source tasks’ models. And the notion of transferability is defined as the optimal convex combination coefficients, quantifying the contributions of each source task to the target tasks. Theoretical results relate the transferability to three quantities: (a) the similarity between the source and target tasks, (b) the sample sizes of the source and target tasks, and (c) the models’ complexity.   Originality & Significance: The idea of studying task transferability under the assumption that the target task’s model is a convex combination of the source tasks’ models is new to me. The linearity assumption allows for a very simple and nice solution: the optimal coefficients can be computed in closed-form (in finite domain cases). The simplicity of the solution allows for the great applicability of the methods. However, I find the paper does not provide enough information to convince the reader that this is a good transferability measure? I will elaborate in the below sections.

Technical quality:
- While I am convinced that the technical theorems are correct, I am unsure why this would be a good framework to measure transferability across tasks. 
- Firstly, the paper is structured in a definition-theorem-proof style and does not provide enough explanation or motivation behind each design choice. More concretely, equations 2, 3, and definition 1 expound the basic structure of the framework, yet no explanation is given to motivate these choices. For example, one could provide citations to convince readers that this is common practice or argue that certain assumptions are weak and generally hold in practice.
- Secondly, the paper does not mention any properties enjoyed by the proposed transferability notion (Eq. 4). 
- Thirdly, the experiments are designed to demonstrate the performance of Algorithm 1 and not the validity of the proposed notion of transferability (Eq. 4). The most glaring evidence is that, although the optimal coefficients \alpha^\ast is introduced as the transferability measure in the previous section, \alpha^\ast were never mentioned or discussed in the experiments, and the paper solely focuses on the test accuracy of the model.
- The introduction of the transferability \alpha^\ast seems unnecessary. I feel that the focus of the paper is a new algorithm for transfer learning (Algorithm 1), and not a mathematical framework for measuring transferability across tasks. I would appreciate the contribution here if the author can demonstrate that the proposed notion of transferability is useful. For example, one could construct a synthetic experiment in which one of the source tasks is more intimately related to the target task than the others, and demonstrate that the coefficient of that task is large.

Clarity: The main paper is clear and well-written. I can follow and understand the ideas most of the time. Notations are consistent and correct. 

======== After rebuttal ========

Thank you for the detailed response. For my questions, I am happy with the response by the authors, and I am willing to increase my score from 5 to 6. See the main review above.

 Thanks to the authors for the detailed response. 

I understand that $V^{(0)}$ is a measure of the problem size, but I am not sure I totally agree with the argument that the dimensionality of the parameter space could be called "learning capacity". For example, we can have a model with only one tunable parameter yet the VC dim is infinity, e.g., the set of sinusoids. It still seems to me that this interpretation is a bit far-fetched. 

Thank you for clarifying the solution of the optimal $\alpha$ in the multi-source setting. This makes sense to me. 

Overall, I'd like to keep my original rating. 

 We thank the reviewer for the valuable comments. First, regarding the concerns on the transferability quantification, our responses are as follows.

a) We appreciate the reviewer's consideration of the generality of the convex combination model, which we choose for the following two reasons. Firstly, the convex combination form is a classical approach combining different models, widely adopted in related theoretic analyses, e.g, *Ben-David et al. A theory of learning from different domains*, which also corresponds to the transfer learning practice that uses a mixture of datasets. Secondly, as illustrated in the manuscript, the convex combination model provides much mathematical convenience throughout our analyses. As a consequence, while there are other forms of model combinations in recent literature, e.g. the form based on Wasserstein metric, our analyses on this basic form can provide theoretical guidance and insights in understanding some fundamental properties of transfer learning, which provides us the possibility to understand much more complex ML/DL settings in practice.

b) Though the convex combination form coincides with the result of minimizing a weighted KL divergence loss, this is not the main motivation of we choosing such form, as mentioned above. Indeed, one can easily verify that the convex combination form also coincides with the result of minimizing a weighted chi-square distance. Therefore, the usage of convex combination form and chi-squared divergence would not cause a consistency issue.
     
We would also like to mention that, the usage of chi-squared divergence instead of other information measures, e.g., KL-divergence, is mainly for the convenience of mathematical analyses. By exploiting the relationship between different information measures, one can readily express similar results expressed in other possible measures, with slightly different mathematical properties.      

c) We also appreciate the reviewer's consideration of the model considered in Section 3. Specifically, under our model Eq. (12), when the same feature extractor $\mathbf{f}$ is shared among different tasks, the knowledge transfer among different tasks is reduced to the knowledge transfer among different $\mathbf{g}_i$'s, represented as the weighted sum of $\mathbf{g}_i$ [cf. line 163]. Therefore, the $\mathbf{g}$ has already incorporated the transferable knowledge from different tasks.

In addition, we appreciate the reviewer's suggestions in demonstrating the usefulness/properties of our transferability measure by experimental results. Indeed, these properties have been observed from our experiments on practical datasets, e.g., CIFAR-10, and are quite consistent with our theoretical developments. For example, in Section 4.1, we show in Table 1 the similarities between the target task and different source tasks, represented as the test accuracies (in percentage) obtained from each single source task: % which are (in percentage) 66.5 (A), 59.7 (B), 56.2 (C), and 77.1 (D). As a comparison, the corresponding values of our transferability of source tasks A, B, C, and D are 0.19, 0.01, 0.09, 0.56, which demonstrates a consistent tendency. 

 We thank the reviewer for the valuable comments. Specifically, regarding the concerns on technical quality and transferability framework, our responses are as follows.

*[Question] Firstly, the paper is structured in a definition-theorem-proof style and does not provide enough explanation or motivation behind each design choice. More concretely, equations 2, 3, and definition 1 expound the basic structure of the framework, yet no explanation is given to motivate these choices. For example, one could provide citations to convince readers that this is common practice or argue that certain assumptions are weak and generally hold in practice.*

**[Response]** We appreciate the reviewer's careful consideration of the motivations for basic structures of our framework, which we would like to address as follows. Specifically, the assumption of convex combination form introduced in Eq. (2) has followed the convention of related works, e.g., *Ben-David et al.'s paper 'A theory of learning from different domains'*, which can also be regarded as a generalization of the usage of mixture models. Similarly, Definition 1 on $\chi^2$-divergence is an information-theoretic measurement that has drawn great interest recently, e.g., *Tao, C., Chen, L., Henao, R., Feng, J. amp; Duke, L.C.. (2018). Chi-square Generative Adversarial Network. ICML.* The testing loss defined in Eq. (3) follows the empirical risk minimization (ERM) framework for machine learning theory (*Vapnik, V. The Nature of Statistical Learning Theory. Information Science and Statistics.*). While each single element of the framework is common practice in the ML literature, the combination of these choices allows us to develop a closed-form transferability measure, which illustrates different impacting factors in transfer learning and has also been empirically validated in our experiments. Further theoretic discussions regarding the validity of these assumptions are possible, but obviously beyond the scope of this paper, which we plan to put in a follow-up paper.

*[Question] Secondly, the paper does not mention any properties enjoyed by the proposed transferability notion (Eq. 4).*

**[Response]** Secondly, regarding the properties of the proposed transferability measure, we have provided explanations on its key impacting factors and general properties. For example, following Theorem 2 (cf. line 106-118), we demonstrate that our transferability measure is in fact proportional to the number of model parameters, and is inversely proportional to the number of samples in source tasks and the similarity between source and target tasks. In addition, by exploiting Theorem 2, more specific properties can be readily obtained as straightforward corollaries, e.g., the transferability for a certain family of distributions. However, we did not include such discussions in our manuscript for the sake of compact presentations. 

*[Question] Thirdly, the experiments are designed to demonstrate the performance of Algorithm 1 and not the validity of the proposed notion of transferability (Eq. 4). The most glaring evidence is that, although the optimal coefficients $\alpha^\ast$ is introduced as the transferability measure in the previous section, $\alpha^\ast$ were never mentioned or discussed in the experiments, and the paper solely focuses on the test accuracy of the model.*

**[Response]** We would like to mention that the proposed transferability measure $\alpha^*$ plays a central role in Algorithm 1, essentially designed for estimating $\alpha^*$. Indeed, all experimental results were obtained based on the usage of $\alpha^*$. In presenting these results, we choose to report the empirical test accuracy (i.e, the accuracy evaluated on testing samples) instead of the value of $\alpha^*$ with the following considerations:

1. The transferability measure $\alpha^*$ is defined to minimize the theoretic testing loss, which in turn leads to a small empirical testing loss, and thus gives a high empirical testing accuracy. Therefore, from the theoretic perspective, the validity of our transferability measure can be reflected in the value of empirical test accuracy.
2. Our experimental result shown in Fig. 2 suggests that the empirical test accuracy is lower when we use a combining coefficient different from the transferability measure $\alpha^*$, which further validates the usage of empirical test accuracy to indicate the transferability measure.  
3. The empirical test accuracy can also serve as a performance measure for the transfer learning algorithm.

*[Question] I would appreciate the contribution here if the author can demonstrate that the proposed notion of transferability is useful. For example, one could construct a synthetic experiment in which one of the source tasks is more intimately related to the target task than the others, and demonstrate that the coefficient of that task is large.*

**[Response]** As mentioned above, the transferability measure plays a central role in our framework and cannot be separated from the algorithm. We also appreciate the reviewer's suggestions in demonstrating the usefulness/properties of our transferability measure by synthesized data. Indeed, these properties have been observed from our experiments on practical datasets, e.g., CIFAR-10, and are quite consistent with our theoretical developments. For example, in Section 4.1, we showed in Table 1 the similarities between the target task and different source tasks, represented as the test accuracies (in percentage) obtained from each single source task:  66.5 (A), 59.7 (B), 56.2 (C), and 77.1 (D). As a comparison, the corresponding values of our transferability of source tasks A, B, C, and D are 0.19, 0.01, 0.09, 0.56, which demonstrate a consistent tendency. 

 We thank the reviewer for the valuable comments, and our responses are as follows.

First, regarding the originality, one main contribution of this paper is the proposed mathematical framework that quantifies roles of impacting factors in transfer learning, including the task similarities, the sample complexity, and the model complexity, which were considered either in an implicit and empirical manner or by applying some indirect bounds of performance, e.g., [4-6] mentioned in the manuscript. Furthermore, our development also characterizes the contribution of the used parameterized model in determining the final performance, which was less addressed in previous works.

*[Question] However, the proposed solution is relatively straightforward without rigorous, principled justifications. While the proposed objective function and algorithm intuitively makes sense, the proposal is not validated in a rigorous, principled manner, ideally with certain theoretical justification.*

**[Response]** Regarding the comment on ``rigorous, principled manner, ideally with certain theoretical justification'', we do not quite understand what specific concern the reviewer was referring to, but would like to briefly summarize our theoretical justification as follows. The only assumption we made in our mathematical framework is the convex combination form of Eq. (2), the validity of which is beyond the scope of this paper and we will have a follow-up paper to discuss the details. Under this assumption, we provide the closed-form solution of the optimal combining coefficients, as shown in Theorem 2. Furthermore, this solution enables us to interpret it as a transferability measure, which illustrates the roles of different impacting factors in transfer learning. In addition, the theoretical framework is also empirically validated by our experimental results on real datasets.

*[Question] It is unclear to me whether a simple first-order method to solve the parameters jointly will suffice, making the proposed algorithm unnecessary.*

**[Response]** On the specific question about the algorithm, we would like to mention that Algorithm 1 cannot be replaced by a simple first-order method with joint optimization, since the two steps of alternating iterations have intrinsically different objectives; specifically, in one step we minimize the theoretic prediction of testing loss based on our developments, while the other step is to learn a parameterized model that fits the training data.  

*[Question] Experimental results are also somewhat limited both in terms of datasets and in terms of competing methods. It is unclear to me why the datasets and methods compared are representative enough to be compared against.*

**[Response]** Regarding the experiments, we considered the multi-source transfer learning applications with few samples from the target task. As for the experimental design, e.g., the choices of competing methods, our primary goal is to illustrate how to apply our theoretic transferability measurements to improve the performance of practical learning tasks, instead of providing comparisons with a comprehensive list of empirical approaches for transfer tasks.

 We thank the reviewer for the valuable comments, and our responses are as follows.

First, regarding the discrete analyses in Section 2, the seemly strong discrete assumption provides conveniences in mathematical analyses, while capturing the main impacting factors in transfer learning, including task similarities, sample size, and the model complexity, as illustrated in Theorem 2 and Theorem 3.

To be more specific, the discrete setting corresponds to the special case with no priors posed on learned distributions. Therefore, the learning parameters are the values of probability mass functions, and the parameter space coincides with the distribution space. Hence, the learning capacity, characterized as the size of parameter space, is equal to the cardinality of the alphabet, as characterized by $V^{(0)}$.
  
When we proceed to the general continuous setting, the only difference is that we have introduced some parameterized models to describe the distribution, see, e.g., Eq. (12). Since the number of learning parameters is always finite, the space of all parameterized models is a much smaller subset of the entire distribution space with infinite dimensions. Thus, the model complexity is also finite. Specifically, for the parameterized model considered in Section 3, such model complexity is characterized by $\tilde{V}^{(i)}$ in Theorem 4, with details illustrated in Eq. (32) of the supplementary material. 

For the two questions that follow:

$\bullet$ There have been a number of model complexity measures developed for different learning objects, e.g., VC dimension, Natarajan dimension,  Pollard's pseudo-dimension, and Rademacher complexity. Our framework that minimizes the expected testing loss leads to the model complexity described as $|\mathcal{X}||\mathcal{Y}| - 1$. The independence between the model complexity and the underlying  distribution is a consequence of the fact that, the variance of each probability mass is proportional to the underlying value, and is canceled out in the computation.

$\bullet$ The expression of optimal $\alpha$ in the multi-source setting in general does not admit a closed-form solution, due to the nonlinear constraints on non-negative entries. However, the optimal solution enjoys similar properties and interpretations as the analytical solution introduced in Theorem 2, and its numerical value can be efficiently computed for practical usages. 

Finally, we would like to thank the reviewers for suggesting improvements on the presentation and pointing out typos.

This paper proposes a new measure to quantify transferability in multi-source transfer learning problems, which takes into account three factors: 1). the similarity between domains, 2). the sample size of each domain, 3). the model complexity. Overall the paper is very clear, and I also enjoy some of the discussion that reveals the underlying insights from the authors. However, I do have several technical questions. See more detailed comments below. 
  The assumption that the input space $X$ is discrete is quite strong. I was wondering is this assumption crucial for the analysis? The authors mention that the definition of $V^0$ in Eq. (7) only works when $|X|$ is finite, otherwise $V^0$ would become infinity. I know that later in Section 3 the authors discuss such an extension, however, it seems to me that the results in Theorem 4 no longer shares the characterization that $V_i$ measures the size of the problem? Could the authors comment on this? 

I like the interpretation of the result in Eq. (5) as a decomposition of bias and variance. This is quite interesting. 

Questions:
-   In the discussion between Line 108 - 109: why the authors describe $|X||Y| - 1$ as the model complexity? I would think model complexity often refers to VC dimension, covering number, etc. It seems to me this explanation is a little bit forced. Shouldn't the $|X||Y| - 1$ solely depend on the problem specification itself? 

-   The authors briefly mention that the optimal solution of $\alpha$ in the multi-source setting could be attained by solving a nonnegative quadratic programming problem. It would better if the authors could also derive the closed form solutions (if possible) in this case, as what has been done in Theorem 2. 

Other comments:
-   Line 25: the citation of the GAN paper on describing it as transferring techniques is not accurate. 
-   Line 145 - 146: "the previous analyzing framework" -> "the previous analysis framework".
 Yes

This paper proposes a notion to quantify the transferability and derive the analytical form for this notion. An alternating iterative algorithm based on this notion is then proposed, and shown to outperform other transfer learning algorithms through experiments.   Pros:
1. This paper proposes a notion to quantify transferability and derive the analytic form, which depends on sample size, chi-sq distance between the source and target tasks, and the complexity of the model.
2. A new algorithm is then proposed based on this notion for transfer learning.
 
Cons:
1. To quantify the transferability, this paper considers the setting where the learned model is a convex combination of source distributions and then measures the chi-sq distance between this learned model and source/target tasks. This setting limits the use of this notion: 

  a). in the recent existing ML and DL literature, the learned models produced by the commonly used methods have very different forms from the convex combination of source distributions, so only considering the model family of convex combinations seems to be too limited. 

  b). In the paper, this convex combination form is motivated by minimizing a weighted KL-divergence loss function, but the transferability is measured by chi-sq distance. Is there a way to make this more consistent? (eg. measure the transferability by the KL divergence.) 

  c). In the model considered in Section 3, this model basically assumes that different source tasks share f, but have different g's. The convex combination form (which is obtained by minimizing the weighted chi-square divergence (17)) does not utilize this structure of sharing g's. Is there a way to incorporate such knowledge and therefore improve learning?

2. For the experiment. Since the paper's goal is to propose a notion to quantify the transferability, it would be better to add some experiments to compute the transferability using the analytical form derived in this paper for different datasets, and then compare these quantities with the accuracy improvement of using transfer learning. The proposed notion of transferability would be more convincing if it can be shown that there is a monotonic relationship between the proposed quantity and accuracy gain. 


Meta-review: This paper proposes a new measure to quantify transferability in multi-source transfer learning problems, which takes into account the similarity between domains, sample size of each domain and model complexity. The authors have not been successful in responding to all the concerns raised by the reviewers. mainly, two important concerns remain. one theoretical side, it would surely enhance the paper to have justification/proofs/lower bound results on the measure. if not, on the empirical side, the authors need to provide more evidence on success of the proposed measure. 

Decision: accept