A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning
Authors: Xinyi Tong
Xiangxiang Xu
Shao-Lun Huang
Lizhong Zheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transfer Learning
Transferability
Sample Complexity

Abstract: Current transfer learning algorithm designs mainly focus on the similarities between source and target tasks, while the impacts of the sample sizes of these tasks are often not sufficiently addressed. This paper proposes a mathematical framework for quantifying the transferability in multi-source transfer learning problems, with both the task similarities and the sample complexity of learning models taken into account. In particular, we consider the setup where the models learned from different tasks are linearly combined for learning the target task, and use the optimal combining coefficients to measure the transferability. Then, we demonstrate the analytical expression of this transferability measure, characterized by the sample sizes, model complexity, and the similarities between source and target tasks, which provides fundamental insights of the knowledge transferring mechanism and the guidance for algorithm designs. Furthermore, we apply our analyses for practical learning tasks, and establish a quantifiable transferability measure by exploiting a parameterized model. In addition, we develop an alternating iterative algorithm to implement our theoretical results for training deep neural networks in multi-source transfer learning tasks. Finally, experiments on image classification tasks show that our approach outperforms existing transfer learning algorithms in multi-source and few-shot scenarios.


A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning
Xinyi Tong
Xiangxiang Xu" Massachusetts Institute of Technology xuxx@mit.cm
Tsinghua-Berkee Shenzhen Institute Tsinghua University txy 10yy18mmlls..mmmm tsinghua. edu. cn
Shao-Lun Huang!
Lizhong Zheng Massachusetts Institute of Technology lizhongêmit. edu
Tsinghua-Berkelor Shenzhen Institute Tsinghua University
shaolun. huangnss tsinghua edu. cn
Abstract
Current transfer learning algorithm designs mainly focus on the similarities between source and target tasks, while the impacts of the sample sizes of these tasks are often sufficiently addressed. This paper proposes mathematical framework for quantifying the transferability in multi-source transfer learning problems, with both the task similarities and the sample complexity of learning models taken into account. In particular, we consider the setup where the models learned from different tasks are linearly combined for learning the target task, and use the optimal combining coefficients to measure the transferability. Then, we demonstrate the analytical expression of this transferability measure, characterized by the sample sizes, model complexity, and the similarities between source and target tasks, which provides fundamental insights of the knowledge transferring mechanism and the guidance for algorithm designs. Furthermore, learning tasks, and establish quantifiable transferability measure by exploiting a parameterized model. In addition, we develop an alternating iterative algorithm implement our theoretical results for training deep neural networks in multi mouti-source transfer learning tasks. Finally, experiments image classification tasks show that our approach outperforms existing transfer learning algorithms in multi-source and few-shot scenarios.
apply our analyses for practical
Introduction
Transfer learning is nowadays an active research area in machine learning focusing on solving target learning tasks by the knowledge of learnable source tasks. The transferability between source and target tasks the central topic in transfer learning for understanding the knowledge transferring mechanisms and the algorithm designs (I]. In general, the transferability can be affected by several factors, including: (i) the similarities between source tasks and the target task 2 (ii) the sample sizes of the tasks; and (iii) the complexity dimensionality of the machine learning model. Most of the existing transfer strategies are designed based on how similar the source and target tasks are [ without considering the impacts of the training sample sizes the complexity of the models.
Work performed at Tsinghua- Berkeay Shenzhen Institute Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In theoretical analyses 4 5 6 sample sizes and model complexity are often included in deriving upper bounds for the transferability or the performance of transfer learning algorithms. However, pointed out that such bounds derived under general learning settings are often relatively loose under numerical simulations 77, and hence the algorithms designed by directly applying theoretical results can hardly achieve satisfactory performance in practical applications. Thus, the gap between theory and practice opposes the fundamental understandings of transfer learning algorithms.
In this paper, we propose a mathematical framework to investigate the transferability in multi-source transfer learning problem, and establish a quantifiable transferability measure for practical learning tasks. Specifically, for given source tasks, we learn the target task by class of learning model which linearly combines the models learned from individual tasks by some designable coefficients. In addition, the performance of this combined model is measured by the empirical risk of only the resting data of the target task, considered as the testing loss. Then, we adopt the optimal combining coefficients that achieve the minimum testing loss as the transferability measure, which illustrates the contribution of each model in learning the target task, and effectively quantifies the knowledge transferable among different tasks.
In our development, we establish an analytical solution of the transferability measure, which jointly quantified by sample sizes, model complexity, and a similarity measure between source and target tasks. particular, we demonstrate that the transferability of a particular source task is typically proportional to the number of samples and the measure of similarity to the target task, and is inversely proportional to the model complexity. This coincides with the intuition that when more training samples are available for a source task that is highly similar to the target task, more knowledge will be transferable from the source task to the target task. On the other hand, when the model is very complicated or high-dimensional, it is typically harder to train the model well, and less knowledge can be acquired and transferred. More importantly, our theoretical results can be applied for designing effective and efficient algorithms for real transfer learning problems, which are especially useful for multi-source transfer learning with large number of source tasks that are generally difficult to deal with.
The contribution of this paper can be summarized follows:
We propose a mathematical framework for transfer learning analyses, and establish transferability measure on discrete data, quantified by the number of samples, the complexity of the model, and the x--distance between source and target tasks.
We extend the transferability analyses to the continuous data, and establish a similar transfer- ability measure that can be evaluated in practical tasks, by exploiting paramerized models.
We apply our theoretical results to develop an iterative algorithm for training deep neural networks in general supervised transfer learning scenarios. Morcover, our algorithm can be practically applied for multi-source transfer learning.
The experiments in real datasets validate our proposed algorithm, in which we show that our approach outperforms many existing transfer learning algorithms.
Due the space limitations, the proofs of theorems and propositions are presented in the supplemental material.
Problem Formulation and Analysis
Let X and Y be the random variables denoting the data and label with domains and y, respectively, and let P denote the set of all distributions on X y. For the convenience of illustration, here we assume to be discrete, and will extend our analyses to continuous cases later. Throughout our analyses, we will use Ar A ((on....,a)) Σi as 1, > 0, = O,...,k} to denote the k-dimensional simplex.
2.1 Single-Source Transfer Learning
To begin, we consider the transfer learning setting with one source task and one target task, denoted as task 1 and 0, respectively. Specifically, for each task i 0, 1, we assume that n, training
2

---

(0) )2 are i.i.d. generated from some underlying joint distribution XY E P with samples {(x]] :38
(x, y) > 0, for all 2, y, and the empirical distributions
((s)
XY E P of the samples are defined as
P (2,
(9) ( = I, I, = y).
The
where 11 {.) denotes the indicator function 8 Then, the empirical distributions P and PXY can be regarded the models learned from the target task and the source task, respectively, when all the entries of the mass functions are required to determine.
To develop the transferability measure, our proposed framework focuses convex combination of both learned model
Qey a0) (x og ((0) (x,y) y) + PCL (I, v). for all (xr, v) E x y,
(2)
where (ag, a1) E A1 are parameters to be designed. Notice that these parameters characterize the knowledge transferred from the source task to target task, and the designing of these parameters will be affected by the sample sizes and the task similarities, which essentially leads to a transferability measure adjusted by the sample complexity.
Then, the performance of the model (a is evaluated by the testing loss, measured by its empirical risk on the testing data of target task. Conventionally, such empirical risk is often computed by the logarithm loss. However, the logarithm risk can ill-defined in our setting. Therefore, alternatively apply the referenced distancce the measure, defined as follows.
Definition 1. Given a reference distribution Rxy, for any distribution Pxy and Qxy, the referenced disttancce between them is defined as
(Pxy(x.u) - Qxy(x,)))
XRx (Pxy,Qxx)
Σ
Rxy (2,
TEX.NEY
Specifically, we denote (Pxx, XPxy (Pxy,Qxy), which corresponds the Pearson x" diverrngee
We choose the underlying target distribution Pxy as the reference, and define the testing loss as the averaged Pearson x² 2arroonn==ddiggegeeee --divergence
L
(ao.as) test
&
E
x2
XY (a a)
))
(3)
where the expectation is taken over all i.d.d samples generated from the source and target distributions. Moreover, define the optimal coefficients
(a a;j) A arg min
Liest
(1g.01)
(4)
(ap,an
as our transferability measure, which effectively quantifies the contributions of the source and target tasks in obtaining the optimal performance.
Then, we have the following characterization.
Theorem 2. The testing loss as defined B is
L L (a0.as)
a²x² (PSt), p(t
ag (()
aj
(5)
Ykn
'The assumption positive entries is without loss of generality, since in practice such joint distributions typically modeled by some positive parameterized families, e.g., the sofumax function.
*We shall emphasize that such combination forms are naturally led by the transfer learning model proposed by
4
Section l] from optimizing convex combination of Log-Loss, where we refer to Section A of supplementary
material for detailed discussion.
'Nte that when some (x, y) pair is missing in training samples, have (00):01) (x, v) 0 while X
XY (a. y) 0, which would lead to infinite logarithm risk.

---

Xy, and the corresponding empirical and the transferabilitty measures as defined in are
Rg (
0j (P P ( Lva
and og aj:
(6)
m
where, for each i 0, V(i) defined as
P (x,
(1 PXr (2,
V A
Σ
PX (2,
TET
From (6). and the fact that V | - 1, the transferability is determined by three key factors: (i)
the similarity between source and target tasks, measured by the <-viieereence x² ( P (ii)
the sample sizes no and n for source and target tasks; and (iii) the model complexity, characterized by the number of model parameter ([C)) | 1) in (0) K
Current transfer learning algorithm designs often focus on the similarities between source and target tasks, while the sample sizes and model complexity are often not sufficiently addressed. In Theorem 2 show that the transferability is in fact proportional the number of model parameters, and is inversely proportional to the number of samples source tasks and the similarity between source and target tasks. Therefore, for source task with a complex model few training samples, even though it is similar to the target task, the knowledge transferable from this source task can still be very limited. Such insight was not well captured in many existing transfer learning algorithms, and our result essentially provides the optimal characterization of the task transferability adjusted by the sample complexity in transfer learning.
The established transferability measure is also related to the optimal bias variance trade-off 9 of this transfer learning problem. Indeed, note that the bias-variance trade-off in testing loss (5) is tuned by og and (1, as
(10)01) test
oîp² (PXY. PX
V
v
(8)
70
71
bias term
variance term(s)
where the bias term does not decay with the sample sizes no. A1, while the variance terms vanish with sufficient samples. Then, the transfcrability measure corresponds to the coefficients ag of that achieve the optimal bias-variance trade-off, such that the testing loss is minimized.
2.2 Multi-source Transfer Learning
Theorem 2 can be readily generalized to multi-source transfer learning problems. Specifically, suppose that there are & source tasks, referred to as task i, for = 1,... k, and target task, referred use (+() xY, { (x((") Me
(i) ) and xx to denote
as task O. Similarly, for each task O.... ,
the underlying distribution, i.i.d. samples generated from ((i) distribution as defined in (T. respectively.
Similar to we consider the convex combination of the models learned from different tasks
Q
OXY
((0) XY
E Ar.
==0
Then, we define the testing loss (a) and the corresponding transferability measure a", as [cf.
(()) A E x2 (Pt 08)) and a f arg min T.(a)
(10)
aε
Similar to Theorem we have the following result for multi-source transfer learning.
'When evaluating (1) this quantity is related to the source distribution. From this perspective, the model complexity reflects how hard the task is.

---

[(a)
g
g
Input
(a) Feature Extractor
(b) Classitier
Figure 1: A pre-trained neural network for classification can be divided into (a) feature extractor
which generates feature f(x) = (z), ... fa(x)]]* Rd and (b) classifier with the weights g- With f fixed, our framework optimizes the weights g in the topmost layer for each task, to obtain the corresponding parameterized representation.
Theorem 3. For the model ( the testing loss under the target task is
(p.))
I [
x2
Σ as p(1)
(0
(11)
n2
where 's)'s are as defined in for all i.
From Theorem 3 the transferability measure a defined in (TO) can be computed by solving non-negative quadratic programming problem [OO]. Similar to the discussions in Section [21] such
transferability measure quantífies the knowledge transferable from different source tasks the target task with the sample complexity being considered.
3 Parametric Models and Transfer Learning Algorithm
3.1 Transferability Measure with Pre-trained Neural Network
This section extends the analyses in the discrete data domain to continuous data in practical problems. In such cases, the previously adopted learning model (I) has infinite parameters due to the infinite cardinality |30, and thus not be effectively represented. In order apply the previous analyzing framework, we first propose a parameterized representation for modeling features of the continuous data by exploiting pre-trained model.
As shown in Figure | a pre-trained network can be divided into two parts: (a) the previous layers for extracting d-dimensional features f(x) = [fi (z), fa(z)]" from the data variable and (b) the topmost layer for linear classification, with weights g = [g1 (v), ... (r()) indexed by label y. When the feature fax) is given and fixed, the models learned from different tasks can be effectively represented by a finite collection of parameters, i.e., ((1))...gg(())...
In particular, our framework considers the discriminative model in the factorization form
p(f.a) Y|x (y|z) A P( (v) (1 f 1 (z)g(v))
(12)
which is similar to the ones introduced in factorization machines [I] and natural language processing applications [2]. Then, for each task = O..... k. we learn corresponding weights g., such that the ()) fits the training sample The weight ğ: can be formally defined as
learned model YIX
(P ((0) p(f.g)) gi arg min XRX xY, YIX
(13)
"The approach of retraining (fine-tune) tuning) the topmost layer is sometimes referred to as the retrain-head method [13]. which has also been widely adopted in transfer learning applications.
5

---

where the fitness is measured as the referenced x² -iitaance [cf. Definition between the empirical Y|X For convenience, we adopt unified
distribution XY and the joint distribution (0) p(T.9)
reference Rxy - PX Py in fitting different tasks.
From Px P().)) plays the role in the continuous case corresponding P(0 in the discrete
case. This allows us to apply previous analyses and focus on the discriminative model p(f.))) Analogous to (9. we consider the convex combination of these discriminative models
FYxx
(a) yix
a
p( p(f.9)
(14)
with 9 - E
Li=o a,g, Then, we define the testing loss and corresponding transferability measure as [ef. (TOM]
L
test
E
XRx
PS x and a"
- arg min L L
(15)
aEAc
for which we have the following characterization.
Theorem 4. The testing loss
15
associated with the model (1) is
I XRxY
p(0)
pl) YIX
Σ
a.
(0) p(..9.) YIX
a
v() + xRxx
(Phr: ((0)
p(t)
p(S. YIX
n,
(16)
where 9: - arg ming Xaxx ( characterized in the supplementary material [ef. (32)).
Px
p(o) p(f-s)), and where V(6) is a constant independent of a YIX
Moreover, note that from the definition of gi the joint distribution p00) p(f-g.) VIX can be interpreted
as a projection of p(t) XY onto the distribution family x p(5.9) YIX y with referenced 2 distance used the distance measure. Therefore, the terms of (I6) share similar interpretations as their counterparts in Theorem with the distances measured in the projected space. Again, a can be efficiently computed by solving non-negative quadratic programming problem.
3.2 Multi-source Transfer Learning Algorithm
With our theoretic analyses in Theorem 4
develop knowledge transfer algorithm for multi- source transfer learning. Different from the previous analyses where f is fixed, our algorithm jointly optimizes the extracted feature f. the weights g, together with the combining coefficients obtain better performance.
To begin, for given f. and a. we introduce the loss function
k
((a Σ aixaxr (PSr, p (()p(()))
(17)
The following result illustrates that, the 9 can be computed via directly minimizing this loss, without evaluating each ği individually.
Proposition 5. The 9 as defined satisfies
9 arg min I (aa.f''
*The joint distribution ((() (5 is defined as [P(C] (D) p(r-9) (x, v) 2 p(o) (x) (vl=), for all (z,y). YIX Note that when discriminative model p( of the target distribution P())
fixed, p(o) p( YIX
corresponds to the optimal approximation

---

Table 1:
Test accuracies (%) the target task, with the network trained on samples from single source. All reported accuracies are averaged over repeated experiments.
Source Task
1
2
3
4
Acc. on the target task
66.5
59.7
56.2
77.1
Table 2:
Test accuracies (%) on the target task, compared with the combining coefficients a deter- mined by 20 rounds of random searches (RS).
Target Sample Size
20
100
Acc. with only target samples
70.9
74.4
81.5
Average acc. by RS
67.8
73.9
75.4
Highest acc. by 20 RS
74.4
78.0
80.8
Acc. by Algorithm !
78.9
81.2
83.7
Then, with training samples from different tasks, our algorithm alternates between two different kinds of optimizations: (i) the optimization of a for given g) to minimize the testing loss L as defined in (T6), via solving non-negative quadratic programming problem; and (ii) the optimization of (f, g) for given a to minimize the loss L (a.f.)) as defined in (17) via training the neural network. We summarize the procedures as Algorithm
Specifically, can shown that both the testing loss T,(a) and the loss f can be represented by some expectations of features f and g. In computing these losses, these expectations approximated by corresponding empirical means, with details provided the supplementary material.
Algorithm 1 Multi- Source Knowledge Transfer Algorithm
1: Input: target and source data samples {<x)", . ( (i -...... .k)
2: Randomly initialize a²
3: repeat
4: (f",g") + arg min 9.9 Lia' .f))
5: a" <- arg min [(a)
6: until a" converges
7:
+ arg min f.g ((aa ffg)
8:
return f"
With the fr and g" computed by the algorithm, for newly observed target sample the predicted label ý is given by the MAP (maximum a posterior) decision rule
((x)
arg max
p(f g¹) (y/zz) x
arg max pS (v) (1+ (TT
(18)
yEy
DEy
4 Experiments
To validate the effectiveness of our algorithms multi-source learning and few-shot transfer learning scenarios, conduct series of experiments common datasets for image recognition, including CIFAR-10 [14], Office-31 and Office-Caltech [15). In all experiments, the g in the classifier is simply generated by an embedding layer.
4.1 Multi-source Transfer Learning
We conduct multi-source transfer learning experiments on CIFAR-1O, which contains 000 training images and 000 testing images in 10 classes. To begin, construct the source tasks and target task by dividing the original CIFAR-16 10 dataset into five disjoint subdatasets, each containing two classes of the original data, which corresponds to a binary classification task. Then, choose as our target task (task O), and use the other four as source tasks for transferring knowledge, referred to task 1, 2, 3, 4.
Moreover, for each source task, 2000 images are used for training, with 1000 images per binary class, and set target sample size ng to no 20, 100, respectively. Throughout this experiment,

---

Table 3: Test accuracies for target tasks under different transfer settings (source - target) Office-31
Method
A++D
A
D-W
D->A
W-+A
W-D
SDT 20
86.1
82.7
95.7
66.2
65.0
97.6
DAMA 21
86.3
84.5
95.5
66.5
65.7
97.5
FADA [22
88.2
88.1
96,4
68.1
71.1
97.5
UDDA 2X
89.0
88.2
96.4
71.8
72.1
97.6
Ours
90.0
87.3
96.5
72.4
72.1
97.2
Table 4: Test accuracies for target tasks under different transfer settings Office-Caltech
Method
A-C
W-C
D-C
C-A
C-#
C-+
GFK LS
68.4
68.4
64.5
83.8
78.7
74.6
TLDA
24
76.1
71.0
65.4
84.2
85.2
78.9
DTML
25
72.0
71.6
67.1
86.0
85.0
79.6
CPNN
78.5
73.5
68.0
86.3
86.2
80.1
Ours
80.3
72.9
72.2
88.4
85.9
83.5
the feature f is of dimensionality d = 10, generated by GoogLeNet [16]. followed by two fully connected layers for further dimension reduction.
Unlike common transfer learning settings where the labels for the source and target tasks are closely related, here the binary labels for these 5 sub-datasets are in general irrelevant. Therefore, first establish the correspondences between labels as follows. For each given source task, we first train the network on its training samples, while the accuracy is evaluated on the samples from the target task. Then, we flip the original binary label for this source, if the test accuracy is less than 50%. The resulting accuracies on the target are summarized Table T
In our implementation of Algorithm I we use the CVXPY 17 18 package for solving the non- negative quadratic programming in line 5. In addition, the alternating iteration is stopped when the element-wise differences for ax computed in two successive iterations are at most 0.05.
Then, the test accuracies of our algorithm on the target set are shown in Table 2 where we have compared our performance with random search (RS) strategy. Specifically, in the RS strategy, we generate the coefficients from the log-uniform distribution [19] in [0.001, 1]. for 20 rounds. The results indicate that our approach outperforms the random search method. Also, the difference in Table 2 and Table also shows the performance gain of multi-source transfer learning over single-source.
4.2 Few-shot Transfer Learning
To validate the effectiveness of algorithm for few-shot learning tasks, conduct experiments Caltech-31 and Office-Caltech datasets. These datasets provide typical transfer learning tasks with few available training samples, where the influence of sample complexity is shown.
4.2.1 Caltech-31
Caltech-31 dataset contains images of 31 categories, which come from 3 sub-datasets: Amazon (2817 images), Dsir (498 images), and Webcam (795 images). Then, different transfer settings among these sub-datasets are denoted by the "source target",, as: A-DD A->,, D-W, D-*, W-A, and W-DD, We adopt the few-shot transfer learning setting [20), illustrated as follows. Specifically, 3 target samples per category are used for training, and the training sample size (per category) for source task is 20 8, depending on whether the source task Amazon not. Moreover, also adopt five train-test splits introduced in 20
In experiment, the feature is 64-dimensional vector, extracted by VGG-16 (77 network pre- pre-trained on the ImageNet, succeeded by two fully connected layers for dimension reduction.
Table 3 summarizes test accuracies for target tasks under different transfer settings, where all reported accuracies are averaged over five train-test splits. The results indicate that our algorithm generally outperforms existing few-shot transfer learning methods.

---

100
Coefficient oz Accuracy
0.6
Value of on Accuracy
0.4
0.0
2
6
8
10
lteration
Figure 2: The combining coefficient on of the task ADD and test accuracies under testing samples of Dslr during iterations.
In addition, we also investigate the convergence of the coefficient a" in Algorithm As an example, for the A+D task, the changes of the coefficient α (the coefficient of the source loss) and accuracy on the Dsir test dataset during iterations are shown in Figure 2 From the figure, the value of on converges under our stopping criterion, where the optimal testing accuracy is obtained.
4.2.2 Office-Caltech
Office- CalteCh dataset is composed of 10 common categories in Office-31 and Caltech-256, divided four sub-datasets: Amazon (958 images), Caltech (1123 images), Webcam (295 images), and DsIr (157 images). We focus on the 6 transfer settings depending on C, i.e. A-C, W-C, D-C, C-A, C+W, and C-+D, which have few common categories between source and target tasks.
In addition, we follow the setting introduced in [5] sionality d = 10, based on the DeCAF feature 28 reduction.
for train-test split. The feature f is dimen- with 2 fully connected layers for dimension
29
Table shows the performance for our algorithm, in comparison with several sepi-supe and few-shot domain adaptation algorithms. is worth mentioning that, though our approach does not use the unlabeled data samples in training, it provides competitive performance the semi-supervised algorithm CPNN [26), and can be better on specific tasks.
5 Related Work
Theoretical Analyses of Transfer Learning and Transferability. Most of the theoretical works about transfer learning focus on deriving upper bounds for the transferability or the performance of transfer learning. For example, the generalization сттоr be bounded by the VC-dimension of
the hypothesis space 33), the total variance distance [4] or the mutual information between training
samples and outputs (5). and the Censen-Shanoor distance between domains (6]. The choices of measures are determined mostly by the problem settings. Furthermore, some of these different measures and the x2 diverrege used in work closely related, which generalized as f-divergence [31]. However, there can exist a significant gap between the theoretical bounds and the performance for real tasks 7
There are also works concentrating on defining a transferability measurement in an empirical way. For instance, the empirical log- liceeiood the target data under the network trained by source samples can measure how much the source samples would help improve the target task [33 32). Compared with these works, establish transterability measure and provide an analytical expression, for guiding algorithm designs.
Transfer Learning Algorithms. Transfer learning algorithms based on the insights from theoretical works intuitively measure similarities between different domains. The similarity measures include the low-rank common information [3 K-L. divergence (34
35
36
37 la-distance 38 39 40)
and Wasserstein distance 4T). Additionally, transfer learning problems also share the similar framework with meta- meta-learning, which concentrates obtaining generalized model for different

---

tasks, especially when lacking enough samples for all categories 42 works, our algorithm takes the sample sizes and model complexity into consideration, and can be more applicable for general learning tasks, including the few-shot setting.
In comparison with the above
Multi-source Domain Adaptation and Few-shot Domain Adaption. Multi-source domain adapta- tion considers the approaches of combining multiple tasks together. Conventional methods mainly include instance weighting [43 and domain weighting [4 [15, which re-weight samples or loss functions in training, respectively. Based on deep learning, cutting-edge algorithms attempt to maximize the domain confusion
46 or learn the domain-invarte invariian representations [47].
Semi-supeervied and few-shot domain adaptation focus on transfer learning algorithms under few labeled target samples, which is of our work's application scenarios. Compared with conventional domain adaptation, this field pays attention to embedding samples into an intrinsic low-dimensional subspace [1]. Common algorithms aim at learning domain-invartie representations, including
simultaneous deep transfer (SDT) method [20] and semantic alignment method [22 23
Compared with these empirical studies, our characterization provides practical learning algorithm under theoretical guarantees.
Conclusion
This paper introduces a mathematical framework for quantifying the transferability in multi-source transfer learning problems. Our characterization reveals the essential roles of sample sizes and model complexity in knowledge transferring, which demonstrates potentials in establishing a unified understanding of various transfer learning algorithms. In addition, we develop multi multi-source transfer learning algorithm based the theoretical analyses. Experiments on practical multi-source learning tasks show the effectiveness of our proposed algorithm.
Acknowledgements
The research of Shao-Lun Huang is supported in part by the National Natural Science Foundation of China under Grant 61 807021, and the Shenzhen Science and Technology Program under Grant KQTD201 708101 50821 146.
The research of Lizhong Zheng is supported in part by the National Science Foundation (NSF) under Award CNS-2002908 and the Office of Naval Research (ONR) under grant N00014-19-1-2621.
References
[1] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions on knowledge and data engineering, 22(10):1345-1335 2009.
[2] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270-279. Springer, 2018.
[3] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27:2672-2680, 2014.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151-175, 2010.
[5] Y. Bu, S. Zou, and V. V. Veeravalli. Tightening mutual information based bounds on gener- alization error. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 587-591, 2019.
[6] Han Zhao, Remi Tacher Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019,
10

---

[7] Xuetong Wu, Jonathan H. Manton. Uwe Aickelin, and Jingge Zhu. Information-theow analysis for transfer learning. 2020 IEEE International Symposium on Information Theory (ISTT), Jun 2020.
[8] Willliam Feller. An introduction to probability theory and its applications, vol John Wiley Sons, 2008.
[9] Richard Duda, Peter E Hart, et al. Puttern classification and scene analysis, volume Wiley New York, 1973.
[10] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[11] Steffen Rendle. Factorization machines. In 2010 IEEE International Conference on Data Mining, pages 995-1000, IEEE, 2010.
[12] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27:2177-2185, 2014.
[13] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR, 2020.
[14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[15] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066-2073 IEEE, 2012.
[16] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[17] Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
[18] Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision, 5(1):42-60, 2018.
[19] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10)281-335, 2012.
[20] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4068-400000 40764 2015.
[21] Piotr Koniusz, Yusuf Tas, and Fatih Porikli. Domain adaptation by mixture of alignments of second-or higher-order scatter tensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4478 088-4487, 2017.
[22] Saeid Motian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In Advances in neural information processing systems, pages 6670-6688) 2017.
[23] Sacid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. Proceedings of the IEEE International Conference Computer Vision, pages 5715-5725, 2017.
[24] Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Deep transfer metric learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 325-333, 2015.
11

---

[25] Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. Supervised represen- tation learning: Transfer learning with deep autoencoders. In Twenty-Found Fourth International Joint Conference Artificial Intelligence, 2015.
[26] Z. Ding, N. M. Nasrabadi, and Y. Fu. Semi-supervised deep domain adaptation via coupled neural networks. IEEE Transactions on Image Processing, 27(11):5214-5222 -2224444,, 2018.
[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[28] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffmman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647 2014.
[29] Judy Hoffman, Erik Rodner, Jeff Donahue, Trevor Darrell, and Kate Saenko. Efficient learning of domain-invariant image representations. arXiv preprint arXiv:1301.322 2013.
[30] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pages 129-136, 2008
[31] Peter Harremoës and Igor Vajda. On pairs of f-divergences and their joint range. IEEE Transactions on Information Theory, 57(6):3230-322 -2225, 2011.
[32] Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 1395-1405, 2019.
[33] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213-226. Springer, 2010.
[34] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: /4/2.3444 2014.
[35] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[36] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarte training of neural networks. Journal of Machine Leurning Research, 17(59):1-35, 2016.
[37] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176. 2017.
[38] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip Yu. Transfer joint matching for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410-1417, 2014.
[39] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pages 97-105. PMLR, 2015.
[40] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in neural information processing systems, pages 136-144, 2016.
[41] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[42] Ruohan Wang, Yiannis Demiris, and Carlo Ciliberto. Structured prediction for conditional meta-learning. In Advances Neural Information Processing Systems, volume 33, pages 2587-2598, 2020.
12

---

[43] Qian Sun, Rita Chattopadhyay, Sethuraman Panchanathan, and Jieping Ye. A two-stage weighting framework for multi-souree domain adaptation. Advances in neural information processing systems, 24:505-513, 2011.
[44] Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Domain adaptation from multiple sources: domain-dependent regularization approach. IEEE Transactions on neural networks and learning systems. 23(3):504-5) 518, 2012.
[45] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3964-3973, 2018.
[46] Han Zhao, Shanghang Zhang, Guanhang Wu, Geoffrey Gordon, et Multiple source domain adaptation with adversarial learning. 2018.
[47] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specifer distribution and classifier for cross-domain classification from multiple sources. Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 5989-5996, 2019.
[48] Xiangxiang Xu and Shao-Lun Huang. Maximal correlation regression. IEEE Access, 8:26591- 26601, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section T
(b) Did you describe the limitations of your work? [Yes] We clearly state the assumptions we use in Section
(c) Did you discuss any potential negative societal impacts of your work? [No] We have
done theoretic work that has few societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary material.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] We think the error bars are not related to the core result of our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We use few computation resources in our work.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
We don't use new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "hW9G5KXZBv", "reviewer": "Reviewer_NvNX", "summary": "This paper proposes a new measure to quantify transferability in multi-source transfer learning problems, which takes into account three factors: 1). the similarity between domains, 2). the sample size of each domain, 3). the model complexity. Overall the paper is very clear, and I also enjoy some of the discussion that reveals the underlying insights from the authors. However, I do have several technical questions. See more detailed comments below. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The assumption that the input space $X$ is discrete is quite strong. I was wondering is this assumption crucial for the analysis? The authors mention that the definition of $V^0$ in Eq. (7) only works when $|X|$ is finite, otherwise $V^0$ would become infinity. I know that later in Section 3 the authors discuss such an extension, however, it seems to me that the results in Theorem 4 no longer shares the characterization that $V_i$ measures the size of the problem? Could the authors comment on this? \n\nI like the interpretation of the result in Eq. (5) as a decomposition of bias and variance. This is quite interesting. \n\nQuestions:\n-   In the discussion between Line 108 - 109: why the authors describe $|X||Y| - 1$ as the model complexity? I would think model complexity often refers to VC dimension, covering number, etc. It seems to me this explanation is a little bit forced. Shouldn't the $|X||Y| - 1$ solely depend on the problem specification itself? \n\n-   The authors briefly mention that the optimal solution of $\\alpha$ in the multi-source setting could be attained by solving a nonnegative quadratic programming problem. It would better if the authors could also derive the closed form solutions (if possible) in this case, as what has been done in Theorem 2. \n\nOther comments:\n-   Line 25: the citation of the GAN paper on describing it as transferring techniques is not accurate. \n-   Line 145 - 146: \"the previous analyzing framework\" -> \"the previous analysis framework\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "G0r5IrL0CiB", "reviewer": "Reviewer_pdrf", "summary": "This paper proposes a notion to quantify the transferability and derive the analytical form for this notion. An alternating iterative algorithm based on this notion is then proposed, and shown to outperform other transfer learning algorithms through experiments. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper proposes a notion to quantify transferability and derive the analytic form, which depends on sample size, chi-sq distance between the source and target tasks, and the complexity of the model.\n2. A new algorithm is then proposed based on this notion for transfer learning.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nCons:\n1. To quantify the transferability, this paper considers the setting where the learned model is a convex combination of source distributions and then measures the chi-sq distance between this learned model and source/target tasks. This setting limits the use of this notion: \n\n  a). in the recent existing ML and DL literature, the learned models produced by the commonly used methods have very different forms from the convex combination of source distributions, so only considering the model family of convex combinations seems to be too limited. \n\n  b). In the paper, this convex combination form is motivated by minimizing a weighted KL-divergence loss function, but the transferability is measured by chi-sq distance. Is there a way to make this more consistent? (eg. measure the transferability by the KL divergence.) \n\n  c). In the model considered in Section 3, this model basically assumes that different source tasks share f, but have different g's. The convex combination form (which is obtained by minimizing the weighted chi-square divergence (17)) does not utilize this structure of sharing g's. Is there a way to incorporate such knowledge and therefore improve learning?\n\n2. For the experiment. Since the paper's goal is to propose a notion to quantify the transferability, it would be better to add some experiments to compute the transferability using the analytical form derived in this paper for different datasets, and then compare these quantities with the accuracy improvement of using transfer learning. The proposed notion of transferability would be more convincing if it can be shown that there is a monotonic relationship between the proposed quantity and accuracy gain. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5X6bthScND2", "reviewer": "Reviewer_FaWn", "summary": "The paper proposed a new objective function for transfer learning that accounts for the similarity between tasks and model complexity. An iterative algorithm is also proposed to solve this objective. The proposed method is validated on three real-world datasets empirically, comparing against various alternative methods for multi-source and few-shot transfer learning.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper aims to address an important machine learning problem in the field of transfer learning. The paper is clearly written, and the authors make an effort to factor in various important aspects of transfer learning in their solutions including sample complexity and task similarity. However, the proposed solution is relatively straightforward without rigorous, principled justifications. Experimental results are limited on three datasets.\n\nOriginality: the originality of the paper lies in the proposal of a new objective function for transfer learning that accounts for both task similarity and sample complexity. I find such a contribution somewhat limited.\n\nQuality: while the proposed objective function and algorithm intuitively makes sense, the proposal is not validated in a rigorous, principled manner, ideally with certain theoretical justification. I find the claim that the objective provides “fundamental insights” somewhat unsubstantiated. Furthermore, the iterative algorithm makes sense because each sub-step is concerned with solving a different subproblem. It is unclear to me whether a simple first-order method to solve the parameters jointly will suffice, making the proposed algorithm unnecessary. Experimental results are also somewhat limited both in terms of datasets and in terms of competing methods. It is unclear to me why the datasets and methods compared are representative enough to be compared against.\n\nClarity: I think the presentation in this paper is clear. Mathematical notations are well setted-up. I can follow the major argument of the paper easily.\n\nSignificance: based on my concern in originality and quality, I find the paper is of limited significance.\n\n======== After rebuttal ========\n\nThank you for the detailed response. My questions are partially answered by the authors, and I would like to keep my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is a lack of discussion of the limitations of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "21-W1c5ZFrI", "reviewer": "Reviewer_HgrD", "summary": "The paper proposes a framework for evaluating transferability, i.e., to quantify how much knowledge one can pass from a source task to a target task. Task transferability is studied under the assumption that the target task’s model is a convex combination of the source tasks’ models. And the notion of transferability is defined as the optimal convex combination coefficients, quantifying the contributions of each source task to the target tasks. Theoretical results relate the transferability to three quantities: (a) the similarity between the source and target tasks, (b) the sample sizes of the source and target tasks, and (c) the models’ complexity. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality & Significance: The idea of studying task transferability under the assumption that the target task’s model is a convex combination of the source tasks’ models is new to me. The linearity assumption allows for a very simple and nice solution: the optimal coefficients can be computed in closed-form (in finite domain cases). The simplicity of the solution allows for the great applicability of the methods. However, I find the paper does not provide enough information to convince the reader that this is a good transferability measure? I will elaborate in the below sections.\n\nTechnical quality:\n- While I am convinced that the technical theorems are correct, I am unsure why this would be a good framework to measure transferability across tasks. \n- Firstly, the paper is structured in a definition-theorem-proof style and does not provide enough explanation or motivation behind each design choice. More concretely, equations 2, 3, and definition 1 expound the basic structure of the framework, yet no explanation is given to motivate these choices. For example, one could provide citations to convince readers that this is common practice or argue that certain assumptions are weak and generally hold in practice.\n- Secondly, the paper does not mention any properties enjoyed by the proposed transferability notion (Eq. 4). \n- Thirdly, the experiments are designed to demonstrate the performance of Algorithm 1 and not the validity of the proposed notion of transferability (Eq. 4). The most glaring evidence is that, although the optimal coefficients \\alpha^\\ast is introduced as the transferability measure in the previous section, \\alpha^\\ast were never mentioned or discussed in the experiments, and the paper solely focuses on the test accuracy of the model.\n- The introduction of the transferability \\alpha^\\ast seems unnecessary. I feel that the focus of the paper is a new algorithm for transfer learning (Algorithm 1), and not a mathematical framework for measuring transferability across tasks. I would appreciate the contribution here if the author can demonstrate that the proposed notion of transferability is useful. For example, one could construct a synthetic experiment in which one of the source tasks is more intimately related to the target task than the others, and demonstrate that the coefficient of that task is large.\n\nClarity: The main paper is clear and well-written. I can follow and understand the ideas most of the time. Notations are consistent and correct. \n\n======== After rebuttal ========\n\nThank you for the detailed response. For my questions, I am happy with the response by the authors, and I am willing to increase my score from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the main review above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new measure to quantify transferability in multi-source transfer learning problems, which takes into account the similarity between domains, sample size of each domain and model complexity. The authors have not been successful in responding to all the concerns raised by the reviewers. mainly, two important concerns remain. one theoretical side, it would surely enhance the paper to have justification/proofs/lower bound results on the measure. if not, on the empirical side, the authors need to provide more evidence on success of the proposed measure. 
Decision: Accept (Poster)