A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning
Authors: Xinyi Tong
Xiangxiang Xu
Shao-Lun Huang
Lizhong Zheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transfer Learning
Transferability
Sample Complexity

Abstract: Current transfer learning algorithm designs mainly focus on the similarities between source and target tasks, while the impacts of the sample sizes of these tasks are often not sufficiently addressed. This paper proposes a mathematical framework for quantifying the transferability in multi-source transfer learning problems, with both the task similarities and the sample complexity of learning models taken into account. In particular, we consider the setup where the models learned from different tasks are linearly combined for learning the target task, and use the optimal combining coefficients to measure the transferability. Then, we demonstrate the analytical expression of this transferability measure, characterized by the sample sizes, model complexity, and the similarities between source and target tasks, which provides fundamental insights of the knowledge transferring mechanism and the guidance for algorithm designs. Furthermore, we apply our analyses for practical learning tasks, and establish a quantifiable transferability measure by exploiting a parameterized model. In addition, we develop an alternating iterative algorithm to implement our theoretical results for training deep neural networks in multi-source transfer learning tasks. Finally, experiments on image classification tasks show that our approach outperforms existing transfer learning algorithms in multi-source and few-shot scenarios.


A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning
Xinyi Tong
Xiangxiang Xu" Massachusetts Institute of Technology xuxx@mit.cm
Tsinghua-Berkee Shenzhen Institute Tsinghua University txy 10y118mmlls..mmmm tsinghua. edu. cn
Shao-Lun Huang!
Lizhong Zheng Massachusetts Institute of Technology lizhongêmit. edu
Tsinghua-Berkelor Shenzhen Institute Tsinghua University
shaolun. huangnss. tsinghua edu. cn
Abstract
Current transfer learning algorithm designs mainly focus on the similarities between source and target tasks, while the impacts of the sample sizes of these tasks are often sufficiently addressed. This paper proposes mathematical framework for quantifying the transferability in multi-source transfer learning problems, with both the task similarities and the sample complexity of learning models taken into account. In particular, we consider the setup where the models learned from different tasks are linearly combined for learning the target task, and use the optimal combining coefficients to measure the transferability. Then, we demonstrate the analytical expression of this transferability measure, characterized by the sample sizes, model complexity, and the similarities between source and target tasks, which provides fundamental insights of the knowledge transferring mechanism and the guidance for algorithm designs. Furthermore, learning tasks, and establish quantifiable transferability measure by exploiting a parameterized model. In addition, we develop an alternating iterative algorithm implement our theoretical results for training deep neural networks in multi mouti-source transfer learning tasks. Finally, experiments image classification tasks show that our approach outperforms existing transfer learning algorithms in multi-source and few-shot scenarios.
apply our analyses for practical
Introduction
Transfer learning is nowadays an active research area in machine learning focusing on solving target learning tasks by the knowledge of learnable source tasks. The transferability between source and target tasks the central topic in transfer learning for understanding the knowledge transferring mechanisms and the algorithm designs (I]. In general, the transferability can be affected by several factors, including: (i) the similarities between source tasks and the target task 2 (ii) the sample sizes of the tasks; and (iii) the complexity dimensionality of the machine learning model. Most of the existing transfer strategies are designed based on how similar the source and target tasks are [ without considering the impacts of the training sample sizes the complexity of the models.
Work performed at Tsinghua- Berkeay Shenzhen Institute Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In theoretical analyses 4 5 6 sample sizes and model complexity are often included in deriving upper bounds for the transferability or the performance of transfer learning algorithms. However, pointed out that such bounds derived under general learning settings are often relatively loose under numerical simulations 77, and hence the algorithms designed by directly applying theoretical results can hardly achieve satisfactory performance in practical applications. Thus, the gap between theory and practice opposes the fundamental understandings of transfer learning algorithms.
In this paper, we propose a mathematical framework to investigate the transferability in multi-source transfer learning problem, and establish a quantifiable transferability measure for practical learning tasks. Specifically, for given source tasks, we learn the target task by class of learning model which linearly combines the models learned from individual tasks by some designable coefficients. In addition, the performance of this combined model is measured by the empirical risk of only the resting data of the target task, considered as the testing loss. Then, we adopt the optimal combining coefficients that achieve the minimum testing loss as the transferability measure, which illustrates the contribution of each model in learning the target task, and effectively quantifies the knowledge transferable among different tasks.
In our development, we establish an analytical solution of the transferability measure, which jointly quantified by sample sizes, model complexity, and a similarity measure between source and target tasks. particular, we demonstrate that the transferability of a particular source task is typically proportional to the number of samples and the measure of similarity to the target task, and is inversely proportional to the model complexity. This coincides with the intuition that when more training samples are available for a source task that is highly similar to the target task, more knowledge will be transferable from the source task to the target task. On the other hand, when the model is very complicated or high-dimensional, it is typically harder to train the model well, and less knowledge can be acquired and transferred. More importantly, our theoretical results can be applied for designing effective and efficient algorithms for real transfer learning problems, which are especially useful for multi-source transfer learning with large number of source tasks that are generally difficult to deal with.
The contribution of this paper can be summarized follows:
We propose a mathematical framework for transfer learning analyses, and establish transferability measure on discrete data, quantified by the number of samples, the complexity of the model, and the x--distance between source and target tasks.
We extend the transferability analyses to the continuous data, and establish a similar transfer- ability measure that can be evaluated in practical tasks, by exploiting paramerized models.
We apply our theoretical results to develop an iterative algorithm for training deep neural networks in general supervised transfer learning scenarios. Morcover, our algorithm can be practically applied for multi-source transfer learning.
The experiments in real datasets validate our proposed algorithm, in which we show that our approach outperforms many existing transfer learning algorithms.
Due the space limitations, the proofs of theorems and propositions are presented in the supplemental material.
Problem Formulation and Analysis
Let X and Y be the random variables denoting the data and label with domains and y, respectively, and let P denote the set of all distributions on X y. For the convenience of illustration, here we assume to be discrete, and will extend our analyses to continuous cases later. Throughout our analyses, we will use Ar A ((on....,a)) Σi ds 1, > 0, = O,...,k} to denote the k-dimensional simplex.
2.1 Single-Source Transfer Learning
To begin, we consider the transfer learning setting with one source task and one target task, denoted as task 1 and 0, respectively. Specifically, for each task i 0, 1, we assume that n, training
2

---

(0) )2 are i.i.d. generated from some underlying joint distribution XY E P with samples {(x]] :2
(x, y) > 0, for all 2, y, and the empirical distributions
((s)
XY E P of the samples are defined as
P (2,
(9) ( = I, I, = y).
The
where 11 {.) denotes the indicator function 8 Then, the empirical distributions P and PXY can be regarded the models learned from the target task and the source task, respectively, when all the entries of the mass functions are required to determine.
To develop the transferability measure, our proposed framework focuses convex combination of both learned model
Qey an) vx og ((0) (x,y) y) + PCL (I, v). for all (xr, v) E x y,
(2)
where (ag, a1) E A1 are parameters to be designed. Notice that these parameters characterize the knowledge transferred from the source task to target task, and the designing of these parameters will be affected by the sample sizes and the task similarities, which essentially leads to a transferability measure adjusted by the sample complexity.
Then, the performance of the model (a is evaluated by the testing loss, measured by its empirical risk on the testing data of target task. Conventionally, such empirical risk is often computed by the logarithm loss. However, the logarithm risk can ill-defined in our setting. Therefore, alternatively apply the referenced distancce the measure, defined as follows.
Definition 1. Given a reference distribution Rxy, for any distribution Pxy and Qxy, the referenced disttancce between them is defined as
(Pxy(x.u) - Qxy(x,)))
XRx (Pxy,Qxx)
Σ
Rxy (2,
TEX.NEY
Specifically, we denote (Pxx, XPxy (Pxy,Qxy), which corresponds the Pearson x" diverrngee
We choose the underlying target distribution Pxy as the reference, and define the testing loss as the averaged Pearson x² 2 --divergence
L
(ao.as) test
&
E
x2
XY (a a)
))
(3)
where the expectation is taken over all i.d.d samples generated from the source and target distributions. Moreover, define the optimal coefficients
(a a;j) A arg min
Liest
(1g.01)
(4)
(ap,an
as our transferability measure, which effectively quantifies the contributions of the source and target tasks in obtaining the optimal performance.
Then, we have the following characterization.
Theorem 2. The testing loss as defined B is
L L (a0.as)
a²x² (PSt), p(t
ag (()
aj
(5)
Ykn
'The assumption positive entries is without loss of generality, since in practice such joint distributions typically modeled by some positive parameterized families, e.g., the sofumax function.
*We shall emphasize that such combination forms are naturally led by the transfer learning model proposed by
4
Section l] from optimizing convex combination of Log-Loss, where we refer to Section A of supplementary
material for detailed discussion.
'Nte that when some (x, y) pair is missing in training samples, have (00):01) (x, v) 0 while X
XY (a. y) 0, which would lead to infinite logarithm risk.

---

Xy, and the corresponding empirical and the transferabilitty measures as defined in are
Rg (
0j (P P ( Lva
and og aj:
(6)
m1
where, for each i 0, V(i) defined as
P (x,
(1 PXr (2,
V A
Σ
PX (2,
TET
From (6). and the fact that V | - 1, the transferabilityy is determined by three key factors: (i)
the similarity between source and target tasks, measured by the <-viieereence x² ( P (ii)
the sample sizes no and n for source and target tasks; and (iii) the model complexity, characterized by the number of model parameter ([C)) | 1) in (0) K
Current transfer learning algorithm designs often focus on the similarities between source and target tasks, while the sample sizes and model complexity are often not sufficiently addressed. In Theorem 2 show that the transferability is in fact proportional the number of model parameters, and is inversely proportional to the number of samples source tasks and the similarity between source and target tasks. Therefore, for source task with a complex model few training samples, even though it is similar to the target task, the knowledge transferable from this source task can still be very limited. Such insight was not well captured in many existing transfer learning algorithms, and our result essentially provides the optimal characterization of the task transferability adjusted by the sample complexity in transfer learning.
The established transferability measure is also related to the optimal bias variance trade-off 9 of this transfer learning problem. Indeed, note that the bias-variance trade-off in testing loss (5) is tuned by og and (1, as
(10)01) test
oîp² (PXY. PX
V
v
(8)
70
71
bias term
variance term(s)
where the bias term does not decay with the sample sizes no. A1, while the variance terms vanish with sufficient samples. Then, the transfcrability measure corresponds to the coefficients ag of that achieve the optimal bias-variance trade-off, such that the testing loss is minimized.
2.2 Multi-source Transfer Learning
Theorem 2 can be readily generalized to multi-source transfer learning problems. Specifically, suppose that there are & source tasks, referred to as task i, for = 1,... k, and target task, referred use (+() xY, { (x((") Me
(i) ) and xx to denote
as task O. Similarly, for each task O.... ,
the underlying distribution, i.i.d. samples generated from ((i) distribution as defined in (T. respectively.
Similar to we consider the convex combination of the models learned from different tasks
Q
OXY
5(0) XY
E Ar.
==0
Then, we define the testing loss (a) and the corresponding transferability measure a", as [cf.
(()) A E x2 (P 08)) and a f arg min T.(a)
(10)
aε
Similar to Theorem we have the following result for multi-source transfer learning.
'When evaluating (1) this quantity is related to the source distribution. From this perspective, the model complexity reflects how hard the task is.

---

[(a)
g
g
Input
(a) Feature Extractor
(b) Classitier
Figure 1: A pre-trained neural network for classification can be divided into (a) feature extractor
which generates feature f(x) = (z), ... fa(x)]]* Rd and (b) classifier with the weights g- With f fixed, our framework optimizes the weights g in the topmost layer for each task, to obtain the corresponding parameterized representation.
Theorem 3. For the model ( the testing loss under the target task is
(p.)) Σ as p(1)
I [
x2
(0
(11)
n2
where 's)'s are as defined in for all i.
From Theorem 3 the transferability measure a defined in (TO) can be computed by solving non-negative quadratic programming problem [OO]. Similar to the discussions in Section [21] such
transferability measure quantífies the knowledge transferable from different source tasks the target task with the sample complexity being considered.
3 Parametric Models and Transfer Learning Algorithm
3.1 Transferability Measure with Pre-trained Neural Network
This section extends the analyses in the discrete data domain to continuous data in practical problems. In such cases, the previously adopted learning model (I) has infinite parameters due to the infinite cardinality |30, and thus not be effectively represented. In order apply the previous analyzing framework, we first propose a parameterized representation for modeling features of the continuous data by exploiting pre-trained model.
As shown in Figure | a pre-trained network can be divided into two parts: (a) the previous layers for extracting d-dimensional features f(x) = [fi (z), fa(z)]" from the data variable and (b) the topmost layer for linear classification, with weights g = [g1 (v), ... (r()) indexed by label y. When the feature fax) is given and fixed, the models learned from different tasks can be effectively represented by a finite collection of parameters, i.e., ((1))...gg(())...
In particular, our framework considers the discriminative model in the factorization form
p(f.a)
Y|x (y|z) A P( (v) (1 f 1 (z)g(v))
(12)
which is similar to the ones introduced in factorization machines [I] and natural language processing applications [2]. Then, for each task = O..... k. we learn corresponding weights g., such that the ()) fits the training sample The weight ğ: can be formally defined as
learned model YIX
(P ((0) p(f.g)) gi arg min XRX xY, YIX
(13)
"The approach of retraining (fine-tune) tuning) the topmost layer is sometimes referred to as the retrain-head method [13]. which has also been widely adopted in transfer learning applications.
5

---

where the fitness is measured as the referenced x² -iitaance [cf. Definition between the empirical distribution XY and the joint distribution (0) p(T.9) Y|X For convenience, we adopt unified
reference Rxy - PX Py in fitting different tasks.
From Px P().)) plays the role in the continuous case corresponding P(0 in the discrete
case. This allows us to apply previous analyses and focus on the discriminative model p(f.0)) Analogous to (9. we consider the convex combination of these discriminative models
FYxx
(a) yix
a
p( p(f.9)
(14)
with 9 - E
Li=o a,g, Then, we define the testing loss and corresponding transferability measure as [ef. (TOM]
L
test
E
XRx
PS x and a"
- arg min L L
(15)
aEAc
for which we have the following characterization.
Theorem 4. The testing loss
15
associated with the model (1) is
I XRxY
p(0)
pl) YIX
Σ
a.
(0) p(..9.) YIX
a
v() + xRxx
(Phr: ((0)
p(t)
p(S. YIX
n,
(16)
where 9: - arg ming Xaxx ( characterized in the supplementary material [ef. (32)).
Px
p(o) p(f-s)), and where V(6) is a constant independent of a YIX
Moreover, note that from the definition of gi the joint distribution p00) p(f-g.) VIX can be interpreted
as a projection of p(t) XY onto the distribution family x p(5.9) YIX y with referenced 2 distance used the distance measure. Therefore, the terms of (I6) share similar interpretations as their counterparts in Theorem with the distances measured in the projected space. Again, a can be efficiently computed by solving non-negative quadratic programming problem.
3.2 Multi-source Transfer Learning Algorithm
With our theoretic analyses in Theorem 4
develop knowledge transfer algorithm for multi- source transfer learning. Different from the previous analyses where f is fixed, our algorithm jointly optimizes the extracted feature f. the weights g, together with the combining coefficients obtain better performance.
To begin, for given f. and a. we introduce the loss function
k
((a Σ aixaxr (PSr, p (()p(()))
(17)
The following result illustrates that, the 9 can be computed via directly minimizing this loss, without evaluating each 9i individually.
Proposition 5. The 9 as defined satisfies
9 arg min I (aa.f''
*The joint distribution ((() (5 is defined as [P(C] (D) p(r-9) (x, v) 2 p(o) (x) (vl=), for all (z,y). YIX Note that when discriminative model p( of the target distribution P())
fixed, p(o) p( YIX
corresponds to the optimal approximation

---

Table 1:
Test accuracies (%) the target task, with the network trained on samples from single source. All reported accuracies are averaged over repeated experiments.
Source Task
1
2
3
4
Acc. on the target task
66.5
59.7
56.2
77.1
Table 2:
Test accuracies (%) on the target task, compared with the combining coefficients a deter- mined by 20 rounds of random searches (RS).
Target Sample Size
20
100
Acc. with only target samples
70.9
74.4
81.5
Average acc. by RS
67.8
73.9
75.4
Highest acc. by 20 RS
74.4
78.0
80.8
Acc. by Algorithm !
78.9
81.2
83.7
Then, with training samples from different tasks, our algorithm alternates between two different kinds of optimizations: (i) the optimization of a for given g) to minimize the testing loss L as defined in (T6), via solving non-negative quadratic programming problem; and (ii) the optimization of (f, g) for given a to minimize the loss L (a.f.)) as defined in (17) via training the neural network. We summarize the procedures as Algorithm
Specifically, can shown that both the testing loss T_(a) and the loss f can be represented by some expectations of features f and g. In computing these losses, these expectations approximated by corresponding empirical means, with details provided the supplementary material.
Algorithm 1 Multi- Source Knowledge Transfer Algorithm
1: Input: target and source data samples {<x)", .2 ( (i -...... .k)
2: Randomly initialize a²
3: repeat
4: (f",g") + arg min 9.9 Lia' .f))
5: a" <- arg min, [(a)
6: until a" converges
7:
+ arg min f.g ((aa ffg)
8:
return f"
With the fr and g" computed by the algorithm, for newly observed target sample the predicted label ý is given by the MAP (maximum a posterior) decision rule
((x)
arg max
p(f g¹) (y/zz) x
arg max pS (v) (1+ (TT
(18)
yEy
DEy
4 Experiments
To validate the effectiveness of our algorithms multi-source learning and few-shot transfer learning scenarios, conduct series of experiments common datasets for image recognition, including CIFAR-10 [14], Office-31 and Office-Caltech [15). In all experiments, the g in the classifier is simply generated by an embedding layer.
4.1 Multi-source Transfer Learning
We conduct multi-source transfer learning experiments on CIFAR-1O, which contains 000 training images and 000 testing images in 10 classes. To begin, construct the source tasks and target task by dividing the original CIFAR-16 10 dataset into five disjoint subdatasets, each containing two classes of the original data, which corresponds to a binary classification task. Then, choose as our target task (task O), and use the other four as source tasks for transferring knowledge, referred to task 1, 2, 3, 4.
Moreover, for each source task, 2000 images are used for training, with 1000 images per binary class, and set target sample size ng to no 20, 100, respectively. Throughout this experiment,

---

Table 3: Test accuracies for target tasks under different transfer settings (source - target) Office-31
Method
A++D
A
D-W
D->A
W-+A
W-D
SDT 20
86.1
82.7
95.7
66.2
65.0
97.6
DAMA 21
86.3
84.5
95.5
66.5
65.7
97.5
FADA [22
88.2
88.1
96,4
68.1
71.1
97.5
UDDA 2X
89.0
88.2
96.4
71.8
72.1
97.6
Ours
90.0
87.3
96.5
72.4
72.1
97.2
Table 4: Test accuracies for target tasks under different transfer settings Office-Caltech
Method
A-C
W-C
D-C
C-A
C-#
C-+
GFK LS
68.4
68.4
64.5
83.8
78.7
74.6
TLDA
24
76.1
71.0
65.4
84.2
85.2
78.9
DTML
25
72.0
71.6
67.1
86.0
85.0
79.6
CPNN
78.5
73.5
68.0
86.3
86.2
80.1
Ours
80.3
72.9
72.2
88.4
85.9
83.5
the feature f is of dimensionality d = 10, generated by GoogLeNet [16]. followed by two fully connected layers for further dimension reduction.
Unlike common transfer learning settings where the labels for the source and target tasks are closely related, here the binary labels for these 5 sub-datasets are in general irrelevant. Therefore, first establish the correspondences between labels as follows. For each given source task, we first train the network on its training samples, while the accuracy is evaluated on the samples from the target task. Then, we flip the original binary label for this source, if the test accuracy is less than 50%. The resulting accuracies on the target are summarized Table T
In our implementation of Algorithm I we use the CVXPY 17 18 package for solving the non- negative quadratic programming in line 5. In addition, the alternating iteration is stopped when the element-wise differences for ax computed in two successive iterations are at most 0.05.
Then, the test accuracies of our algorithm on the target set are shown in Table 2 where we have compared our performance with random search (RS) strategy. Specifically, in the RS strategy, we generate the coefficients from the log-uniform distribution [19] in [0.001, 1]. for 20 rounds. The results indicate that our approach outperforms the random search method. Also, the difference in Table 2 and Table also shows the performance gain of multi-source transfer learning over single-source.
4.2 Few-shot Transfer Learning
To validate the effectiveness of algorithm for few-shot learning tasks, conduct experiments Caltech-31 and Office-Caltech datasets. These datasets provide typical transfer learning tasks with few available training samples, where the influence of sample complexity is shown.
4.2.1 Caltech-31
Caltech-31 dataset contains images of 31 categories, which come from 3 sub-datasets: Amazon (2817 images), Dsir (498 images), and Webcam (795 images). Then, different transfer settings among these sub-datasets are denoted by the "source target",, as: A-DD A->,, D-W, D-*, W-A, and W-DD, We adopt the few-shot transfer learning setting [20), illustrated as follows. Specifically, 3 target samples per category are used for training, and the training sample size (per category) for source task is 20 8, depending on whether the source task Amazon not. Moreover, also adopt five train-test splits introduced in 20
In experiment, the feature is 64-dimensional vector, extracted by VGG-16 (77 network pre- pre-trained on the ImageNet, succeeded by two fully connected layers for dimension reduction.
Table 3 summarizes test accuracies for target tasks under different transfer settings, where all reported accuracies are averaged over five train-test splits. The results indicate that our algorithm generally outperforms existing few-shot transfer learning methods.

---

100
Coefficient oz Accuracy
0.6
Value of on Accuracy
0.4
0.0
2
6
8
10
lteration
Figure 2: The combining coefficient on of the task ADD and test accuracies under testing samples of Dslr during iterations.
In addition, we also investigate the convergence of the coefficient a" in Algorithm As an example, for the A+D task, the changes of the coefficient α (the coefficient of the source loss) and the accuracy on the Dsir test dataset during iterations are shown in Figure 2 From the figure, the value of on converges under our stopping criterion, where the optimal testing accuracy is obtained.
4.2.2 Office-Caltech
Office- CalteCh dataset is composed of 10 common categories in Office-31 and Caltech-256, divided four sub-datasets: Amazon (958 images), Caltech (1123 images), Webcam (295 images), and DsIr (157 images). We focus on the 6 transfer settings depending on C, i.e. A-C, W-C, D-C, C-A, C+W, and C-+D, which have few common categories between source and target tasks.
In addition, we follow the setting introduced in [5] sionality d = 10, based on the DeCAF feature 28 reduction.
for train-test split. The feature f is dimen- with 2 fully connected layers for dimension
29
Table shows the performance for our algorithm, in comparison with several sepi-supe and few-shot domain adaptation algorithms. is worth mentioning that, though our approach does not use the unlabeled data samples in training, it provides competitive performance the semi-supervised algorithm CPNN [26), and can be better on specific tasks.
5 Related Work
Theoretical Analyses of Transfer Learning and Transferability. Most of the theoretical works about transfer learning focus on deriving upper bounds for the transferability or the performance of transfer learning. For example, the generalization сттоr be bounded by the VC-dimension of
the hypothesis space 33), the total variance distance [4] or the mutual information between training
samples and outputs (5). and the Censen-Shanoor distance between domains (6]. The choices of measures are determined mostly by the problem settings. Furthermore, some of these different measures and the x2 diverrege used in work closely related, which generalized as f-divergence [31]. However, there can exist a significant gap between the theoretical bounds and the performance for real tasks 7
There are also works concentrating on defining a transferability measurement in an empirical way. For instance, the empirical log- liceeiood the target data under the network trained by source samples can measure how much the source samples would help improve the target task [33 32). Compared with these works, establish transterability measure and provide an analytical expression, for guiding algorithm designs.
Transfer Learning Algorithms. Transfer learning algorithms based on the insights from theoretical works intuitively measure similarities between different domains. The similarity measures include the low-rank common information [3 K-L. divergence (34
35
36
37 la-distance 38 39 40)
and Wasserstein distance 4T). Additionally, transfer learning problems also share the similar framework with meta- meta-learning, which concentrates obtaining generalized model for different

---

tasks, especially when lacking enough samples for all categories 42 works, our algorithm takes the sample sizes and model complexity into consideration, and can be more applicable for general learning tasks, including the few-shot setting.
In comparison with the above
Multi-source Domain Adaptation and Few-shot Domain Adaption. Multi-source domain adapta- tion considers the approaches of combining multiple tasks together. Conventional methods mainly include instance weighting [43 and domain weighting [4 [15, which re-weight samples or loss functions in training, respectively. Based on deep learning, cutting-edge algorithms attempt to maximize the domain confusion
46 or learn the domain-invarte invariian representations [47].
Semi-supeervied and few-shot domain adaptation focus on transfer learning algorithms under few labeled target samples, which is of our work's application scenarios. Compared with conventional domain adaptation, this field pays attention to embedding samples into an intrinsic low-dimensional subspace [1]. Common algorithms aim at learning domain-invartie representations, including
simultaneous deep transfer (SDT) method [20] and semantic alignment method [22 23
Compared with these empirical studies, our characterization provides practical learning algorithm under theoretical guarantees.
Conclusion
This paper introduces a mathematical framework for quantifying the transferability in multi-source transfer learning problems. Our characterization reveals the essential roles of sample sizes and model complexity in knowledge transferring, which demonstrates potentials in establishing a unified understanding of various transfer learning algorithms. In addition, we develop multi multi-source transfer learning algorithm based the theoretical analyses. Experiments on practical multi-source learning tasks show the effectiveness of our proposed algorithm.
Acknowledgements
The research of Shao-Lun Huang is supported in part by the National Natural Science Foundation of China under Grant 61 807021, and the Shenzhen Science and Technology Program under Grant KQTD201 708101 50821 146.
The research of Lizhong Zheng is supported in part by the National Science Foundation (NSF) under Award CNS-2002908 and the Office of Naval Research (ONR) under grant N00014-19-1-2621.
References
[1] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions on knowledge and data engineering, 22(10):1345-1335 2009.
[2] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270-279. Springer, 2018.
[3] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27:2672-2680, 2014.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151-175, 2010.
[5] Y. Bu, S. Zou, and V. V. Veeravalli. Tightening mutual information based bounds on gener- alization error. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 587-591, 2019.
[6] Han Zhao, Remi Tacher Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019,
10

---

[7] Xuetong Wu, Jonathan H. Manton. Uwe Aickelin, and Jingge Zhu. Information-theow analysis for transfer learning. 2020 IEEE International Symposium on Information Theory (ISTT), Jun 2020.
[8] Willliam Feller. An introduction to probability theory and its applications, vol John Wiley Sons, 2008.
[9] Richard Duda, Peter E Hart, et al. Puttern classification and scene analysis, volume Wiley New York, 1973.
[10] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[11] Steffen Rendle. Factorization machines. In 2010 IEEE International Conference on Data Mining, pages 995-1000, IEEE, 2010.
[12] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27:2177-2185, 2014.
[13] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR, 2020.
[14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[15] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066-2073 IEEE, 2012.
[16] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[17] Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
[18] Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision, 5(1):42-60, 2018.
[19] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10)281-335, 2012.
[20] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4068-400000 40764 2015.
[21] Piotr Koniusz, Yusuf Tas, and Fatih Porikli. Domain adaptation by mixture of alignments of second-or higher-order scatter tensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4478 088-4487, 2017.
[22] Saeid Motian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In Advances in neural information processing systems, pages 6670-6688) 2017.
[23] Sacid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. Proceedings of the IEEE International Conference Computer Vision, pages 5715-5725, 2017.
[24] Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Deep transfer metric learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 325-333, 2015.
11

---

[25] Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. Supervised represen- tation learning: Transfer learning with deep autoencoders. In Twenty-Found Fourth International Joint Conference Artificial Intelligence, 2015.
[26] Z. Ding, N. M. Nasrabadi, and Y. Fu. Semi-supervised deep domain adaptation via coupled neural networks. IEEE Transactions on Image Processing, 27(11):5214-5222 -2224444,, 2018.
[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[28] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffmman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647 2014.
[29] Judy Hoffman, Erik Rodner, Jeff Donahue, Trevor Darrell, and Kate Saenko. Efficient learning of domain-invariant image representations. arXiv preprint arXiv:1301.322 2013.
[30] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pages 129-136, 2008
[31] Peter Harremoës and Igor Vajda. On pairs of f-divergences and their joint range. IEEE Transactions on Information Theory, 57(6):3230-322 -2225, 2011.
[32] Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 1395-1405, 2019.
[33] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213-226. Springer, 2010.
[34] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: /4/2.3444 2014.
[35] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[36] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarte training of neural networks. Journal of Machine Leurning Research, 17(59):1-35, 2016.
[37] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176. 2017.
[38] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip Yu. Transfer joint matching for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410-1417, 2014.
[39] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pages 97-105. PMLR, 2015.
[40] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in neural information processing systems, pages 136-144, 2016.
[41] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[42] Ruohan Wang, Yiannis Demiris, and Carlo Ciliberto. Structured prediction for conditional meta-learning. In Advances Neural Information Processing Systems, volume 33, pages 2587-2598, 2020.
12

---

[43] Qian Sun, Rita Chattopadhyay, Sethuraman Panchanathan, and Jieping Ye. A two-stage weighting framework for multi-souree domain adaptation. Advances in neural information processing systems, 24:505-513, 2011.
[44] Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Domain adaptation from multiple sources: domain-dependent regularization approach. IEEE Transactions on neural networks and learning systems. 23(3):504-5) 518, 2012.
[45] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3964-3973, 2018.
[46] Han Zhao, Shanghang Zhang, Guanhang Wu, Geoffrey Gordon, et Multiple source domain adaptation with adversarial learning. 2018.
[47] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specifer distribution and classifier for cross-domain classification from multiple sources. Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 5989-5996, 2019.
[48] Xiangxiang Xu and Shao-Lun Huang. Maximal correlation regression. IEEE Access, 8:26591- 26601, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section T
(b) Did you describe the limitations your work? [Yes] We clearly state the assumptions we use in Section
(c) Did you discuss any potential negative societal impacts of your work? [No] We have
done theoretic work that has few societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary material.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] We think the error bars are not related to the core result of our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We use few computation resources in our work.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
We don't use new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "k54P4Tp3AX", "reviewer": "Reviewer_thNZ", "summary": "The paper introduces a transferability measure that takes into account the task similarity as well as the sample size and model complexity. The theoretical formulation is then applied to develop an iterative algorithm to train deep neural networks in a supervised transfer learning setting. It is further shown that the algorithm can be applied to multi-source transfer.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe proposed mathematical formulation is novel as it takes into account the sample size and model complexity on top of task similarity which is lacking in previous methods. However, the relation between recently introduced transferability measures e.g. LEEP (Nguyen et al. ICML 2020), NCE (Tran et al. ICCV 2019), and their derivatives are neither discussed nor cited. The results are compared to domain adaptation methods making it unclear whether this work is more suitable as a domain adaptation paper or transferability.  \n\n**Quality**\nI appreciate the author's effort in deriving a mathematical formulation of transferability that takes into account sample size and model complexity. However, the experiments performed in the paper do not seem sufficient due to the following reasons:\n\n1. The method introduces a transferability measure. Therefore ideally it should be compared to methods such as LEEP and more recent state-of-the-art transferability measures. In case, the proposed method is not suitable to be compared with these methods, this should be clearly explained in the paper.\n2. In the current results, the datasets used to evaluate the performance are more suitable for domain adaptation. A more recent standard benchmark VTAB (Zhai et al. arxiv 2019) might be more suitable or the ones used in LEEP, NCE papers making the results of the current method comparable to state-of-the-art.\n3. Even in the domain adaptation benchmark the methods compared are not current state-of-the-art e.g. in Office Caltech dataset Wang et al. AAAI 20 is current state of the art. IT is not necessary to beat state-of-the-art methods but at least these methods should be mentioned and the relation between these methods to the current method should be discussed.\n\n**Clarity**\nThe mathematical formulation is described well and augmented with the algorithm. Although some parts of the description may be improved e.g. in line 109 it is not clear why that term accounts for model parameters.  \n\nThe related literature section lacks citations to multiple relevant papers as mentioned above making it difficult to assess the contribution of the current paper when put in context with recent papers on transferability measures. \n\n**Significance**\nTheoretical estimates of transferability are crucial in order to assess the application of off-the-shelf models in real-world scenarios. In this aspect, the paper does a good job in obtaining an estimate that takes into account the sample size of source and target tasks which is a crucial factor.\n\nHowever, the experiments performed in the current version do not provide sufficient evidence of the applicability of the proposed measure. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "W6jDOW-vyAE", "reviewer": "Reviewer_HA1J", "summary": "The authors propose a mathematical analysis aiming at quantifying the transferability in Transfer Learning.  More specifically, they consider a transferability measure and they show that such a quantity is characterized  by the number of samples, the complexity or dimensionality of the machine learning model and the similarity (according to the \\Chi-squared distance) between source and target tasks (see Thm. 2 and Thm. 3). For given source tasks, the authors learn the target task by a class of learning model which linearly combines the models learned from individual tasks by some designable coefficients. Then, they use the optimal combining coefficients achieving the minimum testing loss on the target task as the transferability measure. By doing this, they manage to illustrate the contribution of each model in learning the target task, and effectively quantify the knowledge transferable among different tasks. They start from discrete data setting and then, they extend and adapt the analysis to the continuous data setting by introducing a related transferability measure that can be computed by solving a non-negative quadratic programming problem. Exploiting this, the authors develop an alternating iterative algorithm for training deep neural networks in standard supervised transfer learning settings. The numerical experiments on real image classifications tasks show the effectiveness of the proposed method in comparison to many existing state-of-the-art algorithms in multi-source and few-shot settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose an interesting and novel analysis on a topic that could provide new insights on the design of new Transfer Learning algorithms. In particular, they manage to show the natural finding according to which the transferability of a particular source task is typically proportional to the number of samples and the measure of similarity to the target task and it is inversely proportional to the model complexity.\n\nThe paper is quite well written.\n\nIn lines 28-33, the authors say that “In theoretical analyses [4, 5, 6], sample sizes and model complexity are often included in deriving upper bounds for the transferability or the performance of transfer learning algorithms. However, it is pointed out that such bounds derived under general learning settings are often relatively loose [7], and hence the algorithms designed by directly applying theoretical results can hardly achieve satisfactory performance in practical applications.” Explain to the reader why such bounds are loose. This point is important for understanding the contribution of the paper.\n\nExplain more the intuition of Definition 1. Which is the relation between the similarity measured the authors consider in such a definition and those usually used in standard multi-task and meta-learning framework, such as the variance among the true risk minimizers of the tasks or the complexity of the shared common representation? Is it possible to extend the analysis to more general types of similarity measures, maybe non-heterogeneous measures such as those considered in [1,2,3] below?\n\nWhich is the difference between multi-source transfer learning and online meta-learning (see e.g. [4,5,6,7] below) in which we interpret the source tasks as training tasks? Discuss this point and associated literature.\n\nAdd a sketch of the proof of the main statements could help the reader in understanding the main tools used during the proofs.\n\n1.   A structured prediction approach for conditional meta-learning, Wang et al. 2020\n2.   Conditional Meta-Learning of Linear Representations, Denevi et al. 2021\n3.   The Advantage of Conditional Meta-Learning for Biased Regularization and Fine Tuning, Denevi et al. 2020.\n4.   Provable Guarantees for Gradient-Based Meta-Learning, Khodak et al. 2019\n5.   Online-Within-Online Meta-Learning, Denevi et al. 2019\n6.   Learning-to-Learn Stochastic Gradient Descent with Biased Regularization, Denevi et al. 2019.\n7.   Adaptive gradient-based meta-learning methods, Khodak et al. 2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "no ethical issue in my opinion", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48"}, "extra_text": {}}, {"review_id": "UXIYg0lEBXf", "reviewer": "Reviewer_9X81", "summary": "In this paper, the authors propose a mathematical framework for quantifying the transferability in multi-source transfer learning problem. Specifically, the target model is considered to be a linear combination of the source model, and the combination coefficients is used to denote the transferability. Analyses further show that the transferability is characterized by the sample size, model complexity, and source-target similarity. A parameterized model is then proposed to achieve the quantifiable transferability measure. Experiments on image classification task are done to verify the effectiveness of the proposed algorithms. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are the comments:\n\nPros:\n\n(1)\tA mathematical framework that establishes a transferability measure quantified by sample size, model complexity, and source-target X^2-distance. \n\n(2)\tA parameterized model based on the theoretical framework, and an iterative algorithm for training deep neural network.\n\n(3)\tThe experiments show the effectiveness of the proposed algorithm in some extent. \n\nCons:\n\n(1)\tThe analyses are limited to a stacking-based target model and the proposed X^2-distance, which constrains its generality. The problem setting studied requires the target labelled data. It is more interesting to have discussions on the popular unsupervised domain adaptation setting where the target labels are not available. Moreover, [ref1] studies the similar problem setting (multi-source setting with limited target labelled data) and also proposes a stacking-based target model with the base model coefficients as the domain similarity. It is necessary to discuss with this work. \n\n[ref1] Source-target similarity modelings for multi-source transfer gaussian process regression. ICML. 2017.\n\n(2)\tThe experiments are done on the classification tasks. Is the analysis applicable to the transfer regression problem? \n\n(3)\tRegarding Eq.(12), is it only for the target task (as the P_Y^{(0)}(y) is used)? What is the dimensionality of \\mathbf{g}_i? How to obtain P_{XY}^{(0)} and P_{X}^{(0)} in Eq. (15)?\n\n(4)\tThe improvements of the proposed method compared with the other baselines are not significant as shown in Table 3 and Table 4. Note that some baselines are unsupervised domain adaptation method, which cannot leverage the target labels. \n\n(5)\tThe empirical evaluation will be more convincing if promising results are obtained on the more challenging dataset, e.g., office-home dataset.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "An explicity limitation section is encrouged. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "31upbgv1Hk", "reviewer": "Reviewer_CHLY", "summary": "The paper aims to quantify transferability in multi-task in multi-source transfer learning in terms of model complexity, sample size, and similarity of the tasks. The analysis is done for the discrete case and later generalized for a specific model with a fixed embedding structure and a particular readout layer that resembles the factorization machines. The results are validated on a number of transfer learning datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Developing a measure of transferability is an important problem for understanding the dynamics of transfer learning. However, the claims made in the paper are not consistent with the analysis. The paper claims quantifying a notion of transferability in terms of the model complexity and sample size. In statistics and ML, model complexity is generally shown in terms of standard quantities such as Rademacher complexity or VC dimension. Also, sample complexity generally imposes a lower-bound on the number of examples that a certain algorithm requires to learn a particular task. Instead, the provided analysis only shows the optimal loss in terms of the discretized probability densities of the tasks. This quantity is not related to the complexity of the model that is used for transfer learning nor the number of training samples. Therefore, the results do not provide any insights on transferability other than how much the densities of the two tasks are similar.\n\nOn the experimental side, it is hard to tell whether the small improvements are significant, given that no error margins are provided. \n\nIn summary, the measure of transferability that is developed in the paper is not related to model complexity or sample size. It simply quantifies how close the discretized distributions of two given tasks are to each other. On the experimental side, it is unclear whether this quantity provides any significant improvements. \n\nMinor: the analysis for the binary case is redundant and included in the multi-task setup. This part can be removed without hurting the construction.\n\n=========================================================\n\nAfter rebuttal:\nThe authors have addressed most of my questions/concerns regarding the technical details of the paper. The construction in the paper is interesting and non-trivial, especially for the continuous case. Therefore, I would like to raise my score to 7 and vote for acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work, a novel mathematical formulation is proposed for the problem of multi-source transfer learning. The theoretical analysis is interesting. Empirical evaluation is good but can be more convincing by conducting experiments on more challenging datasets.

In the rebuttal, most of the concerns raised by reviewers have been well addressed. Though one reviewer sticks to his originally negative rating of 5, I found that the authors have given clear explanations in their responses. Therefore, I recommend an acceptance for this work.

However, the authors are encouraged to make the descriptions clearer in the revision based on reviewers' comments as well as the authors' responses in the rebuttal.
Decision: Accept (Poster)