Transfer Learning with Dynamic Distribution Adaptation
JINDONG WANG, Microsoft Research Asia, China
YIQIANG CHEN and WENJIE FENG, Institute of Computing Technology,
Chinese Academy of Sciences, China
HAN YU, School of Computer Science and Engineering, Nanyang Technological University, Singapore
MEIYU HUANG, Qian Xuesen Laboratory of Space Technology, China Academyof Space Technology, China
QIANG YANG, Department of Computer Science and Engineering, Hong Kong University of Science andTechnology, Hong Kong
Transfer learning aims to learn robust classifiers for the target domain by leveraging knowledge from asource domain. Since the source and the target domains are usually from different distributions, existing
methods mainly focus on adapting the cross-domain marginal or conditional distributions. However, in realapplications, the marginal and conditional distributions usually have different contributions to the domain
discrepancy. Existing methods fail to quantitatively evaluate the different importance of these two distributions, which will result in unsatisfactory transfer performance. In this article, we propose a novel concept
called Dynamic Distribution Adaptation (DDA), which is capable of quantitatively evaluating the relative
importance of each distribution. DDA can be easily incorporated into the framework of structural risk minimization to solve transfer learning problems. On the basis of DDA, we propose two novel learning algorithms:(1) Manifold Dynamic Distribution Adaptation (MDDA) for traditional transfer learning, and (2) Dynamic Distribution Adaptation Network (DDAN) for deep transfer learning. Extensive experiments demonstrate that
MDDA and DDAN significantly improve the transfer learning performance and set up a strong baseline overthe latest deep and adversarial methods on digits recognition, sentiment analysis, and image classification.
More importantly, it is shown that marginal and conditional distributions have different contributions to thedomain divergence, and our DDA is able to provide good quantitative evaluation of their relative importance,
which leads to better performance. We believe this observation can be helpful for future research in transfer
learning.
CCS Concepts: • Computing methodologies → Transfer learning; Learning under covariate shift;
Dimensionality reduction and manifold learning;This work is supported in part by National Key R & D Program of China (2016YFB1001200), NSFC (61572471, 61972383), thesupport of grants from Hong Kong CERG projects 16209715 and 16244616, Nanyang Technological University, Nanyang
Assistant Professorship (NAP), and Beijing Municipal Science & Technology Commission (Z171100000117017).
Authors’ addresses: J. Wang, Microsoft Research Asia, No. 5 Danling Street, Beijing, 100080, China; email: jindong.
wang@microsoft.com; Y. Chen (corresponding author) and W. Feng, Institute of Computing Technology, Chinese Academy of Sciences, No. 6 Kexueyuan South Road, Beijing, 100190, China; emails: {yqchen, fengwenjie}@ict.ac.cn; H. Yu,
School of Computer Science and Engineering, Nanyang Technological University, Singapore; email: han.yu@ntu.edu.sg;
M. Huang, Qian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing, China; email:huangmeiyu@qxslab.cn; Q. Yang, Department of Computer Science and Engineering, Hong Kong University of Scienceand Technology, Kowloon, Hong Kong; email: qyang@cse.ust.hk.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2020 Association for Computing Machinery.
2157-6904/2020/02-ART6 $15.00
https://doi.org/10.1145/3360309
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6

6:2

J. Wang et al.

Additional Key Words and Phrases: Transfer learning, domain adaptation, distribution alignment, deeplearning, subspace learning, kernel method
ACM Reference format:
Jindong Wang, Yiqiang Chen, Wenjie Feng, Han Yu, Meiyu Huang, and Qiang Yang. 2020. Transfer Learningwith Dynamic Distribution Adaptation. ACM Trans. Intell. Syst. Technol. 11, 1, Article 6 (February 2020), 25
pages.
https://doi.org/10.1145/3360309

1 INTRODUCTION
Supervised learning is perhaps the most popular and well-studied paradigm in machine learning
during the past years. Significant advance has been achieved in supervised learning by exploitinga large amount of labeled training data to build powerful models. For instance, in computer vision,large-scale labeled datasets such as ImageNet [18] for image classification and MS COCO [36]for object detection and semantic segmentation have played an instrumental role to help train
computer vision models with superior performance. In sentiment analysis, a lot of reviews for all
kinds of products are available to train sentiment classification models. Unfortunately, it is often
expensive and time-consuming to acquire sufficient labeled data to train these models. Furthermore, there is often dataset bias in newly emerging data, i.e., the existing model is often trainedon a particular dataset and will generalize poorly in a new domain. For example, the images of an
online product can be very different from those taken at home. The product review for electronic
devices is likely to be different from that of clothes. Under this circumstance, it is necessary andimportant to design algorithms that can handle both the label scarcity and dataset bias challenges.
Domain adaptation, or transfer learning [46, 61], has been a promising approach to solve suchproblems. The main idea of transfer learning is to leverage the abundant labeled samples in someexisting domains to facilitate learning in a new target domain by reducing the dataset bias. The
domain with abundant labeled samples is often called the source domain, while the domain forwhich a new model is to be trained is target domain. However, due to the dataset bias, the data distributions on different domains are usually different. In such a circumstance, traditional machinelearning algorithms cannot be applied directly, since they assume that training and testing data areunder the same distributions. Transfer learning is able to reduce the distribution divergence [46]such that the models on the target domain can be learned.
To cope with the difference in distributions between domains, existing works can be summarized
into two main categories: (a) instance reweighting [16, 68], which reuses samples from the source
domain according to some weighting technique; and (b) feature matching, which either performs
subspace learning by exploiting the subspace geometrical structure [20, 25, 52, 64] or distributionalignment to reduce the marginal or conditional distribution divergence between domains [40, 62,74]. Recently, the success of deep learning has dramatically increased the performance of transfer
learning either via deep representation learning [8, 31, 41, 66, 70, 78] or adversarial learning [22,23, 38, 51, 75]. The key idea behind these works is to learn more transferable representations using
deep neural networks. Then, the learned feature distributions can be aligned such that their domain
discrepancy can be reduced.
However, despite the great success achieved by traditional and deep transfer learning methods, there is still a challenge ahead. Existing works only attempt to align the marginal [37, 45,58] or the conditional distributions [38, 48]. Although recent advance has suggested that aligning
both distributions will lead to better performance [40, 41, 62], they only give the two distributions
equal weights, which fails to evaluate the relative importance of these two distributions. In their assumptions, both the marginal and conditional distributions are contributing equally to the domain
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:3

Fig. 1. (a) The different effect of marginal and conditional distributions. (b) Performance comparison between our dynamic distribution adaptation and the latest transfer learning methods.

divergence. However, in this article, we argue that this assumption is not practical in real applications. For example, when two domains are very dissimilar (e.g., transfer learning between (1) and(3) in Figure 1(a)), the marginal distribution is more important. When the marginal distributions
are close (transfer learning between (1) and (4) in Figure 1(a)), the conditional distribution of eachclass should be given more weight. Ignoring this fact will likely to result in unsatisfactory transfer
performance. There is no method that can quantitatively account for the relative importance of
these two distributions in conjunction.
In this article, we propose a novel concept of Dynamic Distribution Adaptation (DDA) todynamically and quantitatively adapt the marginal and conditional distributions in transfer learning. To be specific, DDA is able to dynamically learn the distribution weights through calculating
the H ΔH divergence [5] between domains when learning representations. Then, the relative importance of marginal and conditional distributions can be obtained, which in turn can be utilized
to learn more transferable feature representations. This dynamic importance learning and featurelearning are being optimized iteratively to learn a domain-invariant transfer classifier eventually.
To the best of our knowledge, DDA is the first work to dynamically and quantitatively evaluate theimportance of both distributions. The significant improvements of DDA over the latest method on
different kinds of tasks is shown in Figure 1(b).
To enable good representation learning, we propose two novel learning methods based on DDA
with the principle of Structural Risk Minimization (SRM) [59]. For traditional transfer learning, wedevelop the Manifold Dynamic Distribution Adaptation (MDDA) method to utilize the Grassmann
manifold [30] in learning non-distorted feature representations. For deep transfer learning, wedevelop Dynamic Distribution Adaptation Network (DDAN) to use the deep neural work in learning end-to-end transfer classifier. We also develop respective learning algorithms for MDDA andDDAN.
To sum up, this work makes the following contributions:(1) We propose the DDA concept for domain adaptation. DDA is the first quantitative evaluation framework for the relative importance of marginal and conditional distributions indomain adaptation. This is useful for a wide range of future research on transfer learning.
(2) On top of DDA, we propose two novel methods: MDDA for traditional transfer learningand DDAN for deep transfer learning. Both methods can be efficiently formulated and,
finally, learn the domain-invariant transfer classifier.
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:4

J. Wang et al.
(3) We conduct extensive experiments on digit classification, object recognition, and sentiment classification datasets. Experimental results demonstrate that both MDDA and
DDAN are significantly better than many state-of-the-art traditional and deep methods.
More importantly, empirical results have also demonstrated that the different effect ofmarginal and conditional distributions do exist, and our DDA is able to give them quantitative weights, which facilitates the performance of transfer learning.

This article is an extension of our previous oral paper at ACM Multimedia conference[65]. Our
extensions include: (1) A more general and clear concept of dynamic distribution adaptation andits calculation. (2) We extend DDA in both manifold learning and deep learning methods, andthen we formulate these algorithms and propose respective learning algorithms. (3) We extend theexperiments in digit classification, sentiment analysis, and image classification, which have shown
the effectiveness of our methods. And (4) we extensively analyze our calculation of DDA in newexperiments.
The remainder of this article is structured as follows. We review the related work in Section 2. In
Section 3, we introduce some previous knowledge before introducing the proposed method. Section 4 thoroughly presents our proposed DDA concept and its two extensions: MDDA and DDAN.
Extensive experiments are shown in Section 5, where we extensively evaluate the performance of
MDDA and DDAN. Finally, Section 6 concludes this article.
2 RELATED WORK
Domain adaptation, or transfer learning, is an active research area in machine learning. Apartfrom the popular survey by Pan and Yang [46], several recent survey papers have extensively investigated specific research topics in transfer learning, including visual domain adaptation [64,65], heterogeneous transfer learning [17, 21], multi-task learning [76], and cross-dataset recognition [73]. There are also several successful applications using transfer learning for activity recognition [13, 14, 63, 66], object recognition [24], face recognition [49], speech recognition [69],speech synthesis [33], and text classification [19]. Interested readers are recommended to refer
to http://transferlearning.xyz to find out more related works and applications.
From the perspective of transfer learning methods, there are three main categories: (1) Instance
re-weighting, which reuses samples according to some weighting technique [15, 16, 56]; (2) Feature transformation, which performs representation learning to transform the source and target
domains into the same subspace [38, 45, 62, 74]; and (3) Transfer metric learning [42–44], whichlearns transferable metric between domains. Since our proposed methods are mainly related tofeature-based transfer learning, we will extensively introduce the related work in the following
aspects.
2.1 Subspace and Manifold Learning
One category of feature-based transfer learning is subspace and manifold learning. The goal is
to learn representative subspace or manifold representations that are invariant across domains.
Along this line, subspace alignment (SA) [20] aligned the base vectors of both domains but failed
to adapt feature distributions. Subspace distribution alignment (SDA) [53] extended SA by adding
subspace variance adaptation. However, SDA did not consider the local property of subspaces
and ignored conditional distribution alignment. Correlation Alignment (CORAL) [52] aligned subspaces in second-order statistics, but it did not consider the distribution alignment. Scatter component analysis (SCA) [24] converted the samples into a set of subspaces (i.e., scatters) and thenminimized the divergence between them.
However, some work used the property of manifold to further learn tight representations. Geodesic flow kernel (GFK) [25] extended the idea of sampled points in manifold [27] and proposed
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:5

to learn the geodesic flow kernel between domains. The work in Reference [3] used a Hellinger
distance to approximate the geodesic distance in Riemann space. Reference [2] proposed to use
Grassmann for domain adaptation, but ignored the conditional distribution alignment. Different
from these approaches, DDA can learn a domain-invariant classifier in the manifold and align bothmarginal and conditional distributions.
2.2 Distribution Alignment
Another category of feature-based transfer learning is distribution alignment. The work of this category is pretty straightforward: Find some feature transformations that can minimize the distribution divergence. Along this line, existing work can be classified into three subcategories: marginal
distribution alignment, conditional distribution alignment, and joint distribution alignment.
DDA substantially differs from existing work that only aligns marginal or conditional distribution [45]. Joint distribution adaptation (JDA) [40] matched both distributions with equal weights.
Others extended JDA by adding regularization [39], sparse representation [67], structural consistency [32], domain invariant clustering [55], and label propagation [74]. The work of Balanced
Distribution Adaptation [62] first proposed to manually weight the two distributions. The maindifferences between DDA (MDDA) and these methods are as follows: (1) These works treat thetwo distributions equally. However, when there is a greater discrepancy between both distributions, they cannot evaluate their relative importance and thus lead to undermined performance.
Our work is capable of evaluating the quantitative importance of each distribution via considering
their different effects. (2) These methods are designed only for the original space, where feature
distortion will negatively affect the performance. DDA (MDDA) can align the distributions in the
manifold to overcome the feature distortion.
2.3

Domain-invariant Classifier Learning

Different from the above two types of work that further need to learn a classifier for the target domain, some research is able to learn the domain-invariant classifier while simultaneously
performing subspace learning or distribution alignment. Recent work such as adaptation regularization for transfer learning (ARTL) [39], domain-invariant projection (DIP) [1, 2], and distribution
matching machines (DMM) [10] also aimed to build a domain-invariant classifier. However, ARTL
and DMM cannot effectively handle feature distortion in the original space. Nor can they accountfor the different importance of distributions. DIP mainly focused on feature transformation andonly aligned marginal distributions. DDA (MDDA) is able to mitigate feature distortion and quantitatively evaluate the importance of marginal and conditional distribution adaptation.
2.4 Deep and Adversarial Transfer Learning
Recent years have witnessed the advance of deep transfer learning. Compared to traditional shallow learning, deep neural networks are capable of learning better representations [70]. Deep domain confusion (DDC) [58] first added the MMD loss to a deep network to adapt the network.
Similarly to DDC, deep adaptation networks (DAN) adopted the multiple-kernel MMD [29] to the
network. Instead, Deep CORAL [54] added CORAL loss [52] to the network. CORAL is a secondorder loss compared to MMD, which is a first-order loss. Furthermore, Zellinger et al. introduce
the central moment discrepancy [71] to the network, which is a higher-order distance.
Different from the above deep transfer learning methods, adversarial learning [26] also helps tolearn more transferable and discriminative representations. Domain-adversarial neural network(DANN) was first introduced by Ganin et al. [22, 23]. The core idea is to add a domain-adversarial
loss to the network instead of the predefined distance function such as MMD. This has dramatically
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:6

J. Wang et al.

enabled the network to learn more discriminative information. Following the idea of DANN, a lotof work adopted domain-adversarial training [8, 11, 38, 41, 75].
The above-discussed work all ignore the different effect of marginal and conditional distributions in transfer learning, while our proposed DDA (DDAN) is fully capable of dynamically evaluating the importance of each distribution.
3 PRELIMINARIESLet Ω ∈ Rd be an input measurable space of dimension d and C the set of possible labels. We
use P (Ω) to denote the set of all probability measures over Ω. In standard transfer learning setns
with known labels yis ∈ C and a target domainting, there is a source domain Ωs = {xsi , yis }i=1ntstΩt = {xj }j=1 with unknown labels. Here, x ∼ P (Ωs ) and xt ∼ P (Ωt ) are samples from either
source or the target domain. Different from existing work that assumes that either the marginalor conditional distributions of two domains are different, in this work, we tackle a more general case that both distributions are different, i.e., P (xs )  P (xt ), P (y s |xs )  P (y t |xt ). The goal
is to learn a transferable classifier f such that the risk on the target domain can be minimized:
ϵt = min P (x,y )∼Ωt ( f (x)  y).
3.1

Structural Risk Minimization

From a statistical machine learning perspective, the above problem can be formulated and solvedby the SRM principle [59]. In SRM, the prediction function f can be formulated asf =arg minf ∈HK , (x,y )∼Ωl

J ( f (x), y) + λR( f ),(1)

where the first term indicates the loss on data samples with J (·, ·) as the loss function, the second
term denotes the regularization term, and HK is the Hilbert space induced by kernel function
K (·, ·). λ is the tradeoff parameter. The symbol Ωl denotes the domain that has labels.
In our problem, we have Ωl = Ωs , since there are no labels in the target domain. Specifically,
to effectively handle the different distributions between Ωs and Ωt , we can further divide theregularization term as
R( f ) = λD f (Ωs , Ωt ) + ρR f (Ωs , Ωt ),(2)

where D f (·, ·) represents the distribution divergence between Ωs and Ωt with λ, ρ the tradeoff
parameters and R f (·, ·) denotes other regularization.
3.2 Maximum Mean Discrepancy
There are a variety of means to measure the distribution divergence between two domains such as
Kullback−Leibler divergence and cross-entropy. With respect to efficiency, we adopt the maximum
mean discrepancy (MMD) [5] to empirically calculate the distribution divergence between domains.
As a non-parametric measurement, MMD has been widely adopted by many existing methods [24,45, 74], and its effectiveness has been proven analytically in Reference [28].
Formally, the MMD distance between distributions P and Q is defined as [28]MMD(Hk , P, Q ) :=sup

| |f | |Hk ≤1

EX ∼P f (X ) − EY ∼Q f (Y ),(3)where Hk is the Reproduced kernel Hilbert space with Mercer kernel K (·, ·), || f || Hk ≤ 1 is its unit
norm ball, and E[·] denotes the mean of the embedded samples.
This is known as an integral probability metric in the statistics literature. To compute this divergence, a biased empirical estimate of MMD is obtained by replacing the population expectations
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:7

Fig. 2. The main idea of MDDA and DDAN.

with empirical expectations computed on the samples X and Y ,
m
n

1
1
f (X i ) −f (Yi )  ,n i=1| |f | |Hk ≤1  m i=1
where m, n are sample numbers of P and Q, respectively.

MMDb (Hk , P, Q ) =

4

sup

(4)

DYNAMIC DISTRIBUTION ADAPTATIONIn this section, we present the general dynamic distribution adaptation framework and its twolearning algorithms in detail.
4.1

The General Framework

Transfer learning is to learn transferable representations that can generalize well across different
domains. The key idea of DDA is to dynamically learn the relative importance of marginal andconditional distributions in transfer learning. Therefore, the dynamic importance learning andtransfer feature learning are not independently but quite involved. Accordingly, DDA first performs feature learning to learn more transferable representations. Then, it can perform dynamic
distribution adaptation to quantitatively account for the relative importance of marginal and conditional distributions to address the challenge of unevaluated distribution alignment. These two
steps are iteratively optimized via several iterations. Eventually, a domain-invariant classifier f
can be learned by combining these two steps based on the principle of SRM.
Recall the principle of SRM in Equation (1). If we use д(·) to denote the feature learning function,then f can be represented as
f = arg
min J ( f (д(xi )), yi ) + η|| f ||K2 + λD f (Ωs , Ωt ) + ρR f (Ωs , Ωt ),f∈n
i =1 HK(5)

where || f ||K2 is the squared norm of f . The term D f (·, ·) represents the proposed dynamic distribution alignment. Additionally, we introduce R f (·, ·) as a Laplacian regularization to further
exploit the similar geometrical property of nearest points in manifold G [4]. η, λ, and ρ are theregularization parameters.
In the next sections, we first introduce the learning of dynamic distribution adaptation. Then,
we show how to learn the feature learning function д(·) either through manifold learning (i.e.,
MDDA in Figure 2(a)) and deep learning (i.e., DDAN in Figure 2(b)).
4.2 Dynamic Distribution Adaptation
The purpose of dynamic distribution adaptation is to quantitatively evaluate the importance ofaligning marginal (P) and conditional (Q) distributions in domain adaptation. Existing methods [40,74] assume that both distributions are equally important. However, this assumption may not be
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:8

J. Wang et al.

valid in real-world applications. For instance, when transferring from (1) to (3) in Figure 1(a), there
is a large difference between datasets. Therefore, the divergence between Ps and Pt is more dominant. In contrast, from (1) to (4) in Figure 1(a), the datasets are similar. Therefore, the distribution
divergence in each class (Q s and Q t ) is more dominant.
In view of this phenomenon, we introduce an adaptive factor to dynamically adjust the importance of these two distributions. Formally, the dynamic distribution alignment D f is defined as
D f (Ωs , Ωt ) = (1 − μ)D f (Ps , Pt ) + μ

C

c=1

D f(c ) (Q s , Q t ),(6)

where μ ∈ [0, 1] is the adaptive factor and c ∈ {1, . . . , C} is the class indicator. D f (Ps , Pt ) denotes
the marginal distribution alignment, and D f(c ) (Q s , Q t ) denotes the conditional distribution alignment for class c.
When μ → 0, it means that the distribution distance between the source and the target domains is large. Thus, marginal distribution alignment is more important ((1) → (3) in Figure 1(a)).
When μ → 1, it means that feature distribution between domains is relatively small, such that the
distribution of each class is dominant. Thus, the conditional distribution alignment is more important ((1) → (4) in Figure 1(a)). When μ = 0.5, both distributions are treated equally as in existing
methods [40, 74]. Hence, the existing methods can be regarded as special cases of the dynamic
distribution alignment. By learning the optimal adaptive factor μ opt (which we will discuss later),
MDDA can be applied to different domain adaptation problems.
We use the MMD [5] introduced in the previous section to empirically calculate the distribution
divergence between domains. To be specific, the marginal and conditional distribution distances
can be respectively computed as2,
D f (Ps , Pt ) = E[f (zs )] − E[f (zt )] HK2
D f(c ) (Q s , Q t ) = E[f (zs(c ) )] − E[f zt(c ) ] H
.
K

(7)
(8)

Then, DDA can be expressed as2
D f (Ωs , Ωt ) = (1 − μ)E[f (zs )) − E[f (zt )] H
+μ
K

C

c=1

 
 
2
E[f zs(c ) ] − E[f zt(c ) ] H
.
K

(9)

Note that since Ωt has no labels, it is not feasible to evaluate the conditional distribution Q t =Q t (yt |zt ). Instead, we follow the idea in Reference [62] and use the class conditional distribution
Q t (zt |yt ) to approximate Q t . To evaluate Q t (zt |yt ), we apply prediction to Ωt using a base classifier
trained on Ωs to obtain soft labels for Ωt . The soft labels may be less reliable, so we iteratively refine
the prediction. Note that we only use the base classifier in the first iteration. After that, MDDA canautomatically refine the labels for Ωt using results from previous iterations.
4.2.1 Quantitative Evaluation of Adaptive Factor μ. We can treat μ as a parameter and tune
its value by cross-validation techniques. However, there are no labels for the target domain in
unsupervised transfer learning problems. There are two indirect solutions to apply the value of μ
in DDA rather than estimating its value: by Random guessing and by Max-min averaging. Random
guessing is technically very intuitive. We can randomly pick a value of μ in [0, 1] and then perform
MDDA using the corresponding μ r and to get the transfer learning result. If we repeat this process
t times and denote the tth transfer learning result as r t , then the final result can be calculated as
1 t
i=1 r t . Max-min averaging is also simple to implement. We can search the value of μ in [0, 1]twith the step of 0.1, which will generate a candidate set of μ: [0, 0.1, . . . , 0.9, 1.0]. Then, similarly
1 11
to random guessing, we can also obtain the averaged results as 11
i=1 r i .
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:9

Although the random guessing and max-min averaging are both feasible and simple solutions
to estimate μ, they are computationally prohibitive. More importantly, there are no guarantees of
their results. It is extremely challenging to calculate the value of μ.
In this work, we make the first attempt toward calculating μ (i.e., μ̂) by exploiting the global
and local structure of domains. We adopt the A-distance [5] as the basic measurement. The Adistance is defined as the error of building a linear classifier to distinguish two domains (i.e., abinary classification). Formally, we denote ϵ (h) the error of a linear classifier h discriminating thetwo domains Ωs and Ωt . Then, the A-distance can be defined asd A (Ωs , Ωt ) = 2(1 − 2ϵ (h)).

(10)

We can directly compute the marginal A-distance using the above equation, which is denoted asd M . For the A-distance between conditional distributions, we use dc to denote the A-distance forthe cth class. It can be calculated as dc = d A (Ds(c ) , Dt(c ) ), where Ds(c ) and Dt(c ) denote samples fromclass c in Ωs and Ωt , respectively. Note that d M denotes the marginal difference, while sumCc=1dc
denotes the conditional difference on all classes. In this article, our assumption is that the domaindivergence is caused by both the marginal and conditional distributions. Therefore, d M + Cc=1 dc
can represent the whole divergence. Eventually, μ can be estimated asμ̂ = 1 −dM.
d M + Cc=1 dc(11)Note that the number of labeled samples in the source domain is often much larger than that in
the target domain. Therefore, to solve this imbalanced classification problem, we perform upsampling [34] on the target domain to make the samples with almost the same size. We also notice that
this upsampling process is random, and thus we repeat this step several times to get the averaged
μ value.
This estimation has to be computed during every iteration of the dynamic distribution adaptation, since the feature distribution may vary after evaluating the conditional distribution eachtime. To the best of our knowledge, this is the first solution to quantitatively estimate the relative
importance of each distribution. In fact, this estimation can be significant for future research intransfer learning and domain adaptation.
Remark. Currently, the quantitative evaluation of μ only supports the situation where the label
space of the source and the target domains are the same. However, it is important to note that
this evaluation is also open for open set domain adaptation [47] or partial transfer learning [72],where the label spaces are not the same. In such cases, we should consider the different similarity
between each class in two domains. For instance, we can regard the classes that do not belong
to the target domain as outliers and perform outlier detection before calculating μ. Then, we canselect the most similar samples and classes in both domains and perform DDA. We leave this partfor future research.
4.3 Manifold Dynamic Distribution Adaptation
In this section, we introduce the learning of DDA through manifold learning. We propose MDDA,
as shown in Figure 2(a). Manifold feature learning can serve as the feature learning step to mitigate the influence of feature distortion [2] in transfer learning. The features in manifold space
can reflect more detailed structure and property of the domains, thus avoiding feature distortion.
MDDA learns д(·) in the Grassmann manifold G(d ) [30], since features in the manifold have somegeometrical structures [4, 30] that can avoid distortion in the original space. In addition, G(d )can facilitate classifier learning by treating the original d-dimensional subspace (i.e., feature vector) as its basic elements. Feature transformation and distribution alignment often have efficient
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:10

J. Wang et al.

numerical forms (i.e., they can be represented as matrix operations easily) and facilitate domain
adaptation on (d ) [30]. There are several approaches to transform the features into G [3, 27]. We
embed Geodesic Flow Kernel (GFK) [25] to learn д(·) for its computational efficiency.
When learning manifold features, MDDA tries to model the domains with d-dimensional subspaces and then embed them into G(d ). Let Ps and Pt denote the PCA subspaces for the source
and the target domain, respectively. G can thus be regarded as a collection of all d-dimensional
subspaces. Each original subspace can be seen as a point in G. Therefore, the geodesic flow
{Φ(t ) : 0 ≤ t ≤ 1} between two points can be used to establish a path between the two subspaces,
where t denotes the calculus variant between two domains. If we let Ps = Φ(0) and Pt = Φ(1),then finding a geodesic flow from Φ(0) to Φ(1) equals to transforming the original features into an
infinite-dimensional feature space, which eventually eliminates the domain shift. This approach
can be seen as an incremental way of “walking” from Φ(0) to Φ(1). Specifically, the new features
can be represented as z = д(x) = Φ(t )T x. From Reference [25], the inner product of transformedfeatures zi and zj gives rise to a positive semidefinite geodesic flow kernel,
 1(Φ(t )T xi )T (Φ(t )T xj ) dt = xTi Gxj .
(12)zi , zj  =0

The geodesic flow can be parameterized asΦ(t ) = Ps U1 Γ(t ) − Rs U2 Σ(t ) = [ Ps

Rs ]

U1
0

0
U2




Γ(t )
,
−Σ(t )(13)

where Rs ∈ RD×d presents the orthogonal complements to Ps . U1 ∈ RD×d and U2 ∈ RD×d are two
orthonormal matrices that can be computed by singular value decomposition (SVD),
PST PT = U1 ΓVT , RST PT = −U2 ΣVT .
According to GFK [25], the geodesic flow kernel G can be calculated by
 T T


U1 Ps
Λ1 Λ2
G = Ps U1 Rs U2,
Λ2 Λ3 U2 RsT

(14)

(15)

where Λ1 , Λ2 , Λ3 are three diagonal matrices with elements
λ 1i = 1 +sin (2θ i )cos (2θ i ) − 1
sin (2θ i ), λ 2i =, λ 3i = 1 −
.
2θ i
2θ i
2θ i

(16)

Thus,√the features in the original space can be transformed into Grassmann manifold with z =д(x) = Gx.
After manifold feature learning and dynamic distribution alignment, f can be learned by summarizing SRM over Ωs and distribution alignment. By adopting the square loss l 2 , f can be expressed as
f = arg min
f ∈HK

n


(yi − f (zi )) 2 + η|| f ||K2 + λD f (Ωs , Ωt ) + ρR f (Ωs , Ωt ).

(17)

i=1

To perform efficient learning, we now further reformulate each term.
SRM on the Source Domain: Using the representer theorem [4], f can be expanded as
f (z) =n+m
βi K (zi , z),

i=1

ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

(18)

Transfer Learning with Dynamic Distribution Adaptation

6:11

where β = (β 1 , β 2 , . . .)T ∈ R (n+m)×1 is the coefficients vector and K is a kernel. Then, SRM on Ωs
becomesnn+m(yi − f (zi )) 2 + η|| f ||K2 =Aii (yi − f (zi )) 2 + η|| f ||K2 = ||(Y − β T K)A||F2 + ηtr(β T Kβ ),
i=1

i=1

(19)
where || · ||F is the Frobenious norm. K ∈ R (n+m)×(n+m) is the kernel matrix with Ki j = K (zi , zj ),and A ∈ R (n+m)×(n+m) is a diagonal domain indicator matrix with Aii = 1 if i ∈ Ωs ; otherwise,
Aii = 0. Y = [y1 , . . . , yn+m ] is the label matrix from the source and the target domains, tr(·) denotes
the trace operation, and η is the shrinkage parameter. Although the labels for Ωt are unavailable,
they can be filtered out by the indicator matrix A.
Dynamic distribution adaptation: Using the representer theorem and kernel tricks, dynamic
distribution alignment in Equation (9) becomes
D f (Ωs , Ωt ) = tr β T KMKβ ,(20)
Cwhere M = (1 − μ)M0 + μ c=1 Mc is the MMD matrix with its element calculated by1⎧,zi , zj ∈ Ωs
⎪
n2
⎪
⎪
1
⎨
(M0 )i j = ⎪ 2 ,zi , zj ∈ Ωt ,
m
⎪
⎪−
1
,
⎩ mn otherwise
1⎧,
zi , zj ∈ Ds(c )
⎪
⎪
nc2
⎪
⎪
⎪
⎪
1
⎪
⎪
,
zi , zj ∈ Dt(c )
⎪
mc2
⎪
⎨
(Mc )i j = ⎪⎧⎪ zi ∈ Ds(c ) , zj ∈ Dt(c )⎪⎪− mc1nc , ⎨⎪⎪⎪ zi ∈ D (c ) , zj ∈ D (c )
⎪
⎪
s
⎪
t
⎩
⎪
⎪
otherwise
⎩ 0,

(21)

,

(22)

where nc = |Ds(c ) | and mc = |Dt(c ) |.
Laplacian Regularization: Additionally, we add a Laplacian regularization term to further
exploit the similar geometrical property of nearest points in manifold G [4]. We denote the pairwise affinity matrix assim(zi , zj ), zi ∈ Np (zj ) or zj ∈ Np (zi )Wi j =,(23)
0,otherwise
where sim(·, ·) is a similarity function (such as cosine distance) for measuring the distance between
two points. Np (zi ) denotes the set of p-nearest neighbors to point zi . p is a free parameter and must
be set in the method. By introducing Laplacian matrix L = D − W with diagonal matrix Dii =n+mj=1 Wi j , the final regularization can be expressed as
R f (Ωs , Ωt ) =n+m
i, j=1

Wi j ( f (zi ) − f (zj )) 2 =

n+m




f (zi )Li j f (zj ) = tr β T KLKβ .

(24)i, j=1

Overall Reformulation: By combining Equations (19), (20), and (24), f in Equation (17) canbe reformulated as
f = arg min ||(Y − β T K)A||F2 + η tr(β T Kβ ) + tr β T K(λM + ρL)Kβ .
(25)
f ∈HK

Setting derivative ∂ f /∂β = 0, we obtain the solutionβ  = ((A + λM + ρL)K + ηI) −1 AYT .

(26)

ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:12

J. Wang et al.

ALGORITHM 1: MDDA: Manifold Dynamic Distribution Adaptation
Input: Data matrix X = [Xs , Xt ], source domain labels ys , manifold subspace dimension d, regularization
parameters λ, η, ρ, and #neighbor p.
Output: Classifier f .
√1: Learn manifold feature transformation kernel G via Equation (12), and get manifold feature Z = GX.
2: Train a base classifier using Ωs , then apply prediction on Ωt to get its soft labels ŷt .
3: Construct kernel K using transformed features Zs = Z1:n,: and Zt = Zn+1:n+m,: .
4: repeat5:
Calculate the adaptive factor μ̂ using Equation (11). and compute M0 and Mc by Equation (21) and (22).
6:Compute β  by solving Equation (26) and obtain f via the representer theorem in Equation (18).
7:
Update the soft labels of Ωt : ŷt = f (Zt ).
8: until Convergence
9: return Classifier f .

4.4 Dynamic Distribution Adaptation Network
In this section, we propose DDAN to perform end-to-end learning of not only the feature learning
function д(·) but also the classifier f . DDAN is able to leverage the ability of the recent advance ofdeep neural networks in learning representative features through end-to-end training [6]. Specifically, we exploit a backbone network to learn useful feature representations, while simultaneously
performing domain adaptation using DDA.
The network architecture of DDAN is shown in Figure 2(b). First, the samples from the source
and target domains serves as the inputs into the deep neural networks. Second, the CNN network(the purple part) such as AlexNet [35] and ResNet [31] can extract high-level features from the
inputs. Third, the features are going through a fully connected layer (the blue part) to perform
soft-max classification to obtain the labels y. The novel contribution here is to align the source
and target domain features using the dynamic distribution alignment (DDA, the yellow part).
Adopting the DDA, the learning objective of DDAN can be expressed as:
f = minΘnJ ( f (xsi ), yis ) + λD f (Ωs , Ωt ) + ρR f (Ωs , Ωt ),

(27)

i=1

where J (·, ·) is the cross-entropy loss function and Θ = {w, b} containing the weight and bias
parameters of the neural network. Note that DDAN is based on the deep neural network, so insteadof using the whole domain data, we use the batch data by following the mini-batch stochastic
gradient descent (SGD) training procedure. Therefore, the dynamic distribution adaptation is only
calculated between batches rather than whole domains. This is more practical and efficient in realapplications where the data are coming in a streaming manner.
Most MMD based deep transfer learning methods [58] are based on Equation (4) and only
adopted the linear kernel for simplicity. Since the formulation in Equation (4) is based on pairwise similarity and is computed in quadratic time complexity, it is prohibitively time-consuming
for using mini-batch SGD in CNN-based transfer learning methods. Gretton et al. [29] further
suggest an unbiased approximation of MMD with linear complexity. Without loss of generality,
by assuming M = N , MMD can then be computed as
2 hl (zi ),
M i=1
M /2

MMDl2 (s, t ) =ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

(28)

Transfer Learning with Dynamic Distribution Adaptation

6:13

ALGORITHM 2: DDAN: Dynamic Distribution Adaptation Network
Input: Source domain (xs , ys ), target domain data xt , regularization parameters λ, ρ, and #neighbor p.
Output: Classifier f .
1: repeat2:
Sample a mini-batch data from both the source and target domain
3:
Feed the mini-batch data into the network and get the pseudo labels for Ωt
4:Update the parameters {Θ, b} by computing the mini-batch gradient according to Equation (30).
5:After an epoch, calculate μ using Equation (1) and calculate the loss
6: until Convergence
7: return Classifier f .

where hl is an operator defined on a quad-tuple zi = (xs2i−1 , xs2i , xt2j−1 , xt2j ),







hl (zi ) = k xs2i−1 , xs2i + k xt2j−1 , xt2j − k xs2i−1 , xt2j − k xs2i , xt2j−1(29)

The approximation in Equation (28) takes a summation form and is suitable for gradient computation in a mini-batch manner.
The gradient of the parameters can be computed as:∂D f (·, ·)∂R f (·, ·)∂J (·, ·)
+λ
+ρ
.
(30)
∂Θ
∂Θ
∂Θ
Updating μ: Another important aspect is to dynamically update μ in DDAN. Similarly to theabove mini-batch learning of MMD distance, it seems natural to calculate μ after each mini-batchlearning. However, the labels on the source domain and the pseudo labels on the target domain
are likely to be inconsistent after mini-batch learning. For instance, assume that the batch size forboth domains is bsize = 32 and the total class number |C| = 31. Then, after a forward operationfor one batch, we can obtain 32 pseudo labels for the target domain. Since bsize is rather close to|C|, it is highly likely that the mini-batch labels for both domains do not match, which will easily
lead to the mode collapse or gradient exploding problem.
To avoid this problem, we propose to update μ after each epoch of iteration, rather than eachmini-batch data. In fact, this step is very similar to that of MDDA, which uses all the data to perform
learning. The learning process of DDAN is summarized in Algorithm 2.
ΔΘ =

4.5

Discussions

Both MDDA and DDAN are generic learning methods that are suitable for all transfer classification
problems. In this section, we briefly discuss their differences.
MDDA is for traditional learning, while DDAN is for deep learning. Compared to DDAN, which
is designed for deep neural networks, MDDA can be easily applied on the small resource constraint devices. One possible limitation of MDDA maybe that it relies on certain feature extraction
methods. For instance, on image dataset, we probably need to extract SIFT, SURF, or HOG features. Luckily, MDDA can also use the deep features extracted by deep neural networks such as
AlexNet [35] and ResNet [31]. MDDA has a very useful property: It can learn the cross-domainfunction directly without the need for explicit classifier training. This makes it significantly moreadvantageous compared to most existing work such as Joint Geometrical and Statistical Alignment(JGSA) [74] and SCA [24], which needs to train an extra classifier after learning transferrable features.
However, DDAN is suitable for cloud computing. The model of DDAN can be trained in an endto-end manner and then be used for inference on the device. DDAN does not need extra feature
extraction and classifier training procedure. All steps can be unified in one single deep neural
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:14

J. Wang et al.

Fig. 3. Samples from the datasets in this article.
Table 1. Statistics of the Five Benchmark Datasets

Dataset
USPS+MNIST
Amazon review
Office-31
ImageCLEF DA
Office-Home#Sample
3,800
1,123
4,110
1,800
15,500#Feature for MDDA
256
400
2,048
2,048
2,048

#Class
10
2
31
12
65

DomainU, M
B, D, E, K
A, W, D
C, I, PAr, Cl, Pr, Rw

Type
Digit
Text
Image
Image
Image

network. This advantage makes it useful for large-scale datasets, which will probably result inprohibitive computations for MDDA.
5EXPERIMENTS AND EVALUATIONSIn this section, we evaluate the performance of MDDA through extensive experiments on largescale public datasets. The source code for MDDA is available at http://transferlearning.xyz. We will
focus on evaluating the performance of MDDA, since most of our contributions can be covered
by MDDA. In the last part of this section, we will evaluate the performance of the deep version of
MDDA.
5.1 Experimental Setup
5.1.1 Datasets. We adopted five public image datasets: USPS+MNIST, Amazon review [7],Office-31 [50], ImageCLEF-DA [41], and Office-Home [60]. These datasets are popular for benchmarking domain adaptation algorithms and have been widely adopted in most existing work suchas in References [11, 38, 74, 75]. Figure 3 shows some samples of the datasets, and Table 1 lists
their statistics.
USPS (U) and MNIST (M) are standard digit recognition datasets containing handwritten digits
from 0 to 9. Since the same digits across two datasets follow different distributions, it is necessary
to perform domain adaptation. USPS consists of 7,291 training images and 2,007 test images ofsize 16 × 16. MNIST consists of 60,000 training images and 10,000 test images of size 28 × 28. We
construct two tasks: U → M and M → U. In the rest of the article, we use A → B to denote theknowledge transfer from source domain A to the target domain B.
Amazon review [7] is the benchmark dataset for cross-domain sentiment analysis. This dataset
includes reviews about the Kitchen appliances (K), DVDs (D), Books (B), and Electronics (E). The
reviews of each product can be regarded as data from the same domain. There are 1,000 positive
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:15

and 1,000 negative instances on each domain. Transfer learning can be conducted between any
two domains, leading to 12 tasks.
Office-31 [50] consists of three real-world object domains: Amazon (A), Webcam (W), andDSLR (D). It has 4,652 images with 31 categories. Caltech-256 (C) contains 30,607 images and 256
categories. We constructed six tasks: A → D, A → W, D → A, D → W, W → A, W → D.
ImageCLEF-DA [41] is a dataset presented in the ImageCLEF 2014 domain adaptation challenge. It is composed by selecting the 12 common classes shared by three public datasets (domains):
Caltech-256 (C), ImageNet ILSVRC 2012 (I), and Pascal VOC 2012 (P). There are 50 images in eachcategory and 600 images in each domain, while Office-31 has different domain sizes. We permute
domains and build six transfer tasks: C → I, C → P, I → C, I → P, P → C, P → I.
Office-Home [60] is a new dataset that consists of 15,588 images from four different domains:
Artistic images (Ar), Clip Art (Cl), Product images (Pr), and Real-World images (Rw). For eachdomain, the dataset contains images of 65 object categories collected from office and home settings.
We use all domains and construct 12 transfer learning tasks: Ar → Cl, . . ., Rw → Pr.
In total, we constructed 2 + 12 + 6 + 6 + 12 = 38 tasks.
5.1.2 State-of-the-art Comparison Methods. We compared the performance of MDDA with several state-of-the-art traditional and deep transfer learning approaches.
Traditional transfer learning methods:• NN, SVM, and PCA• TCA: Transfer Component Analysis [45], which performs marginal distribution alignment• GFK: performs manifold feature learning [25]• JDA: adapts marginal & conditional distribution [40]• CORAL: performs second-order subspace alignment [52]
• SCA: adapts scatters in subspace [24]
• JGSA: aligns marginal & conditional distributions with label propagation [74]
Deep transfer learning methods:• AlexNet [35] and ResNet [31], as baseline networks• DDC: a single-layer deep adaptation method [58]• DAN: a multi-layer adaptation method [37]
• DANN: an adversarial deep neural network [22]• ADDA: Adversarial Discriminative Domain Adaptation [57], which is a general framework
for adversarial transfer learning• JAN: Joint Adaptation Networks [41], which is a deep network with joint MMD distance• CAN: Collaborative and Adversarial Network [75], which is based on joint training• CDAN: Conditional Domain Adversarial Networks [38], which is a conditional network
5.1.3 Implementation Details. MDDA requires to extract features from the raw inputs. For
USPS+MNIST datasets, we adopted the 256 SURF features by following existing work [62, 74].
For Amazon review dataset, we follow the feature generation method to exploit marginalized denoising autoencoders [12] to improve the feature representations. For Office-31, ImageCLEF DA,
and Office-Home datasets, we adopted the 2,048 fine-tuned ResNet-50 features for fair comparison.
As for DDAN, we only report its results on three image datasets, since USPS+MNIST and Amazon review datasets are rather simple to transfer. DDAN is able to take the original image data as
inputs. We also adopted ResNet-50 as the baseline network for fair comparison [38, 75].
For the comparison methods, we either cite the results reported in their original papers or
run experiments using their publicly available codes. As for MDDA, we set the manifold feature
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:16

J. Wang et al.
Table 2. Classification Accuracy (%) on USPS-MNIST Datasets with SURF Features

Task
U→M
M→U
AVG

1NN
44.7
65.9
55.3

SVM
62.2
68.2
65.2

TCA
51.2
56.3
53.8

GFK
46.5
61.2
53.9

JDA
59.7
67.3
63.5

CORAL
30.5
49.2
39.9

SCA
48.0
65.1
56.6

JGSA
68.2
80.4
74.3

MDDA
76.8
89.6
83.2

Table 3. Classification Accuracy (%) on Amazon Review Dataset

Method
B→D
B→E
B→K
D→B
D→E
D→K
E→B
E→D
E→K
K→B
K→D
K→E
AVG

1NN
49.6
49.8
50.3
53.3
51.0
53.1
50.8
50.9
51.2
52.2
51.2
52.3
51.3

TCA
63.6
60.9
64.2
63.3
64.2
69.1
59.5
62.1
74.8
64.1
65.4
74.5
65.5

GFK
66.4
65.5
69.2
66.3
63.7
67.7
62.4
63.4
73.8
65.5
65.0
73.0
66.8

SA
67.0
70.8
72.2
67.5
67.1
69.4
61.4
64.9
70.4
64.4
64.6
68.2
67.3

JDA
64.2
62.1
65.4
62.4
66.3
68.9
59.2
61.6
74.7
62.7
64.3
74.0
65.5

CORAL
71.6
65.1
67.3
70.1
65.6
67.1
67.1
66.2
77.6
68.2
68.9
75.4
69.1

JGSA
66.6
75.0
72.1
55.5
67.3
65.6
51.6
50.8
55.0
58.3
56.4
51.7
60.5

MDDA
77.8
80.0
79.9
74.7
80.4
81.0
63.8
62.5
84.4
63.5
72.2
80.7
75.1

dimension d = 30, 30, 50, 60, 200 for the five datasets, respectively. The number of iteration is
set to T = 10. We use the RBF kernel with the bandwidth set to be the variance of inputs.
The regularization parameters are set as p = 10, λ = 4.5, η = 0.1, and ρ = 1. Additionally, theexperiments on parameter sensitivity and convergence analysis in Section 5.5 indicate that theperformance of MDDA and DDAN stays robust with a wide range of parameter choices. For
DDAN, we set the learning rate to be 0.01 with the batch size to be 32 and a weight decay of
5e − 4. Other parameters are tuned by following transfer cross validation [77].
Although MDDA is easy to use, and its parameters do not have to be fine-tuned. For research
purpose, we also investigate how to further tune those parameters. We choose parameters according to following rules. First, SRM on source domain is very important. Thus, we prefer a small η tomake sure MDDA does not degenerate. Second, distribution alignment is required by SRM. Thus,
we choose a slightly larger λ to make it effective. Third, we choose ρ by following the existing
work [4]. Fourth, p is set following Reference [9].
We adopt classification accuracy on Ωt as the evaluation metric, which is widely used in existing
literature [25, 45, 62]:x : x ∈ Ωt ∧ ŷ(x) = y(x) .
(31)
Accuracy = |x : x ∈ Ωt |5.2

Results and Analysis

5.2.1 Results on Digit Datasets. The classification results on USPS+MNIST datasets are shownin Table 2. On the digit recognition tasks, MDDA outperforms the best method JGSA by a large
margin of 8.9%. These results clearly indicate that MDDA significantly outperforms existing
methods.
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:17

Table 4. Classification Accuracy (%) on Office-31 Dataset with ResNet-50 as the Baseline
MethodBaseline

Traditional transfer learning

Task ResNet 1NN SVM TCA GFK JDA
A → D 68.9 79.1 76.9 74.1 77.9 80.7
A → W 68.4 75.8 73.3 72.7 72.6 73.6
D → A 62.5 60.2 64.1 61.7 62.3 64.7
D → W 96.7 96.0 96.5 96.7 95.6 96.5
W → A 60.7 59.9 64.9 60.9 62.8 63.1
W → D 99.3 99.4 99.0 99.6 99.0 98.6
AVG

76.1

78.4 79.1 77.6 78.4 79.5

CORAL
81.5
77.0
65.9
97.1
64.3
99.6

Deep transfer learning

DDA

DDC DAN DANN ADDA JAN CAN MDDA DDAN
76.5 78.6 79.7
77.8 84.7 85.5 86.3
84.9
75.6 80.5 82.0
86.2 85.4 81.5 86.0
88.8
62.2 63.6 68.2
69.5 68.6 65.9 72.1
65.3
96.0 97.1 96.9
96.2 97.4 98.2 97.1
96.7
61.5 62.8 67.4
68.9 70.0 63.4 73.2
65.0
98.2 99.6 99.1
98.4 99.8 99.7 99.2
100

80.9

78.3 80.4

82.2

82.9

84.3 82.4

85.7

83.5

Table 5. Classification Accuracy (%) on ImageCLEF DA with ResNet-50 as Baseline
MethodBaseline

Traditional transfer learning

Task ResNet 1NN SVM TCA GFK JDA
C→I
78.0 83.5 86.0 89.3 86.3 90.8
C → P 65.5 71.3 73.2 74.5 73.3 73.6
I→C
91.5 89.0 91.2 93.2 93.0 94.0
I→P
74.8 74.8 76.8 77.5 75.5 75.3
P → C 91.2 76.2 85.8 83.7 82.3 83.5
P→I
83.9 74.0 80.2 80.8 78.0 77.8
AVG

80.7

78.1 82.2 83.2 81.4 82.5

CORAL
83.0
71.5
88.7
73.7
72.0
71.3
76.7

Deep transfer learning

DDA

DAN DANN JAN CAN CDAN MDDA DDAN
86.3 87.0 89.5 89.5 91.2
92.0
91.0
69.2 74.3 74.2 75.8 77.2
78.8
76.0
92.8 96.2 94.7 94.2 96.7
95.7
94.0
74.5 75.0 76.8 78.2 78.3
79.8
78.0
89.8 91.5 91.7 89.2 93.7
95.5
92.7
82.2 86.0 88.0 87.5 91.2
91.5
91.0
82.5

85.0

85.8 85.7

88.1

88.9

87.2

Moreover, the performances of distribution alignment methods (TCA, JDA, and JGSA) and subspace learning methods (GFK, CORAL, and SCA) were generally inferior to MDDA. Each method
has its limitations and cannot handle domain adaptation in specific tasks, especially with degenerated feature transformation and unevaluated distribution alignment. After manifold or subsapce
learning, there still exists large domain shift [2]; while feature distortion will undermine the distribution alignment methods.
5.2.2 Results on Sentiment Analysis Dataset. The results on Amazon review datasets are shownin Table 3. From the results, we can observe that our proposed MDDA outperforms the best baseline
method CORAL by a large margin of 6.0%. This clearly indicated that MDDA is able to dramaticallyreduce the divergence between different text domains.
5.2.3 Results on Image Datasets. The classification accuracy results on the Office-31, ImageCLEF DA, and Office-Home datasets are shown in Tables 4, 5, and 6, respectively. From thoseresults, we can make the following observations.
First, MDDA outperforms all other traditional and deep comparison methods in most tasks(20/24 tasks). The average classification accuracy achieved by MDDA on all the image tasks is77.3%. Specifically, on the hardest Office-Home dataset, MDDA significantly outperforms the latest
deep transfer learning method CDAN [38] by 4.5%, which clearly demonstrates the effectivenessof MDDA. The results indicates that MDDA is capable of significantly reducing the distribution
divergence in domain adaptation problems.
Second, DDAN also substantially outperforms all the traditional and deep methods on mosttasks. Note that DDAN is only based on deep neural network without adversarial training, while
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:18

J. Wang et al.
Table 6. Classification Accuracy (%) on Office-Home Dataset with ResNet-50 as Baseline

MethodBaseline

Task
ResNet 1NN
Ar → Cl
34.9 45.3
Ar → Pr
50.0 60.1
Ar → Rw 58.0 65.8
Cl → Ar
37.4 45.7
Cl → Pr
41.9 57.0
Cl → Rw 46.2 58.7
Pr → Ar
38.5 48.1
Pr → Cl
31.2 42.9
Pr → Rw 60.4 68.9
Rw → Ar 53.9 60.8
Rw → Cl 41.2 48.3
Rw → Pr 59.9 74.7
Avg

46.1

56.4

Traditional transfer learning
SVM
45.3
65.4
73.1
43.6
57.3
60.2
46.8
39.1
69.2
61.1
45.6
75.9

TCA
38.3
58.7
61.7
39.3
52.4
56.0
42.6
37.5
64.1
52.6
41.7
70.5

GFK
38.9
57.1
60.1
38.7
53.1
55.5
42.2
37.6
64.6
53.8
42.3
70.6

JDA
38.9
54.8
58.2
36.2
53.1
50.2
42.1
38.2
63.1
50.2
44.0
68.2

56.9

51.3 51.2 49.8

CORAL
42.2
59.1
64.9
46.4
56.3
58.3
45.4
41.2
68.5
60.1
48.2
73.1
55.3

Deep transfer learning

DDA

DAN DANN JAN CDAN MDDA DDAN
43.6
45.6 45.9 46.6
54.9
51.0
57.0
59.3 61.2 65.9
75.9
66.0
67.9
70.1 68.9 73.4
77.2
73.9
45.8
47.0 50.4 55.7
58.1
57.0
56.5
58.5 59.7 62.7
73.3
63.1
60.4
60.9 61.0 64.2
71.5
65.1
44.0
46.1 45.8 51.8
59.0
52.0
43.6
43.7 43.4 49.1
52.6
48.4
67.7
68.5 70.3 74.5
77.8
72.7
63.1
63.2 63.9 68.2
67.9
65.1
51.5
51.8 52.4 56.9
57.6
56.6
74.3
76.8 76.8 80.7
81.8
78.9
56.3

57.6

58.3

62.8

67.3

62.5

Fig. 4. (a) Performance of several tasks when searching μ in [0, 1]. (b) Performance comparison of Random
guessing (Random), average search (AVSE), and our DDA.

other deep methods such as CAN [75] and CDAN [38] all require to train an adversarial neuralnetwork, which clearly needs more time to converge. In this way, DDAN is much more efficientthan these networks.
Third, we also note that traditional methods such as TCA, JDA, and CORAL can also achieve
good performance compared to the ResNet, 1NN, and SVM. This clearly indicates the necessityof transfer learning when building models from two domains. Again, our proposed MDDA and
DDAN can achieve the best performances.
5.3

Evaluation of Dynamic Distribution Adaptation

We verify the effectiveness of dynamic distribution adaptation in this section. We answer two important questions: (1) Does the different effect of marginal and conditional distributions exist intransfer learning? and (2) Is our evaluation algorithm for DDA effective? It is worth noting that
there is no ground truth for μ. Therefore, to verify our evaluation, we record the performance of
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:19

Table 7. Comparison of the Performance between Our Evaluation of μ and Average Search (AVSE)
Task M → U B → E E → D A → W W → A C → P P → C Ar → Rw Cl → Pr Rw → Cl AVG
AVSE −11.83 −1.95 −1.40 −4.03−4.69 −0.66 −0.67
−1.49
−3.56
−1.21
−3.15
−0.14 −0.00 −0.00
−0.02
−0.29
−0.02
−0.37
Ours +0.22 −1.25 −0.80 −1.26
Suppose the results of grid search are 0.

Fig. 5. Ablation study of MDDA and DDAN. “M” denotes manifold learning, and “Lap” denotes Laplace
regularization.

DDA by searching different μ values. The hint is that better μ value contributes better performance.
Specifically, we run DDA by searching μ ∈ {0, 0.1, . . . , 0.9, 1.0}. To answer the first question, we
draw the results of DDA under different values of μ in Figure 4(a). To answer the second question, we compare the error made by random search (Random), average search (Average), and ourevaluation in Figure 4(b).
First, it is clear that the classification accuracy varies with different choices of μ. This indicatesthe necessity to consider the different effects between marginal and conditional distributions. We
can also observe that the optimal μ value varies on different tasks (μ = 0.2, 0, 1 for the three tasks,
respectively). Thus, it is necessary to dynamically adjust the distribution alignment between domains according to different tasks. Moreover, the optimal value of μ is not unique for a given task.
The classification results may be the same even for different values of μ.
Second, we also report the results of our evaluation and average search in Table 7. Combining
the results in Figure 4(b), we can conclude that our evaluation of μ is significantly better thanrandom search and average search. Additionally, both random search and average search require
to run the whole MDDA or DDAN algorithm several times to get steady results, our evaluationis only required once in each iteration of the algorithm. This means that our evaluation is more
efficient. It is worth noting that our evaluation is extremely close to the results from grid search.
Note that on task M → U of Table 7, our evaluation exceeds the results of grid search. Consideringthat there is often few or none labels in the target domain, grid search is not actually possible.
Therefore, our evaluation of μ can be used to approximate the ground truth in real applications.
Third, we noticed that on image classification datasets, the performance of MDDA is slightly
better than DDAN. MDDA is a shallow learning method, which is much easier to tune hyperparameters than DDAN, which is based on deep learning. We think that after a more extensive
hyperparameter tuning process, the performance of DDAN will be the same or better as MDDA.
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:20

J. Wang et al.

Fig. 6. Parameter sensitivity analysis and convergence of MDDA.

5.4

Ablation StudyIn this section, we conduct ablation study of MDDA and DDAN. MDDA mainly consists of four
components: SRM, manifold learning, DDA, and Laplacian regularization. DDAN is composed ofa deep network, DDA, and Laplacian regularization. We extensively analyze the performance of
MDDA and DDAN on some tasks from each dataset and present the results in Figure 5.
The results clearly indicate that each component is important to DDA. Of all the components,
it is shown that our proposed DDA component is the most important part, which dramaticallyincreases the results of transfer learning. For MDDA, manifold feature learning shows marginal
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:21

Fig. 7. Parameter sensitivity and convergence analysis of DDAN.
Table 8. Running Time of MDDA and DDAN

Task
U→M
B→E
W→A
C→P
Cl → Pr

ARTL
29.1
22.8
45.6+763.6
124.2 + 1321.4
187.4 + 1768.7

JGSA
14.6
18.7
66.5 + 763.6
198.3 + 1321.4
244.3 + 1768.7

DANN
—
—
1567.32342.12877.7

CDAN——
1873.22451.22956.5

MDDA
31.4
23.5
48.8 + 7663.6
156.7 + 1321.4
207.4 + 1768.7

DDAN
—
—
1324.12109.82698.1

improvement, while it could help to eliminate the feature distortion of the original space [2]. For
DDAN, we can clearly see that our DDA component is better than DAN, which is only adapting
the marginal distributions. This again clarifies the importance of the proposed DDA framework.
Finally, it seems that Laplacian regularization also generates marginal improvements except on
the digit datasets (USPS+MNIST). We add Laplacian regularization, since it helps the algorithm toconverge.
5.5

Parameter Sensitivity and Convergence Analysis

As with other state-of-the-art domain adaptation algorithms [24, 39, 74], MDDA and DDAN also
involve several parameters. In this section, we evaluate the parameter sensitivity of them. Experimental results demonstrated the robustness of MDDA and DDAN under a wide range of parameterchoices.
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:22

J. Wang et al.

5.5.1 MDDA. We investigated the sensitivity against manifold subspace dimension d and#neighbor p through experiments with a wide range of d ∈ {10, 20, . . . , 100} and p ∈ {2, 4, . . . , 64}
on randomly selected tasks. From the results in Figure 6(a) and 6(b), it can be observed that MDDA
is robust with regard to different values of d and p. Therefore, they can be selected without in-depth
knowledge of specific applications.
We ran MDDA with a wide range of values for regularization parameters λ, η, and ρ on several random tasks and compare its performance with the best baseline method. We only report
the results of λ in Figure 6(c), and the results of ρ and η are following the same tendency. We
observed that MDDA can achieve a robust performance with regard to a wide range of parameter values. Specifically, the best choices of these parameters are λ ∈ [0.5, 1, 000], η ∈ [0.01, 1], andρ ∈ [0.01, 5].
We evaluate the convergence of MDDA through experimental analysis. From the results in
Figure 6(f), it can be observed that MDDA can reach a steady performance in only a few (T < 10)
iterations. It indicates the training advantage of MDDA in cross-domain tasks.
5.5.2 DDAN. DDAN involves three key parameters: λ, p, and ρ. Similarly to MDDA, we report
the parameter sensitivity and convergence in Figure 7. It is clear that DDAN is robust to theseparameters. Therefore, in real applications, the hyperparameters of DDAN do not have to be cherry
picked. This is extremely important in deep learning, since it is rather time-consuming to tune thehyperparameters.
We also extensively evaluate the convergence of DDAN in Figure 7(d). It is shown that DDAN
is able to convergence quickly and reach steady performance.
5.5.3 Time complexity. We empirically check the running time of MDDA and DDAN andpresent the results in Table 8. Noe that for image classification tasks, the running time of ARTL,JGSA, and MDDA are the summation of deep feature extraction and algorithm running time, sincethese algorithms require to extract features before transfer learning. It is shown that both MDDA
and DDAN can achieve efficient computing time while achieving better performance compared tothese comparison methods.
6 CONCLUSIONS AND FUTURE WORKIn this article, to solve the transfer learning problem, we propose the novel DDA concept. DDA isable to dynamically evaluate the relative importance between the source and target domains. Basedon DDA, we propose two novel methods: the manifold DDA (MDDA) for traditional transfer learning, and DDAN for deep transfer learning. Extensive experiments on digit recognition, sentiment
analysis, and image classification have demonstrated that both MDDA and DDAN could achieve
the best performance compared to other state-of-the-art traditional and deep transfer learning
methods.
In the future, we plan to extend the DDA framework into the heterogeneous transfer learning
areas as well as apply it to more complex transfer learning situations.
REFERENCES[1] Mahsa Baktashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. 2016. Distribution-matching embedding for visual domain adaptation. J. Mach. Learn. Res. 17, 1 (2016), 3760–3789.
[2] Mahsa Baktashmotlagh, Mehrtash T. Harandi, Brian C. Lovell, and Mathieu Salzmann. 2013. Unsupervised domain
adaptation by domain invariant projection. In Proceedings of the IEEE International Conference on Computer Vision.
769–776.
[3] Mahsa Baktashmotlagh, Mehrtash T. Harandi, Brian C. Lovell, and Mathieu Salzmann. 2014. Domain adaptation on
the statistical manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2481–2488.
ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:23[4] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. 2006. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res. 7 (Nov 2006), 2399–2434.
[5] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain
adaptation. In Advances in Neural Information Processing Systems. 137–144.
[6] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives.
IEEE Trans. Pattern Anal. Mach. Intell. 35, 8 (2013), 1798–1828.
[7] John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 120–128.
[8] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain
separation networks. In Advances in Neural Information Processing Systems. 343–351.
[9] Deng Cai, Xiaofei He, Jiawei Han, and Thomas S Huang. 2011. Graph regularized nonnegative matrix factorization
for data representation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 8 (2011), 1548–1560.
[10] Yue Cao, Mingsheng Long, and Jianmin Wang. 2018. Unsupervised domain adaptation with distribution matchingmachines. In Proceedings of the 2018 AAAI International Conference on Artificial Intelligence (AAAI’18).
[11] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. 2019. Joint domain alignment and discriminative featurelearning for unsupervised deep domain adaptation. In Proceedings of the 2018 AAAI International Conference on Artificial Intelligence (AAAI’19).
[12] Minmin Chen, Zhixiang Xu, Kilian Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain
adaptation. In Proceedings of the International Conference on Machine Learning (ICML’12).
[13] Yiqiang Chen, Jindong Wang, Meiyu Huang, and Han Yu. 2019. Cross-position activity recognition with stratified
transfer learning. Perv. Mob. Comput. 57, July (2019), 1–13.
[14] Yiqiang Chen, Jindong Wang, Chaohui Yu, Wen Gao, and Xin Qin. 2019. FedHealth: A federated transfer learning
framework for wearable healthcare. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’19) Workshop on Federated Machine Learning.
[15] Wenyuan Dai, Gui-Rong Xue, Qiang Yang, and Yong Yu. 2007. Co-clustering based classification for out-of-domain
documents. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
ACM, 210–219.
[16] Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. 2007. Boosting for transfer learning. In Proceedings of the24th International Conference on Machine Learning (ICML’07). ACM, 193–200.
[17] Oscar Day and Taghi M. Khoshgoftaar. 2017. A survey on heterogeneous transfer learning. Journal of Big Data 4, 1
(2017), 29.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image
database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2009 (CVPR’09). IEEE,
248–255.
[19] Chuong B. Do and Andrew Y. Ng. 2006. Transfer learning for text classification. In Advances in Neural Information
Processing Systems. 299–306.
[20] Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. 2013. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE International Conference on Computer Vision. 2960–2967.
[21] Magda Friedjungová and Marcel Jirina. 2017. Asymmetric heterogeneous transfer learning: A survey. In Proceedingsof the 6th International Conference on Data Science, Technology and Applications (DATA’17). 17–27.
[22] Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings ofthe International Conference on Machine Learning (ICML’15).
[23] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario
Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17,
59 (2016), 1–35.
[24] Muhammad Ghifary, David Balduzzi, W. Bastiaan Kleijn, and Mengjie Zhang. 2017. Scatter component analysis: A
unified framework for domain adaptation and domain generalization. IEEE Trans. Pattern Anal. Mach. Intell. 39, 7(2017), 1414–1430.
[25] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. 2012. Geodesic flow kernel for unsupervised domain adaptation. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 2066–
2073.
[26] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, andYoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672–2680.
[27] Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. 2011. Domain adaptation for object recognition: An unsupervised approach. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV’11). IEEE,
999–1006.

ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

6:24

J. Wang et al.

[28] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel
two-sample test. J. Mach. Learn. Res. 13 (Mar. 2012), 723–773.
[29] Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu,
and Bharath K. Sriperumbudur. 2012. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural
Information Processing Systems. 1205–1213.
[30] Jihun Hamm and Daniel D. Lee. 2008. Grassmann discriminant analysis: A unifying view on subspace-based learning.
In Proceedings of the 25th International Conference on Machine Learning. ACM, 376–383.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.
[32] Cheng-An Hou, Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang. 2016. Unsupervised domain adaptation with label and structural consistency. IEEE Trans. Image Process. 25, 12 (2016), 5552–5562.
[33] Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez
Moreno, Yonghui Wu, et al. 2018. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In Advances in Neural Information Processing Systems. 4480–4490.
[34] Bartosz Krawczyk. 2016. Learning from imbalanced data: Open challenges and future directions. Progr. Artif. Intell. 5,4 (2016), 221–232.
[35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neuralnetworks. In Advances in Neural Information Processing Systems. 1097–1105.
[36] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C.
Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proceedings of the European Conference on
Computer Vision. Springer, 740–755.
[37] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015. Learning transferable features with deep adaptation networks. In Proceedings of the International Conference on Machine Learning (ICML’15). 97–105.
[38] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. 2018. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems. 1645–1655.
[39] Mingsheng Long, Jianmin Wang, Guiguang Ding, Sinno Jialin Pan, and S Yu Philip. 2014. Adaptation regularization:
A general framework for transfer learning. IEEE Trans. Knowl. Data Eng. 26, 5 (2014), 1076–1089.
[40] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. 2013. Transfer feature learning withjoint distribution adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 2200–2207.
[41] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. 2017. Deep transfer learning with joint adaptation
networks. In Proceedings of the International Conference on Machine Learning. 2208–2217.
[42] Yong Luo, Tongliang Liu, Dacheng Tao, and Chao Xu. 2014. Decomposition-based transfer distance metric learningfor image classification. IEEE Trans. Image Process. 23, 9 (2014), 3789–3801.
[43] Yong Luo, Tongliang Liu, Yonggang Wen, and Dacheng Tao. 2018. Online heterogeneous transfer metric learning. In
Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’18). 2525–2531.
[44] Yong Luo, Yonggang Wen, Lingyu Duan, and Dacheng Tao. 2018. Transfer metric learning: Algorithms, applications
and outlooks. arXiv preprint arXiv:1810.03944 (2018).
[45] Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. 2011. Domain adaptation via transfer component
analysis. IEEE Trans. Neur. Netw. 22, 2 (2011), 199–210.
[46] Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 10 (2010),
1345–1359.
[47] Pau Panareda Busto and Juergen Gall. 2017. Open set domain adaptation. In Proceedings of the IEEE International
Conference on Computer Vision. 754–763.
[48] Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. 2018. Multi-adversarial domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence.
[49] Chuan-Xian Ren, Dao-Qing Dai, Ke-Kun Huang, and Zhao-Rong Lai. 2014. Transfer learning of structured representation for face recognition. IEEE Trans. Image Process. 23, 12 (2014), 5440–5454.
[50] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. 2010. Adapting visual category models to new domains. In
Proceedings of the European Conference on Computer Vision. Springer, 213–226.
[51] Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. 2018. Generate to adapt: Aligning
domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR’18).
[52] Baochen Sun, Jiashi Feng, and Kate Saenko. 2016. Return of frustratingly easy domain adaptation. In Proceedings of
the AAAI Conference on Artificial Intelligence (AAAI’16), Vol. 6. 8.
[53] Baochen Sun and Kate Saenko. 2015. Subspace distribution alignment for unsupervised domain adaptation. In Proceedings of the British Machine Vision Conference (BMVC’15). 24–1.

ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.

Transfer Learning with Dynamic Distribution Adaptation

6:25[54] Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In Proceedingsof the European Conference on Computer Vision. Springer, 443–450.
[55] Jafar Tahmoresnezhad and Sattar Hashemi. 2017. Visual domain adaptation via transfer feature learning. Knowl. Inf.
Syst. 50, 2 (2017), 586–605.
[56] Ben Tan, Yangqiu Song, Erheng Zhong, and Qiang Yang. 2015. Transitive transfer learning. In Proceedings of the 21th
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1155–1164.
[57] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In
Proceedings of the Computer Vision and Pattern Recognition (CVPR’17), Vol. 1. 4.
[58] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014. Deep domain confusion: Maximizing
for domain invariance. arXiv preprint arXiv:1412.3474 (2014).
[59] Vladimir Naumovich Vapnik and Vlamimir Vapnik. 1998. Statistical Learning Theory. Vol. 1. Wiley, New York, NY.
[60] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. 2017. Deep hashing
network for unsupervised domain adaptation. In Proceedings of the Computer Vision and Pattern Recognition (CVPR’17).
5018–5027.
[61] Jindong Wang et al. 2018. Everything about Transfer Learning and Domain Adapation. Retrieved from http://transferlearning.xyz.
[62] Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, and Zhiqi Shen. 2017. Balanced distribution adaptation fortransfer learning. In Proceedings of the 2017 IEEE International Conference on Data Mining (ICDM’17). IEEE, 1129–
1134.
[63] Jindong Wang, Yiqiang Chen, Lisha Hu, Xiaohui Peng, and Philip S. Yu. 2018. Stratified transfer learning for crossdomain activity recognition. In Proceedings of the 2018 IEEE International Conference on Pervasive Computing andCommunications (PerCom’18).
[64] Jindong Wang, Yiqiang Chen, Han Yu, Meiyu Huang, and Qiang Yang. 2019. Easy transfer learning by exploiting
intra-domain structures. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’19).
[65] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. 2018. Visual domain adaptation
with manifold embedded distribution alignment. In Proceedings of the 2018 ACM Multimedia Conference on Multimedia
Conference. ACM, 402–410.
[66] Jindong Wang, Vincent W. Zheng, Yiqiang Chen, and Meiyu Huang. 2018. Deep transfer learning for cross-domain
activity recognition. In Proceedings of the 3rd International Conference on Crowd Science and Engineering. ACM, 16.
[67] Yong Xu, Xiaozhao Fang, Jian Wu, Xuelong Li, and David Zhang. 2016. Discriminative transfer subspace learning via
low-rank and sparse representation. IEEE Trans. Image Process. 25, 2 (2016), 850–863.
[68] Yonghui Xu, Sinno Jialin Pan, Hui Xiong, Qingyao Wu, Ronghua Luo, Huaqing Min, and Hengjie Song. 2017. A unified
framework for metric transfer learning. IEEE Trans. Knowl. Data Eng. 29, 6 (2017), 1158–1171.
[69] Jiangyan Yi, Jianhua Tao, Zhengqi Wen, and Ye Bai. 2019. Language-adversarial transfer learning for low-resource
speech recognition. IEEE/ACM Trans. Aud. Speech Lang. Process. 27, 3 (2019), 621–630.
[70] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems. 3320–3328.
[71] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. 2017.
Central moment discrepancy (cmd) for domain-invariant representation learning. In Proceedings of the International
Conference on Learning Representations (ICLR’71).
[72] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. 2018. Importance weighted adversarial nets for partial
domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8156–8164.
[73] Jing Zhang, Wanqing Li, and Philip Ogunbona. 2017. Cross-dataset recognition: A survey.arXiv preprint
arXiv:1705.04396 (2017).
[74] Jing Zhang, Wanqing Li, and Philip Ogunbona. 2017. Joint geometrical and statistical alignment for visual domain
adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).
[75] Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. 2018. Collaborative and adversarial network for unsupervised
domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3801–3809.
[76] Yu Zhang and Qiang Yang. 2017. A survey on multi-task learning. arXiv preprint arXiv:1707.08114 (2017).
[77] Erheng Zhong, Wei Fan, Jing Peng, Kun Zhang, Jiangtao Ren, Deepak Turaga, and Olivier Verscheure. 2009. Cross
domain distribution adaptation via kernel mapping. In Proceedings of the 15th ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining. ACM, 1027–1036.
[78] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Jingwu Chen, Zhiping Shi, Wenjuan Wu, and Qing He. 2019. Multirepresentation adaptation network for cross-domain image classification. Neur. Netw. 119 (2019), 214–221.

Received March 2019; revised July 2019; accepted September 2019

ACM Transactions on Intelligent Systems and Technology, Vol. 11, No. 1, Article 6. Publication date: February 2020.