2019 IEEE International Conference on Data Mining (ICDM)

Transfer Learning with Dynamic Adversarial
Adaptation Network
Chaohui Yu∗† , Jindong Wang‡ , Yiqiang Chen∗†( ) , Meiyu Huang§∗ Beijing Key Lab. of Mobile Computing and Pervasive Device, Inst. of Comp. Tech., Chinese Academy of Sciences, Beijing, China† University of Chinese Academy of Sciences, Beijing, China‡ Microsoft Research Asia, Beijing, China§ Qian Xuesen Lab. of Space Technology, China Academy of Space Technology, Beijing, China

Email:{yuchaohui17s,yqchen}@ict.ac.cn, jindong.wang@microsoft.com
6RXUFH

Abstract— The recent advances in deep transfer learning revealthat adversarial learning can be embedded into deep networks
to learn more transferable features to reduce the distribution
discrepancy between two domains. Existing adversarial domain
adaptation methods either learn a single domain discriminator toalign the global source and target distributions, or pay attention
to align subdomains based on multiple discriminators. However,in real applications, the marginal (global) and conditional (local)
distributions between domains are often contributing differently
to the adaptation. There is currently no method to dynamicallyand quantitatively evaluate the relative importance of these two
distributions for adversarial learning. In this paper, we propose
a novel Dynamic Adversarial Adaptation Network (DAAN) to
dynamically learn domain-invariant representations while quantitatively evaluate the relative importance of global and local
domain distributions. To the best of our knowledge, DAAN is
the ﬁrst attempt to perform dynamic adversarial distribution
adaptation for deep adversarial learning. DAAN is extremely
easy to implement and train in real applications. We theoreticallyanalyze the effectiveness of DAAN, and it can also be explainedin an attention stragegy. Extensive experiments demonstrate that
DAAN achieves better classiﬁcation accuracy compared to stateof-the-art deep and adversarial methods. Results also imply thenecessity and effectiveness of the dynamic distribution adaptationin adversarial transfer learning.
Index Terms—domain adaptation, dynamic, global and local,
adversarial learning

0DUJLQDO $GDSWDWLRQ

7DUJHW,,$Q8QNQRZQ7DUJHW&RQGLWLRQDO $GDSWDWLRQ Marginal or Conditional?

Fig. 1. The different effects of marginal and conditional distributions intransfer learning applications
TABLE I
Comparison between the recent methods on Ofﬁce-Home [20] dataset (Cdenotes number of classes)
MethodDynamic adaptation
Hyperparameter
Extra classiﬁer
Accuracy (%)

DANN [18]
No
λ
No
57.6

JAN [21]
No
λ
No
58.3

MEDA [7]Yes
λ, p, η, ρ
C+1
60.2

DAAN
Yes
λ
No
61.8

source and target samples into the same feature space [7],[8], [9], [10], [11]. Recent studies have indicated that deep
networks can learn more transferable features for domain
adaptation [12], [13]. The latest advances have been achievedby embedding domain adaptation modules in the pipeline ofdeep feature learning to extract domain-invariant representations [14], [15], [16], [17], [18], [19].
Recently, adversarial learning [22] has been successfully
embedded into deep networks to reduce distribution discrepancy between the source and target domains. Prior advanced
adversarial adaptation methods [18], [23], [24], [25] haveshown promising results in several domain adaptation tasks.
Most of them either learn a single domain discriminator toalign the global source and target distributions, or pay attention
to align subdomains based on multiple discriminators. For
instance, Domain-adversarial Neural Network (DANN) [18],[26] focuses on the global adversarial learning, while Multiadversarial Domain Adaptation (MADA) [25] pays attention
to the subdomain adaptation by training several domain classiﬁers. However, in real applications, the marginal (global)
and conditional (local) distributions between domains are often
contributing differently to the adaptation. For example, when
two domains are very dissimilar (source → target I in Fig. 1),
the global distribution is more important. When the global

I. I NTRODUCTION
Deep neural networks have signiﬁcantly improved the performance of diverse data mining and computer vision applications [1], [2]. In order to avoid overﬁtting and achieve better
performance, a large amount of labeled data is needed to train
a deep learning model. Unfortunately, it is often expensive
and time-consuming to acquire sufﬁcient labeled data. Thus,a natural idea is to leverage the abundant labeled samples inan existing domain (i.e. source domain) to facilitate learningin the domain to be learned (i.e. target domain).
A promising approach to solve such cross-domain learning
problems is called transfer learning, or domain adaptation [3].
The key to successful adaptation is to learn a discriminativemodel to reduce the distribution discrepancy between the twodomains. Traditional methods perform adaptation by either
reweighting samples from the source domain [4], [5], [6], or
seeking an explicit feature transformation that transforms the2374-8486/19/$31.00 ©2019 IEEE
DOI 10.1109/ICDM.2019.00088

7DUJHW ,778

distributions are close (source → target II in Fig. 1), the local
distribution should be given more attention. Two more recent
work called Balanced Distribution Adaptation (BDA) [8] andManifold Embedded Distribution Alignment (MEDA) [7] proposed to adaptively align these two distributions, while it isbased on kernel method with high computational cost. In addition, MEDA is incapable of handling large-scale data. To date,there is no effort that could dynamically evaluate the relative
importance of the marginal and conditional distributions foradversarial domain adaptation.
In this paper, we propose a novel Dynamic Adversarial Adaptation Network (DAAN) for unsupervised domain
adaptation. DAAN is able to learn domain-invariant features
through end-to-end adversarial training. The key component in
DAAN is the Dynamic Adversarial Factor, which is capableof easily, dynamically, and quantitatively evaluating the relative importance of the marginal and conditional distributions.
The adaptation can be achieved by Stochastic Gradient Descent (SGD) with the gradients computed by backpropagationin linear-time. To the best of our knowledge, DAAN is theﬁrst adversarial domain adaptation method that is able todynamically learn the relationship between the marginal andconditional distributions. Extensive experiments demonstratethat DAAN outperforms state-of-the-art methods on standard
domain adaptation benchmarks. More importantly, it is shownthat there does exist the relative importance of two distributions, of which DAAN could make accurate evaluation.
The contributions of this paper are four-fold:(1) We propose a novel dynamic adversarial adaptation
network to learn domain-invariant features. DAAN is accurate
and robust, and can be easily implemented by most deeplearning libraries.
(2) We propose the dynamic adversarial factor to easily, dynamically, and quantitatively evaluate the relative importanceof the marginal and conditional distributions in adversarial
transfer learning.
(3) We theoretically analyze the effectiveness of DAAN,
and it can also be explained in an attention stragegy.
(4) Extensive experiments on public datasets demonstrate
the signiﬁcant superiority of our DAAN in both classiﬁcation
accuracy and the estimation of the dynamic adversarial factor.

Analysis (TCA) [11] to align the marginal distributions between domains. Based on TCA, Joint Distribution Adaptation (JDA) [30] is proposed to match both marginal andconditional distributions. But these works treat the two distributions equally and fail to leverage the different importanceof distributions. Recently, Wang et al. proposed Balanaced
Distribution Adaptation (BDA) [8] and Manifold Embedded
Distribution Alignment (MEDA) [7] approaches to dynamically evaluate the different effect of marginal and conditional distributions and achieved the state-of-the-art resultson domain adaptation. However, MEDA is based on kernel
method and requires to train several linear classiﬁers in each
iteration. DAAN is signiﬁcantly different from MEDA in
two folds as shown in Table I. Firstly, MEDA uses shallow
features to learn the adaptive factor, while DAAN uses deep
adversarial representations for end-to-end learning. Secondly,
MEDA uses extra linear classiﬁers to learn the adaptive factor,
while DAAN directly uses the adversarial features, which ismore efﬁcient.
In recent years, deep networks can learn more transferable
features for domain adaptation [12], [13], by disentangling
explanatory factors of variations behind domains compared to
traditional methods. Most work on deep domain adaptation
is based on discrepancy measurement. For instance, Correlation Alignment (CORAL) [17], Kullback-Leibler (KL) divergence [31], Maximum Mean Discrepancy (MMD) [32], [21],[33], [34], [15], and Central Moment Discrepancy (CMD) [35]are used to reduce the distribution divergence between domains. However, there is no effective deep learning methodthat can dynamically align the marginal and conditional distributions.
B. Domain-adversarial Learning
As a special case of deep domain adaptation, domainadversarial learning has been popular in recent years. In thiscase, a domain discriminator that classiﬁes whether a data
point is drawn from the source or target domain is used toencourage domain confusion through an adversarial objective to minimize the distance between the source and target
distributions [18]. Adversarial learning has been explored in
Generative Adversarial Networks (GANs) [22]. And Generative Multi-Adversarial Network (GMAN) [36] extends GANs
to multiple discriminators including formidable adversary andforgiving teacher, which signiﬁcantly eases model training.
Recently, we have witnessed considerable research [18],[25], [37], [38] for adversarial domain adaptation. DANN [18]aligns the whole source and target distributions with a global
domain discriminator. MADA [25] captures multi-mode structures to enable ﬁne-grained alignment of different data distributions based on multiple domain discriminators. Co-DA [37]constructs multiple diverse feature spaces and aligns source
and target distributions in each of them individually. The
proposed DAAN is also based on adversarial learning, while
it signiﬁcantly outperforms existing methods by dynamicallyevaluating the relative importance of the marginal and conditional distributions.

II. R ELATED W ORK
A. Unsupervised Domain Adaptation
Unsupervised domain adaptation is a speciﬁc area of transfer learning [3], which is to learn a discriminative model inthe presence of the domain shift between domains. There aremainly two categories: traditional (shallow) learning and deeplearning.
Traditional (shallow) learning methods can mainly be divided into two categories: (1) Subspace learning. Subspace
Alignment (SA) [27] aligns the base vectors of both domains
and Subspace Distribution Alignment (SDA) [28] extends SA
by adding the subspace variance adaptation. CORAL [29]aligns subspaces in second-order statistics. (2) Distributionalignment. Pan et al. proposed the Transfer Component779

III. DYNAMIC A DVERSARIAL A DAPTATION N ETWORK

would lead to better performance [21] since both distributions
are helpful in learning domain-invariant features. However, in
real applications, these two different distributions may havetotally different contributions to the domain discrepancy. In
real applications, it is extremely challenging to account for
the relative importance of these two distributions. Therefore,
we need to dynamically and quantitatively evaluate theirimportance in domain adaptation.
Recently, a Manifold Embedded Distribution Alignment (MEDA) [7] approach has been proposed to computethe weights of marginal and conditional distributions. MEDA
learns a domain-invariant classiﬁer in the Reproducing Kernel
Hilbert Space (RKHS) while evaluating the weights of thetwo distributions using the proxy A-distance [39]. However,
MEDA has to train 1 + C extra linear classiﬁers in each iteration, which is computationally expensive and time-consuming.
Furthermore, MEDA can only be applied to small-scale data
since it calculates the pseudo-inverse of all the samples eachtime thus it cannot be deployed online. To sum up, it is extremely challenging to easily, dynamically, and quantitatively
evaluate the relative importance of both distributions, while
the system still remains scalable to large-scale data.
In this paper, we make key technical improvements
by proposing the Dynamic Adversarial Adaptation Network (DAAN) to address the above challenge. As shown in
Fig. 2, DAAN is based on the well-established generative
adversarial networks (GAN) [22] that aims at learning domaininvariant features via adversarial training. In DAAN, high-levelfeatures f are extracted by a feature extractor (Gf , the bluepart). Then, the adaptation of marginal and conditional distributions are achieved by the Global domain discriminator (Gd ,the purple part) and Local domain discriminator (Gcd , the greenpart), respectively. Most importantly, DAAN proposes a novel
Dynamic Adversarial Factor (ω, the yellow part) to performeasy, dynamic, and quantitative evaluation of these two distributions. Along with the label classiﬁer (Gy , the orange part),
the parameters of DAAN can be trained efﬁciently with theGradient Reversal Layer (GRL) [26].
In the next sections, we will ﬁrst introduce the label
classiﬁer, global domain discriminator, and local domain discriminator. Then, the dynamic adversarial factor is presented
in Section III-D. Finally, we show the loss function of DAAN
and how to train DAAN.
1) Label Classiﬁer: The label classiﬁer (Gy , the orange
part in Fig. 2) is trained to discriminate the label of the
input samples from the source domain. Thus, the supervised
information on Ds can be utilized. Its training objective is across-entropy loss, which can be formulated as:

In this section, we introduce the proposed Dynamic Adversarial Adaptation Network (DAAN).
A. Problem Deﬁnition
In unsupervised domain adaptation, we are given a sourcesdomain Ds = {(xsi , yis )}ni=1
of ns labeled examples and atof nt unlabeled examples. Ds
target domain Dt = {xtj }nj=1
and Dt have the same label space, i.e. xi , xj ∈ Rd whered is the dimensionality. The marginal distributions between
two domains are different, i.e. Ps (xs ) = Pt (xt ). The goal ofdeep UDA is to design a deep neural network that enables
learning of transfer classiﬁers y = f (x) that formally reduces
the shifts in the distributions of two domains such that thetarget risk t (f ) = E(x,y)∼q [f (x) = y] can be bounded byusing the source domain while achieving better performanceon the target domain.
B. Adversarial Learning for Domain Adaptation
Domain adversarial adaptation methods borrow the idea of
GAN [22] to help learn transferable features. The adversarial
learning procedure is a two-player game, where the ﬁrst player
is the domain discriminator Gd trained to distinguish thesource domain from the target domain, and the second player
is the feature extractor Gf that tries to confuse the domain
discriminator by extracting domain-invariant features. The twoplayers are trained adversarially: the parameters θf of feature
extractor Gf are learned by maximizing the loss of domain
discriminator Gd , while the parameters θd of Gd are trained byminimizing the loss of the domain discriminator. In addition,
the loss of the label classiﬁer Gy is also minimized. The loss
function can be formalized as:
1 
L (θf , θy , θd ) =Ly (Gy (Gf (xi )) , yi )nsxi ∈Dsλ
Ld (Gd (Gf (xi )) , di ) ,−ns + ntxi ∈(Ds ∪Dt )(1)
where λ is a trade-off parameter and Ly and Ld denote thelabel classiﬁer loss and domain discriminator loss. Since there
are no labels for the target domain, di means the domain labelof the input samples (di for source domain is 0, di for target
domain is 1). After the training converges, the parametersθ̂f , θ̂y , θ̂d will deliever a saddle point of Eq. (1):θ̂f , θ̂y = arg minθf ,θy L (θf , θy , θd ) (2)θ̂d = arg maxθd L (θf , θy , θd )C. Dynamic Adversarial Adaptation Network
Most state-of-the-art domain adaptation approaches [18],[25], [37], [38] are adversarial learning methods. Competitive
results are achieved by either aligning the marginal distributions [32], [26] (Source → Target I in Fig. 1), or aligning theconditional distributions [25] (Source → Target II in Fig. 1). It
has been shown that aligning these two distributions together

Ly = −C
1  
Pxi →c log Gy (Gf (xi )),
ns
c=1

(3)

xi ∈Dswhere C is the number of classes, Pxi →c is the probability of
xi belonging to class c, Gy is the label classiﬁer and Gf is
the feature extractor.

780

/DEHO&ODVVLILHU
6RIWPD[)HDWXUH([WUDFWRU*OREDO'RPDLQ'LVFULPLQDWRUP*5//RFDO6XEGRPDLQ'LVFULPLQDWRU
RP
*5/

*5/

Fig. 2. (Best viewed in color) The architecture of the proposed Dynamic Adversarial Adaptation Network (DAAN). DAAN consists of a deep feature extractor
Gf (blue), a label classiﬁer Gy (orange), a global domain discriminator (Gd , purple), C local subdomain discriminators (Gcd for c ∈ {1, · · · , C}, green),
and a dynamic adversarial factor module (ω, yellow). ⊕ denotes the plus operator while ⊗ is the product operator. f is the extracted deep features, ŷ is thepredicted label, Ly and Ld are the classiﬁcation loss and domain loss. dˆ and dˆc are the predicted domain label. GRL stands for Gradient Reversal Layer.

2) Global Domain Discriminator: The global domain discriminator (Gd , the purple part in Fig. 2) is designed to alignthe marginal (global) distributions between the source and target domains. The general idea of global domain discriminatorfollows the Domain-adversarial Neural Network (DANN) [26],which has been described in the previous section. In DAAN,
we calculate the loss of the global domain discriminator as:
Lg =1
ns + n t

Ld (Gd (Gf (xi )), di ),D. Dynamic Adversarial Factor ω
In this section, we introduce how to dynamically evaluate
the global and local distributions. It is extremely challenging to design such a dynamic scheme for adversarial learning. Intuitively, there are two natural ideas to acquire ω:
Random guessing and Average search. Random guessing
randomly picks a value of ω in [0, 1], then performs DAAN
using the corresponding value to get the result. This process
can be repeated t times and the ﬁnal result can be obtainedby averaging all the results. Average search picks the value ofω = 0, 0.1, · · · , 1.0 to perform DAAN 11 times and uses theaverage results as the ﬁnal result. However, both of these twoideas are computationally expensive for adversarial domain
adaptation.
In this paper, we propose the dynamic adversarial factor (ω,the yellow part in Fig. 2) to easily, dynamically, and quantitatively evaluate the relative importance of the marginal andconditional distributions. Compared to MEDA [7] that needs
to build 1 + C binary classiﬁers for the calculation of its
adaptive factor, DAAN is able to update the value of the
dynamic adversarial factor within the network. Firstly, insteadof using the shallow features, we use the deep adversarial
representations to learn and update ω, which makes DAAN
more robust and accurate. Secondly, DAAN directly uses theloss of the domain discriminators to automatically ﬁne-tune thedynamic adversarial factor, which is easier and more efﬁcient.
To be more speciﬁc, the global domain distributions andthe local domain distributions can be seen as the marginal andconditional distributions, respectively. Therefore, in DAAN,
we denote the global A-distance of the global domain discriminator as:(4)xi ∈Ds ∪Dt

where Ld is the domain discriminator loss (cross-entropy), Gf
is the feature extractor, and di is the domain label of the input
sample xi .
3) Local Domain Discriminator: The local domain discriminator (Gcd , the green part in Fig. 2) is designed to alignthe conditional (local) distributions between the source andtarget domains. Compared to the global domain discriminator,
local domain discriminator is able to align the multi-mode
structure in two distributions, thus it can perform more ﬁnegrained domain adaptation.
To be concrete, the domain discriminator Gd can be split
into C class-wise domain discriminators Gcd , each is responsible for matching the source and target domain data associatedwith class c. The output of the label predictor Gy (xi ) to eachdata point xi can be used to indicate how much each data
points xi should be attended to the C domain discriminators
Gcd , c = 1, ..., C. The loss function of the local domain
discriminator can be calculated as:
Ll =

C

1
ns + nt c=1



Lcd (Gcd (ŷic Gf (xi )), di ),(5)xi ∈Ds ∪Dt

dA,g (Ds , Dt ) = 2(1 − 2(Lg )).

where Gcd and Lcd are the domain discriminator and its crossentropy loss associated with class c, respectively. ŷic is thepredicted probability distribution over the class c of the input
sample xi , and di is the domain label of the input sample xi .

(6)

And we calculate the local A-distance as:
dA,l (Dsc , Dtc ) = 2(1 − 2(Lcl )),

781

(7)

where Dsc and Dtc denote samples from class c and Lcl is the
local subdomain discriminator loss over class c. Eventually,
the dynamic adversarial factor ω can be estimated as:ω̂ =dA,g (Ds , Dt ).
C
dA,g (Ds , Dt ) + C1 c=1 dA,l (Dsc , Dtc )optimization of DAAN. We can either optimize Eq. (11) directly according to [26], or we can optimize the two objectives
in Eq. (2) iteratively.
F. Discussions(8)

In theory, the risk of DAAN can be bounded by thefollowing theorem since it is designed by directly minimizing
the target risk according to [39]:Theorem 1 Let h ∈ H be a hypothesis, s (h) and t (h)be the expected risks on the source and target domain,
respectively, thenNote that there is no need to explicitly build extra classiﬁers
in order to compute the local distances such as MEDA [7]. In
DAAN, they can be easily implemented by taking advantagesof the global and local domain discriminators. More speciﬁcally, ω is initialized as 1 in the ﬁrst epoch. After each epoch,
the pseudo labels of the target domain can be obtained. Then,the local distance for class c can be easily computed as:
Lcl = CrossEntropy(dˆc , dc ),t (h)  s (h) + dH (p, q) + C0 ,where C0 is a constant for the complexity of hypothesis
and plus the risk of an ideal hypothesis for both domains.
More importantly, according to [39], dH (p, q) is H-divergence
between domains, which can be approximately measured by
the A-distances in Eq. (4) and Eq. (5). In fact, we can regard
the dynamic distribution adaptation of DAAN as the dynamic
version of H-divergence, which could learn both global andlocal divergences between domains. Therefore, the risk of
DAAN can be theoretically bounded.
DAAN can also be explained using the attention mechanism [40]. Attention plays a critical role in human visualexperience. In a computer vision task, attention tries to learn
the important factors of the images. In a machine translationtask, attentions helps to learn the important hidden statesof the encoders. In transfer learning, we can regard that
DAAN is learning the dynamic importance of the marginal
and conditional distributions using neural networks. Therefore,
it perceives the accurate information about distributions using
the dynamic adversarial factor.
DAAN is signiﬁcantly different from existing adversarial
adaptation methods. Speciﬁcally, compared with global domain adaptation methods [18], [26] and the local subdomain
adaptation methods [25], DAAN is able to perform dynamic
adversarial distribution alignment by quantitatively calculating the relative importance of global and local distributions
with a novel dynamic adversarial factor ω. Compared with
MEDA [7], DAAN uses deep adversarial representations to
ﬁne-tune ω without training extra classiﬁers, which makes ourestimation of ω signiﬁcantly more accurate, easy, and efﬁcient.

(9)

where dˆc = [dˆcs ; dˆct ] is the concatenation of the predictions
output by the c-th domain discriminator dc , and dc = [0; 1]ˆcˆcwith 0 ∈ R|ds |×1 and 1 ∈ R|dt |×1 is the concatenation of thetrue domain labels (suppose the source domain has label 0 andtarget domain has label 1). Similarly, the global distances canbe obtained. The calculation of the dynamic adversarial factor
can be performed after each epoch of iteration. Eventually,
DAAN will learn a rather robust dynamic adversarial factor
as the training converges.
E. Learning Procedure
DAAN mainly consists of three components: Label Classiﬁer (Eq. (3)), Global Domain Discriminator (Eq. (4)), andLocal Subdomain Discriminator (Eq. (5)). Integrating allcomponents, the learning objective of DAAN can ﬁnally beformulated as:
L(θf , θy , θd , θdc |Cc=1 ) = Ly − λ((1 − ω)Lg + ωLl ),

(12)

(10)

where λ is a trade-off parameter.
It is worth noting that although DAAN involves two hyperparameters (λ and ω), the value of ω can be self-calculatedby the network. Therefore, DAAN remains the same sample
and efﬁcient as other popular adversarial methods [26], [21].
When ω → 0, it means that the global distribution alignment
is more important (Target I in Fig. 1), and DAAN will
degenerate to DANN [26]. When ω → 1, it means that
global distributions between two domains are relatively small,so the local subdomain distributions of each class is dominant (Target II in Fig. 1). In this case, DAAN will degenerate
to MADA [25]. Note that in real applications, the marginal
and conditional distributions are not determined. Therefore,
by learning the dynamic adversarial factor ω, DAAN can beapplied to diverse domain adaptation scenarios.
Denoting Θ = {θf , θy , θd , θdc |Cc=1 } as all the parameters tobe learned, the gradient of Eq. (10) can be computed as:

IV. E XPERIMENTS
In this section, we evaluate the proposed DAAN against
several state-of-the-art transfer learning methods on unsupervised domain adaptation problems. DAAN is validated on two
popular datasets: ImageCLEF-DA [21] and Ofﬁce-Home [20].
The code of DAAN is released at http://transferlearning.xyz.
A. Datasets
Examples of the two datasets are shown in Figure 3.
ImageCLEF-DA is a benchmark dataset for ImageCLEF
2014 domain adaptation challenge, and it is collected by
selecting the 12 common categories shared by the following public datasets and each of them is considered as aΔ((1 − ω)Lg + ωLl )
ΔLy
−λ
(11)
ΔΘ
ΔΘ
DAAN can be trained efﬁciently by the Stochastic Gradient
Descent (SGD) algorithm. There are two alternatives for the
ΔΘ =

782

TABLE II
Accuracy(%) on Ofﬁce-Home for unsuperevised domain adaptation.

MethodResNet [1]
DAN [32]DANN [18]
JAN [21]
MEDA [7]
DAAN

A→C
34.9
43.6
45.6
45.9
46.6
50.5

A→P
50.0
57.0
59.3
61.2
68.9
65.0

A→R
58.0
67.9
70.1
68.9
68.8
73.7

C→A
37.4
45.8
47.0
50.4
49.0
53.7

C→P
41.9
56.5
58.5
59.7
66.4
62.7

C→R
46.2
60.4
60.9
61.0
66.1
64.6

P→A
38.5
44.0
46.1
45.8
51.8
53.5

P→C
31.2
43.6
43.7
43.4
45.0
45.2

P→R
60.4
67.7
68.5
70.3
72.9
74.0

R→A
53.9
63.1
63.2
63.9
61.2
66.3

R→C
41.2
51.5
51.8
52.4
50.3
54.0

R→P
59.9
74.3
76.8
76.8
76.0
78.8

AVG
46.1
56.3
57.6
58.3
60.2
61.8

TABLE III
Accuracy(%) on ImageCLEF-DA for unsuperevised domain adaptation.

&DOWHFK,PDJH1HW

MethodResNet [1]
DDC [33]
DAN [32]
RTN [41]DANN [18]D-CORAL [17]
JAN [21]
MADA [25]
CAN [14]
MEDA [7]DAAN

3DVFDO92&

Fig. 3. Datasets. Up: ImageCLEF-DA; Down: Ofﬁce-Home

I→P
74.8
74.6
75.0
75.6
75.0
76.9
76.8
75.0
78.2
78.1
78.3

P→I
83.9
85.7
86.2
86.8
86.0
88.5
88.0
87.9
87.5
90.4
91.3

I→C
91.5
91.1
93.3
95.3
96.2
93.6
94.7
96.0
94.2
93.1
94.4

C→I
78.0
82.3
84.1
86.9
87.0
86.8
89.5
88.8
89.5
86.4
88.0

C→P
65.5
68.3
69.8
72.7
74.3
74.0
74.2
75.2
75.8
73.2
73.5

P→C
91.2
88.8
91.3
92.2
91.5
91.6
91.7
92.2
89.2
91.7
94.3

AVG
80.7
81.8
83.3
84.9
85.0
85.2
85.8
85.8
85.7
85.5
86.6

C. Implementation Details

domain: Caltech − 256 (C), ImageN et ILSV RC 2012(I), P ascal V OC 2012 (P). There are 50 images in eachcategory and 600 images in each domain. We use all domain
combinations and build 6 transfer tasks: I→P, P→I, I→C,
C→I, P→C and C→P.
Ofﬁce-Home is a new dataset which consists 15,588 images, which is much larger than ImageCLEF-DA. It consists of images from 4 different domains: Artistic images(A), Clip Art (C), P roduct images (P) and Real −W orld images (R). For each domain, the dataset contains
images of 65 object categories collected in ofﬁce and homesettings. Similarly, we use all domain combinations and construct 12 transfer tasks.

We implement all deep methods based on the PyTorch [42]framework, and ﬁne-tune from ResNet-50 [1] models pretrained on the ImageNet dataset [43]. We obtain the results of
MEDA by running it on the features pre-trained by ResNet.
For all the unsupervised domain adaptation tasks, we ﬁne-tune
all convolutional and pooling layers and train the classiﬁer
layer via backpropagation. Since the classiﬁer is trained fromscratch, we set its learning rate to be 10 times that of the otherlayers. The mini-batch Stochastic Gradient Descent (SGD)with momentum of 0.9 is taken as optimization scheme, andthe learning rate changing strategy follows existing work [18]:the learning rate is not selected by a grid search due to highcomputational cost, it is adjusted during SGD using theseη0formulas [26]: ηk = (1+αk)
β , where k is the training progress
linearly changing from 0 to 1, η0 = 0.01, α = 10 and
β = 0.75. We ﬁx λ = 1, batchsize = 32 in DAAN all
the time. Other Hyperparameters are tuned via transfer crossvalidation [44]. Following [26], [41], classiﬁcation accuracy
is used as the evaluation metric. The labels for the target
domain are only used for evaluation. The results are obtainedby running the method 10 times to get the average accuracy.

B. Baselines
We compare our proposed Dynamic Adversarial Adaptation
Network (DAAN) with several state-of-the-art deep unsupervised domain adaptation methods:
Deep residual learning [1]Deep Domain Confusion (DDC) [33]• Deep Adaptation Network (DAN) [32]• Residual Transfer Network (RTN) [41]• Domain Adversarial Neural Networks (DANN) [18]• Deep CORAL (D-CORAL) [17]• Joint Adaptation Networks (JAN) [21]• Multi-Adversarial Domain Adaptation (MADA) [25]• Collaborative and Adversarial Network (CAN) [14]• Manifold Embedded Distribution Alignment (MEDA) [7]••

D. Results
The classiﬁcation accuracy on the ImageCLEF-DA dataset
based on ResNet is shown in Table III. DAAN outperforms allcomparison methods on most transfer tasks. It is noteworthy
that DAAN outperforms DANN and MADA. Table II shows
the results of DAAN and several baselines on the morechallenging Ofﬁce-Home dataset. DAAN also outperforms allcomparison methods on most tasks.

783

TABLE IV
Error comparison between our evaluation of ω and Average search and MEDA (the results of grid search are 0)
Task
Avg search
MEDA
DAAN

I→P
1.50
0.48
0.15

P→I
2.30
0.67
0.24

I→C
1.70
0.34
0.1

C→I
2.51
0.50
0.32

Fig. 4. Performance of several tasks when searching ω ∈ [0, 1].

A→R
1.57
1.49
0.20

R→A
1.20
0.89
0.29

A→C
1.32
0.57
0.42

C→A
2.00
1.32
0.36

AVG
1.76
0.78
0.26

Fig. 5. (Best viewed in color) The performance w.r.t different calculation
method of ω: Random guessing, Average search, MEDA, and our DAAN

Combining these results, more insightful conclusions canbe made. (1) The adversarial based methods (DANN [18],
MADA [25], and our DAAN) usually perform better than
the non-adversarial based methods (DDC [33], DAN [32],
RTN [41]), which indicates that domain-adversarial learning
is important for domain adaptation. (2) Compared with therecent domain adaptation methods MEDA and CAN, DAAN
achieves better performance which proves our method is more
effective. (3) In contrast to other latest adversarial methods, especially DANN [18] and MADA [25], our DAAN shows better
performance. During training, we also noticed that our DAAN
is able to converge quickly (within ¡ 30 epoches) comparedto other methods. This implies its fast training performancewith steady results (which will be shown in later experiments).
This indicates that DAAN is capable of performing dynamic
adversarial distribution alignment by quantitatively calculating
the relative importance of global and local distributions.

can be veriﬁed in BDA [8] and MEDA [7]), but also inadversarial transfer learning. Moreover, We ﬁnd that the valueof optimal ω varies on different tasks and even for the sametask, ω may have several optimal values. This may be becauseof different feature representations learned by calculating ω.
Again, it implies the importance of ω in adversarial domain
adaptation problems.
To answer the second question, we compare the accuracyof domain adaptation tasks contributed by different calculation
method of ω: Random guessing (t = 20), Average search,
MEDA [7], and our DAAN. For a fair study, the resultsof MEDA is obtained by replacing the dynamic adversarial
factor in DAAN with the adaptive factor in MEDA. Note that
there is no ground truth for ω. Instead, we run DAAN andrecord its accuracy by grid search ω ∈ {0, 0.1, ..., 0.9, 1.0}
to ﬁnd the optimal results as the ground truth. We use thelabels of the target domain only for evaluation. The results
are shown in Fig. 5. Additionally, we also list some results inTable IV to show the errors of different calculation methods.
Combining the results, we can conclude that our evaluation ofω signiﬁcantly and consistently outperforms other comparison
methods. In addition, our evaluation is more efﬁcient than theother three methods since it only requires to run the wholenetwork once while other methods require to run DAAN
several times to get stable results. Compared with MEDA,
our evaluation is more efﬁcient and accurate since it uses theadversarial representations and does not need to train extra
linear classiﬁers. Furthermore, it is worth noting that ourevaluation of ω is extremely close to the grid search results(which can never be reached in real applications). Therefore,
the proposed dynamic adversarial factor is necessary and ourE. Analysis of the Importance of the Dynamic Adversarial
Factor ω
In this section, we evaluate the importance of dynamic
adversarial factor ω in DAAN. To this end, there are twoquestions to be answered: 1) Is it necessary to consider
the different effects of marginal and conditional distributionsin adversarial domain adaptation? And 2) Is our evaluation
method of ω effective?
To answer the ﬁrst question, we randomly pick two tasks
from Ofﬁce-Home and ImageCLEF-DA and draw the resultsof DAAN under different ω in Fig. 4. It can be seen thatthe classiﬁcation accuracy varies with different values of ω,which indicates the necessity to consider the different effectsof the marginal (global) distributions and conditional (local)
distributions not only in shallow domain adaptation (which784

(a) JAN: P → I(b) DAAN: P → I

Fig. 6. (Best viewed in color) The t-SNE visualization of network activation.
(a) and (b) are the learned representations on task P → I, respectively.

(a) ω(b) Loss

Fig. 7. Value change of dynamic adversarial factor ω and loss w.r.t. iterations
TABLE V
Ablation study of DAAN

DatasetImageCLEF-DA
Ofﬁce-Home

DANN(ω = 1)
85.0
57.6

MADA
(ω = 0)85.8-

JAN(ω = 0.5)
85.8
58.3

importance of global domain distributions and local subdomain
distributions.
3) Convergence Analysis: In this section, we evaluate theconvergence of DAAN. On the same DAAN architecture, we
compare the change of ω between DAAN and MEDA w.r.t.
iterations in Fig. 7(a). Additionally, their loss can be seen in
Fig. 7(b). From these results, we can observe: (1) DAAN canreach a quick and steady convergence after 20 epochs. (2) The
dynamic adversarial factor ω can also reach a steady value after several iterations, while the adaptive factor of MEDA takes
more iterations. These results demonstrate that the proposed
DAAN can not only reach competitive performances, it canalso be trained easily with steady results.

DAAN
86.4
61.6

evaluation of ω is more effective and efﬁcient.
F. Effectiveness Analysis
In this section, we analyze the effectiveness of DAAN
from several aspects: ablation study, feature visualizations, andconvergence analysis.
1) Ablation Study: We compare the performance of DAAN
with DANN (ω = 0), MADA (ω = 1), and JAN (ω = 0.5).
All these methods can be seen as special cases of our DAAN.
The average results on each dataset in Table V indicate that
it is not enough to only align the marginal or conditional
distributions, or aligning them with equal weights. Therefore,
the proposed DAAN is able to perform dynamic distributionalignment between domain and achieve better performance.
This property of DAAN is extremely important in real applications since given an unknown target domain, we cannever know the contributions of either marginal or conditional
distribution in domain divergence. DAAN makes it possible toeasily, dynamically, and quantitatively evaluate their relative
importance in adversarial learning.
2) Feature Visualization: To further evaluate the performance of DAAN, we visualize the network activations on
task P → I (12 classes) learned by JAN and DAAN using
t-SNE embeddings [12] in Fig. 6(a)-6(b). Red circles are
the source samples and blue triangles are the target samples.
The visualization results reveal some important observations.
(1) As for the results of JAN, the distributions between thesource and target domains are not aligned very well anddifferent categories are not well discriminated clearly. (2) In
contrast, for the representations learned with our DAAN,
not only the distributions between the source and target
domains are aligned very well, different categories can also
be discriminated more clearly. This ensures that our DAAN
can achieve better performance. The above observations suggest that DAAN is able to learn more representative andtransferable features by quantitatively calculating the relative

V. C ONCLUSIONS AND F UTURE W ORK
In this paper, we propose a novel Dynamic Adversarial
Adaptation Network (DAAN) for adversarial transfer learning. DAAN is able to learn a domain-invariant network
while performing dynamic adversarial distribution alignment
to quantitatively evaluate the relative importance of marginal(global) domain distributions and conditional (local) subdomain distributions. To the best of our knowledge, DAAN is
the ﬁrst attempt to perform easy, dynamic, and quantitativeevaluation of these two distributions in adversarial neuralnetworks. DAAN can be easily implemented and used in realdomain adaptation tasks. Experimental results demonstrate that
DAAN achieves superior performance compared to state-ofthe-art deep methods.
DAAN is a general transfer learning and domain adaptation
approach and it can be applied to a large amount of classiﬁcation related applications such as object detection, image
segmentation, and visual tracking. In the future, we plan toextend DAAN for the more challenging cross-domain data
mining problems.
VI. ACKNOWLEDGMENT
The ﬁrst two authors contributed equally. This work
is supported in part by National Key R & D Planof China (No.2017YFB1002802), National Natural Science
Foundation of China (No.61572471), Beijing Municipal Science & Technology Commission (No.Z171100000117001)
and Chinese Academy of Sciences Research Equipment Development Project under Grant (No.YZ201527).

785

R EFERENCES[23] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2017, pp. 7167–7176.
[24] S. Motiian, Q. Jones, S. Iranmanesh, and G. Doretto, “Few-shot adversarial domain adaptation,” in Advances in Neural Information Processing
Systems, 2017, pp. 6670–6680.
[25] Z. Pei, Z. Cao, M. Long, and J. Wang, “Multi-adversarial domain
adaptation,” in Thirty-Second AAAI Conference on Artiﬁcial Intelligence,
2018.
[26] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training ofneural networks,” The Journal of Machine Learning Research, vol. 17,
no. 1, pp. 2096–2030, 2016.
[27] B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars, “Unsupervised
visual domain adaptation using subspace alignment,” in Proceedings ofthe IEEE international conference on computer vision, 2013, pp. 2960–
2967.
[28] B. Sun and K. Saenko, “Subspace distribution alignment for unsupervised domain adaptation.” in BMVC, 2015, pp. 24–1.
[29] B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy domain
adaptation.” in AAAI, vol. 6, no. 7, 2016, p. 8.
[30] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu, “Transfer featurelearning with joint distribution adaptation,” in Proceedings of the IEEE
international conference on computer vision, 2013, pp. 2200–2207.
[31] F. Zhuang, X. Cheng, P. Luo, S. J. Pan, and Q. He, “Supervised
representation learning: Transfer learning with deep autoencoders,” in
Twenty-Fourth International Joint Conference on Artiﬁcial Intelligence,
2015.
[32] M. Long, Y. Cao, J. Wang, and M. I. Jordan, “Learning transferable
features with deep adaptation networks,” in International Conference
on Machine Learning (ICML), 2015.
[33] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell, “Deepdomain confusion: Maximizing for domain invariance,” arXiv preprint
arXiv:1412.3474, 2014.
[34] H. Yan, Y. Ding, P. Li, Q. Wang, Y. Xu, and W. Zuo, “Mind the classweight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2017, pp. 2272–2281.
[35] W. Zellinger, T. Grubinger, E. Lughofer, T. Natschläger, and
S. Saminger-Platz, “Central moment discrepancy (cmd) for domaininvariant representation learning,” in International Conference on Learning Representations (ICLR), 2017.
[36] I. Durugkar, I. Gemp, and S. Mahadevan, “Generative multi-adversarial
networks,” arXiv preprint arXiv:1611.01673, 2016.
[37] A. Kumar, P. Sattigeri, K. Wadhawan, L. Karlinsky, R. Feris, B. Freeman, and G. Wornell, “Co-regularized alignment for unsupervised
domain adaptation,” in Advances in Neural Information Processing
Systems, 2018, pp. 9367–9378.
[38] S. Xie, Z. Zheng, L. Chen, and C. Chen, “Learning semantic representations for unsupervised domain adaptation,” in International Conference
on Machine Learning, 2018, pp. 5419–5428.
[39] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of representations for domain adaptation,” in Advances in neural information
processing systems, 2007, pp. 137–144.
[40] S. Zagoruyko and N. Komodakis, “Paying more attention to attention:
Improving the performance of convolutional neural networks via attention transfer,” in International Conference on Learning Representations(ICLR), 2016.
[41] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Unsupervised domain
adaptation with residual transfer networks,” in Advances in Neural
Information Processing Systems, 2016, pp. 136–144.
[42] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation inpytorch,” 2017.
[43] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large
scale visual recognition challenge,” International journal of computer
vision, vol. 115, no. 3, pp. 211–252, 2015.
[44] E. Zhong, W. Fan, Q. Yang, O. Verscheure, and J. Ren, “Cross validation
framework to choose amongst models and datasets for transfer learning,”
in Joint European Conference on Machine Learning and Knowledge
Discovery in Databases. Springer, 2010, pp. 547–562.

[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classiﬁcation
with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[3] S. J. Pan, Q. Yang et al., “A survey on transfer learning,” IEEE
Transactions on knowledge and data engineering, vol. 22, no. 10, pp.
1345–1359, 2010.
[4] J. Huang, A. Gretton, K. Borgwardt, B. Schölkopf, and A. J. Smola,“Correcting sample selection bias by unlabeled data,” in Advances inneural information processing systems, 2007, pp. 601–608.
[5] Y. Chen, J. Wang, M. Huang, and H. Yu, “Cross-position activity
recognition with stratiﬁed transfer learning,” Pervasive and Mobile
Computing, vol. 57, pp. 1–13, 2019.
[6] J. Wang, Y. Chen, H. Yu, M. Huang, and Q. Yang, “Easy transfer
learning by exploiting intra-domain structures,” in IEEE International
Conference on Multimedia and Expo (ICME), 2019.
[7] J. Wang, W. Feng, Y. Chen, H. Yu, M. Huang, and P. S. Yu, “Visual
domain adaptation with manifold embedded distribution alignment,” in2018 ACM International Conference on Multimedia (ACM MM). ACM,
2018, pp. 402–410.
[8] J. Wang, Y. Chen, S. Hao, W. Feng, and Z. Shen, “Balanced distribution
adaptation for transfer learning,” in 2017 IEEE International Conferenceon Data Mining (ICDM). IEEE, 2017, pp. 1129–1134.
[9] B. Gong, Y. Shi, F. Sha, and K. Grauman, “Geodesic ﬂow kernel
for unsupervised domain adaptation,” in 2012 IEEE Conference on
Computer Vision and Pattern Recognition. IEEE, 2012, pp. 2066–2073.
[10] J. Wang, Y. Chen, L. Hu, X. Peng, and S. Y. Philip, “Stratiﬁed
transfer learning for cross-domain activity recognition,” in 2018 IEEE
International Conference on Pervasive Computing and Communications(PerCom). IEEE, 2018, pp. 1–10.
[11] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation via
transfer component analysis,” IEEE Transactions on Neural Networks,
vol. 22, no. 2, pp. 199–210, 2011.
[12] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and
T. Darrell, “Decaf: A deep convolutional activation feature for genericvisual recognition,” in International conference on machine learning,
2014, pp. 647–655.
[13] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable arefeatures in deep neural networks?” in Advances in neural information
processing systems, 2014, pp. 3320–3328.
[14] W. Zhang, W. Ouyang, W. Li, and D. Xu, “Collaborative and adversarial
network for unsupervised domain adaptation,” in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, 2018,
pp. 3801–3809.
[15] Y. Zhu, F. Zhuang, J. Wang, J. Chen, Z. Shi, W. Wu, and Q. He,“Multi-representation adaptation network for cross-domain image classiﬁcation,” Neural Networks, 2019.
[16] C. Yu, J. Wang, Y. Chen, and Z. Wu, “Accelerating deep unsupervised domain adaptation with transfer channel pruning,” arXiv preprint
arXiv:1904.02654, 2019.
[17] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep
domain adaptation,” in European Conference on Computer Vision.
Springer, 2016, pp. 443–450.
[18] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by
backpropagation,” in International Conference on Machine Learning(ICML), 2015.
[19] J. Wang, V. W. Zheng, Y. Chen, and M. Huang, “Deep transfer
learning for cross-domain activity recognition,” in Proceedings of the 3rdInternational Conference on Crowd Science and Engineering. ACM,
2018, p. 16.
[20] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan,“Deep hashing network for unsupervised domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2017, pp. 5018–5027.
[21] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learningwith joint adaptation networks,” in ICML, 2017.
[22] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” inAdvances in neural information processing systems, 2014, pp. 2672–
2680.

786