This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

1Deep Subdomain Adaptation Network for
Image Classification
Yongchun Zhu, Fuzhen Zhuang , Jindong Wang, Guolin Ke, Jingwu Chen,
Jiang Bian, Hui Xiong, Fellow, IEEE, and Qing He

Abstract— For a target task where the labeled data areunavailable, domain adaptation can transfer a learner from a different source domain. Previous deep domain adaptation methods
mainly learn a global domain shift, i.e., align the global source
and target distributions without considering the relationships
between two subdomains within the same category of different
domains, leading to unsatisfying transfer learning performance
without capturing the fine-grained information. Recently, more
and more researchers pay attention to subdomain adaptation that
focuses on accurately aligning the distributions of the relevant
subdomains. However, most of them are adversarial methods that
contain several loss functions and converge slowly. Based on this,
we present a deep subdomain adaptation network (DSAN) thatlearns a transfer network by aligning the relevant subdomain
distributions of domain-specific layer activations across different
domains based on a local maximum mean discrepancy (LMMD).
Our DSAN is very simple but effective, which does not needadversarial training and converges fast. The adaptation canbe achieved easily with most feedforward network models byextending them with LMMD loss, which can be trained efficiently
via backpropagation. Experiments demonstrate that DSAN canachieve remarkable results on both object recognition tasks
and digit classification tasks. Our code will be available athttps://github.com/easezyc/deep-transfer-learning.
Index Terms— Domain adaptation, fine grained, subdomain.

I. I NTRODUCTIONIN RECENT years, deep learning methods have achieved
impressive success in computer vision [1], which, however,usually needs large amounts of labeled data to train a good
Manuscript received August 13, 2019; revised December 11, 2019; accepted
April 13, 2020. This work was supported in part by the National Key Researchand Development Program of China under Grant 2018YFB1004300, in partby the National Natural Science Foundation of China under Grant U1836206,
Grant U1811461, and Grant 61773361, and in part by the Project of YouthInnovation Promotion Association CAS under Grant 2017146. (Corresponding
author: Fuzhen Zhuang.)
Yongchun Zhu, Fuzhen Zhuang, and Qing He are with the Key Laboratoryof Intelligent Information Processing of Chinese Academy of Sciences (CAS),
Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China, and also with the University of Chinese Academyof Sciences, Beijing 100049, China (e-mail: zhuyongchun18s@ict.ac.cn;
zhuangfuzhen@ict.ac.cn; heqing@ict.ac.cn).
Jindong Wang, Guolin Ke, and Jiang Bian are with Microsoft
Research, Beijing, China (e-mail: jindong.wang@microsoft.com; guolin.ke@
microsoft.com; jiang.bian@microsoft.com).
Jingwu Chen is with ByteDance (e-mail: chenjingwu@bytedance.com).
Hui Xiong is with Rutgers, The State University of New Jersey, New
Brunswick, NJ USA (e-mail: hxiong@rutgers.edu).
Color versions of one or more of the figures in this article are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TNNLS.2020.2988928

Fig. 1.
Left: global domain adaptation might lose some fine-grained
information. Right: relevant subdomain adaptation can exploit the local affinity
to capture the fine-grained information for each category.

deep network. In the real world, it is often expensive andlaborsome to collect enough labeled data. For a target taskwith the shortage of labeled data, there is a strong motivation
to build effective learners that can leverage rich labeled datafrom a related source domain. However, this learning paradigm
suffers from the shift of data distributions across different
domains, which will undermine the generalization ability ofmachine learning models [2], [3].
Learning a discriminative model in the presence of the shift
between the training and test data distributions is known asdomain adaptation or transfer learning [2], [4], [5]. Previous shallow domain adaptation methods bridge the source
and target domains by learning invariant feature representations [6]–[8] or estimate instance importance without using
target labels [9]. Recent studies have shown that deep networks can learn more transferable features for domain adaptation [10], [11], by disentangling explanatory factors ofvariations behind domains. The latest advantages have beenachieved by embedding domain adaptation modules in thepipeline of deep feature learning to extract domain-invariant
representations [12]–[16].
The previous deep domain adaptation methods [13], [16],[17] mainly learn a global domain shift, i.e., aligning the
global source and target distributions without considering
the relationships between two subdomains in both domains(a subdomain contains the samples within the same class.).
As a result, not only all the data from the source andtarget domains will be confused, but also the discriminative
structures can be mixed up. This might loss the fine-grained
information for each category. An intuitive example is shownin Fig. 1 (left). After global domain adaptation, the distributions of the two domains are approximately the same, but thedata in different subdomains are too close to be classified accurately. This is a common problem in previous global domain

2162-237X © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

adaptation methods. Hence, matching the global source andtarget domains may not work well for diverse scenarios.
With regard to the challenge of global domain shift, recently,
more and more researchers [14], [15], [18]–[21] pay attention
to subdomain adaptation (also called semantic alignment ormatching conditional distribution) which is centered on learning a local domain shift, i.e., accurately aligning the distribution of the relevant subdomains within the same category inthe source and target domains. An intuitive example is shownin Fig. 1 (right). After subdomain adaptation, with the local
distribution that is approximately the same, the global distribution is also approximately the same. However, all of them
are adversarial methods that contain several loss functions andconverge slowly. We list the comparison of the subdomain
adaptation methods in Experiment IV.
Based on the subdomain adaptation, we propose a deep subdomain adaptation network (DSAN) to align the relevant subdomain distributions of activations in multiple domain-specificlayers across domains for unsupervised domain adaptation.
DSAN extends the feature representation ability of deep
adaptation networks (DANs) by aligning relevant subdomain
distributions as mentioned earlier. A key improvement overprevious domain adaptation methods is the capability of
subdomain adaptation to capture the fine-grained informationfor each category, which can be trained in an end-to-end
framework. To enable proper alignment, we design a local
maximum mean discrepancy (LMMD), which measures theHilbert–Schmidt norm between kernel mean embedding of
empirical distributions of the relevant subdomains in source
and target domains with considering the weight of different
samples. The LMMD method can be achieved with mostfeedforward network models and can be trained efficiently
using standard backpropagation. In addition, our DSAN is very
simple and easy to implement. Note that the most remarkable
results are achieved by adversarial methods recently. Experiments show that DSAN, which is a nonadversarial method, canobtain the remarkable results for standard domain adaptation
on both object recognition tasks and digit classification tasks.
The contributions of this article are summarized as follows.
1) We propose a novel deep neural network architecture
for subdomain adaptation, which can extend the abilityof DANs by capturing the fine-grained information foreach category.
2) We show that DSAN, which is a nonadversarial method,
can achieve the remarkable results. In addition, ourDSAN is very simple and easy to implement.
3) We propose LMMD to measure the discrepancy between
kernel mean embedding relevant subdomains in source
and target domains and successfully apply it to DSAN.
4) A new local distribution discrepancy measure dAL is
proposed to estimate the discrepancy between two
subdomain distributions.
II. R ELATED W ORK
In this section, we will introduce the related work inthree aspects: domain adaptation, maximum mean discrepancy (MMD), and subdomain adaptation methods.

1) Domain Adaptation: Recent years have witnessed many
approaches to solve the visual domain adaptation problem,
which is also commonly framed as the visual data set bias
problem [2], [3]. Previous shallow methods for domain adaptation include reweighting the training data so that they can moreclosely reflect those in the test distribution [22], and findinga transformation in a lower dimensional manifold that draws
the source and target subspaces closer [6]–[8], [23], [24].
Recent studies have shown that deep networks can learn
more transferable features for domain adaptation [10], [11],by disentangling explanatory factors of variations behinddomains. The latest advances have been achieved byembedding domain adaptation modules in the pipeline ofdeep feature learning to extract domain-invariant representations [12]–[16], [25]. Two main approaches are identified among the literature. The first is statistic moment
matching-based approach, i.e., MMD [13], [26], [27], central
moment discrepancy (CMD) [28], and second-order statistics
matching [16]. The second commonly used approach is basedon an adversarial loss, which encourages samples from different domains to be nondiscriminative with respect to domainlabels, i.e., domain adversarial net-based adaptation methods [17], [29], [30] borrowing the idea of GAN. Generally,
the adversarial approaches can achieve better performance than
the statistic moment matching-based approaches. In addition,
most state-of-the-art approaches [14], [29], [31] are domain
adversarial net-based adaptation methods. Our DSAN is an
MMD-based method. We show that DSAN without adversarial
loss can achieve remarkable results.
2) Maximum Mean Discrepancy: MMD has been adopted
in many approaches [8], [13], [27] for domain adaptation.
In addition, there are some extensions of MMD [7], [26].
Conditional MMD [7] and joint MMD [26] measure theHilbert–Schmidt norm between kernel mean embedding of
empirical conditional and joint distributions of the source
and target data, respectively. Weighted MMD [32] alleviates
the class weight bias by assigning class-specific weights tosource data. However, our local MMD measures the discrepancy between kernel mean embedding relevant subdomains
in source and target domains with considering the weight ofdifferent samples. CMMD [7], [23], [33] is a special case of
our LMMD.
3) Subdomain Adaptation: Recently, we have witnessedconsiderable interest and research [14], [15], [18], [20] for subdomain adaptation that focuses on accurately aligning the distributions of the relevant subdomains. Multiadversarial domain
adaptation (MADA) [15] captures the multimode structures toenable fine-grained alignment of different data distributions
based on multiple-domain discriminators. Moving semantictransfer network (MSTN) [20] learns the semantic representations for unlabeled target samples by aligning labeled
source centroid and pseudolabeled target centroid. CDAN [14]conditions the adversarial adaptation models on discriminative
information conveyed in the classifier predictions. Co-DA [18]constructs multiple diverse feature spaces and aligns source
and target distributions in each of them individually while
encouraging that alignments agree with each other with regard

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ZHU et al.: DEEP SUBDOMAIN ADAPTATION NETWORK FOR IMAGE CLASSIFICATION

to the class predictions on the unlabeled target examples. The
adversarial loss is adopted by all of them. However, compared
DSAN with them, our DSAN that is more simple and easy to
implement can achieve better performance without adversarial
loss.
III. D EEP S UBDOMAIN A DAPTATION N ETWORKIn unsupervised domain adaptation, we are given a sourcesdomain Ds = {(xis , yis )}ni=1of n s labeled samples ( yis ∈ RC
is an one-hot vector indicating the label of xis , i.e., yisj =1 means xis belonging to the j th class, where C is thenumber of classes.) and a target domain Dt = {xtj }nj t=1 of n t
unlabeled samples. Ds and Dt are sampled from different data
distributions p and q, respectively, and p = q. The goal ofdeep domain adaptation is to design a deep neural networky = f (x) that formally reduces the shifts in the distributionsof the relevant subdomains in different domains and learns
transferable representations simultaneously such that the target
risk Rt ( f ) = E(x,y) q [ f (x) = y] can be bounded by leveraging
the source domain supervised data.
Recent studies reveal that deep networks [34] can learn moretransferable representations than traditional handcrafted features [11], [35]. The favorable transferability of deep features
leads to several popular deep transfer learning methods [12],[13], [26], [36], which mainly use adaptation layers with a
global domain adaptation loss to jointly learn a representation.
The formal representation can be
min
f

ns


   1
J f xis , yis + λd̂( p, q)n s i=1(1)

where J (·, ·) is the cross-entropy loss function (classificationloss) and d̂(·, ·) is domain adaptation loss. λ > 0 is the tradeoff
parameter of the domain adaptation loss and the classification
loss.
The common problem with these methods is that theymainly focus on aligning the global source and target distributions without considering the relationships between subdomains within the same category of different domains. Thesemethods derive a global domain shift for the source and target
domains, and the global distribution of the two domains isapproximately the same after adaptation. However, the globalalignment may lead to some irrelevant data too close to
be classified accurately. Actually, while by exploiting therelationships between the subdomains in different domains,just aligning the relevant subdomain distributions can not
only match the global distributions but also the local distributions mentioned earlier. Therefore, subdomain adaptationthat exploits the relationships between two subdomains to
overcome the limitation of aligning global distributions isnecessary.
To divide the source and target domains into multiple
subdomains that contain the samples within the same class,the relationships between the samples should be exploited.
It is well known that the samples within the same category
are more relevant. However, data in the target domain isunlabeled. Hence, we would use the output of the networks
as the pseudolabels of target domain data, which will be

3detailed later. According to the category, we divide Ds and
Dt into C subdomains Ds(c) and Dt(c) where c ∈ {1, 2, . . . , C}
denotes the class label, and the distributions of Ds(c) and Dt(c)are p(c) and q (c) , respectively. The aim of subdomain adaptation is to align the distributions of relevant subdomains thathave samples with the same label. Combining the classification
loss and subdomain adaptation loss, the loss of subdomain
adaptation method is formulated ass
1 
J ( f (xis ), yis ) + λEc [d̂( p(c) , q (c) )]n s i=1

n

min
f

(2)

where Ec [·] is the mathematical expectation of the class.
To compute the discrepancy in 2 between the relevant subdomain distributions based on MMD [37] that is a nonparametricmeasure, we propose LMMD to estimate the distribution
discrepancy between subdomains.
A. Maximum Mean Discrepancy
MMD [37] is a kernel two-sample test, which rejects or
accepts the null hypothesis p = q based on the observed
samples. The basic idea behind MMD is that if the generating
distributions are identical, all the statistics are the same.
Formally, MMD defines the following difference measure:dH ( p, q)  E p [φ(xs )] − Eq [φ(xt )]2H(3)

where H is the reproducing kernel Hillbert space (RKHS)
endowed with a characteristic kernel k. Here, φ(·) denotes
some feature map to map the original samples to RKHS andthe kernel k means k(xs , xt ) = φ(xs ), φ(xt ), where ·, ·
represents inner product of vectors. The main theoretical result
is that p = q if and only if DH ( p, q) = 0 [37]. In practice,an estimate of the MMD compares the square distance between
the empirical kernel mean embeddings as

2



1 

1

φ(xi ) −φ(x j )d̂H ( p, q) = n t x ∈D
 n s x ∈D

i

s

j

t

H

ns
nt
nt 
ns 




1 
1 
= 2
k xis , xsj + 2
k xit , xtj
n s i=1 j =1
n t i=1 j =1
s 
t


2 
k xis , xtj
n s n t i=1 j =1

n

−

n

(4)

where d̂H ( p, q) is an unbiased estimator of dH ( p, q).
B. Local Maximum Mean Discrepancy
As a nonparametric distance estimate between two distributions, MMD has been widely applied to measure thediscrepancy between the source and target distributions. Previous deep MMD-based methods [13], [26], [38] mainly
focus on the alignment of the global distributions, ignoring
the relationships between two subdomains within the same
category. Taking the relationships of the relevant subdomains
into consideration, it is important to align the distributions of
the relevant subdomains within the same category in source

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

Fig. 2. Left: architecture of DSAN. DSAN will formally reduce the discrepancy between the relevant subdomain distributions of the activations in layers L
by using LMMD minimization. Right: LMMD module needs four inputs: the activations zsl and ztl where l ∈ L, the ground-truth label ys , and the predictedlabel ŷt .

and target domains. With the desire to align distributions of
the relevant subdomains, we propose the LMMD asdH ( p, q)  Ec E p(c) [φ(xs )] − Eq (c) [φ(xt )]2H(5)

where xs and xt are the instances in Ds and Dt , and p(c)and q (c) are the distributions of Ds(c) and Dt(c) , respectively.
Different from MMD that focuses on the discrepancy of global
distributions, 5 can measure the discrepancy of local distributions. By minimizing 5 in deep networks, the distributionsof relevant subdomains within the same category are drawn
close. Therefore, the fine-grained information is exploited fordomain adaptation.
We assume that each sample belongs to each class according
to weight wc . Then, we formulate an unbiased estimator of 5
as

2

C 




1
sc
s
tc
t 

d̂H ( p, q) =
w
φ(x
)
−
w
φ(x
)
(6)
i
i
j
j 

C c=1  stxi ∈Dsx j ∈DtHswhere wisc and wtcj denote the weightxtj belonging
n s of xsci and 
to class c, respectively. 
Note that i=1 wi and nj t=1 wtcj areboth equal to one, and xi ∈D wic φ(xi ) is a weighted sum on
category c. We compute wic for the sample xi as

wic = yic
(x j ,y j )∈D y j c(7)where yic is the cth entry of vector yi . For samples in thesource domain, we use the true label yis as a one-hot vector
to compute wisc for each sample. However, in unsupervised
adaptation where the target domain has no labeled data, we can
not calculate 6 directly with the ytj unavailable. We findthat the output of the deep neural network ŷi = f (xi ) is a
probability distribution that well characterizes the probabilityof assigning xi to each of the C classes. Thus, for target
domain Dt without labels, it is a natural idea to use ŷit as
the probability of assigning xit to each of the C classes. Then,
we can calculate wtcj for each target sample. Finally, we cancalculate 6.
It is easy to access the labels of the source domain, while
for the target domain, the label predicted (hard prediction)by the model might be wrong, and using this wrong label
might degrade the performance. Hence, using the probability
prediction (soft prediction) might alleviate the negative impact.
Note that CMMD, which assumes that each sample has thesame weight, is a special case of LMMD, whereas LMMD
takes the uncertainty of target samples into consideration.
To adapt feature layers, we need the activations in thelayers. Given source domain Ds with n s labeled instances andtarget domain Dt with n t unlabeled points drawn independent
identically distributed (i.i.d.) from p and q, respectively,
the deep networks will generate activations in layers l assand {ztlj }nj t=1 . In addition, we cannot compute the φ(·){zsli }ni=1
directly. Then, we reformulate 6 as
⎡
ns
ns 
C


 sl sl 
1⎣wisc wscd̂l ( p, q) =j k zi , z j
C c=1 i=1 j =1
+

nt
nt 




witc wtcj k ztli , ztlj

i=1 j =1

−2

nt
ns 


⎤


wisc wtcj k zsli , ztlj ⎦(8)

i=1 j =1

where zl is the lth (l ∈ L = {1, 2, . . . , |L|}) layer activation.
Equation 8 can be used as the adaptation loss in 2 directly, andthe LMMD can be achieved with most feedforward networkmodels.
C. Deep Subdomain Adaptation Network
Based on LMMD, we propose DSAN as shown in Fig. 2.
Different from previous global adaptation methods, DSAN
not only aligns the global source and target distributions but
also aligns the distributions of the relevant subdomains by
integrating deep feature learning and feature adaptation in
an end-to-end deep learning model. We try to reduce thediscrepancy between the relevant subdomain distributions of
the activations in layers L. We use the LMMD in (8) over thedomain-specific layers L as the subdomain adaptation loss inthe following equation:ns   1 d̂l ( p, q).
J f xis , yis + λ
min
f ns
i=1
l∈L

(9)

Since training deep CNNs requires a large amount of labeleddata that are prohibitive for many domain adaptation applications, we start with the CNN models pretrained on the
ImageNet 2012 data and fine-tune it as [26]. The training of
DSAN mainly follows standard minibatch stochastic gradient

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ZHU et al.: DEEP SUBDOMAIN ADAPTATION NETWORK FOR IMAGE CLASSIFICATION

descent (SGD) algorithm. It is worth noting that, with DSAN
iteration, the labeling for target samples usually becomes moreaccurate. This EM-like pseudolabel refinement procedure isempirically effective, as shown in the experiments.
Remark: The theory of domain adaptation [39], [40] suggests A-distance as a measure of distribution discrepancy,
which, together with the source risk, will bound the target risk.
The proxy A-distance is defined as dA = 2(1 − 2), where is the generalization error of a classifier (e.g., kernel SVM)
trained on the binary problem of discriminating the source andtarget. The A-distance just focuses on the global distribution
discrepancy; hence, we propose the A L -distance to estimate
the subdomain distribution discrepancy. First, we define dA ofclass c as dAc = 2(1 − 2 c ), where  c is the generalization
error of a classifier trained on the same class in different
domains.
Then, we define dAL = E[dAc ] = 2E[1 − 2 c ] =C
2 c=1 p(c)(1 − 2 c ), where E[·] denotes the mathematical
expectation and p(c) denotes the probability of class c in thetarget domain.

D. Theoretical Analysis
In this section, we give an analysis of the effectiveness ofusing the classifier predictions on the target samples, making
use of the theory of domain adaptation [39], [41].
Theorem 1: Let H be the hypothesis space. Given two
domains S and T , we have1∀h ∈ H, RT (h) ≤ RS (h) + dHH (S, T ) + C
2h∈H

C = min RS (h, fS ) + RT (h, fT )h∈H≤ min RS (h, fS ) + RT (h, fS ) + RT ( fS , f T )h∈H≤ min RS (h, fS ) + RT (h, fS ) + RT ( fS , f T̂ )h∈H

+RT ( f T , fT̂ )(13)

where f T̂ is pseudolabeling function for target domain. The
first term RS (h, f S ) and the second term RT (h, f S ) denotes
the disagreement between h and the source labeling functionf S on source and target samples, respectively. Since h islearned with the labeled source samples, the gap between themcan be very small. The last term RT ( fT , f T̂ ) denotes thediscrepancy between the ideal target labeling function f T andthe pseudolabeling function f T̂ , which would be minimized
as learning proceeds. Then, we should focus on the third term
RT ( f S , f T̂ ) = Ex∼T [l( f S (x), f T̂ (x))], where l(·, ·) is typically 0–1 loss function. The source samples of class k would
be predicted with label k by the source labeling function f S .
If the feature of target samples in class k is similar with thesource feature in class k, the target samples can be predicted
the same as the pseudotarget labeling function. Therefore,
if the distributions of subdomains in different domain arematching, RT ( f S , fT̂ ) is expected to be small.
In summary, by aligning relevant subdomain distributions,
our DSAN could further minimize the shared expected loss C.
Hence, utilizing the prediction of the target samples is effective
for unsupervised domain adaptation.
IV. E XPERIMENT(11)

where f S and f T are true labeling functions for source andtarget domain, respectively.
We show our DSAN is trying to optimize the upper boundfor C. From [39], for any labeling functions f 1 , f2 , and f 3 ,we haveR( f 1 , f 2 ) ≤ R( f 1 , f 3 ) + R( f 2 , f3 ).

Then, we have

(10)

where RS (h) and RT (h) are the expected error on thesource samples and target samples, respectively. RS (h) canbe minimized easily with source label information. Besides,
dHH (S, T ) is the domain divergence measure by a discrepancy distance between two distributions S and T . Actually,
there are many approaches to minimize dHH (S, T ), suchas adversarial learning [12], MMD [13], and Coral [16]. C
is the shared expected loss and is expected to be negligibly
small, thus usually disregarded by previous methods [12], [13].
However, it is possible that C tends to be large when the
cross-domain category alignment is not explicitly enforced.
Hence, C needs to be bounded as well. Unfortunately, we cannot directly measure C without target true labels. Therefore,
we utilize the pseudolabels to give the approximate evaluationand minimization.
Definition 1: C is defined as
C = min RS (h, f S ) + RT (h, f T )

5

(12)

We evaluate DSAN against competitive transfer learning
baselines on object recognition and digit classification. The
four data sets, including ImageCLEF-DA, Office-31, OfficeHome, and VisDA-2017, are used for object recognition task,
while for digit classification, we construct the transfer tasks
from MNIST, USPS, and SVHN. We denote all transfer tasks
as source domain → target domain.
A. Setup
ImageCLEF-DA1 is a benchmark data set for ImageCLEF
2014 domain adaptation challenge, which is organized by
selecting 12 common categories shared by the following
three public data sets, each is considered as a domain:
Caltech-256 (C), ImageNet ILSVRC 2012 (I), and Pascal
VOC 2012 (P). There are 50 images in each category and
600 images in each domain. We use all domain combinations
and build six transfer tasks: I → P, P → I, I → C, C → I,
C → P, P → C.
Office-31 [43] is a benchmark data set for domain adaptation, comprising 4110 images in 31 classes collectedfrom three distinct domains: Amazon (A), which contains
images downloaded from amazon.com, and Webcam (W) and
DSLR (D), which contain images taken by Web camera anddigital SLR camera with different photographical settings,
1 http://imageclef.org/2014/adaptation

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

respectively. To enable unbiased evaluation, we evaluate all
methods on all six transfer tasks A → W, D → W, W → D,
A → D, D → A, W → A as in [12], [26], and [38].
Office-Home [44] is a new data set, which consistsof 15 588 images and is much larger than Office-31 andImageCLEF-DA. It consists of images from four different
domains: artistic images (A), clip art (C), product images (P),and real-world images (R). For each domain, the data set
contains the images of 65 object categories collected in officeand home settings. Similarly, we use all domain combinations
and construct 12 transfer tasks.
VisDA-2017 [45] is a challenging simulation-to-real dataset, with two very distinct domains: synthetic, renderingsof 3-D models from different angles and with different lightning conditions, and real, natural images. It contains over 280k
images across 12 classes in the training, validation, and test
domains.
MNIST-USPS-SVHN: We explore three digit data sets:
MNIST [46], USPS, and SVHN [47] for transfer digit classification. Different from Office-31, MNIST contains gray digit
images of size 28×28, USPS contains 16×16 gray digits, and
SVHN contains color 32 ×32 digits images that might containmore than one digit in each image. We conduct experimentson three transfer tasks MNIST → USPS, USPS → MNIST,
and SVHN → MNIST.
Baseline Methods: For ImageCLEF-DA and Office-31,
we compare our model DSAN with several standard deeplearning methods and deep transfer learning methods:
deep convolutional neural network (ResNet) [1], deep
domain confusion (DDC) [38], DAN [13], Deep CORAL(D-CORAL) [16], domain adversarial neural networks(DANNs) [17], residual transfer network (RTN) [26], adversarial discriminative domain adaptation (ADDA) [30], joint adaptation networks (JANs) [26], MADA [15], collaborative andadversarial network (CAN and iCAN) [31], generate to adapt(GTA) [42], and conditional adversarial domain adaptation(CDAN and CDAN+E) [14]. For Office-Home, we compare
DSAN with ResNet, DAN, DANN, JAN, and CDAN, andthe results of all baselines are extracted from [14] and [15].
For VisDA-2017, we compare DSAN with ResNet, DANN,
DAN, JAN, and MCD [48], and the results of all baselines
are extracted from [49].
For MNIST-USPS-SVHN, we compare DSAN with
DANNs [17], deep reconstruction classification networks(DRCNs) [50], coupled generative adversarial networks(CoGANs) [51], ADDA [30], unsupervised image-to-imagetranslation networks (UNIT) [], asymmetric tritraining domain
adaptation (ATDA) [53], GTA [42], and MSTN [20]. The
results of SourceOnly, DANN, DRCN, CoGAN, ADDA, and
GTA are extracted from [42]. For the rest, we refer to theresults in their original articles.
Implementation Details For object recognition tasks,
we employed the ResNet [1]. Following CDAN [14], a bottleneck layer f cb with 256 units is added after the last average
pooling layer for safe transfer representation learning. We use
the output of f cb as inputs to the LMMD. Note that, it is easy
to add LMMD in multiple layers and we only add LMMD
to one layer. Also, image random flipping and cropping are

adopted following JAN [26]. For a fair comparison, all baselines use the same architecture (for VisDA-2017, we use
ResNet101 [1], whereas ResNet50 for others). We fine-tune
all convolutional and pooling layers from ImageNet pretrainedmodels and train the classifier layer via back-propagation.
Since the classifier is trained from scratch, we set its learning
rate to be ten times that of the other layers. For digit classification tasks, we follow the protocols in ADDA [30] and use
the same architecture with ADDA.
For all tasks, we use minibatch SGD with a momentumof 0.9 and the learning rate annealing strategy in Revgrad [12];the learning rate is not selected by a grid search due tohigh computational cost, it is adjusted during SGD using thefollowing formula: ηθ = η0 /(1 + αθ )β , where θ is the training
progress linearly changing from 0 to 1, η0 = 0.01, α = 10,
and β = 0.75. To suppress noisy activations at the earlystages of training, instead of fixing the adaptation factor λ,we gradually change it from 0 to 1 by a progressive schedule:
λθ = 2/exp(−γ θ ) − 1, and γ = 10 is fixed throughout theexperiments [12].
We implement DSAN in PyTorch and report the average classification accuracy and standard error of three random trials. For all MMD-based methods [13], [26], [38]including DSAN, we adopt Gaussian kernel with bandwidth
set to median pairwise squared distances on the training
data [37].
B. Results1) Object Recognition: The classification results ofImageCLEF-DA, Office-31, Office-Home, and VisDA-2017
are, respectively, shown in Tables I–IV. DSAN outperforms
all compared methods on most transfer tasks. In particular,
DSAN substantially improves the average accuracy by large
margins (more than 3%) on Image-CLEF, Office-Home, and
VisDA-2017. The encouraging results indicate the importanceof subdomain adaptation and show that DSAN is able to learn
more transferable representations.
The experimental results further reveal several insightful
observations.
1) In standard domain adaptation, subdomain adaptation
methods (MADA [15], CDAN [14], and our DSAN)
outperform previous global domain adaptation methods. The improvement from previous global domain
adaptation methods to subdomain adaptation methods is
crucial for domain adaptation; previous methods alignglobal distribution without considering the relationshipbetween subdomains, whereas DSAN accurately aligns
the relevant subdomain distributions, which can capture
more fine-grained information for each category.
2) In particular, comparing DSAN with the mostrecent subdomain adaptation methods [14], [15],DSAN achieves better performance. This verifies theeffectiveness of our model.
3) Comparing DSAN with the nonadversarial methods [13], [16], [26], [38], DSAN also largely improves
the average performance on 24 object recognition tasks(6.65% higher than JAN [26]).

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ZHU et al.: DEEP SUBDOMAIN ADAPTATION NETWORK FOR IMAGE CLASSIFICATION

7TABLE IA CCURACY (%) ON I MAGE CLEF-DA FOR U NSUPERVISED D OMAIN A DAPTATION (R ES N ET 50)

TABLE IIA CCURACY (%) ON O FFICE -31 FOR U NSUPERVISED D OMAIN A DAPTATION (R ES N ET 50)

TABLE IIIA CCURACY (%) ON O FFICE -H OME FOR U NSUPERVISED D OMAIN A DAPTATION (R ES N ET 50)

TABLE IVA CCURACY (%) ON V IS DA-2017 FOR U NSUPERVISED D OMAIN A DAPTATION (R ES N ET 101)

4) Comparing DSAN with LMMD, DAN with MMD, and
JAN with JMMD, DSAN achieves the best performance,
which implies that LMMD is more suitable for aligning
distributions than MMD and JMMD.
2) Digit Classification: The classification results of threetasks of MNIST–USPS–SVHN are shown in Table V. Except
for DRCN [50], all other baselines are adversarial ones.

DSAN largely outperforms all baselines except SVHN →MNIST task. Comparing DSAN with MSTN [20] which isalso a subdomain adaptation method, DSAN achieves better
average accuracy and more stable results with lower standard
error.
Overall, all the abovementioned results demonstrate theeffectiveness of the proposed model.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
8

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

Fig. 3. (a) and (b) Visualizations of the learned representations using t-SNE for JAN and DSAN on task A→W, respectively. Red points are source samples
and blue are target samples. (c) A-distance and A L -distance on task A→W. (d) MMD and (e) LMMD on task A→W.
TABLE VA CCURACY (%) ON D IGIT R ECOGNITION TASKS FOR U NSUPERVISEDD OMAIN A DAPTATION . (“−” M EANS T HAT W E D ID N OTF IND THE R ESULT ON THE TASK )Fig. 4. On task D → A (Office31), we further analyze the convergence.
(a) Convergence (iteration). (b) Convergence (time).

C. Analysis
1) Feature Visualization: We visualize in Fig. 3(a) and (b)the network activations of task A → W learned by JAN
and DSAN (both use Gaussian kernel) using t-SNE embeddings [10]. Red points are source samples and blue are target
samples. Fig. 3(a) shows the result for JAN [26], which isa typical statistic moment matching-based approach using
JMMD. We can find that the source and target domains are
not aligned very well and some points are hard to classify.
In contrast, Fig. 3(b) shows the representations learned by
our DSAN using LMMD. It is observed that the source andtarget domains are aligned very well. We not only can see thatthe subdomains in different domains with the same class arevery close but also the subdomains with different classes are
dispersed. This result suggests that our model DSAN is able tocapture more fine-grained information for each category thanJAN, and LMMD is more effective than JMMD to align the
distributions.
2) Distribution Discrepancy: We use A-distance andA L -distance mentioned in Section III-C to measure global
distribution discrepancy and the subdomain distribution discrepancy. Fig. 3(c) shows dA and dAL on task A → W withrepresentations of CNN, JAN, and DSAN. We observe that dA
and dAL using DSAN are much smaller than the ones using
CNN and JAN, which shows that DSAN can not only close thecross-domain gap but also one of relevant subdomains more
effectively.
MMD is a method to measure the discrepancy of global
distributions, whereas LMMD is a method to measure thediscrepancy of local subdomain distributions. We compute
MMD and LMMD across domains on task A → W using
CNN, JAN, and DSAN based on the features in pool layer
and ground-truth labels. Fig. 3(d) shows that both MMD and
LMMD using DSAN activations are much smaller than using
CNN and JAN activations, which again validates that DSAN
successfully reduces the discrepancy of global and local distributions. In addition, LMMD is smaller than MMD for thereason that LMMD can estimate the distribution discrepancy
by eliminating the irrelevant data.
3) Convergence: We testify the convergence of CDAN,
CDAN+E, and DSAN, with the test errors on task D → A(Office31) shown in Fig. 4. From Fig. 4(a), with the same
number of iterations, DSAN achieves faster convergence than
CDAN and CDAN+E. From Fig. 4(b), with the same period,
DSAN also converges faster. Besides, the results further revealthat for each iteration, DSAN runs faster than CDAN and
CDAN+E.
4) Discussion on the Advantage of DSAN: To give an
overview of the results, we further compare our DSAN withseveral adversarial subdomain adaptation methods [14], [15],[18], [20] in Table VI and find some insightful observations.
First, the adversarial subdomain adaptation methods usually
have several loss functions, while DSAN only needs one classification loss and one LMMD loss. In addition, DSAN only
has one hyperparameter, whereas MSTN [20] and Co-DA [18]have several hyperparameters. DSAN has fewer loss terms andhyperparameter, which also indicates the easy implementation.
Second, comparing DSAN with MADA [15] and CDAN [14],DSAN also takes less time to converge. Third, DSAN achieves
the best performance. Especially, DSAN achieves 3% accuracy
higher than CDAN [14] which is one of the most recent
subdomain adaptation methods. Overall, all the results again
validate the advantage of our model DSAN.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ZHU et al.: DEEP SUBDOMAIN ADAPTATION NETWORK FOR IMAGE CLASSIFICATIONTABLE VIC OMPARISON OF THE S UBDOMAIN A DAPTATION M ETHODS . K IN MADAM EANS THE N UMBER OF C LASSES . PARAMETER M EANS THE N UMBER
OF H YPERPARAMETERS IN THE M ETHODS . T IME M EANS THE
AVERAGE C ONVERGENCE T IME ON THE I MAGE CLEF-DA
D ATA S ET (S ECONDS ). T IME I S M EASURED ON A G E F ORCE
GTX 1080 T I GPU BY O URSELVES . A CCURACY M EANS
THE AVERAGE A CCURACY ON THE I MAGE CLEF-DA
D ATA S ET. “−” M EANS T HAT W E D OSE N OTF IND THE R ESULTS F ROM THE
O RIGINAL A RTICLE

V. C ONCLUSION
Unlike the previous methods that align the global source
and target distributions, subdomain adaptation can accuratelyalign the distributions of relevant subdomains within thesame category of the source and target domains. However,most recent subdomain adaptation methods are adversarial
approaches that contain several loss functions and converge
slowly. Based on this, we proposed a new method DSAN,
which is a nonadversarial method and very simple and easy to
implement. Furthermore, to measure the discrepancy between
relevant subdomains within the same category of different
domains, we proposed a new local distribution discrepancy
measure LMMD. Extensive experiments conducted on bothobject recognition and digit classification tasks demonstrate
the effectiveness of the proposed model.
R EFERENCES[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2016, pp. 770–778.
[2] S. Jialin Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans.
Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010.
[3] J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, andN. D. Lawrence, Dataset Shift in Machine Learning. Cambridge,
MA, USA: MIT Press, 2009.
[4] F. Zhuang, X. Cheng, P. Luo, S. J. Pan, and Q. He, “Supervised
representation learning: Transfer learning with deep autoencoders,” inProc. IJCAI, 2015, pp. 4119–4125.
[5] F. Zhuang et al., “A comprehensive survey on transfer learning,” 2019,arXiv:1911.02685. [Online]. Available: http://arxiv.org/abs/1911.02685[6] B. Gong, Y. Shi, F. Sha, and K. Grauman, “Geodesic flow kernel for
unsupervised domain adaptation,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jun. 2012, pp. 2066–2073.
[7] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu, “Transfer featurelearning with joint distribution adaptation,” in Proc. IEEE Int. Conf.
Comput. Vis., Dec. 2013, pp. 2200–2207.
[8] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation
via transfer component analysis,” IEEE Trans. Neural Netw., vol. 22,
no. 2, pp. 199–210, Feb. 2011.
[9] J. Huang, A. J. Smola, A. Gretton, K. M. Borgwardt, and B. Schölkopf,“Correcting sample selection bias by unlabeled data,” in Proc. NIPS,
2007, pp. 601–608.
[10] J. Donahue et al., “Decaf: A deep convolutional activation feature forgeneric visual recognition,” in Proc. ICML, 2014, pp. 647–655.
[11] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable arefeatures in deep neural networks?” in Proc. NIPS, 2014, pp. 3320–3328.
[12] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by
backpropagation,” in Proc. ICML, 2015, pp. 1180–1189.

9[13] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable
features with deep adaptation networks,” in Proc. ICML, 2015,
pp. 97–105.
[14] M. Long, Z. Cao, J. Wang, and M. I. Jordan, “Conditional adversarial
domain adaptation,” in Proc. NIPS, 2018, pp. 1647–1657.
[15] Z. Pei, Z. Cao, M. Long, and J. Wang, “Multi-adversarial domain
adaptation,” in Proc. AAAI, 2018, pp. 3934–3941.
[16] B. Sun and K. Saenko, “Deep CORAL: Correlation alignment for deep
domain adaptation,” in Proc. ECCV, 2016, pp. 443–450.
[17] Y. Ganin et al., “Domain-adversarial training of neural networks,”J. Mach. Learn. Res., vol. 17, no. 1, pp. 2030–2096, 2016.
[18] A. Kumar et al., “Co-regularized alignment for unsupervised domain
adaptation,” in Proc. NIPS, 2018, pp. 9367–9378.
[19] J. Wang, Y. Chen, L. Hu, X. Peng, and P. S. Yu, “Stratified transfer
learning for cross-domain activity recognition,” in Proc. IEEE Int. Conf.
Pervas. Comput. Commun. (PerCom), Mar. 2018, pp. 1–10.
[20] S. Xie, Z. Zheng, L. Chen, and C. Chen, “Learning semantic representations for unsupervised domain adaptation,” in Proc. ICML, 2018,
pp. 5419–5428.
[21] J. Wang, Y. Chen, H. Yu, M. Huang, and Q. Yang, “Easy transfer
learning by exploiting intra-domain structures,” in Proc. IEEE Int. Conf.
Multimedia Expo (ICME), Jul. 2019, pp. 1210–1215.
[22] J. Jiang and C. Zhai, “Instance weighting for domain adaptation in NLP,”
in Proc. ACL, 2007, pp. 264–271.
[23] J. Wang, W. Feng, Y. Chen, H. Yu, M. Huang, and P. S. Yu,“Visual domain adaptation with manifold embedded distribution alignment,” in Proc. ACM Multimedia Conf. Multimedia Conf. (MM), 2018,
pp. 402–410.
[24] J. Wang, Y. Chen, W. Feng, H. Yu, M. Huang, and Q. Yang, “Transfer
learning with dynamic distribution adaptation,” 2019, arXiv:1909.08531.
[Online]. Available: http://arxiv.org/abs/1909.08531[25] Y. Zhu et al., “Multi-representation adaptation network for cross-domain
image classification,” Neural Netw., vol. 119, pp. 214–221, Nov. 2019.
[26] M. Long, J. Wang, and M. I. Jordan, “Deep transfer learning with joint
adaptation networks,” in Proc. ICML, 2017, pp. 2208–2217.
[27] Y. Zhu, F. Zhuang, and D. Wang, “Aligning domain-specific distribution
and classifier for cross-domain classification from multiple sources,” inProc. AAAI Conf. Artif. Intell., vol. 33, Jul. 2019, pp. 5989–5996.
[28] W. Zellinger, T. Grubinger, E. Lughofer, T. Natschläger, and
S. Saminger-Platz, “Central moment discrepancy (CMD) for domaininvariant representation learning,” in Proc. ICLR, 2017, pp. 1–13.
[29] J. Hoffman et al., “Cycada: Cycle-consistent adversarial domain adaptation,” in Proc. ICML, 2018, pp. 1–15.
[30] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), vol. 1, Jul. 2017, p. 4.
[31] W. Zhang, W. Ouyang, W. Li, and D. Xu, “Collaborative and adversarial
network for unsupervised domain adaptation,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., Jun. 2018, pp. 3801–3809.
[32] H. Yan, Y. Ding, P. Li, Q. Wang, Y. Xu, and W. Zuo, “Mind the classweight bias: Weighted maximum mean discrepancy for unsupervised
domain adaptation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jul. 2017, pp. 2272–2281.
[33] J. Wang, Y. Chen, S. Hao, W. Feng, and Z. Shen, “Balanced distribution
adaptation for transfer learning,” in Proc. IEEE Int. Conf. Data Mining(ICDM), Nov. 2017, pp. 1129–1134.
[34] Y. Bengio, A. Courville, and P. Vincent, “Representation learning:
A review and new perspectives,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 35, no. 8, pp. 1798–1828, Aug. 2013.
[35] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014,
pp. 1717–1724.
[36] J. Hoffman, E. Tzeng, T. Darrell, and K. Saenko, “Simultaneous deeptransfer across domains and tasks,” in Proc. ICCV, 2015, pp. 4068–4076.
[37] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola,“A kernel two-sample test,” J. Mach. Learn. Res., vol. 13, pp. 723–773,
Mar. 2012.
[38] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell,“Deep domain confusion: Maximizing for domain invariance,” 2014,arXiv:1412.3474. [Online]. Available: http://arxiv.org/abs/1412.3474[39] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, andJ. W. Vaughan, “A theory of learning from different domains,” Mach.
Learn., vol. 79, nos. 1–2, pp. 151–175, May 2010.
[40] Y. Mansour, M. Mohri, and A. Rostamizadeh, “Domain adaptation withmultiple sources,” in Proc. NIPS, 2009, pp. 1041–1048.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
10

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS[41] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of representations for domain adaptation,” in Proc. NIPS, 2007, pp. 137–144.
[42] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa, “Generate to adapt: Aligning domains using generative adversarial networks,”
in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018,
pp. 8503–8512.
[43] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual categorymodels to new domains,” in Proc. ECCV, 2010, pp. 213–226.
[44] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan,“Deep hashing network for unsupervised domain adaptation,” 2017,arXiv:1706.07522. [Online]. Available: http://arxiv.org/abs/1706.07522[45] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and
K. Saenko, “VisDA: The visual domain adaptation challenge,” 2017,arXiv:1710.06924. [Online]. Available: http://arxiv.org/abs/1710.06924[46] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11,
pp. 2278–2324, Nov. 1998.
[47] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng,“Reading digits in natural images with unsupervised feature learning,”
NIPS workshop DLUFL, 2011.
[48] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada, “Maximum classifierdiscrepancy for unsupervised domain adaptation,” in Proc. IEEE/CVF
Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 3723–3732.
[49] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann, “Contrastive
adaptation network for unsupervised domain adaptation,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019,
pp. 4893–4902.
[50] M. Ghifary, W. B. Kleijn, M. Zhang, D. Balduzzi, and W. Li, “Deepreconstruction-classification networks for unsupervised domain adaptation,” in Proc. ECCV, 2016, pp. 597–613.
[51] M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,” inProc. Adv. Neural Inf. Process. Syst., 2016, pp. 469–477.
[52] M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-imagetranslation networks,” in Proc. NIPS, 2017, pp. 700–708.
[53] K. Saito, Y. Ushiku, and T. Harada, “Asymmetric tri-training for unsupervised domain adaptation,” in Proc. ICML, 2017, pp. 2988–2997.
Yongchun Zhu received the B.S. degree from Beijing Normal University, Beijing, China, in 2018.
He is currently pursuing the M.S. degree with theInstitute of Computing Technology, Chinese Academy of Sciences, Beijing.
He has published some papers in journals andconference proceedings, including Neural Networks,
AAAI, WWW, and PAKDD. His main researchinterests include transfer learning, metalearning, andrecommendation systems.

Fuzhen Zhuang is currently an Associate Professor
with the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China. He has
published more than 100 papers in the prestigious
refereed journals and conference proceedings, suchas the IEEE T RANSACTIONS ON K NOWLEDGE AND
D ATA E NGINEERING, the IEEE T RANSACTIONSON C YBERNETICS , the IEEE T RANSACTIONS ON
N EURAL N ETWORKS AND L EARNING S YSTEMS ,ACM TIST, SIGKDD, IJCAI, AAAI, WWW, andICDE. His research interests include transfer learning, machine learning, data mining, multitask learning, and recommendation
systems.
Jindong Wang received the Ph.D. degree from theInstitute of Computing Technology, Chinese Academy of Sciences, Beijing, China.
He is currently a Researcher with Microsoft
Research, Beijing. His research interests mainly
include transfer learning, machine learning, data
mining, and artificial intelligence.
Dr. Wang is also a member of ACM, AAAI, andCCF. He serves as a Reviewer for many journals andconferences, including the IEEE T RANSACTIONSON PATTERN A NALYSIS AND M ACHINE I NTELLI GENCE, CSUR, ACM CHI, and Neurocomputing.

Guolin Ke is currently a Senior Researcher with
the Machine Learning Group, Microsoft ResearchAsia, Beijing, China. He created one of themost popular decision tree learning tool LightGBM. His research interest mainly includes machinelearning algorithms.

Jingwu Chen received the master’s degree from theInstitute of Computing Technology, Chinese Academy of Sciences, Beijing, China, in 2019.
He is currently a member of the Recommendation Team, ByteDance. His research interests
include machine learning and its applications,such as recommendation systems and computational
advertising.

Jiang Bian is currently a Researcher and Engineer with rich experience in information retrieval,data mining, and machine learning. He is also a
Principal Researcher and a Research Manager with
Microsoft Research, Beijing, China, with researchinterests in AI for finance, AI for logistics, deeplearning, multiagent reinforcement learning, computational advertising, and a variety of machinelearning applications. He was a Senior Scientist with
Yidian Inc., China, a start-up company, where heworked on recommendation and search problems.

Hui Xiong (Fellow, IEEE) received the Ph.D.
degree in computer science from the Universityof Minnesota–Twin Cities, Minneapolis, MN, USA,in 2005.
He is currently a Full Professor with Rutgers, The
State University of New Jersey, New Brunswick,
NJ USA.
Dr. Xiong received the 2018 Ram Charan Management Practice Award as the Grand Prix winnerfrom the Harvard Business Review, the RBS Deans
Research Professorship in 2016, the Rutgers University Board of Trustees Research Fellowship for Scholarly Excellence in 2009,
the IEEE ICDM Best Research Paper Award in 2011, and the IEEE ICDM
Outstanding Service Award in 2017. He is a Co-Editor-in-Chief of Encyclopedia of GIS, an Associate Editor of the IEEE T RANSACTIONS ON B IG D ATA,ACM TKDD, and ACM TMIS. He has served regularly on the organizationcommittees of numerous conferences, such as the Program Co-Chair for
ACM KDD 2018 (research track), ACM KDD 2012 (industry track), andIEEE ICDM 2013, and the General Co-Chair for IEEE ICDM 2015. He is aDistinguished Scientist of ACM.

Qing He received the B.S. degree in mathematics
from Hebei Normal University, Shijiazhang, China,
in 1985, the M.S. degree in mathematics from
Zhengzhou University, Zhengzhou, China, in 1987,
and the Ph.D. degree in fuzzy mathematics and artificial intelligence from Beijing Normal University,
Beijing, China, in 2000.
From 1987 to 1997, he was with the Hebei University of Science and Technology, Shijiazhuang.
He is currently a Doctoral Tutor with the Institute ofComputing and Technology, CAS. He is currently aProfessor with the Institute of Computing Technology, Chinese Academy of
Science (CAS), Beijing, and the Graduate University of Chinese Academyof Sciences (GUCAS). His interests include data mining, machine learning,classification, fuzzy clustering.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 07:58:55 UTC from IEEE Xplore. Restrictions apply.