Unsupervised Domain Adaptation with Imbalanced Cross-Domain Data
Tzu-Ming Harry Hsu1 , Wei-Yu Chen1 , Cheng-An Hou2 , Yao-Hung Hubert Tsai3 ,Yi-Ren Yeh4 , and Yu-Chiang Frank Wang3
1
Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
2
The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
3
Research Center for IT Innovation, Academia Sinica, Taipei, Taiwan
4Department of Mathematics, National Kaohsiung Normal University, Kaohsiung, Taiwan{harry19930924, wyharveychen, c.a.andyhou, y.h.huberttsai}@gmail.com, yryeh@nknu.edu.tw, ycwang@citi.sinica.edu.tw

Abstract
We address a challenging unsupervised domain adaptation problem with imbalanced cross-domain data. For standard unsupervised domain adaptation, one typically obtains
labeled data in the source domain and only observes unlabeled data in the target domain. However, most existing
works do not consider the scenarios in which either the label numbers across domains are different, or the data in thesource and/or target domains might be collected from multiple datasets. To address the aforementioned settings of imbalanced cross-domain data, we propose Closest CommonSpace Learning (CCSL) for associating such data with thecapability of preserving label and structural information
within and across domains. Experiments on multiple crossdomain visual classification tasks confirm that our method
performs favorably against state-of-the-art approaches, especially when imbalanced cross-domain data are presented.

1. Introduction
For pattern recognition problems, one typically trains theclassifiers using pre-collected training data, aiming at recognizing test instances which are not seen during training.
This implies that training and test data exhibit similar dataor feature distributions. However, in real-world applications, training and test data might be collected by different users, using distinct sensors, at dissimilar scenarios, or
during separate time periods. Such data are considered to
be present in different domains, and the difference between
them (or the mismatch between their data distributions) is
thus not negligible.
Domain adaptation addresses the tasks in which training
and test data are collected from source and target domains,respectively. Its goal is to eliminate domain differences for

relating cross-domain data. Depending on the availabilityof labeled data in the target domain during training, one
can generally divide existing techniques into two categories:
semi-supervised and unsupervised domain adaptation.
For semi-supervised domain adaptation, either a small
number of target-domain labeled data or cross-domain data
pairs can be observed during training [21]. For example, Jiang and Zhai [15] apply instance reweighting techniques for adapting classifiers learned from source to target domains. By utilizing the correspondence information
across source and target domains, Huang and Wang [12]advance dictionary learning to derive a common feature
space, which can be applied for the tasks of cross-domainclassification and synthesis. As noted in [23], adaptation
problems with significant domain differences or distribution mismatches (e.g., pose-invariant face recognition [23]or image-to-text classification [28]) generally require semisupervised settings for achieving satisfactory performance.
For unsupervised domain adaptation, one can collect labeled data in the source domain, while only unlabeled data
to be recognized can be observed in the target domain during training (no cross-domain instance pair is available either). Since there is no label information available in thetarget domain, how to transfer such information from thesource domain becomes a challenging task. Based on Maximum Mean Discrepancy (MMD) [9], recent approaches
choose to eliminate the domain difference by matching
cross-domain data distributions [30, 24, 20, 17]. As discussed in Section 2, the basic idea of such methods is to derive a common feature space, in which the marginal and/or
conditional distributions of cross-domain can be matched.
However, existing approaches for unsupervised domain
adaptation typically assume that the label numbers of thesource and target domains are the same (e.g., [20, 18, 17]).
They also expect that the data of each class presented in thesource or target domains exhibit similar data distributions.
In practice, the number of categories in the source domain is14121

Source Domain(s)Data of class 1
Data of class 3

Data of class 2

Target Domain

Closest Common Space

Unlabeled data

Projected target-domain
data with assigned labels

(b)

(a)

(c)

Figure 1: Overview of our approach: (a) Source and target-domain data, (b) exploiting label and latent-domain information
within and across domains, and (c) the resulting space for adaptation and classification. Note that different shapes correspondto different domains/datasets, while the colors denote the object categories.
often larger than that in the target domain. Moreover, both
source and target-domain data might be collected from multiple datasets. In this paper, we refer to the aforementioned
scenarios as the presence of imbalanced cross-domain data.
While some researchers advocate instance selection or
latent domain discovery [6, 7, 27] to handle problems withproblems with mixed source-domain data, they cannot beeasily applied for solving domain adaptation tasks in which
the label numbers do not match across domains. In ourwork, we also propose an MMD-based algorithm of Closest
Common Space Learning (CCSL). The major advantage of
our CCSL is its ability in dealing with imbalanced crossdomain data for unsupervised domain adaptation. We willshow that, by exploiting label and structural information
within and across domains, latent source domains can be
identified for adaptation and recognition purposes.
The contributions of this paper are summarized below:• We propose a novel unsupervised domain adaptation algorithm of Closest Common Space Learning(CCSL), which jointly solves instance reweighting andsubspace learning to learn the latent sub-domains foradaptation. (Section 3)• Our CCSL exploits both label and structural information for data within and across domains. This isachieved by relating latent source-target domain pairs,with the ability to disregard irrelevant source domain
instances during adaptation. (Section 3)• In addition to achieving satisfactory performance onbenchmark cross-domain classification datasets, ourmethod is able to perform favorably against recent unsupervised domain adaptation approaches on problems
with imbalanced cross-domain data. (Section 4)

2. Related Works
In this section, we briefly review recent works on unsupervised domain adaptation. Generally, one can divide existing approaches into three categories: instance reweighting [13, 25], feature space matching [20, 8, 5, 17, 6], andlatent domain discovery [11, 7]. Viewing the importance orcontribution of each source-domain instance different during adaptation, instance reweighting suppresses the difference between source and target domain data by minimizing the MMD [9] or the Kullback-Leibler distances [25].
Classification-based methods like [3] apply selected sourcedomain classifiers to recognize the matched target-domain
instances. Nevertheless, reweighting the source-domain
data might be not sufficient for adapting cross-domain data,
if the domain difference is not simply a domain shift.
Feature space matching is among the popular techniques
for unsupervised domain adaptation. Such strategies aim at
discovering a common feature space which allows matching of data distributions across domains. For example, Panet al. [20] proposed Transfer Component Analysis (TCA)
to project cross-domain data into low dimensional embeddings for matching their marginal distributions. Long etal. developed [17] Joint Distribution Adaptation (JDA),
which adapts both marginal and conditional data distributions when deriving the common feature space. Different from MMD-based approaches, Gong et al. [8, 6] constructed a Riemannian manifold and defined Geodesic Flow
Kernel (GFK) for matching cross-domain data. Similarly,
Baktashmotlagh et al. [1, 2] applied manifold learning toachieve the above goal by minimizing the Hellinger distance between cross-domain data distributions. Dictionarylearning based approaches methods like [19, 29] can also
be considered in this category. With the same goal of associating cross-domain data, they adapt the source-domain
dictionary to the target domain by observing the data in thatdomain accordingly.
24122

To deal with data collected from more than one domain,
latent domain discovery decomposes the observed source ortarget-domain data into multiple sub-domains for improved
adaptation. For example, Hoffman et al. [11] chose to cluster the source-domain data with constraints on their label
information. To minimize the MMD between the sourcedomain data in different sub-domains, Gong et al. [7] pursued their maximally distinctive distributions. Recently, Xu
et al. [27] utilized exemplar SVMs to identify multiple subdomains for source-domain data via low-rank approximation. However, once the sub-domains are determined, theaforementioned works simply select the one closest to thetarget domain for adaptation. Moreover, existing latent domain discovery approaches typically assume that the label
numbers are the same across domains, which would also
limit their practical uses.

3. Our Proposed Method
3.1. Problem Settings
We first define the problem formulation and introduce
the notations which will be used in this paper. Let the train NSing data in the source domain as DS = xSi , yiS i=1 ={XS , yS }, where XS ∈ Rl×NS denotes NS l-dimensionalsource-domain data, and each entry yiS in yS ∈ RNS indicates the corresponding label of C categories. As forthe target domain, the unlabeled data are represented using the same type of features. Thus, we have DT =
 T T  NT
xj , yj j=1 = {XT , yT }, where XT ∈ Rl×NT is theobserved target-domain data, and yT ∈ RNT is the label
vector to be determined.
It is worth repeating that, for unsupervised domain adaptation with imbalanced cross-domain data, we not only deal
with possible mixed source or target domain data (i.e., instances in XS or XT of the same class but collected fromdifferent datasets). We also consider that the label number
C of the source domain might be larger than or equal to thatin the target domain.

3.2. Beyond Matching Cross-Domain Marginal andConditional Distributions
Recall that, to eliminate the domain differences, JDA
determines a feature transformation Φ (·), which projects
source and target domain data to a common subspace formatching cross-domain marginal and conditional data distributions. In other words, the goal of JDA is to satisfy PS (φ(XS )) ≈ PT (φ(XT )) and PS (φ(XS )|yS ) ≈PT (φ(XT )|yT ) by minimizing the following MMD distance Mφ :Mφ (PS (XS , yS ), PT (XT , yT ))Since only unlabeled data can be observed in the target domain, JDA applies source-domain classifiers to predict thepseudo labels of the target-domain data, which allows thematching of cross-domain conditional data distributions foradaptation purposes.
Despite promising performance, JDA and most MMDbased approaches regard each data domain as an atomic distribution. In practice, source or target-domain data can becollected by different users using distinct sensors, and thus
there would exist latent sub-domains for the collected data.
Moreover, the number of categories in the source domain
might be larger than that in the target domain.
To address unsupervised domain adaptation with imbalanced cross-domain data, we propose a novel algorithm ofClosest Common Space Learning (CCSL). Instead of assuming that the data in each domain exhibit atomic distributions, our CCSL considers a latent domain variable d for
exploiting both label and structural information within andacross domains during adaptation. Thus, our CCSL aims at
minimizing the following MMD distance Mφ,d (d denotes
domain-dependent MMD) :Mφ,d (PS (XS , yS ), PT (XT , yT ))≈ Mφ (PS (XS ), PT (XT ))+ Mφ,d (PS (XS |yS ), PT (XT |yT )) .

(1)

The first term in (1) denotes the matching of cross-domain
marginal distributions, and the second term takes both label
and latent-domain information for matching cross-domain
conditional distributions. That is, different from matching
cross-domain conditional distributions using class means
with pseudo labels, we propose to exploit both label andlatent structure similarities within and across domains foradaptation. This is achieved by jointly solving the tasks
of instance reweighting and subspace learning in a unified
framework, as detailed in the following subsection.

3.3. Closest Common Space Learning
As noted in Section 3.2, we propose to exploit both label
and latent-domain information within and across domains
for matching cross-domain conditional distribution. Morespecifically, we define the second term in (1) as:

where≈ Mφ (PS (XS ), PT (XT )) + Mφ (PS (XS |yS ), PT (XT |yT )) .

Mφ,d (PS (XS |yS ), PT (XT |yT ))X
2mST
P SS ijP T T φ̂(xSi ) − φ̂(xTj ) ,
=
mki
mlj
i,j

M=

34123



k

l

MSS
MT S

MST
MT T



∈ R(NS +NT )×(NS +NT )

(2)

and
φ̂(xSi ) =

P
l

S
mSS
li φ(xl )P SS , φ̂(xTj ) =
mli
l

P
l

mTljT φ(xTl )
P TT
.
mlj
l

(NS +NT )×(NS +NT )In (2), the similarity matrix M ∈ R
associates each within and cross-domain data pair. Each enST
try mST
meaij in the cross-domain similarity matrix M
sures the label and latent-domain similarities for each crossTT
domain data pair, while mSSli and mlj exploit the latent
structures for the associated data within source and target
domains, respectively (see Section 3.3.1 for the derivationof M). As a result, minimizing (2) is equivalent to thematching of cross-domain data distributions based on theconditions of the observed labels and latent domains.
3.3.1

Observing label and latent-domain similarities

We now explain how we determine M in (2). Given labeled source-domain and unlabeled target-domain data, weapply a set of linear discriminators wi , each is trainedby a source or target-domain instance of interest in the
resulting feature space (via φ). Thus, we have W =[w1 , ..., wNS , ...wNS +NT ] ∈ Rk×(NS +NT ) , where k indicates the dimensions of our closest common space. To learn
each wi , we follow the strategies below:• If wi is trained by a projected source-domain instanceφ(xi ), we take a portion p (0 6 p 6 1) of the projected
data with the same label as xj as positive instances (selected by nearest neighbors), while the remaining ones
with distinct labels will viewed as negative samples.
• If wi is trained by a projected target-domain instanceφ(xi ), we follow JDA and apply source-domain SVMs
to predict its pseudo label ŷiT . The procedure of selecting positive and negative samples to train wi for xi isthe same as the case above.
Once wi for each instance is derived, we apply them to predict the output scores p for each instance xj , which is in
the same or different domain as xi is. Finally, this scorewill be normalized to [0, 1] as the corresponding entry in
M using a sigmoid function σ (g) = 1/ (1 + e−g ), whereg = wiT φ(xj ). Once the similarity matrices of MSS ,MT T , and MST are determined, φ̂(xSi ) and φ̂(xTj ) can bederived based on their definitions in (2).
It can be seen that, instead of measuring the difference
between cross-domain instance pairs, the use of φ̂(xSi ) andφ̂(xTj ) in (2) allows us to take local structures of each projected source or target-domain instance into consideration,
while class labels are implicitly embedded in M.
It is worth noting that, while matching cross-domain
marginal distributions in (1) can be viewed as eliminating

the domain/dataset bias (as TCA does), matching crossdomain data distributions based on the observed label andsub-domain information (i.e., minimizing (2)) introduces
the CCSL the ability to handle imbalanced cross-domain
data during adaptation. In Section 4, we will verify the effectiveness of our CCSL for unsupervised domain adaptation with both balanced and unbalanced cross-domain data.
3.3.2

CCSL as TCA or JDA

We note that, both TCA [20] and JDA [17] can be regarded
as special cases of our proposed CCSL. For TCA, neither
label nor latent domain information are considered whenmatching cross-domain data distributions. Thus, disregarding (2) would simplify our CCSL as TCA. On the other
hand, JDA views each cross-domain pair equally important,
if the target-domain instance of this data pair is predicted
as the same category as the corresponding source-domain
instance is. In other words, if we simply let mST
ij = 1 ifSTyi = ŷj without identifying latent domains for adaptation,
our propose formulation of (2) would turn into JDA.

3.4. Optimization
To solve the minimization of (1), we first rewrite (1) into
the following form:
Mφ,d (PS (XS , yS ), PT (XT , yT )) = tr (Kφ L) ,(3)⊤

where Kφ ≡ φ (X) φ (X) is the kernel matrix constructed
over cross-domain data. The matrix L in (3) is derived as:X⊤
mij vij vij,
L = v0 v0⊤ +
i,j

#⊤
e⊤
e⊤
NT
NS
,−
where v0 =
NS
NT
"
⊤ #⊤
⊤
mTj T
mSS
i
vij =
,−
.
mSS
mTj T 1i
1"Note that eN is a N dimensional vector of ones. mSSandi
mTj T represent the i and jth column vectors of MSS and
MT T , respectively.
As pointed out in [20], it is computationally expensive to solve the optimization problem of (3). Therefore, following [20, 17], we utilize the Empirical Kernel Mapping [22] and predefine a kernel matrix K =(KK−1/2 )(K−1/2 K). Next, we determine projections Ã
and A (both of size (NS + NT ) × k) for deriving a lower
k dimensional space in terms of K. This is achieved byhaving Kφ = (KK−1/2 Ã)(Ã⊤ K−1/2 K) = KAA⊤ K,
where A = K−1/2 Ã, where Ã is to transform the corresponding feature vectors to a lower k-dimensional space.
44124

Algorithm 1 CCSL: Closest Common Space Learning

USPS

Input: Data matrix K, source-domain label yS , dim. k, and α
1. Initialize Initialize: M ←− eN e⊤
N
while not converged do
2. A ←− solution of (4)3. Data embedding Z = [ZS , ZT ] ←− A⊤ K
4. Train classifier f ←− {ZS , yS } and ŷT ←− f (ZT )5. Train linear discriminators
W
6. M ←− σ W⊤ Z
end while
Output: Target-domain label ŷT

Finally, by rewriting Kφ in (3), we solve the following objective function for CCSL:
min tr A⊤ KLK⊤ A + αkAk2F
As.t.

A⊤ KHK⊤ A = I,

(4)

where α controls the regularization of A, and H = I −eN e ⊤
N /N is the centering matrix which preserves data variance after the projection.
By applying Λ = diag (λ1 , · · · , λk ) ∈ Rk×k as the Lagrange multiplier, solving (4) is equivalent to minimizing
the following function: 
L = tr A⊤ KLK⊤ + αI A
(5)
 
⊤
⊤
+ tr I − A KHK A Λ .

By setting ∂L/∂A = 0, the above problem turns into a
generalized eigen-decomposition task. In other words, we
calculate the k smallest eigenvectors of the following problem for determining the optimal A:KLK⊤ + αI A = KHK⊤ AΛ.

As summarized in Algorithm 1, we apply the techniqueof iterative optimization to calculate the projection A, linear
discriminators w, and similarity matrix M for CCSL. Oncethe closest common space is derived, one can perform classification using projected cross-domain data accordingly.

4. Experiments
4.1. Datasets and Settings
4.1.1

Cross-domain datasets for visual classification

In our experiments, we evaluate the recognition performance of our proposed method on several cross-domain visual classification tasks. We first consider two handwritten
digit datasets of MNIST [16] and USPS [14] (denoted as M
and U, respectively). The former contains a training set of60, 000 images of 10 digits, and 10, 000 images are available for testing. The resolution of each image is of size

Amazon

MNIST

DSLR

WebcamCaltech-256

Figure 2: Example images of different datasets for crossdomain visual classification.
28 × 28 pixels. As for USPS, there are 7291 and 2007 images available for training and testing, respectively. Each
image in this dataset is of size 16 × 16 pixels.
We also consider cross-domain object recognition, using the datasets of Caltech-256 (C) [10] and Office [21]datasets. The former consists of real-world object images of
256 categories with at least 80 instances per category, while
the latter contains 31 object categories from three different
domains, i.e., Amazon (A), DSLR (D), and webcam (W).
As suggested by [6, 17], 10 overlapping categories acrossthe above four domains are selected for experiments. Example images of the above datasets are shown in Figure 2.
For fair comparisons, we follow the setting of [20] andrandomly sample 2000 and 1800 images from MNIST and
USPS (scaled to the same 16 × 16 pixels), respectively.
And, we use pixel intensities are the associated image features. As for cross-domain object recognition, DeCAF6
features [4] with 4096 dimensions are adopted, since theuse of such deep-learning based features have shown very
promising results for visual classification [4].
4.1.2

Settings and parameters

To compare our CCSL with existing unsupervised domain
adaptation approaches, we consider the methods of Transfer
Component Analysis (TCA) [20], Joint Distribution Adaptation (JDA) [17], and Transfer Joint Matching (TJM) [18]in our experiments. We also apply standard SVM trainedby source-domain data, which indicates direct recognition
without adaptation (denoted as SVM). Although the recent
approach of [7] is able to handle mixed source-domain data,
its focus is to identify the best subset of the source-domain
data, followed by using GFK [8, 6] for performing adaptation. Moreover, the label numbers are assumed to be thesame across domains in [7].
It is worth noting that, since no labeled data can be observed in the target domain, performing cross-validation
for parameter selection is not applicable. Thus, we sim54125

Table 1: Accuracy (%) for cross-domain hand written digit
and object classification with balanced cross-domain data.
Note that CCSL performs comparably as JDA and TJM do(* indicates cross-domain object recognition only).

Table 2: Accuracy (%) for cross-domain object recognition with imbalanced label numbers (the best performanceis highlighted in bold). Note that the label numbers are 10
and 5 for source and target domain data, respectively.

S→T

SVM

TCA

JDA

TJM

CCSL

S → T5

SVM

TCA

JDA

TJM

CCSL

M→U
U→M

44.28
39.30

52.33
46.90

51.78
57.80

60.83
47.50

53.78
58.10

C→A
D→A
W→A
A→C
D→C
W→C
A→D
C→D
W→D
A→W
C→W
D→W

91.54
87.06
75.78
85.13
79.07
72.84
85.99
89.17
99.36
76.95
80.00
98.64

90.92
88.62
80.27
82.37
79.52
74.71
87.26
89.81
100.00
74.58
78.98
99.32

90.92
90.28
87.02
86.33
83.88
83.64
88.54
90.36
100.00
83.78
85.08
97.98

89.77
89.46
86.12
79.43
78.90
75.78
82.17
85.99
100.00
75.93
78.64
98.98

93.32
90.92
89.98
87.18
84.06
82.90
87.26
87.90
96.18
83.05
82.37
96.27

C → A5
D → A5
W → A5
A → C5
D → C5
W → C5
A → D5
C → D5
W → D5
A → W5
C → W5
D → W5

90.88
85.96
74.10
83.62
77.17
74.09
79.66
90.66
99.24
75.12
80.85
98.38

90.55
87.90
79.90
80.84
78.45
73.32
80.95
87.69
100.00
73.06
71.11
99.59

90.20
80.42
76.92
77.49
78.24
75.04
73.69
83.05
99.70
71.34
68.85
98.16

86.93
83.24
78.91
69.63
65.00
72.63
85.35
87.09
100.00
73.45
78.09
98.81

93.32
95.04
91.61
91.88
89.00
83.22
86.55
85.27
99.13
85.01
84.10
96.65

Average

84.14

83.61

81.09

81.59

90.07

Average*

85.13

85.53

88.98

85.10

88.45

SVM

TCA

JDA

TJM

CCSL

L

ply choose linear SVMs for all approaches (i.e., linear
SVMs are trained using projected source-domain data for
all MMD-based approaches). For data embedding in TCA,
JDA, and CCSL, we apply linear kernels for constructing
the kernel matrix as suggested by [17, 20]. As for the remaining parameters, we set the regularization parameter αin (3) as 0.1 and 1 for cross-domain digit and object recognition, respectively. To fix the reduced dimensions for all
MMD-based approaches for comparisons, we have k = 15
and k = 100 for the above two tasks.

Accuracy
Accuracy (%)
(%) 

0.95
0.9
0.85
0.8
0.75
4

5

6

7

Label
L SizeLabelSubset
numbersC

8

9

Figure 3: Average results over different target-domain label
numbers C for cross-domain object recognition. Note thatthe source-domain label number is fixed as 10.

4.2. Evaluation
4.2.1

Classification with balanced cross-domain data

For cross-domain handwritten digit recognition, two classification tasks need to be addressed, i.e., M → U and U → M(S → T indicates adapting data from S to T domains). As
for cross-domain object recognition, we have a total of 12
cross-domain pairs to be evaluated.
Table 1 lists the recognition results of all methods on theabove cross-domain tests. Since all the cross-domain pairs
are balanced, i.e., the label and domain numbers acrosssource and target domains are the same, our CCSL produced comparable performance as JDA did. Since TCA and
TJM did not utilize any label information during adaptation,
degraded performances were obtained.
From Table 1, we see that our CCSL is favorable fortarget domains with larger sizes (e.g., |A| = 958 and|C| = 1123). This is due to the fact that our CCSL isable to identify proper local data structures for adaptation.
Nevertheless, the following experiments using imbalanced

cross-domain data will further verify the effectiveness androbustness of our method.
4.2.2

Classification with imbalanced label numbers

For the experiments with imbalanced cross-domain data,
we first consider the scenario of imbalanced label numbers
across domains. More specifically, we consider the taskof cross-domain object recognition, in which the sourcedomain label number is larger than that in the target domain.
Among the 10 overlapping object categories for Caltech256 and Office, we randomly select C = 4 ∼ 9 as the label
numbers in the target domain. And, all labeled data of all
10 categories are applied as the source-domain data. Dueto space limit, we only present the classification results of
C = 5 for all domain pairs in Table 2. From this table,
we see that TCA, JDA, and TJM were not able to produce
satisfactory results, while improved performance was stillobtained by our CCSL. The degraded performance of exist64126

Table 3: Accuracy (%) for cross-domain object recognition
with mixed-domain data. Note that the best performancefor each mixed domain pair is highlighted in bold.

TCA

JDA

TJM

85

90

SVM

TCA

JDA

TJM

LM

CCSL

80

C+D+W → A
A+D+W → C

91.65
85.66

91.34
83.17

91.34
86.02

89.98
80.50

91.75
87.00

93.75
87.98

75

D+W → A+C
C+W → A+D
C+D → A+W
A+W → C+D
A+D → C+W
A+C → D+W
Average

80.29
93.16
93.44
88.44
89.20
87.78
89.99

80.98
92.96
92.64
87.84
87.57
88.75
89.75

88.66
93.26
92.55
89.04
89.67
91.69
91.58

84.31
91.77
91.40
85.62
86.09
91.69
89.73

86.35
93.45
93.62
88.83
89.77
91.69
90.31

89.18
93.86
93.62
89.38
89.98
91.69
92.15

Classification with mixed-domain data

Finally, we consider cross-domain object recognition using
mixed-domain data. Table 3 lists and compares the performances of different approaches, including LM [6]. The firsttwo rows in Table 3 represent the scenarios of mixed sourcedomain data, with unlabeled data to be recognized collectedfrom a single target domain. As for the remaining rows inTable 3, both labeled and unlabeled data are collected frommultiple domains, and thus multiple latent domains are expected for both source and target domains.
From Table 3, we observe that improved recognition results were obtained by our CCSL. It can also be seen that,the difference between our CCSL and other recent/baseline
approaches was not as significant as those presented in theprevious subsection. This implies that, for practical unsupervised domain adaptation task, solving imbalanced label
numbers across domains is a more challenging task than thatwith mixed-domain data. Nevertheless, training data (andtheir labels) collected in real-world scenarios are typicallynoisy and imbalanced across domains. As verified above,a robust unsupervised domain adaptation with the ability tohandle imbalanced cross-domain data would be preferable.

4.3. Remarks
4.3.1

Convergence analysis and parameter sensitivity

We first provide remarks on the convergence issue for our
proposed algorithm. For both cross-domain digit and ob-

JDA

TJM

CCSL

80
70

2

4

6

8

10

2

4

(a)

6

8

10

(b)

Figure 4: Convergence analysis (accuracy (%) vs. numberof iterations) on (a) balanced domain pair of A → C and (b)imbalanced domain pair of A → C5 .
A→C

M→U

4.2.3

TCA
100

S→T

ing MMD-based approaches is due to their assumption ofbalanced label numbers across source and target domains.
In addition to Table 2, Figure 3 further compares the average performances (over all 12 domain pairs) of different
methods using different label numbers C with 10 random
trials. From Figure 3, we see that our CCSL performed favorably against existing MMD-based methods, especiallywhen C became smaller. This suggests that the advantage
of our CCSL would become clearer if highly imbalancedlabel numbers are expected to be present across domains.

CCSL

90

100

80

80

60

60

40
−3
10

−2

10

−1

10

α

(a)

0

10

A→C

M→U

5

100

1

10

5

40
0.2

0.4

0.6

0.8

p

(b)

Figure 5: Parameter sensitivity. We show the recognition
accuracy (%) over (a) α and (b) p on selected domain pairs.
ject recognition, we observe that the optimization of CCSL
always converged within 5 iterations for both balanced andimbalanced settings (as shown in Figures 4a and b). We also
observe that, when dealing with imbalanced cross-domain
data, the convergence of existing MMD-based methods
like JDA and TJM does not necessarily correspond to
non-decreasing performance improvements. Such trends
were not observed for the experiments with balanced crossdomain data. This further verifies our advantages in identifying sub-domains for improved adaption.
Figures 5a and b further verify the sensitivity of α in (4)and p in Section 3.3.1. In our experiments, we fix p = 0.5
and set α = 0.1 and 1 for cross-domain digit and object
recognition, respectively. From Figures 5a and b, we seethat performance would not be sensitive to the parameters
around our choices.
4.3.2

Visualization of adapting imbalanced crossdomain data

In Sections 4.2, we provide experimental results whichquantitatively verify the effectiveness of our approach forcross-domain visual classification. To qualitatively support
the use of our CCSL for unsupervised domain adaptation(especially for imbalanced cross-domain data), we now discuss the resulting cross-domain data similarity and visualize
the data embedding for the adapted data using t-distributed
stochastic neighbor embedding (t-SNE) [26].
Figure 6 shows the cross-domain similarity and data embedding analysis for the imbalanced domain pair of A →C5 . To plot the cross-domain similarity, we construct the
74127

1

(a)

(b)

(c)

(d)

Figure 6: Analysis of cross-domain similarity and data embedding for the imbalanced domain pair of A → C5 . For crossdomain similarity, we show the affinity matrices of cross-domain data derived by (a) JDA and (b) CCSL. For data embedding,we present the 2D visualization of t-SNE for projected cross-domain data derived by (c) JDA and (d) CCSL. In (c) and (d),instances in different colors denote data of different object categories.
affinity matrix, in which each entry denotes the inner product of the associated cross-domain data pair. Once the affinity matrix is obtained, a threshold of 0.8 is applied to binarize this matrix for visualization purposes. Comparing
Figures 6(a) and (b), we see a large number of irrelevant
entries were nonzero in the affinity matrix of JDA, while
the dominant (non-zero) ones in our affinity matrix mainly
corresponded to the object categories to be transferred.
Figure 6(c) and (d) illustrate the 2D visualization of tSNE for adapted cross-domain data (i.e., those projected
into the common spaces derived by JDA or CCSL). Fromthese two figures, it is clear that CCSL was able to preserve
the label and structural information for cross-domain datawith the same class. As for JDA, the separation between
projected data of different classes was not sufficient. From
the quantitative experiments presented in Sections 4.2, together with the qualitative and visual comparisons provided
in this subsection, the effectiveness and robustness of our
proposed method can be successfully verified.

5. Conclusion
In this paper, we presented Closest Common Space
Learning (CCSL) for unsupervised domain adaptation. In
particular, our CCSL is designed to handle mixed-domain
data or imbalanced label numbers across domains during adaptation. Solving our proposed algorithm can beviewed as jointly optimizing the tasks of instance reweighting and subspace learning, which exploits label and subdomain information for data within and across domains.
In addition to providing the optimization details for deriving CCSL solutions, we also relate CCSL with popular MMD-based approaches of TCA and JDA. This showsthat our CCSL is a robust unsupervised domain adaptation
approach for both scenarios of balanced and imbalanced

cross-domain data. Finally, we conducted experiments on
multiple cross-domain visual classification problems. The
empirical results confirmed that our CCSL performs favorably against state-of-the-art unsupervised domain adaptation approaches, especially when imbalanced cross-domain
data are presented.

6. Acknowledgement
This work was supported in part by the Ministry of Science and Technology of Taiwan under Grants MOST1032221-E-001-021-MY2 and MOST104-2221-E-017-016.

References[1] M. Baktashmotlagh, M. T. Harandi, B. C. Lovell, and
M. Salzmann. Unsupervised domain adaptation by domain invariant projection. In IEEE ICCV, 2013.
[2] M. Baktashmotlagh, M. T. Harandi, B. C. Lovell, and
M. Salzmann. Domain adaptation on the statisticalmanifold. In IEEE CVPR, 2014.
[3] L. Bruzzone and M. Marconcini. Domain adaptation
problems: A DASVM classification technique and acircular validation strategy. PAMI, 2010.
[4] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. arXiv
preprint arXiv:1310.1531, 2013.
[5] B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars. Unsupervised visual domain adaptation using
subspace alignment. In IEEE ICCV, 2013.
[6] B. Gong, K. Grauman, and F. Sha. Connecting the dotswith landmarks: Discriminatively learning domaininvariant features for unsupervised domain adaptation.
In ICML, 2013.
84128

[7] B. Gong, K. Grauman, and F. Sha. Reshaping visual
datasets for domain adaptation. In NIPS, 2013.
[8] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic
flow kernel for unsupervised domain adaptation. 2012.
[9] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf,
and A. J. Smola. A kernel method for the two-sampleproblem. In NIPS, 2006.
[10] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. 2007.
[11] J. Hoffman, B. Kulis, T. Darrell, and K. Saenko. Discovering latent domains for multisource domain adaptation. In ECCV. Springer, 2012.
[12] D.-A. Huang and Y.-C. F. Wang. Coupled dictionary
and feature space learning with applications to crossdomain image synthesis and recognition. In IEEE
ICCV, 2013.
[13] J. Huang, A. Gretton, K. M. Borgwardt, B. Schölkopf,
and A. J. Smola. Correcting sample selection bias byunlabeled data. In NIPS, 2006.
[14] J. J. Hull. A database for handwritten text recognition
research. PAMI, 1994.
[15] J. Jiang and C. Zhai. Instance weighting for domain
adaptation in NLP. In ACL, volume 7, pages 264–271,
2007.
[16] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.
Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 1998.
[17] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu.
Transfer feature learning with joint distribution adaptation. In IEEE ICCV, 2013.
[18] M. Long, J. Wang, G. Ding, J. Sun, and P. S.
Yu. Transfer joint matching for unsupervised domain
adaptation. In IEEE CVPR, 2014.
[19] J. Ni, Q. Qiu, and R. Chellappa. Subspace interpolation via dictionary learning for unsupervised domain
adaptation. In IEEE CVPR, 2013.
[20] S. J. Pan, I. W. Tsang, J. T. Kwok, Q. Yang, et al.
Domain adaptation via transfer component analysis.
IEEE Transactions on Neural Networks, 2011.
[21] K. Saenko et al. Adapting visual category models to
new domains. In ECCV. 2010.
[22] B. Schölkopf, A. Smola, and K.-R. Müller. Nonlinear
component analysis as a kernel eigenvalue problem.
Neural computation, 1998.
[23] A. Sharma, A. Kumar, H. Daume, and D. W. Jacobs.
Generalized multiview analysis: A discriminative latent space. In IEEE CVPR, 2012.
[24] S. Si, D. Tao, and B. Geng. Bregman divergence-based
regularization for transfer subspace learning. IEEE
TKDE, 22, 2010.

[25] M. Sugiyama, S. Nakajima, H. Kashima, P. V. Buenau,
and M. Kawanabe. Direct importance estimation withmodel selection and its application to covariate shift
adaptation. In NIPS, 2008.
[26] L. Van der Maaten and G. Hinton. Visualizing datausing t-SNE. Journal of Machine Learning Research,
9(2579-2605):85, 2008.
[27] Z. Xu, W. Li, L. Niu, and D. Xu. Exploiting low-rank
structure from latent domains for domain generalization. In ECCV. Springer, 2014.
[28] Y. Yeh, C. Huang, and Y. Wang. Heterogeneous domain adaptation and classification by exploiting thecorrelation subspace. 2014.
[29] C. Zhang, Y. Zhang, S. Wang, J. Pang, C. Liang,
Q. Huang, and Q. Tian. Undo the codebook bias bylinear transformation for visual applications. In ACM
MM, 2013.
[30] E. Zhong, W. Fan, J. Peng, K. Zhang, J. Ren,
D. Turaga, and O. Verscheure. Cross domain distribution adaptation via kernel mapping. In ACM KDD,
2009.

94129