Learning Cross-Domain Landmarks for Heterogeneous Domain Adaptation
Yao-Hung Hubert Tsai1 , Yi-Ren Yeh2 , Yu-Chiang Frank Wang1
1
Research Center for IT Innovation,Academia Sinica, Taipei, Taiwan
2
Department of Mathematics, National Kaohsiung Normal University, Kaohsiung, Taiwan
y.h.huberttsai@gmail.com, yryeh@nknu.edu.tw, ycwang@citi.sinica.edu.tw

Abstract
While domain adaptation (DA) aims to associate thelearning tasks across data domains, heterogeneous domain
adaptation (HDA) particularly deals with learning fromcross-domain data which are of different types of features.
In other words, for HDA, data from source and target domains are observed in separate feature spaces and thus
exhibit distinct distributions. In this paper, we propose anovel learning algorithm of Cross-Domain Landmark Selection (CDLS) for solving the above task. With the goalof deriving a domain-invariant feature subspace for HDA,
our CDLS is able to identify representative cross-domain
data, including the unlabeled ones in the target domain, forperforming adaptation. In addition, the adaptation capabilities of such cross-domain landmarks can be determinedaccordingly. This is the reason why our CDLS is able toachieve promising HDA performance when comparing tostate-of-the-art HDA methods. We conduct classification
experiments using data across different features, domains,and modalities. The effectiveness of our proposed method
can be successfully verified.

1. Introduction
When solving standard machine learning and patternrecognition tasks, one needs to collect a sufficient amountof labeled data for learning purposes. However, it is not always computationally feasible to collect and label data foreach problem of interest. To alleviate this concern, domain
adaptation (DA) aims to utilize labeled data from a source
domain, while only few (or no) labeled data can be observedin the target domain of interest [27]. In other words, the goalof DA is to transfer the knowledge learned from an auxiliary domain, so that the learning task in the target domain
can be solved accordingly.
A variety of real-world applications have been benefited
from the recent advances of DA (e.g., sentiment analysis[7, 33], Wi-Fi localization [25, 26], visual object classification [43, 30, 34, 8], and cross-language text classification [10, 29]). Generally, most existing DA approaches

like [26, 24, 31, 6] consider that source and target-domain
data are collected using the same type of features. Suchproblems can be referred to homogeneous domain adaptation, i.e., cross-domain data are observed in the same feature space but exhibit different distributions.
On the other hand, heterogeneous domain adaptation(HDA) [32, 38, 21, 28] deals with a even more challenging
task, in which cross-domain data are described by different
types of features and thus exhibit distinct distributions (e.g.,training and test image data with different resolutions or encoded by different codebooks). Most existing approaches
choose to solve HDA by learning feature transformation,
which either projects data from one domain to the other,or to project cross-domain data to a common subspace foradaptation purposes [32, 38, 21, 12, 18, 42, 19].
As noted in [39, 23, 40, 41], exploiting unlabeled targetdomain instances during adaptation would be beneficial for
HDA. Following their settings, we choose to address semisupervised HDA in this paper. To be more precise, a sufficient amount and few labeled data will be observed in
the source and target domains, respectively. And, the remaining target-domain data (to be classified) will also bepresented during the learning process. In our work, we
propose a learning algorithm of Cross-Domain Landmark
Selection (CDLS) for solving HDA. Instead of viewing all
cross-domain data to be equally important during adaptation, our CDLS derives a heterogeneous feature transformation which results in a domain-invariant subspace for associating cross-domain data. In addition, the representative source and target-domain data will be jointly exploitedfor improving the adaptation capability of our CDLS. Oncethe adaptation process is complete, one can simply projectcross-domain labeled and unlabeled target domain data into
the derived subspace for performing recognition. Illustration of our CDLS is shown in Figure 1.
The contributions of this paper are highlighted below:• We are among the first to exploit heterogenous source
and target-domain data for learning cross-domain
landmarks, aiming at solving semi-supervised HDA
problems.
15081

Source-Domain Data

: Source-Domain Data
: Labeled Target-Domain Data
: Unlabeled Target-Domain Data

: Unlabeled Target Domain Datawith Predicted Labels

Target-Domain Data

Figure 1. Illustration of Cross-Domain Landmark Selection (CDLS) for heterogeneous domain adaptation (HDA). Note that instances
shown in different sizes indicate cross-domain landmarks with different adaptation abilities.

• By learning the adaptability of cross-domain data (including the unlabeled target-domain ones), our CDLS
derives a domain-invariant feature space for improved
adaptation and classification performances.
• Experiments on classification tasks using data acrossdifferent features, datasets, and modalities data areconsidered. Our CDLS is shown to perform favorably
against state-of-the-art HDA approaches.

2. Related Work
Most existing HDA methods consider a supervised setting, in which only labeled source and target-domain data
are presented during adaptation. Generally, these HDA approaches can be divided into two categories. The first groupof them derive a pair of feature transformation (one foreach domain), so that source and target-domain data can beprojected into a common feature space for adaptation andrecognition [32, 38, 12, 39, 23, 41]. For example, Shi et al.
[32] proposed heterogeneous spectral mapping (HeMap),
which selects source-domain instances for relating crossdomain data via spectral embedding. Wang and Mahadevan[38] chose to solve domain adaptation by manifold alignment (DAMA), with the goal of preserving label information during their alignment/adaptation process. Duan et al.
[12] proposed heterogeneous feature augmentation (HFA)
for learning a common feature subspace, in which SVM
classifiers can be jointly derived for performing recognition.
On the other hand, the second category of existing HDA
methods learn a mapping matrix which transforms the datafrom one domain to another [21, 18, 19, 42, 40]. For example, Kulis et al. [21] proposed asymmetric regularizedcross-domain transformation (ARC-t) for associating crossdomain data with label information guarantees. Hoffman etal. [18, 19] considered a max-margin domain transformation (MMDT) approach, which adapts the observed prediction models (i.e., SVMs) across domains. Similarly, Zhou
et al. [42] presented an algorithm of sparse heterogeneous
feature representation (SHFR), which relate the predictive
structures produced by the prediction models across do-mains for addressing cross-domain classification problems.
Nevertheless, the above HDA approaches only considered labeled source and target-domain data during adaption.
As noted in literature on homogeneous domain adaptation[13, 26, 22], unlabeled target-domain data can be jointly exploited for adaptation. Such semi-supervised settings havebeen shown to be effective for DA. Recent HDA approaches
like [39, 23, 40, 41] also benefit from this idea. For example, inspired by canonical correlation analysis, Wu etal. [39] presented an algorithm of heterogeneous transfer
discriminant-analysis of canonical correlations (HTDCC).
Their objective is to minimize canonical correlation between inter-class samples, while that between the intra-classones can be maximized. Li et al. [23] extended HFA[12] to a semi-supervised version (i.e., SHFA), and further exploited projected unlabeled target-domain data into
their learning process. Recently, Xiao and Guo [40] proposed semi-supervised kernel matching for domain adaptation (SSKMDA), which determines a feature space while
preserving cross-domain data locality information. They
also proposed a semi-supervised HDA approach of subspace co-projection (SCP) in [41], and chose to minimize
the divergence between cross-domain features and their prediction models. However, both SSKMDA and SCP require
additional unlabeled source-domain data for HDA.
It is worth noting that, techniques of instance reweighting or landmark selection have been applied for solving domain adaptation problems [20, 9, 15, 1]. Existing methods typically weight or select source-domain data, and only
homogeneous DA settings are considered. Moreover, nolabel information from either domain is taken into consideration during the adaptation process. Based on the above
observations, we address semi-supervised HDA by proposing CDLS for learning representative landmarks from crossdomain data. Our CDLS is able to identify the contributions of each landmark when matching cross-domain classconditional data distributions in the derived feature space.
This is why improved HDA performance can be expected(see our experiments in Section 4).
5082

3. Our Proposed Method
3.1. Problem Settings and NotationsS
Let DS = {xis , ysi }ni=1= {XS , yS } denote sourcedomain data, where each instance xs ∈ RdS is assigned with label ys ∈ L = {1, . . . , C} (i.e., a Ccardinality label set). For HDA with a semi-supervisedLsetting, we let DL = {xil , yli }ni=1= {XL , yL } andi nUi
DU = {xu , yu }i=1 = {XU , yU } as labeled and unlabeled target-domain data, respectively. We have xl , xu ∈RdT and yl , yu ∈ L. Note that dS 6= dT , since source
and target-domain data are of different types of features.
In a nutshell, for solving semi-supervised HDA, we observe dS -dimensional source-domain data DS , few labeleddT -dimensional target-domain data DL , and all unlabeled
target-domain data DU (to be recognized) during adaptation. Our goal is to predict the labels yU for XU .
For semi-supervised HDA, only few labeled data are presented in the target domain, while the remaining unlabeled
target-domain instances are seen during the adaptation process. Nevertheless, a sufficient number of labeled data will
be available in the source domain for performing adaptation. Different from [40, 41], we do not require any additional unlabeled source-domain data for adaptation.

Recall that, as discussed in Section 2, most existing HDA
works consider a supervised setting. That is, only labeleddata in the target domain are utilized during training. In order to match cross-domain data distributions for such standard supervised HDA settings, we formulate the problem
formulation as follows:2

min EM (A, DS , DL ) + EC (A, DS , DL ) + λ kAk , (1)A

where EM and EC measure the differences between crossdomain marginal and conditional data distributions, respectively. The last term in (1) is penalized by λ to prevent
overfitting A.
To solve (1), we resort to statistics to measure the probability density for describing data distributions. As suggested in [26, 24], we adopt empirical Maximum Mean Discrepancy (MMD) [16] to measure the difference between
the above cross-domain data distributions. As a result,
EM (A, DS , DL ) can be calculated asnSnL
1 X ⊤ i
1 X i
bl
EM (A, DS , DL ) =A xs −xnS i=1nL i=1

Matching cross-domain data distributions

EC (A, DS , DL ) =

C
X
c=1

Inspired by the recent HDA approaches of [21, 18, 19, 42,40], we learn a feature transformation A which projects
source-domain data to the subspace derived from the targetdomain data. In this derived subspace, our goal is toassociate cross-domain data with different feature dimensionality and distinct distributions. Such association isachieved by eliminating the domain bias via matching
cross-domain data distributions (i.e., marginal distributions PT (XT ) and PS (A⊤ XS ), and the conditional ones
PT (yT |XT ) and PS (yS |A⊤ XS )). Unfortunately, direct
estimation of PT (yT |XT ) and PS (yS |A⊤ XS ) is not possible. As an alternative solution suggested in [24], we choose
to match the estimated PT (XT |yT ) and PS (A⊤ XS |yS ) by
Bayes’ Theorem [4].
As noted above, we first project all target-domain data
into a m-dimensional subspace via PCA (note that m ≤blmin{dS , dT }). In the remaining of this paper, we use x
bu to denote reduced-dimensional labeled and unlaand x
beled target-domain data, respectively. As a result, the linear transformation to be learned is A ∈ Rm×dS . It is worth
noting that, this dimension reduction stage is to prevent possible overfitting caused by mapping low-dimensional data
in a high-dimensional space. Later in our experiments, wewill vary the dimension number m for the completeness of
our evaluation.

.

(2)

For matching cross-domain conditional data distributions,
we determine EC (A, DS , DL ) as:

3.2. Cross-Domain Landmark Selection
3.2.1

2

c

c

nS
nL
1 X i,c
1 X ⊤ i,c
b
A
x
−
x
s
ncS i=1
ncL i=1 l
c

2

c

nS nL
2
1 XX
bj,c
A⊤ xi,c
,
+ c c
s −x
l
nS nL i=1 j=1

(3)

bi,c
where xl indicates the labeled target-domain instance ofclass c, and ncL denotes the number of labeled target-domain
instances in that class. Similarly, xi,cs is the source-domain
data of class c, and ncS represents the number of sourcedomain data of class c.
In (3), the first term calculates the difference between
class-wise means for matching the approximated classconditional distributions (see [24] for detailed derivations),
while the second term enforces the embedding of transformed within-class data [37]. We note that, since only
labeled cross-domain data are utilized in (1), its solutionfor HDA can be considered as a supervised version of our
CDLS (denoted as CDLS sup in Section 4).
3.2.2

From supervised to semi-supervised HDA

To adopt the information of unlabeled target-domain datafor improved adaptation, we advocate the learning of thelandmarks from cross-domain data when deriving the aforementioned domain-invariant feature subspace.
5083

Extended from (1), our proposed algorithm crossdomain landmark selection (CDLS) exploits heterogenous
data across domains, with the ability to identify the adaptation ability of each instance with a properly assigned
weight. Instances in either domain with a nonzero weight
will be considered as a landmark.

where
c

Econd =

 2
 c
nL
nc
nc
U
S
X
X
1 X
1
i,c
i,c 
⊤ i,c

b
b
,
+
x
β
x
α
A
x
−
i
i
u
s
l
δncS i=1
ncL + δncU
i=1
i=1
nc

c

Eembed =

nc

S X
L
X

i=1 j=1

With the above goal, the objective function of our CDLS
can be formulated as follows:nc

min EM (A, DS , DL , XU , α, β)+2

EC (A, DS , DL , XU , α, β) + λ kAk ,(4)

βc ⊤ 1αc ⊤ 1== δ,
s.t. {αic , βic } ∈ [0, 1] ,ncS
ncU
where α = [α1 ; · · · ; αc ; · · · ; αC ] ∈ RnS and β =[β 1 ; · · · ; β c ; · · · ; β C ] ∈ RnU are the weights observed
for all labeled and unlabeled data in source and target domains, respectively. We have αc = [α1c ; · · · ; αnc cS ] andβ c = [β1c ; · · · ; βnc cU ], where ncS and ncU denote the total
numbers of the corresponding instances of or predicted asclass c in the associated domain.
In (4), δ ∈ [0, 1] controls the portion of cross-domain
data in each class to be utilized for adaptation. If δ = 0, our
CDLS would turn into its supervised version as described in
Section 3.2.1. While we fix δ = 0.5 in our work, additional
analysis on the selection and effect of δ will be provided
in our experiments. We also note that, since the numberof labeled target-domain instances XL is typically small insemi-supervised HDA, all of such data will be viewed as themost representative landmarks to be utilized during adaptation (i.e., no additional weights required for XL ).
The EM in (4) matches marginal cross-domain data disbl and x
bu indicating reducedtributions for HDA. With x
dimensional labeled and unlabeled target-domain data, we
calculate EM by:

1
δnS

P nS

i=1 αi A1⊤ i
xs − nL +δn
U

P nL

bil +
i=1 x

P nU

biu
i=1 βi x

 2 (5).

To match cross-domain conditional data distributions via
EC , we apply SVM trained from labeled cross-domain data
eui for xbiu (as described laterto predict the pseudo-labels yi nUyu }i=1 assigned for XU , the EC
in Section 3.3.3). With {eterm in (4) can be expressed as:

EC (A, DS , DL , XU , α, β) =

C
X
c=1

c
Econd
+

1 c
E
,
ec embed
(6)

j,c

bl
−x

2

+

nc

L X
U
X

i=1 j=1bj,cbi,c− βj x
x
u
l

2

+

i,c

⊤

j,c

b u − αj A x s
βi x

2

,

It can be seen that, EC term in (6) is extended from (3) byutilizing unlabeled target-domain data with pseudo-labels.
Similar to (3), we match cross-domain class-conditional
distributions and preserve local embedding of transformedccdata of each class using via Econdand Eembed, respectively. The normalization term in (6) is calculated as ec =δncS ncL + δncL ncU + δ 2 ncU ncS .
With both EM and EC defined, we address semisupervised HDA by solving (4). This allows us to learn
the proper weights α and β for the representative instancesfrom both domains (i.e., cross-domain landmarks). As a result, our derived feature transformation A will result in afeature subspace with improved adaptation capability. This
will be verified later by our experiments in Section 4.

3.3. Optimization
We now discuss how we solve the optimization problemof (4). Similar optimization process can be applied to solve
the supervised version of (1), which is presented in the Supplementary.
It can be seen that, the semi-supervised learning problem of (4) is a non-convex joint optimization problem withrespect to A, α, and β. By advancing the technique of iterative optimization, we alternate between the learning offeature transformation A and landmark weights α and β.
The latter also involves the prediction of pseudo-labels forunlabeled target-domain data. We now describe the optimization process below.
3.3.1

EM (A, DS , DL , XU , α, β) =

i,c

nc

U X
S
X

i=1 j=1

A,α,β

nc

⊤

αi A xs

Optimizing A

Given the weights of α and β fixed, we take the derivativeof (4) with respect to A and set it equal to zero. The closed
form solution of A can be derived as:

−1 


b ⊤ + HU X
b ⊤ , (7)A = λIdS + XS HS XS ⊤X S HL X
L
U

where IdS is a dS -dimensional identity matrix, matricesb L ∈ Rm×nL , X
b U ∈ Rm×nU } rep{XS ∈ RdS ×nS , X
resent source, transformed labeled and unlabeled targetdomain instances, respectively. Each entry (HS )i,j in
HS ∈ RnS ×nS denotes the derivative coefficient associated⊤with xis xjs . Similar remarks can be applied to the matrices
HL ∈ RnS ×nL and HU ∈ RnS ×nU . Detailed derivations
are available in the Supplementary.
5084

Algorithm 1 Cross-Domain Landmark Selection (CDLS)
Input: Labeled source and target-domain data DS =S
L, DL = {xil , yli }ni=1; unlabeled target{xis , ysi }ni=1
i nU
domain data {xu }i=1 ; feature dimension m; ratio δ;
parameter λnL
1: Derive an m-dim. subspace via PCA from {xil }i=1
andi nU{xu }i=1
Uyui }ni=1
2: Initialize A by (1) and pseudo-labels {e3: while not converge do
4:Update transformation A by (7)5:Update landmark weights {α,β} by (9)U
6:Update pseudo-labels {eyui }ni=1
by Section 3.3.3
7: end while
U
U
Output: Predicted labels {yui }ni=1
of {xiu }ni=1
3.3.2

Optimizing α and βWith transformation A observed, we reformulate (4) as:
1 ⊤1
α KS,S α + β ⊤ KU,U β − α⊤ KS,U βα,β 22s.t. {αic , βic } ∈ [0, 1] ,(8)

αc ⊤ 1βc ⊤ 1
=
= δ.
c
nS
ncU

In (8), each entry (KS,S )i,j in KS,S ∈ RnS ×nS is the coefficient associated with (A⊤ xis )⊤ A⊤ xjs , and each entry(kS,L )i in kS,L ∈ RnS denotes the sum of the coefficientsbjl over all x
bjl . Similar remarks can be appliedof (A⊤ xis )⊤ x
nS ×nU
to KS,U ∈ R, KU,U ∈ RnU ×nU , and kU,L ∈ RnU(see the Supplementary for detailed derivations).
With the above formulation, one can apply existing
Quadratic Programming (QP) solvers and optimize the following problem instead:minzi ∈[0,1],Z⊤ V=W

where

1 ⊤
Z BZ + b⊤ Z,
2

(9)

 




KS,S
−KS,U
α
−kS,L
,B =
,
,
b
=
−K⊤
KU,U
kU,L
β
S,U


VS
0nS ×CV=∈ R(nS +nU )×2C with
0nU ×CVU(
1 if xis ∈ class j(VS )ij =0 otherwise(
1 if xiu predicted as class j(VU )ij =,0 otherwise(
δncS
if c ≤ C
1×2C
W∈R
with (W)c =.
δnc−C
if c > C
U

Z=

DSLR

Figure 2. Example images of Caltech-256 and Office dataset.

3.3.3

Label prediction for XU

Given the feature transformation and the observed landmarks, we train linear SVMs [5] using all transformed labeled cross-domain data with the associated weights. As a
U
result, the labels of unlabeled target-domain data {bxiu }ni=1
can be predicted accordingly.
To start our optimization process, we apply the supervised version of CDLS (i.e., Section 3.2.1) to initializeU
the transformation A and the pseudo-labels {eyui }ni=1. The
pseudo code of our CDLS is summarized in Algorithm 1.

4. Experimentsmin− kS,L ⊤ α + kU,L ⊤ β

Webcam

4.1. Datasets and Settings
We first address the task of cross-domain object recognition, and consider the use of Office + Caltech-256 (C)datasets [30, 17]. The Office dataset contains images collected from threee sub-datasets : Amazon (A) (i.e., images
downloaded from the Internet), Webcam (W) (i.e., lowresolution images captured by web cameras), and DSLR (D)(i.e., high-resolution images captured by digital SLR cameras). This dataset has objects images of 31 categories, andCaltech-256 contains 256 object categories. Among theseobjects, 10 overlapping categories are selected for our experiments. Figure 2 shows example images of the categoryof laptop computer. Following [14], two different types offeatures are considered for cross-domain object recognition:
DeCAF6 [11] and SURF [3]. We note that, the latter is described in terms of Bag-of-Words (BOW) features via kmeans clustering. The feature dimensions of DeCAF6 andSURF are 4096 and 800, respectively.
We also solve the problem of cross-lingual text categorization, and apply the Multilingual Reuters Collection [35, 2] dataset for experiments. This dataset contains
about 11, 000 articles from 6 categories in 5 languages (i.e.,
English, French, German, Italian, and Spanish). To extract the features from these articles, we follow the settings in [12, 42, 23] in which all the articles are represented
by BoW with TF-IDF, followed by PCA for dimension reduction (with 60% energy preserved). The reduced dimensions with respect to different language categories English,
French, German, Italian, and Spanish are 1, 131, 1, 230,
1, 417, 1, 041, and 807, respectively.
For simplicity, we fix the ratio δ = 0.5 for all our ex5085

Table 1. Classification results (%) with standard deviations forcross-feature object recognition.
S,T

SVMt

A, A
W, W
C, C

87.3±0.587.1±1.176.8±1.1

A, A
W, W
C, C

43.4±0.957.9±1.0
29.1±1.5

DAMA

MMDT
SHFA
SURF to DeCAF6
87.4±0.5 89.3±0.4 88.6±0.3
87.2±0.7 87.3±0.8 90.0±1.0
73.8±1.2 80.3±1.2 78.2±1.0
DeCAF6 to SURF
38.1±1.1 40.5±1.3 42.9±1.0
47.4±2.1 59.1±1.2 62.2±0.7
18.9±1.3 30.6±1.7 29.4±1.5

CDLS sup

CDLS

S,T

SVMt

86.7±0.688.5±1.474.8±1.1

91.7±0.2
95.2±0.981.8±1.1

A, D
W, D
C, D

90.9±1.1

45.6±0.760.9±1.1
31.6±1.546.4±1.0
63.1±1.1
31.8±1.2

A, D
W, D
C, D

54.4±1.0

periments. We follow [21, 18, 19] and set λ = 21 for regularizing A. We also fix m = 100 as the reduced dimensionality for the derived feature subspace. Later, we will
provide analyses on convergence and parameter sensitivity
in Section 4.4 for verifying the robustness of our CDLS.

4.2. Evaluations
For evaluation, we first consider the performance ofthe baseline approach of SVMt , which indicates the useof SVMs learned from only labeled target-domain datab L . To compare our approach with state-of-the-art HDA
X
methods, we consider HeMap [32], DAMA [38], MMDT
[18, 19], and SHFA [23]. As noted earlier, since both
SSKMDA [40] and SCP [41] require additional unlabeleddata from the source domain for adaptation, we do not include their results for comparisons. Finally, we consider thesupervised version of our CDLS (denoted as CDLS sup,see Section 3.2.1), which simply derives feature transformation without learning cross-domain landmarks.

4.2.1

Table 2. Classification results (%) with standard deviations for object recognition across feature representations and datasets.

Object recognition across feature spaces

To perform cross-feature object recognition on Office +Caltech-256 (C), two different types of features are available: SURF vs. DeCAF6 . If one of them is applied for describing the source-domain data, the other will be utilized
for representing those in the target domain. For sourcedomain data with SURF features, we randomly choose 20
images per category as labeled data. As for the target domain with DeCAF6 , we randomly choose 3 images per object category as labeled data, and the rest images in eachcategory as the unlabeled ones to be recognized. The sameprocedure is applied to the uses of DeCAF6 and SURF fordescribing source and target-domain data, respectively.
Table 1 lists the average classification results with 10
random trials. Since the number of images in DSLR (D)is much smaller than those in other domains (i.e., Amazon(A), Webcam (W), and Caltech-256 (C)), we only report
the recognition performance on A, W, and C. Also, theperformance of HeMap was much poorer than all other approaches (e.g., 11.7% on C for DeCAF6 to SURF), so we
do not include its performance in the table.

DAMA

MMDT
SHFA
SURF to DeCAF6
91.5±1.2 92.1±1.0 93.4±1.1
91.2±0.9 91.5±0.8 92.4±0.9
91.0±1.3 93.1±1.2 93.8±1.0
DeCAF6 to SURF
53.2±1.5 53.5±1.3 56.1±1.0
51.6±2.2 54.0±1.3 57.6±1.1
51.9±2.1 56.7±1.0 57.3±1.1

CDLS supCDLS

92.0±1.2
91.0±1.1
91.9±1.3

96.1±0.795.1±0.8
94.9±1.554.3±1.3
57.8±1.054.8±1.058.4±0.860.5±1.059.4±1.2

From Table 1, it can be seen that the methods of DAMA
and CDLS sup did not always achieve comparable performance as the baseline approach of SVMt did. A possible
explanation is that these two HDA approaches only related
cross-domain heterogeneous data by deriving feature transformation for alignment purposes. For other HDA ones like
MMDT, and SHFA, they additionally adapted the prediction
models across-domain and thus produced improved results.
Nevertheless, our CDLS consistently achieved the best performance across all categories. This supports the use of our
HDA approach for cross-feature object recognition.
4.2.2

Object recognition across features and datasets

In this part of the experiments, we have source and targetdomain data collected from distinct datasets and also described by different types of features. We adopt the samesettings in Section 4.2.1 for data partition, while the only
difference is that we choose DSLR as the target domain withothers as source domains.
Table 2 lists the average classification results from 10
random trials. From this table, we see that existing HDA approaches generally produced slightly improved results thanthe use of SVMt . Again, our proposed CDLS is observed toperform favorably against all HDA methods. It is worth repeating that, our improvement over CDLS sup verifies theidea of exploiting information from unlabeled samples toselect representative cross-domain landmarks for HDA.
4.2.3

Cross-lingual text categorization

To perform cross-lingual text categorization, we view articles in different languages as data in different domains.
In our experiments, we have {English, French, German, or
Italian} as the source domain, and randomly selected 100
articles from each as labeled data. On the other hand, we
have Spanish as the target domain; for the target-domain
data, we vary the number of labeled articles (from {5, 10,
15, 20}) per category, and randomly choose 500 articles
from the remaining ones (per category) as the unlabeleddata to be categorized.
Figure 3 illustrates the results over different numbers oflabeled target-domain data. From this figure, it can be seen
5086

80.0

75.0

75.0

70.0

CDLS
CDLS_sup
sup
SHFA
MMDT
DAMA
SVMt

65.0
60.0
55.0
50.0
45.0
5

10

15

70.0

CDLS
CDLS_sup
sup
SHFA
MMDT
DAMA
SVMt

65.0
60.0
55.0
50.0
45.0

20

# labeled target-domain data per class
(a)

5

10

15

Accuracy (%)

80.0

75.0

Accuracy (%)

80.0

75.0

Accuracy (%)

Accuracy (%)

80.0

70.0

CDLS
CDLS_sup
sup
SHFA
MMDT
DAMA
SVMt

65.0
60.0
55.0
50.0
45.0

20

5

# labeled target-domain data per class
(b)

10

15

70.0

CDLS
CDLS_sup
sup
SHFA
MMDT
DAMA
SVMt

65.0
60.0
55.0
50.0
45.0

20

# labeled target-domain data per class
(c)

5

10

15

20

# labeled target-domain data per class(d)Figure 3. Classification results on cross-lingual text categorization using Multilingual Reuters Collection. Note that Spanish is viewed as
the target domain, while the source domains are selected from (a) English, (b) French, (c) German, and (d) Italian, respectively.
Table 3. Classification results with standard deviations for cross-lingual text categorization with Spanish as the target domain.
SourceArticlesEnglishFrenchGermanItalianSVMt
66.7±0.9# labeled target domain data / category = 10
DAMA
MMDT
SHFA
CDLS sup67.4±0.7 68.2±1.0 67.8±0.7
69.6±0.8
67.8±0.6 68.3±0.8 68.1±0.6
69.3±0.867.3±0.6 67.9±1.0 68.4±0.7
69.3±0.766.2±1.0 67.0±0.9 68.0±0.7
69.5±0.8CDLS70.8±0.771.2±0.8
71.0±0.871.7±0.8SVMt
73.1±0.6# labeled target domain data / category = 20
DAMA
MMDT
SHFA
CDLS sup73.6±0.7 74.2±0.8 74.1±0.4
76.2±0.673.7±0.7 74.6±0.7 74.5±0.6
76.4±0.473.7±0.7 74.9±0.8 74.7±0.5
76.6±0.673.6±0.7 74.6±0.7 74.5±0.5
76.1±0.5CDLS77.4±0.777.6±0.7
78.0±0.677.5±0.7

that the performances of all approaches were improved with
the increase of the number of labeled target-domain data,
while the difference between HDA approaches and SVMt
became smaller. We also list the categorization results inTable 3 (with 10 and 20 labeled target-domain instances per
category). From the results presented in both Figure 3 andTable 3, the effectiveness of our CDLS for cross-lingual text
categorization can be successfully verified.

4.3. Remarks on Cross-Domain Landmarks
Recall that the weights α and β observed for instancesfrom each domain indicates its adaptation capability. Thatis, a labeled source-domain instance with a large αic andan unlabeled target-domain instance with a large βic would
indicate that they are related to the labeled target-domain
data of the same class c.
In Figure 4, we visualize the embedding for the adapted
data and cross-domain landmarks using t-distributed
stochastic neighbor embedding (t-SNE) [36]. Moreover,in Figure 5, we show example images and landmarks observed with different weights from two categories laptop computer and bike on Caltech-256 → DSLR (withSURF to DeCAF6 ). It can be seen that, the source-domain
images with larger weights were not only more representative in terms of the category information, they were alsovisually more similar to the target-domain images of theclass of interest. The same remarks can also be applied tothe unlabeled target-domain data viewed as landmarks. It isalso worth noting that, unlabeled target-domain images withsmallers weights would be more likely to be mis-classified.
For example, the image bounded by the red rectangle in Figure 5 was recognized as a similar yet different object category of monitor.

Figure 4. Landmarks (with different weights) observed fromsource and target-domain data in the recovered subspace.

4.4. Convergence and Parameter Sensitivity
To further analyze the convergence property of our
CDLS and assess its parameter sensitivity, we conduct additional experiments on cross-domain object recognition using C to D (with SURF to DeCAF6 ). As discussed in Section 3.3, we alternate between the variables to be optimized
when solving our CDLS.
We report the recognition performance with respect tothe iteration number in Figure 6(a), in which we observethat the performance converged within 10 iterations. Thisobservation is consistent to those of our other HDA experiments in this paper. As for the parameters of interest, wefirst discuss the selection of the dimension m of the reduced
feature space. We vary the dimension number m and perform cross-domain recognition on the above cross-domain
pairs. The results are presented in Figure 6(b). From
this figure, we see that the performance increased when m
was larger, while such performance improvements became
marginal for m beyond 100. Thus, m = 100 would be apreferable choice for our experiments.
5087

Labeled Target-Domain Images

Labeled Target-Domain Images

DSLR

DSLR

Source-Domain Images with Larger Weights

Caltech‐256

Source-Domain Images with Larger Weights

Caltech‐256

Source-Domain Images with Smaller Weights

Unlabeled Target-Domain Images with Larger Weights

DSLR

Source-Domain Images with Smaller Weights

Unlabeled Target-Domain Images with Larger Weights

DSLR

Unlabeled Target-Domain Images with Smaller Weights

Unlabeled Target-Domain Images with Smaller Weights

(a)

(b)

Figure 5. Example images and landmarks (with the observed weights) for Caltech-256 → DSLR (with SURF to DeCAF6 ). The object
categories are (a) laptop computer and (b) bike. Note that the image bounded by the red rectangle was mis-classified as monitor.

93
91

CDLS

89

CDLS_sup
sup

87
1

2

3

4

97

95
93
91

CDLS

89

CDLS_sup
sup

87
5

6

7

Iteration Number
(a)

8

9

10

5

10

95
93
91

CDLS

89

CDLS_sup
sup

87

50

100

200

400
500 800

Accuracy (%)

95

Accuracy (%)

97

Accuracy (%)

Accuracy (%)

970

0.1 0.2 0.3 0.4 0.5 0.6 0.7

δ
(c)

m
(b)

0.8 0.9

1

100
95
90
85
80
75
70
65
60
0.0005

CDLS
CDLS_sup
sup
0.005

0.05

0.5

5

50

λ
(d)

Figure 6. Analysis on convergence and parameter sensitivity using Caltech-256 with SURF to DSLR with DeCAF6 . The parameters ofinterests are (a) iteration number, (b) reduced dimensionality m, (c) ratio δ, and (d) regularization parameter λ.

Recall that the ratio δ in (4) controls the portion of crossdomain landmarks to be exploited for adaptation. Figure6(c) presents the performance of CDLS with different δ values. It is worth repeating that, CDLS would be simplified
as the supervised version CDLS sup if δ = 0, as illustrated
by the flat dotted line in Figure 6(c). As expected, using
all (δ = 1) or none (δ = 0) of the cross-domain data as
landmarks would not be able to achieve satisfactory HDA
performance. Therefore, the choice of δ = 0.5 would bereasonable in our experiments.
Finally, Figure 6(d) presents the results of CDLS and
CDLS sup with varying λ values for regularizing A. From
this figure, it can be seen that fixing λ = 12 as suggested
by [21, 18, 19] would also be a reasonable choice.

5. Conclusion
We proposed Cross-Domain Landmark Selection(CDLS) for performing heterogeneous domain adaptation(HDA). In addition to the ability to associate heterogeneous data across domains in a semi-supervised setting,our CDLS is able to learn representative cross-domain
landmarks for deriving a proper feature subspace foradaptation and classification purposes. Since the derived
feature subspace matches cross-domain data distribution
while eliminating the domain differences, we can simply
project labeled cross-domain data to this domain-invariant
subspace for recognizing the unlabeled target-domain
instances. Finally, we conducted experiments on threeclassification tasks across different features, datasets, andmodalities. Our CDLS was shown to perform favorably
against state-of-the-art HDA approaches.

5088

Acknowledgements
This work was supported in part by the Ministry of Science and Technology of Taiwan under Grants MOST1032221-E-001-021-MY2, MOST104-2221-E-017-016, andMOST104-2119-M-002-039.

References[1] R. Aljundi, R. Emonet, D. Muselet, and M. Sebban.
Landmarks-based kernelized subspace alignment for unsupervised domain adaptation. In Computer Vision and Pattern
Recognition (CVPR), 2015. 2[2] M. Amini, N. Usunier, and C. Goutte. Learning from multiple partially observed views-an application to multilingual
text categorization. In Neural Information Processing Systems (NIPS), 2009. 5[3] H. Bay, T. Tuytelaars, and L. Van Gool. Surf: Speeded up
robust features. In European Conference on Computer Vision(ECCV). 2006. 5[4] C. M. Bishop. Pattern recognition and machine learning.
Springer, 2006. 3[5] C.-C. Chang and C.-J. Lin. Libsvm: A library for support
vector machines. ACM Transactions on Intelligent Systems
and Technology (TIST), 2011. 5[6] R. Chattopadhyay, W. Fan, I. Davidson, S. Panchanathan,and J. Ye. Joint transfer and batch-mode active learning.
In International Conference on Machine Learning (ICML),
2013. 1
[7] M. Chen, K. Q. Weinberger, and J. Blitzer. Co-training fordomain adaptation. In Neural Information Processing Systems (NIPS), 2011. 1[8] B. Chidlovskii, G. Csurka, and S. Gangwar. Assembling heterogeneous domain adaptation methods for image classification. In Cross Language Evaluation Forum (CLEF), 2014.
1[9] W.-S. Chu, F. De la Torre, and J. F. Cohn. Selective transfer machine for personalized facial action unit detection. In
Computer Vision and Pattern Recognition (CVPR), 2013. 2[10] W. Dai, Y. Chen, G.-R. Xue, Q. Yang, and Y. Yu. Translated
learning: Transfer learning across different feature spaces. In
Neural Information Processing Systems (NIPS), 2008. 1[11] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International
Conference on Machine Learning (ICML), 2014. 5[12] L. Duan, D. Xu, and I. Tsang. Learning with augmented features for heterogeneous domain adaptation. In International
Conference on Machine Learning (ICML), 2012. 1, 2, 5
[13] L. Duan, D. Xu, and I. W. Tsang. Domain adaptationfrom multiple sources: A domain-dependent regularization
approach. In IEEE Transactions on Neural Networks andLearning Systems (T-NNLS), 2012. 2[14] N. Farajidavar, T. de Campos, and J. Kittler. Transductive
transfer machine. In Asian Conference of Computer Vision(ACCV), 2014. 5[15] B. Gong, K. Grauman, and F. Sha. Connecting the dots withlandmarks: Discriminatively learning domain-invariant features for unsupervised domain adaptation. In International
Conference on Machine Learning (ICML), 2013. 2[16] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, andA. J. Smola. A kernel method for the two-sample-problem.
In Neural Information Processing Systems (NIPS), 2006. 3[17] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. 2007. 5[18] J. Hoffman, E. Rodner, J. Donahue, T. Darrell, and
K. Saenko. Efficient learning of domain-invariant image representations. In International Conference on Learning Representation (ICLR), 2013. 1, 2, 3, 6, 8
[19] J. Hoffman, E. Rodner, J. Donahue, B. Kulis, and K. Saenko.
Asymmetric and category invariant feature transformations
for domain adaptation. In International Journal of Computer
Vision (IJCV), 2014. 1, 2, 3, 6, 8
[20] J. Huang, A. Gretton, K. M. Borgwardt, B. Schölkopf, andA. J. Smola. Correcting sample selection bias by unlabeled
data. In Neural Information Processing Systems (NIPS),
2006. 2[21] B. Kulis, K. Saenko, and T. Darrell. What you saw is notwhat you get: Domain adaptation using asymmetric kernel
transforms. In Computer Vision and Pattern Recognition(CVPR), 2011. 1, 2, 3, 6, 8
[22] A. Kumar, A. Saha, and H. Daume. Co-regularization based
semi-supervised domain adaptation. In Neural Information
Processing Systems (NIPS), 2010. 2[23] W. Li, L. Duan, D. Xu, and I. W. Tsang. Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation. In IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2014. 1,
2, 5, 6[24] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu. Transferfeature learning with joint distribution adaptation. In International Conference on Computer Vision (ICCV), 2013. 1,
3[25] S. J. Pan, J. T. Kwok, Q. Yang, and J. J. Pan. Adaptive localization in a dynamic wifi environment through multi-view
learning. In Association for the Advancement of Artificial Intelligence (AAAI), 2007. 1[26] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. In IEEE
Transactions on Neural Networks and Learning Systems (TNNLS), 2011. 1, 2, 3[27] S. J. Pan and Q. Yang. A survey on transfer learning. In IEEE
Transactions on Knowledge and Data Engineering (TKDE),
2010. 1[28] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa. Visual
domain adaptation: A survey of recent advances. In IEEE
Signal Processing Magazine, 2015. 1[29] P. Prettenhofer and B. Stein. Cross-language text classification using structural correspondence learning. In Association
for Computational Linguistics (ACL), 2010. 1[30] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), 2010. 1, 55089

[31] S. Shekhar, V. M. Patel, H. Nguyen, and R. Chellappa. Generalized domain-adaptive dictionaries. In Computer Vision
and Pattern Recognition (CVPR), 2013. 1[32] X. Shi, Q. Liu, W. Fan, P. S. Yu, and R. Zhu. Transfer learning on heterogenous feature spaces via spectral transformation. In International Conference on Data Mining (ICDM),
2010. 1, 2, 6
[33] V. Sindhwani, P. Niyogi, and M. Belkin. A co-regularization
approach to semi-supervised learning with multiple views.
In International Conference on Machine Learning (ICML)
workshop, 2005. 1[34] E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultaneous deep transfer across domains and tasks. In International
Conference in Computer Vision (ICCV), 2015. 1[35] N. Ueffing, M. Simard, S. Larkin, and J. H. Johnson. NRC’s
PORTAGE system for WMT 2007. In Association for Computational Linguistics (ACL) workshop, 2007. 5[36] L. Van der Maaten and G. Hinton. Visualizing data using
t-SNE. In Journal of Machine Learning Research (JMLR),
2008. 7[37] C. Wang and S. Mahadevan. Manifold alignment without
correspondence. In International Joint Conference on Artificial Intelligence (IJCAI), 2009. 3[38] C. Wang and S. Mahadevan. Heterogeneous domain adaptation using manifold alignment. In International Joint Conference on Artificial Intelligence (IJCAI), 2011. 1, 2, 6
[39] X. Wu, H. Wang, C. Liu, and Y. Jia. Cross-view action recognition over heterogeneous feature spaces. In International
Conference in Computer Vision (ICCV), 2013. 1, 2[40] M. Xiao and Y. Guo. Feature space independent semisupervised domain adaptation via kernel matching. In IEEE
Transactions on Pattern Analysis and Machine Intelligence(T-PAMI), 2015. 1, 2, 3, 6[41] M. Xiao and Y. Guo.
Semi-supervised subspace coprojection for multi-class heterogeneous domain adaptation.
In European Conference on Machine Learning (ECML),
2015. 1, 2, 3, 6[42] J. T. Zhou, I. W. Tsang, S. J. Pan, and M. Tan. Heterogeneous domain adaptation for multiple classes. In International Conference on Artificial Intelligence and Statistics(AISTATS), 2014. 1, 2, 3, 5[43] Y. Zhu, Y. Chen, Z. Lu, S. J. Pan, G.-R. Xue, Y. Yu, andQ. Yang. Heterogeneous transfer learning for image classification. In Association for the Advancement of Artificial
Intelligence (AAAI), 2011. 1

5090