Learning Cross-Domain Landmarks for Heterogeneous Domain Adaptation

IEEE 2016 Conference on
Computer Vision and PatternRecognition

Yao-Hung Hubert Tsai , Yi-Ren Yeh , and Yu-Chiang Frank Wang
12
Research Center for IT Innovation, Academia Sinica, Taipei, Taiwan ; Dept. Mathematics, National Kaohsiung Normal University, Kaohsiung, Taiwan

Source Domain• Illustration: Source-Domain Data
: Labeled Target-Domain Data
: Unlabeled Target-Domain Data

Source-Domain Dataii nSii nL
Input: Labeled source and target-domain data DS = {xs , ys }i=1 , DL = {xl , yl }i=1 ;i nU
unlabeled target-domain data {xu }i=1 ; feature dimension m; ratio ; parameteri nLi nU
1: Derive an m-dimensional subspace via PCA from {xl }i=1 and {xu }i=1nU
ieu }i=1
2: Initialize A by Supervised CDLS and pseudo-labels {y: Unlabeled Target Domain Datawith Predicted Labels

3: while not converge do4:
Update transformation A
5:
Update landmark weights {↵,nUieu }i=1
6:
Update pseudo-labels {y7: end while

Target-Domain DataTarget Domain

Output:• Supervised CDLS
– Only XS , XL available and no landmarks/weights to be learned– Objective function:
min EM (A, DS , DL ) + EC (A, DS , DL ) + kAkRelated Works• Instance Re-weighting or Landmark Selection– TJM [1] and LM [2] (only homogeneous DA considered)• Supervised HDA– Only labeled source and target-domain data available for adaptation– HeMap [3], DAMA [4], ARC-t [5], HFA [6], MMDT [7], and SHFR [8]• Semi-supervised HDA– Unlabeled target-domain data can be jointly exploited during adaptation.
– HTDCC [9], SHFA [10], SSKMDA [11], SCP [12], and Ours

Notations and Motivations• Notations
1 1nsns
Labeled Source-Domain Data DS = {(xs , ys ), . . . , (xs , ys )} = {XS , yS }

nlnl
1 1
Labeled Target-Domain Data DL = {(xl , yl ), . . . , (xl , yl )} = {XL , yL }11nunu
Unlabeled Target-Domain Data DU = {(xu , yu ), . . . , (xu , yu )} = {XU , yU }⇤ Cross-domain marginal data distributions:nSEM (A, DS , DL ) =nS i=1A>

EC (A, DS , DL ) =ncSCX

1 X

ncS i=1c=1

A>i,cxs

ncL i=1nL i=12nc
L

1 X

1 Xixs⇤ Cross-domain conditional data distributions:

ixl

cncn
S X
LX

1> i,c+ c c
A xs
nS nL i=1 j=1i,cxl

j,cxl

2

• Semi-supervised CDLS
– Jointly exploit XS , XL , and XU for learning cross-domain landmarks {↵, }i nUe– Need to assign pseudo-labels {yu }i=1 for XU .
– Objective function:
min EM (A, DS , DL , XU , ↵, ) + EC (A, DS , DL , XU , ↵, ) + kAk⇤ 2 [0, 1] controls the portion of cross-domain data in each class for adaptation.
⇤ Cross-domain marginal data distributions:
EM (A, DS , DL , XU , ↵, ) =1n L + nUnL
X

ixl +i=1

nUXii xu

i=1! 2EC (A, DS , DL , XU , ↵, ) =cwith e =c
Econd =c
Eembed =ccccn S nL + n L nU +nc
S1 X> i,c↵AxsicnS i=1cncnS X
LX

i=1 j=1↵i A>

i,cxs2cc
Econd +

c=11 cEembedcencU

X i,c X
1@xl +ccnL + n U
i=1i=1j,cxl

2+cncnL X
UX

i=1 j=1i,cxl

i,cj,cj xu

2+

cncn
U X
SX

DAMAA, A
W, W
C, C

87.3±0.587.1±1.176.8±1.187.4±0.587.2±0.773.8±1.2

A, D
W, D
C, D

1 2i xu A

SVMt

S,Tcnc
L

S,T

43.4±0.957.9±1.0
29.1±1.538.1±1.147.4±2.1
18.9±1.3• Across Features & DatasetsnU n S080.080.0

75.075.0

75.075.070.0

CDLS
CDLS_supsupSHFA
MMDTDAMA
SVMt

65.060.055.0

45.05101570.0

CDLS
CDLS_supsup
SHFA
MMDT
DAMA
SVMt

65.060.0
55.0
50.0
45.0

20

# labeled target-domain data per class(a)

5101570.0

CDLS
CDLS_supsup
SHFA
MMDT
DAMA
SVMt

65.0
60.0
55.0
50.0
45.020# labeled target-domain data per class(b)

51015i,ci xu↵j Ai=1 j=1cwhere Eembedenforces the projected (cross-domain) data similarity

>

j,cxs

2A, D
W, D
C, DSVMt

DAMA

90.9±1.1

91.5±1.2
91.2±0.9
91.0±1.354.4±1.053.2±1.551.6±2.2
51.9±2.1

70.0

CDLS
CDLS_supsup
SHFA
MMDT
DAMASVMt

65.0
60.0
55.0
50.0
45.020# labeled target-domain data per class(c)

5101520# labeled target-domain data per class(d)

• Visualizing through t-SNE & Example Images/Landmarks– for Caltech-256 ! DSLR (with SURF to DeCAF6 )– Target-domain image bounded by the red rectangle was mis-classified as monitor.
Labeled Target-Domain Images

DSLR

DS

Source-Domain Images with Larger Weights

Caltech,256

Source-Domain Images with Smaller Weights

MMDTSHFASURF to DeCAF6
89.3±0.4 88.6±0.3
87.3±0.8 90.0±1.0
80.3±1.2 78.2±1.0
DeCAF6 to SURF
40.5±1.3 42.9±1.0
59.1±1.2 62.2±0.7
30.6±1.7 29.4±1.5

CDLS_supCDLS

86.7±0.688.5±1.474.8±1.1

91.7±0.2
95.2±0.981.8±1.1

45.6±0.760.9±1.1
31.6±1.546.4±1.063.1±1.1
31.8±1.2

MMDTSHFASURF to DeCAF6
92.1±1.0 93.4±1.1
91.5±0.8 92.4±0.9
93.1±1.2 93.8±1.0
DeCAF6 to SURF
53.5±1.3 56.1±1.0
54.0±1.3 57.6±1.1
56.7±1.0 57.3±1.1

CDLS_supCDLS

92.0±1.2
91.0±1.1
91.9±1.3

96.1±0.795.1±0.8
94.9±1.554.3±1.3
57.8±1.054.8±1.058.4±0.860.5±1.059.4±1.2

CaltecUnlabeled Target-Domain Images with Larger Weights

DSLR

Unlabeled Target-Domain Images with Smaller Weights(a)
laptop_computerConclusion

Evaluation I - Object Recognition

A, AW, WC, C⇤ Cross-domain conditional data distributions:
CX

80.0

• Object Recognition: Oﬃce and Caltech-256 Datasets [14, 15]– Images from Amazon (A), Webcam (W), DSLR (D), Caltech-256 (C)– Feature: DeCAF6 (4096 dimensions) versus SURF (800 dimensions)– 10 overlapping object categories– Randomly select 3 images per category for labeled target-domain instances XL .
• Text Categorization: Multilingual Reuters Dataset [16]– Source domain: English, French, German, and Italian– Target domain: Spanish– Feature: BoW + TF-IDF with 60% energy preserved via PCA– 6 categories in 5 languages– Randomly select 5/10/15/20 articles per category for XL .

• Across Feature Spacesc>
c>
1
↵1cc
=
=s.t. {↵i , i } 2 [0, 1] andccnSnUnSX
1> i↵ i A xsnS i=180.0

50.0i nUi nU
Predicted labels {yu }i=1 of {xu }i=12A,↵,

DT = DL [ DU , nt = nl + nu , and ds 6= dt• Motivations– Learn a domain-invariant mapping A 2 Rm⇥ds .
– Determine landmark weights ↵ 2 RnS for XS and 2 RnU for XU .
– View XL as the most representative landmarks (i.e., weights = 1).
• Matching Cross-Domain Data Distributions as in JDA [13]>– Marginal distributions: P (XT ) and P (A XS )>– Conditional distributions: P (XT |yT ) and P (A XS |yS )1 X• Text Categorization– Source domain: (a) English, (b) French, (c) German, and (d) Italian

22nL}

Experiment SetupA• Our Method: Cross-Domain Landmark Selection (CDLS)– We propose to exploit heterogeneous source and target-domain data for learning
cross-domain landmarks.
– By learning the adaptability of cross-domain data (including the unlabeled
target-domain data), we derive a domain-invariant feature space for HDA.

Evaluation II - Text Categorization

Pseudo-Code

Accuracy (%)• Heterogeneous Domain Adaptation (HDA)– Cross-domain data with distinct feature representations (e.g., BoW v.s. CNN)

Cross-Domain Landmarks Selection (CDLS)

Accuracy (%)• Domain Adaptation– Aim to transfer knowledge across domains (from source to target)– Typically, labeled data available in the source domain– Only few or no labeled data in the target domain

1

Accuracy (%)Introduction

2

Accuracy (%)1

• We presented Cross-Domain Landmarks Selection (CDLS) for HDA.
• Our CDLS is able to learn representative cross-domain landmarks for derivinga proper feature subspace for adaptation and classification purposes.
• Our CDLS performs favorably against state-of-the-art HDA approaches.

References[1] M. Long, et al. Transfer joint matching for unsupervised domain adaptation. In IEEE CVPR, 2014.
[2] B. Gong, et al. Connecting the dots with landmarks: Discriminatively learning domain-invariant features
for unsupervised domain adaptation. In ICML, 2013.
[3] X. Shi et al. Transfer learning on heterogeneous feature spaces via spectral transformation. In IEEE
ICDM, 2010.
[4] C. Wang and S. Mahadevan. Heterogeneous domain adaptation using manifold alignment. In IJCAI, 2011.
[5] B. Kulis et al. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms.
In IEEE CVPR, 2011.
[6] L. Duan et al. Learning with augmented features for heterogeneous domain adaptation. In ICML, 2012.
[7] J. Hoﬀman et al. Eﬃcient learning of domain-invariant image representations. In ICLR, 2013.
[8] J. T. Zhou et al. Heterogeneous domain adaptation for multiple classes. In AISTATS, 2014.
[9] X. Wu et al. Cross-view action recognition over heterogeneous feature spaces. In IEEE ICCV, 2013.
[10] W. Li et al. Learning with augmented features for supervised and semi-supervised heterogeneous domain
adaptation. In IEEE T-PAMI, 2014.
[11] M. Xiao et al. Feature space independent semi-supervised domain adaptation via kernel matching. InIEEE T-PAMI, 2015.
[12] M. Xiao et al. Semi-supervised subspace co-projection for multi-class heterogeneous domain adaptation.
In ECML, 2015.
[13] M. Long, et al. Transfer feature learning with joint distribution adaptation. In IEEE ICCV, 2013.
[14] K. Saenko, et al. Adapting visual category models to new domains. In ECCV, 2010.
[15] G. Griﬃn, el al. Caltech-256 object category dataset. 2007.
[16] N. Ueﬃng, el al. NRC’s PORTAGE system for WMT 2007. In ACL workshop, 2007.

DS