IEEE TRANSACTIONS ON MULTIMEDIA

1

Instance-level Heterogeneous Domain Adaptation
for Limited-labeled Sketch-to-Photo Retrieval
Fan Yang, Member, IEEE, Yang Wu, Member, IEEE, Zheng Wang, Member, IEEE, Xiang Li,
Sakriani Sakti, Member, IEEE, and Satoshi Nakamura, Fellow, IEEE

Abstract—Although sketch-to-photo retrieval has a wide range
of applications, it is costly to obtain paired and rich-labeled
ground truth. Differently, photo retrieval data is easier to acquire.
Therefore, previous works pre-train their models on rich-labeled
photo retrieval data (i.e., source domain) and then fine-tune themon the limited-labeled sketch-to-photo retrieval data (i.e., target
domain). However, without co-training source and target data,
source domain knowledge might be forgotten during the finetuning process, while simply co-training them may cause negativetransfer due to domain gaps. Moreover, identity label spaces ofsource data and target data are generally disjoint and therefore
conventional category-level Domain Adaptation (DA) is not directly applicable. To address these issues, we propose an Instancelevel Heterogeneous Domain Adaptation (IHDA) framework. We
apply the fine-tuning strategy for identity label learning, aiming
to transfer the instance-level knowledge in an inductive transfer
manner. Meanwhile, labeled attributes from the source data are
selected to form a shared label space for source and target
domains. Guided by shared attributes, DA is utilized to bridgecross-dataset domain gaps and heterogeneous domain gaps, whichtransfers instance-level knowledge in a transductive transfer
manner. Experiments show that our method has set a new stateof the art on three sketch-to-photo image retrieval benchmarks
without extra annotations, which opens the door to train more
effective models on limited-labeled heterogeneous image retrieval
tasks.
Index Terms—Domain Adaptation, Cross-modal Image Retrieval, Sketch, Person Re-identification

I. I NTRODUCTION
Domain Adaptation (DA) [1] is used to transfer theknowledge of the rich-labeled source domain (Ds ) to a
unlabeled/limited-labeled target domain (Dt ) and improve theeffectiveness of a given task in the target domain. Conventional
DA studies mainly focus on the category-level cases, suchas image classification and segmentation tasks [2]–[6]. In
category-level DA, the target label space (Y t ) and the sourcelabel space (Y s ) are either fully shared or partially shared,
which can be represented as Y s ∩ Y t 6= ∅ (see Fig. 1 and thenotations used in this section are summarized in TABLE I).
Manuscript received on February 4, 2020, revised on May XX, 2020, andaccepted on July XX, 2020. (Corresponding author: Yang Wu, Zheng Wang)
Fan Yang, Sakriani Sakti, and Satoshi Nakamura are with Nara Instituteof Science and Technology & RIKEN, Center for Advanced Intelligence
Project, Nara, Japan (e-mail: yang.fan.xv6@is.naist.jp; ssakti@is.naist.jp; snakamura@is.naist.jp). Zheng Wang is with the Digital Content and Media
Sciences Research Division, National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, Japan (e-mail: wangz@nii.ac.jp). Xiang Li iswith the School of Computer Science and Engineering, Nanyang Technological University, Singapore (e-mail: me@shawnli.me). Yang Wu is with
the Computer Vision Lab., Kyoto University, Kyoto, Japan, and also with
Nara Institute of Science and Technology, Nara 630-0192, Japan (e-mail:wu.yang.8c@kyoto-u.ac.jp).

Conventional Category-level
DA for Image Classification

Instance-level Heterogeneous DA for
Sketch-photo Retrieval

Richlabeled

Richlabeled

Rich-labeled
Chair

Bike

Chair

Bike

Unlabeled

Mug

id-1

id-2

id-4

id-5

id-3

Limited-labeled

Male,
Hat,
Bag
...

Male,
Hat,
Bag
...

Unlabeled

Target Training Set

Fig. 1. Comparison of Category-level DA (left) and IHDA (right). The
solid two-way arrow connector and the dotted two-way arrow connector
represent labeled and unlabeled, resp. The orange two-way arrow connector
and the purple two-way arrow connector represent inductive transfer flow andtransductive transfer flow, resp.

In real-world applications, the needs of DA are not limited
to category-level tasks. For instance, the instance-level sketchto-photo retrieval, which has been investigated in many studies [7]–[14], is suffering lack of training data as it is costly
to draw sketches. Meanwhile, plenty of rich-labeled photo
retrieval datasets are available. In many existing photo retrieval
datasets [15]–[19], even instance attributes are annotated [20],[21]. Accordingly, majority of sketch-to-photo retrieval methods [7], [10], [22]–[26] choose to pre-train their models on arich-labeled photo retrieval data (i.e., source domain) and thenfine-tune them on the limited-labeled sketch-to-photo retrieval
data (i.e., target domain). However, without co-training source
and target data, source domain knowledge might be forgotten
during the fine-tuning process, while simply co-training them
may cause negative transfer due to domain gaps.
DA methods are commonly used to reduce domain gaps on
co-training source and target datasets. sketch-to-photo retrieval
data consists of two heterogeneous modalities as sketch andphoto. Accordingly, we divide the target data into two subdomains as Dt1 (i.e., photos) and Dt2 (i.e., sketches), where
Dt = Dt1 ∪ Dt2 . The feature spaces of Ds , Dt1 , and Dt2 are
X s , X t1 and X t2 , respectively. If we regard each identity asIEEE TRANSACTIONS ON MULTIMEDIA

one class, the identity label space of X s is Y s while X t1 andX t2 share the same identity label space Y t1,t2 . Nonetheless,the source data is annotated by other contributors and identity
label spaces of the source and target domains are generally
disjoint (i.e., Y s ∩ Y t1,t2 = ∅). It is challenging for categorylevel DA methods to figure out “what to transfer and where totransfer” in this case. It shows that coarsely reducing domain
gaps may incur a target error [27]. It is insufficient to simply
match the marginal distributions, conditional distributions ofboth domains should also be matched.
To address this issue, we raise an Instance-level Heterogeneous Domain Adaptation (IHDA) framework, which breaks
through limitations of simple fine-tuning and category-level
DA. We select attributes that can jointly characterize instancelevel properties for all domains to form a shared label spaceas Z s = Z t1,t2 , where Z s and Z t1,t2 are proposed attributespaces for source domain and target domain, respectively. For
instance-level retrieval problems, since all of the instances
belong to the same category, the existence of shared attributes
is guaranteed. We assume that the attributes of the source
domain are observed. For some photo-sketch retrieval applications, a lot of related photo retrieval datasets already offer
attributes [16], [28]. When attributes of the source dataset
are not available, it is still possible to collect it from large
web resources at an acceptable cost. As many objects areaccompanied by attributes on website resources, both photos
and attributes can be downloaded to construct the source data.
For instance, the UT-Zap50K dataset [15], a source data in
our experiment, was collected from a shopping website.
After Z s and Z t1,t2 are formed, our IHDA framework seeks
to maximally transfer domain knowledge from source data totarget data. Due to Y s ∩Y t1,t2 = ∅, Y s and Y t1,t2 are distinct
but related. Learning to identify Y s is inductive to identify
Y t1,t2 . Thus, pre-training identify labels on Y s and then finetuning identity labels on Y t1,t2 can transfer the instance-levelknowledge from an inductive transfer manner. Since Z s =Z t1,t2 , referring to them, cross-dataset domain gaps (i.e., Ds
to Dt1 and Dt2 ) and heterogeneous domain gaps (i.e., Ds
and Dt1 to Dt2 ) can be reduced via an unsupervised Domain
Adaptation [2]–[5], which transfers instance-level knowledge
from a transductive transfer manner.
Although the ground truth of Z t1,t2 is unknown, it isleveraged to associate source and target domains by using an
entropy minimization criterion [29]. The entropy minimization
criterion does penalize low-confident predictions of Z t1,t2 . In
order to minimize it, target domain features are forced to match
source domain features at an instance level other than roughly
reducing the cross-domain gaps. Such an approach remarkably
improves the IHDA performance.
In summary, our work has three contributions:• We highlight the challenges for improving limited-labeled
sketch-to-photo retrieval performance from the transfer
learning perspective.
• We propose a novel IHDA framework to maximallyutilize the source domain knowledge to benefit the target
task at instance-level image retrieval. It overcomes thelimitations of applying simple fine-tuning and conventional DA.

2•

Our IHDA framework contributes new state-of-the-art
results on instance-level sketch-to-photo retrieval task. It
opens the door to train more effective cross-modal image
retrieval models by using related rich-labeled singlemodal image data.
TABLE IS UMMARY OF N OTATIONS .

SymbolDescription

Ds
Dt
Dt1
Dt2
Xt
X t1
X t2
Ys
Y t1,t2
Zs
Z t1,t2

Rich-labeled source domain
Limited-labelled target domain, Dt = Dt1 ∪ Dt2
Target photo domain, Dt1 ∈ Dt
Target sketch domain, Dt2 ∈ Dt
Target data space, X t = X t1 ∪ X t2
Target photo data space, X t1 ∈ X t
Target sketch data space, X t2 ∈ X t
Source identity label space
Target photo-sketch identity label space, which is shared for Dt1 and Dt2
Source attribute label space, which is shared for Ds , Dt1 , and Dt2
Target photo-sketch attribute label space, which is shared for Ds , Dt1 , and Dt2

The rest of this paper is organized as follows. In Section II, abrief review of related works is given. In Section III, we illustrate the details of the proposed IHDA framework. Section IV
reports experimental results to prove the effectiveness of our
IHDA framework. In Section V, we discuss the application
scope of our method. Finally, Section VI concludes this paper.
II. R ELATED W ORKS
We summarize related works and clarify their differences
and relationships in TABLE II.
TABLE II
A COMPARISON OF RELATED WORKS .
Methods#Modalities[23] [24] [25] [26]2DA &TargetIdentities#Domains Attributes Y t ∩ Y s = ∅w/o DA--[10] [22] [8] [30]2

w/o DA

Annotated-[2] [3] [4] [5]1, 2

w/ DA, 2-

No[31] [32]1

w/ DA, 2

Unknown

No[33] [34] [35] [36]1

w/ DA, 2

Unknown

Yes

Ours

2

w/ DA, 3

Unknown

Yes

Owing to the shortage of training data, studies [10], [22]–[26] take rich-labeled photo retrieval data for pre-training andthen fine-tune their model on the sketch-to-photo retrieval data.
However, there could be a dilemma in their approaches: performing a long-term fine-tuning may forget the source domain
knowledge, which is known as catastrophic forgetting [37],[38], while a short-term fine-tuning may under-fit on the target
data.
Alternatively, both the source and target data could be cotrained together. Due to the across-domain discrepancy, simply
co-train both source and target data may lead to negative transfer [1] and impair the performance. To alleviate this problem,
Domain Adaptation (DA) is desired. Unlike coarse categorylevel DA [2]–[5], or fine-grained category-level DA [31], [32],applying DA in instance-level heterogeneous retrieval faces thechallenge that Y s ∩ Y t1,t2 = ∅. To alleviate this challenge,some instance-level domain adaptation methods [35], [36]IEEE TRANSACTIONS ON MULTIMEDIA

3

Step 2: Co-training with Source Data and Target Training Data

Step 1: Training with Source Data

Homogeneous

Pretrained 𝜽𝑬𝟏 is resumed for 𝜽𝑬𝟏 and 𝜽𝑬𝟐 in Step 2
Pretrained 𝜽𝑪𝒂𝒕𝒕 is resumed in Step 2
SourceHeterogeneous

𝝏𝓛

𝟏

𝟐

GRL

𝑣5

𝓛𝒅

Sharedweights 𝑣 23

𝓛𝒔𝒂𝒕𝒕

𝓛𝒔,𝒕𝟏,𝒕𝟐
𝒂𝒕𝒕

E1

E1

Paired

𝓛𝒔𝒕𝒓𝒊

𝑣 24

𝓛𝒔𝒊𝒅

Target 2

𝝏𝓛 𝒅
𝝏𝑪 𝒅

𝑣5

E1

UnpairedTarget 1

𝝏𝓛− 𝝏𝑬𝒅 − 𝝏𝑬𝒅

SourcePhoto
Dataset𝓛𝒕𝟏,𝒕𝟐𝒊𝒅

E2TargetSketch-photo
Training Set

𝓛𝒕𝟏,𝒕𝟐𝒕𝒓𝒊

Fig. 2. The framework of Instance-level Heterogeneous Domain Adaptation (IHDA). E1 and E2 are encoders for photos (i.e., X s and X t1 ) and sketchess and C t1,t2 stand for the identity classifier of source and target domains, resp;(i.e., X t2 ), resp, and v s , v t1 , and v t2 denotes their embedding features; Cid
id
Catt and Cd denote the shared-attribute classifier and domain classifier, resp; GRL is the Gradient Reversal Layer. Best viewed in color.

transferred the source domain knowledge to the target domain
by using pseudo labels created by clustering algorithms on
the target domain. However, clustering algorithms can easily
generate noise in pseudo labels and impair the model performance. In our sketch-to-photo task, the noise label issue ismore serious due to the heterogeneous domain gap. Therefore,
instead of using pseudo labels, we specifically propose an
IHDA framework, which selects annotated attributes in source
data to form a shared label space for all domains.
Using attributes to learn the instance feature is particularly
prominent in instance-level retrieval tasks. Before us, studies [8], [10], [22], [30] had annotated attributes on both source
and target data to improve retrieval performance. Comparedwith their methods, our IHDA is more practical since nolabeled attribute is required for the target data, which mitigates
the burden of manual labeling. Moreover, IHDA even outperforms their methods in our experiments. Previous works [33],[34] also applied instance-level DA without annotated target
attributes. However, their approaches focus on homogeneous
DA scenario while our IHDA can tackle the more challenging
instance-level heterogeneous DA problem. Besides, [34] only
adapts the marginal distribution while [33] and ours jointly
adapts the marginal distribution and conditional distribution,
which can learn feature embedding that is robust for substantial distribution difference [39], [40].
We align conditional distribution by applying conditional
entropy minimization [40], [41], which can take the taskspecific boundaries into account. IHDA forces source andtarget domain distributions to be matched at an instance levelwith the guidance of attributes. We further demonstrate suchan approach does help to improve the model performance in
our ablation studies.
III. M ETHODOLOGY
We propose an Instance-level Heterogeneous Domain Adaptation (IHDA) framework as Fig. 2 shows. Since X s and X t1
are heterogeneous to X t2 , we construct two separated encoder
neural networks E1 and E2 respectively for (X s , X t1 ) andX t2 . ResNet-50 [42] is used as the backbone for both them andwe denote their network parameters as θE1 and θE2 . The inputs
are represented as xs , xt1 and xt2 , where xs ∈ X s , xt1 ∈ X t1
and xt2 ∈ X t2 . IHDA takes two steps to transfer domain
knowledge from the source domain to target domains. In thesstep 1, the encoder E1 , along with an identity classifier Cid
and an attribute classifier Catt , are pre-trained on the source
data. In the step 2, the weights of the pre-trained attributeclassifier are resumed, and the weights of E1 are reloaded
for both E1 and E2 . The source and target training data areused to co-train the whole network by coupling an adversarial
domain adaptation. We implement the adversarial domain
adaptation by the Gradient Reversal Layer [43]. Related codes
are available at https:// github.com/ fandulu/ IHDA.
A. Identity Learningt1,t2sand Cid
to perform
We build identity classifiers Cid
identity classification. Each of them is a single dense layer,
where the dimension of the output is equal to the numbersof identities. We denote the number of identities as Nid
and
t1,t2
Nid
for the source and target training data, respectively. The
t1,t2
s
s and
network parameters of Cid
and Cidare denoted as θCidt1,t2
θC
, respectively.
id
The identify classification losses Lsid and Lt1,t2
are definedid
as follows:s
E1 (xs ) ,ψ̂ s = Sof tmax Cid



t1,t2
ψ̂ t1 = Sof tmax Cid
E1 (xt1 ) ,



t1,t2
t2
t2
ψ̂ = Sof tmax Cid
E2 (x ) ,
(1)
s
Nid
X
Lsid = −yis log(ψ̂is ),
i=1
t1,t2
Nid

t1,t2
Lid
=−

X
i=1

t1,t2
Nid

yit1 log(ψ̂it1 ) −X
i=1

yit2 log(ψ̂it2 ),IEEE TRANSACTIONS ON MULTIMEDIA

4

where y s , y t1 and y t2 are one-hot ground truth labels of xs , xt1
and xt2 , respectively. The subscript i indicates the ith element.
ψ̂ s , ψ̂ t1 and ψ̂ t2 are the corresponding predicted identity probabilities. Here, y s , ψ̂ s ∈ Y s and y t1 , y t2 , ψ̂ t1 , ψ̂ t2 ∈ Y t1,t2 .
In the step 1, Lsid is used for the source data. In the step2, since Y s ∩ Y t1,t2 = ∅, Lt1,t2
is solely used for the target
id
data. Apart from the identity classification, we also use a triplet
loss to simultaneously learn joint embedding features that canbetter represent similarity and difference of identities. The
joint embedding feature is a global average pooling feature
with 2048 dimensions. The triplet loss Lstri , used in step 1,
is the conventional triplet loss [44] and we will not give its
details here.
We extend the conventional triplet loss to be a heterogeneous triplet loss in step 2. For simplicity, let v t1 = E1 (xt1 )and v t2 = E2 (xt2 ) stand for embedding features of photos andsketches, respectively. We represent the learned embedding
feature space as V, where v t1 , v t2 ∈ V. For feature-label pairs
in a mini-batch of target training data, we sample a sketch
anchor (vat2 , yat2 ), a photo positive sample (vpt1 , ypt1 ) and aphoto negative sample (vnt1 , ynt1 ) withypt1 = yat2 and ynt1 6= yat2 ,(2)where yat2 , ypt1 , ynt1 ∈ Y t1,t2 .
The heterogeneous triplet loss function is as follows,
i
h
t2
t1 2
t2
t1 2
v
−
v
−
v
−
v
+
α
,
Lt1,t2
=
a
p 2
a
n 2
tri
+

(3)

and target domains. It does penalize low-confident predictionsof target attributes. In order to minimize it, target domains
are forced to match with the source domain at an instancelevel other than roughly reducing domain discrepancies, whichleads to a better DA performance. On the target domains, theattribute classification loss is



t1
t1
φ̂ = Sigmoid Catt E1 (x ) ,



t2
t2
φ̂ = Sigmoid Catt E2 (x ) ,(5)
Lt1,t2att = −

N
att
X

t1
φ̂t1
i log(φ̂i ) −

i=1

N
att
X

t2
φ̂t2
i log(φ̂i ),i=1

where φ̂t1 and φ̂t2 are estimated attributes of xt1 and xt2 ,respectively. Here, φ̂t1 , φ̂t2 ∈ Z t1,t2 .
Moreover, when the photo and sketch are paired in target
data, their predicted attributes should be identical. Using thisproperty, we further construct an attribute-consistent loss forpaired target data as
t1
t2
Lt1,t2
con = 1y t1 =y t2 φ̂ − φ̂

,
2

(6)

where 1yt1 =yt2 is 1 when y t1 = y t2 (i.e., sketch and photoare paired), and 0 elsewhere.
Overall, in step 1, we only use Lsatt for attribute classification; in step 2, the entire attribute classification loss is
t1,t2= Lsatt + Latt
+ Lt1,t2
Ls,t1,t2
att
con .

where α indicates the margin between the positive and negative sketch-to-photo pairs. We empirically set α = 0.3.

(7)

C. Adversarial Domain Adaptation Learning
B. Attribute Learning
Since selected attributes are shared among three domains,
an attribute classifier Catt is used for these domains in bothstep 1 and step 2. Catt consists of three dense layers. For the
first two layers, each has 512 dimensions. The dimension of
the output layer is Natt , which is the number of attributes.
The network parameters of Catt is represented as θCatt .
Generally, each identity can be assigned to multiple attributes at the same time, we treat it as a multi-label classification task. Therefore, we apply a Binary Cross-Entropy loss
for each kind of attribute and then sum them up as the multilabel classification loss. On the source domain, the attributeclassification loss is



s
s
φ̂ = Sigmoid Catt E1 (x ) ,Lsatt =

N
att
X

(4)
−zis log(φ̂si ) − (1 − zis ) log(1 − φ̂si ),i=1
swhere z is one-hot ground truth label of xs and the subscript
i indicates the ith element; φ̂s is the estimated attributes. Here,z s , φ̂s ∈ Z s .
Without increasing annotations, attributes are unknown for
the target data and therefore the loss function above cannot
be applied directly. Inspired by previous works [5], [29], [40],[45], a multi-label entropy minimization criterion is proposed
to associate the attribute learning between the source domain

Coupling adversarial training with deep learning introduces
a powerful tool to harness domain adaptation. It has beensuccessfully applied to plenty of tasks [4], [5], [43], [46], [47].
In IHDA, X s and X t1 are heterogeneous to X t2 . Althoughs
X and X t1 are closed to homogeneous, there are differencesin their illuminations, texture styles and image resolutions. As
a result, we accommodate adversarial domain adaptation fromthe traditional two domains to three domains in this task.
We build a domain classifier Cd to distinguish domains s,
t1 and t2. Cd consists of three dense layers. For the first twolayers, each has 512 dimensions. The dimension of the output
layer is Nd , which is the number of domains. The network
parameters of Cd is represented by θCd . Considering the adversarial domain adaptation is applied, a Reverse Categorical
Cross-Entropy Loss is applied to form the loss function:ρ̂s = Sof tmax Cd E1 (xs ) ,ρ̂t1 = Sof tmax Cd E1 (xt1 ) ,



t2
t2
ρ̂ = Sof tmax Cd E2 (x ) ,
Ld =

Nd
X
i=1

dsi log(ρ̂si ) +

Nd
X
i=1

t1
dt1
i log(ρ̂i ) +

Nd
X

t2
dt2
i log(ρ̂i ),

i=1

(8)

IEEE TRANSACTIONS ON MULTIMEDIA

5where Nd stands for the number of domains used for heterogeneous domain adaptation, and Nd = 3 in this work. ds , dt1 anddt2 are one-hot domain labels of xs , xt1 and xt2 , respectively;
the subscript i indicates the ith element; ρ̂s , ρ̂t1 and ρ̂t2 areestimated domains of xs , xt1 and xt2 , respectively.
The adversarial domain adaptation procedure is a twoplayer game. One player Cd intends to maximize Ld so that
ds , dt1 and dt2 could be distinguished. Another player isthe combination of encoders E1 and E2 , which attempts tominimize Ld and confuse Cd by reducing the cross-domain
gaps.
The Gradient Reversal Layer (GRL) [43] is applied between
the encoder and the domain classifier, which simplifies thetwo-player adversarial optimization by one step in an iteration.
D. Optimization Goals of IHDA Framework
By integrating aforementioned Equations, we have the loss
functions of step 1 and step 2 as
Lstep1 = Lsid + λ1 Lstri + λ2 Lsatt ,s,t1,t2
Lstep2 = Lt1,t2+ λ1 Lt1,t2+ λ3 Ld ,tri + λ2 Lattid(9)where λ1 , λ2 and λ3 are trade-off parameters.
In step 1, the optimization goal is to find the networks and θ̂Cparameters θ̂E1 , θ̂Cidthat satisfy
att
s , θ̂C
θ̂E1 , θ̂Cid=
attargminLstep1 .

θE1 ,θC s ,θCatt

(10)

id

In step 2, pretrained θ̂E1 and θ̂Catt are resumed. Then
t1,t2
E1 , E2 , Cid
and Catt are simultaneously optimized byminimizing Lstep2 . While Cd is optimized by maximizing Ld .
The optimization goals are to find the network parametersθ̂E1 , θ̂E2 , θ̂C t1,t2 , θ̂Catt and θ̂Cd that satisfyidθ̂E1 , θ̂E2 , θ̂C t1,t2 , θ̂Catt =Lstep2 ,argminid

θE1 ,θE2 ,θC

t1,t2 ,θCattid

θ̂Cd = argmax Ld .

(11)

θCd

These two optimization goals are jointly updated in eachtraining step.
E. Insight of Optimization Goals
In IHDA framework, we have X 7→ V (by E1 and E2 ),t1,t2s
V 7→ Y (by Cid, Cid), and V 7→ Z (by Catt ). We denotest
P (v ) and P (v ) as the marginal distributions of embedding
features, P (y s |v s ) and P (y t |v t ) as the conditional distributions based on identity labels, and P (z s |v s ) and P (z t |v t ) as
the conditional distributions based on attribute labels, in thesource and target domain, respectively.
As theoretically and experimentally analyzed in [27], [39],[48]–[51], it is insufficient to obtain good embedding representations V by simply reducing the marginal domain distribution discrepancy. Conditional domain distribution discrepancy
should also be considered.
Therefore, other than coarsely minimizing the marginal
domain distribution discrepancy, which is represented bymin{ P (v s ) − P (v t ) },(12)

we aim to jointly reduce marginal domain discrepancy andconditional domain distribution discrepancy:
min{kP (v s ) − P (v t )k},(13)
min{kP (y s |v s ) − P (y t |v t )k}.
Ideally, we should minimize kP (y s |v s ) − P (y t |v t )k, however, as we have analysed, in instance-level retrieval tasks,
we have Y s ∩ Y t = ∅, it is challenging to minimizekP (y s |v s ) − P (y t |v t )k.
Since attribute learning has a main training objective on
attributes but also learn representations that can identify individuals as a side effect, we substitute kP (y s |v s ) − P (y t |v t )kwith kP (z s |v s ) − P (z t |v t )k, where z is the attribute and wehave Z s = Z t . Then we obtain the following optimizationgoal:
min{kP (v s ) − P (v t )k} ⇔ Ld ,(14)
min{kP (z s |v s ) − P (z t |v t )k} ⇔ Ls,t1,t2.
att
They are identical to the last two items of Lstep2 (seeEquation 9). We minimize the conditional distribution discrepancy Ls,t1,t2
by a semi-supervised learning approach [40],att[41]. Be the same as balance factors in Balanced Distribution
Adaptation [50], λ2 and λ3 are used to adjust the importance
reducing the marginal distribution discrepancy and conditional
distribution discrepancy.
Consequently, we approach jointly optimize the marginal
distribution and conditional distribution in our IHDA. We also
experimentally show the effectiveness of jointly optimize themarginal distribution and conditional distribution in DA in
ablation studies.
IV. E XPERIMENTS
A. Datasets
In our experiments, source datasets include UTZap50K [15], CelebFaces [16], and Market-1501 [17], theircorresponding target datasets are QMUL-Shoes [10], IIIT-D
Viewed Sketch [52] and PKU-Sketch [7], respectively.
Since sketches do not contain color information, we selectnon-color attributes from the annotated source to form theshared label space Z s = Z t1,t2 .
1) The UT-Zap50K dataset is a large-scale shoe photodataset consisting of 50K images and around 20K identities.
We select all non-color attributes to form the shared label
space, such as “Wide Toe”, “Snip Toe”, “Toggle Closure”,
and so on. The QMUL-Shoes dataset contains 419 sketchto-photo pairs of shoes, in which 304 pairs are for training
and 115 pairs are for testing. Since sketches are collectedfrom non-professional drawers, it encounters more challenges
to recognize sketch-to-photo pairs than others. Although thetarget QMUL-Shoes also includes shoes attributes, they are
different from the ones defined in UT-Zap50K dataset. To
perform supervised learning on target attributes (e.g., [30]),
we need to add new attributes to the target, which is laborious.
Our method takes advantage of unsupervised DA and does notuse labeled attributes in the target dataset.
2) The CelebFaces dataset is a large-scale face photo datasetwith around 10K identities, 202K face images, and 40 binary

IEEE TRANSACTIONS ON MULTIMEDIA

attributes. We select all non-color attributes to form the sharedlabel space, such as “Wearing-Hat”, “Bald”, “Eyeglasses”,
and so on. The IIIT-D Viewed Sketch dataset includes 238
sketch-to-photo pairs of faces, in which the sketches are drawn
by professional artists. Unlike our source dataset CelebFaces,
attribute annotation is not available in IIIT-D Viewed Sketch
dataset. We take the same evaluation protocols as [24], [26]proposed.
3) The Market-1501 dataset is a large-scale person photodataset with around 1.5K identities, 32K person images. In
work [28], [53], 27 binary attributes are annotated for Market1501 dataset. We select all non-color attributes to form theshared label space, as “gender”, “hair”, “length of sleeve”,“length of lower-body clothing”, “style of clothing”, “hat”,“age”, and “bag” (“hand bag” and “pack bad” are regarded as“bag”). The PKU-Sketch dataset consists of samples of 200
persons, in which each person has one sketch and two photos
taken from disjoint cameras. Besides, there is no attributeannotation for it. We take the evaluation protocols [7], and
150 identities are randomly chosen for training while the rest
50 are used for testing. We report average values of 10 times
experiments as the final results.
Note that, we utilize the whole source dataset and a small
portion of the annotated target training set for the training,
while the target testing set is untouched in the training process.
B. Experimental Settings
Following the procedure in Fig. 2, we start with pre-training
the E1 branch on the source data with 60 epochs in step 1. The
batch size is 64. Then, in step 2, by reloading the pre-trained
θE1 and θCatt weights, we co-train the whole IHDA network
on both source and target training data with another 60 epochs.
The batch size is 96 (32 sketch-to-photo pairs from the target
dataset and 32 samples from the source dataset). We choose
Adam optimizer [54] for the training and the initial learning
rate is set to be 1 × 10−4 . To properly utilize pre-trained
network parameters, we apply a warming-up learning rateschedule, which is modified from [55], to adjust the learning
rate. We set λ1 = 1, λ2 = 0.1 and λ3 = 0.1 to balance thewhole network. Note that the input image size for shoe, face,and person samples are 96 × 96, 144 × 128, and 384 × 128,
respectively. In the inference stage, a L2 Normalization isapplied to each embedding feature before calculating theirrelative distances.
C. Comparison with Methods of Sketch-to-photo Retrieval
We compare our method with state-of-the-art methods ofsketch-to-photo retrieval on three types of target datasets,
respectively for the sketch-to-photo retrieval of shoes, faces,and persons.
For the QMUL-Shoes dataset, we compare our methodwith Triplet SN [10], MAR-FBIR [22], CD-AFL [7], and
DSSA Triplet [23]. They are pre-trained on external datasets
TU Berlin Sketch [56] and Edge-style ImageNet [57]. Ourmodel takes the UT-Zap50K dataset as the source dataset
since shoe attributes are annotated in it. The rank-1 and rank10 re-identification accuracy of each method are reported in
TABLE III.

6

TABLE IIIT HE PERFORMANCE OF SKETCH - TO - PHOTO RETRIEVAL ON THE
QMUL-S HOES DATASET.
Model

w/ external Data

Triplet SN [10](pre-train w/o att + fine-tune)
MAR-FBIR [22](pre-train w/o att + fine-tune)
CD-AFL [7](pre-train w/o att + fine-tune)
DSSA Triplet [23](pre-train w/o att + fine-tune)

TU Berlin Sketch&Edge-style ImageNet
TU Berlin Sketch&Edge-style ImageNet
TU Berlin Sketch&Edge-style ImageNet
TU Berlin Sketch&Edge-style ImageNet

Our IHDA(Instance-level heterogeneous DA w/ att)

R1

R10

39.1 87.8
50.4 91.3
56.4 92.6
61.7 94.8
68.7 95.7

UT-Zap50K

For the IIIT-D Viewed Sketch dataset, we compare ourmethod with Deep Face [58], Light CNN [24], CDL [25], andRCN [26]. The CelebFaces dataset is used in their pre-training.
Our model takes the CelebFaces dataset as the source dataset
and uses the IHDA framework. The rank-1 re-identification
accuracy of each method is listed in TABLE IV.
TABLE IV
T HE PERFORMANCE OF SKETCH - TO - PHOTO RETRIEVAL ON THE IIIT-D
V IEWED S KETCH DATASET.

Model

w/ external Data

R1

CelebFaces

80.9

CelebFaces

84.0

CelebFaces

85.4

CelebFaces

90.3

CelebFaces

95.7

Deep Face [58](pre-train w/o att + fine-tune)
Light CNN [24](pre-train w/o att + fine-tune)
CDL [25](pre-train w/o att + fine-tune)
RCN [26](pre-train w/o att + fine-tune)
Our IHDA(Instance-level heterogeneous DA w/ att)

For the PKU-Sketch dataset, we compare our method with
Triplet SN [10], GN Siamese [59], and CD-AFL [7]. Amongthem, GN Siamese and CD-AFL are pre-trained on the Market1501 dataset. Note that CD-AFL only applied adversarial
supervised DA on PKU-Sketch dataset. When considering
Market-1501 dataset together, it simply applied pre-training
and fine-tuning. Our model takes the Market-1501 dataset as
the source dataset and uses the IHDA framework. The rank-1,
rank-5, rank-10 and rank-20 re-identification accuracy of eachmethod are reported in TABLE V.
TABLE VT HE PERFORMANCE OF SKETCH - TO - PHOTO RETRIEVAL ON THE
PKU-S KETCH DATASET.
Model
Triplet SN [10](pre-train w/o att + fine-tune)
GN Siamese [59](Ppe-train w/o att + fine-tune)
CD-AFL [7](pre-train w/o att + fine-tune)
Our IHDA(Instance-level heterogeneous DA w/ att)

w/ external Data

R1

R5

R10

R20

×

9.0

26.8 42.2

65.2

Market-1501

28.9 54.0 62.4

78.2

Market-1501

34.0 56.3 72.5

84.7

Market-1501

85.6 94.8 98.0 100.0

IEEE TRANSACTIONS ON MULTIMEDIA

Query

Gallery retrieving results by ranking
4
1
2
3
5

7

Predicted Probabilities
for Three Attributes
Boots Slippers Wide-toe
0. 3

0. 2

0. 5

0. 8

0. 1

0. 8

0. 5

0. 1

0. 3

Male

Bald

Wavy-hair

0. 6

0. 1

0. 7

0. 8

0. 1

0. 4

0. 1

0. 1

0. 2

Male

Bag

Hat

0. 8

0. 5

0. 3

0. 1

0. 3

0. 2

0. 9

0. 6

0. 4

Fig. 3. Visualization of retrieving results on QMUL-Shoes, IIIT-D, and PKUSketch datasets. In the same row, gallery images with green bounding boxes
are identical to the query image, while others with the red bounding box are
different identities. Predicted probabilities are shown for three attributes here.

On QMUL-Shoes dataset, using IHDA framework outperforms the previous state-of-the-art method by 7.0% andreaches 68.7% in rank-1 retrieving accuracy. A 5.4% gain ofrank-1 retrieving accuracy is made on IIIT-D Viewed Sketch
dataset, with a value as high as 95.7%. On PKU-Sketch
dataset, IHDA surpasses the previous state-of-the-art method
up to 51.6% and obtains 85.6% in rank-1 retrieving accuracy.
This is not happened by coincidence and we will go deep into
the essential mechanism of IHDA in ablations studies.
The improvement in PKU-Sketch dataset is more significant
than the other two datasets. We consider there are two main
reasons: (1) Sketches of QMUL-Shoes dataset are drawn byamateurs while sketches of IIIT-D and PKU-Sketch datasets
are drawn by artists with high qualities and more details (seeFig. 3), we can achieve high performance on IIIT-D and PKUSketch datasets. (2) The body poses within a body sketch-tophoto pair could be remarkably different, which causes larger
distortion than face sketch-to-photo pairs. Therefore, the finetuning strategy can reach a good performance on the IIIT-D
dataset, but on the more challenging PKU-Sketch dataset, our
IHDA shows its superiority.
In Fig. 3, we illustrate several retrieving results on target
testing data. It can be observed that each retrieved photo hasa similar appearance and structure as the query sketch. This

proves that the joint heterogeneous embedding is well learnedin our IHDA framework. For some abstract sketches, it is evendifficult to identify which is the correct corresponding photoby eyes. This explains why our IHDA framework is hindered
to further improve sketch-to-photo retrieving performance.
D. Comparison of Using Different Source Datasets and Training Strategies
Due to the lack of annotations, most of sketch-photo retrieval works [7], [22]–[26], [58], [59] utilize external datasets
to pre-train and then fine-tune their model on the target dataset.
However, there two questions worth to be further explored: 1)
though unrelated sketch-photo source data (e.g., Sketchy [60])cannot fit our complete IHDA framework, could an incomplete
IHDA framework obtain satisfactory results by using it as
the source dataset? 2) since attribute learning is applied in
our IHDA framework, could it also improve the pre-training
and fine-tuning framework? To answer these questions, weconstruct three baselines by using part of our IHDA framework
components and perform the same training strategy on threeexperimental datasets.
Specifically, in Baseline A, we conduct experiments byusing Sketchy dataset [60] as the source dataset, and QMULShoes, IIIT-D Viewed Sketch, and PKU-Sketch as the target
datasets. Referring to the training process in Fig. 2, we firstpre-train two encoders (for photo and sketch separately) using
Sketchy in Step 1, and then fine-tune both encoders using
target datasets in Step 2. In Baseline B and C, whose components and source dataset are the same as IHDA framework,
excepting that attribute learning is removed or only used in
Step 1. The results are shown in TABLE VI, TABLE VII, andTABLE VIII.
TABLE VI
T HE SOURCE DATASET SETTING AND SKETCH - TO - PHOTO RETRIEVALPERFORMANCE ON THE QMUL-S HOES DATASET.
Model

w/ external Data

R1

R10

Our Baseline A(pre-train w/o att + fine-tune)
Our Baseline B(pre-train w/o att + fine-tune)
Our Baseline C(pre-train w/ att + fine-tune)

Sketchy59.2 93.3

UT-Zap50K

57.5 92.8

UT-Zap50K

59.0 93.1

Our IHDA(Instance-level heterogeneous DA w/ att)

UT-Zap50K

68.7 95.7

The results of Baseline A consistently demonstrate when thenaive pre-training and fine-tuning strategy is applied, using
Sketchy dataset for pre-training leads to better performance
than using a homogeneous dataset. Since Sketchy dataset
contains sketch and photo pairs, using it in pre-training might
improve low-level feature representation in both encoders.
However, samples in Sketchy dataset and our target datasets
may not belong to the same category, not matter the same identity. Such a content divergence hinders the model’s capability
on transferring the high-level semantic knowledge to target
datasets. To some extent, an ideal source dataset might satisfy:
1) including sketch-photo pairs (Sketchy dataset); 2) includingIEEE TRANSACTIONS ON MULTIMEDIA8TABLE VII
T HE SOURCE DATASET SETTING AND SKETCH - TO - PHOTO RETRIEVALPERFORMANCE ON THE IIIT-D V IEWED S KETCH DATASET.

Model

w/ external Data

TABLE IX
T HE PERFORMANCE OF PHOTO - TO - SKETCH RETRIEVAL ON THE
QMUL-S HOES DATASET.

R1

Model

Our Baseline A(pre-train w/o att + fine-tune)
Our Baseline B(pre-train w/o att + fine-tune)
Our Baseline C(pre-train w/ att + fine-tune)

Sketchy

90.5

DSSA Triplet [23](pre-train w/o att + fine-tune)

CelebFaces

88.3

Our IHDA(Instance-level heterogeneous DA w/ att)

CelebFaces

88.6

Our IHDA(Instance-level heterogeneous DA w/ att)

CelebFaces

95.7

w/ external Data

T HE SOURCE DATASET SETTING AND SKETCH - TO - PHOTO RETRIEVALPERFORMANCE ON THE PKU-S KETCH DATASET.
Model

w/ external Data

R1

R5

R10

R20

Our Baseline A(pre-train w/o att + fine-tune)
Our Baseline B(pre-train w/o att + fine-tune)
Our Baseline C(pre-train w/ att + fine-tune)

Sketchy61.0 82.6 93.6

95.2

Market-1501

58.8 78.0 90.0

94.2

Market-1501

59.4 79.2 91.2

95.8

Our IHDA(Instance-level heterogeneous DA w/ att)

Market-1501

85.6 94.8 98.0 100.0

different identities but the same category as the target dataset(our selected source datasets). However, it is difficult to findsuch an ideal source dataset for all target datasets, we haveto make trade-offs. Luckily, our proposed IHDA framework
significantly improves the performance of target datasets byassuming the availability of attributes but dropping the needfor category-matched sketch-photo data.
The performances of Baseline B and C are similar, whichindicates that solely including attribute learning in existing
approaches [7], [23], [26] may not remarkably improve theretrieval performance on the target data. More than using
attributes in pre-training, our IHDA uses attributes to forma shared label space and perform semi-supervised attributelearning to intermediately reduce the domain gaps in termsof marginal distributions and conditional distributions. As aresult, our IHDA can significantly improve the performancein three target datasets.

E. Comparison with Methods of Photo-to-sketch Retrieval
Although the sketch-to-photo retrieval (query: sketch;gallery: photo) is a dominant application scenario, we perform
an inverse experiment to investigate the performance of phototo-sketch retrieval (query: photo; gallery: sketch) on threebenchmarks used in our paper. The results are shown in
Tables IX, X, and XI (Note, some unpublished codes are reimplemented by us, bias could exist).
The photo-to-sketch retrieval performance is similar tosketch-to-photo retrieval performance, which indicates that
distinguishable embedding is obtained for sketches and photos
in a symmetrical way.

R10 R20

69.6 97.4 99.1

UT-Zap50K

TABLE XT HE PERFORMANCE OF PHOTO - TO - SKETCH RETRIEVAL ON THE IIIT-D
V IEWED S KETCH DATASET.
Model

TABLE VIIIR1

TU Berlin Sketch
61.9 95.1 98.6
&Edge-style ImageNet

w/ external Data

R1

R10 R20

RCN [26](pre-train w/o att + fine-tune)

CelebFaces

90.8 95.2 97.8

Our IHDA(Instance-level heterogeneous DA w/ att)

CelebFaces

96.2 98.6 99.2

TABLE XIT HE PERFORMANCE OF PHOTO - TO - SKETCH RETRIEVAL ON THE
PKU-S KETCH DATASET.
Model

w/ external Data

R1

R10

R20

CD-AFL [7](pre-train w/o att + fine-tune)

Market-1501

37.6

76.2

92.8

Our IHDA(Instance-level heterogeneous DA w/ att)

Market-1501

88.2 100.0 100.0

F. Comparison with Methods of Instance-level Domain Adaptation on Sketch-to-photo Retrieval Tasks
Existing instance-level domain adaption methods mainly
perform their experiments on cross-dataset photo (i.e., homogeneous) retrieval tasks. In this part, we compare our IHDA
framework with two open-source works [35], [36] on PKUSketch dataset. The results are shown in Tables XII.
TABLE XIIT HE PERFORMANCE OF SKETCH - TO - PHOTO RETRIEVAL ON THE
PKU-S KETCH DATASET ( COMPARED WITH DA METHODS ).
Model

R10

R20

Market-1501

42.8 62.4 71.2

88.2

UDA-Reid [36](Instance-level homogeneous DA w/o att)

Market-1501

40.4 60.0 72.8

90.4

Our incomplete IHDA(Instance-level heterogeneous DA w/o att)

Market-1501

70.4 82.0 92.6

96.2

Our complete IHDA(Instance-level heterogeneous DA w/ att)

Market-1501

85.6 94.8 98.0 100.0

MMT [35](Instance-level homogeneous DA w/o att)

w/ external Data

R1

R5

Though MMT [35] and UDA-Reid [36] have achieved great
successes on cross-dataset homogeneous retrieval tasks (e.g.,synthetic/real photo), heterogeneous (e.g., sketch-to-photo) domain gaps may be out of the capabilities of their models.
Even without attribute learning, our incomplete IHDA still
can significantly outperform MMT and UDA-Reid on the
PKU-Sketch dataset. In particular, their domain adaptationis specifically designed to improve feature representation incross-dataset photo (homogeneous) retrieval tasks, and thus asingle encoder is applied. However, we consider that applyingIEEE TRANSACTIONS ON MULTIMEDIA

9domain adaption on sketch-to-photo (heterogeneous) retrieval
task is different. To customize instance-level domain adaptation to the heterogeneous case, our IHDA framework applies
two encoders for sketch and photo separately. Besides, ourdomain adaptation consists of three domains, as Ds , Dt1 and
Dt2 , and corresponding objective functions are designed forjointly reducing domain gaps within them. Moreover, unlike
MMT and UDA-Reid that need to include the target testing set
in their training, our IHDA framework keeps the target testing
set untouched during the training process, which demonstrates
its better generalization.

E1 ) and attribute classifier (i.e., Catt ) from the scratch. Finally,
we estimate the source attributes and compare them with theground truth. The comparison results are used for attributeselection.
SourcePhotosTargetTraining
Photos
SourceAttributes

TargetTesting
Photos
E1 + 𝑪𝒂𝒕𝒕

train byour IHDA
framework

TABLE XIIIC OMPARISON OF USING COLOR ATTRIBUTES ON PKU-S KETCH DATASET.

Model

R1

R5

R10

R20

IHDA (w/o color attributes)
IHDA (w/ color attributes)

85.6
54.4

94.8
76.2

98.0
88.6

100.0
90.2

Aside from color attributes, there are several non-color attributes could be selected. Since we may not have the groundtruth attributes on the target dataset, it is challenging to apply
traditional feature selection methods [61] to select the mostuseful ones in our task. As an alternative approach, we assume
that if the semantic information of selected attributes can bewell learned in the target domain, a new attribute classifier,which is trained from scratch using predicted target attributes,
should correctly recognize the attribute in the source domain.
Based on this assumption, we propose a novel procedure
to select attributes as Fig. 4 shows. First, we utilize allavailable non-color attributes to run our IHDA framework withthe default training setting. Then, we estimate attributes fortarget testing photos, which are untouched during the training
process. After that, target testing photos and corresponding
predicted attributes are used to train the photo encoder (i.e.,

E1 + 𝑪𝒂𝒕𝒕

predict
Predicted
TargetTesting
Attributes
compareG. Attribute Selection
Since the source datasets are not originally designed forthe target datasets, some attributes of source datasets may
not suitable for the target dataset. We would like to highlightthat task-irrelevant attributes, which is against our assumption
Z s = Z t1,t2 , should be excluded. in terms of sketch-to-photo
retrieval task, color-related attributes are task-irrelevant. As
all attributes are defined at the semantic level, excluding color
attributes of the source dataset is deterministic rather thanheuristic.
What will happen if we select color attributes to apply
IHDA framework on the sketch-to-photo retrieval task? Taking
Market-1501 and PKU-Sketch datasets as an example, we doexperiments to prove that using color attributes can cause anegative transfer. As TABLE XIII shows, the sketch-photo
retrieval performance significantly decreases after adding color
attributes. The color attributes may work as a noise, make themodel has no incentive to learn the real relevant semantic information between source and target domains. Compared withmanually annotating sketch-photo datasets, it worth spendinga few seconds to exclude color attributes at the beginning.

SourcePhotostrain fromthe scratch

predict
Predicted
SourceAttributes

Fig. 4. The procedure of selecting attributes for IHDA framework.

We use the PKU-Sketch dataset to demonstrate such an attribute selection procedure. The attribute recognition accuracy
on the source dataset is reported in TABLE XIV.
TABLE XIVATTRIBUTE RECOGNITION ACCURACY ON SOURCE DATASET ( TRAINED ON
TARGET PHOTO TRAINING SET ). “L. SLV ”, “L. LOW ”, “S. CLTH ” DENOTE“ LENGTH OF SLEEVE ”, “ LENGTH OF LOWER - BODY CLOTHING ”, “ STYLE
OF CLOTHING ”, RESP.

Attributes

L.slv L.low bag gender hair S.clth hat

Accuracy (%)

81.6

75.9

73.7

72.1

70.5

68.4

age

62.1 37.5

Based on our assumption, the attribute with a higher
recognition accuracy might contribute more in our IHDA
framework. To verify this assumption, we further select twogroups of attributes, as “L.slv, L.low, bag” and “S.clth, hat,age”, to run IHDA framework. The results in TABLE XV
agree with our assumption.
TABLE XVT HE SKETCH - TO - PHOTO RETRIEVAL PERFORMANCE BY USING DIFFERENTATTRIBUTES ON PKU-S KETCH DATASET.

Model

R1

R5

R10

R20

IHDA (w/ L.slv, L.low, bag)
IHDA (w/ S.clth, hat, age)

84.2
80.0

94.4
93.2

97.8
96.6

100.0
98.0

Although we can select important attributes through theabove method, it is inefficient. Considering that we havechosen closely related source datasets for target datasets, noncolor attributes may generally be shared between them and thetotal number of attributes should not be a large number. Could
we just use all non-color attributes?
We do another experiment on PKU-Sketch dataset to explore this question. Within all non-color attributes (8 in total),
we randomly select k (range from 1 to 8) of them and repeat
for 10 times to obtain average rank-1 value and uncertainty.
As Fig. 5 shows, in the beginning, with more attributes, therank-1 value is steadily increasing while the uncertainty isdecreasing. When more than 6 attributes are used, there is asmall difference in the rank-1 value and uncertainty. It suggests
that, within a certain range, increasing the number of attributes

IEEE TRANSACTIONS ON MULTIMEDIA

helps to improve the performance of IHDA framework. However, above a certain number, adding more shared attributes
may not significantly affect the model performance. Therefore,
for the sake of convenience, we may just utilize all of the noncolor attributes of the source dataset.

Fig. 5. Analysis on attribute selection.

Since each non-color attribute is defined as a binary label, it
has both positive and negative samples in the source dataset. In
the domain adaptation process, the framework will assign anattribute all to be negative if positive samples of this attributedo not exist in the target dataset. For such reason, within thesame category, any non-color attributes of the source dataset
can be used in forming the shared-label space, although some
attributes defined in the source data might all be negative inthe target data.
H. Trade-off Parameter Selection
In Fig. 6, we explore the sensitivity of trade-off parameters
defined in Equation 9. From value 0.001 to 10, we vary
one of them by fixing others as our default setting. It showsthat λ1 is not sensitive but very useful in IHDA framework.
When λ2 and λ3 are assigned to 0.001, it yields a poor
retrieval performance compared with assigning them to 0.1.
This implies that attribute learning and domain adaptation are
greatly beneficial to IHDA framework.

Fig. 6. Analysis on trade-off parameter selection.

I. Ablation Studies
To confirm the effectiveness of our IHDA framework, we
conduct more detailed ablation studies by using PKU-Sketch
and Market-1501 datasets. The experimental results are shownin TABLE XVI. In TABLE XVI, there are totally 11 variant
settings for ablation studies. Among them, variant settings

10from (1) to (5) are used to investigate the IHDA performanceat the different proportions of source and target data; variant
settings from (6) to (12) are designed to study the effectivenessof each component in IHDA framework. We have the complete
setup (i.e., components and datasets) of IHDA framework in(13), which considerably surpasses others.
Only using the pre-trained weights on ImageNet generates
results that are almost identical to random guessing (see(1)). Due to the huge domain gaps between sketches andphotos, solely using the pre-trained weights on Market-1501(even with attributes) leads to poor performance (see (2)).
Nevertheless, without external data, only training the modelon PKU-Sketch also cannot obtain satisfactory results (see (3))
since lacking training data. Therefore, it is highly desirable
to properly transfer the domain knowledge from Market-1501
to PKU-Sketch data. With complete components, the IHDA
framework serves for such a purpose and generates promising
results, even with 50% and 80% of PKU-Sketch training data
(see (4) and (5)).
It is critical to design the proper framework to transfer thedomain knowledge from Market-1501 to PKU-Sketch data.
Satisfactory retrieval accuracy cannot be approached throughpre-training our model on Market-1501 data and then finetuning it on PKU-Sketch data (see (6)). In (7), although weapply attribute learning on source data, it does not significantly
improve the target retrieval performance by a fine-tuning on
target data.
The alternative approach is to co-train the source and target
data and reduce the domain gap between them. Without
forming a shared label space via attributes, simply applying domain adaptation can roughly alleviate cross-domain
discrepancies. Nonetheless, the instance-level source domain
knowledge may not be properly transferred and leads tolimited improvement (see (8)). On the other hand, without
the domain adaptation, solely guided by shared attributes also
cannot accomplish promising results (see (12)). These results
are consistent with the analysis in Fig. 5. In (9), we showthat target unsupervised attribute learning plays an importantrole in IHDA. The entropy-minimization loss and attributeconsistent loss work jointly to achieve optimal attribute-guided
domain adaptation. Specifically, we verify the effectiveness ofentropy-minimization and attribute-consistent in setting (10)t1,t2and (11), respectively. The Lattcontributes dominant effects
t1,t2
while Lcon can further improve the effectiveness.
Fig. 8 shows the Rank-1 accuracy on the PKU-sketch target
set as the number of training epochs goes up. By using source
data X , The retrieval performance can increase faster than only
using target data. Although there are many potential ways toutilize the source data, which includes aligned sketch-photopairs and attributes, we propose the most effective one in thefull setting proposal.
Furthermore, we analyze how embedding feature distributions change in different learning settings. For visualization
purposes, we only select the gender attribute, as ‘Male’ and‘Female’ in IHDA. By randomly picking up samples of 10
identities from the testing set of PKU-Sketch, we project
their embedding features into a 2D space and show them inFig. 7. In simple fine-tuning strategy (i.e., TABLE XVI.(6)),

IEEE TRANSACTIONS ON MULTIMEDIA

11

TABLE XVIT HE EXPERIMENTAL SETTINGS AND RESULTS OF ABLATION STUDIES . I N THE TABLE , ‘M ARK .’ STANDS FOR THE M ARKET-1501 DATASET, AND ‘PKU.’
STANDS FOR THE PKU-S KETCH DATASET.
Index Description

E1

E2

Modules
t1,t2
s
Cid
Cid
Catt

Cd

Lossest1,t2
sLsid Lt1,t2
Lstri Lt1,t2
Ld
tri Latt Lattid

Lt1,t2conTraining Data

R1

R5 R10 R20

ImageNet2.0

8.0 20.0 40.0

(1)

w/o X s &X t1,t2

X

X

(2)

w/o X

t1,t2

X

X

(3)

w/o X sXX

(4)

w/ 50% X t1,t2

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Mark.+PKU.(50%) 75.4 90.6 92.0 94.8

(5)w/ 80% X t1,t2

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Mark.+PKU.(80%) 82.2 92.6 95.2 98.0

(6)

w/o Ls,t1,t2
&Ld
att

X

X

X

X

X

X

X

X

(7)

t1,t2
w/o Lt1,t2
att &Lcon &Ld

X

X

X

X

X

X

X

X

(8)

w/o Ls,t1,t2
att

X

X

X

X

X

X

X

X

X

(9)

t1,t2
w/o Lt1,t2
att &Lcon

X

X

X

X

X

X

X

X

X

X

(10)

w/o Lt1,t2
con

X

X

X

X

X

X

X

X

X

X

X

(11)

w/o Lt1,t2
att

X

X

X

X

X

X

X

X

X

X

X

(12)

w/o Ld

X

X

X

X

X

X

X

X

X

X

X

(13)

IHDA

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X
X

X

X

X
X

Mark.

8.0 30.0 44.0 58.0

PKU.(100%)32.0 58.6 68.2 82.6

Mark.+PKU.(100%) 58.8 78.0 90.0 94.2
X

Mark.+PKU.(100%) 59.4 79.2 91.2 95.8

X

X

Mark.+PKU.(100%) 70.4 82.0 92.6 96.2

X

Mark.+PKU.(100%) 70.6 83.2 92.4 95.8

X

Mark.+PKU.(100%) 82.8 94.2 96.8 98.4

X
X

X

Mark.+PKU.(100%) 73.2 85.6 92.6 96.8

X

Mark.+PKU.(100%) 72.4 84.2 93.2 96.6

X

Mark.+PKU.(100%) 85.6 94.8 98.0 100.0

M
M
M

M
M

M

M

M
M

M M
M
F
F

F

F
F

F

F
F

M

F
M

M

F F
F

F
F

F

Fig. 7. Visualization of embedding features for 10 identities in PKU-Sketch testing set by t-SNE [62]. Each color represents a specific identity by ground-truth
annotation. Triangles and circles denote the photo and sketch embedding, respectively. Character “M” and “F” indicate predicted “gender” attribute as “Male”
and “Female”, resp. Red dot lines highlight where to look at.

the attribute guidance.

Fig. 8. Rank 1 accuracy on PKU-Sketch target set as the number of training
epochs goes up.

although both Market-1501 dataset and PKU-Sketch training
set are utilized, photo embedding has a boundary to sketch
embedding. By applying a DA without attribute guidance (i.e.,
TABLE XVI.(7)), the boundary between photo and sketch iseliminated but the relative distance between some sketch-tophoto pairs could be incorrect. By using the complete IHDA,
sketch-to-photo pairs have become more distinguishable with

V. D ISCUSSION
It is hard to judge “pros” or “cons” in our methods without
considering the application scenario. Our IHDA framework is
specifically designed for instance-level heterogeneous image
retrieval tasks, it may not be applicable for category-level
retrieval tasks. In instance-level retrieval tasks, the shared
attributes are guaranteed since all instances belong to thesame category. In contrast, it is challenging to find shared
attributes for all instances in category-level retrieval tasks,
such as Sketchy dataset [60], PACS dataset [63] and M3SDA
dataset [64], Nonetheless, as the key contribution, our IHDA
framework has overcome the limitations of applying simple
fine-tuning strategy and conventional DA. It enables usingrich-labeled photo retrieval data to remarkably improve theretrieval performance on limited-labeled sketch-photo retrieval
tasks, which opens the door to train more effective cross-modal
image retrieval models.
VI. C ONCLUSION
Nowadays, although a large amount of rich-labeled datasets
have been made available to the public, how to utilize them

IEEE TRANSACTIONS ON MULTIMEDIA

to benefit a related limited-labeled task may vary case by caseand is waiting for further exploration. Under such background,
we analyze challenges for limited-labeled sketch-to-photo retrieval from the transfer learning perspective and raise an
Instance-level Heterogeneous Domain Adaptation framework
to tackle them. It demonstrates an important insight: attributes,
which are only annotated in the source domain, can be used
to form a shared label space for both source and target
domains in instance-level retrieval task. With such a sharedlabel space, instance-level domain knowledge can be welltransferred across datasets and heterogeneous modalities. It
opens the door to train more effective cross-modal image
retrieval models by using related rich-labeled single-modal
image data.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Numbers JP17H06101, and a MSRA Collaborative Research 2019
Grant Awarded by Microsoft Research Asia.
R EFERENCES[1] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans.
Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, 2010.
[2] Z. Pei, Z. Cao, M. Long, and J. Wang, “Multi-adversarial domain adaptation,” in Proceedings of the AAAI Conference on Artificial Intelligence,
2018, pp. 3934–3941.
[3] A. Rozantsev, M. Salzmann, and P. Fua, “Beyond sharing weights
for deep domain adaptation,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 41, no. 4, pp. 801–814, 2019.
[4] Z. Cao, L. Ma, M. Long, and J. Wang, “Partial adversarial domain
adaptation,” in Proceedings of the European Conference on Computer
Vision, 2018, pp. 139–155.
[5] H. Zou, Y. Zhou, J. Yang, H. Liu, H. P. Das, and C. J. Spanos,“Consensus adversarial domain adaptation,” in Proceedings of the AAAI
Conference on Artificial Intelligence, 2019, pp. 5997–6004.
[6] Y. Zou, Z. Yu, B. V. K. V. Kumar, and J. Wang, “Unsupervised domain
adaptation for semantic segmentation via class-balanced self-training,”
in Proceedings of the European Conference on Computer Vision, 2018,
pp. 297–313.
[7] L. Pang, Y. Wang, Y.-Z. Song, T. Huang, and Y. Tian, “Cross-domain
adversarial feature learning for sketch re-identification,” in Proceedingsof the ACM international Conference on Multimedia, 2018, pp. 609–617.
[8] S. Ouyang, T. M. Hospedales, Y. Song, and X. Li, “Forgetmenot:
Memory-aware forensic facial sketch matching,” in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, 2016,
pp. 5571–5579.
[9] S. Wang, J. Zhang, T. X. Han, and Z. Miao, “Sketch-based image
retrieval through hypothesis-driven object boundary selection with hlr
descriptor,” IEEE Transactions on Multimedia, vol. 17, no. 7, pp. 1045–
1057, 2015.
[10] Q. Yu, F. Liu, Y.-Z. Song, T. Xiang, T. M. Hospedales, and C.-C.
Loy, “Sketch me that shoe,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2016, pp. 799–807.
[11] Y. Zhang, X. Qian, X. Tan, J. Han, and Y. Tang, “Sketch-based
image retrieval by salient contour reinforcement,” IEEE Transactions
on Multimedia, vol. 18, no. 8, pp. 1604–1615, 2016.
[12] S. D. Bhattacharjee, J. Yuan, Y. Huang, J. Meng, and L. Duan,“Query adaptive multiview object instance search and localization using
sketches,” IEEE Transactions on Multimedia, vol. 20, no. 10, pp. 2761–
2773, 2018.
[13] J. Choi, H. Cho, J. Song, and S. M. Yoon, “Sketchhelper: Real-timestroke guidance for freehand sketch retrieval,” IEEE Transactions on
Multimedia, vol. 21, no. 8, pp. 2083–2092, 2019.
[14] Z. Wang, Z. Wang, Y. Zheng, Y. Wu, W. Zeng, and S. Satoh, “Beyond
intra-modality: A survey of heterogeneous person re-identification,”
arXiv preprint arXiv:1905.10048, 2019.
[15] A. Yu and K. Grauman, “Semantic jitter: Dense supervision for visualcomparisons via synthetic images,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5570–5579.

12[16] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes
in the wild,” in Proceedings of the IEEE International Conference on
Computer Vision, 2015, pp. 3730–3738.
[17] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable
person re-identification: A benchmark,” in Proceedings of the IEEE
International Conference on Computer Vision, 2015, pp. 1116–1124.
[18] Z. Wang, J. Jiang, Y. Yu, and S. Satoh, “Incremental re-identification
by cross-direction and cross-ranking adaption,” IEEE Transactions on
Multimedia, vol. 21, no. 9, pp. 2376–2386, 2019.
[19] Z. Zeng, Z. Wang, Z. Wang, Y. Zheng, Y.-Y. Chuang, and S. Satoh,“Illumination-adaptive person re-identification,” IEEE Transactions on
Multimedia, 2020. [Online]. Available: http://dx.doi.org/10.1109/TMM.
2020.2969782[20] Z. Wang, R. Hu, Y. Yu, C. Liang, and W. Huang, “Multi-level fusion
for person re-identification with incomplete marks,” in Proceedings ofthe 23rd ACM international conference on Multimedia, 2015, pp. 1267–
1270.
[21] Z. Wang, X. Bai, M. Ye, and S. Satoh, “Incremental deep hidden attributelearning,” in Proceedings of the 26th ACM international conference on
Multimedia, 2018, pp. 72–80.
[22] J. Song, Y. Song, T. Xiang, T. M. Hospedales, and X. Ruan, “Deepmulti-task attribute-driven ranking for fine-grained sketch-based image
retrieval,” in Proceedings of the British Machine Vision Conference,
2016.
[23] J. Song, Q. Yu, Y. Song, T. Xiang, and T. M. Hospedales, “Deep spatialsemantic attention for fine-grained sketch-based image retrieval,” inProceedings of the IEEE International Conference on Computer Vision,
2017, pp. 5552–5561.
[24] X. Wu, R. He, Z. Sun, and T. Tan, “A light CNN for deep face
representation with noisy labels,” IEEE Trans. Information Forensics
and Security, vol. 13, no. 11, pp. 2884–2896, 2018.
[25] X. Wu, L. Song, R. He, and T. Tan, “Coupled deep learning forheterogeneous face recognition,” in Proceedings of the AAAI Conferenceon Artificial Intelligence, 2018, pp. 1679–1686.
[26] Z. Deng, X. Peng, and Y. Qiao, “Residual compensation networks forheterogeneous face recognition,” in Proceedings of the AAAI Conferenceon Artificial Intelligence, 2019, pp. 8239–8246.
[27] K. Saito, Y. Ushiku, and T. Harada, “Asymmetric tri-training for
unsupervised domain adaptation,” in Proceedings of the International
Conference on Machine Learning, 2017, pp. 2988–2997.
[28] Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang,“Improving person re-identification by attribute and identity learning,”
Pattern Recognition, vol. 95, pp. 151–161, 2019.
[29] Z. Luo, Y. Zou, J. Hoffman, and L. F. Fei-Fei, “Label efficient learning oftransferable representations acrosss domains and tasks,” in Proceedingsof the Advances in Neural Information Processing Systems, 2017, pp.
165–177.
[30] D. Liu, N. Wang, C. Peng, J. Li, and X. Gao, “Deep attribute guided
representation for heterogeneous face recognition.” in Proceedings ofthe International Joint Conference on Artificial Intelligence, 2018, pp.
835–841.
[31] T. Gebru, J. Hoffman, and L. Fei-Fei, “Fine-grained recognition in thewild: A multi-task domain adaptation approach,” in Proceedings of theIEEE International Conference on Computer Vision, 2017, pp. 1349–
1358.
[32] Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie, “Large scale
fine-grained categorization and domain-specific transfer learning,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2018, pp. 4109–4118.
[33] S. Lin, H. Li, C.-T. Li, and A. C. Kot, “Multi-task mid-levelfeature alignment network for unsupervised cross-dataset person reidentification,” in Proceedings of the British Machine Vision Conference,
2018, pp. 19–32.
[34] J. Wang, X. Zhu, S. Gong, and W. Li, “Transferable joint attributeidentity deep learning for unsupervised person re-identification,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2018, pp. 2275–2284.
[35] Y. Ge, D. Chen, and H. Li, “Mutual mean-teaching: Pseudo label refinery
for unsupervised domain adaptation on person re-identification,” inInternational Conference on Learning Representations, 2020. [Online].
Available: https://openreview.net/forum?id=rJlnOhVYPS[36] L. Song, C. Wang, L. Zhang, B. Du, Q. Zhang, C. Huang, and X. Wang,“Unsupervised domain adaptive re-identification: Theory and practice,”
Pattern Recognition, p. 107173, 2020.
[37] D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual
learning,” in Advances in Neural Information Processing Systems, 2017,
pp. 6467–6476.

IEEE TRANSACTIONS ON MULTIMEDIA[38] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars,“Memory aware synapses: Learning what (not) to forget,” in Proceedingsof the European Conference on Computer Vision (ECCV), 2018, pp.
139–154.
[39] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu, “Transfer featurelearning with joint distribution adaptation,” in Proceedings of the IEEE
International Conference on Computer Vision, 2013, pp. 2200–2207.
[40] K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko, “Semisupervised domain adaptation via minimax entropy,” arXiv preprint
arXiv:1904.06487, 2019.
[41] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy
minimization,” in Proceedings of the Advances in Neural Information
Processing Systems, 2004, pp. 529–536.
[42] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on Computer Vision
and Pattern Recognition, 2016, pp. 770–778.
[43] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by
backpropagation,” in Proceedings of the International Conference on
International Conference on Machine Learning, 2015, pp. 1180–1189.
[44] G. Chechik, V. Sharma, U. Shalit, and S. Bengio, “Large scale online
learning of image similarity through ranking,” Journal of Machine
Learning Research, vol. 11, no. Mar, pp. 1109–1135, 2010.
[45] Y. Shu, Z. Cao, M. Long, and J. Wang, “Transferable curriculum forweakly-supervised domain adaptation,” in Proceedings of the AAAI
Conference on Artificial Intelligence, 2019, pp. 4951–4958.
[46] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,”
Neurocomputing, vol. 312, pp. 135–153, 2018.
[47] P. Dai, R. Ji, H. Wang, Q. Wu, and Y. Huang, “Cross-modality person
re-identification with generative adversarial training.” in Proceedings ofthe International Joint Conference on Artificial Intelligence, 2018, pp.
677–683.
[48] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learningwith joint adaptation networks,” in Proceedings of the International
Conference on Machine Learning, 2017, pp. 2208–2217.
[49] J. Zhang, W. Li, and P. Ogunbona, “Jdeep face recognitionoint geometrical and statistical alignment for visual domain adaptation,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2017, pp. 5150–5158.
[50] J. Wang, Y. Chen, S. Hao, W. Feng, and Z. Shen, “Balanced distribution
adaptation for transfer learning,” in Proceedings of the IEEE International Conference on Data Mining, 2017, pp. 1129–1134.
[51] H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon, “On learning
invariant representations for domain adaptation,” in Proceedings of theInternational Conference on Machine Learning, 2019, pp. 7523–7532.
[52] H. S. Bhatt, S. Bharadwaj, R. Singh, and M. Vatsa, “Memetically
optimized MCWLD for matching sketches with digital face images,”
IEEE Trans. Information Forensics and Security, vol. 7, no. 5, pp. 1522–
1535, 2012.
[53] Z. Wang, J. Jiang, Y. Wu, M. Ye, X. Bai, and S. Satoh, “Learning sparseand identity-preserved hidden attributes for person re-identification,”
IEEE Transactions on Image Processing, vol. 29, no. 1, pp. 2013–2025,
2019.
[54] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[55] H. Luo, W. Jiang, Y. Gu, F. Liu, X. Liao, S. Lai, and J. Gu, “A strong
baseline and batch normalization neck for deep person re-identification,”
IEEE Transactions on Multimedia, pp. 1–1, 2019.
[56] M. Eitz, R. Richter, T. Boubekeur, K. Hildebrand, and M. Alexa,“Sketch-based shape retrieval,” ACM Trans. Graph., vol. 31, no. 4, pp.
31:1–31:10, 2012.
[57] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large
scale visual recognition challenge,” International Journal of Computer
Vision, vol. 115, no. 3, pp. 211–252, 2015.
[58] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,”in Proceedings of the British Machine Vision Conference, vol. 41, 2015,
pp. 1–12.
[59] P. Sangkloy, N. Burnell, C. Ham, and J. Hays, “The sketchy database:learning to retrieve badly drawn bunnies,” ACM Trans. Graph., vol. 35,
no. 4, pp. 119:1–119:12, 2016.
[60] ——, “The sketchy database: learning to retrieve badly drawn bunnies,”
ACM Transactions on Graphics (TOG), vol. 35, no. 4, pp. 1–12, 2016.
[61] I. Guyon and A. Elisseeff, “An introduction to variable and feature
selection,” Journal of machine learning research, vol. 3, no. Mar, pp.
1157–1182, 2003.
[62] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journalof Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008.

13[63] D. Li, Y. Yang, Y. Song, and T. M. Hospedales, “Deeper, broader
and artier domain generalization,” in Proceedings of the International
Conference on Computer Vision, 2017, pp. 5543–5551.
[64] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment
matching for multi-source domain adaptation,” in Proceedings of theIEEE International Conference on Computer Vision, 2019, pp. 1406–
1415.

Fan Yang received a B.S. degree and an M.S
degree from Nanjing University and Nara Institute of
Science and Technology in 2012 and 2018, respectively. He is currently a Ph.D. candidate at the Nara
Institute of Science and Technology. His research
focuses on video processing.

Yang Wu (M’19) received a BS degree and a
Ph.D degree from Xi’an Jiaotong University in 2004
and 2010, respectively. He is currently a programspecific senior lecturer with Department of Intelligence Science and Technology, Graduate School ofInformatics, Kyoto University. He is also a guest
associate professor of Nara Institute of Science andTechnology (NAIST), where he was an assistantprofessor of the NAIST International Collaborative
Laboratory for Robotics Vision, from Dec.2014 toJun. 2019. From 2011 to 2014, he was a program
specific researcher with the Academic Center for Computing and Media
Studies, Kyoto University. His research is in the fields of computer vision,
pattern recognition, and image/video search and retrieval.

Zheng Wang (M’19) received the B.S. and M.S.
degrees from Wuhan University, Wuhan, China, in2006 and 2008, respectively, and the Ph.D. degree from the National Engineering Research Center for Multimedia Software, School of Computer
Science, Wuhan University, Wuhan, China, in 2017.
He is currently a JSPS Fellowship Researcher at
Shin’ichi Satoh’s Lab, National Institute of Informatics, Japan. His research interests focus on person
re-identification and instance search. He received
the Best Paper Award at the 15th Pacific-Rim Conference on Multimedia (PCM 2014) and the 2017 ACM Wuhan Doctoral
Dissertation Award.

Xiang Li received his B.S. degree from Sun Yat-sen
University in 2019 (Distinguished Graduate). He iscurrently a Research Assistant at Nanyang Technological University. His research focus on machinelearning and computer vision.

IEEE TRANSACTIONS ON MULTIMEDIA

Sakriani Sakti received her B.E. degree in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000, she received
DAAD-Siemens Program Asia 21st Century Award
to study in Communication Technology, University
of Ulm, Germany, and received her MSc degreein 2002. During her thesis work, she worked with
Speech Understanding Department, DaimlerChrysler
Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC Labs,
Japan, and during 2006-2011, she worked as anexpert researcher at NICT SLC Groups, Japan. While working with ATRNICT,
Japan, she continued her study (2005-2008) with Dialog Systems Group
University of Ulm, Germany, and received her PhD degree in 2008. She actively involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011). In 2009-2011, she
served as a visiting professor of Computer Science Department, Universityof Indonesia (UI), Indonesia. In 2011-2017, she was an assistant professor at
the Augmented Human Communication Laboratory, NAIST, Japan. She served
also as a visiting scientific researcher of INRIA Paris-Rocquencourt, France, in
2015-2016, under JSPS Strategic Young Researcher Overseas Visits Programfor Accelerating Brain Circulation. Currently, she is a research associate
professor at NAIST, as well as a research scientist at RIKEN, the Centerof for Advanced Intelligent Project AIP, Japan. She is a member of JNS,
SFN, ASJ, ISCA, IEICE and IEEE. She is also the officer of ELRA/ISCA
Special Interest Group on Under-resourced Languages (SIGUL) and a board
Member of Spoken Language Technologies for Under-Resourced Languages(SLTU). Her research interests include statistical pattern recognition, graphical modeling framework, deep learning, multilingual speech recognition &synthesis, spoken language translation, affective dialog system, and cognitive
communication.

Satoshi Nakamura is Director of Data Science Center and Professor at the Graduate School of Scienceand Technology, Nara Institute of Science and Technology, Japan, Team Leader of Riken AIP Tourism
Information Analytics Team, and Honorarprofessorof Karlsruhe Institute of Technology, Germany. He
received his B.S. from Kyoto Institute of Technology
in 1981 and Ph.D. from Kyoto University in 1992.
He was an Associate Professor of the Graduate
School of Information Science at Nara Instituteof Science and Technology in 1994-2000. He was
Director of ATR Spoken Language Communication Research Laboratories
in 2000-2008 and Vice president of ATR in 2007-2008. He was DirectorGeneral of Keihanna Research Laboratories and the Executive Director of
Knowledge-Creating Communication Research Center, National Institute ofInformation and Communications Technology, Japan in 2009-2010. He iscurrently Director of Augmented Human Communication laboratory and a full
professor Nara Institute of Science and Technology, Japan. He is interested inmodeling and systems of speech-to-speech translation and speech recognition.
He is one of the leaders of speech-to-speech translation research and has been
serving various speech-to-speech translation research projects in the world
including C-STAR, IWSLT, and A-STAR. He received the Yamashita ResearchAward, Kiyasu Award from the Information Processing Society of Japan,
Telecom System Award, AAMT Nagao Award, Docomo Mobile ScienceAward in 2007, ASJ Award for Distinguished Achievements in Acoustics. He
received the Commendation for Science and Technology by the Minister of
Education, Science and Technology, and the Commendation for Science andTechnology by the Minister of Internal Affairs and Communications. He also
received the LREC Antonio Zampolli Award 2012. He has been elected Board
Member of International Speech Communication Association, ISCA, in 20112019, IEEE Signal Processing Magazine Editorial Board Member since April
2012-2015, IEEE SPS Speech and Language Technical Committee Member
since 2013-2016. He is ATR Fellow, IPSJ Fellow, ISCA Fellow, and IEEE
Fellow.

14