Cross Domain Image Matching in Presence of Outliers
Xin Liu Seyran Khademi Jan C. van Gemert
Computer Vision Lab, Delft University of Technology
Delft, The Netherlands

Abstract

Source Domain
Target Domain (Inliers)

1.0

0.5

f2

f2

5
0

0.0

−0.5

−1.0

−5
−6

−4

−2

0

f1

2

4

−1.5
−1.5

6

1.5

1.0

1.0

0.5

0.5

f2

1.5

0.0

−0.5

−1.0

−1.5
−1.5

Source Domain
Target Domain (Inliers)
Target Domain (Outliers)
−1.0

−0.5

0.0

f1

0.5

(c) Matching + DA

−1.0

−1.0

1.5

0.0

f1

0.5

1.0

1.5

0.0

−0.5

1.0

−0.5

(b) Matching + DA(a) Original sample distribution

f2

Cross domain image matching between image collections from different source and target domains is challenging in times of deep learning due to i) limited variation ofimage conditions in a training set, ii) lack of paired-image
labels during training, iii) the existing of outliers that makes
image matching domains not fully overlap. To this end, we
propose an end-to-end architecture that can match cross domain images without labels in the target domain and handle
non-overlapping domains by outlier detection. We leverage
domain adaptation and triplet constraints for training a network capable of learning domain invariant and identity distinguishable representations, and iteratively detecting theoutliers with an entropy loss and our proposed weighted
MK-MMD. Extensive experimental evidence on Office [17]dataset and our proposed datasets Shape, Pitts-CycleGAN
shows that the proposed approach yields state-of-the-art
cross domain image matching and outlier detection performance on different benchmarks. The code will be made publicly available.

1.5

Source Domain
Target Domain (Inliers)
Target Domain (Outliers)

10

−1.5
−1.5

Source Domain
Target Domain (Inliers)
Target Domain (Outliers)
−1.0

−0.5

0.0

f1

0.5

1.0

1.5

(d) Our method

1. Introduction

Figure 1: Domain adaptation (DA) and image matching applied
on a 2D toy dataset generated with domain shift between source
and target domains. (a) Original distribution, (b) no outliers, (c)with outliers, (d) our method. The result of (b) and (c) shows thatoutliers affect the alignment of source samples and inlier target
samples. (c) and (d) show that our outlier detection helps separating the outliers from the aligned source samples and inlier target
samples.

Cross domain image matching is about matching two images that are collected from different sources (e.g. photos ofthe same location but captured in different illuminations,seasons or era). It has wide application value in different
areas, with research in location recognition over large timelags [3], e-commerce product image retrieval [8], urban environment image matching for geo-localization [20], etc.
Even using deep feature representation learning, the automated cross domain image matching task remains challenging mainly due to the following difficulties. First, it
is difficult to match varying observations of the same location or object, in general. Second, often the paired-image
examples from two domains are not available for training
neural networks. Third, the image samples in two domains
may not fully overlap due to the existing of outlier images,
which affects the matching performance if such outliers are
not detected.

In this work, we address the problem of domain adaptation for feature learning in a cross domain matching task
when outliers are present. As is common in domain adaptation, we only have labeled image pairs from the source
domain, but no labels from the target domain. To resolve the domain disparity between the train and the test
data, we are inspired from Siamese network [2] for image
matching and domain adaptation used in image classification [13, 18, 22, 23, 26]. We propose a triplet constraints
network to learn the domain invariant and identity distinguishable representations of the samples. This is made possible by utilizing the paired-image information from thesource domain, a weighted multi-kernel maximum mean
discrepancy (weighted MK-MMD) method and an entropy
loss. The setting of the problem and experiment results ofour method are depicted on a 2D toy dataset in Figure 1.
To verify our method, we introduce two new synthetic
datasets as there are no publicly available datasets for our
problem setting. Moreover, we believe outlier-aware algorithms are essential to design practical domain adaptation
algorithms as many real data repositories contain irrelevantsamples w.r.t. the source domain. In summary, our main
contribution is two-fold:• Joint domain adaptation and outlier detection.
• Two new datasets, Pits-CycleGAN dataset and Shapedataset, for cross domain image matching.

2. Related work
2.1. Image matching
Feature learning based matching methods become popular due to its improved performance over hand-crafted features (e.g. SIFT [15]). Siamese network architectures [2]are among the most popular feature learning networks, especially for pairs comparison tasks. We also adopt Siamese
network as part of our framework. The purpose is tolearn feature representations to distinguish matching andunmatching pairs in the source domain, which assists the
network in learning to match cross domain images. In the
cross-domain image matching context, Lin et al. [11] investigated a deep Siamese network to learn feature embedding
for cross-view image geo-localization. Kong et al. [9] applied Siamese architecture to cross domain footprint matching. Tian et al. [20] utilized Siamese network for matching
the building images from street view and bird’s eye view.
Unlike the existing works on cross-domain image matching, we consider labeled paired-image information is only
available in the source domain.

2.2. Domain adaptation
Domain adaptation have been researched over recent
years in diverse domain classification tasks, in which adversarial learning and statistic methods are main approaches.
Ganin et al. [4] proposed domain-adversarial training ofneural networks with input of labeled source domain data
and unlabeled target domain data for classification. In [26],the authors proposed a deep transfer network (DTN), which
achieved domain transfer by simultaneously matching boththe marginal and the conditional distributions with adopting the empirical maximum mean discrepancy (MMD) [5],which is a nonparametric metric. Venkateswara et al. [23]applied MK-MMD [6] to a deep learning framework thatcan learn hash codes for domain adaptive classification. In
this setting MK-MMD loss promotes nonlinear alignmentof data, which generates a nonparametric distance in Reproducing Kernel Hilbert Space (RKHS). The distance between two distributions is the distance between their means

in a RKHS. When two data sets belong to the same distribution, their MK-MMD is zero. Based on the successful
performance of MK-MMD loss, we also adopt it to adapt
different domains, this time for image matching task. Thisrequires the marriage of Siamese network with MK-MMD
loss, as we do later in our paper.

2.3. Outlier detection
Much work exists on outlier detection [1, 12, 16, 25].
Chalapathy et al. [1] proposed an one-class neural network (OC-NN) encoder-decoder model to detect anomalies.
Sabokrou et al. [16] also applied the encoder-decoder architecture as part of their network for novelty detection. Zhang
et al. [25] proposed an adversarial network for partial domain adaptation to deal with outlier classes in the source
domain. Their network is for classification task, and they
do not have the assumption that outliers originate from lowdensity distribution. Instead, we are inspired by the work of
Liu et al. [12] which uses a kernel-based method to learn,jointly, a large margin one-class classifier and a soft label assignment for inliers and outliers. Using the soft label assignment, we implement outlier detection with crossdomain image matching in an iterative sample reweightingway.

3. Domain adaptive image matching
3.1. Siamese loss
We introduce our proposal for domain adaptation for image matching task once labeled data is not available in thetarget domain. Let Xs denote the source domain image set.
A pair of images xi , xj ∈ Xs are used as input to part ofour network, as shown in Figure 2. xi , xj can be a matchingpair or an unmatching pair. The objective is to automaticallylearn a feature representation, f (·), that effectively maps the
input xi , xj to a feature space, in which matching pairs areclose to each other and unmatching pairs are far apart. We
employ the contrastive loss as introduced in [7]:11yD2 + (1 − y){max(0, m − D)}2 , (1)22where y ∈ {0, 1} indicates unmatching pairs with y = 0
and matching pairs with y = 1, D is the Euclidean distance
between the two feature vectors f (xi ) and f (xj ), and m is
the margin parameter acting as threshold to separate matching and unmatching pairs.
L(xi , xj , y) =3.2. Domain adaptation loss
It is known that in deep CNNs, the feature representations transition from generic to task-specific as one goes up
from bottom layers to other layers [24]. Compared to the
convolution layers conv1 to conv5, the fully connected layers are more task-specific and need to be adapted before
they can be transferred [23].

Convolution Layers

Fully Connected Layers
Contrastive Loss

Source Pairs

Weighted MK-MMD

Target Data

Entropy Loss
fc6
conv1

conv2

conv3

conv4

fc7

fc8

conv5

Figure 2: The network for cross domain image matching and outlier detection. The contrastive loss makes the network to learn pairedimage information from the source. The weighted MK-MMD loss trains the network to learn transferable features between the source andthe inliers of the target. The entropy loss helps distinguish inliers and outliers in the target domain.

Accordingly, our approach attempts to minimize the
MK-MMD loss to reduce the domain disparity between thesource and target feature representations for fully connected
layers, F = {f c6, f c7, f c8}. The multi-layer MK-MMD
loss is given by,
X
M(us , ut ) =d2k (uls , ult ),(2)
l∈Ft,l ntnslwhere, uls = {us,li }i=1 and ut = {ui }i=1 are the set of
output representations for the source and target data at layer∗,ll, u∗,li is the output representation of inuput image xi forth2
the l layer. The MK-MMD measure dk (·) is the multikernel maximum mean discrepancy between the source andtarget representations [6]. For a nonlinear mapping φ(·) associated with a reproducing kernel Hilbert space Hk andkernel k(·), where k(x, y) = hφ(x, y)i, the MK-MMD isdefined as,

d2k (uls , ult ) = ||E[φ(us,l )] − E[φ(ut,l )]||Hk .

4.1. Importance weighted domain adaptation
In our implementation, the MK-MMD loss in subsection 3.2 is calculated over every batch of data points during the back-propagation. Let n (even) be the numberof source data points us := {usi }ni=1 and the number oftarget data points ut := {uti }ni=1 in the batch. Then,the MK-MMD can be defined over a set of 4 data points
zi = [us2i−1 , us2i , ut2i−1 , ut2i ], ∀i ∈ {1, 2, ..., n/2}. Thus,
the MK-MMD is given by,d2k (us , ut ) =

κ
X
m=1

n/2

βm

1 X
hm (zi ),n/2 i=1

where, κ is the number of kernels and βm = 1/κ is theweight for each kernel. And we can expand hm (·) as,hm (zi ) = km (us2i−1 , us2i ) + km (ut2i−1 , ut2i )− km (us2i−1 , ut2i ) − km (us2i , ut2i−1 ), (5)(3)

The characteristic kernel k(·), is determined as a convex
combination ofPκ PSD kernels, {km }κm=1 , K := {k : k =
P
κ
κ
β
k
,
m=1 m mm=1 βm = 1, βm ≥ 0, ∀m}. In particular, we follow [14] and set the kernel weights as βm = 1/κ
.

4. Proposed method: Outlier-aware domain
adaptive matching
The task is to match images with the same content but
from different domains where the outliers are present in thetarget domain. We assume that in the source domain there
are sufficient labeled image pairs and in the target domain
low-density outliers are present. As in conventional domain
adaptation setting labeled data is not available in the target
domain. We propose a deep triplet network which is comprised of three instances of the same feed-forward networkwith shared parameters, as shown in Figure 2.

(4)||x−y||2

in which, the kernel is km (x, y) = exp(− σm 2 ).
With equations 4 and 5, we can interpret that in the minimum calculation unit (hm (zi )), two target domain images
contribute to MK-MMD loss calculation. When there are
outliers in the target domain, we only want the inliers tocontribute to the calculation, but not the outliers. Therefore, we could assign the target samples with weights wi as
1 for inliers, and 0 for outliers. Because we have no groundtruth labels, we can only treat the weights as the probabilityof the target samples to be inliers. Hence, we can introduce
the weighted MK-MMD as,
d2wk (us , ut ) =

κ
X
m=1

n/2

βm

1 X
w2i−1 w2i hm (zi ),n/2 i=1(6)

where, w2i−1 and w2i are the weights of the target
data points ut2i−1 and ut2i in hm (zi ) respectively, andw2i−1 , w2i ∈ [0, 1]. We will explain how to obtain theweight for each target domain sample in next subsection.

4.2. Outlier detection

Algorithm 1

Since the inlier-outlier label is not available, we implement an entropy loss to iteratively reassign target domain
sample probability of being an inlier, which provides theweights for the weighted MK-MMD.
We use the similarity measure hui , uj i to learn discriminative inlier-outlier information for the target domain data.
We define three classes of reference data ur for similarity
measure, the source domain class u1 , the pseudo inlier classu2 and the pseudo outlier class u3 . An ideal target output
uti needs to be similar to many of the outputs from one of
the classes, {uck }K
k=1 . We assume K data points for everyclass c, where c ∈ {1, 2, 3} and uck is the k th output fromclass c. Then the probability measure for each target sample
can be outlined as,PK|
exp(uti uck ),(7)
pic = PC k=1
PK
t| c
c=1
k=1 exp(ui uk )Input: source domain and target domain training data
Output: target domain training data probabilities
1: Initialization i = 0, calculate the average Euclidean
distance of each target domain training sample between
all the source domain training samples, sort the distances in ascending order and initialize target domain
training samples’ weights according to the sorted distances, xi ∈ first half: wi = 0.7 (pseudo inlier class),xi ∈ second half: wi = 0.3 (pseudo outlier class). Inlier class consists of source domain training data, which
has the same number of samples with pseudo inlier andpseudo outlier classes.
2: Repeat:
3: i = i + 1
4: make new mini batches
5: minimize the overall loss function objective (10)
6: update the samples’ weights by equation 7 and 9
7: update the sets of pseudo inlier class and pseudo outlier
class
8: Until target samples’ probabilities are unchanged ortraining time ends

where, pic is the probability that a target domain sample
xti is assigned to category c. When the sample output issimilar to one category only, the probability vector pi =[pi1 , ..., pic ]> tends to be a one-hot vector. A one-hot vector
can be viewed as a low entropy realization of pi . Thus, we
introduce a loss to capture the entropy of the probability
vectors. The entropy loss can be given by,nS(ur , ut ) = −

C

t X
1 X
pic log(pic ).
nt i=1 c=1(8)

In subsection 4.1, we discussed the weighted MK-MMD
loss with weights w2i−1 and w2i . With the sample probabilities of target domain data calculated from equation 7, theweights are calculated as,(
pi1 +pi2
if xti is classified as sourcei2 +pi3
wi = pi1 +ppi2. (9)if xti is classified as others
pi1 +pi2 +pi3
If a target domain sample is classified as ”source”, thenit has a high probability of being an inlier, and therefore
should contribute more to reducing the domain disparity.
So we calculate the weight of such a target domain sample
with the sum of pi1 and pi2 .
Algorithm We iteratively update the target domain data
weights after each epoch during training, which works together with domain adaptation for guiding and correcting
the detection of outliers and inliers.
The proposed algorithm for outlier detection is showedin the following. The proposed method is built upon theintuitive assumption that outliers originate from low-density
distribution. Thus, we can assume that the ratio of outliers
to all the target domain data is no more than 50%.

4.3. Overall objective
We propose a model for cross domain image matchingand outlier detection, which incorporates learning image
matching information from source domain (1), weighted
domain adaptation between the source and the target (6) andoutlier detection (8) in a deep CNN. The overall objective
is given by:minu J = L(us ) + γMw (us , ut ) + ηS(ur , ut ),(10)
S
where, u := {us ut } and (γ, η) control the importance ofdomain adaptation (6) and entropy loss (8) respectively.

5. Experiments
5.1. Datasets
There are no publicly available datasets for our task.
Therefore, we propose two datasets for evaluation. Sample images from the three datasets are shown in Figure 3.
Shape is one of the synthetic datasets we generate. It
contains 60k source domain images, 30k target domain images (including 2800 outliers). The outlier images are madeup of single alphabets or digits. The source domain andinlier images are combinations of two geometric shapes,
drawn with black solid lines and colored dot lines, respectively. We define two images are a matching pair if the combination of shapes is the same.
Pitts-CycleGAN is the other synthetic dataset, whichcontains 204k Pittsburgh Google Street View images fromSource

TargetOutlier

Shape

Pitts-CycleGAN

Baseline

Experiment

SIFT + Fisher Vector [15, 19]
Siamese network [2]SiameseDA (upper bound)
SiameseDAOut (lower bound)

Domain adaptive image matching
trained on the source domain data
trained on the source domain image pairs
Effectiveness of outlier detection
trained without outliers
SiameseDA trained with outliers

OfficeTable 1: Baseline methods for our experiments.
Figure 3: Examples from Shape, Pitts-CycleGAN and Office sets.

5.4. Domain adaptive image matching
Pittsburgh dataset [21] as the source domain, and 157k target domain images (including 52k outliers) generated byapplying CycleGAN [27] to the Pittsburgh images. So thetarget domain images are in a painting style. The outliers
are sky images or city views not containing any useful landmark information.
Office [17] consists of 3 domains, Amazon, Dslr, Webcam. We choose Dslr as source domain and Amazon astarget domain. We make pairs with images from the same
category. The outliers come from two randomly chosen categories (’speaker’, ’scissors’) out of the 31 categories.

5.2. Implementation details
For our triplet network, the three sub-networks share thesame architecture and weights. Pre-trained AlexNet [10] isused for the sub-networks. We finetune the weights of conv4- conv5, fc6, fc7, fc8. For the weighted MK-MMD, we use
a Gaussian kernel with a bandwidth σ given by the medianof the pairwise distances in the training data. To incorporate
the multi-kernel, we vary the bandwidth σm ∈ [2−8 σ, 28 σ]with multiplicative factor of 2 [23]. For performance evaluation, we sort the Euclidean distance between the query andall the gallery features (L2-normalized) to obtain the ranking result. Moreover, we employ the standard metric mean
average precision (MAP).

5.3. Baseline methods
There are no available baselines to directly compare with
our method, thus, we separate our experiments to researchon domain adaptive image matching 5.4 and effectivenessof outlier detection 5.5.
In the experiment on domain adaptive image matching,
we assume no outliers exist in the target domain. Our
method is to jointly learn the contrastive loss L(us ) and
MK-MMD loss M (us , ut ). It is trained with pairs from thesource domain and images from the target domain, we callit SiameseDA.
For evaluating the effectiveness of outlier detection, thetarget domain contains outliers. Our method is called
DA+OutlierDetection, which learns on the objective 10.
The baselines for each experiment are shown in Table 1.

In this section, we assume the target domain does notcontain outliers. We explore if applying domain adaptation
improves the performance of cross domain image matching.
In this case, the learning objective isminu J = L(us ) + γM(us , ut ),(11)

where, the MK-MMD loss term M(us , ut ) is the unweighted version as explained in subsection 3.2.
The MAP results are given in Table 2. Our method consistently outperforms the baselines across all the datasets.
With applying MK-MMD loss for domain adaptation, theperformance of matching S → S decreases comparing tothat of Siamese method. This is within our expectation since
the network may need to learn less from the source domain
to be domain adaptive. Moreover, it is worth to notice that
our method also improves the in-domain image matching(T → T ) of the target domain.

5.5. Effectiveness of outlier detection
Here we assume the target domain contains outliers,
which is to show if the presence of outliers reduces the accuracy of cross domain image matching, and our method
could improve it.
The performance of our method (DA+OutlierDetection),
upper bound (SiameseDA) and lowerbound(SiameseDAOut) are given in Table 3. In terms of
testing, we only take the classified inliers in the query set incalculation. From Table 3 we can see, our method outperforms the lower bound for all the three datasets, but is notbetter than the upper bound (except for Pitts-CycleGAN) as
expected. It shows that the presence of outliers reduces theaccuracy of cross domain image matching, and our method
helps improve the performance in this case.
In Figure 4, we also show the retrieval performance interms of the trade-off between precision and recall at different thresholds on our three datasets. The interpolated average precision is used for the precision-recall curves. We cansee that our method gains over the lower bound method.
Impact of outlier proportion We also report the F1 score to measure the performance of outlier detection ofMethod
SIFT + Fisher Vector
SiameseSiameseDA

T →S

Shape
S→S

T →T

T →S

Office
S→S

T →T

T →S

Pitts-CycleGAN
S→S
T →T

2.5 ± 0.4
8.3 ± 0.1
26.4 ± 0.2

3.6 ± 0.3
95.0 ± 0.2
53.1 ± 0.1

3.4 ± 0.3
31.7 ± 0.6
46.2 ± 0.1

3.5 ± 0.2
10.7 ± 0.5
29.1 ± 0.1

12.0 ± 0.5
99.2 ± 0.2
99.7 ± 0.1

3.5 ± 0.1
77.2 ± 0.3
77.5 ± 0.2

0.04
0.2 ± 0.01
0.4 ± 0.01

0.8 ± 0.05
81.3 ± 0.3
80.4 ± 0.1

0.3 ± 0.03
60.6 ± 0.5
59.5 ± 0.1

Table 2: MAP performance for cross domain image matching and in-domain image matching experiments on three datasets. T means
target domain, S means source domain. T → S implies matching target domain images to source domain images, similar for S → S,
T → T . Our method SiameseDA outperforms the baselines across all the datasets.
1

1
SiameseDA
DA+OutlierDetection
SiameseDAOut

0.8

0.7

0.7

0.6

0.6

Precision

Precision

0.8

0.05
SiameseDA
DA+OutlierDetection
SiameseDAOut

0.9

0.5
0.4

0.04
0.035

0.5
0.4

0.03
0.025
0.02

0.3

0.3

0.2

0.2

0.01

0.1

0.1

0.005

0

0.015

0
0

0.2

0.4

0.6

0.8

1

SiameseDA
DA+OutlierDetection
SiameseDAOut

0.045

Precision

0.9

0
0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

RecallRecallRecall(a) Shape(b) Office(c) Pitts-CycleGAN

0.8

1

Figure 4: Precision-Recall results of our method DA+OutlierDetection, SiameseDA and SiameseDAOut for the experiment of crossdomain image matching with outlier detection on the three datasets. Our method gains over the lower bound method.
Method (T → S)

Shape

OfficePitts-CycleGAN

26.4 ± 0.2
11.9 ± 0.1
5.4 ± 0.1

29.1 ± 0.1
15.9 ± 0.2
6.8 ± 0.1

0.4 ± 0.01
1.1 ± 0.03
0.2 ± 0.01

0.6

SiameseDA
DA+OutlierDetection
SiameseDAOut

0.55
Shape
OfficePitts-CycleGAN

Table 3: MAP performance for cross domain image matching withoutlier detection on our three datasets. The proportion of outliersis 10%. Our method DA+OutlierDetection outperforms the lower
bound, but does not surpass the upper bound.

F1 -Score

0.5

0.45

0.4

0.35

our method. Figure 5 shows the F1 -score of our method
as a function of the portion of outlier samples for the threedatasets. As can be seen, with the increase in the number ofoutliers, our method operates consistently robust.
It is important to notice the limitation of our method,which classifies some inlier samples as outliers during training. This is mainly caused by the way of initializing theprobabilities of the target domain training data.

6. Conclusion
We have proposed a network that is trained for crossdomain image matching with outlier detection in an endto-end manner. The two main parts of our approach are(i) domain adaptive image matching subnetwork with contrastive loss and weighted MK-MMD loss, (ii) outlier de-

0.3
10

20

30
40
Percentage of outliers (%)

50

Figure 5: F1 -scores for outlier detection on three datasets withdifferent outlier proportion in the target domain. Our method is
consistently robust.

tection with entropy loss by updating the probability of target domain data during training. The results on several
datasets demonstrate that the proposed method is capable ofdetecting outlier samples and achieving cross domain image
matching at the same time. But our method still needs improvement to overcome the problem of wrongly classifying
inliers as outliers.

References[1] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv:1802.06360,
2018. 2[2] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity
metric discriminatively, with application to face verification.
In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern, pages 539–546, 2005.
1, 2, 5[3] B. Fernando, T. Tommasi, and T. Tuytelaars. Location recognition over large time lags. Computer Vision and Image Understanding, 139:21–28, 2015. 1[4] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle,F. Laviolette, M. Marchand, and V. Lempitsky. Domainadversarial training of neural networks. J. Mach. Learn. Res.,
17:2096–2030, 2016. 2[5] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, andA. J. Smola. A kernel method for the two-sample-problem.
In Proceedings of the 19th International Conference on Neural Information Processing Systems, pages 513–520, 2006.
2[6] A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan,
M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal
kernel choice for large-scale two-sample tests. In Advancesin Neural Information Processing Systems 25, pages 1205–
1213. Curran Associates, Inc., 2012. 2, 3
[7] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In Proceedings of the2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR ’06, pages 1735–1742,
2006. 2[8] X. Ji, W. Wang, M. Zhang, and Y. Yang. Cross-domain image retrieval with attention modeling. 2017 ACM Multimedia
Conference, 2017. 1[9] B. Kong, J. Supancic, D. Ramanan, and C. C. Fowlkes.
Cross-domain image matching with deep feature maps. International Journal of Computer Vision, 2018. 2[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Proceedings of the 25th International Conference on Neural
Information Processing Systems, pages 1097–1105, 2012. 5[11] T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays. Learning deep
representations for ground-to-aerial geolocalization. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015. 2[12] W. Liu, G. Hua, and J. R. Smith. Unsupervised one-classlearning for automatic outlier removal. In Proceedings ofthe 2014 IEEE Conference on Computer Vision and Pattern
Recognition, pages 3826–3833, 2014. 2[13] M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32Nd International Conference on International
Conference on Machine Learning, pages 97–105, 2015. 1[14] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised
domain adaptation with residual transfer networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 136–144, 2016. 3[15] D. G. Lowe. Object recognition from local scale-invariant
features. In Proceedings of the International Conference on
Computer Vision, ICCV ’99, pages 1150–1157, 1999. 2, 5
[16] M. Sabokrou, M. Khalooei, M. Fathy, and E. Adeli. Adversarially learned one-class classifier for novelty detection.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3379–3388, 2018. 2[17] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In Proceedings ofthe 11th European Conference on Computer Vision: Part IV,
ECCV’10, pages 213–226, 2010. 1, 5
[18] K. Saito, Y. Ushiku, and T. Harada. Asymmetric tri-training
for unsupervised domain adaptation. In Proceedings of the34th International Conference on Machine Learning, volume 70, pages 2988–2997, 2017. 1[19] J. Sánchez, F. Perronnin, T. Mensink, and J. Verbeek. Image
classification with the fisher vector: Theory and practice. Int.
J. Comput. Vision, 105(3):222–245, 2013. 5[20] Y. Tian, C. Chen, and M. Shah. Cross-view image matchingfor geo-localization in urban environments. In CVPR, 2017.
1, 2[21] A. Torii, J. Sivic, T. Pajdla, and M. Okutomi. Visual place
recognition with repetitive structures. In Proceedings ofthe 2013 IEEE Conference on Computer Vision and Pattern
Recognition, pages 883–890, 2013. 5[22] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial
discriminative domain adaptation. 2017 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), pages
2962–2971, 2017. 1[23] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain
adaptation. 2017 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 5385–5394, 2017. 1, 2,
5[24] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Proceedingsof the 27th International Conference on Neural Information
Processing Systems, pages 3320–3328, 2014. 2[25] J. Zhang, Z. Ding, W. Li, and P. Ogunbona. Importanceweighted adversarial nets for partial domain adaptation.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018. 2[26] X. Zhang, F. X. Yu, S. Chang, and S. Wang. Deep transfer network: Unsupervised domain adaptation. arXiv:
1503.00591, 2015. 1, 2
[27] J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer
Vision (ICCV), pages 2242–2251, 2017. 5