DEIM Forum 2019 F6-2

Discriminative Objects Discovery for Spot Assessment
陳

天偉†

馬

強†

† School of Informatics, Kyoto University
Yoshida-honmachi, Sakyo-ku, Kyoto, 606–8232, Japan
E-mail: †chentianwei@db.soc.i.kyoto-u.ac.jp, ††qiang@i.kyoto-u.ac.jp
Abstract Discovering discriminative objects that has local characteristics is an important task for sightseeing spot
discovery and assessment. In previous work, discriminative objects are detected in the early stage of whole process
and thus can hardly get discriminative objects when target spots change. Such inflexibility makes it hard to discoverreasonable discriminative objects and will cause a large computational cost for further process. In this paper, we
propose a new framework of discriminative objects discovery. It detects common objects from each sightseeing
spot, ang then generates hierarchical clusters of these objects. Our framework leaves the process of discovering
discriminative objects at the last stage when we want to compare several target spots. When a new requirement of
discover discriminative objects comes, the only thing the proposed framework should do is to compare data fromtwo tree-like clusters and find what the discriminative objects are.
Key words Sightseeing Spot Assessment，Discriminative Object Discovery，Unsupervised Object Detection

1 Introduction

and can get much more reasonable outputs.
But since the processing of finding discriminative elements

Beautiful sceneries, amazing events and memorable experi-of the previous methods is conducted in the early stage, theence, travel is such an attractive activity that draws people

whole framework become inflexible to adapt new data. E.g.

s attention and comes to an important part of people s daily

Those discriminative objects can hardly represent the dif-life. However, there are always the problems like which spots

ference between Kyoto and Nara, but if we add Nara as aare worth to go and why they are of worth, which confusecandidate city to find discriminative objects, we have to run

people who want to have a travel.

the whole process again. It is not the thing we expect, as

Sightseeing spot assessment is the research topic focusing

it is impossible to collect all sightseeing spots around theon this problem, which is to give an assessment to sightsee-

world and send them in one training. In short, it is of time-

ing spots. Current methods of sightseeing spot assessment

consuming and laborious to retrain such a complex model[1, 2, 7, 8, 18, 21] are focusing on assessment image data fromwhen new data come.

spots, which can reflect sightseeing spot s quality directly.

Also, we consider that the discriminability in object-level

However, the utilization of images shared by users is a chal-cannot represent a spot s characteristics and internal uni-

lenging task, as many factors like the quality of the image

versality well. E.g. the building in Kyoto and Nara is very

and the relevance between images and spots could aﬀect thesimilar in the shape, which makes them hard to distinguishresult deeply.

in object-level. However, if we compare them more care-

One reasonable solution of utilizing image data to assess-fully, we may find that most of them are in diﬀerent style,ment sightseeing spot is to discover discriminative elements

which can distinguish them more precisely. Thus, a tinier(e.g. patches [7] or objects [8]) from those images. The corescale should be considered to represent the discriminability

idea is that by comparing images from diﬀerent spots, we canof each spot.

discover some elements that appears much in a specified spot

In this paper, we propose a new framework of discrimi-

and do not appear in others. We thus called them discrimi-

native objects discovery from sightseeing spots to solve thenative elements. By observation, those elements are of highproblem above. It discovers objects from geo-tagged images

local characteristics, which reveal the spot s uniqueness andin an unsupervised manner and avoids finding discrimina-

could be a clue to evaluate the worth of sightseeing spot.

tive objects in the early steps. To represent a spot s char-Also, since the process only concerns about some valuable

acteristics well, we divide the objects into several patches,

subpart of images and remove common elements by compar-

which can represent more discriminative details and is more

ing target spots, this idea has high robustness against noise

human-interpretable. We sampling these millions of patches

and try to reserve those key ones as much as possible. Fur-ing quality, and can apply to estimate obscure spots.

thermore, without the limitation of extract discriminative

2 3 Discriminative sightseeing elements discovery

elements from candidates in the first step, we can collect

Discovering discriminative sightseeing elements is aand organize more data for further use.

2 Related works

subtopic of tourism analysis, which aims to discover elements
in one spot that make the spot distinguishable to other spots.
In this topic, Doersch et al.[7] proposed a method to dis-

Our work is mainly about sightseeing spot assessment andcover discriminative patches of Paris from Google Street

based on the idea of extract discriminative elements from

View. By iteratively comparing Paris image data and otherimages by utilizing unsupervised object discovery method.

cities image data, their model output several patches whichThus, we would introduce those related works before ourappear in Paris and do not appear in other candidate cities,

proposed method.

thus called the things

2 1 Object detection

make Paris look like Paris .

Following this idea of discovering discriminative elements,

Object detection is a fundamental and challenging research

Ge et al.[8] proposed a robust visual object clustering method

topic which may bring benefits to several related topics andand apply it to sightseeing spot assessment. In this paper,

applications. In recent year, object detection methods based

they use the manner of iteratively comparing images fromon CNN made a great breakthrough in this research task [9,Kyoto and other four cities, then return the discriminative

10, 11]. Those methods utilized large amount of labeled im-

objects of Kyoto. The result of discriminative objects include

age data to train a CNN based model in supervised manner,

temples, towers, traditional houses and maple trees, which

which make the model have the capacity to proposed regional

depict characteristics of sightseeing spots in Kyoto.

bounding box to new data. However, due to the supervised

These methods have shown a good performance. In thistraining manner, the model usually can only detect objects

research topic, Ge et al. [8] proposed a discriminative ob-that given in the training set. As a matter of fact, in manyjects discovery framework to discover local characteristics inresearch and application scenario, the data with labeled re-

Kyoto by iteratively comparing Kyoto and other cities. Al-gion and object are not so easy to get, which make themthough the data has lot of noise, the result still clearly shows

hard to share the benefits from recent breakthrough. Therea lot of Kyoto s local characteristics.

are also some research topics focusing on this problem andtry to propose solutions, such as Weak Supervised Object
Detection [12, 13] and Zero-shot Object Detection [14, 15].

3 Methodology
There are two core ideas in our work.

One idea is to

However, the methods mentioned above still aims to detect

avoid discovering discriminative objects in the early step,

limited categories of objects and more or less rely on humanwhich makes the framework more flexible when target spots

annotation.

changing. Another idea is to divide objects into patches andAs our task aims to discover unlimited categories of ob-

discover discriminative patches to represent objects. This

ject and only consider if the object is discriminative, we pre-method can reserve more details of one spot than use object

fer unsupervised object detection methods to get objects we

directly, and can calculate the discriminability in a more rea-need. In this research topic, Tang et al. [16] proposed a joint

sonable and human-interpretable way.

image-box formulation to discover unlabeled objects. Cho et

Based on these two ideas, we propose a new framework

al. [17] apply a probabilistic Hough matching method to deal

which use an unsupervised method to discover candidate ob-with the task of discover and localize objects.

jects for each spot, divide objects to patches, extract features

2 2 Sightseeing spot assessment

and use clustering method to find the common elements from

Sightseeing spot assessment is the task to give a reason-

patches to represent objects and spots. When the require-able evaluation score to sightseeing spot. This task is highly

ment of find discriminative objects comes, we then start torelated to the research of POI (place of interest) discovery

compare target spots to find the discriminative objects. The[3, 4] and recommendation [3, 4, 5] as the assessment could

framework and data flow are showed in Figure 1.

be a kind of clue to discover POIs. However, sightseeing spot

There are mainly two parts in our framework, one is mianly

assessment only focuses on evaluate sightseeing spots while

about preparing common elements (objects and patches), an-

POI discovery not only considers about sightseeing spot, butother one is to discover discriminative elements. In the firstalso hotels, restaurants and other spots.

part, images from each city are processed into patches that

Currently, many methods [3, 8, 18] are focusing on esti-

are common in that city. In the second part, these patches

mate sightseeing spots by geo-tagged images directly, which

will be winnowed down to a set of patches that with highavoid aﬀects from the gap between popularity and sightsee-

discriminability to a city. Then, the discriminative patches

Figure 1 Framework & data flow. The brown box represent what kind of data they are,and the green box represent the processes we do.

are used to train a model to find the most discriminative

tering more reasonable. After that, we cluster all candidates

objects in one city.

in one image to limite that each image can only output 3

To be noticed that as we don t extract discriminative ob-candidate objects, which is from the top-3 largest clusters.

jects in the first step, the process logic changes and eachBy this process, we make the output candidate objects morecomponent are defined with diﬀerent methods. Thus, theclear. Then, we use Kmeans again to cluster the candidates

whole framework is diﬀerent from previous works.

which may still depict same object while from diﬀerent im-

3 1 Region Proposal & Common Object Clusteringages. From each cluster, we select only one as a representative of this cluster. The volume of candidate object thus

According to our task definition, the discriminative objects

reduce slightly. After the process above, we treat these ob-

can be everything that make the candidate distinguishable,

jects as common objects which appear requently in one city.

i.e. we cannot apply any prior knowledges such as which

3 2 Patch Generation & Sampling

categories of object might be discriminative on while which

By the view that the discriminability in object-level di-

might not. Thus, we first apply a well-known unsupervised

rectly cannot represent a spot s characteristics and internal

region proposal method [19] to propose every possible object

universality well, we divide the objects into several patchesfrom geo-tagged images.

to reserve more details for later processing. Thus, we use a

To improve the quality of proposed region and for further

sliding window with the size of [32 × 32], and set the strid touse, we only allow the region that larger than [64 × 64] but

8, to divide each object into several patches. To be noticed

smaller than 90% pixels of the whole image to be valid andthat we orgatized the patches by object index to reserve theonly select the largest one if several regions overlap to eachbelonging information for discovering discriminative objects,

other. After the process, we simply treat those regions as

and thus in Figure 1, the patches from one object are blockedcandidate objects.

in one box.

By observation, there are a lot of candidates don’t catch

In this process, millions of patches are generated from com-

object, or only catch a part of an object. Thus, we use

mon objects and make it almost impossible to do any process

Kmeans algorithm to cluster those candidate objects. Morelater. Thus, we randomly sample a reasonable number ofdetailed, we use a pre-trained VGG-19 [20] model to extract

patches to represent a city. In this paper, we sample about

features from each patch. In the extraction process, we con-2000 patches for each city.

vert all the candidates into [64 × 64] shape and get features

Here, we use a pre-trained VGG-19 [20] model to extractfrom VGG-19 s last pooling layer, which still retain thefeatures from each patch. The size of VGG-19 s output islocation information for each object and thus make the clus-

512, we treat each of them as a vector which represent a

3 4 Discriminative Objects Discovery

patch.
3 3 Discriminative Patch Discovery

In last process, we get a large amount of patches which areWith the clusters generated in last process, get a lot of

discriminative and all comes from one city:s object. However,patches which come from common objects. However, thoseeither retrieve objects by patches or combine patches to ob-patches can only represent the things appears frequently injects is almost impossible, as we only randomly sample a few

one city, but are not discriminative, as they may also ap-of the patches from millions of them. Here, our idea to dis-

pears a lot in other city. Thus, we try to compare themcover discriminative objects is to train a new SVM model aswith patches from other cities and to find those discrimina-

a detector of object’s discriminability.

tive patches from them. The whole process are showed inMore concretely, we set those discriminative patches fromone city as positive set, and set those discriminative patches

Algorithm 1.

belong to other cities as negative set. Then, we initial a new
Algorithm 1
Require:Positive set P,

SVM model and use the data above to train it. After training, we load patches which are generated in section 3 2 as
Negative set N;

test data of the SVM model. Those patches are organisedby object index and thus we can easily find which object the

Begin:

patch belongs to.

1 : P → {P 1, P 2};
2 : Ptrain ← P 1;

In this processing, we treat those patches with the posibil-

Pnext ← P 2;

3 : W hile not converged() do4:

M odel ← SV Mtrain ([Ptrain , N ]);5:

Pnew ← select(M odel, Pnext , > γ);

6:swap(Pnew , Pnext );init(M odel);ity higher than 50% as discriminative patches and calculate
the radio of those patches in one object to detect discriminative object. In this paper, if more than half patches in
one object are discriminative, we then treat this object as adiscriminative one. The calculation of discriminative radioin object On is as follow:

7 : Endwhile
8 : Return [Pnew , Pnext ]On = Pp /Pa ll
Where Pp represents the number of discriminative patches
and Pall represents the number of patches belong to object

There are two steps in this process. At first, we treat thepatches from target city as positive set P , while those patchesfrom other cities are treated as negative set N . Then, we di-On .

4 Experimentvide the positive set into two equal, non-overlapping subset

4 1 Dataset

P1 and P2 for cross-validation.

To verify the utility of our method and compare the perfor-

Given a positive subset P1 , we concatenate P1 with neg-mance with previous work, we first use the dataset provided

ative set N as training set, and the data from positive set

by [8], which consisted of photos of sightseeing spots from fiveare assigned 1 and those negatives are assigned −1. We thencities: Kyoto, Xi an, Beijing, Paris and San Francisco. The

initialize and train a SVM model on this training set. Whendataset collected from approximate 30 most popular sightsee-

the training finished, we send other positive set P2 as test

ing spots from TripAdvisor for each city and 50-100 images

set and let the model to give the probability of the object

are downloaded per spot from Flickr s API. The final dataset

belongs to positive set. Here we set a threshold parameter γ

for each city are formed by randomly select 1000 image from. If an object s probability is smaller than γ, we treat the

downloaded images.

object is not enough discriminative and discard it in later
processing.
After the process, we get a purer positive set P2 . Then,

Additionally, for showing our framework s flexibility, we
collect data from a new city, Nara, in the manner of original
dataset did.

we swap P1 and P2 , concatenate P2 with negative set N as

4 2 Result & Evaluationa new training set and use it to winnow P1 . This process goesTo evaluate the performance of our framework, we are fo-

iteratively until convergence, i.e. the total number of posi-

cusing on three main part: the usability, the accuracy andtive objects doesn t change anymore. At the time iterative

the flexibility. These three part can derive to three questions:process ends, we can get the discriminative patches.

1). Does the model find the discriminative objects?
2). Can we quantitively show the accuracy?

3). Does the model flexibility?

Beijing} and discovers the discriminative objects that ap-

In the later parts of this section, we do the evaluation

pears a lot in Kyoto while not in Beijing. While a tourist
come from Paris then comes to Kyoto, the framework have tobased on these three question.
4 2. 1 The disxriminability of objects

run again to find discriminative objects by the pair {Kyoto,During this part, to make an evaluation of discriminability

Paris}. And even a person who has been to Beijing andof objects discovered by our framwork, we randomly output

Paris (as the pair {Kyoto, [Beijing, Paris]}) comes, the whole

10 discriminative objects of them, which have a discrimina-

framework have to run again to discover discriminative ob-tive radio higher than 0.95. Figure 2 shows the 10 discrim-

jects.

inative objects from Kyoto and the other cities are Beijing,

Assume there will be S groups of tourist with diﬀerent
background (experiments of other cities) come to Kyoto andParis, San Francisco, Xi’an and Nara.

K spots are overlapped. Then, we have to run the wholeframework S times to get all result.
In our framework, we first make the process of finding common objects and patches save them for later process. When aquery (e.g. pair {Kyoto, Beijing}, {Kyoto, Paris} or {Kyoto,[Beijing, Paris]}) comes, we then find the discriminative objects. Assume we face the situation the same as above, ourFigure 2 10 high discriminative radio objects from Kyoto

framework only need to do the time consuming process of

The result shows some objects that we can usually see in

our model are more flexible than Ge et al.[8]’s.

preparing common objects and patches S − K times. Thus,
Kyoto, and most of them are discriminable if comparing with

5 Conclusion

images from other cities.
4 2. 2 The qualify of discriminability

In this paper, we propose a new discriminative object dis-As giving quantitive evaluation to discrimiability is a dif-

covery framework which solve two problem of previous works.

ficult, we set the experiment as to test a new unseen dataset

The first one is about the inflexibility of previous works duewith 2000 patches, which are 50% from target city and 50%

to the discriminative element discovery processing. The sec-from other cities.

ond one is during the process, we try several methods to re-

To make a comparation of Doersch et al.[7] , we implement
their key component of discovering discriminative patches induce the cost while avoid discovering discriminative objects
in the early step.

Python and run our collected dataset throw the component.

However, current discriminative objects are still hard to

Since Ge et al.[8] simpliy apply the model in Doersch etshow a good performance, as some of them may also appears

al.[7], we treat that they use the same component to dis-a lot in other cities (e.g. pine tree also appears a lot in Bei-cover the discriminative patches and only compare the AUC

jing and Nara). The problem may caused by the volume ofresult with Doersch et al.[7]’s work. The result are as follow:

the data we collect and the sampling strateegies we used,

Table 1 Result of evaluating patches’ discriminability (AUC)

which may lose some key discriminative information before
we train the model. Thus, in the later work, we may collectCity

Ours Doersch’s

Kyoto

0.772

0.816

Xi an

0.719

0.798

Beijing

0.662

0.647

Paris

0.854

0.954

San Francisco 0.697

0.696

References

0.798[1] Chen, Wei-Chao, Agathe Battestini, Natasha Gelfand andVidya Setlur, “Visual summaries of popular landmarks from
community photo collections,” Conference Record of the
Forty-Third Asilomar Conference on Signals, Systems andComputers, 1248-1255, 2009.
[2] Berman, Marc G., Michael C. Hout, Elizabeth A Necka,
MaryCarol R. Hunter, Grigori Yourganov, John M. Henderson, Taylor Hanayik, Hossein Karimi and John Jonides,“The Perception of Naturalness Correlates with Low-Level
Visual Features of Environmental Scenes,” PloS one, 2014.
[3] Hasegawa, Keisuke, Qiang Ma and Masatoshi Yoshikawa,

Nara

0.753

4 2. 3 The analysis of flexibility
In Ge et al.[8]’s work, the process of discovering discriminative objects starts as the first step, which makes the wholeframework and its intermedia outputs can hardly be used forcomparing other candidates. E.g. If a tourist from Beijing
comes Kyoto, the framework will process the pair {Kyoto,

more data from each spot and try other sampling strategies
that may reserve more discriminative information.
This work is partly supported by MIC SCOPE(172307001).

[4]

[5]

[6]

[7]

[8]

[9]

[10]
[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

“Trip Tweets Search by Considering Spatio-temporal Continuity of User Behavior,” DEXA, 2012.
Zhuang, Chenyi, Qiang Ma, Xuefeng Liang and Masatoshi
Yoshikawa, “Anaba: An obscure sightseeing spots discovering system,” 2014 IEEE International Conference on Multimedia and Expo (ICME), 1-6, 2014.
Zhao, Pengpeng, Xiefeng Xu, Yanchi Liu, Victor S. Sheng,
Kai Zheng and Hui Xiong, “Photo2Trip: Exploiting VisualContents in Geo-tagged Photos for Personalized Tour Recommendation,” ACM Multimedia, 2017.
Lim, Kwan Hui, Jeﬀrey Chan, Christopher Leckie andShanika Karunasekera, “Personalized Tour Recommendation Based on User Interests and Points of Interest Visit
Durations,” IJCAI, 2015.
Doersch, Carl, Saurabh Singh, Abhinav Gupta, Josef Sivic
and Alexei A, “What makes Paris look like Paris?,” ACM
Trans. Graph. 31, 101:1-101:9, 2012
Min Ge, Chenyi Zhuang, Qiang Ma, “A Ranking Based Approach for Robust Object Discovery from Images of Mixed
Classes,” Asia Information Retrieval Societies (AIRS), 7183, 2017.
Girshick, Ross B., Jeﬀ Donahue, Trevor Darrell and Jitendra Malik, “Rich Feature Hierarchies for Accurate Object
Detection and Semantic Segmentation,” 2014 IEEE Conference on Computer Vision and Pattern Recognition, 580-587,
2014.
Girshick, Ross B., “Fast R-CNN,” 2015 IEEE International
Conference on Computer Vision (ICCV), 1440-1448, 2015.
Ren, Shaoqing, Kaiming He, Ross B. Girshick and Jian Sun,
“Faster R-CNN: Towards Real-Time Object Detection with
Region Proposal Networks,” IEEE Transactions on PatternAnalysis and Machine Intelligence 39, 1137-1149, 2015.
Dong, Xuanyi, Liang Zheng, Fan Ma, Yi Yang and Deyu
Meng, “Few-Example Object Detection with Model Communication,” IEEE transactions on pattern analysis andmachine intelligence, 2018.
Teh, Eu Wern, Mrigank Rochan and Yang Wang, “Attention Networks for Weakly Supervised Object Localization,”
BMVC, 2016.
Bansal, Ankan, Karan Sikka, Gaurav Sharma, Rama Chellappa and Ajay Divakaran, “Zero-Shot Object Detection,”
ECCV, 2018.
Rahman, Shafin, Salman Hameed Khan and Fatih Murat
Porikli, “Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts,” CoRR
abs/1803.06049, 2018.
Tang, Kevin D., Armand Joulin, Li-Jia Li and Li Fei-Fei,“Co-localization in Real-World Images,” 2014 IEEE Conference on Computer Vision and Pattern Recognition, 14641471, 2014.
Cho, Minsu, Suha Kwak, Cordelia Schmid and Jean Ponce,“Unsupervised object discovery and localization in the wild:
Part-based matching with bottom-up region proposals,”2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 1201-1210, 2015.
Shen, Yizhu, Min Ge, Chenyi Zhuang and Qiang Ma,“Sightseeing Value Estimation by Analyzing Geosocial Images,” 2016 IEEE Second International Conference on Multimedia Big Data (BigMM), 117-124, 2016.
Uijlings, Jasper R. R., Koen E. A. van de Sande, Theo Gevers and Arnold W. M. Smeulders, “Selective Search for Object Recognition,” International Journal of Computer Vision 104, 154-171, 2013.
Simonyan, Karen and Andrew Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,”
CoRR abs/1409.1556, 2014.
Shen, Yizhu, Chenyi Zhuang and Qiang Ma, “ElementOriented Method of Assessing Landscape of Sightseeing

Spots by Using Social Images,” APWeb/WAIM, 2017.