Mining Cultural Differencesfrom a Large Number ofGeotagged Photos

1World Wide Web 2009

0011 0010 1010 1101 0001 0100 1011

24

Keiji Yanai and Bingyu Qiu

Department of Computer Science,
The Univ. of Electro-Communications, Tokyo, Japan1

2

Background: geotagged photos
0011 0010 1010 1101 0001 0100 1011

• The number of geo-tagged photos on
the Web grows rapidly: Flickr, panoramio– Flickr has 40,000,000 geotagged photos.

1

2(in the beginning of 2008)

4

A “geo-tag”
represents thecoordinates(latitude,longitude)of a location wherea photo are taken.

3

Motivation : Objects over the World• So with such geotagged photos, we like todiscover specific objects over the world.

0011 0010 1010 1101 0001 0100 1011

 Do you know all kinds of famous “noodles ” in theworld?

1

2

 “Ramen”and“Soba”in Japan, “Thai noodle” in Thailand,“Chinese noodles”, “rice noodle” Taiwan, “Spaghetti”in Italy…

4 How do such scenes as “beach”, “waterfall”,“mountain” look like in different areas in the world?
 How about other objects such as “flower”, “castle”…

As a result, we can discover cultural differenceson specific concepts over the world !

4

Objective of our work
0011 0010 1010 1101 0001 0100 1011

• To mine representative photos forrepresentative areas from geotagged
photos related to generic concepts.

12

noodle, wedding cake, waterfall, car, castle, beach, flower

Raw geo-tagged
photos on Flickr

Relevant photos after
noise image removal

4

Representative photos
for typical regions

5

Related Work
• Automatic selection of landmarks.

0011 0010 1010 1101 0001 0100 1011

Lyndon Kennedy and Mor Naaman: Generating Diverse and Representative Image
Search Results for Landmarks, ACM WWW2008, pp.297-306, (2008).

124Till Quack, Bastian Leibe and Luc Van Gool: World-scale Mining of Objects
and Events from Community Photo Collections, ACM CIVR, pp.47-56, (2008).

They treated with only landmark photos, not generic photos

6

Approach : three steps
1) Select relevant photos and remove noise

0011 0010 1010 1101 0001 0100 1011

 Generate image feature vectors
 Visual clustering Select most relevant clusters

12) Detect representative regions Clustering based on geographic locations243) Generate representative photo sets
for representative regions

 Generate the PLSA topic vectors
 Aggregate photos according to the distribution of mixture
topics7[Image representation]
bag-of-visual-words [csu04]

0011 0010 1010 1101 0001 0100 1011

• Represent an image by a set of local image
features2.
3.

2

Sample keypoints randomly andcompute SIFT descriptor over each keypoint
Vector-quantize SIFT vectors to form visual words(codebook generation)
Build histograms by mapping each feature vector to certain
visual words − regarded as image feature vectors

1frequency

1.

4

codewords (visual words)

8

Approach: (1) noise removal
1) Select relevant photos and remove noise

0011 0010 1010 1101 0001 0100 1011

 Generate image feature vectors
 Visual clustering
k-means clustering over bag-of-visual-words vectors.

12(In the experiments, we set k to 200 for 2000 images.) Select most relevant clusters

4

Evaluate the intra-cluster similarity.

Discard the clusters with small members (m<10)
or small similarity. (select 40 clusters)

9

Approach: (2) location clustering2) Detect representative regions

0011 0010 1010 1101 0001 0100 1011

 Clustering based on geographic locations
K-means clustering over location vectors :(longtude, latitude) (k=5)

124

Approach: (3) select regional

10

representative photos

3) Generate representative photo sets
for representative regions

0011 0010 1010 1101 0001 0100 1011

 Generate the PLSA topic vectors (vector compression) Probabilistic Latent Semantic Analysis (PLSA)

12

P(word , image)  P(image)  p(word | topic) P(topic | image)
topica method to compress a vector to extract “topic” elements like Latent Semantic Analysis

4 Aggregate photos according to the distribution ofmixture topics
K-means clustering over
PLSA topic vectors (k=5) for each region .
Select the largest cluster as a final result.

P(topic|image)
確率的Example
トピック
of a PLSA 0.15
0.240011 0010 1010 1101 0001 0100
1011
result• Mountain

0.03

10 topics

0.05
0.10
0.01

probabilistic
softclustering
110.01
0.01
0.05

0.30

124

12Experimental Results0011 0010 1010 1101 0001 0100 1011• “noodle”,“flower”, “castle”, “waterfall”,“beach”
 For each concept , collect about 2000 geo-tagged photos
from Flickr distributed evenly in the world wide areas

12• Quantitative evaluation for the 1st step4

 Evaluation on our proposed method for extracting the mostrelevant photos
 Precision and Recall
 Color-histogram-based method for comparison• Examples of regional representative
photos13

Quantitative Evaluationfor (1) noise removal• raw vs. color-based vs. BoVW (proposed)

0011 0010 1010 1101 0001 0100 1011

1PrecisionRecall

24

14

[Example of results] “noodle”
0011 0010 1010 1101 0001 0100 1011

124150011 0010 1010 1101 0001 0100 1011

124

16

[Example of results] “flower”
0011 0010 1010 1101 0001 0100 1011

124

17

[Example of results] “waterfall”
0011 0010 1010 1101 0001 0100 1011

“Powerful ” waterfalls
in South America

1“Beautiful” waterfalls
in Asia24

180011 0010 1010 1101 0001 0100 1011

124

19Conclusion
0011 0010 1010 1101 0001 0100 1011

We have proposed a novel topic which isto discover canonical photos of thespecific objects over the world.

1

2

The results help discover cultural differences.

4

For more results, please access:
http://mm.cs.uec.ac.jp/qiu-b/ASRP/