A Visual Analysis of the Relationshipbetween Word Concepts andGeographical Locations
ACM International Conference on Imageand Video Retrieval (ACM CIVR)
Island of Santorini, Greece, July 9th, 2009

Keiji Yanai, Hidetoshi Kawakubo and Bingyu Qiu
Department of Computer Science,
The Univ. of Electro-Communications, Tokyo, Japan1

“Sushi” in Caltech 256

 (Probably) Collected in English keywords

2

“Sushi” in our own dataset

 Collected in Japanese keywords

3

Which do you like to eat ?

4

These two “sushi” image sets are surely
different, although both are image sets
associated with the “sushi” concept !

Caltech “sushi” Japanese “sushi”

“Sushi” over the world5

 Image sets corresponding to the same concept

are changing depending on locations or cultures.

Questions about concepts andlocations (or culture) From this observation,

representative
image sets associated with a given
concept might change (slightly)
depending on locations or cultures.
Questions?

 Which concepts are location-dependent ?
 Which concepts are global (unchanged)

over the world ? (e.g. “sea”, “sky”)
 How concepts change depending onlocations ?

“Sea”:

global concept

The Aegean sea

7The Japanese sea

1. Objective,Background& Related workObjective of this paper Analyze the relationship between word

concepts and locations
using geotagged photos on Flickr
 Consist of two parts:
1.

2.

Entropy-based analysis
A system to detect “cultural differences”
They are relatively independent.

9Background: geotagged photos
 The number of geotagged photos on theWeb grows rapidly: Flickr, panoramio Flickr has 100,000,000 geotagged photos.

(Feb. 2009)

A “geo-tag”
represents thecoordinates(latitude,longitude)of a location wherea photo are taken.

Related work: Geotagged image
Many work used geotags to organize
landmark photos.
 Toyama et al. (2003)
 Jaffe et al. (2006)
 Simon et al. (2007) Kennedy et al. (2008)…… and other many works.
The exception is “IM2GPS” [Hayes et al. 2008]
among works on geotagged photos using image analysis.

11Related work: “IM2GPS”[Hayes et al. 08]
 Estimate the probability distribution over theworld by nearest neighbor search for largescale geotagged image DB. (ignoring “concepts”)

 This work suggests there exists the relation

between visual features and geo-locations.

There are many non-landmark
geotagged photos in Flickr !
fishmosquito

tulip

Deutschland

2. [Part 1]
Entropy-based analysis

Entropy-based analysis
Examine the relation between distributionsof visual features and geo-locations
for many concepts
1.

Entropy-based measure of visual features(Modified method of [Yanai and Barnard 05] )2.

Entropy-based measure of geo-locations

3.

Analysis the relation between two kinds ofentropyFor 230 nouns

Image region entropy [Yanai and Barnard 05]
 A measure of “visualness” of words (concepts)
 Represent the property of the distribution ofimage region features
Biased / uneven:
low entropy
having “visualness”

Random/uniform:high entropy
not having “visualness”“Low entropy” means the concept has visual property,“High entropy” means the concept has less visualproperty.

Low entropy: “scary” [Yanai and Barnard 05][Yanai et al. 05]“Visual” concept

Detected“scary”
regions

High entropy: “famous”[Yanai and Barnard 05]“Non-visual” conceptModified image region entropy
 Follow “image region entropy” [Yanai et al. 05]
1.

Use region-based BoF instead of color, texture

JSEG
2.

Randomly-sampled
SIFT-based bag-offeatures(BoF)(1000-dim)

Use mi-SVM to select relevant regions
For excluding
backgroundand noise regions

3.

4.

Model the distribution of region-based BoF
vectors with pLSA instead of GMM
Calculate entropy based on pLSA vectors

25

Multiple Instance Setting Positive bags / Negative bags

positive ins.
(foreground)negative ins.
(background)

Positive instances of “flower”

pseudo-training images

The rest of regions arenegative regions.

random images

mi-SVM[Andrew et al. NIPS 03]

26

 Apply soft-margin SVM iteratively
 Training → classifying → training →classifying → …… (repeat 5 times)
During the iteration, the hyperplane is approaching the optimalplane to discriminate positive
instances from negative ones.
positive ins.
(foreground)negative ins.
(background)Distribution modeling with the 28
PLSA topic mixture
P ( w, d )  P (d ) p ( w | z ) P ( z | d )z

w: visual words, d: regions, z: topic① Apply PLSA for all the regions of all therandom (background) images in advanceObtain P(w | z) and fix it (based distribution)
10,000 random Web images② Estimate P(z | d) for each regions with fixedP(w | z) using fold-in heuristic [Hofmann 09]

Calculate image region entropy H(X): entropy of the given word “X”

 H(X) can be calculated

from each of 5 iterations of mi-SVM

 Regard the minimum H(X) during

5 iterations as the final entropy H(X)Calculate geo-location entropy
Divide the world into 4 kinds of gridswith every 10 degrees by shifting 5
degrees in terms of both latitude and longitude
2. Build histograms regarding the geotagsof the selected regions
Geo-location entropy
3. Calc entropy
4. Select minimum one
1.

3. [Part 1] Results ofentropy-based
analysisExperiments
 Data 230 nouns including various kinds of words• Gathered photos including the given nouns as their tags
 500 geotagged photos at least/ each tagfrom Flickr(limiting 5 photos for each tag per user ID)

 After selecting relevant regions for each tag,calculate the two entropy:
Image region entropy Hvis(X)
Geo-location entropy Hgeo(X) Analyze relation between themImage region entropy Hvis(X)
名詞

H(X)

sun

3.6497

rainbow

4.5538moon

4.6686

mozart7.8349lincoln

7.7327school7.6173

画像の一部

Geo-entropy Hgeo(X)Deutschland

H geo ( X )  0.2602

mosquito

H geo ( X )  5.9759

Rome

H geo ( X )  0.3843

fish

H geo ( X )  5.7831

Image entropy vs. geo-entropy

Sun, rainbow, moon
• Comcepts related to sky– Image region entropy ： low– Geo-location entropy ： high
They exists everywhere in the world,
and the apperances are similar.

sunRome, Deutschland, Mozart
 Image region entropy： high Geo-location entropy ： low

The geotags concentrates
on specific areas. Theirappearances are various.

mozarttulip

Image region entropy： low
Geo-location entropy ： med.

• Variance of color did not reflect
on image region entropy, since weuse SIFT-based BoF representation.
• Holland and England are main areas.

dolphin

Image region entropy： highGeo-location entropy ： high• Most of dolphins are taken in seaor aquarium• In seaside areas over the worldrice

H geo ( X ) = 5.3425

Image region entropy： highGeo-location entropy ： high

4. [Part 2]
Discovering concept
differences in termsof location

Image entropy vs. geo-entropy

We need to analyze more !

Objective of the Utilize
secondpart
the set of geotagged photos on Flickr• A system to mine representative
photos for representative areas orregions from geotagged photo DB.

Raw geo-tagged
photos on Flickr

Relevant photos after
noise image removal

Representative photos
for typical regions

Motivation : Foods over the world
 So with such geotagged photos, we candiscover specific objects over the world.
Do you know all kinds of famous “noodles ” in theworld?
 “Ramen”and“Soba”in Japan, “Thai noodle” in Thailand,“Chinese noodles”, “rice noodle” Taiwan, “Spaghetti”in Italy…
 How do such scenes as “beach”, “waterfall”,

As a“mountain”
result, welook
canlikediscoverculturalin different
areasdifferencesin the world?
on specific concepts over the world !
 How about other objects such as “flower”, “castle”,“clothes” , “car”, “sushi” ……

Approach : three steps
1) Select relevant photos and remove noise Extract BoF vectors from all the images
 Visual clustering with k-means
 Select most relevant clusters based on the size of clusters2) Detect representative regions Clustering based on geographic locations by k-means

3) Generate representative photo sets
for representative regions
 Generate the PLSA topic vectors
 Aggregate photos according to the distribution of mixture

topics and rank photos for each representative area

Contributions
 Detect canonical photos of a specificobject on each place over the world
 Eg.)“noodle”: Chinese noodle in Asia,

spaghetti in Europe Do not limit to only scene of specific placesor landmarks. Any objects are our targets !
 The method is not very novel, but theobjective of the work is very novel.
 Novel application for geotagged photo DB.

5. [Part 2] Results

Experimental Results
 “noodle”, “flower”, “castle”, “waterfall”,“beach”, “car”
 For each concept , collect about 2000 geo-tagged photos
from Flickr distributed evenly in the world wide areas

 Quantitative evaluation for the 1st step Evaluation on our proposed method for extracting the mostrelevant photos
 Precision and Recall
 Color-histogram-based method for comparison

 Examples of regional representative

photosQuantitative Evaluationfor 1st step

Average recall : 73.0%

[Example of results] “noodle”

[Example of results] “noodle”“spaghetti”
photosin andthe inEuropean
Taiwanese
noodles
spicy Thaiarea
noodles
Manystyle“ramen”
photos[Example of results] “flower”

[Example of results] “flower”

Netherlands
national“Tulip”
Kansas Stateflower flower“Sunflower”

[Example of results] “waterfall”“Powerful ” waterfalls
in South America“Beautiful” waterfalls
in Asia

60

Wedding cake !

6. Conclusions

Conclusions
 In this paper, we pointed out that image

sets associated with the same concepts
are variable depending on locations
 To analysis that, we proposed to use
geotagged photos on the Web.
1.

Entropy-based analysis•

2.

Image region entropy and geo-location entropy

A system to help detect “culturaldifferences” by selecting representative
photos for each location

Future work
 Use cross-language query to getimages
 Actually the obtained results are biased by

English-culture, because we used only
English words when gathering geotagged
images from Flickr.
 Discover (subtle) cultural differences

automatically (hopefully hard for human) Extensive study on cultural differencesusing a large-scale geotagged photo DB63Towards location(culture)specific image recognition
 This analysis will help build location(culture)-specific object/scene
recognition systems.
 For location-dependent concepts, specifictraining image sets are needed, while for
global concepts global image sets are OK.
• To build systems for Japanese people,
a special training set for “Suishi” is needed.
• For people in Santorini, specific image sets for“house” and “building” might be needed.

64Thank you
for your attention !

66

3. Methods