A Visual Analysis of the Relationship between Word
Concepts and Geographical Locations
Keiji Yanai

Hidetoshi Kawakubo

Bingyu Qiu

The University of Electro-Communications
Chofu, Tokyo 182-8585 JAPANBeijing University of Posts and Technology
Beijing, 100876, China{yanai,kawaku-h}@mm.cs.uec.ac.jp

bigyuqiu@bupt.cn

ABSTRACTof consumer-generated photos people uploaded, and make themaccessible via the Web for everyone. Photo sharing sites collectmetadata in addition to photos. While keywords and comments arecommon as metadata, recently some users attach “geo-tags” to their
uploaded photos. A “geo-tag” means metadata which represents alocation where the corresponding photo was taken, which is usually
expressed by a set of a latitude and a longitude.
An accurate geotag can be obtained with a GPS device or alocation-aware camera-phone. However, since it forces us to use
relatively special devices, GPS-based geotags have not been common so far. Instead, map-based geotags have become common,
after Flickr, which is the largest photo sharing site in the world,
launched an online geotagging interface in 2006. Then, Flickr also
became the largest “geotagged” photo database in the world. According to [11], there are currently over 40,000,000 public geotagged photos on Flickr, and 100,000 geotagged photos have been
added every month. These geotagged photos would be valuable
not only for browsing and ﬁnding individual concepts, but also for
helping us understand how speciﬁc objects or scenes are distributedand different over the world.
Our objective is exploring the relationship between word concepts and geographical locations by using a large amount of geotagged images on the photo sharing Web sites such as Flickr.
Firstly, we propose using both image region entropy and geolocation entropy to analyze relations between location and visualfeatures, and in the experiment we found that concepts with low
image entropy tends to have high geo-location entropy and viceversa.
Secondly, we propose a novel method to select representative
photographs for regions in the worldwide dimensions, which helps
detect cultural differences over the world regarding word concepts
with high geo-location entropy. In the proposed method, at ﬁrst, we
extracts the most relevant images by clustering and evaluation on
the visual features. Then, based on geographic information of theimages, representative regions are automatically detected. Finally,
we select and generate a set of representative images for the representative regions by employing the Probabilistic Latent Semantic
Analysis (PLSA) modelling. The results show the ability of ourapproach to mine regional representative photographs and culturaldifferences over the world.

In this paper, we describe two methods to analyze the relationshipbetween word concepts and geographical locations by using a large
amount of geotagged images on the photo sharing Web sites suchas Flickr.
Firstly, we propose using both image region entropy and geolocation entropy to analyze relations between location and visualfeatures, and in the experiment we found that concepts with low
image entropy tends to have high geo-location entropy and viceversa.
Secondly, we propose a novel method to select representative
photographs for regions in the worldwide dimensions, which helps
detect cultural differences over the world regarding word concepts
with high geo-location entropy. In the proposed method, at ﬁrst, we
extracts the most relevant images by clustering and evaluation on
the visual features. Then, based on geographic information of theimages, representative regions are automatically detected. Finally,
we select and generate a set of representative images for the representative regions by employing the Probabilistic Latent Semantic
Analysis (PLSA) modelling. The results show the ability of ourapproach to mine regional representative photographs and culturaldifferences over the world.

Categories and Subject Descriptors
H.3.3 [Information Search and Retrieval]: Miscellaneous

General Terms
Algorithm, Experimentation

Keywords
geotag, entropy, representative image, Flickr

1. INTRODUCTION
In these days, there are a great number of images on the Web.
Especially, photo sharing sites such as Flickr and Picassa are gathering a large number of photo images. They store a huge number

2.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee.
CIVR ’09, July 8-10, 2009 Santorini, Greece.
Copyright 2009 ACM 978-1-60558-480-5/09/07 ...$5.00.

RELATED WORK

Until several years ago, researches on geotagged images focusedon only location-based photo browsing for a personal geotagged
photo collection [21, 15], since it is almost impossible to obtaina large number of geotagged images. However, the situation has
been changed after Flickr launched an online geotagging interface
in 2006. At the present, Flickr has become the largest geotagged1

photo database in the world. Geotagging with GPS devices is too
expensive to spread, but Flickr online geotagging system allows
users to indicate the place where photos are taken by clicking theonline map. In addition, the Flickr database is open to everyone via
FlickrAPI which allows users’ program to search the whole Flickr
photo database for geotagged images.
Therefore, some works on geotagged image recognition withhuge Flickr geotagged image database has been proposed recently.
[4] and [3] proposed methods on event recognition of geotagged
images by integrating visual features and geographical information. In general, a geotag represents a pair of values on latitude
and longitude. It is a just 2-dimensional vector. To convert a 2-d
vector into more rich representation, [14] and [22] converted geotags into visual information from the sky using aerial images, and[10] transformed geotags to words using reverse geo-coding technique. On the other hand, [26] used GPS trace data which is aseries of geotags instead of using just a pinpoint geotag in order toclassify images into several pre-deﬁned events. [25] used time andseasons for geotagged image recognition in addition to visual information and geo-location data. While event or scene recognition
on geotagged images is common, “IM2GPS” project [7] proposeda unique idea of estimating a place from just one non-geotagged
image with 6 million geotagged images gathered from Flickr.
As extension of location-based photo browsing, several recent
researches have considered the problem of selecting representative
or canonical photographs for online image collections. Jaffe et al.
[9] select a summary set of photos from a large collection of geotagged photographs based on only tags and geo-tags. By analyzing
the correlations between tags and geo-tags, a map-based visualization “Tag Map” is developed to help indicate the most importantregions and the concepts represented in those regions. Our work
similarly identiﬁes the most important regions and select representative photos for these regions. A key difference is that in [9], theconcepts are learn which could be mostly affected by users’ photographic behavior. While in our work, we aim to select representative photographs for particular concepts by applying computer
vision techniques. Simon et al. [20] have proposed a method to select canonical views for the landmarks by clustering images basedon the visual similarity between two views. Like [20], Kennedy
et al. [11] attempt to generate representative views for the world’s
landmarks based on the clustering and on the generated link structure. Unlike the works [20] and [11], we choose the general category objects or scenes as our target, but not the identical objects
like landmarks which rely on 3D structure or viewpoint. One ofthe other similar works is the work by Raguram et al. [19]. They
aimed to select iconic images to summarize general visual categories, like “love”, “beauty”, “closeup” and “apple”. Since generalvisual or abstract concepts usually have many semantic “themes”,
their canonical view selection is hence deﬁned as select a small
number of salient images for each semantic “theme”. Our goal isdifferent from theirs in terms of selecting representative photos foreach geographic region. [18] treated with generic concepts like ourwork. However, we select canonical images on generic concepts
regarding several regions in the worldwide dimension, while [18]treated with general concepts within only given regions.
Regarding entropy analysis for relations between word concepts
and locations, as long as we know, there exists no similar works
so far. To analyze the distribution of visual features regarding the
given concepts, Yanai et al.[24] proposed “region image entropy”
to measure visualness of word concepts with images gathered fromthe Web. They claimed that word concepts with low entropy are
more appropriate for automatic image annotation [2]. Koskela etal.[12] used entropy to analysis the large-scale multimedia ontology, LSCOM [16]. In this paper, we use the modiﬁed “region image entropy”[24] which is different from the original methods inimage features and learning methods.

3.

ENTROPY ANALYSIS

In this section, we propose a new method to analyze relations between location and concepts in terms of image features. We compute both image region entropy [23] and geo-location entropy formany concepts using geotagged images gathered from the Flickr.

3.1

Image Region Entropy

Originally, “Image Region Entropy” was proposed in [23], which
is a measure of “visualness” of concepts, that is, what extent concepts have visual characteristics. In the original method to compute “image region entropy”, they perform probabilistic region selection for regions that can be associated with concept “X” fromimages which are labeled as “X” or “non-X”, and then they compute a measure of the entropy of the selected regions based on a
Gaussian mixture model for regions. By introducing a probabilistic region selection method, they can separate foreground regions
from background regions, and compute the entropy using only theforeground regions. Intuitively, if such an entropy is low, then images associated with the concept have typical appearances, and theimage features of the concepts are relatively concentrated. Alternatively, if the entropy is larger, the image features of the concepts
are distributed, and the concept has no typical images.
In this paper, we modify the original method by using the bagof-feature representation (BoF) [5] and the probabilistic latent semantic analysis (PLSA) [8] instead of color and texture features
and the Gaussian mixture model (GMM), since it is regarded thatthe BoF representation has more semantically discriminative power
than other representations [5] and PLSA is more appropriate for
the BoF vectors than GMM. In addition, while the original method
employed the probabilistic generative methods to select foregroundregions which are used for computation of the region entropy, we
use mi-SVM [1], which is a discriminative method, to select positive regions by taking account of the multiple instance learning setting, since discriminative method is superior to generative methodin general in case that much training data is available.
We used the following iterative procedure based on mi-SVM [1]to select foreground regions:
1. Prepare a positive image set gathered from Flickr and a random background image set, carry out region segmentation with
JSEG [6], and construct the region-based BoF vector for eachregion.
2. Sample one third of positive images and negative backgroundimages. Train SVM with them.
3. Classify all the regions of positive images with the trained
SVM.
4. Select one third of regions in the descending order of the output values of the SVM. The selected regions can be regarded
as positive regions.
5. If the number of iteration is more than the pre-deﬁned value r,
ﬁnish the selection of positive images. In the experiment, we
set r as 5.
6. Otherwise select one sixth of positive regions in the ascending order of the output values of the SVM as negative samples. Sample one sixth of negative background images, andadd them to negative samples.
7. Train SVM, and jump back to (3).
2

In the next step, to estimate the entropy of the image features of
selected regions with respect to a generic distribution of image futures. To represent a generic model, we use the probabilistic Latent
Semantic Analysis (PLSA) [8] which is the probabilistic method to
identify latent topics with the give number of topics. PLSA was
originally proposed as a probabilistic model to extract latent topics from text documents represented by bag-of-words. In a similar
consideration, since images can be regarded as “documents” andrepresented by bag-of-features, hence PLSA can be applied to images for discovering the object categories in each image.
We need to obtain generic base topics in advance by the PLSA
for computing the entropy. To obtain the generic base, we used
about ten thousand images randomly picked up from the images
gathered from the Web.
The PLSA model is represented as the generative model of eachword w in a document d:X
P (w, d) = P (d)P (w|z)P (z|d)(1)

Images of the given concepts gathered from Flickr

Pre-processing

Region segmentaion

Construct Bag-of-Features vectors for all the regions

Select repesetative regions with mi-SVM

Calculate pLSA topic weights by fold-in heuristic
in terms of the base topics

where z ∈ Z = (z1 , .., zk ) is a latent topic variable, k is the number of topics, d ∈ D = (d1 , ..., dN ) is an image region expressedby the bag-of-features vector, and w ∈ W = (w1 , ..., wM ) is one
element of the BoF vector, which corresponds to a “visual” word.
The joint probability of the observed variables, w and d, is themarginalization over the k latent topics Z. The parameters are estimated by the EM algorithm. In the experiments, we set 300 to thenumber of base topics k. We carry out this estimation of P(w—z)
in advance which is regarded as training process of the PLSA.
For each positive region i for the concept “X”, we estimate P (z|dX
i )employing “fold-in heuristics” [8]. The entropy for the concept “X”
Himg (X) is given by
X
Himg (X) = −P (zk |X) log2 P (zk |X)(2)
k1|Iselected | i∈IX

P (zk |dXi )Base topic distribution
Compute this by pLSA withrandom image sets in advancez∈Z, where P (zk |X) =SIFT descriptors

Calculate Image Region Entorpy

Figure 1: Flow diagram of the procedure to obtain the “image
region entropy”.

5. Assign all SIFT vectors to the nearest codeword of the codebook, and convert a set of SIFT vectors for each region into
one k-bin histogram vector regarding assigned codewords. In
addition, background images which are prepared as negative
training samples in advance are also divided into regions andconverted the sets of SIFT vectors extracted from regions into
k-bin histograms based on the same codebook.

3.2

(3)

Geo-location Entropy

We can obtain location information of downloaded images whichis represented by a set of an altitude and a longitude from Flickr
with FlickrAPI. In this work, we calculate entropy regarding geolocation in addition to image region entropy. To estimate geolocation entropy, we build a histogram regarding location distribution on each concept by dividing latitude and longitude by every
10 degrees as shown in Figure 2.
Geo-location entropy Hgeo (X) is calculated by the following
equation:
X
Hgeo (X) = −bi log2 bi(4)selectedand |Iselected | is the number of selected positive regions. We summarize the procedure described above in Figure 1.

3.1.1 Image Representation
In this subsection, we describe about the region-based bag-offeatures (BoF) representation [5] we use as an image representation
to estimate image region entropy.
The main idea of the bag-of-features representation [5] is representing images as collections of independent local patches, andvector-quantizing them as histogram vectors. Before constructing
the bag-of-features vector, we apply region segmentation for all theimages. To obtain the region-based BoF vector, we extract the BoF
vector from each region. As a region segmentation method, we
use JSEG [6] after adjusting the parameters so as to generate about
eight regions per image on average.
The main steps of the method are as follows:i

1. Carry out region segmentation with the JSEG algorithm.
2. Sample 3000 patches per image randomly in the same way as[17].
3. Generate feature vectors for the sampled patches by the SIFT
descriptor [13].
4. Construct a codebook with k-means clustering over all the extracted feature vectors. A codebook is constructed for eachconcept independently. We set k as 300.

Figure 2: The overview of “Geo-location entropy”.

3

3.3 Experiments

Table 2: Image region entropy Himg (X) of top 20 and bottom
20 of 230 nouns

We made an experiment on image region entropy and geo-location
entropy for 230 nouns shown in Table 1. As noun concepts, we selected abstract nouns, person names and location names as well as
nouns related to objects and scenes in order to mix words which are
likely to be related to location and words are not likely to be related
to location. In the experiments, we gathered 500 images for eachconcept from Flickr using FlickrAPI.

top 20conceptssunrainbowmoon
dragonﬂysky
mantisegg
airplanes
bee
light
ﬂy
coffeebug
mousebutterﬂy
lemon
dream
lamp
insecttulip

Table 1: 230 noun concepts
africaalexanderalligator america
ant
asia
bach
bananabarbecue battle
beach beauty
beaver bee
beerbeetle
board
boat
bob
book
box
bread
brotherbugbuilding burger bus
california canada candycarcastle
cat
cedar
chair
chalk
cherry chicken
china
circlecity
coffee cokecolor
computercookie
coral
crow
dandelion daughter desert desk
dessert deutschlanddish
doctordolphindragonﬂydream eagle
edison eel
egg
egypt
electionelevator erica
europefather fern
ﬁeld
ﬁsh
ﬂea
ﬂower ﬂyfork
france frog
fruit
game
gates
giraffe goat
gorillagrape
grass
grasshoppergun
half
ham
hawk
heighthibiscus hornethouse
ice creamindia
insectitaliaivy
japan
jellyﬁshjumpkangaroo
killer whalelamp
lavender lawn
leaf
lemon
level
library
lightlincoln lion
lizardlocust love
mangrove manta
mantismarriage milkmintmonkey moon
mosquito moss
moth
mother mountainmouse
mozart museum napoleon new yorkoctopusowl
oyster palm
paris
park
parrotparty
pen
penguin phone pine
pizza
plant
pool
pope
potatopride
rabbit rice
rome
rose
salad
salmonsanta claus school sea
shakespeareshark
ship
shrimpsister
sky
snail
snake
son
sound sport
squarestarﬁsh
steak
sun
sushi
sword
tea
teacher
templetest
thomas
tiger
toad
tokyo
tool
town
tulip
tuna
turtleusa
valley village
whale wine
worm
zoo
airplanes backpackbear
buddhabutterﬂy cactus
cake
canoe
dice
dog
duck
eiffel tower elephant ﬁreworks goose
helicopter
horse
kayak mars
mushroom people pyramid rainbow
skyscraper
socks
spider swan
tripodwatch waterfall

H(X)3.64974.55384.66864.75505.10495.18975.22885.38515.42105.45245.49165.61605.64075.65585.67855.70965.71735.73475.75605.7700bottom 20
concepts
horse
pizza
salad
africajapan
oyster
ﬂea
tiger
rice
rome
usa
backpack
italiatown
santa-claus
housenapoleonschoollincolnmozartH(X)7.30577.30717.30937.3101
7.33877.34357.35907.38747.39067.40137.40207.4086
7.4111
7.51777.5431
7.55987.5704
7.61737.73277.8349

Table 3: Geo-location entropy Hgeo (X) of top 20 and bottom
20 of 230 nouns
top 20concepts
deutschland
rome
tokyo
paris
eiffel-tower
california
new-york
italiafranceegypt
japan
mozartlincolneuropecanadacastlebachnapoleonindia
shakespeare

We show the top 20 and bottom 20 results in terms of region
entropy and geo-location entropy in Table 2 and in Table 3, respectively.
Figure 3 shows the relations between image region entropy (xaxis) and geo-location entropy (y-axis) regarding 230 nouns. Table
4 represents the cross table between image region entropy and geolocation entropy regarding some concepts picked up from Figure 3.
This table shows concepts which has relatively larger or smaller image region entropy and larger or small image geo-location entropy
at the same time. Here, a concept with larger entropy means theconcept is included in top 46 concepts in terms of entropy ranking,and a concept with smaller entropy means the concept is included
in bottom 46 concepts in terms of entropy ranking.

H(X)0.26020.38430.62530.67300.74610.8776
1.02641.31051.4833
1.84762.29732.69042.7962
3.03793.2612
3.39483.4406
3.46863.48743.5342bottom 20
concepts
beetlegrasshopper
ricewaterfallmonkey
boat
sun
pool
bananaparrotsea
motherteacherlizardfruit
hibiscus
ant
coral
ﬁsh
mosquitoH(X)5.32255.3301
5.34255.34495.36005.37555.41095.42255.46745.47675.49365.51145.54175.54485.57795.58565.61475.65655.7831
5.9759(Figure 5) and “rainbow” their image region entropy were smaller,
while geo-location entropy were larger. This is because appearances related to such concepts tends to be very similar or almost
the same everywhere over the world. So geo-location entropy became high, and image region entropy became low.
“Tulip” (Figure 6) was the only concepts which has low image
entropy and low geo-location entropy. “Tulip” was mainly concentrated on the United States and Europe, especially , Holland, andmost of the “tulip” photos included tulip ﬂowers and tulip farms.
Both image and geo-location entropies on “Rice” (Figure 7) werelarge. Although “Rice” is a food concept which are common everywhere over the world, the way to cook is difference depending on
the countries greatly. Moreover it is also the famous person’s nameat the same time. “Dolphin” also were distributed over the world,
and its photos had diverse appearances as shown in Figure 8.

3.4 Discussion
As a prominent tendency shown in Table 4, while geo-location
entropy of location concepts such as “Deutschland”, “Rome” and“Italia” and name concepts of historical persons such as “Mozart”
and “Lincoln” were small, image region entropy of locations andperson names were larger. “Sox” also belongs to this category,since the “Sox” image set gathered from Flickr includes many “RedSox” photos which is a popular baseball team in US.
As shown in Figure 4, geo-location of location names and person names are strongly tied with the concepts themselves, while
images related to them includes various appearance since they are
relatively abstract concepts rather than physical concepts.
We found that for the concepts related to sky such as “sun”

4.

DISCOVERING CULTURAL DIFFERENCES

In this section, we propose a novel method to select representative photographs for regions in the worldwide dimensions, whichhelps detect cultural differences over the world regarding word con4

Figure 9: After collecting geotagged photo related to the given concept by the tag-based search, we remove noise images, cluster
regions and select regional representative images.

Table 4: Cross table between image region entropy and geolocation entropy

cepts with high geo-location entropy.

tinct “topics”, do additional clustering on the entire topic vectors,
and select the “signiﬁcant” cluster as the representative results forthis geographic region. In addition, with the help of map service,a UI is designed to support the browsing photos in context and understanding of the general object concepts.

4.1 Approach

4.2

Figure 3: Relations of visual entropy and geo-entropy.

Our approach for selecting the representative images for representative local regions from geotagged images consists of threemain stages as shown in Figure 9: (1) removing irrelevant images to
the given concept, (2) estimating representative geographic regions,and (3) selecting representative images for each region.
First, we apply clustering techniques to partition the image set
into similar groups, based on bag-of-features (BoF) vectors [5]. Byevaluating the intra-cluster densities as well as the cluster member
numbers, we discard most of the irrelevant images and obtain a reduced set of images which are visually similar each other. This
stage could be regarded as the “Filtering Stage”. The method employed in this stage is based on the method proposed by Raguram
et al. [19].
Then, we geographically cluster the reduced set of images andselect large geographic clusters as representative regions. Here weuse the k-means clustering algorithm based on the geographic latitude and longitude of photos to obtain representative regions in theworld for the given concept.
Finally, for each representative region, we perform the Probabilistic Latent Semantic Analysis (PLSA) [8] to identify the dis-

4.2.1

Filtering Irrelevant Images
Visual Clustering

We adopt the bag-of-features representation [5] as the image representation the detail of which is described in the previous section.
After building bag-of-features representation for all raw images,
we perform clustering using k-means algorithm over the bag-offeatures vectors to partition images into similar groups. In orderto ensure a clear partition, we choose a high number of clusters
k (≈ 200 clusters for a dataset of about two thousand images).
Since most irrelevant and visually unrelated photos tend to fall into
the small clusters, we can discard such small clusters based on aminimum threshold (usually less than 10 cluster members in our
experiment).

4.2.2

Selecting the Most Relevant Clusters

Since there may still exist some clusters with large noises (irrelevant images), in order to detect such irrelevant clusters and selectthe most relevant clusters, we employ the method of evaluating theintra-cluster similarity for the remaining clusters. The intra-cluster
similarity is the average similarity between the images that belong
5to the cluster and the similarity between two images Pi and Pj canbe calculated using the cosine metric between two image vectors
Vi and Vj :sim(Pi , Pj ) = p

Vi · Vj| Vi || Vj |Table 5: Evaluation results. This table describes the number of
raw photos directly collected from Flickr (the numerical value
in the () represents the precision), the number of photos selectedby color-based method and our proposed method (two numerical values in the () represents the precision and the recall).

(5)

Then given a cluster of n photos, C = {P1 , . . . , Pn }, we candeﬁne the intra-cluster similarity as:P
Pi ,Pj ∈C,i6=j sim(Pi , Pj )SIM (C) =(6)
n C2

raw photos
selection by
selection by
conceptsfrom Flickr color-based method proposed method
noodle2080 (42)769 (60, 54)
752 (90, 80)
ﬂower
2225 (60)703 (71, 37)
705 (85, 45)
castle1848 (35)780 (52, 61)
761 (70, 81)
car1908 (43)817 (56, 55)
800 (77, 75)
waterfall1901 (39)689 (63, 59)
672 (78, 70)
beach1917 (38)824 (51, 58)
813 (80, 90)
TOTAL/AVG. 11879 (43)4582 (59, 54)
4503 (80, 74)

which denotes the average similarity between two photos within
one cluster.
By computing the intra-cluster similarity value for each clusterand sorting all clusters in the descending order of the SIM values,
we select several top ones as the most relevant clusters (We selected
40 clusters in our experiments).

4.5

Experimental Results

To test and verify if our approach works in practice, we conducted preliminary experiments with photos collected directly from
Flickr. In the experiments, we used seven “object” concepts andtwo “scene” concepts including “noodle”, “wedding cake”, “ﬂower”,”castle”, “car”, “waterfall” and “beach”. For each concept, wecollected about 2000 most relevant geotagged photos distributed
evenly in the world wide areas.

4.3 Detecting Representative Regions
In this stage, given the remaining most relevant photos, we attempt to detect representative regions based on the photos’ geographic locations. For simplicity, we perform k-means clustering
algorithm, based on the photos’ geographic latitude and longitude(with the help of geo-tags), using geographical distance as the distance metric. Then we select several largest geo-clusters to form therepresentative regions since they have more relevant photos and thenumber of photos taken in a region is an indication of the relative
importance of that region for the particular concept. (In our experiment, for simplicity, we generally select about 4 or 5 representative
regions for each concept.)

4.5.1

Quantitative Evaluation

To evaluate our method for extracting the most relevant photos in the ﬁltering stage, we use the precision, which is deﬁned as
NR /(NR +NIR ), and the recall, which is deﬁned as NRsel /NRcol ,where NR , NIR , NRsel , and NRcol are the number of relevant photos, the number of irrelevant photos, the number of relevant photos
in selected photos, and the number of relevant photos in raw col4.4 Generating Representative Photographs
lected photos, respectively.
At this point, we have obtained the most relevant or visually simFor comparison, we still have applied simple color-historgamilar photos, and the corresponding representative regions. To genbased method to this selection task. First, we quantized the RGB
erate a set of representative photos for these representative regions,color space into 64 (4 for each axis) bins, and made a color distriwe apply the Probabilistic Latent Semantic Analysis (PLSA) [8]bution histogram for each image. Based on the distance (histogram
model which we have already used in Section 2.
intersection) between images, we clustered all images into groups
In terms of images, suppose we have a set of images D = (d1 , . . . , dn ),using k-means algorithm, and ﬁnally selected the largest clusters toeach containing the visual words from the visual vocabulary W =form the most relevant image set. In addition, for more intuitively(w1 , . . . , wm ). By introducing a mediator known as latent topics
comparing, we kept the number of images selected by this method
Z = (z1 , . . . , zk ), we can represent a joint probability model overis almost equivalent to selecting by the proposed method.
images and visual words, deﬁned as:
In Table 5, we present the evaluation results of raw photos diX
rectly collected from Flickr, the precision and recall of photos seP (w, d) = P (d)P (w|z)P (z|d)(7)
lected by color-based method and our proposed method in terms ofz∈Z
six keywords, “noodle”, “ﬂower”, ”castle”, “car”, “waterfall” and“beach”. We obtain an average precision of 80% and an average
where every image is modeled as a mixture of topics, P (z|d), andrecall of 74% by using our proposed method, which outperform
P (w|z) represents probability occurrence of visual words within athe 43% precision of the rae images and the 59% precision and the
topic. We can learn the unobservable mixture parameters P (z|d)
54% recall by the color-based method.
and topic distributions P (w|z) by the EM algorithm.
As in our experiment, for each representative region, we apply
4.5.2 Examples of Regional Representative Photos
the PLSA method to all the photos belonging to the region with agiven number of topics, and get the probability distributions of all
We show the representative photos selected for several representopics over each image, P (Z|d), which can be regarded as topic
tative regions, while these regions were generated automaticallyvectors to represent an image. In the experiment, the number ofbased on geographic locations of the most relevant photos selected
topics was set to 20. After that, we aggregate photos according toin the “Filtering Stage”. Figure 10 and Figure 11 show the results
the distributions of mixture topics by doing an additional step offor the concept “noodle”, each of which presents the most repreclustering the topic vectors, P (Z|d). In our experiments, we obsentative photos generated for the approximate regions: Japan andtained the best results by applying k-means clustering with k = 5.
Europe. Without doubt, these results can help us understand about
Then the set of photos in the largest cluster are selected as reprethe “noodle” in these local areas. For example, Figure 10 demonsentative photos of the given region, which is the ﬁnal output of thestrates many “ramen” photos in Japan and Figure 11 demonstrates
proposed system.
“spaghetti” photos in the European area. In addition, South East
6

Asia, Mideast US and Western US are obtained as other representative regions, representative photos of which also have characteristics such as “noodles” in the South East Asia area containing someTaiwanese style noodles and spicy Thai noodles.
Figure 12 and Figure 13 correspond to “wedding cake” in Europe
and in Mid US, respectively. We can ﬁnd many of the wedding
cakes in Mid US are much taller than ones in Europe.
For the scene concept “waterfall”, we extracted the representative photos for four large regions: Asia, Europe, North America,and South America. Due to the space limitation, we show only two
regions’ results, Figure 14 for the region of Asia and Figure 15 forthe region of South America. From the results, it is clear to ﬁndthat waterfalls in South America seem to be more powerful, whilewaterfalls in the Asian area are somehow more beautiful. Suchkinds of implications would be helpful in guiding travels aroundthe world.
To see more results, please visit the following website:
http://img.cs.uec.ac.jp/yanai/ASRP/ .

Vision and Pattern Recognition, 2008.
[8] T. Hofmann. Unsupervised learning by probabilistic latentsemantic analysis. Machine Learning, 43:177–196, 2001.
[9] A. Jaffe, M. Naaman, T. Tassa, and M. Davis. Generating
summaries and visualization for large collections of
geo-referenced photographs. In Proc. of ACM SIGMM
International Workshop on Multimedia Information
Retrieval, pages 89–98, 2006.
[10] D. Joshi and J. Luo. Inferring generic activities and eventsfrom image content and bags of geo-tags. In Proc. of ACM
International Conference on Image and Video Retrieval,
2008.
[11] L. Kennedy and M. Naaman. Generating diverse andrepresentative image search results for landmarks. In Proc. of
the International World Wide Web Conference, pages
297–306, 2008.
[12] M. Koskela, A. F. Smeaton, and J. Laaksonen. Measuringconcept similarities in multimedia ontologies: Analysis andevaluations. IEEE Transaction on Multimedia, 9(5):912–922,
2007.
[13] D. G. Lowe. Distinctive image features from scale-invariant
keypoints. International Journal of Computer Vision,
60(2):91–110, 2004.
[14] J. Luo, J. Yu, D. Joshi, and W. Hao. Event recognition:
Viewing the world with a third eye. In Proc. of ACM
International Conference Multimedia, 2008.
[15] M. Naaman, Y. J. Song, A. Paepcke, and H. Garcia-Molina.
Automatic organization for digital photographs withgeographic coordinates. In Proc. of ACM International
Conference Multimedia, pages 53–62, 2004.
[16] M. Naphade, J. R. Smith, J. Tesic, S. F. Chang, W. Hsu,
L. Kennedy, A. Hauptmann, and J. Curtis. Large-scaleconcept ontology for multimedia. IEEE Transaction on
Multimedia, 13(3):86–91, 2006.
[17] E. Nowak, F. Jurie, W. Triggs, and M. Vision. Samplingstrategies for bag-of-features image classiﬁcation. In Proc. ofEuropean Conference on Computer Vision, pages
IV:490–503, 2006.
[18] T. Quack, B. Leibe, and L. V. Gool. World-scale mining ofobjects and events from community photo collections. In
Proc. of ACM International Conference on Image and Video
Retrieval, pages 47–56, 2008.
[19] R. Raguram and S. Lazebnik. Computing iconic summaries
of general visual concepts. In Proc. of IEEE CVPR Workshop
on Internet Vision, 2008.
[20] I. Simon, N. Snavely, and S. M. Seitz. Scene summarizationfor online image collections. In Proc. of IEEE International
Conference on Computer Vision, 2007.
[21] K. Toyama, R. Logan, A. Roseway, and P. Anandan.
Geographic location tags on digital images. In Proc. of ACM
International Conference Multimedia, pages 156–166, 2003.
[22] K. Yaegashi and K. Yanai. Can geotags help image
recognition ? In Proc. of Paciﬁc-Rim Symposium on Imageand Video Technology, 2009.
[23] K. Yanai. Image collector II : Anover-one-thousand-image-gathering system. In Proc. of the
Twelfth International World Wide Web Conference, 2003.
[24] K. Yanai and K. Barnard. Image region entropy: A measureof “visualness” of web images associated with one concept.
In Proc. of ACM International Conference Multimedia,
pages 420–423, 2005.
[25] J. Yu and J. Luo. Leveraging probabilistic season andlocation context models for scene understanding. In Proc. of
ACM International Conference on Image and Video
Retrieval, pages 169–178, 2008.
[26] J. Yuan, J. Luo, H. Kautz, and Y. Wu. Mining GPS traces andvisual words for event classiﬁcation. In Proc. of ACM
SIGMM International Workshop on Multimedia Information
Retrieval, 2008.

5. CONCLUSION AND FUTURE WORKIn this paper, we proposed two methods to analyze relationshipbetween word concepts and geographical locations by using a large
amount of geotagged images on the photo sharing Web sites suchas Flickr. Firstly, we proposed using both image region entropy
and geo-location entropy to analyze relations between location andvisual features, and in the experiment we found that concepts with
low image entropy tends to have high geo-location entropy and viceversa. Secondly, we proposed a novel method to select representative photographs for regions in the worldwide dimensions, whichhelps detect cultural differences over the world regarding word concepts with high geo-location entropy.
Currently, the two works described in this paper are independent.
For future work, we plan to integrate two works and to make extensive experiments for more concepts from a larger set of photos,
and think out some other strategies in detecting more representative regions with a more precise and speciﬁc scope. Finally, we
will conduct methods to discriminate concepts which have larger
cultural differences from concepts with low image entropy.

6. REFERENCES[1] S. Andrews, I. Tsochantaridis, and T. Hofmann. Support
Vector Machines for Multiple-Instance Learning. In
Advances in Neural Information Processing Systems, pages
577–584, 2003.
[2] K. Barnard, P. Duygulu, N. d. Freitas, D. Forsyth, D. Blei,
and M. Jordan. Matching words and pictures. Journal of
Machine Learning Research, 3:1107–1135, 2003.
[3] L. Cao, J. Luo, H. Kautz, and T. Huang. Annotatingcollections of geotagged photos using hierarchical event andscene models. In Proc. of IEEE Computer Vision and Pattern
Recognition, 2008.
[4] M. Cristani, A. Perina, U. Castellani, and V. Murino.
Geo-located image analysis using latent representations. In
Proc. of IEEE Computer Vision and Pattern Recognition,
2008.
[5] G. Csurka, C. Bray, C. Dance, and L. Fan. Visual
categorization with bags of keypoints. In Proc. of ECCV
Workshop on Statistical Learning in Computer Vision, pages
59–74, 2004.
[6] Y. Deng and B. S. Manjunath. Unsupervised segmentation ofcolor-texture regions in images and video. IEEE
Transactions on Pattern Analysis and Machine Intelligence,23(8):800–810, 2001.
[7] J. Hays and A. A. Efros. IM2GPS: Estimating geographic
information from a single image. In Proc. of IEEE Computer
7

Figure 10: ”Noodle” in Japan. Chinese-style noodle “ramen”
is popular.
Figure 4: ”Mozart” images Figure 5: ”Sun” images over
the world.
over the world.

Figure 11: ”Noodle” in Europe.
“Spaghetti”.

Most of photos are

Figure 6: ”Tulip” images over the world.
Figure 12: ”Wedding cake” in Mid US. Tall cakes are common.
This is ﬁve-layered.

Figure 13: ”Wedding cake” in Europe. They are much shorterand simpler than US.

Figure 7: ”Rice” images over the world.

Figure 14: ”Waterfall” in Asia. It is somewhat beautiful.

Figure 15: ”Waterfall” is South America. It is more powerful.
Figure 8: ”Dolphin” images over the world.

8