AN ANALYSIS OF THE RELATION BETWEEN VISUAL CONCEPTS AND GEO-LOCATIONSUSING GEOTAGGED IMAGES ON THE WEB
Hidetoshi Kawakubo and Keiji Yanai
Department of Computer Science, The University of Electro-Communications
1-5-1 Chofugaoka, Chofu-shi, Tokyo 182-8585 JAPANABSTRACT
Recently, a large number of geotagged images are available
on photo sharing Web sites such as Flickr. In this paper, we
propose image region entropy and geo-location entropy for
analyzing the relation between visual concepts and geographical locations using a large-scale geotagged image database.
Image region entropy represents to what extent concepts havevisual characteristics, while geo-location entropy represents
to what extent concepts are distributed over the world. In theexperiment, we analyzed relations between image region entropy and geo-location entropy in terms of 230 nouns and 100
adjectives, and we found that the concepts with low image entropy tend to have high geo-location entropy and vice versa.
Index Terms— geotag, entropy, Flickr, PLSA, bag-offeatures
1. INTRODUCTION
Recently, consumer-generated media (CGM) on the Web has
become very popular. Especially, photo sharing sites such as
Flickr and Picassa are representative CGM sites, which store ahuge number of consumer-generated photos people uploaded,
and make them accessible via the Web for everyone. Photosharing sites collect not only photos but also metadata on
uploaded photos. Regarding metadata for photos, textual information such as keywords and comments is common. Recently, in addition to texts, some users attach “geo-tags” totheir uploaded photos. Note that a “geo-tag” means metadata
which represents a location where the corresponding photo
was taken, which is expressed by values of latitude and longitude.
Our objective is exploring relationship between word concepts and geographical locations by using a large number ofgeotagged images on the photo sharing Web sites such as
Flickr. In this paper, we propose bag-of-feature-based image region entropy and geo-location entropy and using bothof them to analyze relations between location and visual features. In the experiment, we analyzed relations between image region entropy and geo-location entropy in terms of 230
nouns and 100 adjectives, and we found that the concepts with
low image entropy tend to have high geo-location entropy andvice versa.
2. ENTROPY ANALYSIS
In this section, we propose a new method to analyze relations
between location and concepts in terms of image features.
We compute both image region entropy [1] and geo-location
entropy for many concepts using geotagged images gatheredfrom the Flickr.

978-1-4244-4291-1/09/$25.00 ©2009 IEEE1644

2.1. Image Region Entropy“Image Region Entropy” is a measure of “visualness” of concepts, that is, to what extent concepts have visual characteristics [1]. In the original method to compute image region entropy, they perform probabilistic region selection for regionsthat can be linked with concept “X” from images which are labeled as “X” or “non-X”, and then they compute a measure of
the entropy of the selected regions based on a Gaussian mixture model for regions. By introducing a probabilistic region
selection method, they can separate foreground regions frombackground regions, and compute the entropy using only theforeground regions. Intuitively, if such an entropy is low, thenimages associated with the concept have typical appearances,and the image features of the concepts are relatively concentrated. Alternatively, if the entropy is larger, the image features of the concepts are distributed, and the concept has no
typical images.
In this paper, we modify the original method by using thebag-of-features representation (BoF) [2] and the probabilistic
latent semantic analysis (PLSA) [3] instead of color and texture features and the Gaussian mixture model (GMM), since
it is regarded that the BoF representation has more semantically discriminative power than other representations [2] and
PLSA is more appropriate for the BoF vectors, which is usually high-dimensional and sparse, than GMM. In addition,
while the original method employed the probabilistic generative methods to select foreground regions which are used
for computation of the region entropy, we use mi-SVM [4],which is a discriminative method, to select positive regions
by taking account of the multiple instance learning setting,since discriminative method is superior to generative methodin general in case that much training data is available.
We used the following iterative procedure based on miSVM [4] to select foreground regions:
1. Prepare a positive image set gathered from Flickr and arandom background image set, carry out region segmentation with JSEG [5], and construct the region-based BoF
vector for each region.
2. Sample one third of positive images and negative background images. Train SVM with them.
3. Classify all the regions of positive images with thetrained SVM.
4. Select one third of regions in the descending order of theoutput values of the SVM. The selected regions can be
regarded as positive regions.
5. If the number of iteration is more than the pre-deﬁned
value r, ﬁnish the selection of positive images. In theexperiment, we set r as 5.
6. Otherwise select one sixth of positive regions in the ascending order of the output values of the SVM as nega-

ICME 2009

tive samples. Sample one sixth of negative backgroundimages, and add them to negative samples.
7. Train SVM, and jump back to (3).
The steps from Step 3 to 7 is iterative steps. It is the same
way as [1] to employ an iterative process for selecting positive
regions.
As the next step, to estimate the entropy of the image features of selected regions with respect to a generic distribution of image futures. To represent a generic model, we use
the probabilistic Latent Semantic Analysis (PLSA) [3] whichis the probabilistic method to identify latent topics with thegiven number of topics. PLSA was originally proposed asa probabilistic model to extract latent topics from text documents.
We need to obtain generic base topics in advance by the
PLSA for computing the entropy. To obtain the generic base,we used about ten thousand images randomly picked up fromthe images gathered from the Web.
The PLSA model is represented as the generative modelof each word w in a document d:
P (w|z)P (z|d)(1)
P (w, d) = P (d)Images of the given concepts gathered from Flickr

Pre-processing

Region segmentaion

Construct Bag-of-Features vectors for all the regionskwhere P (zk |X) =1|Iselected |P (zk |dXi )(3)i∈Iselected

and |Iselected | is the number of selected positive regions. We
summarize the procedure described above in Figure 1.
2.1.1. Image Representation

Base topic distribution
Compute this by pLSA withrandom image sets in advance

Select repesetative regions with mi-SVM

Calculate pLSA topic weights by fold-in heuristic
in terms of the base topics

Calculate Image Region Entorpy

Fig. 1. Flow diagram of the procedure to obtain the “image
region entropy”.
1. Carry out region segmentation with the JSEG algorithm.
2. Sample 3000 patches per image randomly in the same
way as [6].
3. Generate feature vectors for the sampled patches by the
SIFT descriptor [7].
4. Construct a codebook with k-means clustering over all
the extracted feature vectors. A codebook is constructed
for each concept independently. We set k as 300.
5. Assign all SIFT vectors to the nearest codeword of thecodebook, and convert a set of SIFT vectors for each region into one k-bin histogram vector regarding assigned
codewords. In addition, background images which areprepared as negative training samples in advance are also
divided into regions and converted the sets of SIFT vectors extracted from regions into k-bin histograms basedon the same codebook.

z∈Z

where z ∈ Z = (z1 , .., zk ) is a latent topic variable, k is
the number of topics, d ∈ D = (d1 , ..., dN ) is an image region expressed by the bag-of-features vector, and w ∈ W =(w1 , ..., wM ) is one element of the BoF vector, which corresponds to a “visual” word. The joint probability of the observed variables, w and d, is the marginalization over the klatent topics Z. The parameters are estimated by the EM algorithm. In the experiments, we set 300 to the number of base
topics k. We carry out this estimation of P (w|z) in advancewhich is regarded as training process of the PLSA. For full
explanation of the PLSA model refer to [3].
For each positive region i for the concept “X”, we estimate P (z|dXi ) employing “fold-in heuristics” [3]. The entropy for the concept “X” Himg (X) is given byP (zk |X) log2 P (zk |X)(2)
Himg (X) = −SIFT descriptors

2.2. Geo-location Entropy
We can obtain location information of downloaded images
which is represented by a set of values of latitude and longitude from Flickr with FlickrAPI. In this work, we calculate entropy regarding geo-location in addition to image region entropy. To estimate geo-location entropy, we build ahistogram regarding location distribution on each concept bydividing latitude and longitude by every 10 degrees as shownin Figure 2. Geo-location entropy Hgeo (X) is calculated by
the following equation:

In this subsection, we describe about the region-based bagof-features (BoF) representation [2] we use as an image representation to estimate image region entropy.
The main idea of the bag-of-features representation [2]is representing images as collections of independent local
patches, and vector-quantizing them as histogram vectors.
Before constructing the bag-of-features vector, we apply
region segmentation for all the images. To obtain the regionbased BoF vector, we extract the BoF vector from eachregion. As a region segmentation method, we use JSEG [5]after adjusting the parameters so as to generate about eightregions per image on average.
The main steps of the method are as follows:

1645

Hgeo (X) = −bi log2 biiFig. 2. The overview of “Geo-location entropy”.

(4)

3. EXPERIMENTS
We examined image region entropy and geo-location entropy
for 230 nouns shown in Table 1 and 100 adjectives shown in
Table 2. For the experiments, we gathered 500 images foreach concept from Flickr by providing the concept words as
search keywords for FlickrAPI. Totally, we collected 115,000
geotagged photos. In the experiments, we used images including a given concept word as metadata as positive images,
while images randomly selected from all the downloaded
photos are used as negative images. In [1], they used only
150 adjectives and did not make an experiment with nouns.
We imported 100 adjectives from the 150 adjectives, and fornoun concepts we selected abstract nouns, person names andlocation names as well as nouns related to objects and scenesby hand in order to mix the words which are likely to berelated to location and the words are not likely to be related
to location.
Table 1. 230 noun concepts

Table 3. Image region entropy Himg (X) of top 10 and bottom 10 of 230 nouns
10 smallest
conceptssunrainbowmoon
dragonﬂysky
mantisairplanes
egg
bee
light

H(X)
3.64974.55384.66864.75505.10495.18975.38515.22885.42105.452410 largest
conceptsusa
backpack
italiatown
santa-claus
housenapoleonschoollincolnmozartH(X)7.40207.4086
7.4111
7.51777.5431
7.55987.5704
7.61737.73277.8349

Table 4. Geo-location entropy Hgeo (X) of top 10 and bottom
10 of 230 nouns

africaalexander alligatoramerica ant
asia
bach
bananabarbecue battle
beach
beautybeaverbeebeerbeetle
board
boat
bob
book
box
bread
brother bug
building burgerbuscalifornia canada
candy
car
castle
cat
cedar
chair
chalk
cherrychicken china
circlecity
coffeecoke
color
computer cookie coralcrowdandelion daughter desert
desk
dessertdeutschland dishdoctordolphindragonﬂy dreameagle
edison
eel
egg
egypt
election elevator
erica
europefather
fern
ﬁeld
ﬁsh
ﬂea
ﬂower
ﬂy
fork
france
frog
fruit
game
gates
giraffe goat
gorillagrape
grass
grasshopper gun
half
ham
hawk
heighthibiscus hornet
house
icecreamindia
insectitaliaivy
japan
jellyﬁshjumpkangaroo killerwhale lamp
lavender lawn
leaf
lemon
level
library
lightlincolnlion
lizardlocustlove
mangrove manta
mantismarriage milkmintmonkey moon
mosquito moss
moth
mother mountain mouse
mozart museum napoleon newyork octopus owl
oyster
palm
paris
park
parrotparty
pen
penguin phone
pine
pizza
plant
pool
pope
potatopride
rabbit
rice
rome
rose
salad
salmonsantaclaus schoolsea
shakespeare shark
ship
shrimp sister
sky
snail
snake
son
sound
sport
square starﬁsh
steak
sun
sushi
sword
tea
teacher temple
test
thomas tiger
toad
tokyo
tool
town
tulip
tuna
turtleusa
valleyvillage
whale
wine
worm
zoo
airplanes backpack bear
buddhabutterﬂy cactus
cake
canoe
dice
dog
duck
eiffeltower elephant ﬁreworks goose
helicopter
horse
kayak
mars
mushroom people
pyramid
rainbow skyscraper socks
spiderswan
tripodwatch
waterfall

10 smallest
concepts
deutschland
rome
tokyo
paris
eiffel-tower
california
new-york
italiafranceegypt

H(X)0.26020.38430.62530.67300.74610.8776
1.02641.31051.4833
1.847610 largest
concepts
sea
motherteacherlizardfruit
hibiscus
ant
coral
ﬁsh
mosquitoH(X)5.49365.51145.54175.54485.57795.58565.61475.65655.7831
5.9759

Table 5. Image region entropy Himg (X) of top 10 and bottom 10 of 100 adjectives

Table 2. 100 adjective concepts
aerialancient antique badbeautiful best
betterbig
black blue
botanical bottom
brightbrown cherry classic
clean
clear
cold
colourful concrete cool
crazy
cute
dark
digital dryelectric empty famous female
ﬁrst
general good grand
gray
great
green happy hard
heavy highhistoricholy
hothuman iced
interior
international large latest longmale
medieval military
mobile modern more mostnational
naturalnautical new
nice
old
older
oldestopenorange outdoor pinkpresent
publicpurple rainy red
ruralrusted scenic
second sexyshort small
specialsunny
sweet top
traditional tropical twinunderwater urban
vintage warm welcome white
wide
wild
wooden yellow

10 smallest
conceptsorangeyellowdark
latest
white
clean
pink
botanical
happy
nauticalH(X)5.56085.67805.74515.83055.90335.96235.9857
6.09376.1351
6.208510 largest
concepts
woodenoldestconcreteolderinternational
traditional
generalpublicvintagehistoric

H(X)7.34697.37287.38277.3905
7.39947.42687.50157.5253
7.59727.6257

Table 6. Geo-location entropy Hgeo (X) of top 10 and bottom
10 of 100 adjectives

We show the top 10 and bottom 10 results in terms ofregion entropy and geo-location entropy for noun concepts inTable 3 and Table 4, respectively. In the same way, we show
the top 10 and bottom 10 results in terms of region entropy
and geo-location entropy for adjective concepts in Table 5 andTable 6, respectively.
Figure 3 and 4 show the relations between image region
entropy (x-axis) and geo-location entropy (y-axis) regarding
230 nouns and 100 adjectives, respectively. Table 7 represents the cross table between image region entropy and geolocation entropy regarding some concepts picked up from
Figure 3. This table shows concepts which has relatively
larger or smaller image region entropy and larger or small
image geo-location entropy at the same time. In this table, aconcept with larger entropy means the concept is included in
top 46 concepts in terms of entropy ranking, and a concept
with smaller entropy means the concept is included in bottom
46 concepts in terms of entropy ranking.
3.1. Discussion on Nouns
As a prominent tendency, while geo-location entropy of location concepts such as “Rome”, “Africa” and “Japan” andname concepts of historical persons such as “Mozart” and“Napoleon” were small, image region entropy of locations
and person names were larger. “Sox” also belongs to this category, since the “Sox” image set gathered from Flickr includes

1646

10 smallest
concepts
medieval
modern
new
rainy
grand
cherrybig
nicepublichistoric

H(X)3.4364
3.7842
3.89834.1315
4.15684.17824.2196
4.29724.36034.428910 largest
conceptsyellow
blue
cute
small
beautifuldrycolourfultropicalunderwater
traditionalH(X)5.25545.27095.28425.34095.38975.41445.45455.56875.76885.8487many “Red Sox” photos which is a popular baseball team in
US.
As shown in Figure 5, geo-location of location names
and person names are strongly tied with the concepts themselves, while images related to them includes various appearance since they are relatively abstract concepts rather thanphysical concepts.
We found that for the concepts related to sky such as “sun”(Figure 6) and “rainbow” their image region entropy were
smaller, while geo-location entropy were larger. This is because appearances related to such concepts tends to be very
similar or almost the same everywhere over the world. So
geo-location entropy became high, and image region entropy
became low.
“Tulip” (Figure 7) was the only concept which has low
image entropy and low geo-location entropy. “Tulip” wasmainly concentrated on the United States and Europe, espe-

Fig. 5. ”Mozart” images over Fig. 6. ”Sun” images over
the world.
the world.
Fig. 3. Relations of visual and geo- entropy for 230 nouns.

Fig. 7. ”Tulip” images over Fig. 8. ”Rice” images over
the world.
the world.
Fig. 4. Relations of visual and geo- entropy for 230 nouns.
Table 7. Cross table between image region entropy and geolocation entropy for nouns

cially , Holland, and most of the “tulip” photos included tulip
ﬂowers and tulip farms.
Both image and geo-location entropies on “Rice” (Figure8) were large. Although “Rice” is a food concept which arecommon everywhere over the world, the way to cook is different depending on countries greatly. Moreover it also means
the name of the ex-spokesperson of the US government.
3.2. Discussion on Adjectives
Compared Table 3 with Table 5 and compared Table 4 with
Table 6, both image region and geo-location entropies on adjectives were higher than entropies on nouns on the average.
Since adjectives are abstract and general concepts, there areno typical visual appearances and they do not depends onlocations as much as nouns in general. In Table 5, the top
10 adjectives with the smallest entropy include many color
concepts. This is because images related to color concepts
contained many uniform regions ﬁlled with the corresponding color.

1647

4. CONCLUSION AND FUTURE WORKIn this paper, we proposed a novel method to analyze relationship between word concepts and geographical locations
by using a large number of geotagged images on the photosharing Web sites such as Flickr, and we proposed using both
image region entropy and geo-location entropy to analyze relations between visual concepts and their locations. In theexperiment, we analyzed relations between image region entropy and geo-location entropy in terms of 230 nouns and 100
adjectives, and we found that the concepts with low image entropy tend to have high geo-location entropy and vice versa.
For future work, we plan to investigate more deep relations between locations and concepts. We like to propose a
new method to discover differences of a concept depending on
locations like Western-style house is different from Japanesestyle house. Finally we will conduct methods to discriminate
concepts which have larger cultural differences from concepts
with low image entropy.
5. REFERENCES[1] K. Yanai and K. Barnard, “Image region entropy: A measure of “visualness” of web images associated with one concept,” in Proc. of ACM
International Conference Multimedia, 2005, pp. 420–423.
[2] G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual categorization withbags of keypoints,” in Proc. of ECCV Workshop on Statistical Learningin Computer Vision, 2004, pp. 59–74.
[3] T. Hofmann, “Unsupervised learning by probabilistic latent semantic
analysis,” Machine Learning, vol. 43, pp. 177–196, 2001.
[4] S. Andrews, I. Tsochantaridis, and T. Hofmann, “Support Vector Machines for Multiple-Instance Learning,” in Advances in Neural Information Processing Systems, 2003, pp. 577–584.
[5] Y. Deng and B. S. Manjunath, “Unsupervised segmentation of colortexture regions in images and video,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 23, no. 8, pp. 800–810, 2001.
[6] E. Nowak, F. Jurie, W. Triggs, and M. Vision, “Sampling strategies forbag-of-features image classiﬁcation,” in Proc. of European Conference
on Computer Vision, 2006, pp. IV:490–503.
[7] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110,
2004.