Geotagged Photo Recognition using Corresponding Aerial Photos
with Multiple Kernel Learning
Keita Yaegashi
Keiji Yanai
Department of Computer Science, The University of Electro-Communications
1–5–1 Chofugaoka, Chofu-shi, Tokyo, 182–8585 Japanyaegas-k@mm.cs.uec.ac.jp yanai@cs.uec.ac.jp
Abstract“Disneyland” and “Tokyo tower” in the experiments of [7].
On the other hand, in the method (2), we utilize
aerial photo images around the place where a photo
was taken as additional information on the place.
Since “sea” and “mountain” are distributed all overthe world, it is diﬃcult to associate values of latitude and longitude with such generic concepts directly. Then, we regard aerial photo images aroundthe place where the photo is taken as the information expressing the condition of the place, andutilize visual feature extracted from aerial images
as yet another geographic contextual information
associated with geotags of photos. Especially, forgeographical concepts such as “sea” and “mountain”, using feature extracted from aerial photos
was much more eﬀective than using raw values oflatitude and longitude directly [7].
Since in the method (2) of [7] we combine twokinds of feature vectors by simply concatenatingthem into one long vector, the extent of contributions of aerial photos for geotagged photo recognition was unclear. They are expected to vary depending on target concepts. For the concepts which
can be directly recognized from aerial photos suchas “beach” and “mountain”, the contribution ofaerial photos is expected to be much, while it isexpected to be less for the unrecognizable concepts
from aerial photos such as “ﬂower” and “cat”. In
this paper, we introduce Multiple Kernel Learning(MKL) to evaluate contribution of both features forrecognition by estimating the weights of image features of photos and aerial images. MKL is an extension of Support Vector Machine (SVM), and makes
it possible to estimate optimal weights to integrate
diﬀerent features with the weighted sum of kernels.
In the experiments, we evaluate the weights of bothfeatures using MKL for eighteen concepts.
As related work, J. Luo et al.[4] is the most similar to our work. They also proposed to exploit
aerial photos corresponding to where a photo wastaken for recognizing geotagged photos. However,they focused only event concepts such as “baseball”, “at beach” and “in park”, which can be directly recognized from aerial photos, and did notevaluate the contribution of aerial images for recognition with various kinds of concepts.
The rest of this paper is organized as follows:

In this paper, we treat with generic object recognition for geotagged images. As a recognition method
for geotagged photos, we have already proposed exploiting aerial photos around geotag places as additional image features for visual recognition of geotagged photos. In the previous work, to fuse twokinds of features, we just concatenate them. Instead, in this paper, we introduce Multiple Kernel
Learning (MKL) to integrate both features of photos
and aerial images. MKL can estimate the contribution weights to integrate both kinds of features.
In the experiments, we conﬁrmed eﬀectiveness ofusage of aerial photos for recognition of geotagged
photos, and we evaluated the weights of both features estimated by MKL for eighteen concepts.

1Introduction

Recently, the number of geotagged photos on
the Web is increasing rapidly, since people can see
geotagged photos uploaded to some photo sharing
Web sites on a map. Geotags for photos are easy
to obtain with GPS-equipped cameras or carrying portable small GPS devices with digital cameras. Here, “geotag” means a two-dimensional vector consisting of values of latitude and longitude
which represent where a photo is taken.
In this paper, we exploit geotags as additional information for visual recognition of consumer photos
to improve its performance. Geotags have potential
to improve performance of visual image recognition,
since recognition targets are unevenly distributed in
the real world. For example, “beach” photos can betaken near the sea and “lion” photos can be taken
only in a zoo except Africa. In this way, geotags
can restrict concepts to be recognized for images,
so that we expect geotags can help visual image
recognition.
To utilize geotags in visual image recognition, we
have already proposed two methods in [7]: (1) combining values of latitude and longitude with visualfeatures of a photo image, and (2) combining visualfeature extracted from aerial photo images with visual feature extracted from a photo image.
The method (1) is relatively straightforward way,
and it improved recognition performance only forconcepts associated with speciﬁc places such as1

Figure 1. Correspondences between a geotagged photo and aerial images.
Section 2 and 3 explains the overview and the procedure of geotagged image recognition with MKL,
respectively. Section 4 shows the experimental results and discusses them, and we conclude this paper in Section 5.

2Overview

The objective of this paper is to evaluate the contribution weights of both visual features of photo
images and aerial images for image recognition using Multiple Kernel Learning (MKL) regarding various kinds of concepts. In this paper, we assume
that image recognition means judging if an image
is associated with a certain given concept such as“mountain” and “beach”, which can be regarded asa photo detector for a speciﬁc given concept. By
combining many detectors, we can add many kindsof words as word-tags to images automatically.
As representation of photo images, we adopt thebag-of-features (BoF) representation [1]. It has
been proved that it has excellent ability to represent image concepts in the context of visual image
recognition in spite of its simplicity. As representation of geotags, we also adopt the bag-of-features
representation of aerial photos around the geotag
location.
After obtaining feature vectors, we carry out twoclass classiﬁcation by fusing both visual feature vectors with MKL. In the training step of MKL, weobtain optimal weights to fuse both features.

3Methods

In this section, we describe how to recognize images with visual features and geotags. First of all,we need to decide several concepts for the experiments.

3.1

Data Collection

In this paper, we obtain geotagged images forthe experiments from Flickr by searching for images
which have word tags corresponding to the givenconcept. Since the raw images fetched from Flickr
include some noise images which are irrelevant to
the given concepts, we select only relevant imagesby hand. In the experiments, relevant images areused as positive samples, while randomly-sampled
images from all the geotagged images fetched from
Flickr are used as negative samples. We select 200
positive samples and 200 negative samples for eachconcept.

After obtaining geotagged images, we collectaerial photos around the points corresponding to
the geotags of the collected geotagged image withseveral scales from an online aerial map site. In theexperiments, we collect 256 × 256 aerial photos infour diﬀerent kinds of scales for each Flickr photoas shown in Figure 1. The largest-scale one (level4) corresponds to an area of 497 meters square, thenext one (level 3) corresponds to an area of 1.91
kilometers square, the middle one (level 2) corresponds to a 7.64 kilometer-square area, and thesmallest-scale one (level 1) corresponds to a 30.8
kilometer-square area.

3.2

Visual Features

To extract visual feature vectors from photos and aerial images, we use the bag-of-features
method [1]. The main idea of the bag-of-features isrepresenting images as collections of independent
local patches, and vector-quantizing them as histogram vectors.
The main steps to build a bag-of-features vector
are as follows:
1. Sample many patches from all the images. In
the experiment, we sample patches on a regulargrid with every 10 pixels.
2. Generate local feature vectors for the sampled
patches by the SIFT descriptor [3] with four
diﬀerent scales: 4, 8, 12, and 16.
3. Construct a codebook with k-means clusteringover extracted feature vectors. We construct a
codebook for photo images for each given concept independently, while we construct a codebook for aerial images which is common among
all the aerial images for any concepts. We set
the size of the codebook k as 300 in the experiments.
4. Assign all feature vectors to the nearest codeword (visual word) of the codebook, and convert a set of feature vectors for each image into
one k-bin histogram vector regarding assigned
codewords.

3.3

Multiple Kernel Learning

In this paper, we carry out two-class classiﬁcation by fusing visual features of photo images
and aerial images with Multiple Kernel Learning(MKL). MKL is an extension of a Support Vector
Machine (SVM). MKL treats with a combined kernel which is a weighted liner combination of several

single kernels, while a standard SVM treats with
only a single kernel. MKL can estimates optimal
weights for a linear combination of kernels as well as
SVM parameters simultaneously in the train step.
The training method of a SVM employing MKL issometimes called as MKL-SVM.
Recently, MKL-SVM is applied into image recognition to integrate diﬀerent kinds of features suchcolor, texture and BoF [6, 2]. However, the recent
work employing MKL-SVM focuses on fusion of different kinds of features extracted from the same image. This is diﬀerent from our work that MKL isused for integrating features extracted from the different sources, which are photos and aerial images.
With MKL, we can train a SVM with aadaptively-weighted combined kernel which fuses
diﬀerent kinds of image features. The combinedkernel is as follows:KK∑∑
Kcomb (x, y) =βj Kj (x, y) with
βj = 1,j=1

j=1

where βj is weights to combine subkernels Kj (x, y).
As a kernel function, weused a chi-square RBF kernel.
Sonnenburg et al.[5] proposed an eﬃcient algorithm of MKL to estimate optimal weights andSVM parameters simultaneously by iterating training steps of a standard SVM. This implementation
is available as the SHOGUN machine learning toolbox at the Web site of the ﬁrst author of [5]. In theexperiment, we use the MKL library included in theSHOGUN toolbox as an implementation of MKL.

4

These concepts depend on time or seasonsrather than locations.
Location-speciﬁc and recognizable concepts are expected to have correlation to aerial images, while
unrecognizable and time-dependent are to have noor less correlation to aerial images.
We gathered photo images associated to theseconcepts from Flickr, and selected 200 positive sample images for each concepts. As negative sample images, we selected 200 images randomly from
100,000 images gathered from Flickr.
In the experiments, we carried out two-class image classiﬁcation and estimate weights to integratefeatures of photos and four kinds of aerial images infour diﬀerent levels using MKL for the eighteen concepts, and as a baseline we also made experiments
on image classiﬁcation with only image features ofphotos without fusion using a standard SVM.
We evaluated experimental results with ﬁve-foldcross validation using the average precision (AP)which is computed by the following formula:(1)

where P rec(i) is the precision rate of the i positive images from the top, and N is the number of
positive test images for each fold.
Table 1. Average precision for eighteen
concepts by only photos and by fusion ofphotos and aerial images.

Experiments(type) concept(l) Disneyland(r) park(r) shrine(t) festival(r) bridge(r) landscape(r) beach(t) red leaves(l) Tokyo tower(r) castle(u) sushi(r) railroad(u) ﬂower(t) cherry blossom(u) ramen noodle(u) cat(u) vendor machine(t) sunset
AVERAGEIn the experiments, we used eighteen concepts.
To selected the eighteen concepts, we ﬁrst deﬁne
four rough types of concepts and then select several
concepts for each type as follows:
Location-speciﬁc concept (l)[2 concepts]
Disneyland, Tokyo tower
The locations related to these concepts are
speciﬁc to them. Since all the aerial photos
related to these concepts are similar to eachother, using aerial photos are expected to improve recognition performance much.
Recognizable concept (r)[7 concepts]
beach, castle, bridge, railroad, park, shrine,
landscapeSince there are possibility of recognizing theseconcepts on aerial photos directly, improvement of performance is expected.
Unrecognizable concept (u)[5 concepts]
ﬂower, cat, sushi, ramen noodle, vendor machine
These concepts are very diﬃcult to recognize
on aerial images.
Time-dependent concept (t)[4 concepts]
sunset, cherry blossom, red leaves, festival

1 ∑P rec(i),
N i=1
N

AP =

4.1

photo
68.00
67.43
72.79
72.32
69.51
73.71
80.10
79.18
80.84
81.28
80.11
74.70
77.00
80.94
82.34
73.98
83.17
83.01
76.69

MKL
84.06
76.04
78.53
77.75
74.89
78.37
83.85
82.45
83.84
83.53
81.93
76.20
78.48
81.61
82.70
74.26
83.43
83.11
80.28

gain
+16.06
+8.61
+5.74
+5.43
+5.38
+4.66
+3.75
+3.27
+3.00
+2.23
+1.82
+1.50
+1.48
+0.67
+0.36
+0.28
+0.26
+0.10
+3.59

Experimental Results

We show the results by MKL-based fusion andthe baseline method in Table 1 and the estimatedweights for features of photos and features of fourkinds of aerial images in Table 2. In both the tables,
the alphabets just before concept names represent
the types of concepts. (l), (r), (u) and (t) means
location-speciﬁc concepts, recognizable concepts,

Table 2. Weights of features of images and four kinds of aerial images. The results are sortedby the decending order of the weight of photos.
(type) concept
photo(u) ramen noodle
0.873(u) vendor machine 0.794(t) cherry blossom
0.774(u) cat
0.743(t) sunset
0.729(u) ﬂower
0.658(r) railroad
0.604(r) landscape
0.604(u) sushi
0.596level1
0.002
0.058
0.038
0.028
0.055
0.000
0.106
0.078
0.062

level2
0.000
0.009
0.006
0.008
0.058
0.042
0.014
0.024
0.015

level3
0.037
0.074
0.093
0.063
0.016
0.051
0.052
0.093
0.062

level4
0.088
0.065
0.090
0.158
0.142
0.249
0.224
0.199
0.266

unrecognizable concepts, and time-dependent concepts, respectively.
In Table 1, all the AP values of all the concepts were improved by aerial image fusion. Recognizable concepts and location concepts tend toachieve higher gains, while unrecognizable concepts
and time-dependent concepts tend to produce lowgains.
Especially, the gain on “Disneyland” was by far
the best among all the concepts, since “Disneyland”
photos are alway taken inside Disneyland parks,
and aerial photos corresponding to the geotag place
are always the same. The gain on another locationspeciﬁc concept “Tokyo tower” is not so much, sincephotos on “Tokyo tower” were not take inside it,
but usually taken from many other places.
Regarding unrecognizable concepts such as“sushi” and “ﬂower”, small gains were obtained.
This is because the places where “sushi” and“ﬂower” photos are taken might have causal relationship to geographical features which appear directly in aerial photos. The places where “sushi”
and “ﬂower” photos are taken are unevenly distributed, and are usually commercial areas wherethere are many sushi restaurants for “sushi” or noncommercial areas and farming areas for “ﬂower”.
We expect that this indirect causal relation goesfor many unrecognizable concepts other than ﬂowers and sushi.
Regarding the estimated weights shown in Table
2, most of the concepts have the highest weights on
photo features and the second highest weights onlevel 4 aerial images which is the ﬁnest one amongfour levels of aerial images. For location-speciﬁc
concepts such as Disneyland, the area covered by
the ﬁnest level-4 image is too small to represent
speciﬁc features to the Disneyland.
The weights of photos for unrecognizable concepts and time-dependent concepts tend to belarger, while the weights of photos for recognizable for concepts and location concepts tend to besmaller.
From these results shown in Table 1 and Table 2, using ﬁner aerial photos for recognition ismore eﬀective for recognizable concepts, and using medium-level aerial photos rather than theﬁnest ones are more helpful for location-speciﬁc(type) concept(r) bridge(t) red leaves(r) castle(t) festival(r) shrine(r) park(r) beach(l) Disneyland(l) Tokyo tower

photo
0.582
0.5230.523
0.518
0.507
0.437
0.392
0.384
0.364

level1
0.077
0.141
0.166
0.058
0.033
0.073
0.115
0.095
0.008

level2
0.044
0.006
0.004
0.001
0.009
0.012
0.173
0.236
0.002

level3
0.070
0.062
0.099
0.185
0.061
0.045
0.055
0.131
0.396

level4
0.226
0.269
0.208
0.238
0.391
0.433
0.265
0.153
0.231

concepts. Moreover, even for unrecognizable andtime-dependent concepts, aerial photos is still a little beneﬁcial due to indirect causal relation.

5Conclusion

In this paper, we proposed introducing Multiple Kernel Learning (MKL) into geotagged image
recognition to estimate the contribution weights ofboth visual features of photo images and aerial images. In the experiments, we made experimentswith eighteen concepts selected from four diﬀerent
types of concepts. The experimental results showedthat using aerial images can be regarded as very
helpful for recognizable concepts such as “beach”
and “park” and still a little beneﬁcial even for unrecognizable concepts such as “cat” and “noodle”
likely due to indirect causal relation.
For future work, we plan to make more extensive
experiments with much more concepts and more
precise aerial photos, and to use other kinds of data
sources than aerial photos for geotags.

References[1] G. Csurka, C. Bray, C. Dance, and L. Fan. Visual
categorization with bags of keypoints. In Proc. ofECCV Workshop on Statistical Learning in Computer Vision, pages 59–74, 2004.
[2] P. Gehler and S. Nowozin. On feature combination for multiclass object classiﬁcation. In Proc.
of IEEE International Conference on Computer Vision, 2009.
[3] D. G. Lowe. Distinctive image features from scaleinvariant keypoints. International Journal of Computer Vision, 60(2):91–110, 2004.
[4] J. Luo, J. Yu, D. Joshi, and W. Hao. Event recognition: Viewing the world with a third eye. In Proc. ofACM International Conference Multimedia, 2008.
[5] S. Sonnenburg, G. Rätsch, C. Schäfer, andB. Schölkopf. Large Scale Multiple Kernel Learning. The Journal of Machine Learning Research,
7:1531–1565, 2006.
[6] M. Varma and D. Ray. Learning the discriminative
power-invariance trade-oﬀ. In Proc. of IEEE International Conference on Computer Vision, pages
1150–1157, 2007.
[7] K. Yaegashi and K. Yanai. Can geotags help image
recognition ? In Proc. of Paciﬁc-Rim Symposium
on Image and Video Technology, 2009.