WeBCT 9.37

Geotagged Photo Recognition
using Corresponding Aerial Photos
with Multiple Kernel Learning
Keita Yaegashi and Keiji Yanai

The University of Electro-Communications, Tokyo, JAPANBackground & Objective

Experiments
Experiments

Data Collection

 Geotagged Photos are easy to obtain.

◆ 1/0 classification for 18 classes
by 5-fold cross validation
・ 200 positive & 200 negative images for each class
Average・ Evaluated by average precision precision

Given concepts

precision“Mountain”
latitude=42.844202
latitude=42.844202
longitude=140.8528
latitude=42.844202
longitude=140.8528
latitude=42.844202
99
longitude=140.8528
latitude=42.84420
99
longitude=140.8528
99
longitude=140.85289
99

Level 3
Level 4497m x 497m 1.91km x
1.91km256x256
256x256

latitude=42.84422
longitude=140.85289

latitude=23.23202
latitude=37.24423
longitude=143.89249 longitude=137.68353

latitude=40.84423
longitude=135.18353(l) Location concepts

Level 27.64km x
7.64km256x256

Level 130.8km x30.8km
256x256

Web map services

recall(u) Unrecognizable concepts(r) Recognizable concepts

 Gather 4 levels of aerial photos foreach geotagged photo.

(t) Time-dependent concepts

geotag = (latitude, longitude)

FeaturesFeatures

Can a geotag help image recognition ?
No ! when using them as 2D vectors.

photoaerial photos (4 levels)

shown by the following two papers.
codebooks

BoF◆ Bag-of-features (BoF) (local pattern)(1) Sample points by grid sampling (every 10px)[4] J. Luo, J. Yu, D. Joshi, and W. Hao. Event recognition:
Viewing the world with a third eye. In Proc. ofACM International Conference Multimedia, 2008.
Only events such as “baseball” and “on beach”[7] K. Yaegashi and K. Yanai. Can geotags help image
recognition ? In Proc. of Pacific-Rim Symposium

on Image and Video Technology, 2009.
10 categories.
Fusion by concatenating both feature vectors.

Questions ?
1) To what extent can geotags help ?
2) What kinds of categories aregeotags effective for ?
We evaluate the contribution of aerial
photos for image recognition by using
Multiple Kernel Learning (MKL).

latitude=42.844202
longitude=140.852899

Level 4497m x 497m
256x256

Level 3
1.91km x
1.91km256x256

・ Contribution weights by MKL

Grid sampling(2) Describe local patterns around thesampled points with SIFT [Lowe 2004](3) Generate codebooks by k-means(size of a codebook; 1000)(4) Convert images into BoF vectors
by voting to nearest codewords

Five BoF vectors for each image

Learning & Classifying
 Multiple Kernel Learning

MethodLevel 2
Level 17.64km x 7.64km 30.8km x 30.8km
256x256
256x256

・ Is an extenstion of a SVM .
・ Can handle “a combined kernel” which
is a linear combination of kernels.
・ Can estimate kernel weights and SVM
model parameters simultaneously.
・ Can integrate features by assigning
one feature to one kernel.
Combined kernel

BoF

BoF

BoF

BoF

BoF

photo BoF kernel

smaller gain

Represent geographical context

BoF

aerial photo(Lv1) ker.

Aerial photo(Lv4) ker.

Combined kernel

MKL
Kernel weight (to be estimated by MKL)

Photoscorresponding aerial photos

BoF

contribution

latitude=42.84422
longitude=140.85289

BoF

Aerial photos

BoF

gain

Yes ! It can help image recognition
by converting it into aerial photos.

larger gain

・ Classification results (AP, %)Conclusions
Conclusions◆Analyzed contribution ratios ofaerial images for image recognition
using Multiple Kernel Learning (MKL)-AP was improved by 3.59 % on average-Much help to ”location concepts” and“recognizable concepts”-Less help to other kinds of concepts-Detailed aerial photos are more helpful◆Future work
・ More categories
・ More features (e.g. color, HoG, Gabor)・ Other geo-info (e.g. geo-text: country
name, area name, GIS-info: population…)