31

Geotagged Image Recognition
beachsea

golf
store park
highway

by Combining Three Different Kinds ofGeolocation Features

Background & Objective◆ 1/0 classification for 28 classes
by 5-fold cross validation
・ 200 positive & 200 negative images for each class
Average・ Evaluated by average precision precision

Given concepts

precision“Mountain”
latitude=42.844202
latitude=42.844202
longitude=140.8528
latitude=42.844202
longitude=140.8528
latitude=42.844202
99
longitude=140.8528
latitude=42.84420
99
longitude=140.8528
99
longitude=140.85289
99

Level 3
Level 4497m x 497m 1.91km x
1.91km256x256
256x256

Level 27.64km x
7.64km256x256

latitude=40.84423
longitude=135.18353

Level 130.8km x
30.8km256x256

Can a geotag help image recognition ?
No ! when using them only as 2D vectors.

Yes ! It can help image recognition
by converting it into aerial photos.

Web map services[1] J. Luo, J. Yu, D. Joshi, and W. Hao.
Event recognition: Viewing theworld with a third eye. In Proc. ofACM International Conference
Multimedia, 2008.

sunset, cherry blossom,statue, car, bicycle, graffiti,
red leaves, festival,
vending machine
costume-play festivalpark, garden,landscape(FD) Foodsushi, ramen noodle

・ Classification results (AP, %)+6.69 by VT+3.53
by VAT+3.30
by VAT

+0.84
by VT

+1.69
by VT
+4.35
by VAT +2.89
by VT+3.16
by VT
+4.11
by VATD

[3] K. Yaegashi and K. Yanai. Can geotags
help image recognition ? In Proc. ofPacific-Rim Symposium on Image andVideo Technology, 2009.

Geo-text(T) is more helpful than aerial(A) in many cases.
visual words

BoF[2] Joshi, D., Luo, J.:. In: Proc. of Inferring generic activities and events fromimage content and bags of geo-tagsACM International Conference on Imageand Video Retrieval.

Question ?
1) To what extent can geotags help ?
2) What kinds of categories are
geotags effective for ?

We evaluate the contribution of three
features for image recognition byusing Multiple Kernel Learning (MKL).

BoF

BoF

LatitudeLongitude(35.486795,140.423641)

Geo-information texts

building officepost temple
school・・・

Split intowords

Top 2000words (bins)

Histogramat (35.486795,140.423641)
Shirahama inn
Shirahama shrine
Hotel Izukyu
Shirahama Ohama beachFirmiana simplex habitat
Shimoda prince hotel
Shimoda Shirahama post office
Shirahama nursery
Municipal Shirahama elem. school
Shirahama central beachShimoda view hotel
Route 135
Jomo gas station

Learning & Classifying
 Multiple Kernel Learning

Level 2
Level 17.64km x 7.64km 30.8km x 30.8km
256x256
256x256

Combined kernel
photo BoF kernel

aerial photo(Lv1) ker.

Aerial photo(Lv4) ker.

Combined kernel

BoF

BoF

BoF

BoF
Kernel weight (to be estimated by MKL)

Aerial(A)Geo-location(G)mountain rodge
mountain highway
convenience store

Reverse geo-coding
Text(T) by local search API

Date(D)MKL

Histogram for month and hour

Jan 0.25
Feb 0.5
Mar 0.25・・・
Dec 0.25
1 0.5
2 0.25
3・・・
24

Time and date the photo was taken

BoF HSV Aerial Geo Text Date
Vis (BoF + HSV)✔ ✔
Vis + G (VG)✔ ✔
✔
Vis + T (VT)✔ ✔
✔
BoF + A (BA)✔
✔
Vis + A (VA)✔ ✔
✔
Vis + A + T (VAT)✔ ✔
✔
✔
Vis + G + T + D (VGTD) ✔ ✔
✔
✔
✔
Vis + A + T + D (VATD) ✔ ✔
✔
✔
✔
All (All)
✔ ✔
✔
✔
✔
✔

Featurecombinations ofinput for MKL

Somecombinations

・ Contribution weights by MKL

Yahoo! Japan Local Search API

・ Can handle “a combined kernel” which is a linear
combination of kernels.
・Can estimate kernel weights and SVM model
parameters simultaneously.
・Can integrate features by assigning one feature
to one kernel.

latitude=42.844202
longitude=140.852899

Level 3
1.91km x
1.91km256x256

BoF

In all the cases, text(T) weights are larger than aerial(A)weights. In LS and GE, aerial(A) weights are relatively large.

・ Examples ofgeo-text features(top 5 elements of theaverage word vector
over the concept)

・ Is an extenstion of a SVM .

MethodLevel 4497m x 497m
256x256

BoF◆ Geo-information textual features

Yes ! It can also help image recognition
by reverse geo-coding.

BoF(SP) Space

Five BoFvectors
for each image

frequency

Represent geographical context

HSV(TD) Timedependent(OA) Outdoor artifact

cat, bird, flower(1) Sample points by grid sampling (every 10px)
and describe local patterns aroundthe sampled points with SIFT [Lowe 2004](2) Generate codebooks by k-means(size; 1000) andconvert images into BoF vectors
by voting to nearest codewords

lake, river, beachbridge, shrine, building,castle, railroad(CR) Creature◆ Bag-of-features (BoF) (local pattern)(GE) Geographicalcorresponding aerial photos

latitude=42.84422
longitude=140.85289

BoF(LM) Landmark

Disneyland
Tokyo Tower

FeaturesFeaturesgeotag = (latitude, longitude)

recall(LS) Location
specific

 Gather 4 levels of aerial photos foreach geotagged photo.
latitude=23.23202
latitude=37.24423
longitude=143.89249 longitude=137.68353

forestExperiments
Experiments

Data Collection

 Geotagged Photos are easy to obtain.

lodge

Keita Yaegashi and Keiji Yanai

The University of Electro-Communications, Tokyo, JAPAN

latitude=42.84422
longitude=140.85289

ski parkpark
lodge
mountain

Locationwords

Conclusions
Conclusions◆Analyzed contribution ratios ofaerial images for image recognition
using Multiple Kernel Learning(MKL)-AP was improved by 3.16 % on average-Geo-textual features and aerial features
improved AP for most of the concepts-Less helpful for “food” concepts-Texts are more helpful than aerial photos
for our dataset (limited within Japan)(depending on availability of rev-geocoding)◆Future work
・ More large-scale experimentswith much more categories
・ multi-class experiments