Visualization of Real-world Eventswith Geotagged Tweet Photos
IEEE International Conference onMultimedia and Expo (ICME)
Melbourne , Australia, July 9th, 2012

Yusuke Nakaji and Keiji Yanai
Department of Computer Science,
The Univ. of Electro-Communications, Tokyo, Japan
1

Background2

 Wide spread of mobile devices having GPSand cameras
 iPhone, Android
 Microblog service: Users can send images withgeotagged messages on the spot.
 Twitter : new source of geotagged photos

3

Twitter vs Flickr: the most common geo-photo sources

is different fromgreatly.

WhenFlickr
Offline uploadTwitterInstant (online) upload

WhereUpload at home

On the spot

How many Many photos at once One photo (or small num.)
WhatSpecial personal
Everyday life (e.g. foods),event (e.g.travel)
Something special photoPurpose

Making online album Posting a photo message

Geo-Twitter photos helps us to understandwhat and where happens intuitively.

4Demo World Seer: A Real-time

Geo-Tweet Photo Mapping System[ICMR2012] http://mm.cs.uec.ac.jp/geotwphoto/

Geotag tweet photos are very helpful to understandwhat and where happens intuitively over the world.
• Photos are posted on the spot instantly.
• Photos are sometime reflected on the current“events”.

Objective

5

Detect geo-photos related to the specific eventsfrom large number of geo-tweets with photos“Event”＝ related to many people in broad areas
 Natural disaster Bad weather
 Festivals
 Something unusual Representative photos for each place =correspond to representative event in the place We can get to know what happens intuitively for eachplace.
Users of geo-tweet photos =“social distributed-cameras”

Related Work :Event detection from Twitter

Many works on event detection from geo-tweets
 Sakaki et al. [WWW2010]•Detect “earthquake”and“typhoon”,and estimate their location•Twitter = “Social sensor”
 Lee et al. [SIGSPATIAL WS 2010]•Detect local events such as festival
by considering #tweets, #users and user movments•Monitor the status of small local regions•If the status is normal or not (“burst of tweets”)

Existing works focused on analysis of geotagged texts.
They did not use photos at all.

6

Related work :Selection of representative photos


VisualRank [Jing et al. PAMI2008]
 Image ranking method based on PageRank Image similarity is used instead of link structure.

GeoVisual Rank [Kawakubo et al. WWW2010]
 Extension of VisualRank for geotagged photos.
 Consider location proximity as well as similarity
 Bias vector is computed based on geotags
We use this method in this work.

7

Overview of the proposed system

8

Image cluster : looks similar

Image collection
2. Image feature extraction
 Bag-of-Features with SURF
 RGB color histogram
3. Clustering of locations
 Mean-shift clustering4. Selection of representative
 GeoVisualRank
5. Showing similar images tothe representative
photos as “event photos”
West area
1.

East area

Basically the method is similar to the work on landmark photo selection [Cradall et al WWW2009].

Difference of the photos = Difference of the situations

9Data collection of geo-tweet photos
 Via Twitter Streaming API provided by [Data Collection System] Collecting geo-photo tweets from theTwitter stream continuously for 18 months
 Currently it has about 32,000,000 geophoto tweets (200,000 photos/days )•[Yanai ICMR 2012] World Seer: A Real-time
Geo-Tweet Photo Mapping System Photos are downloaded from twitter

photo sites such as pic.twitter.com,t, yfrom.co and.

Image features10 Before extracting features, we search

the DB from images with given
conditions related to locations, time
periods, and keywords.
 Bag-of-Features with SURF
 RGB color histogram

We use them by fusing with equal weights.

11

Clustering of geotagged locations
 Mean-Shift clustering Clustering based on kernel density

estimation
 Given the radius of clusters instead of “k”

r=50kmr=100km12
GeoVisualRank [Kawakubo et al. WWW2010]
 Modification of VisualRank (random walk on link-graph) r ＝ αSr + (1－α)p(0≦α≦1)

commonly α≧0.8
S: similaritymatrix

p:bias vector

 Simliarity: histogram intersection of BoF andcolor Bias: based on the locations (proportional tothe distance to the given reference point.)

13

Temporal-extended GeoVisualRank

Modify GeoVisualRank so as to take into accounttime proximity
 Bias vector: pgeo,ptime• 𝐶 𝑔𝑒𝑜 : imgs in given geo cluster• 𝐶 𝑡𝑖𝑚𝑒 :imgs in given time cluster• 𝑛𝑔𝑒𝑜 = 𝐶𝑔𝑒𝑜 , 𝑛𝑡𝑖𝑚𝑒 = |𝐶 𝑡𝑖𝑚𝑒 |𝒑 = 𝛽𝒑𝒈𝒆𝒐 + 1 − 𝛽 𝒑𝑡𝑖𝑚𝑒(𝛽 = 0.5)

Time cluster (6 hour/each -> 4clusters)
 late-night…… 0am～6am
 morning…… 7am～12pm
 afternoon…… 12pm～6pm
 night…… 6pm～0pm(midnight)

Experiments: case study

14Case study for the three kinds of “events”with Japan “Typhoon”•Given keyword: “typhoon”, term: Sept. 2011
•Location: Japan•# hit geo-tweet photos : 616
 “New year’s day”

•Keywords: Japanese words related to “new year”•(正月|おせち|日の出|はつひので| 初詣|はつもうで）•Term: Jan. 1st – 20th 2013•#hit geo-tweet photos : 1400
 “Big earthquake”(March 11th 2011)•Keywords: none, location: Japan•Term: March 11th-12th 2011•#hit geo-tweet photos : 1080

“Typhoon”

15
2011/09/01～
2011/09/30

 Difference depending on locations

＠Nagoya＠Akita

“New year’s day”
Transition of event photos
 “New year’s day” in Tokyo16
2012/01/01～
2012/01/20
Late-nightmorning
afternoonnight“Big earthquake”
17
2011/03/11～
2011/03/12

Difference of the situation
 Eastern part of Japan was unusual and much confused.

＠Okayama

＠Tokyo19Conclusions
 We proposed using geotagged tweet photos

for event photo detection.
 “social distributed-cameras”
 We used time-location VisualRankto detect event photos
on the given location and the given time.
 We can get to know about“events” intuitively .
 Different from text-based event detection.

Future work20

 Real-time image-based event

detection
 Use all the images over the world

e.g. comparison of “new year’s event”
over the world Use non-geotagged tweet photos as well
 About 50～100 times as many photos as

geo-photos are being posted.

21