Visual Event Mining
from Geo-tweet Photos
Takamu Kaneko and Keiji Yanai
Department of Informatics,
The University of Electo-Communications,Tokyo, Japanⓒ 2013 UEC Tokyo.

Background• Spread of smart phones-Geotagged photos• Spread of Twitter-Real-time posting

Increase of “geo-tweet photos”ⓒ 2013 UEC Tokyo.

Objective• Detect events from Twitter stream-Weather, natural events-Festivals, sport games• Understand events visually-Select representative photos-Map in a map

Mapping events with the photoⓒ 2013 UEC Tokyo.

Related Work: Twitter Event Mining• Use only textual information• Sakaki et al. [WWW 2010]-Regard Twitter users as social sensors-Estimate the location of natural events• Lee et al. [ACM SIGSPATIAL WS 2010]-Divide target area into small sub-regions-Monitor the number of tweetsⓒ 2013 UEC Tokyo.

Related Work: Twitter Photo Mining• Use geo-tweet photos• Yanai [ACM ICMR 2012]-Monitor the Twtiiter stream in real-time-Map geo-tweet photos on the map• Nakaji et al. [ICME WS 2012]-Mine representative photos related keywords-Compare difference on places and timeⓒ 2013 UEC Tokyo.

Overview
1. Event keyword detection
2. Unification and complement

3. Event photo clustering
4. Mapping event with photosⓒ 2013 UEC Tokyo.

Event Keyword Detection• Examine change of daily frequencyⓒ 2013 UEC Tokyo.

Event Keyword Detection• Part-of-Speech tagger-MeCab(http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html)-TweetNLP(http://www.ark.cs.cmu.edu/TweetNLP/)• Divide target area into sub-regions-Grids by 1 degree latitude and longitude
USJapanⓒ 2013 UEC Tokyo.

Event Keyword Detection• Difference of user population-Difference of increase scalehigh population areaⓒ 2013 UEC Tokyo.

low population area

Event Keyword Detection• Evaluate area weight#𝑢𝑠𝑒𝑟𝑠𝑚𝑎𝑥 + 𝑠𝑑
𝑊𝑎𝑟𝑒𝑎 =#𝑢𝑠𝑒𝑟𝑠𝑎𝑟𝑒𝑎 + 𝑠𝑑
• Evaluate keyword score-over 50(JPN), 200(USA)𝑆𝑘𝑤,𝑑𝑎𝑡𝑒,𝑎𝑟𝑒𝑎= (𝑁𝑘𝑤,𝑑𝑎𝑡𝑒,𝑎𝑟𝑒𝑎 − 𝑁𝑘𝑤,𝑑𝑎𝑡𝑒−1,𝑎𝑟𝑒𝑎 )𝑊𝑎𝑟𝑒𝑎ⓒ 2013 UEC Tokyo.

Unification and Complement• Unification keyword-more than half of the same tweets-Use the highest score keyword“shuttle”, “Endeavor”“shuttle”• Complement keyword-more than 80% of the same word
-before and after the keyword“Festival”“Music Festival”ⓒ 2013 UEC Tokyo.

Event Photo Clustering• Image features-Bag-of-Features with SURF
-Color histograms• Ward method-a hierarchical clustering method-threshold is 300(both)
𝐸 𝐶 =2𝑤
2𝑤(𝑥−𝑥+𝑥−𝑥
𝐵𝑜𝐹
𝐵𝑜𝐹
𝑅𝐺𝐵𝑅𝐺𝐵𝑅𝐺𝐵 )𝐵𝑜𝐹𝑥∈𝐶ⓒ 2013 UEC Tokyo.

Event Photo Selection• Select a representative cluster-evaluate cluster score on representativeness#𝑝ℎ𝑜𝑡𝑜𝑠𝐶 2
𝑉𝐶 =𝑊𝑎𝑟𝑒𝑎𝐸(𝐶)• Select a representative photo-from the maximum score cluster• Eliminate lower score cluster-less than 5(JPN) , 20(USA)ⓒ 2013 UEC Tokyo.

Experiments• Dataset 1-in Japan-from Feb 10th, 2011 to Sep 30th, 2012
-about 3 million geo-tweet photos• Dataset 2-in US-from Jan 1st, 2012 to Dec 31st, 2012
-about 17 million geo-tweet photosⓒ 2013 UEC Tokyo.

Results of Keyword Detection
Keyword

DateKeyword

Datesnow

11/02/2011

snow

09/01/2012

earthquake

11/03/2011

sunset13/01/2012

fireworks

06/08/2011

Grammy12/02/2012

typhoon21/09/2011

Valentines

14/02/2012

Mt. Fuji

24/09/2011

SXSW

09/03/2012

Apple

06/10/2011

Easter

08/04/2012

eclipse10/12/2011

shuttle17/04/2012

illumination

10/12/2011

WWDC

10/06/2012Christmas

24/12/2011

hurricane

26/08/2012

New years eve

31/12/2011

rainbow

05/09/2012

sunrise01/01/2012

49ers

18/10/2012

firefly

06/05/2012

NYE

31/12/2012

JapanUSAⓒ 2013 UEC Tokyo.

USJapan

Results of Unification & complement
BeforeAfter unification

After complement

Sky, TreeSkySky Treefireworks, Yodo-river

fireworks

fireworks

eclipse, solar

eclipsesolar eclipse

marathon

marathon

Tokyo marathon

Dome

Dome

Sapporo Dome

BeforeAfter unification

After complement

Rangers, Ballpark

Rangers

Rangers Ballpark

West, WWDC, Apple

WWDC

WWDC

Golden, Gate, Bridge

GoldenGolden Gate Bridge

Square, TimesTimes

Times Square

Carnival, Electric, Daisy

Electric

Electric Daisy Carnivalⓒ 2013 UEC Tokyo.

“fireworks” photo clustersⓒ 2013 UEC Tokyo.

“cherry blossoms” photo clusters
daytimenightⓒ 2013 UEC Tokyo.

“Stanford Stadium” photo clusters

soccer gamefootball gameⓒ 2013 UEC Tokyo.

Mapping Results• Map event in a map-Calculate coordinates of event-Correspond information and the photo• Summary of results
JapanUSA# events 258

1676

accuracy 65.5%

72.5%ⓒ 2013 UEC Tokyo.

Result“cherry“Fire“fireworks”
of JapanFestival”
blossoms”
datasetⓒ 2013 UEC Tokyo.

“River’sResult“Balloon
Edge“rainbow”
of USMusicFiesta”
datasetFestival”ⓒ 2013 UEC Tokyo.

“sunset”ⓒ 2013 UEC Tokyo.

Conclusion• Event detection with geo-tweet photos-Detection of keywords-Selection of representative photos• Detection results-258 events in Japan dataset-1676 events in US dataset• Accuracy of representative photos-65.5% in Japan dataset-72.5% in US datasetⓒ 2013 UEC Tokyo.

Future Works• Flexible detection-Variable size of grid
-Variable term for detection• Real-time detection• Improvement of methods for selection
representative photosⓒ 2013 UEC Tokyo.

More results…
• ICME demo session on July 18thⓒ 2013 UEC Tokyo.