Visual Event Mining from Geo-tweet Photos Takamu Kaneko and Keiji Yanai Department of Informatics, The University of Electo-Communications, Tokyo, Japan
The University of Electro-Communications
he University of Electro-Communications
Background •  Spread of smart phones -Geotagged photos •  Spread of Twitter -Real-time posting Increase of “geo-tweet photos”
◎ 2013 UEC Tokyo.
University of Electro-Communications

•  Detect events from Twitter stream -Weather, natural events -Festivals, sport games •  Understand events visually -Select representative photos -Map in a map Mapping events with the photo
© 2013 UEC Tokyo.
he University of Electro-Communication
Related Work: Twitter Event Mining •  Use only textual information •  Sakaki et al. [WWW 2010] -Regard Twitter users as social sensors -Estimate the location of natural events •  Lee et al. [ACM SIGSPATIAL WS 2010] -Divide target area into small sub-regions -Monitor the number of tweets
© 2013 UEC Tokyo.
University of Electro-Communication
Related Work: Twitter Photo Mining •  Use geo-tweet photos •  Yanai [ACM ICMR 2012] -Monitor the Twtiiter stream in real-time -Map geo-tweet photos on the map •  Nakaji et al. [ICME WS 2012] -Mine representative photos related keywords -Compare difference on places and time
© 2013 UEC Tokyo.
he University of Electro-Communications

1. Event keyword detection 2. Unification and complement 3. Event photo clustering 4. Mapping event with photos
© 2013 UEC Tokyo.
UEE  The University of Electro-Communications

day
◎ 2013 UEC Tokyo.

•  Part-of-Speech tagger -MeCab( http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html ) -TweetNLP( http://www.ark.cs.cmu.edu/TweetNLP/ ) • Divide target area into sub-regions -Grids by 1 degree latitude and longitude
•  Difference of user population -Difference of increase scale high population area low population area
© 2013 UEC Tokyo.
Event Keyword Detection • Evaluate area weight #userSmax + sd Warea #userSarea + sd •  Evaluate keyword score -over 50(JPN), 200(USA) Skw,date,area = (Nkw,date,area — Nkw,date–1,area)Warea
© 2013 UEC Tokyo.
he University of Electro-Communication
Unification and Complement •  Unification keyword -more than half of the same tweets -Use the highest score keyword “shuttle”, “Endeavor” ￼ “shuttle” •  Complement keyword -more than 80% of the same word -before and after the keyword “Festival” 🌸 “Music Festival”
© 2013 UEC Tokyo.

•  Image features -Bag-of-Features with SURF -Color histograms •  Ward method -a hierarchical clustering method -threshold is 300(both) E(C) = ∑ xec((xbof — xbof)²Wbof + (xrgb — xrgb)²Wrgb)
© 2013 UEC Tokyo.

•  Select a representative cluster -evaluate cluster score on representativeness #photosc² Vc Warea E(C) • Select a representative photo -from the maximum score cluster •  Eliminate lower score cluster -less than 5(JPN) , 20(USA)
© 2013 UEC Tokyo.
University of Electro-Communic
Experiments • Dataset 1 -in Japan -from Feb 10th, 2011 to Sep 30th, 2012 -about 3 million geo-tweet photos •  Dataset 2 -in US -from Jan 1st, 2012 to Dec  31st, 2012 -about 17 million geo-tweet photos
© 2013 UEC Tokyo.

Japan USA
© 2013 UEC Tokyo.

apa
S n
Cluster No.1 num="32" b_score="89.4698" c_score="127.6658" weight="1.9642" score="9.2631" Cluster No.3 num="1" b_score="0" c_score="0" weight="1.9642" score="0.0002" –
Cluster No.1 num="33" bof="142.56" color="167.23" weight="10.95" score="38.5"
he University of Electro-Communica

•  Map event in a map -Calculate coordinates of event -Correspond information and the photo •  Summary of results

© 2013 UEC Tokyo.
Re'sc'hEifise'aortisse't's set © 2013 UEC Tokyo. “Riveressed getWord’s ta’s stival” © 2013 UEC Tokyo. IEC  The University of Electro-Communications “sunset” © 2013 UEC Tokyo.
he University of Electro-Communication

• Event detection with geo-tweet photos -Detection of keywords -Selection of representative photos •  Detection results -258 events in Japan dataset -1676 events in US dataset •  Accuracy of representative photos -65.5% in Japan dataset -72.5% in US dataset
© 2013 UEC Tokyo.
University of Electro-Communical

• Flexible detection -Variable size of grid -Variable term for detection • Real-time detection •  Improvement of methods for selection representative photos
© 2013 UEC Tokyo.