Visual Event Mining from the Twitter Stream
The University of Electro-Communications, Tokyo, Japan
Takamu Kaneko and Keiji Yanai

ObjectiveBackground

Spread of smart phones (camera & GPS)- helpful to understand events intuitively- easy to obtain photos and geotags


Detect event photos with location info. from Twitter

Spread of Twitter- Real-time posting of event-related tweets

A lot of “geotags” or “photos” on Twitter. However,the number of geotagged photo tweets are limited (2-3%).

Use not only geotagged photo tweets but alsogeotagged non-photo tweets andnon-geotagged photo tweets
Only-geo (w/o photo) tweets
for event photo detection
Only-photo (w/o geo) tweets

Main contribution of this work

Proposed method
1. Event burst detection

Evaluationof“event-ness”
ofnon-geophotosEvent detection on words and areasDivide target area into sub-regions(text-based analysis with geotagged tweets)- Grids by 0.5 degree latitude
and longitude

Event geotagged photos

2. Location estimation of
non-geotagged photo tweets

Build N-gram (N≦5) of all the geo-tweet
texts regarding # of unique users
within each grid in each unit time (1day)(geotag and visual feature based analysis)e.g. “I’m in Japan Rock Festival.”⇒ Japan, Rock, Festival , Japan Rock,Rock Festival, Japan Rock Festival

Augmented event geotagged photos

3. Event photo clustering andrepresentative photo selection
4．Visualizing events on the map

Burst detection- compare each 1-day 1-gird N-gram with
1-year average: their ratio = burst scoreness” in given area
- text: Naïve Bayes (NB)
- image: Naïve Bayes Nearest Neighbor (NBNN)

Binary classification (if given photo is related to events or not)nv

d j  NN ci 1j 1

d j NN ccˆ  arg max P (c) P ( xi | c)
ctext𝑐 ∈ {𝑒𝑣𝑒𝑛𝑡 𝑝ℎ𝑜𝑡𝑜,non-event 𝑝ℎ𝑜𝑡𝑜}

imageEvent Photo ClusteringImage features- DCNN activation features (4096 dim)
(DCNN: Deep Convolutional Neural Network)(Alexnet pretrained with ImageNet1000)

Adjusting weight# usersmax  s
Wi , j # usersi , j  s

Location estimation of non-geo photo tweets
containing event words by combining
both texts and image features Evaluating “event-1-year avg.
Photo clustering: Ward method- a hierarchical clustering method

Commonness score

Com (w)

Burst score ≥ 𝑡ℎ(= 200)
Event wordcandidates

Burst score

𝐶𝑜𝑚(𝑤)



Representative photo selection- a photo which is the closest to the centerof the largest cluster

Experimental Results
Comparison with the baseline (= using only geo-photo tweets)

Data collected via Twitter Streaming API
Tweets within Japan on August. (Many festivals in August. )2012 whole Aug. 2012(training for geo. estimation)(for evaluation)

2,645,709
24,715,962

255,455
2,102,151
3,367,169

Geo-photo tweets
Only-geo tweets
Only-photo tweets

Examples of detected keywords
Event word
Firework
rainbow
ROCK IN JAPANAyu Festical
Nebuta Festival
Awa Festival
Thunder storm
Blue moondate
20120801
20120801
20120803
20120804
20120806
20120814
20120818
20120831

Loc. (lat,lng)
33.0, 129.5
34.0, 134.5
36.0, 140.0
34.5, 135.5
40.5, 140.0
34.0, 134.0
34.0, 135.0
34.5, 136.0# events
Event precision (%)
Photo precision (%)

proposed
310
81.3
88.7

Event map with representative photos

baseline
3577.1
65.5↑ increased↑ improved↑ improved

Event photos

Burst score
297.7
229.1
430.3
265.1
255.7
589.8
367.5
269.7

“Nebuta Festival”“Biwa lake fireworks”

More event photos can be seen at
http://mm.cs.uec.ac.jp/kaneko-t/tw/adv/Conclusion
A new method on Twitter event photo detection
using geo-photo/only-geo/only-photo tweets
with location estimation of non-geo photos
 Outperformed the baseline regarding # and prec.
Future Work
Variable grid size and time unit for event detection More large-scale experiments with various languages


Building world-wide event photo database (regular, accidental)Real-time event photo detection