Twitter Event Photo Detection
Using Geotagged Tweets andNon-geotagged Photo Tweets
Takamu Kaneko and Keiji Yanai
Department of Informatics,
The University of Electro-Communications,Tokyo, Japanⓒ 2015 UEC Tokyo.

Background• Spread of smart phones-Geotagged photos• Spread of Twitter-Real-time posting

A huge number of “geotags” and “photos”
in the Twitter streamⓒ 2013 UEC Tokyo.

Twitter Event Photo Mining[ICME WS 2013]• Mine the photos related to the events
happened in the specific areas and times

The results of detected event photos in 2012
ⓒ 2013 UEC Tokyo.

Objective of
Twitter Event Photo Detection• Detect events from Twitter stream- Weather, natural events- Festivals, sport games• Understand events visually-Select representative photos-Visualize eventson an on-line map
Mapping events with the photoⓒ 2013 UEC Tokyo.

Related Work: Twitter Event Mining• Many works which use only text analysis• Sakaki et al. [WWW 2010]-Regard Twitter users as social sensors-Estimate the location of natural events• Lee et al. [ACM SIGSPATIAL WS 2010]-Divide target area into small sub-regions-Monitor the number of tweets
Detect events and their locations from geotagged tweetsⓒ 2013 UEC Tokyo.

Related Work: Twitter Photo Mining• Use geo-tweet photos• Nakaji et al. [ICME WS 2012]– Mine representative photos for given event words– Compare differences of events depending onplaces and time

• Our previous work [ICME WS 2013]– Event keyword detection + photo clusterting– Use only “geotagged photo tweets”• The number of images are limited.
• About 2 or 3% ofⓒ 2013
allUECthetweets
Tokyo.

Contributions of this work
• Extend the previous work so as to use
not only geotagged photo tweets but also
geotagged non-photo tweets andnon-geotagged photo tweets

Geotagged phototweets

Geotagged
text tweets
For event word detectionⓒ 2013 UEC Tokyo.

Non-geotagged
photo tweets
For event photo detectionDifferences to the previous work
Types of tweets

Geotagged phototweets
Geotagged texttweets
Non-geotagged
photo tweet
Text analysis method
Image features

Previous[1]This work○○×

20 timeslargervolume○×○Morphologicalanalysis
BoF + color hist.

N-gramDCNN[1] T. Kaneko and K. Yanai. Visual event mining from geo-tweet photos. In
Proc. of IEEE ICME Workshop on Social Multimedia Research (SMMR) ,2013.
ⓒ 2013 UEC Tokyo.

Overview
1. Event keyword detection
2. Location estimation ofnon-geotagged photo tweets

3. Event photo clustering
4. Mapping event with photosⓒ 2013 UEC Tokyo.

Event Keyword Detection• Examine change of daily frequency⇒ burst detectionⓒ 2013 UEC Tokyo.

Event Word and Region Detection• N-gram: words by more than 5 unique users
e.g. “I’m in Japan Rock Festival.”⇒ Japan, Rock, Festival , Japan Rock,Rock Festival, Japan Rock Festival• Divide target area into sub-regions-Grids by 0.5 degree latitude and longitude
USJapanⓒ 2013 UEC Tokyo.

Event word detection• Use both geotagged photo tweets andgeotagged non-photo text tweets
1. Count unique users
who tweets given word
combinations
within the specific region in the given day
2. Compare the user counts with theprevious day’s counts⇒ burst detectionⓒ 2013 UEC Tokyo.

Estimating locations of nongeotagged photo tweets• The number of geotagged photo tweets isvery limited (2-3%) ⇒ use non-geo photo as well
• Judge if a non-geotagged photo tweet
including the given event words is posted
from the area of the event- two class classification (in area/out of area)• Use geotagged photo tweets as training data
- positive: event-related tweets- negative: tweets including event words posted
from out of the areaⓒ 2013 UEC Tokyo.

Estimate Locations of Nongeotagged Photos using bothtexts and image features• Text: Naive Bayes(NBNN, Boiman et al. CVPR2008)• Image: Naïve Bayes Nearest Neighbour– Local feature(SIFT) based image classification– The sum of cos similarity between local featuresand nearest features in training sets.
nv

d j  NN ci 1j 1d j NN ccˆ  arg max P(c) P( xi | c)
ctext
UEC Tokyo.
𝑐 ∈ {𝑒𝑣𝑒𝑛𝑡 𝑡𝑤𝑒𝑒𝑡, 𝑢𝑛𝑟𝑒𝑙𝑎𝑡𝑒𝑑ⓒ 2013𝑡𝑤𝑒𝑒𝑡}

NN cimageEvent Photo Clustering• Image features- DCNN activation features (4096 dim)(Alexnet pretrained with ImageNet1000)• Photo clustering: Ward method- a hierarchical clustering method
𝐸 𝐶 =𝑥∈𝐶 ( 𝑥𝐵𝑜𝐹 − 𝑥𝐵𝑜𝐹

2𝑤𝐵𝑜𝐹 + 𝑥𝑅𝐺𝐵 − 𝑥𝑅𝐺𝐵 2 𝑤𝑅𝐺𝐵 )• Representative photo– A photo which is the closest to the center ofthe largest clusterⓒ 2013 UEC Tokyo.

Expetiments• Tweets Dataset- Collect via Twitter Streaming API- Limit Tweets posted only from Japan- train: Whole, 2012 (training for NB/NBNN)- test: August, 2012 (target for event detection)(We have many festivals in August.)
Geotagged photo
Only Geotag
Only Photo2012 whole
2,645,709
24,715,962ⓒ 2013 UEC Tokyo.

Aug. 2012
255,455
2,102,151
3,367,169

Detected Event Words (8/2012)
Event word

date

Locatio (lat,lng)

Eval. score

Firework

20120801

33,129.5

297.7

rainbow

20120801

34,134.5

229.1

ROCK IN JAPAN20120803

36,140430.3

Ayu Festical20120804

34.5,135.5

265.1

Nebuta Festival

20120806

40.5,140

255.7

Awa Festival

20120814

34,134

589.8

Thunder storm

20120818

34,135367.5

Blue moon

20120831

34.5,136269.7ⓒ 2013 UEC Tokyo.

Events with representative
event photos on Google Maps

ⓒ 2013 UEC Tokyo.

Event Photos
on “Nagaoka firework festival”ⓒ 2013 UEC Tokyo.

「Nebuta Festival」

Red box:Geo-photo
Yellow box:Non-geo
photosⓒ 2013 UEC Tokyo.

「Biwa Lake Fireworks」

Red box:Geo-photo
Yellow box:Non-geo
photosⓒ 2013 UEC Tokyo.

「Sendai Tanabata Festival」

Red box:Geo-photo
Yellow box:Non-geophotosⓒ 2013 UEC Tokyo.

Summary of event detection# events
Event precision(%)

This work
310
81.3

previous
35
77.1

Photo precision(%)

88.7

65.5

The num. event: 10 times
 イベントの精度は約４％向上
 代表画像の適合率は大幅に改善

ⓒ 2013 UEC Tokyo.

Conclusions• Proposed a method to detect events andcorresponding photos from Twitter• Proposed a method to detect event photos
from non-geotagged photos- Use both text and visual featureswith NB and NBNN

• Future work
- Detailed evaluation- Realtime event detectionⓒ 2013 UEC Tokyo.

ⓒ 2013 UEC Tokyo.

Unification and Complement• Unification keyword-more than half of the same tweets-Use the highest score keyword“shuttle”, “Endeavor”“shuttle”• Complement keyword-more than 80% of the same word
-before and after the keyword“Festival”“Music Festival”ⓒ 2013 UEC Tokyo.

Event Photo Clustering• Image features-Bag-of-Features with SURF
-Color histograms• Ward method-a hierarchical clustering method-threshold is 300(both)
𝐸 𝐶 =

22𝑤
(𝑥−𝑥𝑤
+𝑥−𝑥
𝐵𝑜𝐹
𝐵𝑜𝐹
𝑅𝐺𝐵𝑅𝐺𝐵𝑅𝐺𝐵 )𝐵𝑜𝐹𝑥∈𝐶ⓒ 2013 UEC Tokyo.

Event Photo Selection• Select a representative cluster-evaluate cluster score on representativeness#𝑝ℎ𝑜𝑡𝑜𝑠𝐶 2
𝑉𝐶 =𝑊𝑎𝑟𝑒𝑎𝐸(𝐶)• Select a representative photo-from the maximum score cluster• Eliminate lower score cluster-less than 5(JPN) , 20(USA)ⓒ 2013 UEC Tokyo.

“River’sResult“BalloonEdge“rainbow”
of US
MusicFiesta”
datasetFestival”ⓒ 2013 UEC Tokyo.

“sunset”ⓒ 2013 UEC Tokyo.

Conclusion• Event detection with geo-tweet photos-Detection of keywords-Selection of representative photos• Detection results-258 events in Japan dataset-1676 events in US dataset• Accuracy of representative photos-65.5% in Japan dataset-72.5% in US datasetⓒ 2013 UEC Tokyo.

Future Works• Flexible detection-Variable size of grid
-Variable term for detection• Real-time detection• Improvement of methods for selection
representative photosⓒ 2013 UEC Tokyo.

More results…
• ICME demo session on July 18thⓒ 2013 UEC Tokyo.