VISUAL EVENT MINING FROM GEO-TWEET PHOTOS
Takamu Kaneko and Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585, Japan
ABSTRACTIn this paper, we propose a system to mine events visuallyfrom the Twitter stream by making use of “geo-tweet photos”
which are tweets including both geotags and photos. Someworks on event mining which utilize geotagged tweets havebeen proposed so far. However, they used no images but only
textual analysis of tweet texts. In this work, we detect eventsusing visual information as well as textual information. In
the experiments, we show some examples of detected eventsand their photos such as “rainbow”, “ﬁreworks” and “Tokyo
ﬁreﬂy festival”.
Index Terms— Twitter, Geotagged Images, Event Mining, Geo-Photo Tweet
1. INTRODUCTION
Twitter is a unique microblog, which is different from conventional social media in terms of its quickness and on-the-spotness. Many Twitter’s users send messages, which is commonly called “tweets”, to Twitter on the spot with mobile
phones or smart phones, and some of them send photos andgeotags as well as tweets. Most of the photos are sent to Twitter soon after taken, and geotags attached to tweets generally
represent the current locations of the Twitter’s users.
In fact, we collected about 200 thousand geotagged photos a day on average in December 2012 via Twitter Streaming
API. Such geotagged photo tweets are sometimes posted fromthe places where some events happen such as festival, sport
game and earthquake. Then, we can regard Twitter’s users as
distributed image sensors on events.
We think that Twitter is a promising data source of geotagged of photos, while Flickr has been the most popular
data source of geotagged photos in the research communityof multimedia so far. Since the characteristic of Twitter isquickness and on-the-spot-ness, the photos on Twitter are different from the photos on Flickr. Flickr has many travel photos, while Twitter has many photos related to everyday life
such as food, weather, street scene and some events. Therefore, we think that geo-tweet photos are more useful to understand what happens currently over the world than Flickr
geo-photos.

In this paper, we propose a system to mine events visuallyfrom the Twitter stream. To do that, we pay attention to thetweets having both geotags and photos. We call such tweets
as “geo-tweet photos”. So far some works on event mining
which utilize geotagged tweets have been proposed. However, they used no images but only textual analysis of tweet
texts. On the other hand, in this work, we detect events using visual information as well as textual information. In theexperiments, we show some examples of detected events andtheir photos such as “rainbow”, “ﬁreworks” and “Tokyo ﬁreﬂy festival”.

2. RELATED WORK
Sakaki et al. [1] regarded Twitter users as social sensors whichmonitor and report the current status of the places where theusers are. They proposed a system which estimates the location of natural events such as typhoons and earthquakes. They
used geotagged tweets to estimate event locations.
Lee et al. [2] proposed an event detection system fromgeotagged tweets. They divided target areas into small subregions, and monitor the number of tweets posted from each
sub-region. They regarded the areas where the number oftweets rose suddenly as the event areas where some events
happened. In our work, we also examine the daily changes on
the number of tweets of each area to detect events.
In these two works, they used textual information extracted from tweets and geo-location information embedded
in geotags, and did not use visual information which can beextracted from tweet photos.
As works on geo-tweet photos, Yanai proposed World
Seer [3] which can visualize geotagged photo tweets on the
online map in the real-time way by monitoring the Twitter
stream. Nakaji et al. [4] proposed a system to mine representative photos related to the given keyword or term from alarge number of geo-tweet photos. They extracted representative photos related to events such as “typhoon” and “NewYear’s Day”, and successfully compared them in terms of thedifference on places and time. However, their system needs
to be given event keywords or event term by hand. Then, in
this paper, we integrate a method to select representative event
photos with automatic detection of event keywords.

3. OVERVIEW
To detect events visually from Twitter stream, we monitor
Twitter stream to pick up tweets having both geotags andphotos, and store them into a geo-photo tweet database. We
apply to this database the proposed visual event mining system which consists of event keyword detection, event photoclustering and representative photo selection. The processing
steps of the proposed system are as follows:(1) Detect event keyword candidates which frequently appear in the tweets posted from speciﬁc areas in speciﬁc
days.
(2) Unify and complement detected event keywords(3) Select geo-tweet photos corresponding to the event keywords by image clustering(4) Select a representative photo to each event(5) Show the detected events with their representative photos on the map
Note that the current system assumes the tweet messages
written by Japanese language, since keyword extraction needs
to be taken into account of the characteristics of target language. However, it is not so difﬁcult to extend the proposedsystem to other languages.

Fig. 1. The grids dividing the Japanese Islands. Each of themis a unit area for event detection.

who posted tweets in the indicated day and area, respectively.
Wi,j represents a weight to adjust the scale of the number ofdaily tweet users, which is deﬁned in the following equation:
Wi,j =

4. PROPOSED METHOD
In this section, we explain the detail of each step of the proposed system.
4.1. Event keyword detection
Tweet messages are written in sentences in general. To detect event easier, we extract noun keywords from each tweet
message. To do this, we apply the Japanese morphologicalanalyzer, MeCab 1 , and extract only noun words as keywordsof each tweet after stop-word removal.
To detect events, we search for bursting keywords by
examining change of the daily frequency of each keywords
within each unit area. The area which is a location unit todetect events are deﬁned in the grids by one degree latitude
and one degree longitude as shown in Fig.1. In case that thedaily frequency of the speciﬁc keyword within one grid area
increases greatly, we consider that an event related to the speciﬁc keyword happened within the area in that day.
We set up the following equation to decide if an event
related to the given keyword happens in the given area. We
consider that an event happens if Sk,d,i,j is more than the predeﬁned threshold, which was set as 50 in the experiments.
Sk,d,i,j = (Nk,d,i,j − Nk,d−1,i,j )Wi,j ,(1)where k, d, i, j and Nk,d,i,j represent an index of a keyword,
an index of date, an index of area grids, and the number users
1 http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html

M +s,Ni,j + s(2)

where i, j, N , M and s represents the index of grids, the number of unique users in the given grid, the maximum numberof unique users among all the grids (which is equivalent to
the number of Tokyo area users), and the standard deviationof user number over all the grids. With this adjusting weight,
we can detect events from the areas where tweet users are notso many as well as the areas where so many tweets are always
being posted such as Tokyo area.
4.2. Keyword uniﬁcation and complement
In the previous step, we limited an event keyword to a single noun keyword. However, since some events are represented by compound keywords, the same event are sometimes
detected by several keywords independently. In such case,
we unify them into a compound keyword related to the same
event according to the following heuristics:(1) In case that more than half of the tweets related to a
speciﬁc event keyword overlaps the tweets related toanother event keyword, the former keywords are integrated and replaced with the latter keywords.
• E.g. “rain” and “typhoon” ⇒ “typhoon”(2) In case that words just after or before the detected event
keyword are the same in more than 80% tweets including the keyword, such words are regarded as being partof a compound event keyword.
• E.g. “Tokyo”, “sky” and “tree” ⇒ “Tokyo Skytree”

4.3. Event photo clustering and representative photo selection
Until the previous step, event keywords and their corresponding tweets have been selected. In this step, we carry out clustering of the photos embedded in the selected event tweets
and selecting representative ones from them.
As image features, we use bag-of-features (BoF) with
densely-sampled SURF local features and 64-dim RGB color
histograms. SURF keypoints are sampled with every 10 pixels in the scale 5, 10 and 15. The size of the codebook for BoF
was set as 1000. Both feature vectors are L1-normalized.
For clustering photos, we use the Ward method which isone of agglomerative hierarchical clustering methods. It creates clusters so to minimize the total distance between thecenter of each cluster and the cluster members. It merges thecluster pairs which bring the minimum total error calculated
in the following equation one by one.
d(C1 , C2 ) = E(C1 ∪ C2 ) − E(C1 ) − E(C2 )(3)

In general, E(C) is deﬁned as the total distance between thecenter and the members of the cluster. Since we use two kindsof visual features, we deﬁned E(C) to combine them in thefollowing equation.
E(C) =∑((xBoF − xBoF )2 wBoF
(4)

x∈C

+(xColor − xColor )2 wColor ) ,where xBoF , xColor , x and w represent a BoF feature vector, a RGB color histogram vector, a vector of the center ofthe cluster, and the weight which is deﬁned as a reciprocal
number of the dimension of each feature vector.
We evaluate each of the obtained clusters in terms of visual coherence with the following equation. We designed thisequation so that the score of the cluster the member photos ofwhich are similar to each other becomes larger.
VC =

n2C
Wi,j ,
E(C)(5)

where nC represents the number of photos in cluster C, andWi,j is a adjustment weight deﬁned in Eq.(2). When VC ishigh, the corresponding cluster is likely to strongly related tothe event. On the other hand, in case that VC is lower, thecluster is expected to be a noise one which is less related tothe event. In the experiments, we set the threshold of VC as
5 which was decided based on the results in the preliminary
experiments.
In addition, the cluster having the maximum value of VC
is regarded as a representative cluster, and the photo the visualfeature vector of which is the closest to the cluster center isselected as a representative photo for the corresponding event.

5. EXPERIMENTS
In the experiment, we used about three million geo-tweet photos posted from Japan which were collected from the Twitter
stream from February 10th, 2011 to September 30th, 2012.
5.1. Experimental results on keyword selection
As results of event keyword extraction for the given dataset,
we obtained 306 keywords related to natural phenomena suchas “rainbow” and “typhoon” and local events related to “ﬁreworks” and “festival”. Part of the results are shown in Table 1. In the table, “Area”, “Weight” and “Score” represent
the bounds of the grid in terms of latitude and longitude, thevalue of Eq.(2), and the value of Eq.(1), respectively. Since
the area where there are the largest number of unique users
who posted geo-photo tweets was Tokyo, the weight value of
the Tokyo area become 1.0. Because the other areas have lessusers than Tokyo, the adjusting weight value become morethan 1.0.
As results of keyword uniﬁcation and complement, thewords which originally come from the same compound wordsuch as “ﬁreworks festival” are uniﬁed and converted into acompound keyword. Part of the results of uniﬁcation is shownin Table 2, and part of the results of complement is shonw inTable 3.
5.2. Experimental results on photo clustering
Next, we show some example results of event photo clusteringcorresponding to ﬁve keywords, “ﬁreworks”, “tree”, “cherry
blossoms”, “rainbow” and “ﬁreﬂy” (See Table 4 for the detail) in Fig.2, 3, 4 5 and 6. The numbers shown on the right ofeach photo cluster represent cluster scores. The clusters (with
red boxes) having the score which is with more than 5.0 are
regarded as event photo clusters, while the rest clusters (withblue boxes) are regarded as non-event clusters unrelated to thecorresponding event keyword. Within each cluster, photos are
sorted in the ascending order of the distance to the cluster center. From the results, scoring of clusters worked successfully
to place more visual clusters in the higher rank.
In Fig.3 (“tree”), the ﬁrst cluster represents Christmas
trees, while the second cluster represents Tokyo Skytree in
the night. In Fig.5 (“rainbow”), the score of the second cluster was less than 5, since it contains many of the photos including relatively large and various background regions. In
Fig.6 (“ﬁreﬂy”), the ﬁrst cluster represents illumination eventof Tokyo Skytree which was called “Tokyo ﬁreﬂy”
Some detected events are shown on the map with theirrepresentative photos in Fig.7. This map is an interactive system based on Google Maps API, and a user can see any event
photos by clicking markers.
Finally, 258 events were detected in this experiment. All
the 258 detected events can be regarded as being related tosome of various kind of actual “events” including weathercondition, natural events, festivals and sport games. We evaluated if all the presentative photos of the detected events are
relevant to the event or not subjectively by hand, and the precision of the representative photos were 65.5%.
6. CONCLUSIONSIn this paper, we proposed a visual event mining system fromthe Twitter stream using visual information as well as textual
and location information. The system enables us to discoverand understand events visually, which is the novel contribution of this work.
For future work, we plan to propose more sophisticatedvisual event mining methods which integrate visual, textual
and location information more closely and more comprehensively. In the current system, the grid size and the term to
extract events are ﬁxed to one degree and one day, respectively. We will extend the system so that the grid size andthe time unit for detecting events are adjusted automaticallydepending on events.
In addition, real-time visual event detection is also one of
our future works. We also plan to analyze the difference between Tweet photos and Flickr photos in terms of their characteristic.
7. REFERENCES[1] T. Sakaki, M. Okazaki, and Y. Matsuo, “Earthquake
shakes twitter users: real-time event detection by social
sensors,” in Proc. of the International World Wide Web
Conference, 2010, pp. 851–860.
[2] R. Lee and K. Sumiya, “Measuring geographical regularities of crowd behaviors for twitter-based geo-social event
detection,” in Proc. of the 2nd ACM SIGSPATIAL International Workshop on Location Based Social Networks,
2010, pp. 1–10.
[3] K. Yanai, “World seer: a realtime geo-tweet photo mapping system,” in Proc. of ACM International Conference
on Multimedia Retrieval, 2012.
[4] Y. Nakaji and K. Yanai, “Visualization of real world
events with geotagged tweet photos,” in Proc. of IEEEICME Workshop on Social Media Computing (SMC),
2012.

Table 1. Part of the list of the extracted keywords
keywordsnow
earthquakeﬁreworks
festival
Yodo-river
dome
rain
typhoon
Mt.FujiApple
Ginza
SuzukaCircuitAge
eclipsetotal
Christmas
New-Year’s-Eve
sunrise
New-Yesr’s-Day
Meiji
Eho
ski
Valentine
Marathon
Roppongicherry-blossoms
super
moon
ﬁreﬂy
motherecliseannular
Tanabata
Gion-Festival
Tohoku-Denryoku
sky
tree
peace
Makuhari Messe
Awa
OkuribiDaimonjiOkinawawindlakeside
Nihon-dairadate
2011/2/11
2011/3/11
2011/8/6
2011/8/6
2011/8/6
2011/8/10
2011/8/19
2011/9/21
2011/9/24
2011/10/6
2011/10/6
2011/10/9
2011/10/9
2011/10/23
2011/12/10
2011/12/10
2011/12/24
2011/12/31
2012/1/1
2012/1/1
2012/1/1
2012/2/3
2012/2/11
2012/2/14
2012/2/26
2012/3/24
2012/4/28
2012/5/5
2012/5/5
2012/5/6
2012/5/13
2012/5/21
2012/5/21
2012/7/7
2012/7/14
2012/7/14
2012/7/28
2012/7/28
2012/8/6
2012/8/11
2012/8/12
2012/8/16
2012/8/16
2012/9/8
2012/9/15
2012/9/15
2012/9/15

area
34,35,135,136
35,36,139,140
34,35,135,136
34,35,135,136
34,35,135,136
43,44,141,142
35,36,139,140
35,36,139,140
35,36,138,139
35,36,139,140
35,36,139,140
34,35,136,137
34,35,136,137
35,36,135,136
34,35,135,136
34,35,135,136
35,36,136,137
35,36,139,140
35,36,139,140
35,36,139,140
35,36,139,140
35,36,139,140
36,37,138,139
35,36,139,140
35,36,139,140
35,36,139,140
37,38,140,141
35,36,139,140
35,36,139,140
35,36,139,140
35,36,139,140
35,36,139,140
35,36,140,141
34,35,135,136
35,36,135,136
37,38,139,140
35,36,139,140
35,36,139,140
34,35,132,133
35,36,140,141
34,35,134,135
35,36,135,136
35,36,135,136
26,27,127,128
38,39,140,141
38,39,140,141
34,35,138,139

weight
1.96
1
1.96
1.96
1.96
3.96
1
1
3.35
1
1
3.94
3.94
3.46
1.96
1.96
2.9
1
1
1
1
1
3.69
1
1
1
4.18
1
1
1
1
1
3.18
1.96
3.46
4.4
1
1
4.08
3.18
3.91
3.46
3.46
4.17
3.9
3.9
4.15

score
135.5
55
149.2
68.7
72.6
51.5
60
62
67
70
51
78.8
67
55.4
84.4
58.9
55.2
68
84
69
50
63
77.5
58
77
88
121.4
93
96
59
63
314
60.5
56.9
104
79.2
54
53
77.5
168.9
54.8
104
83.2
66.8
58.5
66.3
78.8

Table 2. Results of keyword uniﬁcation.
keywords before uniﬁcation
ﬁreworks, festival
ﬁreworks, festival, Tama-riverﬁreworks, festival, Edo-riverﬁreworks, festival, Lake-Suwa
Apple, Ginza
eclipe, Total
Roppongi, Hills
wind, raincherry blossoms, beautiful
super, moon
blue, moon
Sky, tree
Suzuka, circuit

keyword after uniﬁcation
ﬁreworks
ﬁreworks
ﬁreworks
ﬁreworks
Apple
eclipseRoppongi
windcherry blossoms
super
blue
sky
circuit

Table 3. Results of keyword completion.
keywords before completion
dome
Apple
Meiji
Eho
ski
MarathonMarathon
super
blue
ﬁreﬂy
mothergone
Tohoku-denryoku
skycircuitPeace
MarineNihon-daira

compound keyword after completion
Sappro DomeApple Store
Meiji Shrine
Eho-makiski park
Tokyo Marathon
Kyoto Marathon
super moon
blue moonTokyo ﬁreﬂy
mother’s day
Typhoon gone
Tohoku-denryoku Big One Studium
sky-treeSuzuka Circuit
Peace Memorial Park
QVC Marine Filed
Outsourcing Studium Nihon-dairaFig. 3. “Tree” photo clusters.

Table 4. Summary for the example results.
event keyword
ﬁreworks
treecherry blossoms
rainbow
ﬁreﬂydate
2011/12/23
2011/12/23
2012/04/21
2012/05/04
2012/05/06

grid (lat,lng)
35,36,139,140
35,36,139,140
34,35,135,136
35,36,139,140
35,36,139,140

area
Tokyo
Tokyo
Osaka
Tokyo
Tokyo

Fig. 2. “Fireworks” photo clusters.

# photos

91
91
57
93
93

Fig. 4. “Cherry blossoms” photo clusters.

Fig. 5. “Rainbow” photo clusters.

Fig. 7. Some detected events are shown on the map with theirrepresentative photos.

Fig. 6. “Fireﬂy” photo clusters.