[DEMO PAPER] TWITTER VISUAL EVENT MINING SYSTEM
Takamu Kaneko, Hiroyoshi Harada and Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan{kaneko-t,harada-h,yanai}@mm.inf.uec.ac.jp

ABSTRACTIn this demo, we demonstrate a system to mine events visuallyfrom the Twitter stream by making use of “geo-tweet photos”. Some works on event mining which utilize geotagged
tweets have been proposed so far. However, they used no images but only textual analysis of tweet texts. In this work,
we detect events using visual information as well as textualinformation, which is the first work to mine event photos automatically from a huge number of Twitter photos, as longas we know. In the experiments, we show some examples
of detected events and their photos such as “blooming cherryblossom” and “Tokyo firefly festival”.

We think that Twitter is a promising data source of geotagged of photos, while Flickr has been the most popular
data source of geotagged photos in the research communityof multimedia so far. Since the characteristic of Twitter isquickness and on-the-spot-ness, the photos on Twitter are different from the photos on Flickr. Flickr has many travel photos, while Twitter has many photos related to everyday life
such as food, weather, street scene and some events. Therefore, we think that geo-tweet photos are more useful for event
photo mining than Flickr geo-photos, which are used in theMediaEval Social Event Mining task 1 as data source.
In this demo, we show a system to mine events visuallyfrom the Twitter stream. To do that, we pay attention to thetweets having both geotags and photos. We call such tweets
as “geo-tweet photos”. So far some works on event mining
which utilize geotagged tweets have been proposed. However, they used no images but only textual analysis of tweet
texts. On the other hand, in this work, we detect events using visual information as well as textual information. As longas we know, this is the first system that leverages the use ofimages along with the geotagged tweets, while the previoussystems used Flickr geotagged photos for the same objective,
or mined events with only textual information from Tweets.
2. OVERVIEW

Fig. 1. Some detected events are shown on the Google mapwith their representative photos.

1. INTRODUCTION
Twitter can be regarded as “socially distributed cameras”,
since many of the Twitter’s users send send photos with geotags as well as messages to Twitter with mobile phones or
smart phones from various kinds of places. In fact, we collected about 200 thousand geotagged photos a day on average in December 2012 via Twitter Streaming API. Such geotagged photo tweets are sometimes posted from the places
where some events happen such as festival, sport game andearthquake. Then, we can regard Twitter’s users as distributed
image sensors on events.

To detect events visually from Twitter stream, we monitor
Twitter stream to pick up tweets having both geotags andphotos, and store them into a geo-photo tweet database. We
apply to this database the proposed visual event mining system which consists of event keyword detection, event photoclustering and representative photo selection. The processing
steps of the proposed system are as follows:(1) Detect event keyword candidates which frequently appear in the tweets posted from specific areas in specificdays.
To detect events, we search for bursting keywords by
examining change of the daily frequency of each keywords within each unit area. The area which is a location unit to detect events are defined in the grids by one
degree latitude and one degree longitude.
(2) Unify and complement detected event keywords.
In the previous step, we extract a single noun as an event
keyword. However, since some events are represented
by compound keywords, the same event are sometimes
detected by several keywords independently. In suchcase, we unify them into a compound keyword related
to the same event.
1 http://www.multimediaeval.org/mediaeval2012/sed2012/

(3) Select geo-tweet photos corresponding to the event keywords by image clustering.
As image features, we use bag-of-features (BoF) with
densely-sampled SURF local features and 64-dim RGB
color histograms. For visual clustering of photos, weuse the Ward method which is one of agglomerative hierarchical clustering methods.
(4) Select a representative photo to each event
We evaluate each of the obtained clusters in terms ofvisual coherence based on average similarity among thephotos in each cluster.
In addition, the cluster having the maximum inner similarity is regarded as a representative cluster, and thephoto the visual feature vector of which is the closest to
the cluster center is selected as a representative photofor the corresponding event. Moreover, representative
photos can be selected by VisualRank as well.
(5) Show the detected events with their representative photos on the map as shown in Figure 1.
This work will be presented at ICME Workshop on Social
Multimedia Research (SMMR2013) as a workshop paper [1].
Please refer to it for the detail.

Table 1. Summary for the example results.
event keyword
cherry blossoms
firefly
fireworks
tree
rainbowdate
2012/04/21
2012/05/06
2011/12/23
2011/12/23
2012/05/04

grid (lat,lng)
34,35,135,136
35,36,139,140
35,36,139,140
35,36,139,140
35,36,139,140

area
OsakaTokyoTokyoTokyoTokyo# photos

Fig. 2. “Cherry blossoms” photo clusters. The clusters withred boxes are relevant, while one with a blue box in which theevaluation score is less than 5.0 are irrelevant.

3. EXPERIMENTSIn the experiment, we used about 3,000,000 geo-tweet photos posted from Japan which were collected from the Twitter
stream from February 10th, 2011 to September 30th, 2012.
As results of event keyword extraction for the given
dataset, we obtained 306 keywords related to natural phenomena such as “rainbow” and “typhoon” and local events related
to “fireworks” and “festival”.
We show some example results of event photo clustering corresponding to the two keywords, “cherry blossoms”
and “firefly” (See Table 1 for the detail) in Figure 2 and3. The numbers shown on the right of each photo cluster
represent cluster scores. The clusters (with red boxes) having
the score which is with more than 5.0 are regarded as event
photo clusters, while the rest clusters (with blue boxes) areregarded as non-event clusters unrelated to the corresponding
event keyword. Within each cluster, photos are sorted in theascending order of the distance to the cluster center. From theresults, scoring of clusters worked successfully to place morevisual clusters in the higher rank.
In Figure 2 (“cherry blossom”), the first cluster shows
fully-blooming cherry blossoms, which the second cluster
shows cherry blossom photos mostly taken in night. In Figure 3 (“firefly”), the first cluster represents illumination eventof Tokyo Skytree which was called “Tokyo firefly”, while theother clusters with blue boxes are judged as being irrelevant
to “Tokyo firefly”.
Some detected events are shown on the map with theirrepresentative photos in Figure 1. This map is an interactive
system based on Google Maps API, and a user can see any
event photos by clicking markers.
Finally, 258 events were detected in this experiment, andthe precision of the representative photos were 65.5%. All the258 detected events can be regarded as being related to someof various kind of actual “events”.

Fig. 3. “Firefly” photo clusters.
4. CONCLUSIONSIn this demo paper, we demonstrated a novel visual event mining system from the Twitter stream using visual information
as well as textual and location information. The system enables us to discover and understand events visually, which isthe novel contribution of this work.
For future work, we will extend the system for real-timevisual event detection. We also plan to analyze the differences
between Tweet geo-photos and Flickr geo-photos in terms of
their characteristic.
In the ICME demo, we will show the extended results
which are mined from more than 10,000,000 geo-tweet photos posted from all over the world which were collected fromthe Twitter stream since 2011 till now.
5. REFERENCES[1] T. Kaneko and K. Yanai, “Visual event mining from geo-tweet
photos,” in Proc. of IEEE ICME Workshop on Social Multimedia
Research (SMMR), 2013.

5793
919193