VISUALIZATION OF REAL-WORLD EVENTS WITH GEOTAGGED TWEET PHOTOS
Yusuke Nakaji and Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan{nakaji-y,yanai}@mm.cs.uec.ac.jp
ABSTRACT
Recently, microblogs such as Twitter have become very common, which enable people to post and read short messages
from anywhere. Since microblogs are different from traditional blogs in terms of being instant and on the spot, they
include much more information on various events happened
over the world. In addition, some of the messages posted toTwitter include photos and geotags as well as texts. Fromthem, we can get to know what and where happens intuitively.
Then, we propose a method to select photos related to
the given real-world events from geotagged Twitter messages(tweets) taking advantage of geotags and visual features ofphotos. We implemented a system which can visualize realworld events on the online map.
Index Terms— Twitter, Tweet photo, event visualization,
geotag, mean-shift
1. INTRODUCTION
Recently, smart phones such as iPhone and Android phones
have become very common. At the same time, microblogs
such as Twitter have become to gather many users, since microblogs enable people to post and read short messages fromanywhere using smart phones. Since microblogs are different from traditional blogs in terms of being instant and on thespot, they includes much more information on events happened over the world. In addition, some of the messages
posted to Twitter, which are commonly called as “tweets”,
include photos and geo-tags as well as texts. By using photos
and geotags embedded in tweets, we can get to know what
and where happened over the world intuitively.
So far, there has been many works on Twitter. Some ofthem handle geotags. By analyzing geotagged tweets, we
can detect events with their locations. For example, Sakaki
et al. proposed a method to track typhoon move and monitor
earthquake using geotagged tweets [1]. They regarded Twitter users as “social sensors” which react real-world events including natural phenomenon such as earthquake and typhoonand human events such as festivals and sport events.
On the other hand, Lee et al. proposed a method to detect local events automatically by monitoring the number oftweets, the number of tweet users, and movement of tweet
users within each local area [2]. If a sudden rise in these numbers is detected, some kinds of “events” are regarded as being

taking place. Their methods succeeded in detecting special
local festivals and famous seasonal festivals.
In this way, most of the works on Twitter utilize texts aswell as geotags embedded in tweets. On the other hand, noneof the works on Twitter focus on photos attached to tweets, aslong as we know. Then, in this paper, we propose a method
to detect representative photos related to the given events in
the real world automatically from tweets containing geotags
and photos using visual analysis as well as geotag analysis.
To provide events to the system, we can use time periods andlocations as well as keywords. By comparing representative
tweet photos over the time periods or over some locations, we
can get to know transition of events regarding locations andtime.
Our ﬁnal objective is to implement a system which can let
us know what happens currently and in the past over the world
visually, which includes detection of events and selection ofrepresentative photos corresponding to the detected events.
In this paper, we focus on only selection of representative photos related to events which are assumed to be given
by hand. We propose a method to do that and show somepreliminary results.
2. OVERVIEW
At ﬁrst, we have to decide areas and time periods where andwhen we like to detect events. In addition, we can use optional keywords to limit tweets as well, which helps examine speciﬁc events such as typhoon. Since this paper focuseson only selecting representative photos corresponding to thegiven event, detecting salient events will be one of our future
works.
Firstly, we gather geo-photo-tweets from Twitter and store
them into the in-house database. Secondly, we download photos contained in the gathered tweets. Thirdly, we extract visual features from the downloaded images. Fourthly, we cluster images and geotags embedded in the tweets related to thegiven events, and ﬁnally select representative photos regarding the given locations and time periods by using a method
based on the GeoVisualRank method [3].
To summarize it, the method to detect representative event
photos from geo-photo-tweets is as follows:
1. Gather tweets with geotags and photos using Twitter
Streaming API.

2. Extract visual features from tweet photos using color
histograms and SURF-based bag-of-feature (BoF)
representation.
3. Perform k-means clustering over visual features andmean-shift clustering over geotags.
4. Select representative tweet photos with time-extended
GeoVisualRank regarding each place and each time
period.
2.1. Collecting Geo-tweets with Photos
We use Twitter Streaming API to collect geo-tweets with photos. We have been collecting tweets for about one years so far.
Since February 2011 for one year, we collected about 18 million geo-photo-tweets and stored all of them into our in-house
geo-photo-tweet database.
Since each tweet itself includes not photos, but URLs ofphotos, we extract URLs and download the corresponding
photos from photo storage sites for twitter such as Twitpic,
yfrog and instagr.am.
In fact, we limit geo-photo-tweets to download corresponding photos with only the tweets posted within Japan
currently, since we can catch a geo-photo-tweet every two
seconds and the total amount is too huge to download all thephotos.
2.2. Extraction of Visual Features
As visual features to represent tweet photos, we use color
histogram and Bag-of-Features representation of SURF features [4] with the codebook the size of which is 2000.
2.3. Clustering Photos and Geotags
Before performing clustering, we search the geo-photo-tweet
database for the geo-photo-tweets which meet the given conditions on locations, time periods and optional keywords.
And then, we perform k-means clustering for the selected
tweet-photos on visual features with the number of clusters
set as n/10 where n is the number of the selected photos. Thisvisual clustering result is used for showing similar photos torepresentative photos on the system.
At the same time, we carry out mean-shift clustering [5]on geotags of the selected geo-photo-tweets in order to selectrepresentative areas. Mean-shift clustering is based on kernel density estimation, which consists of iteration of setting
circular regions Th (x) with radius h where its center is a data
point x and updating the value of the point x to the mean value
weighted by Gaussian kernel function K(xi ; x, h) shown inEquation (1). This iteration is repeated until being convergent.
m(x) =K(xi ; x, h) =∑xi ∈Th (x)K(xi ; x, h)w(xi )xi∑ni=1 K(xi ; x, h)w(xi )( ||x − x ||2 )
i
exp −
h2

(1)

In the experiment, we set w(xi ) as 1 and h so that the radiumof each cluster is 50km.
2.4. Select Representative Photos
We use a method based on the GeoVisualRank method [3] toselect representative photos regarding the given locations andtime periods.
GeoVisualRank [3] is the extension of VisualRank [6]which is a method to select representative images from a setof images. VisualRank is based on the widely-known Web
page ranking algorithm, PageRank [7], which is based on
Markov chain. In PageRank, the transition matrix is computed based on links between Web pages, while the transition
matrix of the Markov chain in VisualRank and GeoVisualRank is computed based on visual similarity between images.
The rank of Web pages or images are estimated according to
the probability of the steady state distribution of the Markov
chain.
VisuakRank is computed with the following equation:r = αSr + (1 − α)p,(3)where S is the column-normalized similarity matrix of images, p is a damping vector, and r is the ranking vector eachelement of which represents a ranking value of each image.
α plays a role to control the extent of effect of p. In the experiments, α is set as more than 0.85 in the same as [6]. The
ﬁnal value of r is estimated by updating r iteratively with
Equation (3). Because S is column-normalized and the sumof elements of p is 1, the norm of ranking vector r does notchange. Although p is set as a uniform vector in VisualRankas well as normal PageRank, it is known that p can plays abias vector which affects the ﬁnal value of r [8].
In GeoVisualRank [3], a geo-location-based bias vector isused in calculation of VisualRank instead of a uniform damping vector to take account of geo-location proximity. GeoVisualRank computes ranking of geotagging images considering both visual similarity and geo-based bias. In this paper,
we extend it by making it take account of time proximity. We
compute a bias vector based on time differences between thetime when the corresponding tweet is posted and a given time
as well as distances between geotagged location of a tweet
photo and a given reference location.
In the extended GeoVisualRank we proposed here, reference locations and reference times need to be given to calculate a bias vector. It gives higher PageRank values to geotagged images the geotag location of which are closer to thegiven reference locations and the taken time of which arecloser to the given reference time. To do that, we set eachelement of the bias vector as follows:{
1/ngeo (gi ∈ C geo )pgeo (i) =(4)0(gi 6∈ C geo ){(2)(0 ≤ α ≤ 1)

p

time

(i) =

1/ntime0(ti ∈ C time )(ti 6∈ C time )(5)where C geo represents a location cluster containing the givenreference location, C time represents time cluster containing
the given time, and ngeo and ntime represents the number ofimages belonging to C geo and C time , respectively. Note thattime clusters are constructed by dividing times every six hours
as shown in Table 1.

No.
0
1
2
3

Table 1. Time clusters
time period
Midnight∼6 A.M. (after midnight)
6 A.M. ∼Noon (morning)Noon∼6 P.M. (afternoon)
6 P.M. ∼Midnight (night)

Finally the bias vector p is obtained by linear combinationof pgeo and ptime as shown in Equation (6).
p = βpgeo + (1 − β)ptime(0 ≤ β ≤ 1)

(6)

In the experiments, we set β as 0.5.
Since we use color histogram and Bag-of-Features representation of SURF features as visual features, we calculate
histogram intersections as visual similarity between image
features. As shown in Equation (7), we generate a similarity matrix S by combining a color histogram similarity matrix Scolor with Bag-of-Features similarity matrix SBoF witha weighting constant β. In the experiment, we set γ as 0.5.
Scombine = γ Scolor + (1 − γ) SBoF

(0 ≤ γ ≤ 1)

(7)

3. EXPERIMENTS
As the experiments, we tried to select event photos related totyphoon on September 2011 (Event no.1), new year habits inJapan (Event no.2) and the huge earthquake hit on NorthernEast Japan in March 11th 2011 (Event no.3) as shown in Table2.
3.1. Selected Photos for Each Event
For Event No.1, before the processing, we selected 616 geophoto-tweets with keyword search for tweet messages andtime periods limitation on September 2011. And then, we applied clustering on geotags and visual features, and computethe extended GeoVisualRank. Finally we obtained representative photos as shown in the upper left of Figure 1. Thisﬁgure shows the representation photos on “typhoon” selectedin the largest time cluster in the largest geotag cluster whichcorresponds to the area around Tokyo. Most of the selected
photos shows the black sky which is typical way of the sky
when typhoon is approaching.
For Event No.2, we selected photo-tweets in the same
way as Event No.1 at ﬁrst. The upper left ﬁgure of Figure
1 shows representative photos related to “New Year’s life inJapan” (Event No.2), which represents photos taken at shrines

or temples. We have a traditional custom to go shrines or temples to pray safe life of the new year in Japan. The selected
photos reﬂected it.
For Event No.3, we did not use keyword search to pickup geo-photo-tweets from the database. We used only thecondition on time and area, since we liked to examine thedifference of the kinds of representative photos depending on
locations in the day when the biggest event for these ten years
happened in Japan. The time period was set from the day,March 11th, for two days.
The bottom ﬁgure of Figure 1 shows the representative
photos taken in Tokyo, which expresses the scene that thetrains stopped and people walks to home instead of getting
trains.
3.2. Comparison of the Representative Photos Depending
on Locations
In this subsection, we examine the differences of the representative photos on the same event depending on locations. Figure 2 shows results of the four location, Fukushima,
Tokyo, Okayama and Fukuoka, on Event No.3 (earthquake)in the afternoon of March 11th 2011. Some of the photos in
Fukushima and Tokyo which the earthquake affected greatly
show the rooms where many books are scattered, brokenbuilding, and many people walking to home due to train
stop. On the other hand, most of the photo in Okayama andFukuoka where they are enough far from seismic center ofthe earthquake consists of everyday life photos such as foods
and normal street scene.
From these results, we can get to know areas where theearthquake affected intuitively.
3.3. Comparison of the Representative Photos Dependingon Times
In this subsection, we examine the differences of the representative photos on the same event depending on times. Figure
3 shows the representative photos on the four time periods in
the New Year’s day (Event No.2) in Tokyo. From midnight to
6AM (left top), the photo of the ﬁrst sunrise of the year wereselected, from 6AM to 12PM (right top) the photo of the temples and shrines were selected, and from 12PM to 6PM (leftbottom) and from 6PM to 12AM (right bottom) many photos
related to new year’s day foods were selected.
From these results, we can get to know transition of wayof the life depending on times on the New Year’s day in Tokyovisually.
4. CONCLUSIONSIn this paper, we proposed a method to select photos related to
the given real-world events from geotagged Twitter messages(tweets) taking advantage of geotags and visual features ofphotos. We implemented a system which can visualize realworld events on the online map. The experiments showedthat the method was helpful to visualize the differences of theFig. 1. Representative tweet photos in the largest time clusters in Tokyo (upper left: typhoon (Event No.1), upper right: New
Year (Event No.2), bottom: earthquake (Event No.3)

representative photos related to the given event depending on
locations and times.
In this paper, we focused on only selection of representative photos related to events which are assumed to be given
by hand . As future work, we will work on a method to detect
event automatically and in the real-time way, and combine itwith the proposed method in this paper to implement a system
which can let us know what happens currently and in the pastover the world visually.
5. REFERENCES[1] T. Sakaki, M. Okazaki, and Y. Matsuo, “Earthquake shakes twitter users:
real-time event detection by social sensors,” in Proc. of the International
World Wide Web Conference, pp. 851–860, 2010.
[2] R. Lee and K. Sumiya, “Measuring geographical regularities of crowd
behaviors for twitter-based geo-social event detection,” in Proc. of the

2nd ACM SIGSPATIAL International Workshop on Location Based Social Networks, pp. 1–10, 2010.
[3] H. Kawakubo and K. Yanai, “Geovisualrank: A ranking method of geotagged images considering visual similarity and geo-location proximity,”in Proc. of the ACM International World Wide Web Conference, 2011.
[4] H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded up robustfeatures,” in Proc. of European Conference on Computer Vision, pp.
404–415, 2006.
[5] D. Comaniciu and P. Meer, “Mean shift analysis and applications,” inProc. of International Conference on Computer Vision. Ieee, vol. 2, pp.
1197–1203, 2002.
[6] Y. Jing and S. Baluja, “Visualrank: Applying pagerank to large-scale
image search,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 30, no. 11, pp. 1877–1890, 2008.
[7] S. Brin and L. Page, “The anatomy of a large-scale hypertextual web
search engine,” in Proc. of the Seventh International World Wide Web
Conference, 1998.
[8] T.H. Haveliwala, “Topic-sensitive PageRank: A context-sensitive ranking algorithm for web search,” IEEE trans. on Knowledge and Date
Engneering, vol. 15, no. 4, pp. 784–796, 2003.

Fig. 2. Representative tweet photos related to “earthquake” (left top: Fukushima, left bottom: Tokyo, right top: Okayama, rightbottom: Fukuoka)

No.
1
2

3

Search keyword (OR search)
typhoonNew year’s day
ﬁrst-shrine-visit
ﬁrst-sunrise(none)

Table 2. The conditions of three kinds of experiments
time periods
area# tweet photos
Sep. 1 2011 to Sep. 30 2011
posted from Japan 616
Jan. 1 2012 to Jan. 20 2011
posted from Japan 1400

Mar. 11 2011 to Mar. 12 2011

posted from Japan1080

Fig. 3. Representative tweet photos related to “New Year” on the New Year’s day (Jan. 1st) (left top: 0AM∼6AM, right top:
6AM∼12PM, left bottom:12PM∼6PM, right bottom:6PM∼0AM)