On-site likelihood identification of tweets for tourism information analysis
Kazutaka Shimada, Shunsuke Inoue and Tsutomu Endo
Department of Artificial Intelligence, Kyushu Institute of Technology
680-4 Iizuka Fukuoka 820-8502 Japan{shimada. endo}@pluto.ai.kyutech.ac.jp

Abstract—Tourism is one of the most important key industries. The Web contains much information for the tourism,
such as impressions and sentiments about sightseeing areas.
Analyzing the information is a significant task for tourism
informatics. One approach to extract tourism information is
to extract sentences with keywords related to target facilities
and events. However, all sentences with keywords might be nottourism information. In this paper, we propose a method formeasuring tourism information likelihood. The target resource
for the analysis is information on Twitter. The task is to identify
whether each tweet has high on-site likelihood. We introduce
a filtering process and a machine learning technique for thetask. Our method obtained 80.5% on the precision rate.
Keywords-Tourism information on the Web, Twitter, On-site
likelihood

I. I NTRODUCTION
Tourism for many local cities is one of the most important
key industries. The activation of tourism leads to the activation of the local industries and communities. In this situation,
the World Wide Web plays a large role [6]. Although ahuge number of online documents are easily accessible on
the Web, the quality of the information is a mixture of
the good and bad. Finding important information relevantto the target needs has become increasingly significant. We
develop a tourism information analysis system which extracts information about tourism from the Web, analyzes theextracted information in various perspectives, and visualizes
the output of the analysis [8]. Figure 1 shows the outlineof the system. By using this system, people involved in
the tourism can easily understand and organize significant
information of the target city. The target resource for thesystem is information on Twitter1 . It is one of the most
famous microblogging services and text-based posts of up
to 140 characters. The posted sentences are described as“tweets”. In microblogging services such as Twitter, users
tend to post tweets in real time. It denotes that tweets
often contain significant information of events for tourism
as lifelog data.
One approach to extract tourism information is to extract
tweets with keywords related to target facilities and events.
However, all tweets with keywords might be not tourism
information. For example, the tweet “I’m on the way towork now. near LOCATION” is not suitable as an input
1 http://twitter.com

for an opinion analysis system because it does not includesentiments about sightseeing as experience. Therefore, weneed to judge the adequateness of each tweet.
In this paper, we propose a method for evaluating the adequateness for the tourism information analysis system. We
focus on on-site likelihood of tweets. The on-site likelihood
estimation is to identify whether a tweet is posted at the target facility. High on-site likelihood denotes the adequatenessas the input for the analysis system. In addition, tweets withhigh on-site likelihood are useful to analyze the behaviorof tourists because their location is identified. The on-site
likelihood identification contains two processes; a filtering
process and a machine learning process.
In this paper, we explain the outline of the basic information extraction process in our system in Section II. Then,
we describe the on-site likelihood identification method in
Section III. The method consists of a filtering process anda classification task with a machine learning technique. In
Section IV, we evaluate the performance of our method, andconclude the paper in Section V.
II. T OURISM INFORMATION EXTRACTION
In this section, we describe a basic method to extract
tourism information.
The extraction process is basically as follows:
Step1: Acquisition of basic queries,
Step2: Selection of related words,
Step3: Query generation and retrieval.
The basic information for this process is extracted fromportal sites for tourism which the city and tourist association
established. Here “basic query” denotes tourist facilities,
restaurants and events, such as festival, which are written in
the portal sites. Figure 2 shows an example of the tourism
portal site about Iizuka city2 . It consists of (1) facility or
event names, (2) a link to detailed information of each entry
and (3) basic information of each entry. We define the facility
or event names as the basic queries. For example, “Kahoperforming theater” and “Ito Den-emon residence” are basic
queries. The number of basic queries is approximately 200
words.
We need to consider a problem of basic queries. Sightseers
do not always mention the basic queries, i.e., facility or event
2 http://portal.kankou-iizuka.jp/ It is one of our target cities.

Sentiment or Opinions from Users

Automatic / Semi-automatic Information Analysis

Twitter

Blog

Portal sites of the target location

Information Extraction

Sentiment Analysis, Demand Analysis,
Behavioral Analysis

Output of Analysis

Visualization / Summarization

Detailed check by human and Improvement* Sentiment analysis* Analysis about Location* Distribution of Positive,
Location A
Neutral and Negative
Location B
Neg
Neu

Pos

Location C
Location D[Beautiful] Japanese garden!! Positive opinion-> It is a strong point
Rest rooms are [dirty] Negative opinion-> Need to improvement
XXX is [delicious] Positive Opinion-> It may becomes a new tourism element?

* Important topics
keyword 1
Keyword 2
Keyword 3
Keyword 4* Demand Analysis[Where] is souvenir shops ... :-(-> need to put out the shop sign
I [faced difficulty] finding a restaurant ... :-O-> need to prepare a restaurant guide

Visualization by charts and graphs* Behavioral Analysis
In the morning, we went to see the city sights.
We went to the ZZZ park after lunch.

Display and highlight important information
generated from analyzers

Figure 1.

The outline of our prototype system.

names, in tweets. Moreover, they might mention information
which is related to the location or event names and does
not appear in the portal site. Therefore, we need to acquire
related words of the basic query, i.e., query expansion. First,
we need to divide each sentence into words. For the process
we use MeCab3 , which is one of the most famous Japanese
language morphological analyzers. For the selection of related words, we introduce a weighting approach, which iswell-known as Okapi-BM25 [5]. The importance of a word

3 http://mecab.sourceforge.net/

Behavioral pattern: [City]→[Lunch]→[ZZZ park]-> Restaurants between the city area and ZZZ park area very important issue!

is computed byscore(D, Q) =n
X
i=1

IDF(qi ) ·f(qi , D) · (k1 + 1)
|D|f(qi , D) + ki · (1 − b + b · avgdl
)

(1)

where
IDF(qi ) = log

N − n(qi ) + 0.5
n(qi ) + 0.5(2)f(qi , D) is the frequency of a word qi in a document D.
n(qi ) and |D| are the number of documents containing qi andExplanation

Facility or event names[1] Kaho performing theater[2] Old Ito Den-emon's residence

Link to detailed information

Basic information about the entry
This was built in 1931.
Two-story wooden house adopting particulararchitectural idea.
......

Figure 2.

The portal site on the Web.

the length of D, respectively. avgdl and n are the average
length of documents and the number of documents. b and k
are constant factors for weighting.
Figure 3 shows an example of explanation about “Old
Ito Den-emon’s residence (the 2nd facility in Fig 2)”. We
obtain a weighted word list from this explanation. In thisexample, appropriate words, such as “Byakuren4 ” and ”Coalmining5 ”, are located in top position in the ranking. Finally
we select suitable words from the weighted word list, as
related words by hand work. This selection process depends
on subjective heuristics. In this process, we tend to selectproper names (e.g., Byakuren) , the head of a noun (e.g.,emon) and attributes of the target (e.g., Coal mining).
We retrieve tweets with Twitter API by using themanually-produced query list, namely the union of basic
queries and related words. Final queries for the retrieval
are combination of words in the list. However, there is aproblem of the final queries. Tweets do not always containthe official name of facilities or events. For example, “Old
Ito Den-emon residence”, which is one of the most famous
facilities in Iizuka, is exceedingly-long words. Therefore, it’sunlikely that users input the official name itself. To solve thisproblem, we manually generate abbreviations of queries. For“Old Ito Den-emon residence”, we add some abbreviations
such as “Ito residence” and “emon residence” to the querylist.
4 She is a wife of Mr. Ito and famous poet.
5 Mr. Ito was a rich coal mine owner and is called Coal Mine King

Ito Den-emon (1860-1947)'s old residence wasbuilt in Meiji Period, and then it extended thebuilding in Taisho and Showa Periods. .... It's
modern Japanese architecture. .... The elaborate
luxurious house has a large garden. .... Den-emon
was the president of a coal mining company inthis area (Iizuka city). Byakuren Yanagihara,
his wife and famous poet, lived in the luxurious
house. ...
Calculation of importance of each word

Word
Okapi-BM25
emon
16.92
Iizuka15.06
Byakuren
14.97
Coal mining
13.33
large garden
13.29
Poet
13.03
Yanagihara
12.80
Lived in
12.37
extended
12.21
Den
11.96

Figure 3.

Selected byhand workemon
Byakuren
Coal mining
Poet
Yanagihara

An example of explanation and the related words.

III. O N - SITE LIKELIHOOD IDENTIFICATION
Not all tweets extracted in the previous section might betourism information. For example, the tweet “I’m going tovisit LOCATION” is not always suitable as an input for an
opinion analysis system because it does not include sentiments about sightseeing as experience. Therefore we need to
identify the on-site likelihood of each tweet. We introduce
a filtering process and a machine learning technique for thetask. Figure 4 shows the outline of the proposed method.
First, we delete obvious noise tweets by using the rule-based
filtering approach. Then, we classify tweets into tweets withon-site likelihood and without on-site likelihood.
A. Filtering
Many tweets on Twitter do not contain information withthe on-site likelihood. Although one solution to identify theon-site likelihood of each tweet is to classify them by using
a machine learning based classifier, biased data usually leads
to the development of a unsuitable classifier. Therefore, weneed to delete the tweets not containing the on-site likelihoodin advance. We apply a rule-based filtering approach to theprocess.
The filtering process contains two types of rules; deletion
rules and non-deletion rules. First, we detect tweets that arematched with the deletion rules. Then we delete the tweets
if they do not contain the non-deletion rules. In other words,

Input: tweets

Positive
Negative

Rule-based filtering
Deletion rules andnon-deletion rules

Hyperplane
MarginSupport Vector (+)

ML-based classification
Support Vector Machines

wx+b=1wx+b=0

Output: tweets withhigh on-site likelihood

Support Vector (-)

wx+b=-1

Figure 5.
Figure 4.

MarginSupport vector machines.

The outline of on-site likelihood identification.

we retain the tweet matched with the non-deletion rules even
if they correspond to the deletion rules.
The deletion rules are as follows:• Linguistic rules: we delete tweets including the following words.
– future: “tomorrow”, “next week”, “someday”. etc.
– tentative: “may”, “guess”, etc.
– indirect: “hear”, etc.
– interaction marks: @ and RT• Time rule: most tweets on late night do not probably
relate to sightseeing information. Therefore we delete
tweets that are posted from 11 p.m. to 3 a.m.
• Length rule: In a preliminary experiment, long tweets
did not tend to include sightseeing information. Therefore, we delete tweets consisting of more than 100
letters.
• # of nouns: In the preliminary experiment, tweets containing many nouns were often advertising information.
We delete tweets containing nouns whose the number
is more than a threshold. The threshold in this paper is
36 nouns. This value was determined experimentally.
On the other hand, there are characteristics expressing onsite information with high probability. We introduce nondeletion rules. We do not delete tweets which contain the
non-deletion rules. They are as follows:• Presence of activity: if tweets contain words related
to the activity of users, they have a potential value
as on-site information. The words are “arrive”, “Herewe are” and so on. Words related to present andprogressive tenses also contain a potential value as onsite information. The words are “looking”, “strolling”
and so on. Tweets with these words indicate user’s
action. Therefore we retain tweets with these words.
• Presence of “NAU (now)”: NAU is a characteristic keyword on Twitter. Although the presence of NAU does
not always indicate on-site information, it seems to sug-gest high potential for on-site information. For example,“LOCATION, NAU” denotes “I’m at LOCATION”.
This tweet includes on-site information. However, “LOCATION on TV, NAU” denotes “LOCATION is
broadcast via television”. It is not on-site information.
In other words, “NAU” is ambiguous and considerable.
Therefore we retain tweets with the word.
By using these rules, we can delete obvious noise tweets
from the tweets extracted by Section II and obtain input
candidates with high accuracy for the classification process.
B. Classification
The purpose of the filtering process is to delete noise
tweets. The filtered tweets consist of tweets with high on-site
likelihood and low on-site likelihood. Therefore, we needto classify them into on-site information and non on-site
information. We apply Support Vector Machines (SVMs)
to the classification task. SVMs are a machine learningalgorithm that was introduced by [9]. An SVM is a binaryclassifier that finds a maximal margin separating hyperplane
between two classes. The hyperplane can be written as:yi = w~ · ~x + b
where ~x is an arbitrary data point, i.e., feature vectors, w~
and b are decided by optimization, and yi ∈ {+1, −1}. The
instances that lie closest to the hyperplane are called support
vectors. Figure 5 shows an example of the hyperplane. In
the figure, the solid line shows hyperplane w~ · ~x + b = 0.
For SVMs, we use 10 features. The features are as
follows:
1) BOW: The bag-of-words is a simple and famousfeature in natural language processing. In our method,
it is a baseline feature set. We use term frequency for
the BOW model.
2) SpecificWords: There are some specific words linked
to on-site information, such as “arrive”, “I’m at” and“NAU (now)”. These words are important features. We
use the presence of them as the features.

3) PostTime: There are appropriate visiting time patterns
for each tourist facility and event. For example, amusement parks are full of customers in the daytime. On theother hand, fireworks shows are events at night. The
appropriate posting time of each tweet depends on thetarget facilities and events. We quantize the postingtime by the hour. For example, the feature value is 12
if the posting time is 12:30. This feature is effective by
combining the presence of tourist facility and events,
namely the BOW feature.
4) PostRange: This is an abstract feature of the PostTime
feature. We classify 24 hours into 3 ranges; (1) 4 a.m.
- 9 a.m., (2) 10 a.m. - 9 p.m., and (3) 10 p.m. - 3 a.m.
5) Length: Users probably post tweets with mobile devices from facilities and event locations. In general,
entering characters by mobile devices is a cumbersome
process. In this situation, the length of posted tweets
tends to be short. Therefore, we utilize the tweet length
as the feature.
6) Tense: The tense is one of the important points forthe on-site likelihood. The past and future tenses are
not usually related to the on-site information. We
focus on the tense linked to adjectives. For example,
we distinguish between “tanoshii (enjoyable)” and“tanoshikatta (enjoyed)”.
7) NumVerb: tweets with high on-site likelihood tend touse relatively little verb in the preliminary experiment.
Therefore, we apply the number of verbs in each tweet
to the feature for SVMs.
8) NumNoun: As we mentioned in the filtering process,the number of nouns in each tweet relates to the onsite likelihood. We also use the number of nouns as
the feature.
9) RT: RT (ReTweet) is a kind of reference structure
on Twitter. In our preliminary experiment, tweets
with RT tended to not contain sentiment or opinion
information of tourism. It leads to the decrease of onsite likelihood. Our method handles the presence of
RT as the feature.
10) LocName: Tweets beginning with a location nameoften contain high on-site likelihood. We use thepresence of a location name at the beginning of a tweet
as the feature.
IV. E XPERIMENT
We evaluated our method with a test data set. The data set
consisted of 2886 tweets. They contained 509 tweets as onsite tweets and 2377 tweets as off-site tweets. There are twopoints in this experiment; the filtering and the classification.
First, we evaluated the filtering process. The criterionof the filtering was a simple accuracy rate. The method
described in Section III-A deleted 1738 tweets from 2886
tweets as noise tweets. In other words, we obtained 1148
tweets as inputs for the next process, namely the classifi-

Table IT HE EXPERIMENTAL RESULT.
FeatureBOW
BOW+SpecificWord
BOW+PostTime
BOW+PostRange
BOW+Length
BOW+Tense
BOW+NumVerb
BOW+NumNoun
BOW+RT
BOW+LocName
ALL

Precision
80.2
79.4
80.4
79.9
80.5
80.3
80.3
80.2
80.3
79.2
80.5

Recall
62.3
64.6
63.1
62.7
62.5
62.3
62.3
62.1
62.3
62.9
65.0

cation. 486 tweets of 1148 were on-site tweets (662 tweets
were off-site tweets). As a result, our method deleted 23 onsite tweets by mistake. The accuracy of the filtering process
was 95.5% (486/509). The mistake of the filtering processleads to the decrease of the accuracy of the classification
process because the output of the filtering become the inputof the classification directly. Therefore, the mistake is a
fatal error of our method even if the error rate is 5%. The
improvement of the filtering is the most important future
work. On the other hand, our method reduced off-site tweets
by approximately 30% (662/2377). This result shows theeffectiveness of our filtering method.
Next, we evaluated the classification task using SVMs.
We used 1148 tweets, which were extracted by the filtering
process, as the input of the classification. We evaluated thetweets with 10-fold cross-validation. Evaluation criteria were
the precision and recall rates. These criteria are computedby:# of correct outputs# of tweets which the method judged as on-site(3)# of correct outputs
Recall =(4)# of on-site tweets in the input data

Precision =The experimental result is shown in Table I. The
mark “+” denotes the combination of each feature; e.g,
“BOW+SpecificWord” denotes SVMs with the bag-of-words
and specific word features. “ALL” denotes the method withall features mentioned in Section III-B. The method withall features produced the best precision and recall rates.
As a feature for the combination with BOW (baseline), theSpecificWord feature was the most suitable in terms of therecall rate. The feature consists of the related word to onsite information, such as “arrive”. It was an intuitive result.
Although the precision rate was relatively favorable (80.5%),
the recall rate was insufficient (65.0%). To improve the recall
rate, we need to add suitable keywords to the SpecificWord
feature.
Finally, we verified the effectiveness of the filtering process. Table II shows the result of a comparison of the method

Table II
T HE EFFECTIVENESS OF THE FILTERING PROCESS .
FeatureWith Filter
Without Filter

Precision
80.5
75.0

Recall65.0
58.2

with filtering process and that without the filtering process.
The method without the filtering denotes the result for 2886
tweets with 10-fold cross-validation. We used all features forthese methods. Applying the filtering led to the improvementof both the precision rate (5.5 points) and the recall rate (6.8
points). This result shows the effectiveness of our methodwith the filtering.
V. D ISCUSSION AND C ONCLUSIONSIn this paper, we focused on on-site likelihood of tweets
for evaluating the adequateness for the tourism information
analysis system. The on-site likelihood identification was
to identify whether a tweet is posted at the target facility. We proposed two processes for the on-site likelihood
identification task. The filtering process was based on two
types of rules; deletion rules and non-deletion rules. The
accuracy was 95.5%. Although the accuracy was relatively
favorable,the filtering errors became a critical problem in
our method because the output was the input of the next
process, namely classification process. The improvement of
the filtering method is important future work. We applied amachine learning technique, namely SVMs with 10 features,to the classification process. For the task, we obtained 80.5%
as the precision rate. We also verified the effectiveness of thefiltering process for the classification task. However, the data
set in the experiment was the closed data, namely known
data for the method, for the filtering and the classification
tasks. Therefore, we need to evaluate our method with theopen data set, namely unknown data for the method.
Several researchers have studied similar tasks. Inui etal. [4] have proposed a task which were called experiencemining. The task included factuality analysis, which was tojudge whether the event indeed took place or not. Aramaki
et al. [1] have proposed a machine learning based method
to extract influenza tweets from Twitter. This task was alsoa kind of fact detection. By using this technique, theydetected the influenza epidemics. Sakaki et al. [7] have alsostudied a event detection task from Twitter. Cheng et al.
[2] have proposed a method for predicting a user’s location
based purely on the content of the user’s tweets. They
identified local words in tweets. Eisenstein et al. [3] havealso discussed lexical variations across geographic areasfrom tweets. Incorporating knowledge of these studies toour method is considerable future work.
We identified the on-site likelihood from a tweet. However, handling only one tweet is not appropriate for theidentification. Tweets that are posted within a time sequenceoften contain the relation between them. For example, thetweet “I enjoyed the place” contains uncertainty as to theon-site likelihood because it might be a tweet about ayesterday’s event. However, if the tweet are posted after thetweet “ LOCATION now!” at short intervals, the tweet has
high on-site likelihood even if it includes the past tense.
The time relation and the content of the previous tweet areimportant to improve the identification accuracy, especiallythe recall rate. We need to handle co-occurrences of tweets
for the on-site likelihood identification task.
R EFERENCES[1] Eiji Aramaki, Sachiko Maskawa, and Mizuki Morita. Twittercatches the flu: Detecting influenza epidemics using twitter. In
Proceedings of Conference on Empirical Methods in Natural
Language Processing (EMNLP), 2011.
[2] Zhiyuan Cheng, James Caverlee, and Kyumin Lee. You are
where you tweet: A content-based approach to geo locating
twitter users. In Proceedings of the 19th ACM international
conference on Information and knowledge management, pages
759–769, 2010.
[3] Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, andEric P. Xing. A latent variable model for geographic lexical
variation. In Proceedings of the 2010 Conference on Empirical
Methods in Natural Language Processing, pages 1277–1287,
2010.
[4] Kentaro Inui, Shuya Abe, Hiraku Morita, Megumi Eguchi,
Asuka Sumida, Chitose Sao, Kazuo Hara, Koji Murakami, andSuguru Matsuyoshi. Experience mining: Building a largescale database of personal experiences and opinions fromweb documents. In Proceedings of the 2008 IEEE/WIC/ACM
International Conference on Web Intelligence, pages 314–321,
2008.
[5] Stephen E. Robertson, Steve Walker, Susan Jones, Micheline
Hancock-Beaulieu, and Mike Gatford. Okapi at trec-3. In
In Proceedings of the Third Text REtrieval Conference (TREC
1994), 1994.
[6] Hajime Saito. Analysis of tourism informatics on web. Journalof the Japanese Society for Artificial Intelligence, 26(3):234–
240, 2011.
[7] Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. Earthquakeshakes twitter users: real-time event detection by social
sensors. In Proceedings of the 19th international conferenceon World Wide Web (WWW2010), 2010.
[8] Kazutaka Shimada, Shunsuke Inoue, Hiroshi Maeda, and Tsutomu Endo. Analyzing tourism information on twitter for a
local city. In Proceedings of SSNE2011, pages 61–66, 2011.
[9] V. N. Vapnik. Statistical Learning Theory. Wiley, 1999.