東海大学紀要情報通信学部
vol.6,No2,2013,pp.22-30
Proc.
Schl. ITE Tokai Univ.

Paper

Paper

Vol. xx,No.xx ,20xx,pp.xxx -xxx

Finding Similar Tweets and Similar Users
by Applying Document Similarity to Twitter Streaming Data byIwao FUJINO *1 and Yuko HOSHINO *2( received on September 28, 2013&

accepted on January 27, 2014 )Abstract
Twitter has grown so rapidly that its users are suffering from an information ove rload. In order to help people to findinteresting tweets and users from the enormous Twitter space, by applying the document similarity of Gerard Salton andChristopher Buckley to Twitter streaming data , we attempt to devise a content-based scheme that compares similarities between
Twitter users by matching their tweets against each other, especially in terms of the new key-value type database environment.
Considering that each tweet is a very short message, we performed data processing of individual twe et level and accumulated
tweet level according to author name in our study. At the individual tweet level, finding similar tweets is functional for finding
retweets of tweeted messages, which is helpful for estimating the information propagation in Twitter. Also in the meaning of
security, it is also effective for finding spam tweets and dishonest copies of tweeted messages.

At the accumulated tweet level,

finding users who use similar words or expressions is functional for finding users who post similar c ontents or topics, which is
helpful for finding friends who have similar preferences and interests. As for the concrete procedures of both levels, we firstuse Japanese morphological analysis to pick up terms from Twitter data. Then we calculate tfidf to provide a weight parameterfor each term. Finally we calculate the document similarity from weight parameter vector between any two documents, whichshows how much the tweets or the authors are similar to each other. As a confirmation work, we build a computer system tosearch tweets by keywords query and to show user similarity between any two users. Distribution graphs of similarity in both
tweet level and author level are also achieved.
Keywords: Twitter, Morphological Analysis, Key-value Database, Tfidf, Document Similarity

users. However nowadays Twitter users are suffering froman information overload as it has grown so rapidly.

1. Introduction

In

order to overcome this problem, a computer system is

Twitter is an online social networking service andintroduced to help Twitter users to find what they want. By

microblogging service started from July 2006. Unlike manymeans of this computer system, Twitter users can searchother social network services, Twitter not only maintains

the information relevant to given keywords or check if anysocial links among users but also broadcasts information to

tweets similar to the tweets of themselves or find the users

crowds like the mass media [1]. Furthermore because these

who

two roles work interactively, users may have massive

expressions similar to themselves. When dealing with anchances to access information which they have interest in

individual tweet, finding similar tweets is functional for

as well as interact with the users who posted thefinding retweets of tweeted messages, which is helpful for

information on Twitter. According to Wikipedia [2], Twitter

estimating the information propagation in Twitter. Also in

has over 500 million registered users as of 2012 andthe meaning of security, it is also effective for finding spam

generating over 340 million tweets every day. Also

tweets and dishonest copies of tweeted message s.

according to the report of Semiocast [3], Japan remains thedealing with tweets accumulated according to author,

second most active country in terms of posted tweets: infinding users who use similar words or expressions is

June 2012, 10.6% of all public tweets were posted fromfunctional for finding users who post similar contents or

Japan, while Japanese users represent 6.7 % of all Twitter

topics, which is helpful for finding friends who have

Engineering,

SchoolofInformationand

Telecommunication Engineering, Tokai University*2 Lecturer, Department ofTechnology,

SchoolInformationoftweetswithsimilar

words

or

When

the document similarity of tweets and Twitter users shouldbe placed as a fundamental theoretical problem, andimplementing the document similarity on a computerand Media

Information

post

similar preferences and interests. From these standpoints*1 Professor, Department of Telecommunication and
Network

generallysystem efficiently is an important problem as well.

andMethodology for Twitter data analysis is classified

Telecommunication Engineering, Tokai University

into

―
1 ―
− 22
−

two

types:

therelation-based

type

and

the

Iwao FUJINO and Yuko HOSHINO

ヘッダー

content-based type. In the field of content -based type,

document similarity between any two users of Twitter.

many previous works focus on ranking and clustering

Calculation results of document similarity distribution at

Twitter users according to the similarity of Twitter contents.

both tweet level and author level are plotted in graphs also.

As leading approaches in this filed, two interesting

The remainder of this paper consists of as follows. In

examples are introduced in the following. S. Petrovic, M.

section 2 we will describe basic processing procedure andOsborneon

calculation of weight parameter tfidf and document“Streaming First Story Detection with application tosimilarity. In section 3 we will give the details for

Twitter” [4]. Their approach is based on locality sensitive

implementing the procedure and the calculation of various

hashing method adapted to the first story detection task byparameters. In section 4 we will deal with individual tweet

introducing

to build a similar tweets search system and also to find

and

V.

a

Lavrenkobackoffreported

towards

exact

their

work

search.

The

adaptation eliminates variance in detection results andsimilar tweets and to figure the distribution of similarity

significantly improves the performance of the system.

according to tweet id. In section 5 we will deal with

Meanwhile C. G. Akcora, M. A. Bayir, M. Demirbas and H.

accumulated tweet data according to author to find similar

Ferhatosmanoglu provided their approach on “Identifying

authors and also to figure the distribution of similarity

Breakpoints in Public Opinion” [5].

In this paper, based

according to author name. Finally in section 6, we willon two observations on Twitter and the streaming tfidf

conclude our works and describe some problems for ouralgorithm, they proposed efficient methods to identify

future research.

break points and classify public opinions in a large streamof information. They also reported a method to detect the

2. Basic Procedure of Proposed
Scheme and Calculation of Document
Similarity

changes over time and find related events that caused theopinion changes from Twitter streaming timeline.
In our study as a basic consideration, we think about
the concept of document similarity proposed by Gerard
Salton and Christopher Buckley [6] for information

2.1

Basic procedure of proposed scheme
Regarding the documents in Japanese, the document

retrieval may be applied to Twitter streaming data. We
attempt to devise a content-based scheme that compares

similarity is performed by the following procedures.

similarities between Twitter users by matching their tweets

1.

morphological analysis.

against each other, especially in terms of the new key-value
type database environment. The key-value type database2.
3.

Calculate

the

idf

(inverse

document

frequency)

parameter for each term.

Twitter data is dispatched in the style of attribute and its
value pairs, the key-value type database becomes the mostCalculate the tf (term frequency) parameter for eachterm and each document.

stores data in a collection of (key, value) pairs, such thateach possible key appears at most once in the collection . As

Extract terms from Japanese document s by using

4.

Multiply the tf parameter and the idf parameter to get

data.

the tfidf (term frequency and inverse document

Considering that each tweet is a very short message, we

frequency) weight parameter for each term and eachperformed two levels of data processing in our study. At the

document.

appropriate

databaseforprocessing

Twitterfirst level, called a tweet level, we intend to deal with5.

Calculatedocument

similarity between

any two

documents using the tfidf weight parameter vector.

individual tweet data just as it is. At the second level,
called an author level, we intend to process accumulated
tweets data which is divided according to authors. For bothof these levels, we first use Japanese morphologicalanalysis to pick up terms from Twitter data. Then we
calculate tfidf to give a weight parameter for each term.
Finally we calculate the document similarity from weightparameter vector between any two documents. As aconfirmation work, we will build a simple search engine todiscover similar tweets for a given query or to calculate

2 ―
―
− 23
−

2.2

Weight parameter of terms
In order to numeralize each extracted term, we utilize

tfidf as the weight parameter in our study. There are several
kinds of calculation expression for tfidf [6][7][8], theexpression we use here will be described in the following.
At first we will give the definition expression of term
frequency tf(k, j) [6][7][8].

tf(k, j) is the appearanceFinding Similar Tweets and Similar Users by Applying Document Similarity to Twitter Streaming Data

ヘッダー

frequency of term 𝑡𝑡𝑘𝑘 . If we denote the appearance numberof term 𝑡𝑡𝑘𝑘 in document 𝐷𝐷𝑗𝑗 as 𝑛𝑛𝑘𝑘𝑘𝑘 , we can give term
frequency tf(k, j) as follows,tf(k, j)

,

∑

(1)

,

where the term 𝑡𝑡𝑘𝑘 will become more important in⃗⃗⃗formula is given as follows [6][7][8].
im( ,𝑡𝑡𝑘𝑘 . When term 𝑡𝑡𝑘𝑘 appeared in document 𝐷𝐷𝑗𝑗 at least one
regarding term 𝑡𝑡𝑘𝑘 as one time. The total number of

3.1

|

|

value of df(k) becomes larger. To make it be proportional

to importance of the terms, we choose the logarithmicvalue of inversed df(k) which is called inverse document

frequency idf(k) [6][7][8] and the expression is given as

follows,

l

(

(

) √∑

)

(

)

(6)

Details of gathering
Twitter streaming dataandaccumulating

the sample of Twitter streaming API and accumulated thesefollows.

whereas the term 𝑡𝑡𝑘𝑘 will become less important if theidf(k)√∑∑

data to CouchDB. The details for this step are shown as

(2)

| |

⃗⃗⃗⃗ ⃗⃗⃗⃗|⃗⃗⃗⃗ ||⃗⃗⃗⃗ |In our study, we gathered all the tweets provided bygive document frequency df(k) as follows,
df(k)

)

3. Implementing the Calculation ofDocument Similarity to Twitter
Streaming Datatime, we count the appearance number of document

number of whole document data set as |𝐷𝐷|, then we can(5))

Therefore we can calculate the similarity of two document
vector (⃗⃗⃗i , ⃗⃗⃗ ) by cosine similarity, whose definition

df(k) [6][7][8]. df(k) is the appearance frequency of term

set and we will denote it as |𝑑𝑑𝑘𝑘 |. Also if we denote the total,

the total number of terms appeared in the whole data set.

we will give the definition equation of document frequency

number of document regarding the whole document in data,

where m indicates the dimension of vector space, which is

document 𝐷𝐷𝑗𝑗 if the value of tf(k, j) becomes larger. Next

document where term 𝑡𝑡𝑘𝑘 appeared is called the appearance

(

( )

l

| |

|

|

streaming data on the conditions as follows.
(a) The language of user account is assigned as
Japanese. That is the attribute of author.lang=‟ja‟.
(b) At least one Japanese character is included in the
2. All gathered data is accumulated to CouchDB. To help

factor here. As a result the term 𝑡𝑡𝑘𝑘 will become more
important if the value of idf(k) becomes larger. At last theweight parameter tfidf(k, j) of term 𝑡𝑡𝑘𝑘 in document 𝐷𝐷𝑗𝑗 isdefined by the product of term frequency tf(k, j) anddocument frequency df(k), which is given as follows,tf(k, j) idf(k)as possible, we only pick out the tweets from Twitter

attribute of text.

(3)

where the logarithm operation is used as a attenuation

tfidf(k, j)

1. In order to gather data from Japanese tweet as accurately(4)find the data easily, we store all Twitter data according
to the gathered date with a database name like“„tweets-streaming-datasetnameNO‟,” where NO indicatesthe number of database.

3.2

Details of extracting terms from Japanese
tweet text
Because there is no separation symbol obviously

we will use this parameter as an importance metric for

between words in Japanese, we have to use morphologicalweighting terms in our study.

analysis to separate passages to words. In our study weused a Japanese morphological analyzer named MeCab [9]2.3

to separate words from passages and pick out all nouns ,Document similarity
By using these methods for extracting and weighting

terms in documents, we can express a document as anumerical vector given as follows,

verbs and adjectives as terms. Because the original Mecab
works only on the condition of standard Japanese and there
appear many non-standard transcribe in Twitter, it may notwork properly sometimes for tweet texts. To improve this

3 ―
―
− 24
−

Iwao FUJINO and Yuko HOSHINO

ヘッダー

weak point, we gather non-standard words, notations and3.3

expressions from Twitter and register them to users‟
dictionary of MeCab so that they can be recognized.
Although many efforts have been done, there still remains
some improper terms in the results, so at last we remove
these terms by using a stop word list. The detail flow chart
for this procedure is shown in Figure 1.

Details of calculating tfidf weight parameter
This procedure consists of two steps, the df calculation

step and the tfidf calculation step. In both of these two
steps, the algorithm description is given on the assumptionthat the input data term_by_author is given by a data
structure of {author, terms} pair, where “author” represents
author name, and the “terms” represents an indefinite
length list of terms. The detail flow chart for calculating df

extract terms

parameter is shown in Figure 2 and the detail flow chart forcalculating tfidf weight parameter is shown in Figure 3.

for each database in given data set

3.4
for each tweet data in database

Details of calculating document similarity
This procedure consists of two steps, the norm

calculation step and the similarity calculation step. In bothof these two steps, the algorithm description is given on theauthor=get author name from tweet data

calculate df

parse tweet text to nodelist

for author, termlist in term_by_author

for each node in nodelist

for term in termlist

if node is noun or verb or
adjectiveyes

if term not in author_terms

if node is not a stop word

yes

yes

author_terms[term]=1

append node to termlist
yes

if term in df

df[term]+=1

nodf[term]=1

noterm_by_author[author]=termlist

end

df[term]=df[term]/len(term_by_author) for all term

Fig. 1 Flow chart for extra cting terms from Twitter streaming data

end

set

Fig. 2 Flow chart for calculating df parameter from extracted
terms

4 ―
―
− 25
−

Finding Similar Tweets and Similar Users by Applying Document Similarity to Twitter Streaming Data

ヘッダー

assumption that the input data termtfidf_by_author is given

utilizing the document similarity between given query andby a data structure of {author, termtfidf} pair, where

any tweet from Twitter streaming data. The second is going“author” represents author name, and the “termtfidf”

to find all similar tweets in the Twitter streaming timeline.

represents an indefinite length {term, tfidf} pair of terms.

The third is going to investigate the distribution of

The detail flow chart for calculating norm is shown insimilarity between any two tweets. The summary of data

Figure 4 and the detail flow chart for calculating similarity

set and the results of these experiments will be given in theis shown in Figure 5.

following subsections.
calculate tfidf

calculate norm

for author, termlist in term_by_author

for author, termlist in termtfidf_by_author

for term in termlist[author]

for term in termlist[author]yes

if term in tf[author]tf[author][term]+=1

if author in norm

no

yes

tf[author][term]=1

norm[author]+=tfidf[author][term]*tfidf[author][term]

nonorm[author]=
tfidf[author][term]*tfidf[author][term]

for term in tf[author]

norm[author]=sqrt(norm[author])tf[author][term]=tf[author][term]/len(termlist[author])
tfidf[author][term]=-tf[author][term]*log(df[term])

endFig. 4 Flow chart for calculating norm of author from term
weight parameter saved according to author name

4.1
end

Summary of data set “sample”
The summary of data set “sample” is shown as

follows.
Fig. 3 Flow chart for calculating tfidf parameter from

Table 1 Summary of data set “sample”

extracted terms

4. Applying Document Similarity to
Individual Tweet Text
In this section, we will show three confirmation
experiments using the proposed procedure regarding thetweet level. The first is a simple search engine application

5 ―
―
− 26
−

Database namesample1

sample2

sample3

Crawling dateSize in CouchDB

13/08/09

13/08/11

13/08/16

49.8MB

105.9MB224.9MB

Number of documents

10,08020,05440,065

Number of authors

9,888

10,93837,923

Number of terms

14,41022,54034,123

Iwao FUJINO and Yuko HOSHINO

ヘッダー

4.2

4.3

Search tweets by weighted keywords
The first experiment in tweet level aims to build a

simple search engine application for Twitter. For a query

Finding similar tweets in Twitter streaming
timeline
The second experiment aims to find the document

query={keyword1:weight1,

similarity between any two tweets. In the above sample

keywordn:weightn},

thisdata set there are 70,199 tweets all together. But for theapplication calculates the document similarity between

reason of precision and elapse time of computation, we

given

in

the

form

keyword2:weight2,

of

…..

query and each tweet in data set and outputs the results inonly adopted the tweet which has more than 15 terms as andescending order of the similarity. A screen shot of theeffective tweet and all the effective tweets are 5810 at thissearch results for query={ 大 学 :1.0, 学 祭 :0.5, 文 化

time. The results are saved into CouchDB, which can be

祭 :0.4} is shown Figure 6.

fetched from other applications. To give an example of ourcalculation results, a screen shot of calculation results in
CouchDB is shown in Figure 7.

calculate similarity

In this figure, the value 1means that it is judged that the two tweets are completely
similar to each other. In other words it is judged that all thefor author1, termlist1 in termtfidf_by_author

six tweets in the field of similar_tweet are completely
similar to the tweet in the field of tweet_id.

for author2, termlist2 in termtfidf_by_authorsimi[author1][author2]=0

for term1 in termlist1[author1]

for term2 in termlist1[author2]

if term1==term2
yessimi[author1][author2]=simi[author1][author2]+tfidf[author1][term1]*tfidf[author2][term2]

Fig. 6 Screen shot of search results for query={大学 :1.0, 学祭 :0.5, 文化祭 :0.4}

simi[author1][author2]=simi[author1][author2]/(norm[author1]*norm[author2])

end

Fig. 5 Flow chart for calculating document similarity from

Fig.7 Calculation results of similarity between any twoterm weight parameter saved according to author nametweets (details of document in CouchDB)

6 ―
―
− 27
−

Finding Similar Tweets and Similar Users by Applying Document Similarity to Twitter Streaming Data

ヘッダー

4.4

Distribution of document similarity between
any two tweets
The third experiment aims to calculate the document

5.1

Summary of data set “bigtest”
The summary of data set “bigtest” is shown as

follows.

similarity between any two tweets and to figure the results
in a distribution graph. The result of this experiment isTable 2 Summary of data set “ bi gte st”

shown in Figure 8. The x axis indicates the document

Database namebigtestCrawling date

13/03/13

Size in CouchDB

7.8GB

Number of documents

1,167,309

Number of authors

739,321

Number of terms

232,135

similarity between any two tweets in percentage and theunit level of similarity is 1%. The y axis, which is
graduated in a logarithm scale, indicates the appearance
frequency in percentage, which is normalized by the squareof the number of tweets. This figure shows there are about
98% appearances for the similarity less than 1% and also
shows a steep descent curve as the similarity level
increases until about 96%. But for the similarity level over

5.2

timeline

96%, it shows a very sharp rise, we think that thisphenomenon is caused by retweet, duplication, quotation of
tweeted messages.

Finding similar authors in Twitter streaming
The first experiment in author level aims to find the

document similarity between any two Twitter users. In theabove data set “bigtest” there are 1,167,309 tweets all
together and 739,321 distinct authors. For the reason of
precision and elapse time of computation, we only adopted
effective author on the following conditions: at least 5
tweets are accumulated for an author document and the
document has more than 30 terms. As a result there are9584 effective authors all together. To give an example ofour calculation result, a screen shot of the similarity resultsin CouchDB is shown in Figure 9. This figure shows that
all the author names in the field of similar_author have
similar contents to the author name in the field of author
with a similarity value shown next to the author ‟s name,whereas we only saved the users who have similarity larger

Fig. 8 Distribution of document similarity between any two
tweetsstored in the format of Figure 9.

5. Applying Document Similarity to
Accumulated Tweets

5.3

In this section, we will show two confirmation
experiments using the proposed procedure regarding author
level. The first is going to find all similar authors fromTwitter streaming timeline. The second is going toinvestigate the distribution of similarity between any two
authors. The summary of the data set and the results oftheseexperiments

will

be

given

than 0.5 to CouchDB this time. For all the calculation
results there are 1492 documents which have details data

in

the

following

subsections.

Distribution of document similarity between
any two authors
The second experiment aims to calculate the document

similarity between any two users and to figure the results ina distribution graph. The result of this experiment is shownin Figure 10. The x axis indicates the document similarity
between any two authors in percentage and the unit level ofsimilarity is 1%. The y axis, which is graduated in alogarithm scale, indicates the appearance frequency inpercentage, which is normalized by the square of thenumber of authors. It is observed that the curve in this

7 ―
―
− 28
−

Iwao FUJINO and Yuko HOSHINO

ヘッダー

ヘッダー

ヘッダー

ヘッダー

ery
graph
to similar
thatis of
very
graph
to
Figuresimilar
that
is 8,
of
very
but
to
Figuresimilarthat
it shows
8,of butto
Figureathat
it shows
8,of butFigurea it shows
8, but a it showsa6. Conclusions
6. Conclusions
6. Conclusions
6. Conclusions

ntle
trelative
compareddescent
gentlerelativetocompared
Figuredescent
gentle8.tocompared
Figuredescent8.tocompared
Figure 8.to Figure 8.
In this paper,
In thisaccordingpaper,In thisaccordingto thepaper,
Indocument
thistoaccordingthepaper,
similarity
document
accordingto the similarity
document
to the document
similarity similarity
methodformethodinformationformethodinformation
retrieval,formethodinformation
retrieval,
wefor proposed
information
retrieval,
we proposeda retrieval,
we proposedawe proposedaacontent-based
content-based
implementing
content-based
implementing
schemecontent-based
implementing
forscheme
finding
implementing
forscheme
similarfindingforscheme
similarfindingforsimilarfinding similar
tweets and tweets
similarand
users
tweetssimilarin Twitter
and
users
tweetssimilarin terms
Twitterandusers
similarofin
ina terms
key-value
Twitterusersofina terms
key-value
Twitterofina terms
key-valueof a key-value
type

databasetype environment.
databasetype environment.
database
Basedtype environment.
databaseon Based
the environment.
proposedon Based
the proposedon Based
the proposedon the proposed

procedures procedures
and calculation
procedures
and calculation
details,
procedures
andwecalculation
details,
producedandwecalculation
adetails,
producedcomputer
weadetails,
producedcomputer
weaproduced
computera computer
system to performsystem toconfirmation
perform
system toconfirmation
perform
systemwork. Intoconfirmationconsequence,
perform
work. Inconfirmationconsequence,work.
the In consequence,work.
the In consequence,thethesystem works
systemaccordingworks
systemtoaccordingitsworks
specification
systemtoaccordingitsworks
specification
andto
accordingthe
its validity
specification
andtotheits validity
specificationand the validity
and the validityof our proposedof our implementing
proposedof ourimplementing
proposedschemeof our implementing
proposedis scheme
confirmed
implementing
is confirmed
schemeat the is scheme
confirmed
at the is confirmed
at theat theprototype prototype
level. Althoughprototypelevel. Althoughtheprototype
level.
confirmation
Althoughthelevel.
confirmation
work
Althoughthe confirmation
is work
the confirmation
is work is work is
achieved successfully,achieved successfully,achieved
there aresuccessfully,achieved
still
there
many
aresuccessfully,
stillpractical
there
many
areissues
stillpractical
there
many
areissues
stillpractical
manyissues
practical issues
we have towedeal
havewith.
towedeal
As
have
we
with.
towe
use
deal
As
have
allwe
with.
nouns,
touse
deal
Asall
verbs
we
with.
nouns,
use
and
Asall
verbs
wenouns,useandallverbs
nouns,and verbs andadjectives adjectives
as terms, thereasadjectives
terms,
are too
there
adjectives
as many
terms,
are terms
too
there
as many
terms,
wearehave
terms
too
there
many
towe
are haveterms
too many
towe havetermstowe have toprocess in process
the subsequentin process
the subsequent
procedures.
in process
the subsequent
procedures.
However,in the subsequent
procedures.
many
However,of procedures.
many
However,of many
However,of many ofextracted terms
extracted
may terms
notextracted
describemay terms
notextracted
thedescribemaycontent
terms
nottheofdescribemaycontent
the tweet
nottheofdescribe
content
the tweet
theof content
the tweetof the tweet
plausibly, so
plausibly,that processingsoplausibly,that processing
theseplausibly,
so thatredundant
processingthese
so thatredundant
terms
processing
thesemayredundanttermsthesemayredundantterms mayterms may
exhaust futile
exhaustcomputation
futileexhaustcomputationtime.
futileexhaustAccordingly
computationtime.
futileAccordinglyascomputationtime.
the future
Accordingly
as time.
the future
Accordingly
as the futureas the future
work we will
work
attempt
we willwork
to attempt
develop
we willwork
tomoreattempt
developweefficient
will
to
more
attempt
developalgorithm
efficienttomoredevelopalgorithm
efficientmorealgorithm
efficient algorithm
for extracting
for extracting
terms. forRegardingterms.
extractingforRegardingthe extracting
terms.
computation
Regardingthe terms.
computation
problem,
Regardingthe computation
problem,the computation
problem, problem,
although italthough
is an approach
italthough
is an toapproachitalthoughintroduceis an toapproach
it moreintroduceis anpowerfultoapproachmore
introduce
powerful
to moreintroduce
powerful
more powerfulcomputer systems
computeror systems
highcomputerspeed
or systems
highdatabasecomputerspeedorsystems
systemshighdatabase
speed
certainly,
orsystems
highdatabase
speed
certainly,
systems
databasecertainly,
systems certainly,
adopting parallel
adoptingcomputation
paralleladoptingcomputation
parallelschemeadoptingcomputationmayparallelschemeprovidecomputationmayschememore
providemay
schememore
providemaymoreprovide more
effective solutions
effective because
solutionseffectiveour
becausesolutionseffective
work ourneedsbecausesolutionswork
to process
our
needs
becausework
to ourprocessneeds
work
to processneeds to process
massive irrelevant
massive irrelevant
document
massive irrelevant
document
records.
massive As
irrelevant
document
records.
anotherAs
document
records.
futureanotherAsrecords.
futureanotherAsfuture
another future
work we will
workendeavor
we willwork
toendeavor
wedevelopwill
work
toendeavor
parallelwedevelopwillalgorithm
toendeavor
paralleldevelopfor
algorithm
toparalleldevelopfor
algorithm
parallel foralgorithm forprocessing processing
Twitter streaming
processing
Twitter data.
streaming
processing
Twitter data.
streaming
Twitter data.
streaming data.
esultslculation
Fig. of9 Calculation
similarity
results
Fig. of9 between
Calculation
similarity
results anyof between
similarity
results
two ofany
betweensimilarity
two any
betweentwo any two
We expect We
thatexpect
our works
We
thatexpect
ourwill works
be
We
that
utilized
expectourwill works
bethat
to utilized
prevent
ourwill works
beto utilized
prevent
will beto utilized
prevent to preventthors
ls of document(details
authorsof indocument(details
CouchDB)authorsof in(details
document
CouchDB)of in
document
CouchDB)in CouchDB)
spamming, spamming,
to analyzespamming,
totheanalyze
information
spamming,
totheanalyze
information
propagation
totheanalyze
information
propagation
andtheto information
propagationand to propagationand toand tofind friends
find
in friends
Twitter.
find
inFurthermore,friends
Twitter.
find
inFurthermore,friends
Twitter.
we expect
inFurthermore,
Twitter.
wethatexpect
theFurthermore,
wethatexpect
the wethatexpect
the that the
influences ofinfluences
our worksofinfluences
may
our works
not influencesofbe mayourlimitedworks
not toofbe Twittermay
ourlimitedworks
notspace,tobemay
Twitter
limitednot space,tobe Twitter
limited space,to Twitter space,but it willbut
alsoit contribute
will but
alsoit contributetowillthebut
alsodevelopment
it contributetowillthealso
development
contributeofto social
the developmentof
to social
the developmentof social of social
relations inrelations
the substantial
inrelations
the substantial
world.
inrelations
the substantial
world.
in the substantial
world.
world.

References
References
References
References[1] H. Kwak,[1] H.
C. Kwak,Lee,[1] H.H.
C.Park
Kwak,
Lee,
[1]
and
H.H.
C.S.
Park
Kwak,
Lee,
Moon:
and
H.C.S.
“What
Park
Lee,
Moon:
and
H.
is S.
“What
Park
Moon:
and
is S.
“What
Moon:
is “What isTwitter,

Twitter,
anetworksocialnetworkor media?”a newsaTwitter,social anetwork
Twitter,social or
anetworksociala newsormedia?”a newsormedia?”a newsththth

media?”

thConference
Conference
on 19
World
International
Conferenceon World Conferenceon World on World
ProceedingsofProceedingsProceedingsof 19 International
Proceedingsof 19 Internationalof 19 International
Wide WebWide(WWW‟10),Web Wide(WWW‟10),
pp591-600,
WebWide(WWW‟10),
pp591-600,
Web
Raleigh,(WWW‟10),
pp591-600,
North
Raleigh, pp591-600,
North
Raleigh, North
Raleigh, North
Carolina, USA
Carolina,(2010)USA
Carolina,(2010)USA
Carolina,(2010)USA (2010)[2] “Twitter”,[2] “Twitter”,[2]
http://en.wikipedia.org/wiki/Twitter“Twitter”,[2]
http://en.wikipedia.org/wiki/Twitter“Twitter”,
http://en.wikipedia.org/wiki/Twitter
http://en.wikipedia.org/wiki/Twitter,,,,(Accessed on(Accessed
July 20,on(Accessed
2013)
July 20, (Accessedon
2013)
July 20,on
2013)
July 20, 2013)
istribution
ofFig.
document
10 Distribution
ofFig.
similarity
document
10 Distribution
ofbetween
similarity
document
any
ofbetween
similarity
document
any
betweensimilarity
any
betweenany[3]“Twitter[3]
reaches“Twitterhalf[3]reachesa“Twitterbillionhalf[3]accounts
reaches“Twittera billionhalf
Morereaches
accounts
a billion
thanhalfMore
140
accounts
a billion
than 140
More
accounts
than More
140 than 140
two authorstwo authorstwo authorstwo authors

millions

millions inmillions inmillions
thein the U.S.”,in the U.S.”, the U.S.”,

U.S.”,

http://semiocast.com/en/publications/2012_07_30_Twi
http://semiocast.com/en/publications/2012_07_30_Twi
http://semiocast.com/en/publications/2012_07_30_Twi
http://semiocast.com/en/publications/2012_07_30_Twi

― 8 ―

― 8 ―

― 8 ―

8 ―
―
− 29
−

Finding Similar Tweets and Similar Users by Applying Document Similarity to Twitter Streaming Data

ヘッダー

ヘッダー

es_half_a_billion_accounts_140m_in_the_U
tter_reaches_half_a_billion_accounts_140m_in_the_U

ed on July 20, 2013)
S, (Accessed on July 20, 2013)

c, M. Osborne[4 ] andS. Petrovic,
V. Lavrenko:
M. Osborne“Streaming
and V. Lavrenko: “Streamingy Detection withFirstapplication
Story Detection
to Twitter”,
with application to Twitter”,

anguage Technologies:
Human Language
The 2010Technologies:
AnnualThe 2010 Annual

e of the North Conference
American Chapterof the Northof ACL,
American Chapter of ACL,

9, Los Angeles,pp.181-189,
California, USA
Los Angeles,(2010) California, USA (2010)

cora, M. A.
[5 ] Bayir,
C. G. M.
Akcora,Demirbas
M. A.
andBayir,
H. M. Demirbas and H.

anoglu: “Identifing
Ferhatosmanoglu:
Breakpoints “Identifingin Public Breakpoints in Public

on Social 1 stMediaworkshopAnalytics
on Social Media Analytics
1 st workshop Opinon”,

0), Washington,(SOMA‟10),
DC, USA (2010)
Washington, DC, USA (2010)

alton

and[6]Christopher
Gerard Salton
Buckley:and “Term
Christopher

Buckley:“TermApproaches inWeighting
AutomaticApproaches
Text Retrieval”,
in Automatic Text Retrieval”,nProcessing Informationand Management,
Processing
25(5),
and3 (1988)

Management,

25(5),

pp.513-523 (1988)

Croft, Donald[7] Metzler
W. Bruce
andCroft,
TrevorDonaldStrohman:
Metzler and Trevor Strohman:

ngines Information“SearchRetrievalEnginesin Information
Practice”, Retrieval in Practice”,

Wesley, pp.237-243
Addison(2010)
Wesley, pp.237-243 (2010)

Baeza-Yates[8] and
RicardoBerthier
Baeza-Yates
Ribeiro-Neto:and BerthierRibeiro-Neto:

Information Retrieval“Modern the
Information
concepts Retrieval
andthe concepts andy behind searchtechnology
Second edition”,
behind search
AddisonSecond edition”, Addisonp.66-79 (2011) Wesley, pp.66-79 (2011)
Ye t

An
[9ot
] h“Meer CabP art: -o Yef-Stp eeAnch ot haerndgi ca lM or ph ol ogi ca l An al yze r ”,

P art -o f-S p ee chandAn al yze r ”,

ab.googlecode.com/svn/trunk/mecab/doc/ind
http://mecab.googlecode.com/svn/trunk/mecab/doc/ind

Accessed on Julyex.html
20, 2013)( Accessed on July 20, 2013)

― 9 ―

9 ―
―
− 30
−