Modeling Social Annotation Datawith Content Relevance using a Topic Model

Tomoharu Iwata
Takeshi Yamada
Naonori Ueda
NTT Communication Science Laboratories
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, Japan{iwata,yamada,ueda}@cslab.kecl.ntt.co.jp

Abstract
We propose a probabilistic topic model for analyzing and extracting contentrelated annotations from noisy annotated discrete data such as web pages storedin social bookmarking services. In these services, since users can attach annotations freely, some annotations do not describe the semantics of the content, thus
they are noisy, i.e. not content-related. The extraction of content-related annotations can be used as a preprocessing step in machine learning tasks such as textclassification and image recognition, or can improve information retrieval performance. The proposed model is a generative model for content and annotations, inwhich the annotations are assumed to originate either from topics that generated
the content or from a general distribution unrelated to the content. We demonstrate
the effectiveness of the proposed method by using synthetic data and real social
annotation data for text and images.

1 Introduction
Recently there has been great interest in social annotations, also called collaborative tagging or
folksonomy, created by users freely annotating objects such as web pages [7], photographs [9],blog posts [23], videos [26], music [19], and scientific papers [5]. Delicious [7], which is a social
bookmarking service, and Flickr [9], which is an online photo sharing service, are two representativesocial annotation services, and they have succeeded in collecting huge numbers of annotations. Since
users can attach annotations freely in social annotation services, the annotations include those that do
not describe the semantics of the content, and are, therefore, not content-related [10]. For example,
annotations such as ’nikon’ or ’canon’ in a social photo service often represent the name of the
manufacturer of the camera with which the photographs were taken, or annotations such as ’2008’
or ’november’ indicate when they were taken. Other examples of content-unrelated annotations
include those designed to remind the annotator such as ’toread’, those identifying qualities such as’great’, and those identifying ownership.
Content-unrelated annotations can often constitute noise if used for training samples in machinelearning tasks, such as automatic text classification and image recognition. Although the performance of a classifier can generally be improved by increasing the number of training samples, noisy
training samples have a detrimental effect on the classifier. We can improve classifier performance
if we can employ huge amounts of social annotation data from which the content-unrelated annotations have been filtered out. Content-unrelated annotations may also constitute noise in information
retrieval. For example, a user may wish to retrieve a photograph of a Nikon camera rather than aphotograph taken by a Nikon camera.
In this paper, we propose a probabilistic topic model for analyzing and extracting content-related
annotations from noisy annotated data. A number of methods for automatic annotation have been
proposed [1, 2, 8, 16, 17]. However, they implicitly assume that all annotations are related to content,
1Symbol
D
W
T
K
Nd
Md
wdn
zdn
tdm
cdm
rdm

Table 1: Notation
Description
number of documents
number of unique words
number of unique annotations
number of topics
number of words in the dth document
number of annotations in the dth document
nth word in the dth document, wdn ∈ {1, · · · , W }topic of the nth word in the dth document, zdn ∈ {1, · · · , K}
mth annotation in the dth document, tdm ∈ {1, · · · , T }topic of the mth annotation in the dth document, cdm ∈ {1, · · · , K}
relevance to the content of the mth annotation of the dth document,rdm = 1 if relevant, rdm = 0 otherwise

and to the best of our knowledge, no attempt has been made to extract content-related annotations
automatically. The extraction of content-related annotations can improve performance of machinelearning and information retrieval tasks. The proposed model can also be used for the automatic
generation of content-related annotations.
The proposed model is a generative model for content and annotations. It first generates content, andthen generates the annotations. We assume that each annotation is associated with a latent variable
that indicates whether it is related to the content or not, and the annotation originates either fromthe topics that generated the content or from a content-unrelated general distribution depending on
the latent variable. The inference can be achieved based on collapsed Gibbs sampling. Intuitively
speaking, this approach considers an annotation to be content-related when it is almost always attached to objects in a specific topic. As regards real social annotation data, the annotations are notexplicitly labeled as content related/unrelated. The proposed model is an unsupervised model, andso can extract content-related annotations without content relevance labels.
The proposed method is based on topic models. A topic model is a hierarchical probabilistic model,
in which a document is modeled as a mixture of topics, and where a topic is modeled as a probability distribution over words. Topic models are successfully used for a wide variety of applications
including information retrieval [3, 13], collaborative filtering [14], and visualization [15] as well as
for modeling annotated data [2].
The proposed method is an extension of the correspondence latent Dirichlet allocation (CorrLDA) [2], which is a generative topic model for contents and annotations. Since Corr-LDA assumes
that all annotations are related to the content, it cannot be used for separating content-related annotations from content-unrelated ones. A topic model with a background distribution [4] assumes
that words are generated either from a topic-specific distribution or from a corpus-wide background
distribution. Although this is a generative model for documents without annotations, the proposedmodel is related to the model in the sense that data may be generated from a topic-unrelated distribution depending on a latent variable.
In the rest of this paper, we assume that the given data are annotated document data, in which thecontent of each document is represented by words appearing in the document, and each document
has both content-related and content-unrelated annotations. The proposed model is applicable to awide range of discrete data with annotations. These include annotated image data, where each image
is represented with visual words [6], and annotated movie data, where each movie is represented byuser ratings.

2

Proposed method

Suppose that, we have a set of D documents, and each document consists of a pair of words anddannotations (wd , td ), where wd = {wdn }Nn=1 is the set of words in a document that represents theMdcontent, and td = {tdm }m=1 is the set of assigned annotations, or tags. Our notation is summarizedin Table 1.
2

α

η

λ

θ

z

w

c

t

r

N

φ
ψ

K

K+1

β
γ

M D

Figure 1: Graphical model representation of the proposed topic model with content relevance.
The proposed topic model first generates the content, and then generates the annotations. The generative process for the content is the same as basic topic models, such as latent Dirichlet allocation(LDA) [3]. Each document has topic proportions θd that are sampled from a Dirichlet distribution.
For each of the Nd words in the document, a topic zdn is chosen from the topic proportions, and thenword wdn is generated from a topic-specific multinomial distribution φzdn . In the generative process for annotations, each annotation is assessed as to whether it is related to the content or not. In
particular, each annotation is associated with a latent variable rdm with value rdm = 0 if annotation
tdm is not related to the content; rdm = 1 otherwise. If the annotation is not related to the content,
rdm = 0, annotation tdm is sampled from general topic-unrelated multinomial distribution ψ0 . If
the annotation is related to the content, rdm = 1, annotation tdm is sampled from topic-specificmultinomial distribution ψcdm , where cdm is the topic for the annotation. Topic cdm is sampledduniform randomly from topics zd = {zdn }N
n=1 that have previously generated the content. Thismeans that topic cdm is generated from a multinomial distribution, in which P (cdm = k) = NNkd,dwhere Nkd is the number of words that are assigned to topic k in the dth document.
In summary, the proposed model assumes the following generative process for a set of annotated
documents {(wd , td )}D
d=1 ,
1. Draw relevance probability λ ∼ Beta(η)
2. Draw content-unrelated annotation probability ψ0 ∼ Dirichlet(γ)
3. For each topic k = 1, · · · , K:(a) Draw word probability φk ∼ Dirichlet(β)(b) Draw annotation probability ψk ∼ Dirichlet(γ)
4. For each document d = 1, · · · , D:(a) Draw topic proportions θd ∼ Dirichlet(α)(b) For each word n = 1, · · · , Nd :i. Draw topic zdn ∼ Multinomial(θd )ii. Draw word wdn ∼ Multinomial(φzdn )(c) For each annotation m = 1, · · · , Md :i. Draw topic cdm ∼ Multinomial({ NNkd
}K
k=1 )
d
ii. Draw relevance rdm ∼ Bernoulli(λ){
Multinomial(ψ0 )if rdm = 0iii. Draw annotation tdm ∼Multinomial(ψcdm ) otherwise
where α, β and γ are Dirichlet distribution parameters, and η is a beta distribution parameter. Figure 1 shows a graphical model representation of the proposed model, where shaded and unshaded
nodes indicate observed and latent variables, respectively.
As with Corr-LDA, the proposed model first generates the content and then generates the annotations
by modeling the conditional distribution of latent topics for annotations given the topics for thecontent. Therefore, it achieves a comprehensive fit of the joint distribution of content and annotations
and finds superior conditional distributions of annotations given content [2].
The joint distribution on words, annotations, topics for words, topics for annotations, and relevance
given parameters is described as follows:
P (W , T , Z, C, R|α, β, γ, η) = P (Z|α)P (W |Z, β)P (T |C, R, γ)P (R|η)P (C|Z), (1)
3

Md
D
D
D
where W = {wd }D
d=1 , T = {td }d=1 , Z = {zd }d=1 , C = {cd }d=1 , cd = {cdm }m=1 , R =MdD{rd }D
d=1 , and rd = {rdm }m=1 . We can integrate out multinomial distribution parameters, {θd }d=1 ,K
K{φk }k=1 and {ψk0 }k0 =0 , because we use Dirichlet distributions for their priors, which are conjugate
to multinomial distributions. The first term on the right hand side of (1) is calculated by P (Z|α) =∏D ∫D
d=1 P (zd |θd )P (θd |α)dθd , and we have the following equation by integrating out {θd }d=1 ,()D ∏ Qk Γ(Nkd +α)
P (Z|α) = Γ(αK)
d Γ(Nd +αK) , where Γ(·) is the gamma function. Similarly, the second
Γ(α)K)K ∏ Q()
w Γ(Nkw +β)
term is given as follows, P (W |Z, β) = Γ(βWk Γ(Nk +βW ) , where Nkw is the number
Γ(β)W∑of times word w has been assigned to topic k, and Nk =w Nkw . The third term is given as)K+1 ∏ Q
(
Γ(M
+γ)
)
0
0
k0 t
t
follows, P (T |C, R, γ) = Γ(γT
k0 Γ(Mk0 +γT ) , where k ∈ {0, · · · , K}, and k = 0
Γ(γ)Tindicates irrelevant to the content. Mk0 t is the number of times annotation t has been ∑identified
as content-unrelated if k 0 = 0, or as content-related topic k 0 if k 0 6= 0, and Mk0 =t Mk0 t .
The Bernoulli parameter λ can also be integrated out because we use a beta distribution for theprior, which is conjugate to a Bernoulli distribution. The fourth term is given as follows, P (R|η) =Γ(2η) Γ(M0 +η)Γ(M −M0 +η), where M is the number of annotations, and M0 is the number of contentΓ(η)2
Γ(M +2η)
0
)Mkd
∏ ∏ (
, whereunrelated annotations. The fifth term is given as follows, P (C|Z) = d k NNkdd0
Mkd
is the number of annotations that are assigned to topic k in the dth document.

The inference of the latent topics Z given content W and annotations T can be efficiently computed
using collapsed Gibbs sampling [11]. Given the current state of all but one variable, zj , wherej = (d, n), the assignment of a latent topic to the nth word in the dth document is sampled from,
(
) 0
(
) 0
Nkd\j + α Nkwj \j + β Nkd\j + 1 Mkd ∏ Nld\j Mld
P (zj = k|W , T , Z\j , C, R) ∝,Nd\j + αK Nk\j + βW
Nd
Nd
l6=k

where \j represents the count when excluding the nth word in the dth document. Given the current
state of all but one variable, ri , where i = (d, m), the assignment of either relevant or irrelevant to
the mth annotation in the dth document is estimated as follows,
P (ri = 0|W , T , Z, C, R\i ) ∝P (ri = 1|W , T , Z, C, R\i ) ∝M0\i + η M0ti \i + γ,
M\i + 2η M0\i + γT

M\i − M0\i + η Mci ti \i + γ.
M\i + 2ηMci \i + γT

(2)

The assignment of a topic to a content-unrelated annotation is estimated as follows,
P (ci = k|ri = 0, W , T , Z, C\i , R\i ) ∝

Nkd
,
Nd

(3)

and the assignment of a topic to a content-related annotation is estimated as follows,
P (ci = k|ri = 1, W , T , Z, C\i , R\i ) ∝Mkti \i + γ Nkd.
Mk\i + γT Nd(4)

The parameters α, β, γ, and η can be estimated by maximizing the joint distribution (1) by the
fixed-point iteration method described in [21].

3Experiments

3.1

Synthetic content-unrelated annotations

We evaluated the proposed method quantitatively by using labeled text data from the 20 Newsgroups
corpus [18] and adding synthetic content-unrelated annotations. The corpus contains about 20,000
articles categorized into 20 discussion groups. We considered these 20 categories as content-related
annotations, and we also randomly attached dummy categories to training samples as contentunrelated annotations. We created two types of training data, 20News1 and 20News2, where the4former was used for evaluating the proposed method when analyzing data with different numbersof content-unrelated annotations per document, and the latter was used with different numbers of
unique content-unrelated annotations. Specifically, in the 20News1 data, the unique number ofcontent-unrelated annotations was set at ten, and the number of content-unrelated annotations per
document was set at {1, · · · , 10}. In the 20News2 data, the unique number of content-unrelated
annotations was set at {1, · · · , 10}, and the number of content-unrelated annotations per document
was set at one. We omitted stop-words and words that occurred only once. The vocabulary size was
52,647. We sampled 100 documents from each of the 20 categories, for a total of 2,000 documents.
We used 10 % of the samples as test data.
We compared the proposed method with MaxEnt and Corr-LDA. MaxEnt represents a maximum
entropy model [22] that estimates the probability distribution that maximizes entropy under theconstraints imposed by the given data. MaxEnt is a discriminative classifier and achieves high performance as regards text classification. In MaxEnt, the hyper-parameter that maximizes the performance was chosen from {10−3 , 10−2 , 10−1 , 1}, and the input word count vector was normalized sothat the sum of the elements was one. Corr-LDA [2] is a topic model for words and annotations that
does not take the relevance to content into consideration. For the proposed method and Corr-LDA,
we set the number of latent topics, K, to 20, and estimated latent topics and parameters by using
collapsed Gibbs sampling and the fixed-point iteration method, respectively.
We evaluated the predictive performance of each method using the perplexity of held-out contentrelated annotations given the content. A lower perplexity represents higher predictive performance.
In the proposed method, we calculated the probability of content-related annotation t in the dth∑
is adocument given the training samples as follows, P (t|d, D) ≈ k θ̂dk ψ̂kt , where θ̂dk = NNkdd
Mkt +γpoint estimate of the topic proportions for annotations, and ψ̂kt = M
is a point estimate of thek +γT
annotation multinomial distribution. Note that no content-unrelated annotations were attached to thetest samples. The average perplexities and standard deviations over ten experiments on the 20News1
and 20News2 data are shown in Figure 2 (a). In all cases, when content-unrelated annotations were
included, the proposed method achieved the lowest perplexity, indicating that it can appropriately
predict content-related annotations. Although the perplexity achieved by MaxEnt was slightly lower
than that of the proposed method without content-unrelated annotations, the performance of MaxEnt
deteriorated greatly when even one content-unrelated annotation was attached. Since MaxEnt is asupervised classifier, it considers all attached annotations to be content-related even if they are not.
Therefore, its perplexity is significantly high when there are fewer content-related annotations per
document than unrelated annotations as with the 20News1 data. In contrast, since the proposed
method considers the relevance to the content for each annotation, it always offered low perplexity
even if the number of content-unrelated annotations was increased. The perplexity achieved by
Corr-LDA was high because it does not consider the relevance to the content as in MaxEnt.

We evaluated the performance in terms of extracting content-related annotations. We considered extraction as a binary classification problem, in which each annotation is classified as either contentrelated or content-unrelated. As the evaluation measurement, we used F-measure, which is the
harmonic mean of precision and recall. We compared the proposed method to a baseline method
in which the annotations are considered to be content-related if any of the words in the annotations
appear in the document. In particular, when the category name is ’comp.graphics’, if ’computer’ or’graphics’ appears in the document, it is considered to be content-related. We assume that the baseline method knows that content-unrelated annotations do not appear in any document. Therefore,
the precision of the baseline method is always one, because the number of false positive samples iszero. Note that this baseline method does not support image data, because words in the annotations
never appear in the content. F-measures for the 20News1 and 20News2 data are shown in Figure 2 (b). A higher F-measure represents higher classification performance. The proposed method
achieved high F-measures with a wide range of ratios of content-unrelated annotations. All of theF-measures achieved by the proposed method exceeded 0.89, and the F-measure without unrelated
annotations was one. This result implies that it can flexibly handle cases with different ratios ofcontent-unrelated annotations. The F-measures achieved by the baseline method were low because
annotations might be related to the content even if the annotations did not appear in the document.
On the other hand, the proposed method considers that annotations are related to the content when
the topic, or latent semantics, of the content and the topic of the annotations are similar even if theannotations did not appear in the document.

5

20News1
45
40

Proposed
Corr-LDA
MaxEntF-measureperplexity

35
30
25
20
15

1

1

0.8

0.8
lambda

50

0.6
0.4
0.2

Estimated
True

0.6
0.4
0.2

Proposed
Baseline

10
5

0
0

2

4

6

8

10

0
0

number of content-unrelated annotations per document

2

4

6

8

10

0

number of content-unrelated annotations per document

2

4

6

8

10

number of content-unrelated annotations per document

20News216

Proposed
Corr-LDA
MaxEntF-measureperplexity

14
12
10

1

1

0.8

0.8
lambda

18

0.6
0.4

8

0.2

6

0

Estimated
True

0.6
0.4
0.2

Proposed
Baseline
0

2

4

6

8

10

number of unique content-unrelated annotations(a) Perplexity

0
0

2

4

6

8

10

number of unique content-unrelated annotations(b) F-measure

0

2

4

6

8

10

number of unique content-unrelated annotations(c) λ̂

Figure 2: (a) Perplexities of the held-out content-related annotations, (b) F-measures of content
relevance, and (c) Estimated content-related annotation ratios in 20News data.
Figure 2 (c) shows the content-related annotation ratios as estimated by the following equation,−M0 +η
λ̂ = MM+2η , with the proposed method. The estimated ratios are about the same as the true
ratios.
3.2

Social annotations

We analyzed the following three sets of real social annotation data taken from two social bookmarking services and a photo sharing service, namely Hatena, Delicious, and Flickr.
From the Hatena data, we used web pages and their annotations in Hatena::Bookmark [12], which
is a social bookmarking service in Japan, that were collected using a similar method to that usedin [25, 27]. Specifically, first, we obtained a list of URLs of popular bookmarks for October 2008.
We then obtained a list of users who had bookmarked the URLs in the list. Next, we obtained a newlist of URLs that had been bookmarked by the users. By iterating the above process, we collected
a set of web pages and their annotations. We omitted stop-words and words and annotations that
occurred in fewer than ten documents. We omitted documents with fewer than ten unique words
and also omitted those without annotations. The numbers of documents, unique words, and uniqueannotations were 39,132, 8,885, and 43,667, respectively. From the Delicious data, we used web
pages and their annotations [7] that were collected using the same method used for the Hatena data.
The numbers of documents, unique words, and unique annotations were 65,528, 30,274, and 21,454,
respectively. From the Flickr data, we used photographs and their annotations Flickr [9] that werecollected in November 2008 using the same method used for the Hatena data. We transformed photo
images into visual words by using scale-invariant feature transformation (SIFT) [20] and k-means as
described in [6]. We omitted annotations that were attached to fewer than ten images. The numbersof images, unique visual words, and unique annotations were 12,711, 200, and 2,197, respectively.
For the experiments, we used 5,000 documents that were randomly sampled from each data set.
Figure 3 (a)(b)(c) shows the average perplexities over ten experiments and their standard deviation
for held-out annotations in the three real social annotation data sets with different numbers of topics.
Figure 3 (d) shows the result with the Patent data as an example of data without content unrelated
annotations. The Patent data consist of patents published in Japan from January to March in 2004,
to which International Patent Classification (IPC) codes were attached by experts according to theircontent. The numbers of documents, unique words, and unique annotations (IPC codes) were 9,557,
6

4000

9000
Proposed
CorrLDA

Proposed
CorrLDA
8000

3500

7000
perplexity

perplexity

3000

2500

6000
5000

2000
4000
1500

3000

1000

2000
0

20

40

60

80

100

0

20

number of topics

4060(a) Hatena

100(b) Delicious

2200

2000
Proposed
CorrLDA

Proposed
CorrLDA

2000

1800

1800

1600
perplexity

perplexity

80

number of topics

1600
1400

1400
1200

1200

1000

1000

800

800

600
0

20

40

60

80

100

0

number of topics

20

40

60

80

100

number of topics(c) Flickr(d) Patent

Figure 3: Perplexities of held-out annotations with different numbers of topics in social annotation
data (a)(b)(c), and in data without content unrelated annotations (d).

canada banking toread
London river london history reference imported England
blog ruby rails cell person misc Ruby plugin cpu ajax javascript exif php
future distribution internet prediction Internet computer computers no_tag bandwidthfilm Art good mindfuck movies list blog
ricette cucina cooking italy search recipes italian food cook news reference searchengine list italiano links
ruby git diff useful triage imported BookmarksBar blog
SSD toread ssd
c# interview programming C# .net todo language tips microsoft
google gmail googlecalendar Web-2.0 Gmail via:mento.info

Figure 4: Examples of content-related annotations in the Delicious data extracted by the proposed
method. Each row shows annotations attached to a document; content-unrelated annotations areshaded.

104,621, and 6,117, respectively. With the Patent data, the perplexities of the proposed method
and Corr-LDA were almost the same. On the other hand, with the real social annotation data, theproposed method achieved much lower perplexities than Corr-LDA. This result implies that it isimportant to consider relevance to the content when analyzing noisy social annotation data. The
perplexity of Corr-LDA with social annotation data gets worse as the number of topics increases
because Corr-LDA overfits noisy content-unrelated annotations.
The upper half of each table in Table 2 shows probable content-unrelated annotations in the leftmost
column, and probable annotations for some topics, which were estimated with the proposed method
using 50 topics. The lower half in (a) and (b) shows probable words in the content for each topic.
With the Hatena data, we translated Japanese words into English, and we omitted words that had
the same translated meaning in a topic. For content-unrelated annotations, words that seemed tobe irrelevant to the content were extracted, such as ’toread’, ’later’, ’*’, ’?’, ’imported’, ’2008’,’nikon’, and ’cannon’. Each topic has characteristic annotations and words, for example, Topic1
in the Hatena data is about programming, Topic2 is about games, and Topic3 is about economics.
Figure 4 shows some examples of the extraction of content-related annotations.
7

Table 2: The ten most probable content-unrelated annotations (leftmost column), and the ten mostprobable annotations for some topics (other columns), estimated with the proposed method using 50
topics. Each column represents one topic. The lower half in (a) and (b) shows probable words in thecontent.
(a) Hatena
unrelatedtoread
web
later
great
documenttroll*?
summary
memo

Topic1
programming
development
dev
webdevphp
java
softwarerubyopensource
softwaredev
development
webserieshp
technology
management
sourceusage
project
systemTopic2game
animation
movie
Nintendo
movie
event
xbox360
DS
PS3
animation
game
animation
movie
story
work
createPG
mrinterestingworld

Topic3
economics
financesocietybusiness
economyreading
investment
japan
money
companyyear
articlefinanceeconomics
investmentcompanyday
management
information
nikkeiTopic4scienceresearch
biology
study
psychology
mathematics
pseudoscienceknowledge
educationmathscienceresearcher
answerspiritquestion
human
ehara
proof
mind
brain

Topic5foodcooking
gourmetrecipecook
life
fooditemfoods
alcohol
foodieeat
useomission
water
decision
broil
face
input
miss
food

Topic6linux
tips
windows
security
server
network
unix
mysql
mail
Apacheinsetting
file
servercase
mailaddressconnectionaccesssecurity

Topic7
politics
internationaloverseasocietyhistory
china
world
international
usa
news
japan
countryusa
china
politics
aso
mr
korea
human
peopleTopic8pc
apple
iphonehardwaregadgetmac
cupiditytechnology
ipodelectronics
yen
product
digitalpc
support
in
note
price
equipment
model

Topic9medicalhealthlie
government
agriculturefoodmentalhealthmentalenvironment
sciencericebananamedicaldiethospitalpoisoneat
incident
korea
jelly

iphonemobilehardwaregames
iPhoneapple
techgamingmac
gameiphoneapple
ipod
mobile
game
games
pc
phone
mac
touch

education
learning
books
book
languagelibraryschoolteaching
Education
research
book
legal
theorybooks
lawuniversity
students
learning
education
language(b) Delicious
referenceweb
imported
designinternet
onlinecool
toread
tools
blog

money
financeeconomics
business
economyFinancefinancialinvestingbailout
financesmoney
financialcreditmarketeconomic
october
economybanks
government
bank

video
music
videosfun
entertainment
funny
movies
media
Video
film
music
video
link
tv
movie
itunesfilm
amazonplay
interview

opensource
softwareprogramming
development
linux
tools
rails
ruby
webdev
rubyonrails
project
code
serverrubyrailssourcefileversionfiles
development

food
recipes
recipecooking
Food
Recipesbakinghealthvegetarian
diy
recipefood
recipes
make
wine
made
add
love
eat
good

windowslinux
sysadminWindows
security
computermicrosoft
networkLinuxubuntuwindows
systemmicrosoft
linux
softwarefileserveruser
files
ubuntuart
photo
photography
photosPhotographyArtinspiration
music
foto
fotografia
art
photography
photoscameravol
digitalimages2008
phototracksshopping
shop
Shoppinghomewishlistbuy
store
fashion
gifts
house
buy
onlinepricecheap
productorder
freeproducts
ratingcard(c) Flickr
2008
nikon
canon
white
yellow
red
photo
italy
california
color

4

dance
bar
dc
digitalconcertbands
music
washingtondc
dancingworksea
sunsetsky
cloudsmountainsocean
panorama
southirelandoregonautumntrees
treemountainfall
gardenbortescristiangeotaggedmudnatura

rock
house
party
park
inn
coach
creature
halloween
mallory
night

beach
travelvacation
campinglandscape
texas
lake
cameraphonemdsunfamilyportrait
cute
baby
boy
kids
brown
closeup08
galvestonislandasia
landscape
rock
blue
tour
plant
tourguidesoma
koh
samui

Conclusion

We have proposed a topic model for extracting content-related annotations from noisy annotateddata. We have confirmed experimentally that the proposed method can extract content-related annotations appropriately, and can be used for analyzing social annotation data. In future work, we will
determine the number of topics automatically by extending the proposed model to a nonparametric Bayesian model such as the Dirichlet process mixture model [24]. Since the proposed method
is, theoretically, applicable to various kinds of annotation data, we will confirm this in additional
experiments.
8References[1] K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D. M. Blei, and M. I. Jordan. Matching words andpictures. Journal of Machine Learning Research, 3:1107–1135, 2003.
[2] D. M. Blei and M. I. Jordan. Modeling annotated data. In SIGIR ’03: Proceedings of the 26th Annual
International ACM SIGIR Conference on Research and Development in Information Retrieval, pages
127–134, 2003.
[3] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning
Research, 3:993–1022, 2003.
[4] C. Chemudugunta, P. Smyth, and M. Steyvers. Modeling general and specific aspects of documentswith a probabilistic topic model. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural
Information Processing Systems 19, pages 241–248. MIT Press, 2007.
[5] CiteULike. http://www.citeulike.org.
[6] G. Csurka, C. Dance, J. Willamowski, L. Fan, and C. Bray. Visual categorization with bags of keypoints.
In ECCV International Workshop on Statistical Learning in Computer Vision, 2004.
[7] Delicious. http://delicious.com.
[8] S. Feng, R. Manmatha, and V. Lavrenko. Multiple Bernoulli relevance models for image and video
annotation. In CVPR ’04: Proceedings of the IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, volume 2, pages 1002–1009, 2004.
[9] Flickr. http://flickr.com.
[10] S. Golder and B. A. Huberman. Usage patterns of collaborative tagging systems. Journal of Information
Science, 32(2):198–208, 2006.
[11] T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of
Sciences, 101 Suppl 1:5228–5235, 2004.
[12] Hatena::Bookmark. http://b.hatena.ne.jp.
[13] T. Hofmann. Probabilistic latent semantic analysis. In UAI ’99: Proceedings of 15th Conference on
Uncertainty in Artificial Intelligence, pages 289–296, 1999.
[14] T. Hofmann. Collaborative filtering via Gaussian probabilistic latent semantic analysis. In Proceedingsof the 26th Annual International ACM SIGIR Conference on Research and Development in Information
Retrieval, pages 259–266. ACM Press, 2003.
[15] T. Iwata, T. Yamada, and N. Ueda. Probabilistic latent semantic visualization: topic model for visualizing
documents. In KDD ’08: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, pages 363–371. ACM, 2008.
[16] J. Jeon, V. Lavrenko, and R. Manmatha. Automatic image annotation and retrieval using cross-media
relevance models. In SIGIR ’03: Proceedings of the 26th Annual International ACM SIGIR Conference
on Research and Development in Information Retrieval, pages 119–126. ACM, 2003.
[17] J. Jeon and R. Manmatha. Using maximum entropy for automatic image annotation. In CIVR ’04:
Proceedings of the 3rd International Conference on Image and Video Retrieval, pages 24–32, 2004.
[18] K. Lang. NewsWeeder: learning to filter netnews. In ICML ’95: Proceedings of the 12th International
Conference on Machine Learning, pages 331–339, 1995.
[19] Last.fm. http://www.last.fm.
[20] D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer
Vision, 60(2):91–110, 2004.
[21] T. Minka. Estimating a Dirichlet distribution. Technical report, M.I.T., 2000.
[22] K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. In Proceedingsof IJCAI-99 Workshop on Machine Learning for Information Filtering, pages 61–67, 1999.
[23] Technorati. http://technorati.com.
[24] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Journal of theAmerican Statistical Association, 101(476):1566–1581, 2006.
[25] X. Wu, L. Zhang, and Y. Yu. Exploring social annotations for the semantic web. In WWW ’06: Proceedings of the 15th International Conference on World Wide Web, pages 417–426. ACM, 2006.
[26] YouTube. http://www.youtube.com.
[27] D. Zhou, J. Bian, S. Zheng, H. Zha, and C. L. Giles. Exploring social annotations for information retrieval.
In WWW ’08: Proceeding of the 17th International Conference on World Wide Web, pages 715–724.
ACM, 2008.

9