Automatic Construction of A Folksonomy-based Visual Ontology
Hidetoshi Kawakubo, Yuuta Akima and Keiji Yanai
Department of Computer Science, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan
Email: {kawaku-h,akima-y,yanai}mm.cs.uec.ac.jp

Abstract—Recently, Folksonomy attracts attentions as a
new method to index large-scale image databases. In theFolksonomy-style image databases, they allows users to attach
keywords to images as “tags”. Since tag words are uncontrolled,they have various and many kinds of tags associated with images. This is much different from conventional image databases.
In this paper, we propose a novel method to extract hierarchicalstructure on relations between tags from Folksonomy. The
tag structure we extract can be used as an ontology forimage database search which reflects both textual and visual
relations between tags. In the proposed method, at first, wecollect millions of tag-attached-images from Flickr which is theworld-largest Folksonomy-style image database, and remove
noise images from them. Next, we estimate concept vectors
for highly-frequent tags based on only visual features, only
tag word features and combined features of both visual andtextual features, and compute JS divergence and entropy forthree kinds of concept vectors. Finally we estimate hierarchicalstructures between tags regarding three kinds of concept
vectors. In the experiments, we show the obtained hierarchicalstructure, and it includes interesting relations which sometimes
are difficult to be discovered by human. In addition, as its
application, we used and evaluated the obtained ontology forquery expansion of text-tag-based image search over Flickr.
These results indicate that the proposed method is promisingand the structure is expected to help image search and someother applications.
Keywords-visual ontology, Folksonomy, PLSA, DAG, social
media, Flickr

I. I NTRODUCTION
Recently, a large number of images have become available
on the Web. Especially, photo sharing sites such as Flickr
and Panoramio gather a huge number of consumer photos
uploaded by many people. Such sites called “social media”
allow users to add keywords as “tags” to uploaded photos tomake easy to search photo databases. Since the vocabularyof tags user can add to photos is not restricted, various andmany kinds of words are used as tags. To index images
with uncontrolled tags is called “Folksonomy” in contrast
to “Taxonomy” where the vocabulary of keywords attached
to images is controlled. Folksonomy is common methods toindex large-scale social image databases in these days.
Uncontrolled vocabulary in Folksonomy is its advantage
and its drawback at the same time. Uncontrolled vocabulary
enables anyone to tag photos easily, while keywords tagged
to photos vary depending on users. For example, someone
might tag a beach photo with “sea”, while the other one
might tag the same photo with “ocean”. In this case, “ocean”
should be identified with a subordinate concept of “ocean”.
To do that, “ontology” is needed, which describes relations
between words including a hierarchical structure. Amongexisting ontologies, “WordNet” is the most common. However, since WordNet is based on the taxonomic hierarchy
constructed by hand, it does not describe co-occurrence ofword concepts in the photo context.
In this paper, we propose a novel method to construct anontology which describes hierarchical structure on relations
between tags from Folksonomy. The tag structure we extract
can be used as an ontology for image database searchwhich reflects both textual and visual relations between tags.
In the proposed method, at first, we collect millions of
tag-attached-images from Flickr which is the world-largest
Folksonomy-style image database, and remove noise images
from them. Next, we estimate concept vectors for highlyfrequent tags based on only visual features, only tag word
features and combined features of both visual and textual
features, and compute JS divergence and entropy for threekinds of concept vectors. Finally we estimate hierarchicalstructures between tags regarding three kinds of concept
vectors.
In the experiments, we show the obtained hierarchicalstructure, and it includes interesting relations which sometimes are difficult to be discovered by human. In addition, as
its application, we used and evaluated the obtained ontology
for query expansion of text-tag-based image search over
Flickr. These results indicate that the proposed method is
promising and the structure is expected to help image searchand some other applications.
II. R ELATED W ORK
It is very challenging work to build a semantichierarchical-structured database automatically from the Web.
In this section, we introduce some related works regarding
both text-based and visual-based hierarchical databases on
relations between word concepts.
A. Text-based Hierarchical Structure
WordNet is the most common hierarchical word dictionary which can be used as an ontology [1]. WordNet was
built manually by the experts, and contains several kinds ofsemantic relations such as part-of and instance-of between
words. There are many studies which employs WordNet
as an ontology. However, WordNet ontology is not always
effective for image databases. In addition, WordNet does
not contain newly-appeared words and proper nouns suchas iPhone.
While WordNet is built manually, several automatic methods to build hierarchical structure of word concepts havebeen proposed so far. Tang et al. [2] built hierarchicalstructure of word concepts using Folksonomy-based datafor movie rating. Tang et al. defined four kinds of distancemeasures, and used them to build hierarchical structure.
Their obtained hierarchical structure represents connectionsof contents of data rather than the meaning of word concepts.
Plangprasopchok et al. [3] proposed a method to extract
hierarchical relations from Flickr for treating images. However, they did not use visual features of images and use
only “collect/set” information provided by Flickr. Although
their result of the hierarchical structure is quite interesting,their method is not automatic and requires some manual
supervision.
B. Image-based Hierarchical Structure
Bart et al. [4] and Sivic et al. [5] proposed unsupervised
methods to build tree structures of word concepts based on
the similarity of visual features. However, their experiments
were carried out with small hand-made image databases, andthey did not use large-scale Folksonomy.
Wu et al. [6] proposed distance measure of word concepts called “Flickr Distance” employing probabilistic latent
topics and JS divergence. In our method, we also use JS
divergence as distance measure and construct a hierarchicaldatabase, while they consider only distances between concepts,
Deng et al. [7] built image datasets over the WordNet
hierarchical structure using Amazon Mechanical Turk [8].
They try to build a high-quality image database quality
using crowd sourcing, while our objective is automaticconstruction of a visual ontology from Folksonomy.
III. P ROPOSED M ETHOD
In this section, we describe a novel method to construct a
visual ontology which describes both visual and semantical
relations between word concepts. The ontology obtained inthis paper is represented in a directed acyclic graph (DAG)
where nodes and edges correspond to word concepts andrelations, respectively. The relation obtained in the ontology
includes several kinds of relations such as co-occurrence,
part-of and instance-of. In this paper, we do not classify akind of relations represented by edges, which will be one of
our future works.
We estimate three kinds of hierarchical structures represented by DAGs with visual features, tag features, andcombined features. The visual DAG is reflected in visual
similarity between concepts, while the tag-based DAG isreflected in semantic similarity which is originally estimatedfrom tag co-occurrency. The combined DAG has their intermediate properties taking into account both visual ans
semantic similarities.
The proposed method consists of four following procedures:(1) Generate visual, tag and combined feature vectors, andcompute PLSA topic vectors.
(2) Remove noise images for each concepts.
(3) Compute concept vectors, JS divergence between tag
concepts and entropy of each tag concepts.
(4) Build a hierarchical structure represented by a Directed
Acyclic Graph (DAG).

A. Representing Images in Three Ways
In the experiment, we collected about 2,000,000 tagged
images from Flickr. From all of them, we extracted thefollowing three kinds of features.
The first representation of an image is bag-of-features(BoF) [9] using SIFT descriptor [10]. We set the size ofa codebook of BoF representation as 1,000. Each image
is expressed in a 1,000 dimensional vector of appearance
frequency histogram of representative local features. We callthis a visual feature.
The second representation of an image is bag-of-words(BoW) of tags attached to each image. In Flickr, morethan two different tags are usually given to each image.
Since BoW represents occurrence of tags for each image,its elements are always one or zero. We use 4,345 kinds oftags to build BoW vectors. We call this a tag feature.
The third representation is a combination of the first
and the second representation using Probabilistic Latent
Semantic Analysis (pLSA)[11]. This representation method
is based on Lienhart et al.’s method [12] called Multimodal
pLSA. Original pLSA is a generation model contain latent
topic variables proposed by Hofmann [11], which is represented by the following equation:
𝑃 (𝑑𝑖 , 𝑤𝑗 ) =𝐾∑

𝑃 (𝑑𝑖 ∣𝑧𝑘 )𝑃 (𝑤𝑗 ∣𝑧𝑘 )𝑃 (𝑧𝑘 )

(1)

𝑘=1

where 𝑑𝑖 , 𝑤𝑗 and 𝑧𝑘 represent an image, a (visual) word
and a latent topic, respectively. We use pLSA as a methodfor dimension reduction of feature vectors. 𝑃 (𝑧𝑘 ∣𝑑𝑖 ) for𝑘 = 1..𝐾 can be regarded as a compressed feature vectorof image 𝑑𝑖 .
Each image 𝑑𝑖 is represented by the combination of
𝑃 (𝑧𝑘𝑣𝑖𝑠 ∣𝑑𝑖 ) and 𝑃 (𝑧𝑘𝑡𝑎𝑔 ∣𝑑𝑖 ), where 𝑧𝑘𝑣𝑖𝑠 and 𝑧𝑘𝑡𝑎𝑔 are a latent
topic about visual vectors and tag vectors, respectively.
Because we set the number of topics for both visual vectors
and tag vectors as 100 in this experiment, each image isexpressed with a 200 dimensional vector. We call this a
combined feature.
To obtain concept vectors for three kinds of representationin the third step, we calculate PLSA topic vectors of a
combined feature 𝑃 (𝑧𝑘𝑐𝑜𝑚𝑏 ∣𝑑𝑖 ) for each image in addition
to PLSA topic vectors of a visual feature and a tag feature.
B. Removal of Noise Images
We use PLSA topic vectors of visual features for removalof noise images. Noise image removal in this paper isbased on the idea that firstly we estimate the probabilityof a given concept over latent topics 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 ) bycomparing topic distributions of images belonging to theconcept and ones not belonging to the concept, and next
we evaluate the probability of the concept over each image
𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝐼𝑚𝑎𝑔𝑒) using estimated with the probability of
the given concept over topics 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 ).
To obtain the probability of whether image 𝑑𝑖 belongs
to a certain concept 𝐶𝑜𝑛𝑐𝑒𝑝𝑡 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑑𝑖 ), at first we
calculate 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 ) which represents a ratio of images
belonging to the given concept 𝐶𝑜𝑛𝑐𝑒𝑝𝑡 regarding the latenttopic 𝑧𝑘 . 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 ) is calculated as follows:
D. Building of Hierarchical Structure∑
As a method to build a hierarchical structure from Folk1
𝑧𝑘
𝑃𝐶𝑜𝑛𝑐𝑒𝑝𝑡=
𝑃 (𝑑𝐶 ∣𝑧𝑘𝑣𝑖𝑠 )sonomytags, we adopt Eda et al’s method [13] which∣𝐷𝐶𝑜𝑛𝑐𝑒𝑝𝑡 ∣𝑑𝐶 ∈𝐷𝐶𝑜𝑛𝑐𝑒𝑝𝑡
employs a Directed Acyclic Graph (DAG). Although it is∑originallya method for tag features, it is applicable for
1𝑣𝑖𝑠
𝑃 (𝑑𝑁 𝐶 ∣𝑧visual
𝑃𝑁𝑧𝑘𝑜𝑡𝐶𝑜𝑛𝑐𝑒𝑝𝑡 =𝑘 ) features and combined features. Since a DAG is a
∣𝐷𝑁 𝑜𝑡𝐶𝑜𝑛𝑐𝑒𝑝𝑡 ∣𝑑𝑁 𝐶 ∈𝑞𝐷𝑁 𝑜𝑡𝐶𝑜𝑛𝑐𝑒𝑝𝑡
graph structure where a node can have multiple parents in𝑧𝑘
contrast to trees that allow each node to have only one
𝑃𝐶𝑜𝑛𝑐𝑒𝑝𝑡
𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 ) =(2) DAG representation is appropriate as a representation
parent,𝑧𝑘
𝑧𝑘
𝑃𝐶𝑜𝑛𝑐𝑒𝑝𝑡 + 𝑃𝑁 𝑜𝑡𝐶𝑜𝑛𝑐𝑒𝑝𝑡
of Folksonomy-based visual ontology. Moreover, since a
DAG has no closed chains, we follow edges of DAG from
where 𝐷𝐶𝑜𝑛𝑐𝑒𝑝𝑡 and 𝐷𝑁 𝑜𝑡𝐶𝑜𝑛𝑐𝑒𝑝𝑡 represent a set of imagesa broader concept to a subordinate concept easily.
belonging to the given concept 𝐶𝑜𝑛𝑐𝑒𝑝𝑡 and a set of images
Given distance between concepts and entropy of eachnot belonging to the given concept, respectively.
concept, we can build hierarchical structure of concepts repGiven 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 ) and 𝑃 (𝑧𝑘 ∣𝑑𝑣𝑖𝑠𝑖 ), the probability
resented by a DAG according to the following procedures:of whether image 𝑑𝑖 belongs to the concept 𝐶𝑜𝑛𝑐𝑒𝑝𝑡
For each concept 𝐶,
𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑑𝑖 ) is calculated as follows:(1)
Select 𝑘 concepts at most to which the distance from𝐾∑
𝐶 is less than 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑.
𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑑𝑖 ) =𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑧𝑘 )𝑃 (𝑧𝑘 ∣𝑑𝑖 ) (3)(2)
Compareentropy values of 𝐶 and selected concepts,
𝑘=1
and classify the concepts with smaller entropy as child
We sort images which have 𝐶𝑜𝑛𝑐𝑒𝑝𝑡 as tags in the denodes and the concepts with larger entropy as parent
scending order of 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑑𝑖 ), and we use only top 100
nodes.
images for each concept for calculating a concept vector
In this paper, we set 𝑘 as 20, 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 as 0.05, 0.15 andremoving the rest of images as noise images.
0.10 for visual features, tag features and combined features,
respectively.
C. Representing Concepts and Extracting Relations between
Concepts
IV. E XPERIMENTS
We represent a concept vector of each concept associatedIn the experiments, we collected about 2,000,000 taggedwith a tag word as an average of PLSA topic vectors of theimages from Flickr, selected the top 5,000 frequent tags
images belonging to the concept in the following equations:as “concepts” for building a concept hierarchy, and built∑
a hierarchical structure by the proposed method. At first, we𝑝(𝑧∣𝑑)𝑑∈𝐶𝑜𝑛𝑐𝑒𝑝𝑡
𝑃 (𝑧∣𝐶𝑜𝑛𝑐𝑒𝑝𝑡) =(4)
evaluate removal of noise images, and then we show part of∣𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣
obtained visual ontology.
As 𝑃 (𝑧∣𝑑), we can use any PLSA topic vectors of visual
A. Results of Noise Removalfeatures, tag features or combined features.
Fig.1 shows comparison on accuracy rate of randomlyWe use JS divergence to estimate distance between conselected
100 images before noise removal and top 100
cept P, Q (Eq.(5), Eq.(6)). JS divergence (Jensen-Shannon
imagesregarding
𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝑑𝑖 ) for 10 concepts after noise
divergence) is a distance measure between two probability
removal.
distributions. JS divergence can be regarded as a symThe average accuracy rate of randomly-selected images
metricalized version of KL divergence (Kullback-Leibler
is70%, while the average accuracy rate of images selected
divergence). If two concept vectors are similar to each other,byproposal noise removal method is 92%. This shows thatthe value of JS divergence will be small.
noise image removal worked well. Fig.2 shows parts of the∑
𝑃 (𝑖)dataset after the noise image removal.
𝐷𝐾𝐿 (𝑃 ∣∣𝑄) =(5)
𝑃 (𝑖) log
𝑄(𝑖)B. Hierarchical Structures of Noun Concepts𝑖
𝐷𝐾𝐿 (𝑃 ∣∣(𝑃/2 + 𝑄/2))
We obtained hierarchical structures about 2,657 concepts
𝐷𝐽𝑆 (𝑃 ∣∣𝑄) =using 2,000,000 tagged images. Here, we explain part of2
obtained DAGs regarding some concepts. At first, we show
𝐷𝐾𝐿 (𝑄∣∣(𝑄/2 + 𝑃/2))(6)+
the DAG which starts from “sand” node. Note that the DAGs
2
shown here are a part of the DAGs extracted for explanation
In addition to JS divergence, we calculate entropy of
due to space limitation.
each concept vector which is needed by the next step to
Fig.3 shows the hierarchical structure under “sand” nodebuild hierarchical structure (Eq.(7)). Basically, high entropy
estimated by visual features. This structure includes “water”
concepts are distributed over broad topics, and low entropy
and “sea” which are expected to be related to “sand” asconcepts are concentrated on specific topics.
subordinate concepts of “sand”. However, it is not natural for∑human that “water” is regarded as the subordinate concept
𝐻(𝑃 ) = −𝑃 (𝑧∣𝐶𝑜𝑛𝑐𝑒𝑝𝑡) log(𝑃 (𝑧∣𝐶𝑜𝑛𝑐𝑒𝑝𝑡)) (7)of “lake”. Fig.6 shows the top 50 images of “sand”, “lake”
𝑧∈𝑍
and “water” regarding 𝑃 (𝐶𝑜𝑛𝑐𝑒𝑝𝑡∣𝐼𝑚𝑎𝑔𝑒). These threeFigure 1.

Figure 3.

The DAG under “sand” node by visual features

Figure 4.

The DAG under “sand” node by tag features

Results of noise removal.

Figure 2. Sample images after noise removal for “fire”, “luna”, “coast”,“sheep” and “piano”.

concepts are similar to each other visually. Since “water”
images are relatively less various among them, “water”
becomes a subordinate concept of “sand” and “lake” in termsof visual features.
Fig.4 shows the hierarchical structure under “sand” estimated by tag features. Tags frequently-used tend to becomesuperordinate concepts. Fig.4 shows the subordinate structure that starts from “sand” estimated by combined features,
which can be regarded as being taken into account bothvisual relations and semantic tag relations between concepts.
Although “beach” does not appear in the DAG of tag
features (Fig.4), it appears in the DAGs of both visualfeatures and combined features. On the other hand, “water”
which appears in the visual DAG does not appear in the
combined DAG, since relation between “water” and “sand”
is not so strong regarding the tag-based structure. In addition,although “lake” appears in the visual DAG (Fig.3), thesemantic relationship between “sand” and “lake” is not
so strong. In this sense, disappearance of “lake” in the
combined DAG can be regarded as a reasonable result.
Next, we explain about the DAG started from the “mountain” node shown in Fig.7. Fig.7 shows some representative
images of “mountain”. In the DAG of “mountain” built

Figure 5.

The DAG under “sand” node by combined features

Figure 6.

Top 50 images of “sand”, “lake” and “water”

Figure 12. A part of the DAG above “Pontevedra” node by visual features
Figure 7. The DAG under “mountain” node (by visual features (on theleft), by tag features (on the center) and by combined features (on theright))

by visual features, “mountain” have subordinate concepts“dawn”, “sunrise” and “sunset”.
Fig.9 shows representative images of “dawn”. These images depict that the sun rises from or go down to the horizonor the mountain, and they resemble “mountain” images verymuch visually. Since most of “dawn” images represents
horizons and are similar to each other, the entropy of “dawn”
becomes low. Therefore, the “dawn” node was placed as asubordinate node of the “mountain” node.
Meanwhile, in the hierarchical structure of “mountain”
built using tag features, “mountain” have subordinate concepts “valley”, “vista”, ”panoramic” and “morning”. These
concepts resemble the “mountain” semantically, but these do
not always resemble “mountain” visually (Fig.10).
Furthermore, in the hierarchical structure of “mountain”
built using combined features, subordinate concepts “berge”,“alps” and “alpen” are extracted which do not appear in
the above-mentioned hierarchical structures. These concepts
resemble the “mountain” semantically and visually (Fig.11).
We expect that this ontology enables us to search for images
tagged with only “alps” by the query keyword “mountain”.

Figure 13. A part of hierarchical structures above “Pontevedra” node by
tag features

Figure 14.

The DAG above “Pontevedra” node by Combined features

Figure 15.
Figure 8.

Sample images of “Pontevedra”

Representative images of “mountain”

Figure 16.
Figure 9.

Sample images of “Toledo”

Representative images of “dawn”

Figure 10.

Representative images of “vista”

Figure 11.

Representative images of “alps”

As the result related to proper nouns, we explain thehierarchical structures above the “Pontevedra” node. “Pontevedra” is a name of the city of the northwest of Spain. In
the result by tag features (Fig.13), broader concepts such as“Galiza” and “acoruña” appeared. Note that “Galiza” is thestate which “Pontevedra” belongs to. “acoruña” means “ACoruña”, the city of the northwest of Spain, which is larger
than “Pontevedra”.
“Toledo”, the city of the center of Spain, appears in the
result only by visual features (Fig.12). Although “Toledo”
may resemble “Pontevedra” visually (Fig.15, Fig.16), thesemantic relation is weak.
The result by combined features is close to
the one by tag features than by visual features(fig:resultPontevedraCombined). It is one of future
works to adjust the weight of the balance to combine visualfeatures and tag features automatically.
C. Hierarchical Structures of Adjective Concepts
In the last, as an example of DAG which starts from anadjective concept, we explain about hierarchical structures
under “cute” node. Fig.17 shows the DAG by visual features.

“Statue” has weak semantic relationship with “cute”, while“puppy” and “child” are apparently subordinate concepts of“cute”. In the result by combined features, “statue” disappears and the DAG was improved. However, the subordinate
node “puppy” also disappeared.

Figure 17.

The DAG under “cute” node by visual features

words three times for each words regarding both sub- andsuperordinate words, while we sent Flickr the original word
to get the baseline result. We compared the result by theoriginal word with the results by the subordinate words andthe ones by the subordinate words.
We evaluated how many irrelevant images we foundamong the top ten images returned from Flickr. Tab.I shows
the result of original and query expansion search. By using
subordinate concepts or superordinate words as additional
query words, search results were improved.
Table IT HE RESULTS OF QUERY EXPANSION . T HE AVERAGE NUMBER OF
IRRELEVANT IMAGES IN THE TOP 10 IMAGES RETURNED FROM F LICKR .
the average number ofirrelevant images
only original query word
2.4
with subordinate word
1.9
with superordinate word
1.9

V. C ONCLUSION
In this paper, we proposed a novel method to build a largescale visual ontology automatically. In the experiments, weconstructed three kinds of ontologies based on visual, tag
and combined features, and we confirmed that the ontology
built from combined features which took into account bothvisual similarity and semantical similarity was more naturalthat the two other ontologies.
As future work, we explore applications of the obtained
ontology. One of the possible applications is query expansion. We plan to study how to incorporate a visual ontology
into a text-based image search system effectively.
R EFERENCES
Figure 18.

Figure 19.

The DAG under “cute” node by tag features

The DAG under “cute” node by combined features

D. Query Expansion Using Ontology By Combined features
As a simple experiment to evaluate effectiveness of ourontology, we made an experiment on query expansion of
text-tag-based image search over Flickr, which is possible
one of applications of the obtained ontology.
We evaluated the number of relevant images among the
top 10 returned images from Flickr with search by original
keywords and expanded keyword sets. As expanded methods, we tried to add one of three subordinate words withthe original word and to add one of three superordinate
words with the original one. We sent Flickr the expanded
query word sets with “AND search” by changing additional[1] C. Fellbaum, Ed., WordNet: An Electronic Lexical Database, The
MIT Press, 2000.
[2] J. Tang, H. Leung, Q. Luo, D. Chen, and J. Gong, “Towards
ontology learning from folksonomies,” in Proc. of International Joint
Conferences on Artificial Intelligence, 2009.
[3] A. Plangprasopchok and K. Lerman, “Constructing folksonomies
from user-specified relations on Flickr,” in Proc. of the 18th
International Conference on World Wide Web, 2009, pp. 781–790.
[4] E. Bart, I. Porteous, P. Perona, and M. Welling, “Unsupervised
learning of visual taxonomies,” in Proc. of IEEE Computer Vision
and Pattern Recognition, 2008.
[5] J. Sivic, B.C. Russell, A. Zisserman, W.T. Freeman, and A.A. Efros,“Unsupervised discovery of visual object class hierarchies,” in Proc.
of IEEE Computer Vision and Pattern Recognition, 2008.
[6] L. Wu, X. S. Hua, N. Yu, W. Y. Ma, and S. Li, “Flickr distance,” inProc. of ACM International Conference Multimedia, 2008.
[7] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei,“ImageNet: a large-scale hierarchical image database,” in Proc. ofIEEE Computer Vision and Pattern Recognition, 2009, pp. 710–719.
[8] Amazon Mechanical Turk, http://aws.amazon.com/mturk/.
[9] G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual categorization
with bags of keypoints,” in Proc. of ECCV Workshop on Statistical
Learning in Computer Vision, 2004, pp. 59–74.
[10] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp.
91–110, 2004.
[11] T. Hofmann, “Unsupervised learning by probabilistic latent semantic
analysis,” Machine Learning, vol. 43, pp. 177–196, 2001.
[12] R. Lienhart, S. Romberg, and E. Horster, “Multilayer pLSA for multimodal image retrieval,” in Proc. of ACM International Conference
on Image and Video Retrieval, 2009.
[13] T. Eda, M. Yoshikawa, T. Uchiyama, and T. Uchiyama, “The
Effectiveness of Latent Semantic Analysis for Building Up a Bottomup Taxonomy from Folksonomy Tags,” World Wide Web, vol. 12, no.
4, pp. 421–440, 2009.