Automatic Construction ofA Folksonomy-based
Visual Ontology
Hidetoshi Kawakubo, Yuuta Akima and Keiji Yanai
Department of Computer Science,
The University of Electro-Communications, Tokyo, Japan

December 13, 2010

OutlineObjective

Related Work

Proposal methodExperimental ResultsConclusions & Future WorkWhat is Folksonomy？
Folk + taxsonomy = Folksonomy
 Folksonomy is Contents taxonomy by Users ofsocial media
 Users add tags to online contents for all users
 Online photo sharing site “Flickr” is an famous socialmedia.

Users add tags toonlinecontentsBackground

A large quantity of Folksonomy images areavailable on the Web.
More efficient image retrieval is necessary.
Ontologies for image retrieval has beenstudied.

ObjectiveAutomatic construction of hierarchicalimage datasets (Visual Ontology)
using Flickr images
FoodLunch

RiceAbout “Ontology”
Ontology is a database to let computers
understand relationships between things in
the real world.

Generally,

Food ⊃ Rice
We use the term “Ontology” as structures
to show relationships of concepts.
 Our “Visual Ontology” contains images of2657 concepts.
Related Works

WordNet It is an ontology database built by handand contains part-of and instance-of relations.
ImageNet [J. Deng et al. CVPR 2009 ] It is an image database according to WordNetSubordinate structure of “Beach”
WordNetOur ontology
beach

beachsky

It includes
technical knowledgeocean

It contains more visual concepts
and predictably-effective
for image retrieval.

Flow of Proposal Method
Generate visual, tag and combined feature vectors,
and compute pLSA topic vectors.

Remove noise images for each concept.
Compute concept vectors, distance between concepts
and entropy of each concept.

Build hierarchical structures.

Image collecting from Flickr

We collected 2 million tagged images randomly
from Flickr using Flickr APItags•bird•goose•swanny•Bali Bird Park

Features of images
Visual Feature SIFT + Bag-of-Features, 1000 dimension



Tag Feature Bag-of-Words (Bag-of-Tags), 4345 dimension

Combined Feature (Visual + Tag) as we will describe laterprobabilistic Latent Semantic Analysis


We use pLSA to represent distribution of images
 Each image are represented as a mixture oflatent topics.
 pLSA is a kind of soft clustering.

Image 𝑑TopicVector
𝒑 𝐳𝐝

0.2
0.1
0.2

0.0
0.5

LatentTopics𝑧Feature Combine Method
Visual Feature

Tag Feature

pLSA

pLSA

Pvisual z | d 
VisualTopic Vector

Ptag z | d 
Tag Topic Vector

CombineCombinedFeature

Pcombined z | d 
pLSA

Combined Topic Vector

Topic vectorization of a concept“moon”“moon”
imagesconcept

Average・・・Distance between concepts
We use JS divergence between topic vectors
as measurement of dis-similarity.

P, Q are topic vectors of concepts

Hierarchical relations of concepts

We use entropy of concepts’ topic vector
to estimate broadness of concepts

AnimalBird( Broader Concept )( Subordinate Concept )Larger Entropy

Smaller Entropy

Construction of hierarchical structures

We use DAG (Directed Acyclic Graph)

For each concepts,
Step1. select 20 neighborhood concepts.
Step2. compare entropy value and classify
the concepts with smaller entropy as child nodes
the concepts with larger entropy as parent nodes.
vista
vista：：

alps

dawn
nationalparkvalley

mountain
mountain・・・ parent nodes

valleyalps
nationaldawn
park ・・・ child nodes

Experiments

We built hierarchical concept structures for
2,657 concepts.
We show several results and consider
differences about feature kinds.

Subordinate structure of “mountain”
By visual feature

mountain

mountain
lake

dawn
sky

By tag feature
mountainvalleyskylinedawn

Subordinate structures of “mountain”
By combined feature
mountain

alps

berge
mountain

alps

alpen

Superordinate structure of “bridge”
By visual feature
australiausabridge

usabridge

Superordinate structure of “bridge”
By tag feature

citywideangle

polarizer

landmark

tripodbridgeSuperordinate structure of “bridge”
By combined featurepolarizer

wideangle

rivercityscapebridgehighway

Subordinate structure of “cute”
cute

cat

statuepuppy

statueBy visualfeature

Conclusions & Future WorkConclusions
 We proposed an automatic construction method ofvisual ontology using online tagged images.
 Ontology by combined features is better thanontology by only visual features and by only tagfeatures.
Future Work We plan to Improve the proposed method.

 Quantitative evaluation of method is yet.

Removal of Noise Images“moon”pLSA
Not “moon”
Image set

Removal of Noise Images

・・・Concept