Visual Analysis on Relationships
between Nouns and Adjectives
Using a Large Number of Web Images
Yuya Kohara and Keiji Yanai
Department of Informatics,
The University of Electro-Communications, Tokyoⓒ 2012 UEC Tokyo.

Research area : Image recognition• We are working on– Automatic recognition of images

Dog

Car
ⓒ 2012 UEC Tokyo.

Sky

Photo database on the Web• Photo sharing Web sites (photo version of Youtube)– e.g.
– Everybody canupload photos
with tags.

ⓒ 2012 UEC Tokyo.

Background• Therearemanytaggedimages
ontheQuery
Query word:word:Web.
blueblue && carcarbut“redcar”“bluecar” add the tags to increase• Contributor“blue sky”

page view.
Relevant
Irrelevant– Most of the tags are independently.
◎×• The irrelevant results appear on theresults of AND-search with the tags.
ⓒ 2012 UEC Tokyo.

Objective• Analysis on visual relationships between
nouns and adjectives“red + flower”“red + dog”

visualrelationships＞• Finding out the tag pair with high visual
relationshipsⓒ 2012 UEC Tokyo.

Objective• The class with high relationships
contains similar images.

flower + red• Creating new dataset with less noise• Improving accuracy of simultaneousrecognition of nouns and adjectives– There is a flower and the color of theflower is red.
ⓒ 2012 UEC Tokyo.

Basic idea
• Prepare many tag pairs of nouns and adjectives– e.g. "red + car", "blue + sky", ...

• Search web image database for the images
corresponding to each of the prepared tag pairs• Detect regions of objects for all the images– Eliminating of background in the images• Evaluate the distribution of the image set of eachtag pair with entropy, and calculate mutualinformationⓒ 2012 UEC Tokyo.

Overview
Image acquisition by
the tag search

Calculation offeature distribution

Imagesegmentation

Calculation ofvisual relation

FeatureExtraction
Object region
determination

Analysis of the relationshipbetween the tags

ⓒ 2012 UEC Tokyo.

Example of gathered images
red + carbeautiful + car

red + dog

beautiful + dogⓒ 2012 UEC Tokyo.

Image acquisition• Image acquisition from Flickr– 20 nouns: car , dog , sky , …
– 16 adjectives : - , red , morning , old , …
• 20 nouns × 16 adjectives = 320 tag pairs– 200 positive images for each tag pair
– 600 negative images (common to all tagpairs)• 64,600 images (=200×320+60)ⓒ 2012 UEC Tokyo.

Results of image segmentation
red + carbeautiful + car

red + dog

beautiful + dogⓒ 2012 UEC Tokyo.

Feature extraction andcreating Bag of Feature• Extract SIFT features from each region• Create Bag of Feature vectors from a setof the SIFT features
Detection of KeyPointsⓒ 2012 UEC Tokyo.

Frequency of representative pattern

Assume good region
corresponding keyword• Multiple instance-SVM– training→ evaluation → changing dataset→training→ …… (5-loops)
High evaluated regions are positive data,
and low evaluated regions are negative
data in the next loop.

car & red regions

Not (car & red)regionsⓒ 2012 UEC Tokyo.

Results of detected regions
red + carred+carbeautiful + car
beautiful+car

red+carbeautiful+car

red + dog

beautiful + dog
red+dog

red+dogⓒ 2012 UEC Tokyo.

beautiful+dog

beautiful+dog

Visual relation by mutual information
Entropy： Value of distribution of local features
𝐻 𝑋 =−High : Wide distribution

𝑃 𝑥 log(𝑃 𝑥 )Low : Narrow distribution

Mutual information： Difference of entropy
𝑀𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻(𝑋|𝑌)High : High relation

Low : Low relationⓒ 2012 UEC Tokyo.

Experiments
Dataset:64,600 images for 320 tagged pairs

1. Evaluate mutual information of tag pairs
2. Compare visual relation andtag co-occurrencyⓒ 2012 UEC Tokyo.

Examples of results
1. Evaluate mutual information of tag pairs
High visual relationship between pair tags
• "red+sun" , "red+car"– Color adjective and object noun• "morning+sun", “night+sun"– Time adjective and noun related skyⓒ 2012 UEC Tokyo.

1. Evaluate mutual information of
tag pairs• Color adjective and object noun
– “blue + car”

carⓒ 2012 UEC Tokyo.

1. Evaluate mutual information oftag pairs• Time adjective and noun related sky– “sun+morning” , “sun+night”
sunⓒ 2012 UEC Tokyo.

Similarity by tag co-occurrence
2. Compare visual relation and tag co-occurrency
The Normalized Google Distance (NGD)– Estimates similarity from number of search results
max log |𝑋| , log |𝑌| − log |𝑋 ∩ 𝑌|
𝑁𝐺𝐷 =log 𝑁 − min{log |𝑋| , log |𝑌|}

|Y|

|Y|

|X|Big: Less co-occurrence|X|

Small: Much co-occurrence|X|, |Y| : The number of search results obtained from Flickrⓒ 2012 UEC Tokyo.

2. Compare visual relation andtag co-occurrency• Low visual relationshiphigh similarity by tag co-occurrence– “beach+summer “, “sky+green”ⓒ 2012 UEC Tokyo.

VisualTag relation
relation

Summary• Analyze visual similarity between the tag pairs• Calculate mutual information from the imagesof 360 tag pairs– High visual relationshipThe pair of color adjectives and object nouns
The pair of time adjectives and related sky nouns• Compare visual relation and tag co-occurrency– Exist the tag pair of Low visual relationship but Highco-occurrenceⓒ 2012 UEC Tokyo.

Future works• Large scale experiments– e.g. 100 nouns×100 adjectives• Creating dataset considered visualrelationship between the noun and theadjective• Simultaneous recognition of the nouns andadjectivesⓒ 2012 UEC Tokyo.

Future works
dogbagredcuteⓒ 2012 UEC Tokyo.