Image Region Entropy: A Measure of “Visualness”
of Web Images Associated with One Concept

5. Experiments
Case study: Finding “visual” adjectives• Collect 250 images per word for 150 adjectives
selected from frequently words of keywords of Hemera image database

Keiji Yanai and Kobus Barnard

C-07

Univ. of Electro-Communications, JAPAN / Univ. of Arizona, USA

4. Method

1. Objective
Select “words” for image-annotation automatically
We have a lot of words, but not all words are appropriate
for image-annotation. We should use “visual” words.
“Visual” : being associated with visual properties of images

Measure “visualness” of word concepts
We propose a new measure: “image region entropy”.
Based on it, we can select words for image-annotation.

2. Motivation
tiger feline cat
mammal animal wildlife
grass forest
Corel ID 108041

Corel keywords [Corel image gallery 1,000,000]• “Mammal” is categorized based on the way of their birth,
not based on their appearance.
• “Animal” is too broad. There are many kinds of animals.

Entropy of “X”

Repeat several times
to refine “X” model

Word “X”

PrepareCalculate
ProbabilisticProbabilistic
Gathergeneric
the entropy
selection(2)selection(1)“X” images
model &of “X” regionswith Google
backgroundregardingSelect highLearn P(X|r)
imagesgeneric modelP(X|r) regions
(A)
(B)
(C)
(D)
(E)
Processing flow to obtain “image region entropy”

A. Prepare the generic model and negative images• To get generic distribution of region features,– Collect a lot of images from the Web randomly.
– Segment all the collected images and extract region features– Model generic distribution of region featureswith GMM using EM
generic base model• Use these images also as negative sample images (backgroundimages) for selecting “X” regions– Color adj.: purple, black, red, blue, white, pink,..
– Emotional adj.: sad, happy, scary, romantic,…
– Physical adj.: heavy, light, solid, sharp, old, electric, oval,…
– Many other kinds of adj.: floral, international, safe, digital,….
• Segmentation by JSEG
• Region features: color (avg. RGB), texture (Gaussian filter), shape• Our model detects regions
related to the conceptsof the given wordwith no prior knowledge• Results for 150 adjectives“yellow” regions after one iteration“yellow” regions after five iterations
Random Web images for building the generic model

B. Gather Web images associated with a given
keyword “X” with Google Image Search• “Mammal” and “animal” should be inferred from “tiger”
rather than be recognized from images directly.

Some words are not appropriate for image recognition.
Words related to “visual properties” are suitable for that.

We need to find “visual” words for image-annotation.

Results of color adj.

• Segment all the images and extract region features.

C. Use EM to learn which regions should be
associated with the concept “X”. (Learn P(X|r))
D. Select regions with high P(X|r) as “X” regions– Base on the modified image-annotation-model [ICCV 01]

3. Image region entropyN• Measure “visualness” of word concepts• Represent the property of the distribution of image features– Biased / uneven: having “visualness” low entropy– Random/uniform:not having “visualness” high entropy
biasedlow entropy

E. Calculate the entropy of “X” regions regarding
the generic model. E ( X ) = ∑ − P( X | c ) log P( X | c )randomhigh entropy

visualnot visual• Need no ground truth data unlike r-p diagram• Use images on the Web with Google– This enables us examine about any words• To make “entropy” meaningful, pick up only “X” regions,excluding backgrounds with a probabilistic method.

j =1

j2

j

Entropy : how much the distribution of region features is biased
compared to the generic distribution of region features• Color adjectives which are expected to be “visual “ are ranked
in the relatively upper ranking.
• “Scary” images have black
no-textured regions“visual” word
• “Famous” images have noprominent tendency.
“non-visual” word

Repeat several times
Probabilisticrclustering
ri+ i ++r− i r with GMM
iri −
ri +
ri+ ri+ ri−
ri−r − ri EM ali‘X’ & ‘non-X’ gorithm
features

Select ‘X’and ‘non-X’
components

r1 r8 r
r5 2a<(1-t)

p = P(c j | X )pXj = P(c j | nonX)
aj =pjX

pXj + pXj

Selected ‘X’ regions

P( X | ri ) > P(nonX | ri )‘X’ modela>tjX

P( X | ri )

a>t

r3

a<(1-t)P(nonX | ri )t=0.85(C) Learn P(X|r) with EMHigh entropy: “famous”

6. Conclusion

Detail of region selection+Low entropy: “scary”‘non-X’ model

r4
r6 r7

Selected ‘non-X’ regions
P( X | ri ) < P(nonX | ri )(D) Calculate entropy• Proposed “image region entropy” to measure“visualness” of words, and applied it to150 adjectives.
• For future work, we will improve and apply it to nouns.
• Incorporate “visual” adjectives with ourimage-annotation models in addition to nouns.
• Select “visual” words from all nouns and build“generic image corpus” by the Web image-gathering.