Evaluation Strategies
for Image Understandingand Retrieval
Keiji Yanai(Univ. Electro-Communications, Japan& former Visiting Scholar at Univ. Arizona)

Nikhil Shirahatti, Prasad Gabbur, andKobus Barnard(Univ. Arizona, USA)2

Our Three Evaluation Projects(1) Evaluation of low and mid-level
algorithms in the word-image-translation[CVPR 01]

Algorithms to extract features(2) Evaluation of image retrieval methods[CVPR 05]

CBIR systems(3) Evaluation of “visualness” of words[ACM MM05 sp]

Words to be annotated withIntroduction:
We need “evaluation” !
 Now we have a huge number of images.
 In MM, CV and IR , we are eagerly develo-

ping methods to infer semantics from them.
 We need “Evaluation Strategies”

to compare many methods
in the comprehensive points of view.

3Projects-1
Evaluation of
low and mid-level algorithms
in the word-image-translation[CVPR 01]

5

Word-image-translation
OutputInput

Word
to imagegrasscattiger[ICCV01]
rockgrass

We proposed two
types of annotation.
word to image
word to regions

tiger
grass

grasstiger[ECCV02]

Words
to regions

6Segmentation & Image features
InputSegmentation

sun sky waves sea

Blobworld Each regions is a large
Mean-shift vector of features
N-Cut• Region size
• Position• Color
9 RGB, L*a*b* or rgS

We have many combinationsof segmentation and features.

• Texture
9Oriented energy (12 filters)
9Response to DOG (4 filters)• Shape features

7

Sample Segmentations
Blobworld[UCB 02]

Mean-Shift[Rutger Univ. 02]

Normalized cuts
[UCB 00]

8Measuring Annotation Performance
predicted words

CAT HORSEGRASS WATER

Compare the ratio of overlap

actual keywordsGRASS TIGER
CAT FOREST

We compare segmentation methods and combinationsof features with this annotation performance.

9

Segmentation evaluation
Semantic based sementation comparison(held out data)

0.1

Preseg matches
N-cuts at 20

Word prediction performance

0.08

The performancevaried from 0.075
to 0.0950.06

 N-cuts

outperformed
Mean-shift &Blobworld.

0.04

Blobworld, held out
Mean-shift, held out
N-cuts, held out
Preseg, held out

0.02

02

4

6

8

Segments used1012

10Feature evaluation
Feature set

Performance

Base set

0.020

Base set, RGB0.057

Base set, L*a*b0.085

Base set, rgS

0.092

Base, rgS, color context

0.094

Base set, texture

0.048

Base, rgS, texture

0.072

Base, RGB, color context,texture

0.073

Base set, shape

0.016

Base set, rgS, shape

0.029

Base,rgS, texture, shape

0.043

Everything

0.055

Base set:SizeLocation
First moment
Area / (Perimeter)2

Varied from 0.016
to 0.094
Color is the most important.
La*b* and rgS are better than RGB.

12 types of combinations of feature sets

Projects-2
Evaluation ofimage retrieval methods[CVPR 05]

12Objective
 Develop a comprehensive method andprovide ground truth data to evaluate
image retrieval algorithms or systems.
 Human-centered evaluation
 Fully automatic evaluation
 Independent of image retrieval system Open calibration/evaluation software andground truth data available at our Web site
http://kobus.ca/research/projects/cbir-eval/

13

Preparation for evaluating your system
 You need: CBIR system to be evaluated COREL image data sets
 System output: many query-result pairs with score(assuming query by one image and no feedback) We provide (at our Web site) : Human-evaluation data

• 20000 pairs of<query COREL image id> <result image id><human score> Calibration and evaluation software• to measure the performance of your CBIR system

14Collecting human-eval. data Collected 20,000

query-result pairs
 32 participants Calibrated forparticipantsvariance.
Common groundtruth data

Web interface for collecting data
(4 pairs on one page)
5-step human evaluation

Mapping CBIR to human score& calculating their correlation15

 Estimate mapping function with

common ground truth data Monotonic constraint
 3 methods to map• Least Mean Square• Correlation Maximization• Bayesian Inference Calculate correlation of GT and estimated human score

Correlation between real and estimated human score
What we want to measure

System performance16

Case-study:
 Evaluate 4 CBIR systems
 GNU Image Finding Tool (GIFT)
 SIMPLIcity[J.Z.Wang 01]
 Our Translation Model [ICCV01]
 Corel keyword-based search (text search)e.g. bear, river, animal

queryfox, river, animal

2/3=0.667

resultsystemscoreResults: correlation between
human and system scores
0.6

0.51

0.5
0.4
0.3

0.17

0.19

0.24

0.2
0.1
0

GIFT

SIMPLIcity

Translation

Keywords

Keywords can represent semantics much better than image features.

17Projects-3
Evaluation of“visualness” of words[ACM MM 05 short paper]

19Motivation
 A lots of words for annotation of images

tiger feline cat
mammal animal wildlife
grass forest
8 words [Corel image gallery 1,000,000]

Corel ID 108041
e.g. “Mammal” is classified based on the way of their birth,
not based on their appearance.

Some words are not appropriate for image recognition.
Words related to “visual properties” are good for that.

We need to evaluate “visualness” of words.

20

Image region entropy A measure of “visualness” of words (concepts)
 Represent the property of the distribution of imagefeatures
Biased / uneven:
low entropy
having “visualness”

Random/uniform:high entropy
not having “visualness”

 Need no ground truth data unlike rec.-prec. diagram
 To get images associated to the given word,

use images on the Web with Google

Enables us examine about any words automatically

To examine “image region entropy”,
we have to provide only a concept keyword at first.

21Method: prepare generic model To make “entropy” meaningful, select“X” regions, excluding backgroundswith a probabilistic method. (same as prob.
Web gathering) Calculate the entropy of the “X” regionswith respect to a generic model Build a generic distribution model of region featuresof randomly collected images in advance

random Web images

22Case study:

Finding “visual” adjectives
 Collect 250 images per

word for 150 adjectivesusing Google ImageSearch Our model detects

regions related to theconcept of the given
word without any priorknowledge
images with “yellow” regions

23

Experimental results
 Low entropy (“visual” adjectives)
 darkvisual rusted purple
black shiny scary…

 High entropy (“non-visual” adjectives)
 medical

famous angry large
open acoustic religious…

24

Low entropy: “scary”“Visual” adjective

Detected“scary”
regions

25

High entropy: “famous”“Non-visual” adjective

26Conclusion
 We introduced our three projects

related to evaluation briefly:
 Segmentation algorithms andcombinations of image features
 Image retrieval systems
 Words to be annotated with27

Thank you!
If you are interested in our projects,
please visit

http://kobus.ca/