Appears

in

Pattern

Recognition

-LRB-

-RRB-

April

special

issue

on

image

databases

Deformable

Prototypes

for

Encoding

Shape

Categories

in

Image

Databases

Stan

Sclaroff

Computer

Science

Department

Boston

University

Cummington

St.

Boston

MA

Abstract

We

describe

a

method

for

shape-based

image

database

search

that

uses

deformable

prototypes

to

represent

categories

Rather

than

directly

comparing

a

candidate

shape

with

all

shape

entries

in

the

database

shapes

are

compared

in

terms

of

the

types

of

nonrigid

deformations

-LRB-

differences

-RRB-

that

relate

them

to

a

small

subset

of

representative

prototypes

To

solve

the

shape

correspondence

and

alignment

problem

we

employ

the

technique

of

modal

matching

an

information-preserving

shape

decomposition

for

matching

describing

and

comparing

shapes

despite

sensor

variations

and

nonrigid

deformations

In

modal

matching

shape

is

decomposed

into

an

ordered

basis

of

orthogonal

principal

components

We

demonstrate

the

utility

of

this

approach

for

shape

comparison

in

image

databases

Introduction

Shape

categories

can

be

represented

as

deformations

from

a

subset

of

standard

or

prototypical

shapes

it

is

thought

that

this

is

one

plausible

mechanism

for

human

perception

-LSB-

-RSB-

This

basic

premise

is

appealing

for

its

descriptive

parsimony

and

has

served

as

inspiration

for

many

of

the

prototypebased

representations

for

machine

vision

robotics

and

simulation

In

the

work

described

in

this

paper

our

aim

is

to

represent

shape

categories

for

interactive

image

database

search

Rather

than

directly

comparing

a

candidate

shape

with

all

shapes

in

the

database

we

propose

a

method

that

rst

indexes

shapes

in

terms

of

their

relationship

to

a

few

shape

prototypes

To

do

this

we

will

employ

modal

matching

a

deformable

shape

decomposition

that

allows

users

to

specify

a

few

example

shapes

and

has

the

computer

ef?ciently

sort

the

set

of

objects

based

on

the

similarity

of

their

shape

If

desired

shapes

can

be

more

closely

compared

in

terms

of

the

types

of

nonrigid

deformations

-LRB-

differences

-RRB-

that

relate

them

to

a

few

prototype

shapes

Our

approach

is

related

to

morphing

a

computer

graphics

technique

that

has

become

quite

popular

in

advertisements

Morphing

is

accomplished

by

an

artist

identifying

a

large

number

of

corresponding

control

points

in

two

images

and

then

incremen

tally

deforming

the

geometry

of

the

rst

image

so

that

its

control

points

eventually

lie

atop

the

control

points

of

the

second

image

Using

this

technique

in-between

or

novel

views

can

be

generated

as

warps

between

example

views

This

suggests

an

important

way

to

obtain

a

low-dimensional

parametric

description

of

shape

interpolate

between

known

prototype

views

For

instance

given

views

of

the

extremes

of

a

motion

-LRB-

e.g.

systole

and

diastole

or

left-leg

forward

and

right-leg

forward

-RRB-

we

can

describe

the

intermediate

views

as

a

smooth

combination

of

the

extremal

views

All

that

is

required

to

determine

this

view-based

parameterization

of

a

new

shape

are

the

prototype

views

point

correspondences

between

the

new

shape

and

the

prototype

views

and

a

method

of

measuring

the

amount

of

-LRB-

nonrigid

-RRB-

deformation

that

has

occurred

between

the

new

shape

and

each

prototype

view

The

prototypes

de?ne

a

polytope

in

the

space

of

the

-LRB-

unknown

-RRB-

underlying

physical

system

's

parameters

By

measuring

the

amount

of

deformation

between

the

new

shape

and

extremal

views

we

locate

the

new

shape

in

the

coordinate

system

de?ned

by

the

polytope

This

coordinate

in

prototype

space

can

be

used

for

database

indexing

and

fast

search

This

general

approach

is

related

in

spirit

to

the

linearcombinations-of-views

paradigm

where

any

object

view

can

be

synthesized

as

a

combination

of

linearly-warped

example

views

of

Ullman

and

Basri

-LSB-

-RSB-

and

Poggio

et

al.

-LSB-

-RSB-

However

it

differs

from

their

proposals

in

two

important

ways

First

we

are

interested

not

only

in

recognizing

shapes

but

also

in

describing

the

types

of

deformations

that

relate

them

We

want

to

derive

a

low-dimensional

parametric

representation

of

the

shape

that

can

be

used

to

recognize

and

compare

shapes

in

the

manner

of

Darrell

and

Pentland

-LSB-

-RSB-

Second

we

can

not

be

restricted

to

a

linear

framework

Nonrigid

motions

are

inherently

nonlinear

although

they

are

often

physically

smooth

Therefore

to

employ

a

combination-of-views

approach

we

must

be

able

to

determine

point

correspondences

and

measure

similarities

between

views

in

a

way

that

takes

into

account

at

least

qualitative

physics

of

nonrigid

shape

deformation

In

computer

graphics

it

is

the

job

of

the

artist

to

enforce

the

constraint

of

physical

smoothness

in

machine

vision

we

need

to

be

able

to

do

the

same

automatically

To

achieve

this

we

will

employ

modal

matching

a

method

for

-LRB-

-RRB-

determining

point

correspondences

using

a

energy-based

model

-LRB-

-RRB-

warping

or

morphing

one

shape

into

another

using

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

-LRB-

f

-RRB-

Figure

The

data

needed

to

build

two

deformable

prototype

shape

models

Support

maps

are

shown

in

-LRB-

b

e

-RRB-

and

edge

maps

in

-LRB-

c

f

-RRB-

The

two

prototype

shape

models

depict

-LRB-

a

-RRB-

a

European

Hare

and

-LRB-

d

-RRB-

a

Desert

Cottontail

Their

associated

The

original

color

images

were

digitized

from

-LSB-

-RSB-

Prototype

Figure

When

a

new

shape

is

encountered

it

is

parameterized

in

terms

of

the

energy

needed

to

nonrigidly

deform

the

prototype

shape

models

into

alignment

with

the

new

shape

The

distance

to

prototypes

is

expressed

as

the

square

root

of

strain

The

resulting

tuple

-LRB-

-RRB-

is

used

to

represent

the

shape

in

a

space

de?ned

in

terms

of

distance

to

prototypes

energy-based

interpolants

and

-LRB-

-RRB-

measuring

the

amount

of

deformation

between

an

object

's

shape

and

prototype

views

-LSB-

-RSB-

Figure

shows

the

information

required

to

build

modal

shape

prototypes

for

two

rabbit

shape

prototypes

employed

in

our

image

database

experiments

In

our

system

a

shape

is

de?ned

by

a

cloud

of

feature

locations

-LRB-

i.e.

edges

corners

high-curvature

points

-RRB-

and

a

region

of

support

that

tells

us

where

the

shape

is

Given

this

input

deformable

prototype

models

are

built

directly

from

feature

data

using

a

nite

element

formulation

that

is

based

on

Gaussian

interpolants

-LSB-

-RSB-

For

ef?ciency

we

can

select

a

subset

of

the

feature

data

as

nodes

for

a

lower-resolution

nite

element

model

and

then

use

the

resulting

eigenmodes

in

nding

the

higher-resolution

feature

correspondences

as

described

in

-LSB-

-RSB-

This

subset

can

be

a

set

of

particularly

salient

features

-LRB-

i.e.

corners

T-junctions

and

edge

mid-points

-RRB-

or

a

randomly

selected

subset

of

-LRB-

roughly

-RRB-

uniformly-spaced

features

cottontailsqrt

-LRB-

modal

strain

-RRB-

from

protoype

rabbitseuropean

hare

Figure

Scatter

plot

of

square-root

modal

strain

energy

for

rabbit

prototypes

used

in

the

image

database

experiment

Each

axis

depicts

the

square-root

of

strain

energy

needed

to

align

a

shape

with

a

rabbit

prototype

Thus

each

rabbit

shape

has

a

coordinate

in

this

space

The

rabbits

are

clustered

in

terms

of

their

shape

appearance

long-legged

standing

rabbits

cluster

at

the

top-left

of

the

graph

while

short-legged

seated

rabbits

cluster

at

the

bottom

right

There

are

two

rabbits

that

map

between

clusters

showing

the

smooth

ordering

from

long-legged

to

medium-legged

to

short-legged

rabbits

in

this

view-space

When

a

new

shape

is

encountered

it

is

parameterized

in

terms

of

the

energy

needed

to

nonrigidly

deform

the

prototype

shape

models

into

alignment

with

the

new

shape

Similarity

is

thus

computed

in

terms

of

the

amount

of

strain

energy

needed

to

deform

each

prototype

to

match

it

to

the

candidate

shape

as

illustrated

in

Figure

The

amounts

of

deformation

are

measured

in

terms

of

strain

energy

and

stored

as

a

n-tuple

where

n

is

the

number

of

prototype

shapes

employed

In

this

case

the

resulting

tuple

is

-LRB-

-RRB-

The

result

is

a

low-dimensional

parametric

representation

that

can

be

used

for

ef?cient

shape-based

image

database

search

Rather

than

directly

comparing

a

candidate

shape

with

all

shape

entries

in

a

database

we

instead

compute

similarity

in

a

distance

to

prototypes

space

Using

this

method

we

compactly

represent

a

category

of

shapes

in

terms

of

a

few

prototype

views

Fig.

shows

a

scatter

plot

of

the

two-dimensional

rabbit

space

spanned

by

two

rabbit

shape

prototypes

The

graph

's

xaxis

depicts

the

square-root

of

strain

energy

needed

to

align

the

European

Hare

prototype

with

each

rabbit

shape

while

the

y-axis

shows

the

energy

needed

to

align

the

Desert

Cottontail

prototype

with

each

rabbit

shape

Each

or

the

rabbit

shapes

has

a

coordinate

in

this

strain-energy-from-prototype

subspace

As

can

be

seen

the

rabbits

are

clustered

in

terms

of

their

shape

appearance

long-legged

standing

rabbits

cluster

at

the

top-left

of

the

graph

while

short-legged

seated

rabbits

cluster

at

the

bottom

right

There

are

two

rabbits

that

map

between

clusters

showing

the

smooth

ordering

from

long-legged

to

medium-legged

to

short-legged

rabbits

in

this

view-space

We

will

demonstrate

the

utility

of

this

approach

for

comparing

shapes

in

image

databases

digitized

from

children

's

eld

guides

and

images

of

hand

tools

Deformable

shape

models

will

be

built

and

compared

using

support

and

silhouette

data

The

methods

described

in

this

paper

are

also

useful

for

recognizing

or

classifying

motions

-LSB-

-RSB-

fusing

data

from

different

sensors

and

for

comparing

data

acquired

at

different

times

or

under

different

conditions

-LSB-

-RSB-

Segmentation

The

work

reported

in

this

paper

addresses

issues

of

shape

categorization

even

when

shapes

within

categories

can

undergo

both

rigid

and

nonrigid

motion

Throughout

this

paper

it

was

assumed

that

gure/ground

segmentation

information

can

be

provided

as

input

to

the

modal

shape

comparison

modules

Since

segmentation

is

not

the

topic

of

this

paper

our

current

databases

contain

images

of

unoccluded

objects

on

uniform

backgrounds

Under

these

circumstances

a

c-Means

clustering

and

thresholding

technique

can

be

used

for

foreground/background

separation

-LSB-

-RSB-

However

for

very

general

query

by

shape

foreground/background

modules

will

be

needed

as

a

front-end

to

the

system

The

rst

solution

would

be

to

use

motion

and

color

to

pull

out

foreground

objects

Such

gure-ground

segmentation

can

be

done

reliably

by

use

of

clustering

in

conjunction

with

optical

ow

-LSB-

-RSB-

and/or

color

information

-LSB-

-RSB-

Background

and

Notation

In

the

last

few

years

researchers

have

made

some

progress

toward

automatic

shape

indexing

for

image

databases

The

general

approach

has

been

to

calculate

some

approximately

invariant

statistic

like

shape

moments

and

use

these

to

stratify

the

image

database

-LSB-

-RSB-

One

problem

with

this

general

approach

is

that

it

discards

signi?cant

perceptual

and

semantic

information

While

indexing

methods

provide

a

means

to

quickly

narrow

a

search

to

a

more

manageable

subset

they

often

do

not

provide

a

method

for

closer

direct

comparison

of

how

they

are

related

Rather

than

discarding

useful

similarity

information

by

employing

only

invariants

we

believe

that

one

should

use

a

decomposition

that

preserves

as

much

semantically

meaningful

and

perceptually

important

information

as

is

possible

while

still

providing

an

ef?cient

encoding

of

the

original

signal

-LSB-

-RSB-

Another

important

problem

with

these

approaches

is

that

most

are

only

robust

for

rigid

shapes

Although

many

things

move

rigidly

in

many

cases

this

rigid-body

model

is

inadequate

For

instance

most

biological

objects

are

exible

and

articulated

To

describe

these

deformations

therefore

it

is

reasonable

to

model

the

physics

by

which

real

objects

deform

This

rationale

led

to

the

physical

modeling

paradigm

of

active

contours

or

snakes

-LSB-

-RSB-

and

deformable

templates

-LSB-

-RSB-

A

snake

has

a

prede?ned

structure

which

incorporates

knowledge

about

the

shape

and

its

resistance

to

deformation

By

allowing

the

user

to

specify

forces

that

are

a

function

of

sensor

measurements

the

intrinsic

dynamic

behavior

of

a

physical

model

can

be

used

to

solve

tting

interpolation

or

correspondence

problems

While

snakes

enforced

constraints

on

smoothness

and

the

amount

of

deformation

they

could

not

in

their

original

form

be

used

to

constrain

the

types

of

deformation

valid

for

a

particular

problem

domain

or

object

class

This

led

to

the

development

of

algorithms

which

include

a

priori

constraints

on

the

types

of

allowable

deformations

for

motion

tracking

-LSB-

-RSB-

Cootes

et

al.

-LSB-

-RSB-

use

trainable

snakes

for

capturing

the

invariant

properties

of

a

class

of

shapes

by

nding

the

principle

variations

of

a

snake

via

the

Karhunen-Loeve

transform

Unfortunately

this

method

relies

on

the

consistent

sampling

and

labeling

of

point

features

across

the

entire

training

set

and

can

not

handle

large

rotations

If

different

feature

points

are

present

in

different

views

or

if

there

are

very

different

sampling

densities

then

the

resulting

models

will

differ

even

if

the

object

's

pose

and

shape

are

identical

Keeping

these

issues

in

mind

we

use

the

Finite

Element

Method

to

alleviate

problems

with

sampling

and

modal

analysis

to

provide

a

principled

way

to

select

the

types

of

nonrigid

deformations

needed

for

exibly

describing

shape

In

the

rest

of

this

section

we

provide

a

brief

review

of

our

representation

In

addition

we

review

our

new

method

of

building

FEM

models

without

imposing

an

a

priori

parameterization

and

how

to

use

the

modes

of

this

model

to

nd

point

correspondences

to

align

objects

and

to

compare

their

shape

This

initial

work

was

applied

in

the

area

of

nding

corresponding

features

in

static

imagery

-LSB-

-RSB-

and

serves

as

the

foundation

for

our

new

representation

for

shape

categories

Finite

Element

Method

The

major

advantage

of

the

nite

element

method

is

that

it

uses

the

Galerkin

method

of

surface

interpolation

This

provides

an

analytic

characterization

of

shape

and

elastic

properties

over

the

whole

surface

and

thereby

alleviates

problems

caused

by

irregular

sampling

of

feature

points

In

Galerkin

's

method

we

set

up

a

system

of

polynomial

shape

functions

that

relate

the

displacement

of

a

single

point

to

the

relative

displacements

of

all

the

other

nodes

of

an

object

u

-LRB-

x

-RRB-

H

-LRB-

x

-RRB-

U

-LRB-

-RRB-

where

H

is

the

interpolation

matrix

x

is

the

local

coordinate

of

a

point

in

the

element

where

we

want

to

know

the

displacement

and

U

denotes

a

vector

of

displacement

components

at

each

element

node

By

using

these

functions

we

can

calculate

the

deformations

which

spread

uniformly

over

the

body

as

a

function

of

its

constitutive

parameters

Solution

to

the

problem

of

deforming

an

elastic

body

to

match

the

set

of

feature

points

then

requires

solving

the

dynamic

equilibrium

equation

M?U

D

U

KU

R

-LRB-

-RRB-

where

R

is

the

load

vector

whose

entries

are

the

spring

forces

between

each

feature

point

and

the

body

surface

and

where

M

D

and

K

are

the

element

mass

damping

and

stiffness

matrices

respectively

-LSB-

-RSB-

Modal

Representation

The

FEM

governing

equations

can

be

decoupled

by

posing

the

equations

in

a

basis

de?ned

by

the

M-orthogonalized

eigenvectors

of

K

These

eigenvectors

and

values

are

the

solution

to

the

generalized

eigenvalue

problem

KOEi

MOEi

-LRB-

-RRB-

The

vector

OEi

is

called

the

ith

mode

shape

vector

and

i

is

the

corresponding

frequency

of

vibration

Each

mode

shape

vector

describes

how

each

node

is

displaced

by

the

ith

vibration

mode

The

mode

shape

vectors

are

M-orthonormal

this

means

that

TK

and

TM

I

The

OEi

form

columns

in

the

transform

and

are

elements

of

the

diagonal

matrix

We

will

assume

Rayleigh

damping

-LRB-

i.e.

D

-RRB-

thus

the

damping

matrix

will

also

be

diagonalized

by

this

transform

-LSB-

-RSB-

This

generalized

coordinate

transform

is

then

used

to

transform

between

nodal

point

displacements

U

and

decoupled

modal

displacements

U

U

U

We

can

now

rewrite

Eq

in

terms

of

these

generalized

or

modal

displacements

obtaining

a

decoupled

system

of

equations

U

D

U

U

TR

-LRB-

-RRB-

allowing

for

closed-form

solution

to

the

equilibrium

problem

-LSB-

-RSB-

Given

this

equilibrium

solution

in

the

two

images

point

correspondences

can

be

obtained

directly

By

discarding

high

frequency

modes

the

amount

of

computation

required

can

be

minimized

without

signi?cantly

altering

correspondence

accuracy

Moreover

such

a

set

of

modal

amplitudes

provides

a

robust

canonical

description

of

shape

in

terms

of

deformations

applied

to

the

original

elastic

body

This

allows

them

to

be

used

directly

for

object

recognition

-LSB-

-RSB-

Modal

Matching

Perhaps

the

major

limitation

of

previous

methods

is

the

requirement

that

every

object

be

described

as

the

deformations

of

a

single

prototype

object

For

instance

in

some

schemes

all

shapes

are

represented

as

deformations

from

an

elliptical

or

circular

prototype

-LSB-

-RSB-

Such

approaches

implicitly

impose

an

a

priori

parameterization

upon

the

sensor

data

and

therefore

implicitly

determine

the

correspondences

between

data

and

prototype

these

are

FEM

nodes

build

physical

model

compute

eigenmodesInput

featuresOutput

strongestfeature

correspondencesKU

MU

R

determine

FEM

mass

and

stiffness

matricesKU

MU

R

determine

FEM

mass

and

stiffness

matricesKf

w

Mfiiisolve

generalizedeigenproblemKf

wMfiiisolve

generalizedeigenproblemfind

correspondence

ingeneralized

feature

spaceimatch

low-order

nonrigid

modes

ffor

both

shapesthese

are

FEM

nodes

Input

featuresuse

the

matched

fias

coordinate

system

Figure

Modal

matching

system

diagram

-LRB-

reprinted

from

-LSB-

-RSB-

-RRB-

Furthermore

an

elliptical

prototype

may

be

inadequate

for

many

shapes

especially

shapes

that

are

not

star-connected

or

those

that

have

long

protrusions

or

deep

concavities

We

would

like

to

avoid

these

problems

as

much

as

possible

by

letting

the

data

determine

the

parameterization

in

a

natural

manner

To

accomplish

this

we

use

the

data

itself

to

de?ne

the

deformable

object

by

building

stiffness

and

mass

matrices

that

use

the

positions

of

image

feature

points

as

the

nite

element

nodes

The

resulting

new

modeling

formulation

is

called

modal

matching

and

is

described

in

detail

in

-LSB-

-RSB-

A

ow-chart

of

our

method

is

shown

in

Fig.

For

each

image

we

start

with

feature

point

locations

which

are

used

as

nodes

in

building

a

nite

element

model

of

the

shape

If

we

are

given

a

support

function

then

we

can

cut

the

nite

element

sheet

into

any

shape

We

do

this

by

de?ning

a

support

function

that

is

zero

anywhere

outside

the

shape

region

and

greater

than

zero

inside

the

shape

region

Thus

the

support

function

can

be

used

to

de?ne

both

the

shape

and

the

thickness

of

the

elastic

model

A

Gaussian

is

then

centered

at

each

node

Together

these

Gaussians

form

a

basis

for

building

the

Galerkin

interpolants

of

Eq

and

are

thus

used

in

constructing

FEM

mass

and

stiffness

matrices

When

there

are

possibly

hundreds

of

feature

points

for

each

shape

computing

the

FEM

model

and

eigenmodes

for

the

full

feature

set

can

become

non-interactive

For

ef?ciency

we

can

select

a

subset

of

the

feature

data

to

build

a

lower-resolution

nite

element

model

and

then

use

the

resulting

eigenmodes

in

nding

the

higher-resolution

feature

correspondences

as

described

in

-LSB-

-RSB-

This

subset

can

be

a

set

of

particularly

salient

features

-LRB-

i.e.

corners

T-junctions

and

edge

mid-points

-RRB-

or

a

randomly

selected

subset

of

-LRB-

roughly

-RRB-

uniformly-spaced

features

We

then

compute

the

modes

of

free

vibration

of

this

model

using

Eq

The

modes

of

an

object

form

an

orthogonal

objectcentered

coordinate

system

for

describing

feature

locations

That

is

each

feature

point

location

can

be

uniquely

described

in

terms

of

how

it

projects

onto

each

eigenvector

i.e.

how

it

participates

in

each

deformation

mode

The

transform

between

Cartesian

feature

locations

-LRB-

x

y

-RRB-

and

modal

feature

locations

-LRB-

u

v

-RRB-

is

accom

plished

by

using

the

eigenvectors

as

a

coordinate

basis

-LSB-

j

j

-RSB-

um

vm

-LRB-

-RRB-

where

m

is

the

number

of

nodes

used

to

build

the

nite

element

model

The

column

vector

OEi

is

the

ith

mode

shape

and

describes

the

modal

displacement

-LRB-

u

v

-RRB-

at

each

feature

point

due

to

the

ith

mode

while

the

row

vector

ui

and

vi

are

the

ith

generalized

feature

vectors

which

together

describe

the

feature

's

location

in

the

modal

coordinate

system

Normally

only

the

n

lowest-order

modes

are

used

in

forming

this

coordinate

system

so

that

-LRB-

-RRB-

we

can

compare

objects

with

differing

numbers

of

feature

points

and

-LRB-

-RRB-

ensure

that

the

feature

point

descriptions

are

insensitive

to

noise

Depending

upon

the

demands

of

the

application

we

can

also

selectively

ignore

rigidbody

modes

or

low-order

projective-like

modes

or

modes

that

are

primarily

local

Consequently

we

can

match

describe

and

compare

nonrigid

objects

in

a

very

exible

and

general

manner

Point

correspondences

can

now

be

determined

by

comparing

the

two

groups

of

generalized

feature

vectors

The

important

idea

here

is

that

the

low-order

vibration

modes

computed

for

two

similar

objects

will

be

very

similar

even

in

the

presence

of

af?ne

deformation

nonrigid

deformation

local

shape

perturbation

noise

or

small

occlusions

The

points

that

have

the

most

similar

and

unambiguous

coordinates

are

then

matched

with

the

remaining

correspondences

determined

by

using

the

physical

model

as

a

smoothness

constraint

-LSB-

-RSB-

Currently

the

algorithm

has

the

limitation

that

it

can

not

reliably

match

largely

occluded

or

partial

objects

Recovering

Modal

Descriptions

Given

point

correspondences

between

two

shapes

we

can

then

determine

the

deformations

required

to

align

them

An

important

bene?t

of

modal

matching

is

that

the

eigenmodes

computed

for

the

correspondence

algorithm

can

also

be

used

to

describe

the

rigid

and

non-rigid

deformation

needed

to

align

one

object

with

another

Once

this

modal

description

has

been

computed

we

can

compare

shapes

simply

by

looking

at

their

mode

amplitudes

or

since

the

underlying

model

is

energy-based

we

can

compute

and

compare

the

amount

of

deformation

energy

needed

to

align

an

object

and

use

this

as

a

similarity

measure

If

the

strain

energy

required

to

align

two

feature

sets

is

relatively

small

then

the

objects

are

very

similar

Our

task

is

to

recover

the

modal

deformation

parameters

U

that

take

the

set

of

points

from

the

rst

image

to

the

corresponding

points

in

the

second

A

number

of

different

methods

for

recovering

the

modal

deformation

parameters

are

described

in

-LSB-

-RSB-

We

will

only

give

an

overview

of

the

strain-minimizing

leastsquares

method

employed

for

the

database

experiments

described

in

this

paper

Given

that

modal

models

have

been

computed

for

both

shapes

and

that

correspondences

have

been

established

we

can

solve

for

the

modal

displacements

directly

if

correspondence

is

known

at

all

nodes

Unfortunately

correspondence

is

not

usually

available

at

all

nodes

and

our

recovery

problem

becomes

underconstrained

Since

the

modal

matching

algorithm

computes

the

strength

for

each

matched

feature

we

would

also

like

to

utilize

these

match-strengths

directly

in

alignment

As

detailed

in

-LSB-

-RSB-

we

can

obtain

a

constrained

weighted

least

squares

solution

if

we

minimize

alignment

error

that

includes

a

modal

strain

energy

term

U

-LRB-

-RRB-

where

entries

of

the

diagonal

weighting

matrix

W

are

inversely

proportional

to

the

af?nity

measure

for

each

feature

match

The

entries

for

unmatched

features

are

set

to

zero

The

strain

term

directly

parallels

the

smoothness

functional

employed

in

regularization

-LSB-

-RSB-

This

measure

allows

us

to

incorporate

some

prior

knowledge

about

how

stretchy

the

shape

is

how

much

it

resists

compression

etc.

Unmatched

nodes

to

move

in

a

manner

consistent

with

the

material

properties

and

the

forces

at

the

matched

nodes

Encoding

Modal

Shape

Categories

We

will

now

describe

how

to

use

modal

models

to

encode

shape

categories

One

key

advantage

in

using

such

a

prototypebased

approach

is

that

of

data

reduction

given

the

multitude

of

possible

viewpoints

and

con?gurations

for

an

object

we

need

to

reduce

this

multitude

down

to

a

more

ef?cient

representation

that

requires

only

a

few

characteristic

views

Shapes

are

compared

in

terms

of

their

relative

distances

to

prototypes

rather

than

directly

compared

with

one

another

Distance

Measures

Once

the

mode

deformation

parameters

U

have

been

recovered

we

can

compute

the

strain

energy

incurred

by

these

deformations

and

use

this

as

a

similarity

metric

In

general

we

will

want

to

compare

the

strain

only

in

a

subset

of

modes

S

that

has

been

deemed

important

in

measuring

similarity

ffi

-LRB-

A

B

-RRB-

X

-LRB-

-RRB-

where

the

modal

displacements

ui

describe

the

deformation

needed

to

align

shape

A

with

shape

B

It

may

be

desirable

to

make

object

comparisons

rotation

and/or

position

independent

To

do

this

we

ignore

displacements

in

the

rigid

body

modes

thereby

disregarding

differences

in

position

and

orientation

In

addition

we

can

make

our

comparisons

robust

to

noise

and

local

shape

variations

by

discarding

higher-order

modes

This

modal

selection

technique

is

also

useful

for

its

compactness

since

we

can

describe

deviation

from

a

prototype

in

terms

of

relatively

few

modes

If

a

metric

distance

function

is

desired

then

this

simple

energy

measure

needs

to

be

modi?ed

strain

does

not

satisfy

one

of

the

three

axioms

for

a

metric

space

-LSB-

-RSB-

These

three

axioms

are

minimality

ffi

-LRB-

A

B

-RRB-

ffi

-LRB-

A

A

-RRB-

symmetry

ffi

-LRB-

A

B

-RRB-

ffi

-LRB-

B

A

-RRB-

and

triangle

inequality

ffi

-LRB-

A

B

-RRB-

ffi

-LRB-

B

C

-RRB-

ffi

-LRB-

A

C

-RRB-

While

the

strain

energy

measure

satis?es

minimality

and

the

triangle

inequality

it

does

not

satisfy

symmetry

The

strain

energy

is

not

symmetric

for

shapes

of

differing

sizes

i.e.

if

the

scales

of

two

objects

A

and

B

differ

then

the

strain

energy

needed

to

align

A

with

B

may

differ

from

that

needed

to

align

B

with

A

The

difference

in

strain

will

be

inversely

proportional

to

the

difference

in

square

of

the

object

scales

Therefore

when

comparing

objects

of

differing

scales

we

divide

strain

energy

by

the

shape

's

area

When

a

support

map

is

available

this

area

can

be

computed

directly

In

the

in?nite

support

case

the

area

can

be

approximated

by

computing

the

minimum

bounding

circle

or

the

moments

for

the

data

There

is

an

additional

property

that

proves

useful

in

de?ning

a

metric

space

segmental

additivity

ffi

-LRB-

A

B

-RRB-

ffi

-LRB-

B

C

-RRB-

ffi

-LRB-

A

C

-RRB-

if

B

is

on

the

line

between

A

and

C.

To

satisfy

segmental

additivity

we

can

take

the

square

root

of

the

strain

energy

ffi

-LRB-

A

B

-RRB-

X

-LRB-

-RRB-

where

a

is

the

shape

's

area

This

results

in

a

weighted

distance

metric

not

unlike

the

Mahalanobis

distance

the

modal

amplitudes

are

decoupled

each

having

a

variance

that

is

inversely

proportional

to

the

mode

's

eigenvalue

As

a

result

this

formulation

could

be

used

as

part

of

a

regularized

learning

scheme

in

which

the

initial

covariance

matrix

is

iteratively

updated

to

incorporate

the

observed

modal

parameter

covariances

along

the

lines

of

-LSB-

-RSB-

Modal

Shape

Prototypes

Instead

of

looking

at

the

strain

energy

needed

to

align

the

two

shapes

we

wish

to

the

compare

mode

amplitudes

needed

to

align

a

third

prototype

object

C

with

each

of

the

two

objects

In

this

case

we

rst

compute

two

modal

descriptions

Ua

and

Ub

that

align

the

prototype

with

each

candidate

object

We

then

utilize

our

strain-energy

distance

metric

to

order

the

objects

based

on

their

similarity

to

that

prototype

We

can

use

distance

to

prototypes

to

de?ne

a

low-dimensional

space

for

ef?cient

shape

comparison

In

such

a

scenario

a

few

prototypes

are

selected

to

span

the

variation

of

shape

within

each

category

Every

shape

in

the

database

is

then

aligned

with

each

of

the

prototypes

using

modal

matching

and

the

resulting

modal

strain

energy

is

stored

as

an

n-tuple

where

n

is

the

number

of

prototypes

Each

shape

in

the

database

now

has

a

coordinate

in

this

strain-energy-from-prototypes

space

shapes

can

be

compared

simply

in

terms

of

their

Euclidean

distance

in

this

space

We

have

used

strain

energy

for

most

of

our

object

comparison

experiments

since

it

has

a

convenient

physical

meaning

however

we

suspect

that

it

may

sometimes

be

necessary

to

weigh

higher-frequency

modes

less

heavily

since

these

modes

typically

only

describe

high-frequency

shape

variations

and

are

more

susceptible

to

noise

For

instance

we

could

directly

measure

distances

between

modal

descriptions

U

Our

preliminary

experiments

in

prototype-based

shape

description

have

shown

that

this

metric

yields

comparable

performance

to

the

strain

energy

metric

Spanning

Categories

with

Prototypes

In

our

current

image

database

system

a

human

operator

selects

a

few

example

shapes

that

approximately

span

each

category

Our

system

performance

is

therefore

dependent

on

the

user

's

ability

to

select

an

adequately

diverse

and

suf?cient

set

of

prototypes

It

may

be

desirable

to

have

a

system

that

could

automatically

select

prototypes

in

an

unsupervised

fashion

An

unsupervised

learning

or

clustering

-LRB-

e.g.

k-means

hiearchical

clustering

iterative

optimization

Bayes

classi?ers

-RRB-

could

be

adapted

for

automatically

selecting

the

prototype

shapes

based

on

modal

matching

and

modal

strain

Using

such

methods

introduces

a

tradeoff

because

for

many

pattern

classi?cation

and

learning

schemes

it

is

critical

that

training

data

sets

be

large

and

diverse

enough

to

characterize

the

variations

within

a

particular

shape

class

-LSB-

-RSB-

This

shifts

the

pressure

from

a

human

selecting

adequate

prototypes

to

a

human

providing

suf?cient

diverse

and

large

training

data

set

-LRB-

and

providing

the

number

of

categories

present

-RRB-

Finding

these

clusters

without

prototypes

would

-LRB-

in

general

-RRB-

require

matching

all

shapes

to

all

other

shapes

before

optimal

clusters

could

be

obtained

In

either

case

qualities

missing

from

either

the

training

data

or

the

prototypes

may

be

ignored

or

misinterpreted

Another

issue

is

orthogonality

It

is

unlikely

that

the

selected

shape

prototypes

will

describe

orthogonal

axes

in

some

idealized

category

space

To

ensure

orthogonality

we

have

employed

a

method

based

on

nding

the

principal

components

Given

a

set

of

prototypes

we

compute

the

strain-to-prototypes

feature

vector

and

its

covariance

matrix

for

a

randomly

selected

subset

of

shapes

in

the

database

The

eigenvectors

of

the

covariance

matrix

are

used

to

transform

all

into

new

coordinates

in

an

orthogonalized

parameter

space

-LRB-

-RRB-

where

is

a

diagonal

matrix

containing

the

eigenvalues

Computing

distances

in

this

new

space

is

equivalent

to

computing

the

Mahalonobis

distance

in

the

original

strain-to-prototypes

space

As

before

variation

orthogonal

to

the

space

spanned

by

the

training

set

will

not

be

represented

This

may

at

rst

seem

like

a

limitation

however

this

property

can

be

exploited

to

constrain

the

allowable

deformations

to

only

those

that

are

statistically

most

likely

Furthermore

principal

components

with

eigenvalues

less

than

a

threshold

can

be

discarded

to

gain

a

lower-dimensional

parameter

space

as

well

as

better

robustness

to

noise

-LSB-

-RSB-

The

transform

to

the

orthogonalized

parameter

space

is

done

as

a

precomputation

-LRB-

prior

to

repeated

database

search

-RRB-

The

method

has

been

tested

in

experiments

with

a

database

of

handtool

images

as

will

be

detailed

in

Section

Comparison

Without

Direct

Correspondence

Computation

In

an

alternative

method

we

can

measure

the

distance

between

modes

and

determine

how

similar

two

shape

's

modes

are

without

actually

computing

feature

correspondences

This

can

be

accomplished

by

measuring

the

Hausdorff

distance

between

the

loworder

mode

vectors

of

the

new

shape

and

the

low-order

modes

of

a

prototype

Given

two

mode

vectors

OEa

and

OEb

one

from

the

rst

shape

A

and

another

from

the

second

shape

B

we

can

de?ne

the

Hausdorff

distance

between

these

mode

vectors

as

H

-LRB-

OEa

OEb

-RRB-

max

-LRB-

h

-LRB-

OEa

OEb

-RRB-

h

-LRB-

OEb

OEa

-RRB-

-RRB-

-LRB-

-RRB-

where

h

-LRB-

OEa

OEb

-RRB-

is

the

directed

Hausdorff

distance

from

OEa

to

OEb

h

-LRB-

OEa

OEb

-RRB-

max

OEa

min

OEb

k

OEa

i

OEb

i

k

-LRB-

-RRB-

The

distance

norm

is

taken

between

generalized

features

of

Eq

-LRB-

ua

i

va

i

-RRB-

and

-LRB-

ub

i

vb

i

-RRB-

In

our

experiments

we

have

used

a

Euclidean

norm

This

measure

requires

no

speci?c

correspondence

between

points

on

the

two

objects

In

this

formulation

we

match

and

compare

modes

consequently

for

each

shape

in

the

database

we

tally

the

number

of

modes

that

match

each

prototype

's

modes

Typically

mode

distances

are

computed

for

only

the

lowest-order

or

fewer

of

the

nonrigid

modes

These

tallies

can

be

stored

as

coordinates

in

an

n-dimensional

similarity

space

thus

shape

similarity

is

proportional

to

the

Euclidean

distance

in

this

space

Finally

if

two

shapes

have

no

modes

falling

within

the

reasonable

tolerance

for

similarity

then

the

shapes

will

be

agged

as

no

similar

modes

This

computation

can

precede

direct

point

correspondence

or

alignment

computation

The

lack

of

modal

similarity

is

a

strong

clue

that

the

shapes

are

probably

from

different

categories

and

therefore

attempting

correspondence

and

alignment

would

be

unreasonable

This

method

is

used

in

the

experiments

with

the

hand

tool

image

database

described

in

the

next

section

Experiments

in

Interactive

Search

In

the

rst

set

of

experiments

our

method

is

used

to

structure

an

image

database

of

sh

The

images

in

this

experimental

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

Figure

The

ve

prototype

shapes

used

in

the

image

database

experiment

-LRB-

a

-RRB-

Squirrel

Fish

-LRB-

b

-RRB-

Spot

Fin

Butter?y?sh

-LRB-

c

-RRB-

Coney

-LRB-

d

-RRB-

Horse

Eye

Jack

and

-LRB-

e

-RRB-

Southern

Sennet

-LCB-

z

-RCB-

prototype

-LCB-

z

-RCB-

sh

that

had

no

modes

matching

this

prototype

Figure

Six

sh

had

no

modes

that

came

within

tolerance

of

matching

modes

for

the

Butter?y

Fish

prototype

in

Figure

-LRB-

b

-RRB-

and

are

clearly

not

in

the

Butter?y

Fish

category

database

were

digitized

from

a

children

's

eld

guide

-LSB-

-RSB-

Currently

there

are

images

of

tropical

sh

in

the

database

Each

image

depicts

a

sh

from

the

canonical

viewpoint

-LRB-

side

view

-RRB-

though

orientation

position

and

scale

vary

Each

sh

is

unoccluded

and

appears

on

a

uniform

background

Images

for

this

and

other

experiments

are

available

for

anonymous

FTP

from

cspub.bu.edu

in

the

compressed

tar

le

sclaroff/pictures.tar

Z

We

used

the

prototype-based

shape

description

method

formulated

in

Sec.

where

each

shape

's

strain-energy

distance

to

the

prototypes

was

precomputed

and

stored

for

interactive

search

later

First

for

each

image

a

support

map

and

edge

image

was

computed

a

nite-support

shape

model

was

built

and

then

the

eigenmodes

were

determined

For

the

shapes

in

this

experiment

approximately

nite

element

nodes

were

chosen

so

as

to

be

roughly-regularly

spaced

across

the

support

region

Each

shape

in

the

database

is

then

modal

matched

to

a

set

of

prototype

images

There

were

ve

sh

prototypes

as

shown

in

Fig.

These

prototype

images

were

selected

by

a

human

operator

so

as

to

span

the

range

of

shapes

in

the

database

For

sh

prototypes

we

chose

prototypes

that

span

the

range

from

skinny

sh

-LRB-

Fig.

-LRB-

e

-RRB-

-RRB-

to

fat

sh

-LRB-

-LRB-

b

-RRB-

-RRB-

and

from

smooth

sh

-LRB-

-LRB-

c

-RRB-

-RRB-

to

prickly

or

pointy-tailed

sh

-LRB-

-LRB-

a

d

-RRB-

-RRB-

Not

all

shapes

in

the

database

have

similar

modes

-LRB-

similarity

is

measured

to

within

a

threshold

-RRB-

This

information

was

quickly

deterined

by

using

the

Hausdorff

distance

measure

described

in

Section

Sometimes

as

is

shown

in

Fig.

even

shapes

within

the

same

category

do

not

have

similar

modes

In

this

particular

case

the

modes

of

the

wide-bodied

Butter?y

Fish

prototype

of

Fig.

did

not

match

well

with

the

modes

of

the

most

narrowbodied

sh

Using

the

more

ef?cient

Hausdorff

distance

we

can

quickly

determine

when

modes

are

nowhere

near

being

similar

and

no

attempt

at

alignment

and

strain

energy

computation

is

made

Such

shapes

are

simply

agged

as

being

not

at

all

similar

to

a

particular

shape

prototype

as

described

in

Section

The

resulting

modal

strain

energy

was

then

used

as

a

similarity

metric

in

Photobook

an

image

database

management

system

developed

at

the

MIT

Media

Lab

-LSB-

-RSB-

Using

Photobook

the

user

selected

the

image

at

the

upper

left

and

the

system

retrieved

the

remaining

images

sorted

by

strain

energy

-LRB-

shape

similarity

-RRB-

from

left

to

right

top

to

bottom

The

similarity

measure

is

shown

below

each

image

The

database

searches

in

Figs.

through

were

conducted

using

distance

in

prototype-space

In

Fig.

a

Banded

Butter?y?sh

was

selected

The

matches

are

shown

in

order

starting

with

the

most

similar

Based

on

mode-similarity-distance

the

system

retrieved

the

animal

shapes

that

were

closest

to

the

Banded

Butter?y

Fish

shape

-LRB-

other

Butter?y

Fish

and

other

fat-bodied

sh

-RRB-

In

the

second

search

shown

in

Fig.

a

Trumpet

Fish

was

selected

In

this

case

the

system

retrieved

similar

long

and

skinny

sh

In

both

searches

the

sh

judged

most

similar

by

the

system

appeared

on

the

same

page

in

the

eld

guide

This

type

of

similarity

judgment

performance

is

an

encouraging

result

since

sh

appearing

under

the

same

heading

are

nearly

always

in

the

same

taxonomic

category

e.g.

Groupers

Jacks

Snappers

Porgies

Squirrel?shes

Butter?y?shes

Hamlets

or

Damsel?shes

In

the

cases

where

sh

listed

under

the

same

heading

are

not

in

the

same

taxonomic

category

it

is

because

they

were

grouped

together

due

to

some

shape

similarity

e.g.

Slim-bodied

shes

is

the

heading

under

which

the

Trumet?sh

Bluespotted

Cornet?sh

Balao

Needle?sh

Ballyhoo

and

Hound?sh

appear

Fig.

continues

this

example

this

time

searching

for

shapes

most

similar

to

a

Crevalle

Jack

and

a

Dog

Snapper

Again

the

matches

are

shown

in

order

starting

with

the

most

similar

The

shapes

most

similar

to

a

Crevalle

Jack

are

other

sh

with

similar

body

and

tail

shapes

In

this

case

the

system

rates

Jolt

Head

Porgy

over

a

closer

relative

-LRB-

Yellow

Jack

-RRB-

This

is

fairly

reasonable

since

all

are

closely-related

open

water

sh

In

the

nal

example

Fig.

the

user

selected

a

Dog

Snapper

Again

the

system

rated

sh

from

the

same

pages

in

the

eld

guide

as

most

similar

In

each

example

search

and

display

took

less

than

a

second

on

an

HP

Database

queries

were

performed

for

each

of

the

sh

images

in

the

database

for

which

there

were

other

sh

under

same

Figure

Searching

an

image

database

for

similarly-shaped

sh

In

this

example

distance

in

mode-similarity-space

was

used

as

a

shape

similarity

metric

The

gure

shows

the

rst

of

four

examples

of

the

ordering

that

resulted

in

searches

for

similar

sh

a

Banded

Butter?y

Fish

The

matches

are

shown

in

order

starting

with

the

most

similar

Based

on

mode-similarity-distance

the

system

retrieved

the

animal

shapes

that

were

closest

to

the

Banded

Butter?y

Fish

shape

-LRB-

other

Butter?y

Fish

and

other

fat-bodied

sh

-RRB-

The

sh

judged

most

similar

by

the

system

appeared

on

the

same

page

in

the

original

eld

guide

book

and

in

the

same

taxonomic

class

heading

in

the

eld

guide

Overall

another

sh

under

the

same

heading

in

the

eld

guide

was

judged

as

most

similar

of

the

time

To

gain

enhanced

performance

in

capturing

animal

taxonomies

we

suspect

that

modal

matching

would

need

to

be

part

of

a

combined

system

that

includes

local

feature

and

color

information

For

comparison

the

same

queries

were

performed

using

moment

invariants

based

on

second

and

third-order

moments

-LSB-

-RSB-

To

gain

better

performance

the

covariance

matrix

for

the

seven-dimensional

feature

vectors

was

computed

and

shapes

where

ordered

in

terms

of

their

Mahalonobis

distances

to

the

selected

shape

In

this

case

another

sh

under

the

same

heading

in

the

eld

guide

was

judged

as

most

similar

of

the

time

Figure

Searching

an

image

database

for

similarly-shaped

sh

-LRB-

continued

-RRB-

Trumpet

Fish

The

matches

are

shown

in

order

starting

with

the

most

similar

In

this

second

search

the

system

retrieved

similar

long

and

skinny

sh

that

are

on

the

same

page

in

the

original

eld

guide

Evaluating

Retrieval

Accuracy

using

AVRR

Thus

far

retrieval

performance

has

been

measured

in

terms

of

the

percentage

of

times

that

a

shape

in

the

same

category

is

retrieved

as

most

similar

over

a

number

of

trials

However

the

Photobook

system

and

other

query

by

example

-LRB-

QBE

-RRB-

image

database

systems

-LRB-

IBM

Virage

Jacob

-RRB-

provide

a

list

of

possible

matches

ordered

in

terms

of

their

similarity

distance

from

the

example

image

This

is

in

contrast

to

retrieval

systems

based

on

exact

match

In

exact

match

systems

the

standard

measures

of

precision

and

recall

can

be

employed

-LSB-

-RSB-

However

as

noted

by

Faloutsos

et

al.

-LSB-

-RSB-

systems

that

offer

a

list

of

items

sorted

by

similarity

do

not

fall

under

the

rubric

of

exact

matching

We

need

a

performance

measure

that

embodies

the

positions

in

which

target

items

appear

in

the

retrieval

Ideally

if

there

were

a

total

of

n

items

of

the

same

category

in

the

database

then

these

n

items

would

appear

in

the

rst

n

positions

for

a

similarity-based

retrieval

To

evaluate

retrieval

performance

we

will

employ

the

normalized

recall

metric

developed

to

evaluate

IBM

's

QBIC

-LSB-

-RSB-

As

Figure

Ordering

that

resulted

in

searches

for

similar

sh

-LRB-

continued

-RRB-

a

Crevalle

Jack

The

shapes

most

similar

to

a

Crevalle

Jack

are

other

sh

with

similar

body

shapes

and

pointed

tails

-LRB-

other

open

water

sh

-RRB-

sume

that

the

number

of

categories

shapes

per

category

and

category

membership

for

each

shape

are

known

We

can

measure

the

average

rank

of

all

relevant

items

-LRB-

AVRR

-RRB-

for

a

particular

retrieval

and

then

compare

this

with

the

ideal

average

rank

-LRB-

IAVRR

-RRB-

when

all

n

images

from

a

particular

shape

category

appear

in

the

rst

n

positions

For

a

database

that

contains

n

shapes

in

each

category

IAVRR

In

general

the

equation

for

ideal

average

rank

is

IAVRR

m

c

X

i

-LRB-

-RRB-

where

m

is

the

total

number

of

shapes

in

the

database

c

is

the

number

of

categories

and

ni

is

the

number

of

shapes

in

the

ith

category

The

AVRR

is

computed

based

on

the

actual

ordered

ranking

of

shapes

for

each

database

retrieval

Thus

the

ratio

of

AVRR

to

IAVRR

can

be

used

to

give

a

measure

of

average

retrieval

accuracy

over

a

number

of

experimental

trials

Using

this

measure

the

retrieval

accuracy

was

evaluated

for

the

previously

described

experiments

with

the

sh

image

database

in

Photobook

The

IAVRR

for

this

database

was

and

the

AVRR

was

This

means

that

on

average

the

relevant

image

appears

in

the

ninth

position

The

ratio

of

AVRR/IAVRR

In

con

Figure

Ordering

that

resulted

in

search

for

sh

similar

to

a

Dog

Snapper

The

system

rated

sh

from

the

same

page

in

the

eld

guide

as

most

similar

In

each

example

search

and

display

took

less

than

a

second

on

an

HP

trast

the

AVRR

was

and

AVRR/IAVRR

when

moment

invariants

were

used

Tool

Image

Database

In

a

second

set

of

image

database

experiments

we

used

a

database

of

grayscale

images

of

real

and

toy

hand

tools

There

were

images

from

each

of

three

tool

categories

wrenches

hammers

and

crescent

wrenches

Figure

shows

example

images

taken

from

this

database

Note

that

because

the

toy

tools

were

made

of

plastic

they

could

be

bent

in

various

ways

Further

tools

appeared

in

a

number

of

orientations

and/or

scales

with

varying

lighting

The

tools

were

placed

on

a

uniform

background

so

that

a

simple

fuzzy

c-Means

clustering

technique

could

be

used

for

foreground/background

separation

-LSB-

-RSB-

For

the

shapes

in

this

experiment

approximately

nite

element

nodes

were

chosen

so

as

to

be

roughly-regularly

spaced

across

the

support

region

Mode

amplitudes

for

the

rst

modes

were

recovered

and

used

to

warp

each

prototype

onto

the

other

tools

As

in

the

sh

database

experiments

the

Hausdorff

distance

method

was

used

to

cull

cases

where

no

modes

matched

The

Figure

Some

example

images

from

the

hand

tools

experimental

image

database

There

are

images

of

children

's

toy

tools

and

adult

tools

in

the

database

each

of

category

hammer

single-ended

wrench

and

double-ended

wrench

Because

the

toy

tools

were

made

of

plastic

they

could

be

bent

in

various

ways

Further

tools

appeared

in

a

number

of

orientations

and/or

scales

with

varying

lighting

comparisons

were

made

translation

and

rotation

invariant

by

ignoring

displacements

in

the

rigid

body

modes

Comparisons

were

made

scale

invariant

by

recovering

the

scale

factor

before

nonrigidly

warping

the

shape

to

each

prototype

-LSB-

-RSB-

Total

CPU

time

for

database

precomputation

-LRB-

match

align

and

store

n-tuple

-RRB-

averaged

seconds

per

prototype

on

an

SGI

workstation

Matching

experiments

were

then

conducted

using

the

coordinates

produced

via

the

orthogonalization

procedure

in

Section

Database

queries

were

performed

for

each

of

the

tool

images

in

the

database

Overall

another

tool

from

the

same

category

was

judged

as

most

similar

of

the

time

compared

with

for

the

moments-based

method

For

orthogonalized

strain-from-prototypes

the

AVRR

was

this

means

that

the

average

relevant

image

appears

in

roughly

in

the

fteenth

position

The

IAVRR

for

this

database

is

Thus

the

ratio

of

AVRR/IAVRR

The

momentsbased

method

produced

AVRR

and

AVRR/IAVRR

As

another

point

of

comparison

performance

for

shape-based

Number

of

prototypes

Closest

shape

in

same

category

-LRB-

-RRB-

Chance

Figure

Graph

showing

how

the

number

of

prototypes

affects

the

average

performance

for

retrieval

given

a

database

of

handtools

tools

from

each

of

three

categories

For

each

trial

n

prototypes

were

chosen

at

random

from

the

database

and

searches

conducted

in

orthogonalized

strain-to-prototypes

space

For

each

n

there

were

up

to

trials

Database

queries

were

performed

for

each

of

the

tool

images

in

the

database

With

only

one

prototype

another

tool

from

the

same

category

was

judged

as

most

similar

of

the

time

As

the

number

of

prototypes

reached

four

performance

began

to

level

off

search

in

QBIC

was

reported

to

have

a

AVRR/IAVRR

ratio

of

in

-LSB-

-RSB-

for

a

database

of

airplane

silhouettes

coarsely

categorized

by

viewpoint

and

overall

shape

properties

Number

of

Prototypes

and

Retrieval

Accuracy

Using

the

tool

image

database

an

experiment

was

conducted

to

evaluate

retrieval

accuracy

as

a

function

of

the

number

of

prototypes

used

Multiple

trials

were

conducted

using

between

one

and

ten

prototypes

These

n

prototypes

were

selected

at

random

-LRB-

uniformly

distributed

-RRB-

in

trials

for

each

n.

Average

matching

perfmance

was

evaluated

using

the

coordinates

produced

via

the

orthogonalization

procedure

in

Section

In

each

trial

database

queries

were

performed

for

each

of

the

tool

images

in

the

database

Figures

and

show

the

resulting

performance

curves

The

graph

in

Figure

shows

how

the

number

of

prototypes

affects

the

average

performance

for

database

queries

performed

for

each

of

the

tool

images

in

the

database

With

only

one

prototype

another

tool

from

the

same

category

was

judged

as

most

similar

of

the

time

As

the

number

of

prototypes

reached

four

performance

began

to

level

off

at

approximately

The

graph

in

Figure

shows

how

the

number

of

prototypes

affects

the

average

AVRR

for

retrieval

of

the

handtools

in

the

same

tool

category

With

only

one

prototype

the

AVRR

averaged

The

average

performance

leveled

out

at

prototypes

where

AVRR

The

ideal

AVRR

would

be

The

ratio

AVRR/IAVRR

is

greater

than

two

Number

of

prototypes

Average

AVRR

over

trials

Ideal

AVRR

Figure

Graph

showing

how

the

number

of

prototypes

affects

the

average

rank

-LRB-

AVRR

-RRB-

for

retrieval

of

the

handtools

in

the

same

tool

category

For

each

trial

n

prototypes

were

chosen

at

random

from

the

database

and

searches

conducted

in

orthogonalized

strain-to-prototypes

space

For

each

n

there

were

up

to

trials

With

only

one

prototype

the

AVRR

averaged

The

average

performance

leveled

out

at

prototypes

where

AVRR

The

ideal

AVRR

would

be

Discussion

One

of

the

main

motivations

for

this

research

was

to

provide

improved

shape

representations

for

query

by

image

content

While

the

shape

comparison

algorithms

developed

in

the

machine

vision

and

pattern

recognition

communities

can

serve

as

a

good

starting

point

for

developing

shape-based

image

database

search

methods

retrieval

by

shape

is

still

considered

to

be

one

of

the

most

dif?cult

aspects

of

content-based

image

search

-LSB-

-RSB-

IBM

's

Query

By

Image

Content

system

-LRB-

QBIC

-RRB-

-LSB-

-RSB-

is

perhaps

the

most

advanced

image

database

system

to

date

it

is

available

as

a

commercial

product

QBIC

can

perform

searches

that

combine

information

about

shape

color

and

texture

As

input

the

system

assumes

non-occluded

planar

shapes

that

are

represented

as

a

binary

image

Shape-based

search

in

QBIC

can

not

deal

well

with

nonrigid

deformation

Algebraic

moment

invariants

-LSB-

-RSB-

were

intended

for

modeling

rigid

objects

only

In

addition

the

higher

moments

are

dominated

by

points

that

are

farthest

from

the

centroid

therefore

they

are

highly

susceptible

to

outliers

Similar

moments

do

not

necessarily

guarantee

perceptually

similar

shapes

Other

shape

indexing

schemes

have

been

based

on

local

boundary

features

-LSB-

-RSB-

and

are

therefore

not

very

robust

to

noise

scale

and

sampling

Another

system

proposed

by

Chen

-LSB-

-RSB-

identi?ed

aircraft

shapes

using

elliptic

Fourier

descriptors

Because

it

is

Fourier

descriptor-based

Chen

's

system

suffers

from

problems

with

sampling

and

parameterization

Jagadish

introduced

a

multidimensional

indexing

scheme

that

offered

the

advantage

that

it

could

index

images

much

faster

than

previous

techniques

-LSB-

-RSB-

However

the

system

had

limited

descriptive

power

because

the

shape

similarity

measure

was

too

simple

-LRB-

the

area

difference

between

two

shapes

-RRB-

and

the

underlying

shape

representation

was

polyhedral

-LRB-

representing

shapes

in

terms

of

K-d-b

trees

of

overlapping

minimum

bounding

rectangles

-RRB-

In

contrast

to

previous

formulations

the

FEM

integrals

used

in

the

modal

model

formulation

provide

greater

robustness

to

sampling

outliers

and

missing

data

Furthermore

modal

models

provide

quasi-invariance

to

different

types

of

nonrigid

deformation

while

also

providing

an

ordered

orthogonal

encoding

of

the

nonrigid

deformation

that

relates

a

candidate

shape

to

a

shape

prototype

or

shape

category

Matching

Human

Similarity

Judgments

For

a

image

database

search

to

be

useful

it

is

critical

that

the

shape

similarity

metric

be

able

to

match

human

judgments

of

similarity

This

is

not

to

say

that

the

computation

must

somehow

mimic

the

human

visual

system

but

rather

that

computer

and

human

judgments

of

similarity

must

be

generally

correlated

Without

this

the

images

the

computer

nds

will

not

be

those

desired

by

the

human

user

For

human

shape

similarity

judgments

sometimes

scale

and

rotation

invariance

are

important

other

times

not

-LSB-

-RSB-

it

is

therefore

desirable

to

duplicate

this

performance

in

our

image

database

search

algorithms

In

QBIC

a

weighted

metric

allows

for

subset

selection

and

thus

it

provides

selective

invariance

to

size

and

orientation

-LSB-

-RSB-

Modal

matching

also

provides

this

invariance

to

size

and

orientation

but

unlike

any

of

the

shape

representations

used

in

QBIC

modal

representations

can

also

be

made

invariant

to

af?ne

deformations

and

thus

selectively

invariant

to

changes

in

camera

viewpoint

More

importantly

the

modal

representation

provides

deformation

control

knobs

that

correspond

qualitatively

with

human

's

notions

of

perceptual

shape

similarity

-LSB-

-RSB-

Shape

is

thought

of

in

terms

of

an

ordered

set

of

deformations

from

an

initial

shape

starting

with

bends

tapers

shears

and

moving

up

towards

higher-frequency

shape

variations

Speed

of

Image

Database

Search

Another

concern

in

image

database

search

is

the

computation

speed

Shape-based

image

database

search

must

be

ef?cient

enough

to

be

interactive

A

search

that

requires

minutes

per

image

is

simply

not

useful

in

a

database

with

millions

of

images

Furthermore

interactive

search

speed

makes

it

possible

for

users

to

recursively

re?ne

a

search

by

selecting

examples

from

the

currently

retrieved

images

and

using

these

to

initiate

a

new

selectsort-display

cycle

Thus

users

can

iterate

a

search

to

quickly

zero

in

on

what

they

are

looking

for

As

demonstrated

in

our

image

database

experiments

searches

on

databases

over

one

hundred

images

take

less

than

a

second

-LRB-

including

image

display

-RRB-

on

an

HP

workstation

In

addition

search

time

in

our

system

scales

linearly

on

the

number

of

shapes

in

the

database

Finally

it

is

possible

that

the

notion

of

build

ing

up

prototype-based

modal

categories

could

be

exploited

to

structure

databases

into

taxonomic

trees

thereby

improving

the

computational

complexity

of

image

database

search

Conclusion

A

new

image

database

search

method

has

been

described

The

method

uses

strain

energy

from

deformable

prototypes

to

encode

shape

categories

Retrieval

accuracy

of

this

approach

has

been

demonstrated

in

a

series

of

experiments

with

image

databases

of

animals

scanned

from

children

's

eld

guides

and

of

deformable

hand

tools

digitized

via

a

video

camera

In

these

experiments

the

method

performed

consistently

better

than

search

on

moment

invariants

Experiments

were

also

conducted

to

evaluate

retrieval

accuracy

as

a

function

of

the

number

of

prototypes

used

Relatively

few

prototypes

were

needed

to

produce

stable

performance

when

a

new

orthogonalization

scheme

was

employed

Acknowledgment

This

work

began

while

the

author

was

at

the

MIT

Media

Laboratory

and

is

now

supported

in

part

by

the

Department

of

the

Navy

Of?ce

of

Naval

Research

Young

Investigator

Award

References

-LSB-

-RSB-

P.

Alden

and

R.

Grossenheider

Peterson

First

Guide

to

Mammals

of

North

America

Houghton

Mif?in

Co.

Boston

-LSB-

-RSB-

K.

Bathe

Finite

Element

Procedures

in

Engineering

Analysis

Prentice-Hall

-LSB-

-RSB-

A.

Baumberg

and

D.

Hogg

Learning

exible

models

from

image

sequences

In

Proc

European

Conference

on

Computer

Vision

pages

Stockholm

Sweden

May

-LSB-

-RSB-

I.

Biederman

Human

Image

Understanding

Recent

Research

and

a

Theory

Computer

Vision

Graphics

and

Image

Processing

-LRB-

-RRB-

-LSB-

-RSB-

M.

Black

Recursive

non-linear

estimation

of

discontinuous

ow

elds

In

Proc

European

Conference

on

Computer

Vision

May

-LSB-

-RSB-

A.

Blake

R.

Curwen

and

A.

Zisserman

A

framework

for

spatiotemporal

control

in

the

tracking

of

visual

contours

International

Journal

of

Computer

Vision

-LRB-

-RRB-

-LSB-

-RSB-

F.

Bookstein

Principal

warps

Thin-plate

splines

and

the

decomposition

of

deformations

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

June

-LSB-

-RSB-

M.

Celenk

A

color

clustering

technique

for

image

segmentation

Computer

Graphics

Vision

and

Image

Processing

-LSB-

-RSB-

Z.

Chen

and

S.

Y.

Ho

Computer

vision

for

robust

aircraft

recognition

with

fast

library

search

Pattern

Recognition

-LRB-

-RRB-

-LSB-

-RSB-

I.

Cohen

N.

Ayache

and

P.

Sulger

Tracking

points

on

deformable

objects

In

Proc

European

Conference

on

Computer

Vision

Santa

Margherita

Ligure

Italy

May

-LSB-

-RSB-

T.

Cootes

D.

Cooper

C.

Taylor

and

J.

Graham

Trainable

method

of

parametric

shape

description

Image

and

Vision

Computing

-LRB-

-RRB-

June

-LSB-

-RSB-

T.

Darrell

and

A.

Pentland

Space-time

gestures

In

Proc

CVPR

pages

June

-LSB-

-RSB-

T.

Darrell

and

A.

P.

Pentland

Robust

estimation

of

multiple

models

using

support

maps

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

May

-LSB-

-RSB-

R.

O.

Duda

and

P.

E.

Hart

Pattern

Recognition

and

Scene

Analysis

John

Wiley

New

York

-LSB-

-RSB-

S.

Dudani

K.

Breeding

and

R.

McGhee

Aircraft

identi?cation

by

moment

invariants

IEEE

Trans

Computers

-LRB-

-RRB-

-LSB-

-RSB-

J.

Duncan

R.

Owen

L.

Staib

and

P.

Anandan

Measurement

of

non-rigid

motion

using

contour

shape

descriptors

In

Proc

CVPR

pages

-LSB-

-RSB-

S.

Edelman

On

learning

to

recognize

objects

from

examples

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LSB-

-RSB-

S.

Edelman

Representation

similarity

and

chorus

of

prototypes

Minds

and

Machines

-LSB-

-RSB-

S.

Edelman

and

D.

Weinshall

A

self-orgainizing

multipleview

representation

for

objects

Biological

Cybernetics

-LSB-

-RSB-

C.

Faloutsos

R.

Barber

M.

Flickner

J.

Hafner

W.

Niblack

D.

Petkovic

and

W.

Equitz

Ef?cient

and

effective

querying

by

image

content

Journal

of

Intelligent

Information

Systems

-LSB-

-RSB-

K.

Fukunaga

and

W.

Koontz

Application

of

the

karhunenloeve

expansion

to

feature

selection

and

ordering

IEEE

Trans

Communications

-LRB-

-RRB-

-LSB-

-RSB-

I.

Greenberg

Guide

to

Corals

and

Fishes

of

Florida

th

Bahamas

and

the

Caribbean

Seahawk

Press

Miami

Florida

-LSB-

-RSB-

W.

Grosky

P.

Neo

and

R.

Mehrotra

A

pictorial

index

mechanism

for

model-based

matching

Data

and

Knowledge

Engineering

-LSB-

-RSB-

G.

Healey

Segmenting

images

using

normalized

color

IEEE

Trans

on

Systems

Man

and

Cybernetics

-LSB-

-RSB-

B.

Horn

Closed-form

solution

of

absolute

orientation

using

unit

quarternions

JOSA-A

-LSB-

-RSB-

M.

A.

Ireton

and

C.

S.

Xydeas

Classi?cation

of

shape

for

content

retrieval

of

images

in

a

multimedia

database

In

Proc

Sixth

International

Conference

on

Digital

Processing

of

Signals

in

Communications

pages

Loughborough

UK

September

-LSB-

-RSB-

H.

V.

Jagadish

A

retrieval

technique

for

similar

shapes

In

Proc

International

Conference

on

Management

of

Data

ACM

SIGMOD

pages

Denver

CO

May

-LSB-

-RSB-

M.

Kass

A.

Witkin

and

D.

Terzopoulos

Snakes

Active

contour

models

International

Journal

of

Computer

Vision

-LSB-

-RSB-

G.

J.

Klinker

S.A.

Shafer

and

T.

Kanade

A

physical

approach

to

color

image

understanding

International

Journal

of

Computer

Vision

-LSB-

-RSB-

M.

Leyton

Perceptual

organization

as

nested

control

Biological

Cybernetics

-LSB-

-RSB-

Young

Won

Lim

and

Sang

Uk

Lee

On

the

color

image

segmentation

algorithm

based

on

the

thresholding

and

the

fuzzy

c-means

techniques

Pattern

Recognition

-LRB-

-RRB-

-LSB-

-RSB-

J.Q.

Liu

and

Y.H.

Yang

Multiresolution

color

image

segmentation

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LSB-

-RSB-

D.

Marr

and

K.

Nishihara

Representation

and

Recognition

of

the

Spatial

Organization

of

Three-dimensional

Shapes

In

Proc

of

the

Royal

Society

London

B

volume

-LSB-

-RSB-

R.

Mehrotra

and

W.

I.

Grosky

Shape

matching

utilizing

indexed

hypotheses

generation

and

testing

IEEE

Transactions

of

Robotics

and

Automation

-LRB-

-RRB-

-LSB-

-RSB-

D.

Mumford

Mathematical

theories

of

shape

Do

they

model

perception

In

Proc

SPIE

Conf

on

Geometric

Methods

in

Computer

Vision

volume

-LSB-

-RSB-

W.

Niblack

R.

Barber

W.

Equitz

M.

Flickner

E.

Glasman

D.

Petkovic

and

P.

Yanker

The

QBIC

project

Querying

images

by

content

using

color

texture

and

shape

In

Proc

SPIE

Conf

on

Storage

and

Retrieval

of

Image

and

Video

Databases

volume

February

-LSB-

-RSB-

Y.

Ohta

T.

Kanade

and

T.

Sakai

Color

information

for

region

segmentation

Computer

Graphics

Vision

and

Image

Processing

-LSB-

-RSB-

E.

Oja

Principal

components

minor

components

and

linear

neural

networks

Neural

Networks

-LSB-

-RSB-

E.

Oja

and

J.

Karhunen

Nonlinear

PCA

Algorithms

and

Applications

Technical

Report

Helsinki

University

of

Technology

Laboratory

of

Computer

and

Information

Sciences

Espoo

Finland

-LSB-

-RSB-

A.

Pentland

Perceptual

organization

and

representation

of

natural

form

Arti?cial

Intelligence

-LRB-

-RRB-

-LSB-

-RSB-

A.

Pentland

Automatic

extraction

of

deformable

part

models

International

Journal

of

Computer

Vision

-LRB-

-RRB-

March

-LSB-

-RSB-

A.

Pentland

R.

Picard

and

S.

Sclaroff

Photobook

Tools

for

content-based

manipulation

of

image

databases

International

Journal

of

Computer

Vision

to

appear

fall

-LSB-

-RSB-

A.

Pentland

and

S.

Sclaroff

Closed-form

solutions

for

physically-based

shape

modeling

and

recognition

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

July

-LSB-

-RSB-

T.

Poggio

and

F.

Girosi

A

theory

of

networks

for

approximation

and

learning

Technical

Report

A.I.

Memo

No.

Arti?cial

Intelligence

Lab

MIT

Cambridge

MA

July

-LSB-

-RSB-

E.

Rosch

Cognitive

representations

of

semantic

categories

Journal

of

Experimental

Psychology

General

-LSB-

-RSB-

G.

Salton

and

M.

J.

McGill

Introduction

to

Modern

Information

Retrieval

McGraw-Hill

-LSB-

-RSB-

B.

Scassellati

S.

Alexopoulos

and

M.

Flickner

Retrieving

images

by

shape

comparison

of

computation

methods

with

human

perceptual

judgements

In

Proc

SPIE

Conf

on

Storage

and

Retrieval

of

Image

and

Video

Databases

II

San

Jose

CA

February

-LSB-

-RSB-

S.

Sclaroff

Modal

Matching

A

Method

for

Describing

Comparing

and

Manipulating

Digital

Signals

PhD

thesis

MIT

Media

Lab

January

-LSB-

-RSB-

S.

Sclaroff

and

A.

Pentland

Physically-based

combinations

of

views

Representing

rigid

and

nonrigid

motion

In

Proc

IEEE

Workshop

on

Nonrigid

and

Articulate

Motion

Austin

TX

Nov

-LSB-

-RSB-

S.

Sclaroff

and

A.

Pentland

Modal

Matching

for

Correspondence

and

Recognition

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LSB-

-RSB-

G.

Taubin

and

D.

Cooper

Recognition

and

positioning

of

rigid

objects

In

Proc

SPIE

Conf

on

Geometric

Methods

in

Computer

Vision

volume

-LSB-

-RSB-

D.

Terzopoulos

Topical

Meeting

on

Machine

Vision

volume

of

Technical

Digest

Series

chapter

On

matching

deformable

models

to

images

Direct

and

iterative

solutions

pages

Optical

Society

of

America

Washington

DC

-LSB-

-RSB-

D.

Terzopoulos

The

Computation

of

Visible

Surface

Representations

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

July

-LSB-

-RSB-

W.

B.

Thompson

Combining

motion

and

constrast

for

segmentation

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LSB-

-RSB-

A.

Tversky

Features

of

Similarity

Psychological

Review

-LRB-

-RRB-

-LSB-

-RSB-

S.

Ullman

and

R.

Basri

Recognition

by

linear

combinations

of

models

IEEE

Trans

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LSB-

-RSB-

J.Y.A.

Wang

and

E.

H.

Adelson

Layered

representation

for

motion

analysis

In

Proc

CVPR

-LSB-

-RSB-

J.

Weber

and

J.

Malik

Robust

computation

of

optical

ow

in

a

multi-scale

differential

framework

In

Proc

International

Conference

on

Computer

Vision

pages

-LSB-

-RSB-

A.

Yuille

D.

Cohen

and

P.

Hallinan

Feature

extraction

from

faces

using

deformable

templates

In

Proc

CVPR

pages

San

Diego

combinations of models
deformable models
deformable templates
modal matching
shape matching