View-Based

Object

Recognition

Using

Saliency

Maps

Ali

Shokoufandehx

Ivan

Marsicy

and

Sven

J.

Dickinsonx

xDepartment

of

Computer

Science

and

Center

for

Cognitive

Science

Rutgers

University

New

Brunswick

NJ

fshokoufa,sveng@cs.rutgers.edu

yDepartment

of

Electrical

and

Computer

Engineering

Rutgers

University

New

Brunswick

NJ

marsic@caip.rutgers.edu

Abstract

We

introduce

a

novel

view-based

object

representation

called

the

saliency

map

graph

-LRB-

SMG

-RRB-

which

captures

the

salient

regions

of

an

object

view

at

multiple

scales

using

a

wavelet

transform

This

compact

representation

is

highly

invariant

to

translation

rotation

-LRB-

image

and

depth

-RRB-

and

scaling

and

offers

the

locality

of

representation

required

for

occluded

object

recognition

To

compare

two

saliency

map

graphs

we

introduce

two

graph

similarity

algorithms

The

first

computes

the

topological

similarity

between

two

SMG

's

providing

a

coarse-level

matching

of

two

graphs

The

second

computes

the

geometrical

similarity

between

two

SMG

's

providing

a

fine-level

matching

of

two

graphs

We

test

and

compare

these

two

algorithms

on

a

large

database

of

model

object

views

A

condensed

version

of

this

paper

will

appear

in

the

IEEE

International

Conference

on

Computer

Vision

Bombay

January

under

the

title

View-Based

Object

Matching

Introduction

The

view-based

approach

to

object

recognition

represents

an

object

as

a

collection

of

views

sometimes

called

aspects

or

characteristic

views

-LSB-

-RSB-

The

advantage

of

such

an

approach

is

that

it

avoids

having

to

construct

a

model

of

an

object

as

well

as

having

to

make

inferences

from

features

Many

approaches

to

view-based

modeling

represent

each

view

as

a

collection

of

extracted

features

such

as

extracted

line

segments

curves

corners

line

groups

regions

or

surfaces

-LRB-

Ikeuchi

and

Kanade

-LSB-

-RSB-

Burns

and

Kitchen

-LSB-

-RSB-

Ullman

and

Basri

-LSB-

-RSB-

Dickinson

et

al.

-LSB-

-RSB-

and

Pope

and

Lowe

-LSB-

-RSB-

-RRB-

The

success

of

these

view-based

recognition

systems

depends

on

the

extent

to

which

they

can

extract

their

requisite

features

With

real

images

of

real

objects

in

unconstrained

environments

the

extraction

of

such

features

can

be

both

time

consuming

and

unreliable

In

contrast

to

the

feature-based

view-based

recognition

paradigm

a

number

of

imagebased

view-based

recognition

systems

have

emerged

Beginning

with

the

eigenface

approach

proposed

by

Turk

and

Pentland

-LSB-

-RSB-

these

image-based

approaches

avoid

extracting

complex

features

from

an

image

instead

they

retain

the

entire

raw

image

as

a

single

feature

in

a

high-dimensional

space

Turk

and

Pentland

focused

on

the

domain

of

faces

and

therefore

did

not

require

a

large

set

of

model

views

for

each

face

Nayar

and

Murase

extended

this

work

to

general

objects

where

a

dense

set

of

views

was

acquired

for

each

object

-LSB-

-RSB-

Although

avoiding

costly

and

often

unreliable

feature

extraction

these

image-based

approaches

pay

the

price

of

sensitivity

to

lighting

conditions

image

translation

image

rotation

depth

rotation

occlusion

and

minor

shape

variation

all

of

which

affect

an

image

's

pixel

values

and

result

in

a

change

in

the

image

's

location

in

some

high-dimensional

space

Recent

results

have

shown

some

progress

towards

solving

these

problems

e.g.

the

work

of

Belheumer

and

Kriegman

-LSB-

-RSB-

-LRB-

limited

invariance

to

illumination

changes

-RRB-

and

the

work

of

Leonardis

and

Bischoff

-LSB-

-RSB-

and

Schmid

and

Mohr

-LSB-

-RSB-

-LRB-

limited

invariance

to

occlusion

-RRB-

Nevertheless

the

lack

of

abstraction

from

raw

image

data

to

the

model

means

that

the

model

defines

a

very

specific

object

instance

The

concept

of

computing

coarse-to-fine

image

descriptions

has

much

support

in

the

computer

vision

community

some

examples

include

-LSB-

-RSB-

In

some

cases

attention

models

have

been

developed

that

use

a

multiscale

description

to

decide

where

in

the

image

to

apply

some

operation

Lindeberg

has

based

this

selection

process

on

a

quantitative

analysis

of

gray-level

blobs

in

scale

space

-LSB-

-RSB-

J?agersand

-LSB-

-RSB-

uses

an

information

theoretic

measure

to

compute

informativeness

of

image

regions

at

different

scales

while

others

have

defined

some

measure

of

importance

and

used

it

to

drive

an

attention

process

-LSB-

-RSB-

Although

suitable

for

locating

objects

in

images

for

further

processing

the

above

multiscale

descriptions

often

called

saliency

maps

lose

the

detailed

shape

information

required

for

object

recognition

Some

multiscale

image

descriptions

have

been

used

to

locate

a

particular

target

object

in

the

image

For

example

Rao

et

al.

use

correlation

to

compare

a

multiscale

saliency

map

of

the

target

object

with

a

multiscale

saliency

map

of

the

image

in

order

to

fixate

on

the

object

-LSB-

-RSB-

Although

these

approaches

are

effective

in

finding

a

target

in

the

image

they

like

any

template-based

approach

do

not

scale

to

large

object

databases

Their

bottom-up

descriptions

of

the

image

are

not

only

global

offering

little

means

for

segmenting

an

image

into

objects

or

parts

but

offer

little

invariance

to

occlusion

object

deformation

and

other

transformations

An

approach

similar

to

the

approach

we

will

present

is

due

to

Crowley

et

al.

-LSB-

-RSB-

From

a

Laplacian

pyramid

computed

on

an

image

peaks

and

ridges

at

each

scale

are

detected

as

local

maxima

The

peaks

are

then

linked

together

to

form

a

tree

structure

from

which

a

set

of

peaks

paths

are

extracted

corresponding

to

the

branches

of

the

tree

During

matching

correspondence

between

low-resolution

peak

paths

in

the

model

and

the

image

are

used

to

solve

for

the

pose

of

the

model

with

respect

to

the

image

Given

this

initial

pose

a

greedy

matching

algorithm

descends

down

the

tree

pairing

higher-resolution

peak

paths

from

the

image

and

the

model

Using

a

log

likelihood

similarity

measure

on

peak

paths

the

best

corresponding

paths

through

the

two

trees

is

found

The

similarity

of

the

image

and

model

trees

is

based

on

a

very

weak

approximation

of

the

trees

topology

and

geometry

restricted

in

fact

to

a

single

path

through

the

tree

In

this

paper

we

present

a

multiscale

view-based

representation

of

objects

that

on

one

hand

avoids

the

need

for

complex

feature

extraction

such

as

lines

curves

or

regions

while

on

the

other

hand

provides

the

locality

of

representation

necessary

to

support

occluded

object

recognition

as

well

as

invariance

to

minor

changes

in

both

illumination

and

shape

In

computing

a

representation

for

a

image

-LRB-

whether

model

image

or

image

to

be

recognized

-RRB-

a

multiscale

wavelet

transform

is

applied

to

the

image

resulting

in

a

hierarchical

saliency

map

of

the

image

that

offers

advantages

over

a

Laplacian

pyramid

This

saliency

map

is

represented

as

a

hierarchical

graph

structure

called

the

saliency

map

graph

that

encodes

both

the

topological

and

geometrical

information

found

in

the

saliency

map

The

similarity

between

a

test

image

and

a

model

image

is

defined

as

the

similarity

between

their

respective

saliency

map

graphs

We

address

the

problem

of

matching

two

saliency

map

graphs

leading

to

two

matching

algorithms

The

first

algorithm

finds

the

best

mapping

between

two

saliency

map

graphs

in

terms

of

their

topological

structure

while

the

second

algorithm

factors

in

the

geometry

of

the

two

graphs

In

each

case

we

present

an

evaluation

function

that

determines

the

overall

quality

of

the

match

i.e.

the

similarity

of

the

two

graphs

We

demonstrate

and

evaluate

our

image

representation

and

our

two

matching

algorithms

using

the

Columbia

University

COIL

image

database

In

addition

we

assess

the

viewpoint

invariance

of

our

representation

and

matching

algorithms

A

Scale-Space

Saliency

Representation

of

an

Image

To

reduce

the

complexity

in

matching

input

image

representations

to

model

view

representations

we

seek

a

scale-space

or

coarse-to-fine

representation

of

images

that

allows

us

to

first

match

or

index

based

on

the

coarse-level

features

in

the

image

Coarse-level

correspondence

can

then

be

used

to

constrain

a

fine-level

matching

of

the

remaining

features

Furthermore

we

would

like

our

image

representation

to

be

invariant

to

slight

variations

in

the

illumination

falling

on

the

object

image-plane

rotation

translation

and

scaling

of

the

object

slight

rotation

in

depth

of

the

object

slight

deformations

of

the

shape

of

the

object

e.g.

stretching

bending

etc.

and

occlusion

of

the

object

Traditional

view-based

object

representations

that

are

image

based

e.g.

-LSB-

-RSB-

are

neither

coarse-to-fine

nor

invariant

to

the

above

transformations

due

to

the

global

nature

of

their

representations

-LRB-

although

some

offer

limited

invariance

to

particular

transformations

-RRB-

However

the

advantage

of

these

approaches

is

that

complex

feature

extraction

grouping

or

abstraction

is

not

required

Systems

based

on

more

invariant

view-based

image

descriptions

e.g.

-LSB-

-RSB-

have

relied

on

complex

feature

extraction

-LRB-

e.g.

edges

lines

regions

etc.

-RRB-

which

is

not

only

unreliable

but

often

requires

domain-specific

parameter

tuning

To

address

these

shortcomings

we

compute

a

scale-space

representation

of

an

image

in

which

image

objects

-LRB-

homogeneous

regions

-RRB-

are

located

at

the

coarsest

scale

which

captures

their

salient

shape

properties

Moreover

both

the

geometrical

and

topological

relations

between

the

regions

will

be

explicitly

encoded

in

the

representation

Finally

computing

these

regions

and

relations

requires

the

setting

of

very

few

parameters

The

Multiscale

Wavelet

Transform

The

scale-space

image

representation

that

we

have

selected

is

based

on

a

multiscale

wavelet

transform

-LSB-

-RSB-

The

advantage

of

the

wavelet

decomposition

lies

in

its

effective

time

-LRB-

space

-RRB-

-LCB-

frequency

-LRB-

scale

-RRB-

localization

Unlike

other

image

transforms

e.g.

-LSB-

-RSB-

which

spread

the

information

across

their

basis

functions

the

wavelet

transform

allows

us

to

compute

better

localized

object

representations

In

the

output

of

the

transform

as

illustrated

in

Figure

the

salient

shape

of

small

objects

is

best

captured

by

small

wavelets

while

the

converse

is

true

for

large

objects

Searching

from

finer

to

coarser

scales

-LRB-

right

to

left

in

Figure

-RRB-

we

select

the

scale

which

captures

the

most

efficient

encoding

of

an

object

's

salient

shape

above

the

chosen

scale

extraneous

information

is

encoded

while

below

the

chosen

scale

the

object

is

overly

blurred

The

region

defining

the

object

at

the

chosen

scale

is

called

the

scale-space

cell

-LRB-

SSC

-RRB-

-LSB-

-RSB-

The

dyadic

wavelet

transform

of

a

function

f

-LRB-

R

-RRB-

at

the

scale

and

at

the

position

k

is

given

by

the

inner

products

of

the

function

with

the

family

of

wavelets

-LRB-

Wf

-RRB-

-LRB-

j

k

-RRB-

hf

j

ki

j

Z

f

-LRB-

x

-RRB-

-LRB-

jx

k

-RRB-

dx

-LRB-

-RRB-

Wavelet

DecompositionInput

Image

xylog

-LRB-

-RRB-

xlog

-LRB-

-RRB-

scoarsescalefinescale

Figure

In

the

proposed

multiscale

description

objects

are

captured

at

the

coarsest

scale

which

captures

their

salient

properties

Note

that

the

frequency

and

the

scale

s

are

inversely

proportional

where

the

overline

denotes

the

complex

conjugate

This

inner

product

may

also

be

viewed

as

a

convolution

product

-LRB-

Wf

-RRB-

-LRB-

j

k

-RRB-

f

j

k

or

as

a

filtering

of

the

function

f

with

a

band-pass

filter

whose

impulse

response

is

j

k.

Detecting

the

scale-space

cells

requires

analysis

of

the

wavelet

transform

response

at

each

scale

The

scale-space

cell

of

an

image

object

is

located

at

the

scale

which

is

approximately

one

octave

below

the

scale

at

which

the

object

's

response

becomes

indistinguishable

from

other

image

objects

of

the

same

size

At

this

scale

the

object

's

response

resembles

the

wavelet

basis

function

impulse

response

The

following

subsections

will

explore

scale-space

cell

detection

in

greater

detail

Scale-Space

Cells

in

One

Dimension

To

illustrate

the

detection

of

scale-space

cells

consider

the

one-dimensional

signal

in

Figure

using

the

wavelets

described

in

-LSB-

-RSB-

Any

object

behaves

like

a

point

at

all

scales

coarser

than

its

characteristic

scale

I

This

means

that

the

wavelet

transform

of

a

signal

at

any

scale

j

I

is

the

same

no

matter

what

the

object

's

shape

is

the

transform

is

completely

determined

by

the

width

of

the

objects

and

their

amplitude

We

can

ignore

the

information

at

the

scales

j

I

and

still

be

able

to

reconstruct

the

original

signal

almost

perfectly

as

illustrated

in

the

right

column

in

Figure

The

reconstructed

signal

f

-LRB-

x

-RRB-

is

obtained

by

replacing

the

responses

-LRB-

W

-RRB-

-LRB-

x

-RRB-

of

objects

A

and

B

with

the

analyzing

wavelet

k

of

the

same

amplitude

Similarly

f

-LRB-

x

-RRB-

f

-LRB-

x

-RRB-

and

f

-LRB-

x

-RRB-

are

obtained

by

successively

replacing

the

corresponding

wavelet

responses

at

the

scales

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

with

the

analyzing

wavelets

The

error

in

reconstruction

is

very

small

if

the

replacement

takes

place

at

scales

greater

than

the

characteristic

scale

the

error

increases

significantly

as

the

replacement

takes

place

at

the

finer

scales

In

the

general

case

interactions

between

the

neighboring

objects

will

distort

the

wavelet

transform

response

However

even

for

complex

signals

each

object

will

eventually

yield

the

-LRB-

approximate

-RRB-

impulse

response

at

the

appropriate

scale

determined

by

the

size

of

the

object

In

order

to

find

the

characteristic

scale

of

an

image

object

in

the

-LCB-

D

case

one

can

measure

the

correlation

between

the

wavelet

transform

of

the

object

and

the

basis

function

at

any

given

scale

At

each

location

-LRB-

x

y

-RRB-

one

can

then

select

the

finest

scale

at

which

the

correlation

exceeds

some

threshold

We

will

now

proceed

to

examine

the

detection

of

a

scale-space

cell

in

two

dimensions

Saliency

Detection

Algorithm

in

Two

Dimensions

In

the

two-dimensional

case

the

characteristic

scale

I

-LRB-

-RRB-

may

be

different

for

any

particular

orientation

of

a

-LCB-

D

cross-section

through

an

object

Any

object

other

than

a

circular

one

-LRB-

disc

-RRB-

will

become

a

point

at

different

scales

in

different

directions

e.g.

an

elongated

object

occluded

object

etc.

In

this

case

the

object

will

extend

over

several

SSC

's

at

any

scale

j

max?fI

-LRB-

-RRB-

g.

Therefore

in

the

-LCB-

D

case

we

apply

the

-LCB-

D

procedure

in

a

number

of

directions

and

search

for

clusters

of

-LCB-

D

centers

as

shown

in

Figure

-LRB-

a

-RRB-

Our

entire

procedure

for

detecting

the

SSC

's

in

an

image

therefore

consists

of

the

following

four

steps

-LSB-

-RSB-

-LRB-

-RRB-

f

r

-LRB-

-RRB-

f

-LRB-

-RRB-

rf

x

-LRB-

-RRB-

f

xWf

x

-LRB-

-RRB-

-LRB-

-RRB-

Wf

x

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Wf

x

-LRB-

-RRB-

-LRB-

-RRB-

Wf

x

-LRB-

-RRB-

-LRB-

-RRB-

Wf

x

ABWf

x

-LRB-

-RRB-

-LRB-

-RRB-

k

-LRB-

-RRB-

f

r

Figure

An

illustration

of

the

object

's

characteristic

scale

The

left

column

shows

the

analyzing

wavelets

i

k

at

the

corresponding

scales

The

middle

column

shows

two

examples

-LRB-

A

and

B

-RRB-

of

a

function

f

-LRB-

x

-RRB-

-LRB-

top

-RRB-

and

their

wavelet

transforms

-LRB-

below

-RRB-

Beginning

with

the

characteristic

scale

I

the

-LRB-

W

-RRB-

-LRB-

I

j

x

-RRB-

takes

the

shape

of

I

j

k

j

The

right

column

shows

the

reconstruction

of

the

original

function

by

replacing

the

-LRB-

W

-RRB-

-LRB-

i

x

-RRB-

with

the

i

k.

Step

Wavelet

Transform

Compute

the

wavelet

pyramid

of

an

image

with

dyadic

scales

using

oriented

quadrature

bandpass

filters

tuned

to

different

orientations

i.e.

ffi

ffi

ffi

See

-LSB-

-RSB-

for

a

detailed

derivation

and

description

of

computing

the

wavelet

pyramid

using

steerable

basis

filters

Step

Local

Energies

Compute

the

oriented

local

energies

using

the

equation

E

-LRB-

s

x

y

-RRB-

h

G

-LRB-

s

x

y

-RRB-

h

H

-LRB-

s

x

y

-RRB-

-LRB-

-RRB-

CbCaCg

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

The

clusters

formed

by

the

centers

of

the

-LCB-

D

SSCs

associated

with

the

crosssections

through

an

object

-LRB-

b

-RRB-

One

of

the

filter

kernels

-LRB-

x

y

-RRB-

used

in

computing

the

saliency

of

the

SSC

's

-LRB-

Eq

-LRB-

-RRB-

-RRB-

The

kernel

is

obtained

by

computing

the

oriented

energy

at

ffi

for

a

disc

where

G

-LRB-

s

x

y

-RRB-

and

H

-LRB-

s

x

y

-RRB-

are

the

outputs

of

a

quadrature

pair

of

analyzing

wavelet

filters

at

the

scale-space

coordinate

-LRB-

s

x

y

-RRB-

oriented

at

the

angle

For

each

image

point

different

oriented

local

energies

are

computed

Step

Saliency

Maps

Compute

saliency

maps

The

saliency

of

each

particular

SSC

is

computed

using

the

convolution

saliency

SSC

-LRB-

s

x

y

-RRB-

X

-LSB-

E

-LRB-

s

x

y

-RRB-

-LRB-

x

y

-RRB-

-RSB-

-LRB-

-RRB-

where

-LRB-

x

y

-RRB-

is

the

filter

kernel

obtained

by

computing

the

sum

of

the

squared

impulse

responses

of

the

two

analyzing

wavelet

filters

G

-LRB-

s

x

y

-RRB-

and

H

-LRB-

s

x

y

-RRB-

as

shown

in

Figure

-LRB-

b

-RRB-

As

discussed

above

circular

shape

has

the

highest

saliency

as

measured

by

this

scheme

Step

Peaks

in

Saliency

Maps

Moving

from

finer

to

coarser

scales

at

every

location

we

select

the

first

saliency

map

for

the

which

a

peak

-LRB-

local

maximum

-RRB-

at

that

location

exceeds

a

given

threshold

By

using

a

series

of

oriented

filters

to

detect

the

characteristic

scale

we

can

detect

objects

that

are

not

perfectly

circular

in

shape

For

example

if

a

non-circular

shape

's

variation

in

diameter

does

not

reach

neighboring

scales

above

or

below

the

current

scale

then

a

circularly-symmetric

filter

such

as

that

used

by

Crowley

-LSB-

-RSB-

will

give

a

weak

response

for

the

shape

In

our

approach

however

the

filters

are

slightly

adjusted

in

width

-LRB-

bounded

by

neighboring

scales

-RRB-

The

result

is

a

cluster

of

oriented

peaks

from

which

we

compute

the

shape

's

location

as

the

centroid

of

these

peaks

The

salience

of

the

shape

is

computed

as

the

sum

of

the

oriented

saliencies

of

the

oriented

peaks

near

this

centroid

Finally

we

apply

a

non-maximum

suppression

process

to

eliminate

closely

overlapping

salient

SSC

's

at

each

scale

The

contents

of

each

scale-space

cell

-LRB-

SSC

-RRB-

is

a

matrix

of

wavelet

coefficients

The

size

of

this

matrix

is

invariant

to

both

the

scale

at

which

the

SSC

is

detected

and

the

complexity

of

the

shape

contained

in

the

SSC

's

corresponding

image

region

In

the

current

implementation

only

the

scale

position

and

saliency

of

a

SSC

is

exploited

during

the

matching

of

two

saliency

maps

However

one

could

include

the

actual

content

of

the

SSC

as

specified

by

the

wavelet

coefficient

matrix

The

fact

that

all

the

SSC

matrices

are

self-similar

and

small

-LRB-

in

our

case

-RRB-

means

that

efficient

comparisons

can

be

made

between

SSC

's

at

different

scales

Figure

illustrates

how

a

complex

object

's

saliency

map

-LRB-

a

-RRB-

is

largely

invariant

to

scaling

-LRB-

b

-RRB-

translation

-LRB-

c

-RRB-

image

plane

rotation

-LRB-

d

-RRB-

and

limited

rotation

in

depth

-LRB-

e

-RRB-

where

the

illuminated

left

side

of

the

face

exhibits

little

change

in

its

saliency

map

Circles

in

the

image

correspond

to

scale-space

cells

while

their

intensity

is

proportional

to

the

their

saliency

Note

that

the

size

of

the

circle

appears

to

be

slightly

larger

than

its

corresponding

image

feature

This

is

due

to

the

fact

that

the

size

of

the

circle

is

determined

by

the

largest

extent

of

the

filter

shown

in

Figure

-LRB-

b

-RRB-

i.e.

where

the

response

approaches

zero

For

different

images

taken

under

different

conditions

there

can

not

be

true

invariance

in

the

sense

that

salient

regions

are

identical

in

different

views

However

approximate

invariance

suffices

for

the

recognition

scheme

that

we

propose

below

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

-LRB-

f

-RRB-

Figure

Extracting

the

most

salient

SSC

's

in

an

image

-LRB-

a

-RRB-

original

image

and

its

saliency

map

-LRB-

b

-RRB-

scale

invariance

-LRB-

c

-RRB-

translation

invariance

-LRB-

d

-RRB-

image

rotation

invariance

-LRB-

e

-RRB-

invariance

to

rotation

in

depth

-LRB-

illuminated

left

side

of

face

exhibits

little

change

in

its

saliency

map

-RRB-

and

-LRB-

f

-RRB-

the

saliency

map

graph

-LRB-

SMG

-RRB-

of

the

original

image

in

-LRB-

a

-RRB-

Limitations

of

the

Representation

Under

normal

circumstances

an

object

-LRB-

or

one

of

its

component

features

-RRB-

should

produce

a

peak

at

its

corresponding

location

in

the

saliency

map

However

there

are

several

exceptions

which

do

not

properly

fit

within

the

SSC

framework

The

evolution

of

the

saliency

map

at

a

single

scale

as

the

object

becomes

degenerate

in

the

sense

of

the

SSC

framework

is

shown

in

Figure

The

exceptions

arise

due

to

wavelet

transform

scale

sub-sampling

crowded

objects

and

elongated

objects

As

expected

combinations

of

these

will

create

even

more

difficulties

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Figure

The

evolution

of

the

saliency

map

at

a

single

scale

for

the

exception

cases

The

left

side

shows

the

original

images

and

the

right

side

shows

the

corresponding

saliency

maps

at

one

scale

-LRB-

a

-RRB-

As

the

object

size

increases

a

peak

in

the

saliency

map

corresponding

to

the

object

turns

into

a

crater

-LRB-

b

-RRB-

As

the

objects

approach

closer

to

one

another

the

anti-object

between

them

becomes

the

most

salient

-LRB-

c

-RRB-

Instead

of

peak

-LRB-

s

-RRB-

in

the

saliency

map

elongated

objects

produce

mountain

ridges

As

shown

in

Figure

-LRB-

a

-RRB-

the

saliency

peak

detector

will

find

one

salient

region

for

the

object

at

the

left

side

whereas

it

will

find

several

salient

regions

for

larger

objects

on

the

right

side

This

effect

will

occur

when

the

size

of

an

object

is

intermediate

between

two

-LRB-

octave

-RRB-

scales

and

can

be

minimized

by

increasing

the

number

of

scales

In

Figure

-LRB-

b

-RRB-

the

detector

will

find

four

salient

regions

for

the

distant

objects

at

left

but

will

find

only

one

region

in

the

center

of

the

four

close

objects

at

right

This

phenomenon

occurs

when

a

set

of

objects

enclose

a

compact

background

region

the

detector

can

not

separate

figure

from

ground

One

could

argue

that

such

a

regular

pattern

represents

a

form

of

texture

that

should

be

treated

as

a

single

object

Again

if

there

were

more

scales

this

composite

object

would

be

detected

at

a

coarser

scale

Finally

in

Figure

-LRB-

c

-RRB-

the

detector

will

find

several

salient

regions

positioned

along

the

elongated

object

In

this

case

a

salient

region

grouper

could

search

for

a

string

of

co-curvilinear

SSC

's

at

a

given

scale

and

group

them

into

a

composite

structure

which

could

easily

be

accommodated

by

our

graph

representation

and

matching

methods

outlined

below

An

approach

similar

to

this

was

proposed

by

Crowley

for

detecting

ridges

in

a

Laplacian

pyramid

-LSB-

-RSB-

The

Saliency

Map

as

a

Graph

The

computed

saliency

map

can

be

represented

as

a

hierarchical

graph

with

nodes

representing

saliency

regions

and

specifying

region

location

-LRB-

in

the

image

-RRB-

region

size

region

saliency

and

scale

level

More

formally

we

define

the

Saliency

Map

Graph

-LRB-

SMG

-RRB-

to

be

a

directed

acyclic

graph

G

-LRB-

V

E

-RRB-

with

each

saliency

region

ri

having

a

vertex

vi

in

V

-LRB-

vi

vj

-RRB-

is

a

directed

edge

in

E

if

and

only

if

the

scale

level

of

region

ri

is

less

than

the

scale

level

of

region

rj

and

the

center

of

the

region

rj

lies

entirely

in

the

interior

of

the

region

ri

All

the

edges

of

G

will

therefore

be

directed

from

vertices

at

a

coarser

scale

to

vertices

at

a

finer

scale

as

shown

in

Figure

-LRB-

lower-right

-RRB-

Finally

to

construct

the

database

of

object

views

a

set

of

views

is

obtained

for

each

object

from

a

fixed

number

of

viewpoints

-LRB-

e.g.

a

regularly

sampled

tesselation

of

a

viewing

sphere

centered

at

the

object

-RRB-

For

each

view

the

Saliency

Map

Graph

is

computed

and

stored

in

the

database

Matching

Two

Saliency

Map

Graphs

Given

the

SMG

computed

for

an

input

image

to

be

recognized

and

an

SMG

computed

for

a

given

model

object

image

-LRB-

view

-RRB-

we

propose

two

methods

for

computing

their

similarity

In

the

first

method

we

compare

only

the

topological

or

structural

similarity

of

the

graphs

a

weaker

distance

measure

designed

to

support

limited

object

deformation

invariance

In

the

second

method

we

take

advantage

of

the

geometrical

information

encoded

in

an

SMG

and

strengthen

the

similarity

measure

to

ensure

geometric

consistency

a

stronger

distance

measure

designed

to

support

subclass

or

instance

matching

It

is

imperative

that

each

method

support

a

measure

of

subgraph

similarity

in

order

to

support

occluded

object

matching

graph

structure

is

more

compact

than

Crowley

's

Laplacian

pyramid

approach

-LSB-

-RSB-

since

he

represents

every

local

peak

in

his

graph

whereas

we

retain

peaks

only

at

their

most

characteristic

scale

Problem

Formulation

Two

graphs

G

-LRB-

V

E

-RRB-

and

-LRB-

V

-RRB-

are

said

to

be

isomorphic

if

there

exists

a

bijective

mapping

f

V

V

satisfying

for

all

x

y

V

-LRB-

x

y

-RRB-

E

-LRB-

f

-LRB-

x

-RRB-

f

-LRB-

y

-RRB-

-RRB-

To

compute

the

similarity

of

two

SMG

's

we

consider

a

generalization

of

the

graph

isomorphism

problem

which

we

will

call

the

SMG

similarity

problem

Given

two

SMG

's

-LRB-

-RRB-

and

-LRB-

-RRB-

and

a

partial

mapping

from

f

let

E

be

a

real-valued

error

function

defined

on

the

set

of

all

partial

mappings

We

say

that

a

partial

mapping

f

is

feasible

if

f

-LRB-

x

-RRB-

y

implies

that

there

are

parents

px

of

x

and

py

of

y

such

that

f

-LRB-

px

-RRB-

py

Our

goal

is

therefore

to

find

a

feasible

mapping

f

which

minimizes

E

-LRB-

f

-RRB-

Choosing

a

Suitable

Error

Function

The

requirement

of

feasibility

assures

that

the

partial

mapping

preserves

the

path

structure

between

and

It

is

essential

that

the

error

function

E

is

carefully

selected

so

as

to

yield

optimal

results

Our

error

function

incorporates

two

components

with

respect

to

any

partial

mapping

-RRB-

we

would

like

to

reward

corresponding

nodes

which

are

similar

in

terms

of

their

topology

geometry

and

salience

and

-RRB-

we

would

like

to

penalize

a

set

of

correspondences

the

more

they

exclude

nodes

from

the

model

Excluding

the

second

component

our

measure

is

similar

to

the

objective

function

used

in

the

integer

programming

formulation

of

the

largest

subgraph

isomorphism

problem

-LSB-

-RSB-

Given

two

SMG

's

-LRB-

-RRB-

and

-LRB-

-RRB-

and

a

partial

mapping

f

we

define

the

mapping

matrix

M

-LRB-

f

-RRB-

between

and

to

be

a

matrix

as

follows

Mu

v

if

u

v

u

f

-LRB-

v

-RRB-

Otherwise

Since

f

is

a

bijective

mapping

M

-LRB-

f

-RRB-

will

satisfy

the

following

conditions

Mu

v

Mu

v

Given

this

formulation

of

the

mapping

f

we

define

the

error

of

f

to

be

E

-LRB-

f

-RRB-

X

X

Mu

v

-LRB-

u

v

-RRB-

js

-LRB-

u

-RRB-

s

-LRB-

v

-RRB-

j

-LRB-

-RRB-

X

-LRB-

X

Mu

j

-RRB-

s

-LRB-

u

-RRB-

X

-LRB-

X

Mi

v

-RRB-

s

-LRB-

v

-RRB-

A

-LRB-

-RRB-

where

-LRB-

f

-RRB-

represents

the

ratio

of

the

number

of

matched

vertices

to

the

number

of

vertices

-LRB-

-RRB-

in

the

model

SMG

-LRB-

with

the

identity

vector

of

the

appropriate

dimension

-RRB-

and

s

-LRB-

v

-RRB-

denotes

the

strength

of

region

v

in

its

saliency

map

For

the

SMG

topological

similarity

algorithm

defined

in

Section

-LRB-

u

v

-RRB-

is

always

one

while

for

the

SMG

geometrical

similarity

algorithm

defined

in

Section

-LRB-

u

v

-RRB-

represents

the

Euclidean

distance

between

the

centers

of

the

regions

u

and

v.

Clearly

in

the

case

of

perfect

similarity

E

-LRB-

f

-RRB-

while

E

-LRB-

f

-RRB-

will

be

s

-LRB-

u

-RRB-

s

-LRB-

v

-RRB-

if

there

is

no

match

-LRB-

-LRB-

f

-RRB-

-RRB-

A

Matching

Algorithm

Based

on

Topological

Similarity

In

this

section

we

describe

an

algorithm

which

finds

an

approximate

solution

to

the

SMG

similarity

problem

The

focus

of

the

algorithm

is

to

find

a

minimum

weight

matching

between

vertices

of

and

which

lie

in

the

same

level

Our

algorithm

starts

with

the

vertices

at

level

Let

and

be

the

set

of

vertices

at

level

in

and

respectively

We

construct

a

complete

weighted

bipartite

graph

G

-LRB-

E

-RRB-

with

a

weight

function

defined

for

edge

-LRB-

u

v

-RRB-

-LRB-

u

and

v

-RRB-

as

w

-LRB-

u

v

-RRB-

js

-LRB-

v

-RRB-

s

-LRB-

u

-RRB-

j.

Next

we

find

a

maximum

cardinality

minimum

weight

matching

in

G

using

-LSB-

-RSB-

All

the

matched

vertices

are

mapped

to

each

other

that

is

we

define

f

-LRB-

x

-RRB-

y

if

-LRB-

x

y

-RRB-

is

a

matching

edge

in

The

remainder

of

the

algorithm

proceeds

in

phases

as

follows

as

shown

in

Figure

In

phase

i

the

algorithm

considers

the

vertices

of

level

i.

Let

Ai

and

Bi

be

the

set

of

vertices

of

level

i

in

and

respectively

Construct

a

weighted

bipartite

graph

G

-LRB-

Ai

Bi

E

-RRB-

as

follows

-LRB-

v

u

-RRB-

is

an

edge

of

G

if

either

of

the

following

is

true

-LRB-

-RRB-

Both

u

and

v

do

not

have

any

parent

in

and

respectively

or

-LRB-

-RRB-

They

have

at

least

one

matched

parent

-LRB-

A

B

E

-RRB-

is

a

weighted

bipartite

graph

with

weight

matrix

W

-LSB-

wij

-RSB-

of

size

jAj

jBj

if

for

all

edges

of

the

form

-LRB-

i

j

-RRB-

E

i

A

j

B

and

-LRB-

i

j

-RRB-

has

an

associated

weight

wi

j.

nodes

and

are

paired

since

they

have

no

parents

and

they

are

the

trivial

solution

to

the

minimum

weight

maximum

cardinality

bipartite

matching

at

level

level

nodes

and

and

nodes

and

are

paired

since

they

represent

the

best

node

correspondences

in

the

minimum

weight

maximum

cardinality

bipartite

matching

at

level

Had

their

parents

not

been

in

the

solution

to

level

they

would

not

have

been

in

the

level

solution

algorithm

iteratively

descends

to

the

leaves

of

the

shallowest

graph

before

terminating

Figure

Illustration

of

the

SMGBM

Algorithm

-LRB-

see

text

for

explanation

-RRB-

of

depth

less

than

i

that

is

there

is

a

parent

pu

of

u

and

pv

of

v

such

that

-LRB-

pu

pv

-RRB-

Mj

for

some

j

i

We

define

the

weight

of

the

edge

-LRB-

u

v

-RRB-

to

be

js

-LRB-

u

-RRB-

s

-LRB-

v

-RRB-

j

The

algorithm

finds

a

maximum

cardinality

minimum

weight

matching

in

G

and

proceeds

to

the

next

phase

The

above

algorithm

terminates

after

phases

where

is

the

minimum

number

of

scales

in

the

saliency

maps

-LRB-

or

SMG

's

-RRB-

of

two

graphs

The

partial

mapping

M

of

SMG

's

can

be

simply

computed

as

the

union

of

all

Mi

's

for

i

Finally

using

the

error

measure

defined

above

we

compute

the

error

of

the

partial

mapping

M

Each

phase

of

the

algorithm

requires

simple

operations

with

the

time

to

complete

each

phase

being

dominated

by

the

time

to

compute

a

minimum

weight

matching

in

a

bipartite

graph

The

time

complexity

for

finding

such

a

matching

in

a

weighted

bipartite

graph

with

n

vertices

is

O

-LRB-

log

log

n

-RRB-

time

using

the

scaling

algorithm

of

Gabow

Gomans

and

Williamson

-LSB-

-RSB-

The

entire

procedure

as

currently

formulated

requires

O

-LRB-

log

log

n

-RRB-

steps

A

Matching

Algorithm

Based

on

Geometric

Similarity

The

SMGBM

similarity

measure

captured

the

structural

similarity

between

two

SMG

's

in

terms

of

branching

factor

and

node

saliency

similarity

no

geometric

information

encoded

in

the

SMG

was

exploited

In

this

section

we

describe

a

second

similarity

measure

called

SMG

Similarity

using

an

Affine

Transformation

-LRB-

SMGAT

-RRB-

that

includes

the

geometric

properties

-LRB-

e.g.

relative

position

and

orientation

-RRB-

of

the

saliency

regions

Given

-LRB-

-RRB-

and

-LRB-

-RRB-

we

first

assume

without

loss

of

generality

that

First

as

shown

in

Figure

the

algorithm

will

hypothesize

a

correspondence

between

three

regions

of

say

-LRB-

-RRB-

and

three

regions

-LRB-

-RRB-

of

The

mapping

f

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

g

will

be

considered

as

a

basis

for

alignment

if

the

following

conditions

are

satisfied

ffl

ri

and

have

the

same

level

in

the

SMG

's

for

all

i

g.

ffl

-LRB-

ri

rj

-RRB-

if

and

only

if

-LRB-

-RRB-

for

all

i

j

g

which

implies

that

selected

regions

should

have

the

same

adjacency

structure

in

their

respective

SMG

's

Once

regions

-LRB-

-RRB-

and

-LRB-

-RRB-

have

been

selected

we

solve

for

the

affine

transformation

-LRB-

A

b

-RRB-

that

aligns

the

corresponding

region

triples

by

solving

the

following

system

of

linear

inequalities

-LRB-

-RRB-

The

affine

transformation

-LRB-

A

b

-RRB-

will

be

applied

to

all

regions

in

to

form

a

new

graph

Next

a

procedure

similar

to

the

minimum

weight

matching

used

in

the

SMGBM

is

applied

to

Choose

region

triple

correspondence

and

solve

for

affine

transformation

T

that

aligns

the

triples

Apply

affine

transform

T

to

one

graph

aligning

it

with

the

other

T

Find

minimum

weight

mapping

in

bipartite

graph

at

each

level

based

on

Euclidean

distance

Figure

Illustration

of

the

SMGAT

Algorithm

-LRB-

see

text

for

explanation

-RRB-

the

regions

in

graphs

and

Instead

of

matching

regions

which

have

maximum

similarity

in

terms

of

saliency

we

match

regions

which

have

minimum

Euclidean

distance

from

each

other

Given

two

regions

u

and

v

the

distance

between

them

can

be

defined

as

the

norm

of

the

distance

between

their

centers

denoted

by

d

-LRB-

u

v

-RRB-

q

-LRB-

xu

xv

-RRB-

-LRB-

yu

yv

-RRB-

In

a

series

of

steps

SMGAT

constructs

weighted

bipartite

graphs

Gi

-LRB-

Ri

Ei

-RRB-

for

each

level

i

of

the

two

SMG

's

where

Ri

and

represent

the

set

of

vertices

of

and

at

the

i-th

level

respectively

The

constraints

for

having

an

edge

in

Ei

is

the

same

as

SMGBM

-LRB-

u

v

-RRB-

is

an

edge

in

Gi

if

either

of

the

followings

holds

ffl

Both

u

and

v

do

not

have

any

parents

in

and

respectively

ffl

They

have

at

least

one

matched

parent

of

depth

less

than

i

The

corresponding

edge

will

have

weight

equal

to

w

-LRB-

u

v

-RRB-

d

-LRB-

u

v

-RRB-

A

maximum

cardinality

minimum

weight

bipartite

matching

Mi

will

be

found

for

each

level

Gi

and

the

partial

mapping

f

-LRB-

A

b

-RRB-

for

the

affine

transformation

-LRB-

A

b

-RRB-

will

be

formed

as

the

union

of

all

Mi

's

Finally

the

error

of

this

partial

mapping

E

-LRB-

f

-LRB-

A

b

-RRB-

-RRB-

will

be

computed

as

the

sum

over

each

Ei

of

the

Euclidean

distance

separating

Ei

's

nodes

weighted

by

the

nodes

difference

in

saliency

Once

the

total

error

is

computed

the

algorithm

proceeds

to

the

next

valid

pair

of

region

triples

Among

all

valid

affine

transformations

SMGAT

chooses

that

one

which

minimizes

the

error

of

the

partial

mapping

In

terms

of

algorithmic

complexity

solving

for

the

affine

transformation

-LRB-

eq

-RRB-

takes

only

constant

time

while

applying

the

affine

transformation

to

to

form

is

O

-LRB-

max

-LRB-

-RRB-

-RRB-

The

execution

time

for

each

hypothesized

pair

of

region

triples

is

dominated

by

the

complexity

of

establishing

the

bipartite

matching

between

and

which

is

O

-LRB-

log

log

n

-RRB-

for

SMG

's

with

n

vertices

and

scales

Although

in

the

worst

case

i.e.

when

both

saliency

map

graphs

have

only

one

level

there

are

O

-LRB-

-RRB-

pairs

of

triples

However

in

practice

the

vertices

of

an

SMG

are

more

uniformly

distributed

among

the

levels

of

the

graph

greatly

reducing

the

number

of

possible

correspondences

of

base

triples

Although

not

yet

implemented

we

can

reduce

the

complexity

of

the

bipartite

matching

step

by

exploiting

the

fact

that

the

edge

weights

of

the

bipartite

graph

represent

the

Eu

clidean

distance

between

the

regions

of

two

SMG

's

and

satisfy

the

triangle

inequality

The

following

algorithm

can

then

be

used

to

find

the

bipartite

matching

at

the

i-th

level

of

Construct

the

Voronoi

diagram

of

the

vertices

at

the

i-th

level

of

-LRB-

O

-LRB-

jRij

log

jRij

-RRB-

time

-RRB-

Project

the

regions

in

into

the

plane

of

this

Voronoi

diagram

In

each

Voronoi

polytope

choose

the

closest

region

of

to

the

vertex

defining

the

Voronoi

polytope

if

one

exists

-LRB-

O

-LRB-

log

jRij

-RRB-

time

for

each

Voronoi

region

-RRB-

Update

the

Voronoi

diagram

by

removing

the

matched

vertices

and

their

corresponding

regions

in

-LRB-

-LRB-

log

jRij

-RRB-

time

-RRB-

Repeat

this

process

until

either

all

the

vertices

at

the

i-th

level

of

are

matched

or

the

remaining

vertices

can

not

be

matched

-LRB-

due

to

path

constraints

-RRB-

The

time

complexity

of

the

above

procedure

for

level

i

is

O

-LRB-

log

n

-RRB-

-LSB-

-RSB-

For

levels

the

total

complexity

is

O

-LRB-

log

n

-RRB-

which

compares

favorably

to

the

O

-LRB-

log

log

n

-RRB-

complexity

of

our

current

SMGAT

algorithm

Limitations

of

the

Matching

Algorithms

There

are

two

major

limitations

of

both

matching

algorithms

First

since

both

algorithms

seek

a

minimum

weight

maximum

cardinality

matching

in

a

bipartite

graph

that

spans

corresponding

levels

of

two

saliency

map

graphs

corresponding

nodes

in

the

two

graphs

must

therefore

lie

at

the

same

levels

in

their

respective

SMG

's

This

implies

that

a

scene

SMG

can

not

be

vertically

expanded

or

compressed

relative

to

a

model

SMG

Furthermore

an

image

object

that

is

detected

at

a

scale

different

from

that

of

its

corresponding

model

object

can

not

be

correctly

matched

To

overcome

this

problem

consider

a

model

SMG

corresponding

to

a

particular

view

of

some

object

and

let

the

initial

scene

SMG

be

exactly

equal

to

the

model

SMG

Next

consider

a

perturbation

of

the

scene

SMG

in

which

any

scene

SMG

node

can

migrate

up

or

down

a

small

number

of

levels

k

provided

that

the

scene

SMG

topology

remains

intact

i.e.

same

parent-child

relationships

with

parents

and/or

children

changing

levels

For

a

fixed

small

k

the

bandwidth

of

the

bipartite

graph

mapping

solution

will

increase

from

to

In

other

words

the

bipartite

graph

previously

generated

at

each

level

will

now

encompass

nodes

at

neighboring

levels

The

resulting

complexity

of

both

algorithms

will

be

the

same

except

for

a

constant

scaling

factor

to

account

for

the

increased

-LRB-

constant

-RRB-

number

of

nodes

in

each

bipartite

graph

The

restriction

that

corresponding

nodes

lie

at

the

same

level

or

scale

has

an

important

implication

for

matching

cluttered

scenes

If

the

scale

of

background

objects

is

comparable

to

the

object

being

recognized

the

saliency

map

graph

corresponding

to

the

scene

is

approximately

the

saliency

map

corresponding

to

the

object

with

additional

nodes

added

to

one

or

more

levels

In

this

case

our

assumption

that

corresponding

nodes

exist

at

the

same

level

is

not

violated

However

if

a

background

object

dominates

the

object

being

recognized

the

effect

will

be

to

push

the

object

down

to

a

finer

scale

while

the

background

object

occupies

the

coarser

scales

This

migration

of

the

target

object

would

violate

our

assumption

that

corresponding

SMG

nodes

lie

at

the

same

scale

If

we

assume

that

some

model

SMG

consisting

of

levels

occupies

any

continuous

levels

of

a

scene

SMG

then

to

overcome

this

second

problem

requires

that

we

apply

either

algorithm

to

each

level

of

the

scene

SMG

This

will

mean

that

the

complexity

of

both

algorithms

will

increase

by

a

multiplicative

factor

of

h

the

number

of

scales

in

the

scene

SMG

In

the

experiments

reported

in

the

following

section

we

assume

a

bipartite

graph

of

bandwidth

and

assume

that

any

background

objects

do

not

dominate

the

target

object

Experiments

To

illustrate

our

approach

to

shape

representation

and

matching

we

apply

it

to

a

database

of

model

object

views

generated

by

Murase

and

Nayar

at

Columbia

University

Views

of

each

of

the

objects

are

taken

from

a

fixed

elevation

every

degrees

-LRB-

views

per

object

-RRB-

for

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

Figure

A

sample

of

views

from

the

database

top

row

represents

original

images

second

row

represents

saliency

maps

while

third

row

represents

saliency

map

graphs

a

total

of

model

views

The

top

row

of

images

in

Figure

shows

three

adjacent

model

views

for

one

of

the

objects

-LRB-

piggy

bank

-RRB-

plus

one

model

view

for

each

of

two

other

objects

-LRB-

bulb

socket

and

cup

-RRB-

The

second

row

shows

the

computed

saliency

maps

for

each

of

the

five

images

while

the

third

row

shows

the

corresponding

saliency

map

graphs

The

time

to

compute

the

saliency

map

averaged

seconds/image

for

the

five

images

on

a

Sun

Sparc

but

can

be

reduced

to

real-time

on

a

system

with

hardware

support

for

convolution

e.g.

a

Datacube

The

average

time

to

compute

the

distance

between

two

SMG

's

is

ms

using

SMGBM

and

second

using

SMGAT

-LRB-

an

average

of

nodes

per

SMG

-RRB-

Algorithm

-LRB-

a

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

SMGBM

SMGAT

Table

Distance

of

Figure

-LRB-

b

-RRB-

to

other

images

in

Figure

Unoccluded

Scenes

To

illustrate

the

matching

of

an

unoccluded

image

to

the

database

we

compare

the

middle

piggy

bank

image

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

to

the

remaining

images

in

the

database

Table

shows

the

distance

of

the

test

image

to

the

other

images

in

Figure

the

two

other

piggy

bank

images

-LRB-

Figures

-LRB-

a

-RRB-

and

-LRB-

c

-RRB-

-RRB-

were

the

closest

matching

views

in

the

entire

database

Table

also

illustrates

the

difference

between

the

two

matching

algorithms

SMGBM

is

a

weaker

matching

algorithm

searching

for

a

topological

match

between

two

SMG

's

SMGAT

on

the

other

hand

is

more

restrictive

searching

for

a

geometrical

match

between

the

two

SMG

's

For

similar

views

the

two

algorithms

are

comparable

however

as

two

views

diverge

in

appearance

their

similarity

as

computed

by

SMGAT

diverges

more

rapidly

than

their

SMGBM

similarity

In

the

second

experiment

we

choose

three

different

objects

and

repeat

the

procedure

as

shown

in

Figure

In

this

case

each

of

the

views

is

compared

to

the

other

views

leading

to

a

by

table

as

shown

in

Table

Entries

above

the

diagonal

are

computed

using

the

SMGBM

algorithm

while

entries

below

the

diagonal

are

computed

using

the

SMGAT

algorithm

Again

both

algorithms

perform

well

with

the

SMGAT

distance

diverging

more

rapidly

as

the

objects

become

more

dissimilar

In

the

third

experiment

we

compare

every

image

to

every

other

image

in

the

database

resulting

in

over

million

trials

There

are

three

possible

outcomes

-RRB-

the

image

removed

from

the

database

is

closest

to

one

of

its

neighboring

views

of

the

correct

object

-RRB-

the

image

removed

from

the

database

is

closest

to

a

view

belonging

to

the

correct

object

but

not

a

neighboring

view

and

-RRB-

the

image

removed

from

the

database

is

closest

to

a

view

belonging

to

a

different

object

The

results

are

shown

in

Table

As

we

would

expect

the

SMGAT

algorithm

due

to

its

stronger

matching

criterion

outperforms

the

SMGBM

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

Figure

A

second

sample

of

views

for

the

second

experiment

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

Table

Pairwise

distance

between

figures

in

Figure

Entries

above

the

diagonal

are

distances

as

computed

by

SMGBM

while

entries

below

the

diagonal

are

computed

by

SMGAT

algorithm

If

we

include

as

a

correct

match

any

image

belonging

to

the

same

object

both

algorithms

-LRB-

SMGBM

and

SMGAT

-RRB-

perform

extremely

well

yielding

success

rates

of

and

respectively

Occluded

Scenes

To

illustrate

the

matching

of

an

occluded

image

to

the

database

we

compare

an

image

containing

the

piggy

bank

occluded

by

the

bulb

socket

as

shown

in

Figure

Table

shows

the

distance

of

the

test

image

to

the

other

images

in

Figure

The

closest

matching

view

is

the

middle

view

of

the

piggy

back

which

is

in

fact

the

view

embedded

in

the

occluded

scene

In

a

labeling

task

the

subgraph

matching

the

closest

model

view

would

be

removed

from

the

graph

and

the

procedure

applied

to

the

remaining

subgraph

After

removing

the

matching

subgraph

we

match

the

remaining

scene

subgraph

to

the

entire

database

as

shown

Algorithm

Hit

Miss

Miss

right

object

wrong

object

SMGBM

SMGAT

Table

An

exhaustive

test

of

the

two

matching

algorithms

For

each

image

in

the

database

the

image

is

removed

from

the

database

and

compared

using

both

algorithms

to

every

remaining

image

in

the

database

The

closest

matching

image

can

be

either

one

of

its

true

neighboring

views

a

different

view

belonging

to

the

correct

object

or

a

view

belonging

to

a

different

object

Figure

Occluded

Object

Matching

-LRB-

a

-RRB-

original

image

-LRB-

b

-RRB-

saliency

map

and

-LRB-

c

-RRB-

saliency

map

graph

in

Table

In

this

case

the

closest

view

is

the

correct

view

-LRB-

Figure

-LRB-

d

-RRB-

-RRB-

of

the

socket

In

a

second

occlusion

experiment

consider

the

duck

occluding

the

toy

cat

as

shown

in

Figure

The

closest

matching

view

in

the

database

is

the

correct

view

of

the

duck

-LRB-

Figure

-LRB-

a

-RRB-

-RRB-

After

removing

the

scene

SMG

subgraph

corresponding

to

the

duck

the

remaining

subgraph

was

matched

to

the

entire

database

as

shown

in

Table

The

closest

image

is

the

correct

view

-LRB-

Figure

-LRB-

d

-RRB-

-RRB-

of

the

cat

Algorithm

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

SMGBM

SMGAT

Table

Distance

of

Figure

-LRB-

a

-RRB-

to

other

images

in

Figure

The

correct

piggy

bank

view

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

is

the

closest

matching

view

Algorithm

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

SMGBM

SMGAT

Table

Distance

of

Figure

-LRB-

a

-RRB-

-LRB-

after

removing

from

its

SMG

the

subgraph

corresponding

to

the

matched

piggy

back

image

-RRB-

to

other

images

in

Figure

Algorithm

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

SMGBM

SMGAT

Table

Distance

of

Figure

to

other

images

in

Figure

Algorithm

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

SMGBM

SMGAT

Table

Distance

of

Figure

-LRB-

after

removing

from

its

corresponding

SMG

the

subgraph

corresponding

to

the

matched

duck

image

-RRB-

to

the

other

images

in

Figure

Figure

Image

of

occluded

object

used

in

second

occlusion

experiment

An

Analysis

of

Viewpoint

Invariance

In

a

view-based

object

recognition

system

an

object

is

represented

by

a

collection

of

views

The

more

viewpoint-invariant

an

image

representation

is

the

fewer

the

number

of

views

needed

to

represent

the

object

In

the

above

experiments

we

computed

the

saliency

map

graphs

for

the

full

set

of

views

for

each

of

the

objects

In

this

section

we

explore

the

viewpoint

invariance

of

our

representation

by

considering

a

smaller

sample

of

views

for

one

of

our

objects

Our

experiment

as

shown

in

Figure

consists

of

successively

removing

every

second

view

-LRB-

model

SMG

's

-RRB-

of

a

given

object

-LRB-

in

this

case

the

piggy

bank

-RRB-

and

computing

the

distance

using

both

SMGBM

and

SMGAT

between

each

removed

view

to

the

remaining

views

Thus

at

the

first

iteration

we

will

remove

every

second

view

from

the

original

set

of

views

leaving

views

of

the

model

object

Each

of

the

views

that

was

removed

will

then

be

compared

to

each

of

the

remaining

model

views

If

the

closest

matching

model

view

is

adjacent

to

the

removed

view

's

position

in

the

original

set

of

views

then

one

can

argue

that

the

intermediate

view

-LRB-

that

was

removed

-RRB-

is

extraneous

At

the

next

iteration

we

remove

every

second

view

from

the

model

views

and

repeat

the

experiment

with

the

removed

views

The

results

are

shown

in

Table

For

example

when

leaving

out

views

of

the

n

views

removed

at

step

are

maximally

distant

from

the

n

remaining

views

there

is

no

need

to

match

the

views

removed

at

step

to

the

views

remaining

at

step

images

for

next

samplingsearch

images

for

next

samplingsearch

images

for

next

sampling

Figure

An

Experiment

Exploring

the

Viewpoint

Invariance

of

the

SMG

Representation

and

Matching

Algorithms

Views

in

Tree

SMGBM

SMGAT

Table

Evaluating

Viewpoint

Invariance

of

the

SMG

Representation

The

first

row

indicates

the

number

of

model

views

remaining

in

the

model

view

set

for

the

piggy

bank

object

after

removing

every

second

view

The

second

and

third

rows

indicate

the

percentage

of

SMGBM-based

and

SMGAT-based

searches

respectively

between

each

of

the

removed

views

and

the

remaining

model

views

that

result

in

a

closest

view

that

is

adjacent

to

the

removed

view

SMGBM

searches

-LRB-

using

a

removed

view

-RRB-

resulted

in

a

closest

view

that

is

adjacent

to

the

removed

view

at

the

next

level

up

-LRB-

views

-RRB-

while

for

SMGAT

of

the

searches

were

successful

Furthermore

this

percentage

gradually

declines

for

SMGAT

and

rapidly

declines

for

SMGBM

As

one

might

expect

when

geometric

information

is

included

in

the

search

neighboring

views

of

a

test

view

exhibit

the

least

geometric

distortion

For

the

SMGBM

algorithm

however

the

topological

structure

of

a

test

view

may

in

fact

be

similar

to

other

views

of

the

object

despite

geometric

differences

With

the

proper

indexing

structure

it

is

clear

that

in

a

recognition

framework

the

number

of

candidates

returned

from

a

topological

index

will

be

higher

than

that

returned

from

a

geometric

index

given

the

ambiguity

inherent

in

a

topological

index

On

the

other

hand

shape

deformations

within

an

object

's

class

may

be

accommodated

by

SMGBM

and

not

by

SMGAT

Finally

it

must

be

pointed

out

that

the

above

analysis

was

performed

on

only

one

object

Although

we

would

expect

the

same

trend

to

occur

with

other

model

object

view

sets

the

percentages

will

vary

with

the

shape

and

appearance

of

the

object

For

example

for

an

object

with

many

degenerate

views

we

would

expect

the

percentages

to

fall

when

a

sample

lies

directly

on

a

degenerate

view

We

are

currently

conducting

more

comprehensive

experiments

in

order

to

to

predict

what

kind

of

view

sampling

resolutions

are

appropriate

for

each

algorithm

Limitations

of

the

Experiments

The

approach

presented

in

this

paper

has

not

addressed

the

indexing

problem

For

the

experiments

each

query

view

was

compared

to

each

and

every

model

view

to

return

the

closest

matching

view

In

current

work

we

are

exploring

the

use

of

recovered

local

SMG

structure

-LRB-

SMG

subgraphs

covering

local

regions

in

the

image

-RRB-

to

index

into

the

database

of

model

views

and

return

objects

whose

model

view

trees

have

similar

structure

at

their

leaves

In

addition

we

are

exploring

hierarchical

representations

of

the

model

views

corresponding

to

a

given

object

leading

to

a

more

efficient

-LRB-

O

-LRB-

log

n

-RRB-

-RRB-

search

of

an

object

's

model

views

than

the

current

linear

search

The

evaluation

of

our

approach

is

also

limited

in

that

by

using

the

Columbia

University

image

database

we

were

unable

to

change

the

lighting

conditions

scale

etc.

of

the

images

In

future

work

we

plan

to

construct

our

own

image

database

allowing

us

to

more

effectively

evaluate

the

transformation

invariance

of

our

representation

Conclusions

There

is

a

gap

in

the

view-based

object

recognition

literature

between

the

image-based

systems

and

the

feature-based

systems

While

the

image-based

systems

have

been

shown

to

work

with

complex

objects

e.g.

faces

they

are

highly

sensitive

to

occlusion

scale

and

deformation

The

feature-based

systems

on

the

other

hand

rely

on

highly

sensitive

feature

extraction

processes

We

have

introduced

an

image

representation

that

fills

this

gap

Our

saliency

map

graph

offers

a

robust

transformation

invariant

multiscale

representation

of

an

image

that

not

only

captures

the

salient

image

structure

but

provides

the

locality

of

representation

required

to

support

occluded

object

recognition

We

have

presented

two

graph

matching

algorithms

SMGBM

and

SMGAT

that

offer

an

effective

mechanism

for

comparing

the

topological

and

geometric

structure

respectively

of

a

test

image

SMG

and

a

database

image

SMG

Our

graph

matching

formulation

in

terms

of

topological

and

geometric

similarity

is

applicable

to

any

multiscale

image

representation

e.g.

a

Laplacian

pyramid

which

can

be

mapped

to

a

vertex-weighted

directed

acyclic

graph

We

are

not

only

seeking

to

improve

our

saliency

map

construction

but

are

exploring

other

multiscale

image

representations

within

this

framework

We

are

also

embedding

our

matching

algorithms

in

an

object

recognition

system

that

uses

SMG

subgraphs

as

an

indexing

structure

Acknowledgements

We

gratefully

acknowledge

Columbia

University

's

Shree

Nayar

for

providing

us

with

the

database

of

model

views

We

also

gratefully

acknowledge

the

support

of

the

National

Science

Foundation

References

-LSB-

-RSB-

P.

Belhumeur

and

D.

Kriegman

What

is

the

set

of

images

of

an

object

under

all

possible

lighting

conditions

In

IEEE

Conference

on

Computer

Vision

and

Pattern

Recognition

pages

-LCB-

San

Francisco

CA

June

-LSB-

-RSB-

J.

Burns

and

L.

Kitchen

Recognition

in

images

of

objects

from

large

model

bases

using

prediction

hierarchies

In

Proceedings

International

Joint

Conference

on

Artificial

Intelligence

pages

-LCB-

Milan

Italy

-LSB-

-RSB-

P.

J.

Burt

Attention

Mechanisms

for

Vision

in

a

Dynamic

World

In

Proceedings

of

the

International

Conference

on

Pattern

Recognition

Vol

pages

-LCB-

The

Hague

The

Netherlands

-LSB-

-RSB-

J.

Crowley

and

A.

Parker

A

representation

for

shape

based

on

peaks

and

ridges

in

the

difference

of

low-pass

transform

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

J.

L.

Crowley

A

Multiresolution

Representation

for

Shape

In

A.

Rosenfeld

editor

Multiresolution

Image

Processing

and

Analysis

pages

-LCB-

Springer

Verlag

Berlin

-LSB-

-RSB-

J.

L.

Crowley

and

A.

C.

Sanderson

Multiple

Resolution

Representation

and

Probabilistic

Matching

of

-LCB-

D

Gray

-LCB-

Scale

Shape

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

S.

Dickinson

A.

Pentland

and

A.

Rosenfeld

shape

recovery

using

distributed

aspect

matching

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

G.

G.

E.

Mjolsness

and

P.

Anandan

Optimization

in

model

matching

and

perceptual

organization

Neural

Computation

-LCB-

-LSB-

-RSB-

E.

Edmonds

Paths

trees

and

flowers

Canadian

Journal

of

Mathematics

-LCB-

-LSB-

-RSB-

H.

Gabow

M.

Goemans

and

D.

Williamson

An

efficient

approximate

algorithm

for

survivable

network

design

problems

Proc

of

the

Third

MPS

Conference

on

Integer

Programming

and

Combinatorial

Optimization

pages

-LCB-

-LSB-

-RSB-

K.

Ikeuchi

and

T.

Kanade

Automatic

generation

of

object

recognition

programs

Proceedings

of

the

IEEE

-LCB-

-LSB-

-RSB-

M.

J?agersand

Saliency

Maps

and

Attention

Selection

in

Scale

and

Spatial

Coordinates

An

Information

Theoretic

Approach

In

Proceedings

of

the

International

Conference

on

Computer

Vision

pages

-LCB-

Boston

MA

June

-LSB-

-RSB-

J.

Kobler

The

Graph

Isomorphism

Problem

Its

Structural

Complexity

Birkhauser

Boston

-LSB-

-RSB-

J.

Koenderink

and

A.

van

Doorn

The

internal

representation

of

solid

shape

with

respect

to

vision

Biological

Cybernetics

-LCB-

-LSB-

-RSB-

A.

Leonardis

and

H.

Bischoff

Dealing

with

occlusions

in

the

eigenspace

approach

In

Proceedings

IEEE

Conference

on

Computer

Vision

and

Pattern

Recognition

pages

-LCB-

San

Francisco

CA

June

-LSB-

-RSB-

T.

Lindeberg

Detecting

Salient

Blob

-LCB-

Like

Image

Structures

and

Their

Scales

With

a

Scale

-LCB-

Space

Primal

Sketch

A

Method

for

Focus

-LCB-

of

-LCB-

Attention

International

Journal

of

Computer

Vision

-LRB-

-RRB-

-LCB-

December

-LSB-

-RSB-

S.

Mallat

and

W.

L.

Hwang

Singularity

Detection

and

Processing

with

Wavelets

IEEE

Transactions

on

Information

Theory

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

I.

Marsic

Data

-LCB-

Driven

Shifts

of

Attention

in

Wavelet

Scale

Space

Technical

Report

CAIP

-LCB-

TR

-LCB-

CAIP

Center

Rutgers

University

Piscataway

NJ

September

-LSB-

-RSB-

H.

Murase

and

S.

Nayar

Visual

learning

and

recognition

of

objects

from

appearance

International

Journal

of

Computer

Vision

-LCB-

-LSB-

-RSB-

B.

Olshausen

C.

Anderson

and

D.

V.

Essen

A

Neurobiological

Model

of

Visual

Attention

and

Invariant

Pattern

Recognition

Based

on

Dynamic

Routing

of

Information

Journal

of

Neurosciences

-LRB-

-RRB-

-LCB-

November

-LSB-

-RSB-

A.

Pope

and

D.

Lowe

Learning

object

recognition

models

from

images

In

Proceedings

IEEE

International

Conference

on

Computer

Vision

pages

-LCB-

Berlin

May

-LSB-

-RSB-

F.

Preparata

and

M.

Shamos

Computational

Geometry

Springer-Verlag

New

York

NY

-LSB-

-RSB-

R.

P.

N.

Rao

G.

J.

Zelinsky

M.

M.

Hayhoe

and

D.

H.

Ballard

Modeling

Saccadic

Targeting

in

Visual

Search

In

D.

Touretzky

M.

Mozer

and

M.

Hasselmo

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

MIT

Press

Cambridge

MA

-LSB-

-RSB-

C.

Schmid

and

R.

Mohr

Combining

greyvalue

invariants

with

local

constraints

for

object

recognition

In

Proceedings

IEEE

Conference

on

Computer

Vision

and

Pattern

Recognition

pages

-LCB-

San

Francisco

CA

June

-LSB-

-RSB-

E.

Simoncelli

W.

Freeman

E.

Adelson

and

D.

Heeger

Shiftable

multi-scale

transforms

IEEE

Transactions

on

Information

Theory

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

E.

P.

Simoncelli

W.

T.

Freeman

E.

H.

Adelson

and

D.

J.

Heeger

Shiftable

Multiscale

Transforms

IEEE

Transactions

on

Information

Theory

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

B.

Taka?cs

and

H.

Wechsler

A

Dynamic

and

Multiresolution

Model

of

Visual

Attention

and

Its

Application

to

Facial

Landmark

Detection

Computer

Vision

and

Image

Understanding

-LRB-

in

press

-RRB-

-LSB-

-RSB-

J.

K.

Tsotsos

An

Inhibitory

Beam

for

Attentional

Selection

In

L.

Harris

and

M.

Jenkin

editors

Spatial

Vision

in

Humans

and

Robots

pages

-LCB-

Cambridge

University

Press

Cambridge

UK

-LSB-

-RSB-

M.

Turk

and

A.

Pentland

Eigenfaces

for

recognition

Journal

of

Cognitive

Neuroscience

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

S.

Ullman

and

R.

Basri

Recognition

by

linear

combinations

of

models

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

-LCB-

October

shape representation and recovery
view-based object recognition