Shape

from

Rotation

Richard

Szeliski

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

December

d

Digital

Equipment

Corporation

has

four

research

facilities

the

Systems

Research

Center

and

the

Western

Research

Laboratory

both

in

Palo

Alto

California

the

Paris

Research

Laboratory

in

Paris

and

the

Cambridge

Research

Laboratory

in

Cambridge

Massachusetts

The

Cambridge

laboratory

became

operational

in

and

is

located

at

One

Kendall

Square

near

MIT

CRL

engages

in

computing

research

to

extend

the

state

of

the

computing

art

in

areas

likely

to

be

important

to

Digital

and

its

customers

in

future

years

CRL

's

main

focus

is

applica

tions

technology

that

is

the

creation

of

knowledge

and

tools

useful

for

the

preparation

of

impor

tant

classes

of

applications

CRL

Technical

Reports

can

be

ordered

by

electronic

mail

To

receive

instructions

send

a

mes

sage

to

one

of

the

following

addresses

with

the

word

help

in

the

Subject

line

On

Digital

's

EASYnet

CRL

TECHREPORTS

On

the

Internet

techreports@crl.dec.com

This

work

may

not

be

copied

or

reproduced

for

any

commercial

purpose

Permission

to

copy

without

payment

is

granted

for

non?profit

educational

and

research

purposes

provided

all

such

copies

include

a

notice

that

such

copy

ing

is

by

permission

of

the

Cambridge

Research

Lab

of

Digital

Equipment

Corporation

an

acknowledgment

of

the

authors

to

the

work

and

all

applicable

portions

of

the

copyright

notice

The

Digital

logo

is

a

trademark

of

Digital

Equipment

Corporation

Cambridge

Research

Laboratory

One

Kendall

Square

Cambridge

Massachusetts

dT

Shape

from

Rotation

Richard

Szeliski

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

December

Abstract

This

paper

examines

the

construction

of

a

surface

model

of

an

object

rotating

in

front

of

a

camera

Previous

research

in

depth

from

motion

has

demonstrated

the

power

of

using

an

incremental

approach

to

depth

estimation

In

this

paper

we

extend

this

approach

to

more

general

motion

and

use

a

full

surface

model

instead

of

a

D

sketch

The

algorithm

starts

with

a

flow

field

computed

using

local

correlation

It

then

projects

individual

measurements

into

points

with

associated

uncertainties

Nearby

points

from

successive

frames

are

merged

to

improve

the

position

estimates

These

points

are

then

used

to

construct

a

finite

element

surface

model

which

is

itself

refined

over

time

We

demonstrate

the

application

of

our

new

techniques

to

several

real

image

sequences

c

Digital

Equipment

Corporation

All

rights

reserved

Contents

i

Contents

Introduction

Previous

work

Framework

Optical

flow

Constrained

flow

and

depth

recovery

Incremental

estimation

-LRB-

points

-RRB-

Local

surface

fitting

Experimental

results

Discussion

Conclusions

A

Inverse

perspective

projection

with

homogeneous

coordinates

ii

Contents

Introduction

Introduction

This

paper

examines

the

construction

of

a

surface

model

from

image

sequences

of

an

object

rotating

in

front

of

a

stationary

camera

Because

the

motion

of

the

object

between

frames

is

known

we

can

use

traditional

depth

from

motion

techniques

to

directly

recover

the

depth

of

points

in

the

image

Our

approach

uses

a

large

number

of

images

where

the

motion

between

successive

images

is

small

This

makes

it

much

easier

to

compute

flow

-LRB-

the

stereo

correspondence

problem

is

avoided

-RRB-

but

makes

individual

flow

measurements

much

less

reliable

To

compensate

for

this

we

use

an

incremental

estimation

algorithm

to

integrate

measurements

from

successive

frames

and

reduce

the

uncertainty

over

time

The

incremental

approach

to

depth

estimation

was

previously

developed

by

Matthies

et

al.

-LSB-

-RSB-

In

this

paper

we

extend

their

work

to

true

surface

models

A

simpler

method

for

creating

such

models

is

to

use

the

object

silhouettes

to

carve

out

a

bounding

volume

for

the

model

-LRB-

this

method

is

presented

in

a

companion

report

-LSB-

Szeliski

-RSB-

-RRB-

However

to

obtain

a

more

detailed

description

we

need

to

use

the

optic

flow

of

the

texture

marks

to

give

us

a

dense

estimate

of

surface

shape

Our

new

shape

from

rotation

algorithm

builds

such

a

model

and

also

provides

us

with

a

framework

within

which

we

can

explore

a

number

of

important

issues

in

computer

vision

These

include

flow

estimation

uncertainty

modeling

incremental

estimation

surface

representation

and

reconstruction

and

massively

parallel

algorithms

In

addition

to

being

an

interesting

research

topic

the

automatic

acquisition

of

object

models

is

important

in

many

applications

These

include

robotics

manipulation

where

the

object

must

first

be

described

and/or

recognized

before

it

can

be

manipulated

Computer

Aided

Design

-LRB-

CAD

-RRB-

where

automatic

model

building

can

be

used

as

an

input

stage

to

the

CAD

system

and

computer

graphics

animation

or

virtual

reality

where

it

facilitates

the

task

of

an

animator

allowing

him

easy

access

to

a

large

catalog

of

real-world

objects

All

of

these

applications

become

much

more

interesting

if

the

acquisition

can

be

performed

Introduction

quickly

and

without

the

need

for

special

equipment

or

environments

Our

aim

is

to

build

such

a

system

by

using

the

motion

of

the

turntable

and

object

to

provide

most

of

the

system

calibration

automatically

Because

we

also

intend

our

system

to

eventually

run

in

real-time

finding

efficient

parallelizable

algorithms

will

be

important

Previous

work

Some

of

the

early

work

in

object

motion

estimation

-LSB-

Hallam

Broida

and

Chellappa

Rives

et

al.

Matthies

and

Kanade

-RSB-

identified

Kalman

filtering

as

a

viable

framework

for

incremental

estimation

because

it

incorporates

representations

of

uncertainty

and

provides

a

mechanism

for

incrementally

reducing

uncertainty

over

time

Applied

to

depth

from

motion

this

framework

was

at

first

restricted

to

estimating

the

positions

of

a

sparse

set

of

trackable

features

such

as

points

or

line

segments

-LSB-

Faugeras

et

al.

Matthies

and

Shafer

-RSB-

-LRB-

see

also

-LSB-

Ullman

-RSB-

for

an

incremental

approach

to

the

related

structure

from

motion

problem

-RRB-

Another

line

of

work

addressed

the

problem

of

extracting

denser

depth

or

displacement

estimates

from

image

sequences

-LRB-

Figure

-RRB-

However

these

approaches

either

were

restricted

to

two

frame

analysis

-LSB-

Horn

and

Schunck

Anandan

-RSB-

or

used

batch

processing

of

the

image

sequence

for

example

via

line

fitting

-LSB-

Bolles

et

al.

Baker

and

Bolles

-RSB-

or

spatio-temporal

filtering

-LSB-

Heeger

-RSB-

The

work

of

-LSB-

Matthies

et

al.

-RSB-

overcame

these

limitations

by

combining

a

recursive

estimation

procedure

with

dense

flow

measurement

This

work

has

recently

been

extended

to

more

general

motion

by

Heel

-LSB-

-RSB-

Because

the

camera

motion

in

-LSB-

Matthies

et

al.

-RSB-

was

a

pure

translation

perpendicular

to

the

line

of

sight

the

resulting

image

flow

was

always

one-dimensional

Under

more

general

camera

motion

the

image

flow

is

two-dimensional

and

has

a

spatially

varying

uncertainty

which

can

be

characterized

using

either

a

two-dimensional

confidence

measure

-LSB-

Anandan

-RSB-

or

a

covariance

matrix

-LSB-

Szeliski

-RSB-

When

the

camera

motion

Previous

work

a

b

Figure

Spatio-temporal

image

sequence

data

-LRB-

a

-RRB-

first

image

in

frame

sequence

-LRB-

b

-RRB-

horizontal

slice

through

spatio-temporal

cube

The

average

inter-frame

rotation

is

ffi

is

known

each

flow

measurement

from

the

image

can

be

converted

into

a

position

estimate

in

the

scene

and

an

associated

uncertainty

-LRB-

covariance

-RRB-

matrix

can

be

computed

As

we

will

show

in

this

paper

these

measurements

can

be

integrated

over

time

-LRB-

along

with

the

intensity

value

associated

with

each

points

-RRB-

and

surfaces

can

be

fitted

to

these

points

shape

modeling

has

long

been

one

of

the

fundamental

research

areas

in

computer

vision

The

problem

is

to

come

up

with

representations

that

are

sophisticated

enough

to

model

interesting

objects

yet

simple

enough

to

permit

recognition

or

construction

from

images

or

other

sensor

data

A

variety

of

surface-based

models

have

been

proposed

including

generalized

cylinders

-LSB-

Brooks

et

al.

-RSB-

superquadrics

-LSB-

Pentland

-RSB-

and

deformable

finite-element

models

-LSB-

Terzopoulos

et

al.

-RSB-

Volumetric

models

such

as

octrees

-LSB-

Jackins

and

Tanimoto

Meagher

-RSB-

have

also

been

used

One

popular

approach

to

constructing

such

volumes

has

been

to

intersect

multiple

silhouettes

of

the

Introduction

object

seen

from

different

views

-LRB-

see

-LSB-

Chen

and

Huang

Szeliski

-RSB-

for

a

review

-RRB-

In

this

paper

we

will

use

locally

parametrized

deformable

surface

models

Our

long-term

goal

is

to

build

higher-level

-LRB-

parts

-RRB-

descriptions

from

these

surfaces

The

study

of

incremental

shape

from

rotation

is

becoming

particularly

interesting

because

of

the

dramatic

increase

in

computer

processing

speed

both

through

the

availability

of

massively

parallel

architectures

-LSB-

Hillis

-RSB-

and

the

appearance

of

fast

RISC

microprocessors

-LSB-

Hennessy

and

Patterson

-RSB-

Eventually

many

of

the

low-level

processing

algorithms

used

in

our

research

could

be

implemented

using

analog

processing

-LSB-

Koch

et

al.

Hutchinson

et

al.

-RSB-

One

of

the

focuses

of

our

research

is

the

use

of

fine-grained

parallel

algorithms

-LSB-

Poggio

et

al.

Little

et

al.

-RSB-

However

unlike

much

of

the

current

research

in

low-level

vision?which

embeds

the

computation

in

a

plane

of

processors?our

models

will

require

more

complex

representations

and

processor

topologies

Framework

The

shape

from

rotation

algorithm

developed

in

this

paper

converts

a

series

of

images

into

a

model

of

the

object

whose

accuracy

improves

with

time

The

initial

estimates

of

the

object

's

shape

are

crude

because

the

object

motion

between

successive

image

pairs

is

small

Fortunately

modeling

the

uncertainty

in

these

estimates

allows

us

to

refine

them

as

more

images

are

seen

Since

we

wish

to

build

a

full

model

we

can

not

just

forget

a

part

of

the

surface

when

it

becomes

occluded

Therefore

a

simple

D

depth

map

such

as

was

used

in

-LSB-

Matthies

et

al.

-RSB-

is

not

an

adequate

representation

On

the

other

hand

as

the

object

continues

to

rotate

we

will

see

each

view

more

than

once

so

it

is

not

necessary

to

make

optimal

use

of

the

information

in

each

image

Before

we

begin

our

surface

model

construction

we

use

a

preprocessing

stage

to

calibrate

the

system

and

to

adapt

to

the

background

The

camera

parameters

-LRB-

relative

to

the

Framework

turntable

-RRB-

are

determined

by

imaging

a

known

reference

model

such

as

a

calibration

cube

-LRB-

Figure

a

-RRB-

We

use

a

binary

Gray

code

painted

on

the

rim

of

the

turntable

to

automatically

determine

its

rotation

angle

without

any

additional

sensors

-LRB-

Figures

and

-RRB-

These

steps

are

described

in

a

companion

report

-LSB-

Szeliski

-RSB-

along

with

an

algorithm

for

computing

a

bounding

volume

for

the

object

from

its

silhouettes

-LRB-

Figure

-RRB-

The

actual

shape

from

rotation

algorithm

operates

in

the

following

stages

First

the

optical

flow

between

successive

image

pairs

is

extracted

over

the

whole

image

-LRB-

Section

-RRB-

The

correlation

surface

corresponding

to

the

Sum

of

Squared

Differences

-LRB-

SSD

-RRB-

measure

is

used

to

compute

both

the

best

flow

estimate

at

each

point

and

its

uncertainty

Next

using

the

known

object

motion

we

project

this

flow

into

a

position

measurement

with

an

associated

uncertainty

at

each

point

-LRB-

Section

-RRB-

This

cloud

of

intensitytagged

depth

values

is

then

refined

by

merging

nearby

points

from

successive

frames

whose

uncertainties

overlap

sufficiently

-LRB-

Section

-RRB-

A

locally

parametrized

surface

is

then

fitted

to

this

collection

of

points

-LRB-

Section

-RRB-

This

stage

reduces

the

noise

in

nearby

measurements

-LRB-

using

a

regularization-based

weak

smoothness

constraint

-LSB-

Poggio

et

al.

-RSB-

-RRB-

and

fills

in

the

data

where

there

is

unreliable

flow

information

-LRB-

e.g.

in

areas

of

uniform

intensity

-RRB-

The

surface

model

along

with

its

associated

intensities

are

then

refined

as

more

images

are

acquired

In

Section

we

present

some

experiments

with

real

image

sequences

acquired

in

our

lab

In

Section

we

compare

our

approach

with

alternative

shape

acquisition

techniques

and

we

suggest

a

number

of

extensions

to

our

work

including

higher-level

surface

models

and

the

merging

of

multiple

object

poses

Introduction

a

b

c

d

Figure

Image

adaptation

thresholding

and

bounding

volume

computation

A

calibration

cube

-LRB-

a

-RRB-

is

used

to

compute

the

camera

parameters

during

a

setup

phase

Next

the

system

is

adapted

to

a

blank

turntable

to

locate

the

Gray-code

position

encoding

ring

and

to

memorize

the

background

Each

image

-LRB-

b

-RRB-

is

then

thresholded

yielding

an

object/background

-LRB-

black/white

-RRB-

segmentation

-LRB-

c

-RRB-

The

Gray-code

ring

is

used

to

compute

the

turntable

angle

An

incremental

octree

construction

algorithm

is

used

to

produce

the

bounding

volume

-LRB-

d

-RRB-

See

-LSB-

Szeliski

-RSB-

for

details

Optical

flow

Optical

flow

Given

two

or

more

images

we

can

compute

a

two-dimensional

vector

field

called

the

optic

flow

which

measures

the

interframe

motion

of

each

pixel

in

the

image

A

number

of

different

algorithms

have

been

developed

previously

for

extracting

the

optic

flow

In

this

paper

we

use

a

variant

of

correlation

called

the

Sum

of

Squared

Differences

-LRB-

SSD

-RRB-

measure

-LSB-

Anandan

-RSB-

since

it

provides

us

not

only

with

flow

estimates

but

also

with

uncertainty

estimates

for

each

measurement

Alternative

approaches

to

computing

optic

flow

include

gradient-based

techniques

-LSB-

Horn

and

Schunck

Lucas

Nagel

-RSB-

spatiotemporal

filtering

-LSB-

Adelson

and

Bergen

Heeger

Fleet

and

Jepson

-RSB-

and

direct

depth

estimation

-LSB-

Heel

-RSB-

-LRB-

see

-LSB-

Nagel

Anandan

-RSB-

for

a

comparison

of

several

of

these

techniques

-RRB-

The

Sum

of

Squared

Differences

method

integrates

the

squared

intensity

difference

between

two

shifted

images

over

a

small

area

to

obtain

an

error

measure

et

-LRB-

u

v

x

y

-RRB-

Z

Z

w

-LRB-

-RRB-

-LSB-

ft

-LRB-

x

y

-RRB-

ft

-LRB-

x

u

y

v

-RRB-

-RSB-

d

d

-LRB-

-RRB-

where

ft

-LRB-

x

y

-RRB-

and

ft

-LRB-

x

y

-RRB-

are

the

two

successive

image

frames

and

w

-LRB-

x

y

-RRB-

is

a

windowing

function

The

SSD

flow

estimator

selects

at

each

pixel

-LRB-

x

y

-RRB-

the

flow

-LRB-

u

v

-RRB-

which

minimizes

the

SSD

measure

In

Anandan

's

algorithm

a

coarse-to-fine

technique

is

used

to

limit

the

range

of

possible

flow

values

In

our

shape

from

rotation

work

a

single-resolution

algorithm

is

used

since

the

range

of

possible

motions

is

small

The

error

surface

et

-LRB-

u

v

x

y

-RRB-

can

be

used

not

only

to

determine

the

best

displacement

estimate

-LRB-

u

v

-RRB-

but

also

to

determine

the

confidence

in

this

estimate

Anandan

and

Weiss

-LSB-

-RSB-

observed

that

the

shape

of

the

error

surface

differs

depending

on

whether

both

one

or

of

the

displacement

components

are

uniquely

computable

-LRB-

corresponding

to

an

intensity

corner

an

edge

or

a

homogeneous

area

-RRB-

They

proposed

a

method

for

computing

the

confidence

measures

based

on

the

principal

curvatures

and

the

directions

of

the

principal

Optical

flow

axes

in

the

vicinity

of

the

error

surface

minimum

Matthies

et

al.

-LSB-

-RSB-

showed

how

for

a

one-dimensional

displacement

the

variance

in

the

displacement

estimate

can

be

computed

from

the

second

derivative

of

a

parabola

fit

to

the

error

curve

This

result

was

extended

to

two

dimensions

in

-LSB-

Szeliski

-RSB-

thus

providing

a

statistical

justification

for

the

heuristics

developed

by

Anandan

and

Weiss

The

derivation

in

-LSB-

Szeliski

-RSB-

involves

modeling

the

two

image

frames

ft

and

ft

as

displaced

versions

of

the

same

image

corrupted

with

additive

white

Gaussian

noise

with

variance

A

quadratic

of

the

form

-LRB-

u

v

x

y

-RRB-

u

u

v

v

A

u

u

v

v

c

-LRB-

-RRB-

is

fitted

to

the

error

surface

defined

by

-LRB-

-RRB-

by

finding

the

values

of

A

u

v

and

c

which

minimize

the

weighted

least

squared

error

from

the

measured

e

-LRB-

u

v

x

y

-RRB-

values

We

then

set

the

disparity

estimate

at

-LRB-

x

y

-RRB-

to

-LRB-

u

v

-RRB-

and

set

the

variance

of

this

measurement

to

This

simple

model

does

not

account

for

occlusions

disparity

gradients

or

other

optical

effects

It

is

thus

only

valid

over

small

windows

and

breaks

down

in

certain

areas

such

as

at

occlusion

boundaries

In

the

context

of

shape

from

rotation

we

expect

the

flow

estimates

to

be

most

reliable

when

a

surface

point

is

locally

translating

in

front

of

the

camera

and

less

reliable

as

it

recedes

and

disappears

-LRB-

because

of

excessive

warping

and

occlusion

effects

-RRB-

The

analysis

presented

in

-LSB-

Szeliski

-RSB-

can

also

be

used

to

derive

the

correlation

between

adjacent

flow

estimates

and

between

flow

estimates

obtained

from

successive

frames

To

help

differentiate

between

pixels

which

are

part

of

the

object

and

those

in

the

background

it

may

be

useful

to

distinguish

valid

flow

measurements

on

the

object

's

surface

from

all

other

measurements

A

very

simple

approach

to

this

problem

is

to

use

the

rotation

almost

every

image

patch

is

warping

-LRB-

undergoing

a

non-translation

affine

transformation

-RRB-

at

every

instant

However

the

amount

of

this

warping

is

usually

very

small

if

the

images

are

tightly

spaced

in

time

Constrained

flow

and

depth

recovery

background

values

before

the

object

was

placed

on

the

turntable

to

threshold

the

image

into

foreground

and

background

regions

-LSB-

Szeliski

-RSB-

This

approach

will

often

fail

however

due

to

effects

such

as

shadows

specularities

and

nearby

object/background

gray

levels

Another

approach

is

to

detect

areas

with

zero

optical

flow

by

computing

the

SSD

measure

for

-LRB-

u

v

-RRB-

-LRB-

-RRB-

and

classifying

the

pixel

as

background

if

this

is

smaller

than

any

other

SSD

value

This

test

may

fail

to

find

some

background

pixels

-LRB-

because

of

imaging

noise

-RRB-

and

may

erroneously

classify

some

object

pixels

as

background

pixels

either

in

homogeneous

areas

or

at

points

where

the

motion

is

purely

vertical

-LRB-

points

lying

on

a

plane

parallel

to

the

image

plane

passing

through

the

rotation

axis

-RRB-

The

latter

kind

of

error

is

fairly

harmless

since

we

do

not

require

or

even

expect

a

truly

dense

estimate

of

flow

over

the

whole

objects

-LRB-

e.g.

areas

of

constant

intensity

will

always

yield

little

or

no

information

-RRB-

Two

additional

indicators

for

suspect

flow

values

suggested

by

Anandan

-LSB-

-RSB-

are

a

high

value

for

the

minimum

of

et

-LRB-

u

v

x

y

-RRB-

and

a

difference

in

shape

between

et

-LRB-

u

v

x

y

-RRB-

and

the

image

autocorrelation

at

-LRB-

x

y

-RRB-

In

practice

we

have

found

it

unnecessary

to

explicitly

compute

regions

of

zero

or

bad

flow

since

we

can

use

the

temporal

integration

phase

-LRB-

Section

-RRB-

to

discard

erroneous

measurements

Constrained

flow

and

depth

recovery

The

general

flow

estimator

described

in

the

previous

section

is

a

useful

first

step

in

determining

shape

from

motion

when

the

object

motion

-LRB-

egomotion

-RRB-

is

unknown

In

shape

from

rotation

however

we

know

the

angular

position

of

the

turntable

in

each

frame

and

therefore

the

relative

motion

of

the

object

-LRB-

or

equivalently

of

the

camera

-RRB-

This

makes

the

problem

of

depth

recovery

easier

and

obviates

the

need

for

additional

assumptions

such

incremental

rigidity

-LSB-

Ullman

-RSB-

Using

the

known

motion

we

can

compute

for

each

pixel

a

constraint

line

for

the

flow

at

that

point

with

the

actual

-LRB-

ideal

-RRB-

flow

observed

Constrained

flow

and

depth

recovery

Z

P

P

Pq

X

Y

object

coordinates

ffi

Y

@RX

Z

camera

coordinates

P

screen

coordinates

ffi

y

@Rx

z

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Xz

t

p

t

q

Figure

Object

camera

and

screen

coordinates

depending

only

on

the

depth

of

the

surface

at

that

pixel

Furthermore

we

can

compute

a

bounded

segment

for

each

flow

constraint

line

from

the

minimum

and

maximum

expected

depth

values

-LRB-

e.g.

from

a

bounding

volume

or

cylinder

-RRB-

The

simplest

way

to

compute

these

constraint

lines

is

to

use

homogeneous

coordinates

-LSB-

Newman

and

Sproull

-RSB-

Given

a

point

in

object-space

p

-LRB-

X

Y

Z

-RRB-

we

can

convert

it

to

screen

coordinates

q

-LRB-

x

y

z

-RRB-

using

a

linear

matrix

transform

M

followed

by

a

projection

operation

P

-LRB-

Figure

-RRB-

First

we

multiply

p

by

M

to

obtain

the

camera-based

coordinates

Mp

X

Y

Z

W

-LRB-

-RRB-

The

transformation

matrix

M

encodes

all

of

the

information

about

the

perspective

and

screen

transformations

such

as

the

focal

length

and

the

aspect

ratio

Next

we

use

the

parameter-free

projection

operator

P

to

compute

q

P

W

X

W

Y

W

Z

W

q

-LRB-

-RRB-

Constrained

flow

and

depth

recovery

The

above

equations

describe

the

forward

projection

from

object

space

to

screen

-LRB-

image

-RRB-

space

If

we

know

the

depth

buffer

value

z

for

a

given

pixel

-LRB-

x

y

-RRB-

we

can

recover

its

location

using

backprojection

-LRB-

Appendix

A

-RRB-

p

P

M

-LRB-

-RRB-

Computing

the

flow

constraint

line

for

each

pixel

is

therefore

straightforward

From

our

knowledge

of

the

camera

calibration

and

turntable

rotation

we

can

precompute

the

projection

matrices

Mt

and

Mt

for

the

previous

and

current

frames

A

pixel

in

the

current

frame

qt

should

appear

at

qt

P

Mt

M

t

qt

P

Mt

t

qt

-LRB-

-RRB-

in

the

previous

frame

-LRB-

Appendix

A

-RRB-

Of

course

for

each

pixel

we

do

not

know

the

correct

value

of

zt

but

we

can

project

the

minimum

and

maximum

expected

depth

values

z?t

and

z

t

-LRB-

e.g.

from

the

depths

at

the

front

and

back

of

the

turntable

-RRB-

We

therefore

obtain

two

endpoints

-LRB-

x?t

y?t

z?t

-RRB-

and

-LRB-

x

t

y

t

z

t

-RRB-

for

the

segment

describing

the

expected

previous

point

position

This

constrains

the

possible

flow

values

to

lie

on

a

line

between

-LRB-

u

v

-RRB-

-LRB-

xt

x?t

yt

y?t

-RRB-

and

-LRB-

u

v

-RRB-

-LRB-

xt

x

t

yt

y

t

-RRB-

Figure

shows

a

set

of

flow

constraint

segments

computed

for

the

standard

imaging

setup

shown

in

Figure

and

a

rotation

of

the

turntable

Notice

how

the

flow

is

generally

upward

at

the

right

edge

of

the

image

and

downward

at

the

left

edge

This

is

as

expected

for

a

scene

spinning

counterclockwise

in

front

of

the

camera

Notice

also

how

the

flow

constraint

lines

in

a

given

row

line

up

almost

perfectly

This

effect

is

even

more

pronounced

for

the

smaller

rotations

-LRB-

ffi

to

ffi

-RRB-

which

we

use

in

practice

In

the

case

of

general

motion

the

flow

constraint

line

at

each

pixel

defines

the

-LRB-

u

v

-RRB-

values

along

which

e

-LRB-

u

v

x

y

-RRB-

should

be

searched

for

a

minimum

For

our

particular

imaging

setup?with

the

vertical

axes

of

the

camera

and

turntable

aligned

and

small

inter-frame

displacements?we

can

use

a

near-epipolar

line

constraint

to

further

reduce

the

Constrained

flow

and

depth

recovery

Figure

Flow

constraint

segments

for

rotation

The

dot

indicates

the

pixel

center

-LRB-

subsampled

every

pixel

-RRB-

and

the

two

ends

of

the

tee

indicate

the

minimum

and

maximum

expected

flow

values

Constrained

flow

and

depth

recovery

computational

complexity

of

our

algorithm

and

to

make

it

more

regular

For

each

pixel

in

a

given

row

of

the

current

image

the

pixel

corresponding

to

a

zero

horizontal

displacement

-LRB-

v

-RRB-

is

extracted

from

the

previous

image

thus

forming

the

approximate

epipolar

The

two

rows

are

then

passed

to

a

flow

extraction

algorithm

similar

to

that

used

in

-LSB-

Matthies

et

al.

-RSB-

which

we

describe

below

The

flow

extraction

algorithm

we

use

is

designed

to

compute

the

flow

estimate

u

to

subpixel

-LRB-

floating-point

-RRB-

precision

and

a

confidence

-LRB-

variance

-RRB-

estimate

for

this

measurement

Each

row

is

first

interpolated

by

a

factor

of

r

using

a

Hermite

cubic

interpolator

-LSB-

Szeliski

and

Ito

-RSB-

resulting

in

a

smoother

error

surface

at

each

point

For

each

horizontal

displacement

in

the

range

-LSB-

u

u

-RSB-

-LRB-

in

r

steps

-RRB-

the

discrete

squared

difference

measure

is

computed

e

-LRB-

u

x

-RRB-

b

-LRB-

r

-RRB-

X

k

b?r

-LSB-

gt

-LRB-

rx

k

-RRB-

gt

-LRB-

r

-LRB-

x

u

-RRB-

k

-RRB-

-RSB-

where

gt

-LRB-

x

-RRB-

and

gt

-LRB-

x

-RRB-

are

the

interpolated

rows

The

weighted

summation

over

a

square

patch

is

implemented

using

iterated

two-dimensional

box

filtering

-LSB-

Burt

-RSB-

e

-LRB-

i

-RRB-

-LRB-

u

x

y

-RRB-

k

l

e

-LRB-

i

-RRB-

-LRB-

u

x

k

y

l

-RRB-

This

gives

us

a

discrete

approximation

to

the

SSD

measure

at

each

pixel

To

extract

the

horizontal

component

of

the

flow

at

each

pixel

we

find

the

discrete

value

umin

which

minimizes

e

-LRB-

u

x

y

-RRB-

A

parabola

fit

to

the

three

points

e

-LRB-

umin

x

y

-RRB-

e

-LRB-

umin

x

y

-RRB-

and

e

-LRB-

umin

x

y

-RRB-

e

-LRB-

u

x

y

-RRB-

a

-LRB-

u

umin

-RRB-

b

-LRB-

u

umin

-RRB-

c

-LRB-

-RRB-

-LRB-

Figure

-RRB-

is

used

to

compute

the

sub-pixel

flow

estimate

u

umin

b

-LRB-

-RRB-

we

know

the

motion

between

the

two

frames

i.e.

we

know

the

relative

orientation

-LSB-

Horn

-RSB-

of

the

two

cameras

we

could

instead

use

the

standard

epipolar

geometry

to

find

the

set

of

corresponding

epipolar

lines

in

the

two

images

-LSB-

Bolles

et

al.

-RSB-

Constrained

flow

and

depth

recovery

umin

umin

umin

u

c

u

e

-LRB-

u

x

y

-RRB-

d

d

d

d

d

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

Figure

Parabolic

fit

to

SSD

error

surface

The

large

circles

indicate

discrete

values

of

e

-LRB-

u

x

y

-RRB-

the

dotted

line

is

the

parabola

fit

to

the

three

lowest

values

and

its

variance

a

-LRB-

-RRB-

where

is

the

variance

of

the

image

noise

-LSB-

Matthies

et

al.

-RSB-

The

image

noise

can

be

estimated

locally

using

c

which

has

the

advantage

of

increasing

the

flow

variance

estimate

in

regions

with

a

large

minimum

SSD

value

Once

the

flow

estimate

has

been

determined

from

the

image

pair

we

can

compute

the

current

screen

depth

zt

by

linear

interpolation

zt

z?t

z

t

z?t

u

u

-LRB-

u

u

-RRB-

-LRB-

-RRB-

and

the

variance

in

this

estimate

from

z

t

z?t

u

u

-LRB-

-RRB-

This

gives

us

a

dense

estimate

of

depth

and

uncertainty

at

each

pixel

in

the

image

At

this

point

we

could

throw

away

the

measurements

corresponding

to

background

points

Constrained

flow

and

depth

recovery

occlusion

boundaries

and

homogeneous

areas

by

thresholding

on

the

variance

We

could

also

try

to

reduce

the

noise

in

the

depth

measurements

by

using

regularization-based

smoothing

as

was

done

in

-LSB-

Matthies

et

al.

-RSB-

In

our

current

experiments

we

are

able

to

obtain

good

results

without

the

use

of

either

background

point

removal

or

image-based

smoothing

It

remains

to

be

seen

if

these

additional

steps

would

improve

the

quality

of

our

estimates

To

convert

these

screen-based

measurements

qt

-LRB-

xt

yt

zt

-RRB-

into

object

space

locations

pt

-LRB-

Xt

Yt

Zt

-RRB-

we

use

backprojection

pt

P

M

t

qt

This

gives

us

a

collection

of

points

in

consistent

with

the

flow

measurements

we

computed

For

each

point

we

also

need

to

compute

a

covariance

matrix

Cpt

D

-LRB-

pt

pt

-RRB-

-LRB-

pt

pt

-RRB-

T

E

which

characterizes

the

shape

and

magnitude

of

the

point

's

positional

uncertainty

Computing

this

covariance

matrix

is

tricky

since

the

projection

operator

is

non-linear

If

the

covariance

in

the

original

measurement

Cqt

is

sufficiently

small

we

can

use

the

approximation

Cpt

@b

@qt

Cqt

@b

@qt

T

-LRB-

-RRB-

where

b

-LRB-

q

-RRB-

P

M

t

q

is

the

backprojection

operator

-LRB-

the

Jacobian

@b

@qt

can

be

decomposed

into

a

gradient

of

the

projection

operator

times

the

inverse

transform

matrix

M

t

-RRB-

In

the

above

formula

we

set

the

positional

uncertainty

in

x

and

y

to

some

small

value

-LRB-

for

example

-LRB-

pixel

-RRB-

-RRB-

Incremental

estimation

-LRB-

points

-RRB-

A

simpler

approach

which

we

used

in

our

experiments

is

to

backproject

the

original

point

plus

one

standard

deviation

q?t

-LRB-

xt

yt

zt

oezt

-RRB-

to

get

the

vector

rt

p?t

pt

P

M

t

q?t

P

M

t

qt

-LRB-

-RRB-

This

vector

is

the

major

axis

of

the

covariance

ellipsoid

The

other

two

axes

of

the

ellipsoid

st

and

tt

can

be

chosen

arbitrarily

and

their

length

-LRB-

standard

deviation

-RRB-

set

to

a

suitably

chosen

constant

value

-LRB-

say

corresponding

to

the

size

of

a

pixel

projected

into

the

middle

of

the

object

-RRB-

We

can

then

form

the

covariance

matrix

using

Cpt

RtRTt

with

Rt

rt

st

tt

-LRB-

-RRB-

Note

that

since

Cpt

can

be

derived

from

rt

and

it

is

sufficient

to

keep

a

list

of

f

-LRB-

pt

rt

-RRB-

g

vector

pairs

to

fully

describe

the

locations

and

uncertainties

of

the

points

computed

from

the

current

optic

flow

field

Incremental

estimation

-LRB-

points

-RRB-

The

result

of

our

two-frame

optic

flow

analysis

and

backprojection

into

object

space

gives

us

a

cloud

of

uncertainty-tagged

points

lying

on

the

surface

of

the

object

-LRB-

each

point

also

carries

along

with

it

the

intensity

of

the

associated

-RRB-

As

the

object

continues

to

rotate

and

more

points

are

acquired

point

collections

from

successive

frames

must

be

merged

in

order

to

reduce

the

noise

in

point

location

estimates

Our

collection

of

surface

points

is

a

less

restrictive

representation

than

the

previously

used

depth

map

representation

-LSB-

Matthies

et

al.

Heel

-RSB-

which

would

not

allow

us

to

build

a

full

model

since

it

is

univalued

at

each

image

pixel

To

represent

the

position

of

the

points

we

use

an

object-centered

coordinate

reference

frame

rather

than

a

camera-centered

frame

The

origin

of

this

frame

is

fixed

to

the

theory

we

could

estimate

the

covariance

between

the

intensity

and

the

point

location

-LRB-

x

y

-RRB-

from

the

local

gradient

Incremental

estimation

-LRB-

points

-RRB-

screen

-LRB-

x

y

-RRB-

A

A

A

A

A

A

A

A

A

z

oez

ss

ss

s

Figure

Merging

uncertainty

ellipses

top

of

the

turntable

and

rotates

with

it

-LRB-

Figure

-RRB-

This

makes

the

estimates

of

position

much

more

reliable

especially

when

information

is

being

integrated

over

multiple

frames

-LSB-

Tomasi

and

Kanade

-RSB-

The

question

of

how

and

when

to

merge

neighboring

points

from

different

frames

is

in

general

quite

difficult

We

start

by

using

an

uncertainty-weighted

distance

measure

dij

-LRB-

pi

pj

-RRB-

T

-LRB-

C

i

C

j

-RRB-

-LRB-

pi

pj

-RRB-

-LRB-

-RRB-

If

this

distance

is

sufficiently

small

we

can

merge

the

two

points

and

replace

them

with

a

single

measurement

pk

Ck

-LRB-

C

i

pi

C

j

pj

-RRB-

-LRB-

-RRB-

with

a

reduced

uncertainty

Ck

-LRB-

C

i

C

j

-RRB-

-LRB-

-RRB-

The

problem

with

this

approach

is

that

there

may

be

many

candidate

matches

for

a

given

point

especially

if

one

elongated

uncertainty

ellipsoid

overlaps

several

other

points

-LRB-

whose

own

ellipsoids

are

distinct

-RRB-

Consider

for

example

the

points

in

the

upper

right

of

Figure

The

points

with

the

nearly

horizontal

uncertainty

ellipses

were

measured

much

earlier

and

the

rotation

of

the

turntable

has

rotated

their

ellipses

It

is

unclear

with

which

point

-LRB-

s

-RRB-

the

new

measurement

-LRB-

with

the

vertical

uncertainty

ellipse

-RRB-

should

be

merged

It

is

better

to

inhibit

merging

in

this

case

since

we

can

not

determine

which

match

is

correct

Local

surface

fitting

A

simpler

and

more

conservative

combination

rule

is

to

limit

merges

to

points

whose

uncertainty

ellipsoid

major

axes

are

nearly

parallel

and

which

also

meet

the

previous

distance

criteria

-LRB-

middle

of

Figure

-RRB-

In

this

case

it

is

much

easier

to

determine

which

of

the

nearby

points

is

the

best

candidate

for

a

merge

In

practice

we

make

the

merging

step

even

simpler

by

re-projecting

the

locations

and

their

uncertainties

into

the

camera

image

plane

-LRB-

-LRB-

x

y

z

-RRB-

and

oez

in

Figure

-RRB-

Two

points

are

merged

if

their

image

plane

centers

lie

within

a

small

distance

of

each

other

-LRB-

say

pixel

-RRB-

and

their

depths

overlap

sufficiently

-LRB-

using

a

version

of

the

uncertainty-weighted

distance

-RRB-

The

thresholds

for

merging

points

are

set

high

enough

so

that

neighboring

measurements

from

the

same

frame

are

not

merged

-LRB-

we

want

our

final

model

to

be

at

least

as

accurate

as

the

input

image

-RRB-

but

low

enough

so

that

oversampling

-LRB-

the

density

of

points

per

image

pixel

-RRB-

is

not

too

great

This

simplified

framework

has

two

additional

advantages

First

the

image

plane

can

be

used

as

a

natural

binning

structure

to

group

nearby

points

together

for

merging

Second

we

can

continue

to

use

the

f

-LRB-

pt

rt

-RRB-

g

-LRB-

location

uncertainty

-RRB-

representation

for

all

of

the

points

What

we

give

up

in

this

case

is

the

ability

to

increase

the

resolution

in

the

point

locations

orthogonal

to

rt

over

time

-LRB-

e.g.

if

the

points

in

the

upper

right

of

Figure

had

been

merged

the

uncertainty

would

be

small

in

all

directions

-RRB-

This

is

not

a

problem

however

because

our

surface

interpolation

stage

will

smooth

the

surface

and

further

reduce

the

positional

uncertainty

Local

surface

fitting

Once

the

point

estimates

acquired

from

multiple

frames

have

been

integrated

sufficiently

to

make

them

reliable

we

can

start

building

a

surface

model

This

model

serves

both

to

reduce

the

noise

in

the

position

estimates

-LRB-

through

smoothing

-RRB-

and

to

fill-in

areas

on

the

object

surface

where

no

reliable

flow

information

is

available

The

surface

model

Local

surface

fitting

Figure

A

surface

model

This

surface

can

either

be

described

using

a

finite

element

model

or

using

a

spring-mass

system

The

behaviors

of

the

two

models

are

similar

not

only

provides

us

with

a

detailed

description

of

the

object

's

shape

but

also

tells

us

the

intensity

-LRB-

albedo

-RRB-

of

each

point

on

the

surface

-LRB-

ignoring

for

now

the

variation

of

shading

with

object

orientation

-RRB-

The

surface

model

which

we

use

is

a

finite

element

model

i.e.

a

collection

of

nodal

variables

roughly

corresponding

to

the

set

of

position

measurements

This

model

can

be

viewed

as

either

a

true

surface

model

composed

of

polygonal

facets

or

simply

as

a

neighborhood

graph

defined

over

the

nodal

variables

-LRB-

Figure

-RRB-

In

either

case

we

start

with

the

position

measurements

and

add

or

remove

points

to

obtain

a

smooth

and

continuous

surface

Each

point

has

a

list

of

neighbors

which

can

be

chosen

either

by

finding

the

closest

neighbors

or

by

using

the

original

topological

relationship

between

the

pixels

that

generated

these

points

Generating

a

complete

mesh

for

the

surface

from

a

sparse

and

scattered

collection

of

points

is

in

general

quite

difficult

For

example

we

could

use

Sha'ashua

's

-LSB-

-RSB-

Structural

Local

surface

fitting

Z

P

P

Pq

X

Y

t

-LRB-

xi

Ri

-RRB-

B

B

BBM

z

x

y

t

-LRB-

xj

Rj

-RRB-

z

P

P

Pq

x

y

rij

Figure

Oriented

particle

system

with

global

and

local

coordinate

frames

Saliency

theory

but

this

requires

a

dense

-LRB-

in

this

case

-RRB-

network

of

points

which

would

be

computationally

prohibitive

A

simpler

solution

is

to

allow

surface

points

to

move

into

gaps

in

the

surface

However

we

have

to

be

careful

not

to

fill

across

true

holes

in

the

model

such

as

the

handle

in

a

cup

-LRB-

here

the

bounding

volume

computed

by

-LSB-

Szeliski

-RSB-

would

be

useful

-RRB-

Another

possibility

is

to

use

the

points

on

the

surface

of

the

bounding

volume

as

candidates

for

mesh

points

To

circumvent

these

difficulties

we

have

developed

a

new

surface

interpolation

model

based

on

interacting

oriented

particles

-LSB-

Szeliski

and

Tonnesen

-RSB-

These

particles

which

represent

local

surface

patches

have

energy

functions

which

favor

the

alignment

of

tangent

planes

of

neighboring

particles

thus

endowing

the

surface

with

an

elastic

resistance

to

bending

The

particles

also

have

a

preferred

inter-particle

spacing

distance

which

encourages

a

uniform

sample

density

over

the

surface

Each

particle

is

represented

by

state

variables

for

position

and

for

orientation

-LRB-

Figure

-RRB-

This

is

similar

to

the

Darboux

frames

used

by

Sander

and

Zucker

-LSB-

-RSB-

except

that

no

local

curvature

information

is

kept

Within

each

particle

's

local

coordinate

frame

the

energy

function

defining

its

interactions

with

other

particles

is

Eij

-LRB-

-RRB-

exp

where

Experimental

results

and

qij

-LRB-

x

y

z

-RRB-

is

the

local

coordinate

of

particle

j

in

particle

i

's

coordinate

frame

qij

R

i

-LRB-

pj

pi

-RRB-

where

Ri

denotes

the

orientation

of

particle

pi

In

addition

to

the

inter-particle

smoothness

forces

we

use

external

forces

to

attract

surface

particles

to

the

original

sparse

data

-LSB-

Szeliski

-RSB-

Once

a

reasonably

accurate

surface

model

has

been

constructed

we

can

dispense

with

the

optic

flow

computation

altogether

As

each

new

image

arrives

it

directly

modifies

the

deformable

surface

model

and

its

associated

intensities

by

making

small

local

changes

which

better

register

the

model

and

the

image

The

data

constraint

energy

between

the

surface

model

and

the

sparse

data

points

is

therefore

replaced

with

a

direct

intensity

matching

energy

EI

Z

f

-LRB-

x

-LRB-

u

v

-RRB-

y

-LRB-

u

v

-RRB-

-RRB-

I

-LRB-

u

v

-RRB-

-LRB-

x

y

-RRB-

-LRB-

u

v

-RRB-

du

dv

-LRB-

-RRB-

where

f

-LRB-

x

y

-RRB-

is

the

new

image

x

-LRB-

u

v

-RRB-

and

y

-LRB-

u

v

-RRB-

are

the

projected

screen

coordinates

of

the

surface

model

and

I

-LRB-

u

v

-RRB-

is

its

intensity

Experimental

results

We

have

performed

a

number

of

experiments

with

our

shape

from

rotation

algorithms

on

both

live

and

off-line

-LRB-

canned

-RRB-

image

sequences

The

experimental

setup

consists

of

a

spring-wound

microwave

turntable

with

a

position

encoding

grid

taped

to

its

side

-LRB-

Figure

-RRB-

and

a

stationary

camera

mounted

on

a

tripod

-LRB-

General

Imaging

Video

Camera

with

a

Fujinon

TV

zoom

lens

-RRB-

A

rough

calibration

of

the

intrinsic

and

extrinsic

camera

parameters

can

be

obtained

by

locating

the

ellipse

that

defines

the

turntable

top

and

measuring

the

camera

to

turntable

distance

A

more

exact

calibration

can

be

obtained

using

multiple

images

of

a

calibration

cube

-LSB-

Szeliski

-RSB-

-LRB-

Figure

-RRB-

Experimental

results

The

live

experiments

involve

building

an

octree

bounding

volume

of

the

object

processing

a

monochrome

image

every

seconds

on

a

RISC-based

workstation

-LSB-

Szeliski

-RSB-

The

algorithm

is

first

adapted

to

the

empty

turntable

while

it

is

spinning

both

to

memorize

the

background

and

to

locate

the

position

encoding

ring

After

the

object

is

placed

on

the

table

each

new

image

is

then

thresholded

and

the

turntable

angle

computed

from

the

binary

codes

averaged

over

columns

-LRB-

accurate

to

about

ffi

-RRB-

The

bounding

volume

is

then

computed

from

the

object

silhouettes

-LRB-

Figures

d

-RRB-

For

the

off-line

experiments

we

first

recorded

onto

videotape

a

number

of

image

sequences

of

different

objects

spinning

on

the

turntable

-LRB-

Figures

-RRB-

We

then

digitized

each

sequence

using

the

single-frame

playback

capabilities

of

our

video

recorder

to

obtain

a

high

resolution

image

sequence

of

about

frames

-LRB-

about

ffi

rotation

between

frames

-RRB-

For

the

experiments

presented

in

this

paper

each

image

was

subsampled

from

to

with

only

every

second

frame

being

used

The

resulting

interframe

rotation

is

about

ffi

with

a

maximum

horizontal

flow

-LRB-

on

the

turntable

edge

-RRB-

of

about

pixels

These

image

sequences

were

input

into

our

optic

flow

extraction

algorithm

whose

output

was

then

backprojected

into

world

coordinates

Figures

and

show

three

of

the

image

sequences

we

are

using

and

the

results

of

these

initial

depth

extraction

stages

The

first

image

-LRB-

a

-RRB-

in

each

figure

shows

the

first

frame

of

the

input

intensity

image

sequence

The

second

image

-LRB-

b

-RRB-

shows

an

intensity-coded

depth

map

extracted

from

the

first

pair

of

images

where

each

local

flow

estimate

has

been

converted

to

a

screen-based

depth

value

z

-LRB-

depth

values

with

high

uncertainty

are

not

shown

-RRB-

The

third

image

-LRB-

c

-RRB-

shows

the

inverse

variance

-LRB-

certainty

-RRB-

at

each

pixel

This

certainty

is

much

higher

in

textured

areas

and

near

strong

intensity

gradients

After

eliminating

the

estimates

whose

variance

is

too

large

we

project

the

depth

estimates

into

position

estimates

These

are

shown

in

the

fourth

part

-LRB-

d

-RRB-

of

each

figure

using

a

top

view

of

the

object

to

better

see

its

structure

-LRB-

the

wireframe

cube

and

axes

are

for

reference

only

-RRB-

Both

the

circular

structure

of

the

turntable

edge

and

the

rectangular

structure

of

the

tea

box

-LRB-

Figure

-RRB-

and

Experimental

results

the

domino

cube

-LRB-

Figure

-RRB-

are

roughly

recovered

The

structure

of

the

dodecahedron

-LRB-

Figure

-RRB-

is

more

difficult

to

see

from

the

top

view

The

next

step

in

the

shape

from

rotation

algorithm

consists

of

merging

neighboring

points

acquired

from

different

viewpoints

Figures

and

show

the

results

of

this

merging

step

operating

incrementally

on

the

complete

image

sequences

We

present

this

data

as

isolated

points

shown

in

different

projections

top

front

side

and

oblique

Unfortunately

it

is

somewhat

difficult

to

gauge

the

true

shape

of

the

object

using

these

flat

two-dimensional

projections

In

our

own

experiments

we

can

rotate

the

object

interactively

to

get

a

good

sense

of

the

depth

and

relationship

of

the

points

-LRB-

kinetic

depth

effect

-RRB-

We

also

use

multiple

colors

to

display

points

from

different

iterations

-LRB-

when

examining

the

merging

step

-RRB-

From

these

figures

we

can

see

that

the

overall

shape

of

the

objects

is

recovered

well

although

the

exact

surface

data

is

not

very

smooth

Adding

a

small

amount

of

image-plane

smoothing

should

help

to

reduce

this

effect

-LSB-

Matthies

et

al.

-RSB-

Of

course

once

a

complete

surface

model

is

fit

to

this

sparse

data

the

resulting

solution

will

also

be

smooth

Figures

d

show

that

in

some

cases

shadows

will

be

incorporated

into

the

object

model

-LRB-

under

many

imaging

and

lighting

geometries

the

shadows

form

non-rigid

structures

and

are

therefore

discarded

by

the

temporal

integration

stage

-RRB-

To

remove

these

shadow

points

we

could

either

cut

off

the

bottom

of

the

model

or

use

a

more

sophisticated

color-based

image

preprocessing

stage

More

detailed

examination

of

the

data

reveals

that

even

after

integration

over

many

frames

the

depth

estimates

near

horizontal

edges

are

still

poor

This

suggests

a

bias

in

our

flow

estimator

which

could

be

reduced

either

by

using

more

temporal

averaging

-LRB-

antialiasing

-RRB-

using

using

spatio-temporal

filters

-LSB-

Fleet

and

Jepson

-RSB-

or

using

variable-sized

windows

-LSB-

Okutomi

and

Kanade

-RSB-

Experimental

results

a

b

c

d

Figure

Flow

computed

from

assam

image

series

-LRB-

a

-RRB-

first

image

in

sequence

-LRB-

b

-RRB-

depth

map

from

flow

-LRB-

darker

is

nearer

-RRB-

-LRB-

c

-RRB-

certainty

in

depth

estimates

oe

z

-LRB-

darker

is

higher

certainty

-RRB-

-LRB-

d

-RRB-

top

view

of

point

cloud

Experimental

results

a

b

c

d

Figure

Flow

computed

from

dodecahedron

image

series

-LRB-

a

-RRB-

first

image

in

sequence

-LRB-

b

-RRB-

depth

map

from

flow

-LRB-

darker

is

nearer

-RRB-

-LRB-

c

-RRB-

certainty

in

depth

estimates

oe

z

-LRB-

darker

is

higher

certainty

-RRB-

-LRB-

d

-RRB-

top

view

of

point

cloud

Experimental

results

a

b

c

d

Figure

Flow

computed

from

domino

image

series

-LRB-

a

-RRB-

first

image

in

sequence

-LRB-

b

-RRB-

depth

map

from

flow

-LRB-

darker

is

nearer

-RRB-

-LRB-

c

-RRB-

certainty

in

depth

estimates

oe

z

-LRB-

darker

is

higher

certainty

-RRB-

-LRB-

d

-RRB-

top

view

of

point

cloud

Experimental

results

a

b

c

d

Figure

Final

merged

data

from

assam

image

series

-LRB-

a

-RRB-

top

view

-LRB-

b

-RRB-

oblique

view

-LRB-

c

-RRB-

front

view

-LRB-

d

-RRB-

side

view

a

b

c

d

Figure

Final

merged

data

from

dodecahedron

image

series

-LRB-

a

-RRB-

top

view

-LRB-

b

-RRB-

oblique

view

-LRB-

c

-RRB-

front

view

-LRB-

d

-RRB-

side

view

a

b

c

d

Figure

Final

merged

data

from

domino

image

series

-LRB-

a

-RRB-

top

view

-LRB-

b

-RRB-

oblique

view

-LRB-

c

-RRB-

front

view

-LRB-

d

-RRB-

side

view

Discussion

Discussion

The

techniques

we

have

described

in

this

paper

perform

a

shape

construction

task

similar

to

that

usually

associated

with

active

range

sensors

-LSB-

Agin

and

Binford

Woodham

Besl

and

Jain

-RSB-

An

example

of

such

a

sensor

which

can

be

used

in

short-distance

indoor

environments

is

structured

light

where

an

encoded

light

pattern

falling

on

the

object

is

used

to

give

direct

-LRB-

and

usually

sparse

-RRB-

measurements

of

depth

-LSB-

Agin

and

Binford

Vuylsteke

and

Oosterlinck

-RSB-

Compared

to

active

range

sensors

our

approach

requires

a

far

less

structured

environment

since

no

special

lighting

sources

are

required

and

the

calibration

of

the

system

is

simple

and

fairly

automatic

Our

technique

also

has

the

potential

for

better

accuracy

since

our

measurements

are

dense

-LRB-

at

least

in

textured

areas

-RRB-

and

because

we

see

more

views

of

the

object

On

the

other

hand

our

flow-based

approach

will

fail

in

areas

where

the

surface

has

a

uniform

albedo

An

experimental

comparison

of

these

two

techniques

should

be

performed

to

better

quantify

these

effects

Combining

structured

light

with

the

shape-from-rotation

paradigm

should

result

in

an

algorithm

that

works

under

a

wider

variety

of

conditions

but

at

the

expense

of

increased

hardware

complexity

An

alternative

to

the

approach

presented

in

this

paper

is

to

use

the

silhouette

of

the

object

in

each

frame

to

construct

a

bounding

volume

for

the

object

-LSB-

Szeliski

-RSB-

This

bounding

volume

can

provide

a

non-linear

-LRB-

inequality

-RRB-

constraint

on

the

position

of

surface

points

Tracking

the

silhouettes

through

three

or

more

images

can

also

be

used

to

estimate

the

location

and

curvature

of

points

on

the

limb

of

the

object

-LSB-

Giblin

and

Weiss

Vaillant

Cipolla

and

Blake

-RSB-

Combining

silhouette-based

and

flow-based

approaches

should

yield

an

algorithm

that

works

for

a

much

wider

variety

of

object

shapes

and

textures

Our

shape

from

rotation

algorithm

would

be

even

more

useful

if

we

could

change

the

position

of

the

camera

and/or

the

object

The

former

case

is

easier

to

handle

we

simply

recalibrate

the

system

and

continue

processing

with

the

new

camera

parameters

The

change

Discussion

in

object

orientation

caused

by

repositioning

it

on

the

turntable

is

also

simple

to

handle

if

the

new

object

pose

is

known

Determining

this

new

pose

from

the

surface

data

itself

is

more

difficult

A

three-dimensional

generalization

of

an

existing

sparse

range

matching

algorithm

-LSB-

Szeliski

-RSB-

should

help

us

here

Additional

visual

cues

for

reconstructing

the

object

's

shape

should

be

added

to

our

algorithm

to

make

it

more

generally

applicable

and

more

robust

Shape

from

shading

-LSB-

Horn

and

Brooks

-RSB-

is

a

particularly

interesting

cue

since

it

provides

information

that

is

usually

complementary

to

optic

flow

-LRB-

it

works

best

in

uniform

albedo

areas

-RRB-

Specularities

-LSB-

Klinker

et

al.

-RSB-

which

create

difficulties

for

the

current

algorithm

would

also

be

a

powerful

cue

for

shape

computation

-LSB-

Healey

and

Binford

-RSB-

To

determine

the

lighting

characteristics

of

our

environment

we

could

use

a

reflective

sphere

or

cube

placed

on

the

turntable

during

our

calibration

phase

While

shape

from

shading

and

shape

from

specularities

could

be

applied

to

a

single

image

at

a

time

they

may

prove

to

be

even

more

powerful

when

applied

to

a

dynamic

sequence

The

algorithm

described

in

this

paper

builds

a

detailed

locally

parameterized

surface

model

of

the

object

The

next

step

in

processing

would

be

to

build

a

higher-level

description

of

the

object

either

for

more

efficient

CAD/graphics

manipulation

or

for

object

recognition

An

example

of

such

a

model

would

be

a

superquadrics

parts

model

which

could

be

fitted

directly

to

our

sparse

collection

of

points

-LSB-

Pentland

-RSB-

However

if

the

part

model

does

not

fit

the

data

well

we

may

wish

to

use

something

in

between

a

finite

element

deformable

model

and

a

globally

parameterized

model

for

example

a

deformable

superquadric

-LSB-

Terzopoulos

and

Metaxas

-RSB-

This

kind

of

model

could

also

snap

into

a

variety

of

preferred

shapes

such

as

cylinders

or

boxes

-LSB-

Terzopoulos

and

Metaxas

-RSB-

Conclusions

Conclusions

Shape

from

rotation

is

a

practical

approach

to

building

models

from

a

sequence

of

images

The

goal

of

this

work

is

to

produce

a

locally

accurate

model

of

shape

and

intensity

of

an

unknown

object

which

could

later

be

used

to

build

a

high-level

parts

description

As

such

this

technique

should

be

useful

in

a

variety

of

robotics

and

CAD

tasks

as

well

as

providing

a

novel

source

of

objects

for

computer

animation

systems

The

design

of

our

algorithm

is

motivated

both

by

the

increasing

availability

of

massively

parallel

architectures

for

computer

vision

tasks

and

the

recent

success

of

incremental

algorithms

in

building

high-quality

depth

maps

from

motion

sequences

This

work

can

be

viewed

as

an

extension

of

this

recent

work

in

D

incremental

depth

estimation

to

full

shape

reconstruction

The

design

of

a

complete

shape

from

rotation

system

requires

the

solution

of

a

number

of

fundamental

computer

vision

problems

These

include

flow

estimation

uncertainty

modeling

incremental

estimation

surface

representation

and

reconstruction

deformable

-LRB-

energy-based

-RRB-

models

and

massively

parallel

algorithms

We

have

implemented

and

tested

the

main

stages

of

processing

-LRB-

flow

constraints

flow

estimation

backprojection

into

and

point

merging

-RRB-

but

much

interesting

work

remains

to

be

done

-LRB-

surface

reconstruction

and

refinement

evaluation

and

enhancements

-RRB-

We

expect

that

shape

from

rotation

will

prove

to

be

an

interesting

and

challenging

problem

to

solve

as

well

as

a

good

framework

for

studying

various

important

computer

vision

techniques

References

-LSB-

Adelson

and

Bergen

-RSB-

E.

H.

Adelson

and

J.

R.

Bergen

Spatiotemporal

energy

models

for

the

perception

of

motion

Journal

of

the

Optical

Society

of

America

A

-LRB-

-RRB-

February

Conclusions

-LSB-

Agin

and

Binford

-RSB-

G.

J.

Agin

and

T.

O.

Binford

Computer

description

of

curved

objects

IEEE

Transactions

on

Computers

-LRB-

-RRB-

April

-LSB-

Anandan

-RSB-

P.

Anandan

A

computational

framework

and

an

algorithm

for

the

measurement

of

visual

motion

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Anandan

and

Weiss

-RSB-

P.

Anandan

and

R.

Weiss

Introducing

a

smoothness

constraint

in

a

matching

approach

for

the

computation

of

displacement

fields

In

Image

Understanding

Workshop

pages

Science

Applications

International

Corporation

Miami

Beach

Florida

December

-LSB-

Baker

and

Bolles

-RSB-

H.

H.

Baker

and

R.

C.

Bolles

Generalizing

epipolar-plane

image

analysis

on

the

spatiotemporal

surface

International

Journal

of

Computer

Vision

-LRB-

-RRB-

-LSB-

Besl

and

Jain

-RSB-

P.

J.

Besl

and

R.

C.

Jain

Three-dimensional

object

recognition

Computing

Surveys

-LRB-

-RRB-

March

-LSB-

Bolles

et

al.

-RSB-

R.

C.

Bolles

H.

H.

Baker

and

D.

H.

Marimont

Epipolar-plane

image

analysis

an

approach

to

determining

structure

from

motion

International

Journal

of

Computer

Vision

-LSB-

Broida

and

Chellappa

-RSB-

T.

J.

Broida

and

R.

Chellappa

Kinematics

of

a

rigid

object

from

a

sequence

of

noisy

images

a

batch

approach

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Press

Miami

Beach

Florida

June

-LSB-

Brooks

et

al.

-RSB-

R.

A.

Brooks

R.

Greiner

and

T.

O.

Binford

The

ACRONYM

modelbased

vision

system

In

Sixth

International

Joint

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

pages

Tokyo

Japan

August

-LSB-

Burt

-RSB-

P.

J.

Burt

Fast

filter

transforms

for

image

processing

Computer

Graphics

and

Image

Processing

Conclusions

-LSB-

Chen

and

Huang

-RSB-

H.

H.

Chen

and

T.

S.

Huang

A

survey

of

construction

and

manipulation

of

octrees

Computer

Vision

Graphics

and

Image

Processing

-LSB-

Cipolla

and

Blake

-RSB-

R.

Cipolla

and

A.

Blake

The

dynamic

analysis

of

apparent

contours

In

Third

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Osaka

Japan

December

-LSB-

Faugeras

et

al.

-RSB-

O.

D.

Faugeras

N.

Ayache

and

B.

Faverjon

Building

visual

maps

by

combining

noisy

stereo

measurements

In

IEEE

International

Conference

on

Robotics

and

Automation

pages

IEEE

Computer

Society

Press

San

Francisco

California

April

-LSB-

Fleet

and

Jepson

-RSB-

D.

Fleet

and

A.

Jepson

Computation

of

normal

velocity

from

local

phase

information

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Press

San

Diego

California

June

-LSB-

Giblin

and

Weiss

-RSB-

P.

Giblin

and

R.

Weiss

Reconstruction

of

surfaces

from

profiles

In

First

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

London

England

June

-LSB-

Hallam

-RSB-

J.

Hallam

Resolving

observer

motion

by

object

tracking

In

International

Joint

Conference

on

Artificial

Intelligence

-LSB-

Healey

and

Binford

-RSB-

G.

Healey

and

T.

O.

Binford

Local

shape

from

specularity

In

First

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

London

England

June

-LSB-

Heeger

-RSB-

D.

J.

Heeger

Optical

flow

from

spatiotemporal

filters

In

First

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

London

England

June

-LSB-

Heel

-RSB-

J.

Heel

Direct

Estimation

of

Structure

and

Motion

from

Multiple

Frames

A.

Conclusions

I.

Memo

Artificial

Intelligence

Laboratory

Massachusetts

Institute

of

Technology

March

-LSB-

Hennessy

and

Patterson

-RSB-

J.

L.

Hennessy

and

D.

A.

Patterson

Computer

Architecture

A

Quantitative

Approach

Morgan

Kaufmann

Publishers

Los

Altos

California

-LSB-

Hillis

-RSB-

W.

D.

Hillis

The

Connection

Machine

MIT

Press

Cambridge

Massachusetts

-LSB-

Horn

-RSB-

B.

K.

P.

Horn

Relative

orientation

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Horn

and

Brooks

-RSB-

B.

K.

P.

Horn

and

M.

J.

Brooks

Shape

from

Shading

MIT

Press

Cambridge

Massachusetts

-LSB-

Horn

and

Schunck

-RSB-

B.

K.

P.

Horn

and

B.

G.

Schunck

Determining

optical

flow

Artificial

Intelligence

-LSB-

Hutchinson

et

al.

-RSB-

J.

Hutchinson

C.

Koch

J.

Luo

and

C.

Mead

Computing

motion

using

analog

and

binary

resistive

networks

Computer

-LRB-

-RRB-

March

-LSB-

Jackins

and

Tanimoto

-RSB-

C.

L.

Jackins

and

S.

L.

Tanimoto

Oct-trees

and

their

use

in

representing

three-dimensional

objects

Computer

Graphics

and

Image

Processing

-LSB-

Klinker

et

al.

-RSB-

G.

J.

Klinker

S.

A.

Shafer

and

T.

Kanade

The

measurement

of

highlights

in

color

images

International

Journal

of

Computer

Vision

-LRB-

-RRB-

June

-LSB-

Koch

et

al.

-RSB-

C.

Koch

J.

Marroquin

and

A.

Yuille

Analog

neuronal

networks

in

early

vision

Proceedings

of

the

National

Academy

of

Sciences

U.S.A.

June

-LSB-

Little

et

al.

-RSB-

J.

Little

G.

E.

Blelloch

and

T.

A.

Cass

Algorithmic

techniques

for

Conclusions

computer

vision

on

a

fine-grained

parallel

machine

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

March

-LSB-

Lucas

-RSB-

B.

D.

Lucas

Generalized

Image

Matching

by

the

Method

of

Differences

PhD

thesis

Carnegie

Mellon

University

July

-LSB-

Matthies

and

Shafer

-RSB-

L.

Matthies

and

S.

A.

Shafer

Error

modeling

in

stereo

navigation

IEEE

Journal

of

Robotics

and

Automation

-LRB-

-RRB-

June

-LSB-

Matthies

and

Kanade

-RSB-

L.

H.

Matthies

and

T.

Kanade

The

cycle

of

uncertainty

and

constraint

in

robot

perception

In

International

Symposium

on

Robotics

Research

MIT

Press

August

-LSB-

Matthies

et

al.

-RSB-

L.

H.

Matthies

T.

Kanade

and

R.

Szeliski

Kalman

filter-based

algorithms

for

estimating

depth

from

image

sequences

International

Journal

of

Computer

Vision

-LSB-

Meagher

-RSB-

D.

Meagher

Geometric

modeling

using

octree

encoding

Computer

Graphics

and

Image

Processing

-LSB-

Nagel

-RSB-

H.-H

Nagel

On

the

estimation

of

optical

flow

relations

between

different

approaches

and

some

new

results

Artificial

Intelligence

-LSB-

Newman

and

Sproull

-RSB-

W.

M.

Newman

and

R.

F.

Sproull

Principles

of

Interactive

Computer

Graphics

McGraw

Hill

New

York

New

York

edition

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

signal

matching

algorithm

an

adaptive

window

based

on

a

brownian

motion

model

In

Third

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Osaka

Japan

December

-LSB-

Pentland

-RSB-

A.

P.

Pentland

Perceptual

organization

and

the

representation

of

natural

form

Artificial

Intelligence

-LRB-

-RRB-

May

-LSB-

Poggio

et

al.

-RSB-

T.

Poggio

V.

Torre

and

C.

Koch

Computational

vision

and

regular

Conclusions

ization

theory

Nature

-LRB-

-RRB-

September

-LSB-

Rives

et

al.

-RSB-

P.

Rives

E.

Breuil

and

B.

Espiau

Recursive

estimation

of

features

using

optical

flow

and

camera

motion

In

Conference

on

Intelligent

Autonomous

Systems

pages

Elsevier

Science

Publishers

December

Also

appeared

in

IEEE

International

Conference

on

Robotics

and

Automation

-LSB-

Sander

and

Zucker

-RSB-

P.

T.

Sander

and

S.

W.

Zucker

Inferring

surface

trace

and

differential

structure

from

images

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

September

-LSB-

Sha'ashua

and

Ullman

-RSB-

A.

Sha'ashua

and

S.

Ullman

Structural

saliency

the

detection

of

globally

salient

structures

using

a

locally

connected

network

In

Second

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Tampa

Florida

December

-LSB-

Szeliski

-RSB-

R.

Szeliski

Estimating

motion

from

sparse

range

data

without

correspondence

In

Second

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Tampa

Florida

December

-LSB-

Szeliski

-RSB-

R.

Szeliski

Bayesian

Modeling

of

Uncertainty

in

Low-Level

Vision

Kluwer

Academic

Publishers

Boston

Massachusetts

-LSB-

Szeliski

-RSB-

R.

Szeliski

Real-Time

Octree

Generation

from

Rotating

Objects

Technical

Report

Digital

Equipment

Corporation

Cambridge

Research

Lab

December

For

ordering

information

please

send

a

message

to

techreports@crl.dec.com

with

the

word

help

in

the

Subject

line

-LSB-

Szeliski

and

Ito

-RSB-

R.

Szeliski

and

M.

R.

Ito

New

Hermite

cubic

interpolator

for

two-dimensional

curve

generation

IEE

Proceedings

E

-LRB-

-RRB-

November

-LSB-

Szeliski

and

Tonnesen

-RSB-

R.

Szeliski

and

D.

Tonnesen

Particle

systems

for

surface

interpolation

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

in

preparation

-RRB-

Conclusions

-LSB-

Terzopoulos

and

Metaxas

-RSB-

D.

Terzopoulos

and

D.

Metaxas

Dynamic

models

with

local

and

global

deformations

deformable

superquadrics

In

Third

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

Osaka

Japan

December

-LSB-

Terzopoulos

et

al.

-RSB-

D.

Terzopoulos

A.

Witkin

and

M.

Kass

Symmetry-seeking

models

and

object

reconstruction

International

Journal

of

Computer

Vision

-LRB-

-RRB-

October

-LSB-

Tomasi

and

Kanade

-RSB-

C.

Tomasi

and

T.

Kanade

Shape

and

motion

without

depth

In

Third

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Osaka

Japan

December

-LSB-

Ullman

-RSB-

S.

Ullman

Maximizing

rigidity

the

incremental

recovery

of

structure

from

rigid

and

nonrigid

motion

Perception

-LSB-

Vaillant

-RSB-

R.

Vaillant

Using

occluding

contours

for

object

modeling

In

First

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

SpringerVerlag

Antibes

France

April

-LSB-

Vuylsteke

and

Oosterlinck

-RSB-

P.

Vuylsteke

and

A.

Oosterlinck

Range

image

acquisition

with

a

single

binary-encoded

light

pattern

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

February

-LSB-

Woodham

-RSB-

R.

J.

Woodham

Analysing

images

of

curved

surfaces

Artificial

Intelligence

A

Inverse

perspective

projection

with

homogeneous

coordinates

A

Inverse

perspective

projection

with

homogeneous

coor

dinates

To

compute

the

world

-LRB-

object

-RRB-

coordinates

of

a

point

p

given

its

screen

coordinates

q

we

can

use

the

projection

operator

P

after

the

inverse

matrix

transformation

M

To

see

that

this

is

the

case

we

note

from

-LRB-

-RRB-

that

W

q

where

the

value

of

W

is

unknown

From

-LRB-

-RRB-

we

have

p

M

W

M

But

since

we

know

that

the

fourth

element

of

p

-LRB-

X

Y

Z

-RRB-

was

originally

we

can

simply

normalize

M

to

obtain

the

formula

in

-LRB-

-RRB-

p

P

M

Similarly

if

we

wish

to

determine

where

the

pixel

at

time

t

denoted

by

screen

coordinates

qt

was

at

time

t

-LRB-

denoted

by

qt

-RRB-

we

have

p

W

t

M

t

qt

and

qt

P

-LRB-

Mt

-RRB-

P

Mt

t

qt

-LRB-

since

the

projection

operator

is

invariant

to

a

uniform

scaling

of

its

input

-RRB-

Thus

even

for

composite

transformations

of

coordinates

we

only

have

to

perform

one

projection

operation

at

the

end

of

the

transformation

even

if

multiple

perspective

or

inverse

perspective

transformations

have

occured

A

Inverse

perspective

projection

with

homogeneous

coordinates

Acknowledgements

I

would

like

to

thank

Gudrun

Klinker

for

her

careful

reading

of

this

manuscript

and

many

helpful

suggestions

and

an

anonymous

CVPR

reviewer

for

his

detailed

comments

and

insightful

observations

computer aided design
computer graphics animation
computer vision
motion
optic flow
surface interpolation

3-d model construction
image sequence analysis
kalman filter