Spline-Based

Image

Registration

Richard

Szeliski

and

James

Coughlan

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

April

Digital

Equipment

Corporation

has

four

research

facilities

the

Systems

Research

Center

and

the

Western

Research

Laboratory

both

in

Palo

Alto

California

the

Paris

Research

Laboratory

in

Paris

and

the

Cambridge

Research

Laboratory

in

Cambridge

Massachusetts

The

Cambridge

laboratory

became

operational

in

and

is

located

at

One

Kendall

Square

near

MIT

CRL

engages

in

computing

research

to

extend

the

state

of

the

computing

art

in

areas

likely

to

be

important

to

Digital

and

its

customers

in

future

years

CRL

's

main

focus

is

applications

technology

that

is

the

creation

of

knowledge

and

tools

useful

for

the

preparation

of

important

classes

of

applications

CRL

Technical

Reports

can

be

ordered

by

electronic

mail

To

receive

instructions

send

a

mes

sage

to

one

of

the

following

addresses

with

the

word

help

in

the

Subject

line

On

Digital

's

EASYnet

CRL

TECHREPORTS

On

the

Internet

techreports@crl.dec.com

This

work

may

not

be

copied

or

reproduced

for

any

commercial

purpose

Permission

to

copy

without

payment

is

granted

for

non?profit

educational

and

research

purposes

provided

all

such

copies

include

a

notice

that

such

copy

ing

is

by

permission

of

the

Cambridge

Research

Lab

of

Digital

Equipment

Corporation

an

acknowledgment

of

the

authors

to

the

work

and

all

applicable

portions

of

the

copyright

notice

The

Digital

logo

is

a

trademark

of

Digital

Equipment

Corporation

Cambridge

Research

Laboratory

One

Kendall

Square

Cambridge

Massachusetts

TM

Spline-Based

Image

Registration

Richard

Szeliski

and

James

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

April

Abstract

The

problem

of

image

registration

subsumes

a

number

of

problems

and

techniques

in

multiframe

image

analysis

including

the

computation

of

optic

flow

-LRB-

general

pixel-based

motion

-RRB-

stereo

correspondence

structure

from

motion

and

feature

tracking

We

present

a

new

registration

algorithm

based

on

spline

representations

of

the

displacement

field

which

can

be

specialized

to

solve

all

of

the

above

mentioned

problems

In

particular

we

show

how

to

compute

local

flow

global

-LRB-

parametric

-RRB-

flow

rigid

flow

resulting

from

camera

egomotion

and

multiframe

versions

of

the

above

problems

Using

a

spline-based

description

of

the

flow

removes

the

need

for

overlapping

correlation

windows

and

produces

an

explicit

measure

of

the

correlation

between

adjacent

flow

estimates

We

demonstrate

our

algorithm

on

multiframe

image

registration

and

the

recovery

of

projective

scene

geometry

We

also

provide

results

on

a

number

of

standard

motion

sequences

c

Digital

Equipment

Corporation

All

rights

reserved

University

Department

of

Physics

Cambridge

MA

Contents

i

Contents

Introduction

Previous

work

General

problem

formulation

Spline-based

motion

model

Local

-LRB-

general

-RRB-

flow

estimation

Global

-LRB-

planar

-RRB-

flow

estimation

Mixed

global

and

local

-LRB-

rigid

-RRB-

flow

estimation

Multiframe

flow

estimation

Experimental

results

Discussion

Future

work

and

Conclusions

ii

LIST

OF

TABLES

List

of

Figures

Displacement

spline

the

spline

control

vertices

f

-LRB-

uj

vj

-RRB-

g

are

shown

as

circles

-LRB-

ffi

-RRB-

and

the

pixel

displacements

f

-LRB-

ui

vi

-RRB-

g

are

shown

as

pluses

-LRB-

-RRB-

Spline

basis

functions

Block

diagram

of

spline-based

image

registration

Example

of

general

flow

computation

Block

diagram

of

global

motion

estimation

Example

of

affine

flow

computation

translation

Example

of

affine

flow

computation

divergence

-LRB-

zoom

-RRB-

Example

of

projective

motion

estimation

Example

of

projective

-LRB-

rigid

-RRB-

motion

estimation

Sinusoid

and

Square

sample

images

Yosemite

sample

image

and

flow

-LRB-

unthresholded

-RRB-

NASA

Sequence

rigid

flow

Rubik

Cube

sequence

sparse

flow

Hamburg

Taxi

sequence

dense

local

flow

List

of

Tables

Summary

of

Sinusoid

results

Summary

of

Square

results

Summary

of

Translating

Tree

results

Summary

of

Diverging

Tree

results

Summary

of

Yosemite

results

Introduction

Introduction

The

analysis

of

image

sequences

-LRB-

motion

analysis

-RRB-

is

one

of

the

more

actively

studied

areas

of

computer

vision

and

image

processing

The

estimation

of

motion

has

many

diverse

applications

including

video

compression

the

extraction

of

scene

geometry

and

camera

motion

robot

navigation

and

the

registration

of

multiple

images

The

common

problem

is

to

determine

correspondences

between

various

parts

of

images

in

a

sequence

This

problem

is

often

called

motion

estimation

multiple

view

analysis

or

image

registration

Motion

analysis

subsumes

a

number

of

sub-problems

and

associated

solution

techniques

including

optic

flow

stereo

and

multiframe

stereo

egomotion

estimation

and

feature

detection

and

tracking

Each

of

these

approaches

makes

different

assumptions

about

the

nature

of

the

scene

and

the

results

to

be

computed

-LRB-

computational

theory

and

representation

-RRB-

and

the

techniques

used

to

compute

these

results

-LRB-

algorithm

-RRB-

In

this

paper

we

present

a

general

motion

estimation

framework

which

can

be

specialized

to

solve

a

number

of

these

sub-problems

Like

Bergen

et

al.

-LSB-

-RSB-

we

view

motion

estimation

as

an

image

registration

task

with

a

fixed

computational

theory

-LRB-

optimality

criterion

-RRB-

and

view

each

sub-problem

as

an

instantiation

of

a

particular

global

or

local

motion

model

For

example

the

motion

may

be

completely

general

it

can

depend

on

a

few

global

parameters

-LRB-

e.g.

affine

flow

-RRB-

or

it

can

result

from

the

rigid

motion

of

a

scene

We

also

use

coarse-to-fine

-LRB-

hierarchical

-RRB-

algorithms

to

handle

large

displacements

The

key

difference

between

our

framework

and

previous

algorithms

is

that

we

represent

the

local

motion

field

using

multi-resolution

splines

This

has

a

number

of

advantages

over

previous

approaches

The

splines

impose

an

implicit

smoothness

on

the

motion

field

removing

in

many

instances

the

need

for

additional

smoothness

constraints

-LRB-

regularization

-RRB-

The

splines

also

remove

the

need

for

correlation

windows

centered

at

each

pixel

which

are

computationally

expensive

and

implicitly

assume

a

local

translational

model

Furthermore

they

provide

an

explicit

measure

of

the

correlation

between

adjacent

motion

estimates

The

algorithm

we

develop

to

estimate

structure

and

motion

from

rigid

scenes

differs

from

previous

algorithms

by

using

a

general

camera

model

which

eliminates

the

need

to

know

the

intrinsic

camera

calibration

parameters

This

results

in

estimates

of

projective

depth

rather

than

true

Euclidean

depth

the

conversion

to

Euclidean

shape

if

required

can

be

performed

in

a

later

post-processing

stage

-LSB-

Szeliski

-RSB-

Previous

work

The

remainder

of

the

paper

is

structured

as

follows

Section

presents

a

review

of

relevant

previous

work

Section

gives

the

general

problem

formulations

for

image

registration

Section

develops

our

algorithm

for

local

motion

estimation

Section

presents

the

algorithm

for

global

-LRB-

planar

-RRB-

motion

estimation

Section

presents

our

novel

formulation

of

structure

from

motion

based

on

the

recovery

of

projective

depth

Section

generalizes

our

previous

algorithms

to

multiple

frames

and

examines

the

resulting

performance

improvements

Section

presents

experimental

results

based

on

some

commonly

used

motion

test

sequences

Finally

we

close

with

a

comparison

of

our

approach

to

previous

algorithms

and

a

discussion

of

future

work

Previous

work

A

large

number

of

motion

estimation

and

image

registration

algorithms

have

been

developed

in

the

past

-LSB-

Brown

-RSB-

These

algorithms

include

optical

flow

-LRB-

general

motion

-RRB-

estimators

global

parametric

motion

estimators

constrained

motion

estimators

-LRB-

direct

methods

-RRB-

stereo

and

multiframe

stereo

hierarchical

-LRB-

coarse-to-fine

-RRB-

methods

feature

trackers

and

feature-based

registration

techniques

We

will

use

this

rough

taxonomy

to

briefly

review

previous

work

while

recognizing

that

these

algorithms

overlap

and

that

many

algorithms

use

ideas

from

several

of

these

categories

The

general

motion

estimation

problem

is

often

called

optical

flow

recovery

-LSB-

Hornand

Schunck

-RSB-

This

involves

estimating

an

independent

displacement

vector

for

each

pixel

in

an

image

Approaches

to

this

problem

include

gradient-based

approaches

based

on

the

brightness

constraint

-LSB-

Horn

and

Schunck

Lucas

and

Kanade

Nagel

-RSB-

correlation-based

techniques

such

as

the

Sum

of

Squared

Differences

-LRB-

SSD

-RRB-

-LSB-

Anandan

-RSB-

spatio-temporal

filtering

-LSB-

Adelson

and

Bergen

Heeger

Fleet

and

Jepson

-RSB-

and

regularization

-LSB-

Horn

and

Schunck

Hildreth

Poggio

et

al.

-RSB-

Nagel

-LSB-

-RSB-

and

Anandan

-LSB-

Anandan

-RSB-

provide

comparisons

and

derive

relations

between

different

techniques

while

Barron

et

al.

-LSB-

-RSB-

provide

some

numerical

comparisons

Global

motion

estimators

-LSB-

Lucas

Bergen

et

al.

-RSB-

use

a

simple

flow

field

model

parameterized

by

a

small

number

of

unknown

variables

Examples

of

global

motion

models

include

affine

and

quadratic

flow

fields

In

the

taxonomy

of

Bergen

et

al.

-LSB-

-RSB-

these

fields

are

called

parametric

motion

models

since

they

can

be

used

locally

as

well

-LRB-

e.g.

you

can

estimate

Previous

work

affine

flow

at

every

pixel

from

filter

outputs

-LSB-

Manmatha

and

Oliensis

-RSB-

-RRB-

Global

methods

are

most

useful

when

the

scene

has

a

particularly

simple

form

e.g.

when

the

scene

is

planar

Constrained

-LRB-

quasi-parametric

-LSB-

Bergen

et

al.

-RSB-

-RRB-

motion

models

fall

between

local

and

global

methods

Typically

these

use

a

combination

of

global

egomotion

parameters

with

local

shape

-LRB-

depth

-RRB-

parameters

Examples

of

this

approach

include

the

direct

methods

of

Horn

and

Weldon

-LSB-

-RSB-

and

others

-LSB-

Hanna

Bergen

et

al.

-RSB-

In

this

paper

we

use

projective

descriptions

of

motion

and

depth

-LSB-

Faugeras

Mohr

et

al.

Szeliski

and

Kang

-RSB-

for

our

constrained

motion

model

which

removes

the

need

for

calibrated

cameras

Stereo

matching

-LSB-

Barnard

and

Fischler

Quam

Dhond

and

Aggarwal

-RSB-

is

traditionally

considered

as

a

separate

sub-discipline

within

computer

vision

-LRB-

and

of

course

photogrammetry

-RRB-

but

there

are

strong

connections

between

the

two

problems

Stereo

can

be

viewed

as

a

simplified

version

of

the

constrained

motion

model

where

the

egomotion

parameters

-LRB-

the

epipolar

geometry

-RRB-

are

given

so

that

each

flow

vector

is

constrained

to

lie

along

a

known

line

While

stereo

is

traditionally

performed

on

pairs

of

images

more

recent

algorithms

use

sequences

of

images

-LRB-

multiframe

stereo

or

motion

stereo

-RRB-

-LSB-

Bolles

et

al.

Matthies

et

al.

Okutomi

and

Kanade

Okutomi

and

Kanade

-RSB-

Hierarchical

-LRB-

coarse-to-fine

-RRB-

matching

algorithms

have

a

long

history

of

use

both

in

stereo

matching

-LSB-

Quam

Witkin

et

al.

-RSB-

and

in

motion

estimation

-LSB-

Enkelmann

Anandan

Singh

Bergen

et

al.

-RSB-

Hierarchical

algorithms

first

solve

the

matching

problem

on

smaller

lower-resolution

images

and

then

use

these

to

initialize

higher-resolution

estimates

Their

advantages

include

both

increased

computation

efficiency

and

the

ability

to

find

better

solutions

-LRB-

escape

from

local

minima

-RRB-

Tracking

individual

features

-LRB-

corners

points

lines

-RRB-

in

images

has

always

been

alternative

to

iconic

-LRB-

pixel-based

-RRB-

optic

flow

techniques

-LSB-

Dreschler

and

Nagel

Sethi

and

Jain

Zheng

and

Chellappa

-RSB-

This

has

the

advantage

of

requiring

less

computation

and

of

being

less

sensitive

to

lighting

variation

The

algorithm

presented

in

this

paper

is

closely

related

to

patchbased

feature

trackers

-LSB-

Lucas

and

Kanade

Rehg

and

Witkin

Tomasi

and

Kanade

-RSB-

In

fact

our

general

motion

estimator

can

be

used

as

a

parallel

adaptive

feature

tracker

by

selecting

spline

control

vertices

with

low

uncertainty

in

both

motion

components

Like

-LSB-

Rehg

and

Witkin

-RSB-

which

is

an

affine-patch

based

tracker

it

can

handle

large

deformations

in

the

spline-based

flow

fields

we

describe

in

the

next

section

can

be

viewed

as

local

parametric

models

since

the

flow

within

each

spline

patch

is

defined

by

a

small

number

of

control

vertices

General

problem

formulation

patches

being

tracked

Spline-based

image

registration

techniques

have

been

used

in

both

the

image

processing

and

computer

graphics

communities

The

work

in

-LSB-

Goshtasby

Goshtasby

-RSB-

applies

surface

fitting

to

discrete

displacement

estimates

based

on

feature

correspondences

to

obtain

a

smooth

displacement

field

Wolberg

-LSB-

-RSB-

provides

a

review

of

the

extensive

literature

in

digital

image

warping

which

can

be

used

to

resample

images

once

the

-LRB-

usually

global

-RRB-

displacements

are

known

Spline-based

displacement

fields

have

recently

been

used

in

computer

graphics

to

perform

morphing

-LSB-

Beier

and

Neely

-RSB-

-LRB-

deformable

image

blending

-RRB-

using

manually

specified

correspondences

Registration

techniques

based

on

elastic

deformations

of

images

-LSB-

Burr

Bajcsy

and

Broit

Bajcsy

and

Kovacic

Amit

-RSB-

also

sometimes

use

splines

as

their

representation

-LSB-

Bajcsy

and

Broit

-RSB-

General

problem

formulation

The

general

image

registration

problem

can

be

formulated

as

follows

We

are

given

a

sequence

of

images

It

-LRB-

x

y

-RRB-

which

we

assume

were

formed

by

locally

displacing

a

reference

image

I

-LRB-

x

y

-RRB-

with

horizontal

and

vertical

displacement

ut

-LRB-

x

y

-RRB-

and

vt

-LRB-

x

y

-RRB-

i.e.

It

-LRB-

x

ut

y

vt

-RRB-

I

-LRB-

x

y

-RRB-

-LRB-

-RRB-

Each

individual

image

is

assumed

to

be

corrupted

with

uniform

white

Gaussian

noise

We

also

ignore

possible

occlusions

-LRB-

foldovers

-RRB-

in

the

warped

images

Given

such

a

sequence

of

images

we

wish

to

simultaneously

recover

the

displacement

fields

ut

and

vt

and

the

reference

image

I

-LRB-

x

y

-RRB-

The

maximum

likelihood

solution

to

this

problem

is

well

known

-LSB-

Szeliski

-RSB-

and

consists

of

minimizing

the

squared

error

X

t

Z

Z

-LSB-

It

-LRB-

x

ut

y

vt

-RRB-

I

-LRB-

x

y

-RRB-

-RSB-

dy

-LRB-

-RRB-

In

practice

we

are

usually

given

a

set

of

discretely

sampled

images

so

we

replace

the

above

integrals

with

summations

over

the

set

of

pixels

f

-LRB-

xi

yi

-RRB-

g

If

the

displacement

fields

ut

and

vt

at

different

times

are

independent

of

each

other

and

the

reference

intensity

image

I

-LRB-

x

y

-RRB-

is

assumed

to

be

known

the

above

minimization

problem

will

use

the

terms

displacement

field

flow

field

and

motion

estimate

interchangeably

Spline-based

motion

model

decomposes

into

a

set

of

independent

minimizations

one

for

each

frame

For

now

we

will

assume

that

this

is

the

case

and

only

study

the

two

frame

problem

which

can

be

rewritten

E

-LRB-

fui

vig

-RRB-

X

i

-LSB-

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

xi

yi

-RRB-

-RSB-

-LRB-

-RRB-

This

equation

is

called

the

Sum

of

Squared

Differences

-LRB-

SSD

-RRB-

formula

-LSB-

Anandan

-RSB-

Expanding

in

a

first

order

Taylor

series

expansion

in

-LRB-

ui

vi

-RRB-

yields

the

the

image

brightness

constraint

-LSB-

Horn

and

Schunck

Anandan

-RSB-

The

above

minimization

problem

will

typically

have

many

locally

optimal

solutions

-LRB-

in

terms

of

the

f

-LRB-

ui

vi

-RRB-

g

-RRB-

The

choice

of

method

for

finding

the

best

estimate

efficiently

is

what

typically

differentiates

between

various

motion

estimation

algorithms

For

example

the

SSD

algorithm

performs

the

summation

at

each

pixel

over

a

window

-LSB-

Anandan

-RSB-

-LRB-

more

recent

variations

use

adaptive

windows

-LSB-

Okutomi

and

Kanade

-RSB-

and

multiple

frames

-LSB-

Okutomi

and

Kanade

-RSB-

-RRB-

Regularization-based

algorithms

add

smoothness

constraints

on

the

u

and

v

fields

to

obtain

good

solutions

-LSB-

Horn

and

Schunck

Hildreth

Poggio

et

al.

-RSB-

Multiscale

or

hierarchical

-LRB-

coarse-to-fine

-RRB-

techniques

are

often

used

to

speed

the

search

for

the

optimum

displacement

field

Another

decision

that

must

be

made

is

how

to

represent

the

-LRB-

u

v

-RRB-

fields

Assigning

an

independent

estimate

at

each

pixel

-LRB-

ui

vi

-RRB-

is

the

most

commonly

made

choice

but

global

motion

descriptors

are

also

possible

-LSB-

Lucas

Bergen

et

al.

-RSB-

-LRB-

see

also

Section

-RRB-

Constrained

motion

models

which

combine

a

global

rigid

motion

description

with

a

local

depth

estimate

are

also

used

-LSB-

Horn

and

Weldon

Jr.

Hanna

Bergen

et

al.

-RSB-

and

we

will

study

these

in

Section

Both

local

correlation

windows

-LRB-

as

in

SSD

-RRB-

and

global

smoothness

constraints

attempt

to

disambiguate

possible

motion

field

estimates

by

aggregating

information

from

neighboring

pixels

The

resulting

displacement

estimates

are

therefore

highly

correlated

While

it

is

possible

to

analyze

the

correlations

induced

by

overlapping

windows

-LSB-

Matthies

et

al.

-RSB-

and

regularization

-LSB-

Szeliski

-RSB-

the

procedures

are

cumbersome

and

rarely

used

Spline-based

motion

model

will

return

to

the

problem

of

multiframe

motion

in

Section

General

problem

formulation

e

e

e

e

-LRB-

uj

vj

-RRB-

-LRB-

ui

vi

-RRB-

Figure

Displacement

spline

the

spline

control

vertices

f

-LRB-

uj

vj

-RRB-

g

are

shown

as

circles

-LRB-

ffi

-RRB-

and

the

pixel

displacements

f

-LRB-

ui

vi

-RRB-

g

are

shown

as

pluses

-LRB-

-RRB-

The

alternative

to

these

approaches

which

we

introduce

in

this

paper

is

to

represent

the

displacements

fields

u

-LRB-

x

y

-RRB-

and

v

-LRB-

x

y

-RRB-

as

two-dimensional

splines

controlled

by

a

smaller

number

of

displacement

estimates

uj

and

vj

which

lie

on

a

coarser

spline

control

grid

-LRB-

Figure

-RRB-

The

value

for

the

displacement

at

a

pixel

i

can

be

written

as

u

-LRB-

xi

yi

-RRB-

X

j

uj

Bj

-LRB-

xi

yi

-RRB-

or

ui

X

j

uj

wij

-LRB-

-RRB-

where

the

Bj

-LRB-

x

y

-RRB-

are

called

the

basis

functions

and

are

only

non-zero

over

a

small

interval

-LRB-

finite

support

-RRB-

We

call

the

wij

Bj

-LRB-

xi

yi

-RRB-

weights

to

emphasize

that

the

-LRB-

ui

vi

-RRB-

are

known

linear

combinations

of

the

-LRB-

uj

vj

-RRB-

In

our

current

implementation

the

basis

functions

are

spatially

shifted

versions

of

each

other

i.e.

Bj

-LRB-

x

y

-RRB-

B

-LRB-

x

xj

y

yj

-RRB-

We

have

studied

five

different

interpolation

functions

block

B

-LRB-

x

y

-RRB-

on

-LSB-

-RSB-

linear

B

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

on

-LSB-

-RSB-

-LRB-

x

-RRB-

on

-LSB-

-RSB-

-LSB-

-RSB-

and

-LRB-

y

-RRB-

on

-LSB-

-RSB-

-LSB-

-RSB-

linear

on

sub-triangles

B

-LRB-

x

y

-RRB-

max

-LRB-

max

-LRB-

jxj

jyj

jx

yj

-RRB-

-RRB-

the

remainder

of

the

paper

we

will

use

indices

i

for

pixels

and

j

for

spline

control

vertices

Spline-based

motion

model

block

linear

-LRB-

square

patch

-RRB-

linear

-LRB-

triangles

-RRB-

bilinear

biquadratic

Figure

Spline

basis

functions

General

problem

formulation

bilinear

B

-LRB-

x

y

-RRB-

-LRB-

jxj

-RRB-

-LRB-

jyj

-RRB-

on

-LSB-

-RSB-

biquadratic

B

-LRB-

x

y

-RRB-

-LRB-

x

-RRB-

-LRB-

y

-RRB-

on

-LSB-

-RSB-

where

-LRB-

x

-RRB-

is

the

quadratic

B-spline

Figure

shows

graphs

of

the

basis

functions

for

the

five

splines

We

also

impose

the

condition

that

the

spline

control

grid

is

a

regular

subsampling

of

the

pixel

grid

xj

mxi

yj

myi

so

that

each

set

of

m

m

pixels

corresponds

to

a

single

spline

patch

This

means

that

the

set

of

wij

weights

need

only

be

stored

for

a

single

patch

How

do

spline

representations

compare

to

local

correlation

windows

and

to

regularization

This

question

has

previously

been

studied

in

the

context

of

active

deformable

contours

-LRB-

snakes

-RRB-

The

original

work

on

snakes

was

based

on

a

regularization

framework

-LSB-

Kass

et

al.

-RSB-

giving

the

snake

the

ability

to

model

arbitrarily

detailed

bends

or

discontinuities

where

warranted

by

the

data

More

recent

versions

of

snakes

often

employ

the

B-snake

-LSB-

Menet

et

al.

Blake

et

al.

-RSB-

which

has

fewer

control

vertices

A

spline-based

snake

has

fewer

degrees

of

freedom

and

thus

may

be

easier

to

recover

The

smooth

interpolation

function

between

vertices

plays

a

similar

role

to

regularization

although

the

smoothing

introduced

is

not

as

uniform

or

stationary

In

our

use

of

splines

for

modeling

displacement

fields

we

have

a

similar

tradeoff

-LRB-

see

Section

for

more

discussion

-RRB-

We

often

may

not

need

regularization

-LRB-

e.g.

in

highly

textured

scenes

-RRB-

Where

required

adding

a

regularization

term

to

the

cost

function

-LRB-

-RRB-

is

straightforward

i.e.

we

can

use

a

first

order

regularizer

-LSB-

Poggio

et

al.

-RSB-

-LRB-

f

uk

l

vk

lg

-RRB-

X

k

l

-LRB-

uk

l

uk

l

-RRB-

-LRB-

uk

l

uk

l

-RRB-

terms

in

vkl

-LRB-

-RRB-

or

a

second

order

regularizer

-LSB-

Terzopoulos

-RSB-

-LRB-

f

uk

l

vk

lg

-RRB-

h

X

k

l

-LRB-

uk

l

uk

l

uk

l

-RRB-

-LRB-

uk

l

uk

l

uk

l

-RRB-

-LRB-

-RRB-

-LRB-

uk

l

uk

l

uk

l

uk

l

-RRB-

terms

in

vkl

where

h

is

the

patch

size

and

we

index

the

spline

control

vertices

with

indices

-LRB-

k

l

-RRB-

Spline-based

flow

descriptors

also

remove

the

need

for

overlapping

correlation

windows

since

each

flow

estimate

-LRB-

uj

vj

-RRB-

is

based

on

weighted

contributions

from

all

of

the

pixels

beneath

the

support

of

its

basis

function

-LRB-

e.g

-LRB-

-RRB-

-LRB-

-RRB-

pixels

for

a

bilinear

basis

-RRB-

As

we

will

show

in

Section

the

spline-based

flow

formulation

makes

it

straightforward

to

compute

the

uncertainty

-LRB-

covariance

matrix

-RRB-

associated

with

the

complete

flow

field

It

also

corresponds

naturally

to

the

Local

-LRB-

general

-RRB-

flow

estimation

optimal

Bayesian

estimator

for

the

flow

where

the

squared

pixel

errors

correspond

to

Gaussian

noise

while

the

spline

model

-LRB-

and

any

associated

regularizers

-RRB-

form

the

prior

-LSB-

Szeliski

-RSB-

Before

moving

on

to

our

different

motion

models

and

solution

techniques

we

should

point

out

that

the

squared

pixel

error

function

-LRB-

-RRB-

can

be

generalized

to

account

for

photometric

variation

-LRB-

global

brightness

and

contrast

changes

-RRB-

Following

-LSB-

Lucas

Gennert

-RSB-

we

can

write

-LRB-

fui

vig

-RRB-

X

i

-LSB-

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

xi

yi

-RRB-

b

-RSB-

-LRB-

-RRB-

where

b

and

c

are

the

-LRB-

per-frame

-RRB-

brightness

and

contrast

correction

terms

Both

of

these

parameters

can

be

estimated

concurrently

with

the

flow

field

at

little

additional

cost

Their

inclusion

is

most

useful

in

situations

where

the

photometry

can

change

between

successive

views

-LRB-

e.g.

when

the

images

are

not

acquired

concurrently

-RRB-

We

should

also

mention

that

the

matching

need

not

occur

directly

on

the

raw

intensity

images

Both

linear

-LRB-

e.g.

low-pass

or

band-pass

filtering

-LSB-

Burt

and

Adelson

-RSB-

-RRB-

and

non-linear

pre-processing

can

be

performed

Local

-LRB-

general

-RRB-

flow

estimation

To

recover

the

local

spline-based

flow

parameters

we

need

to

minimize

the

cost

function

-LRB-

-RRB-

with

respect

to

the

f

uj

vjg

We

do

this

using

a

variant

of

the

Levenberg-Marquardt

iterative

non-linear

minimization

technique

-LSB-

Press

et

al.

-RSB-

First

we

compute

the

gradient

of

E

in

-LRB-

-RRB-

with

respect

to

each

of

the

parameters

uj

and

vj

guj

@E

uj

X

i

eiGxi

wij

gvj

@E

vj

X

i

eiGyi

wij

-LRB-

-RRB-

where

ei

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

xi

yi

-RRB-

-LRB-

-RRB-

is

the

intensity

error

at

pixel

i

-LRB-

Gxi

Gyi

-RRB-

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

-RRB-

techniques

with

overlapping

windows

do

not

have

a

similar

direct

connection

to

Bayesian

techniques

Local

-LRB-

general

-RRB-

flow

estimation

is

the

intensity

gradient

of

at

the

displaced

position

for

pixel

i

and

the

wij

are

the

sampled

values

of

the

spline

basis

function

-LRB-

-RRB-

Algorithmically

we

compute

the

above

gradients

by

first

forming

the

displacement

vector

for

each

pixel

-LRB-

ui

vi

-RRB-

using

-LRB-

-RRB-

then

computing

the

resampled

intensity

and

gradient

values

of

at

-LRB-

-RRB-

-LRB-

xi

ui

yi

vi

-RRB-

computing

ei

and

finally

incrementing

the

guj

and

gvj

values

of

all

control

vertices

affecting

that

pixel

For

the

Levenberg-Marquardt

algorithm

we

also

require

the

approximate

Hessian

matrix

A

where

the

second-derivative

terms

are

left

out

The

matrix

A

contains

entries

of

the

form

auujk

X

i

@ei

uj

@ei

uk

X

i

wijwik

-LRB-

Gxi

-RRB-

auvjk

avujk

X

i

@ei

uj

@ei

vk

X

i

wijwikGxi

Gyi

-LRB-

-RRB-

avvjk

X

i

@ei

vj

@ei

vk

X

i

wijwik

-LRB-

Gyi

-RRB-

The

entries

of

A

can

be

computed

at

the

same

time

as

the

energy

gradients

What

is

the

structure

of

the

approximate

Hessian

matrix

The

sub-matrix

Ajj

corresponding

to

the

terms

auujj

auvjj

and

avvjj

encodes

the

local

shape

of

the

sum-of-squared

difference

correlation

surface

-LSB-

Lucas

Anandan

-RSB-

This

matrix

is

often

used

to

compute

an

updated

flow

vector

by

setting

-LSB-

D

uj

D

vj

-RSB-

T

A

jj

-LSB-

guj

gvj

-RSB-

T

-LRB-

-RRB-

-LSB-

Lucas

Anandan

Bergen

et

al.

-RSB-

The

overall

A

matrix

is

a

sparse

multi-banded

block-diagonal

matrix

i.e.

sub-blocks

Ajk

will

be

non-zero

only

if

vertices

j

and

k

both

influence

some

common

patch

of

pixels

The

Levenberg-Marquardt

algorithm

proceeds

by

computing

an

increment

Du

to

the

current

displacement

estimate

u

which

satisfies

-LRB-

A

diag

-LRB-

A

-RRB-

-RRB-

Du

g

-LRB-

-RRB-

where

u

is

the

vector

of

concatenated

displacement

estimates

f

uj

vjg

g

is

the

vector

of

concatenated

energy

gradients

fguj

gvj

g

and

is

a

stabilization

factor

which

varies

over

time

-LSB-

Press

et

al.

-RSB-

For

systems

with

small

numbers

of

parameters

e.g.

if

only

a

single

spline

patch

is

mentioned

in

-LSB-

Press

et

al.

-RSB-

inclusion

of

these

terms

can

be

destabilizing

if

the

model

fits

badly

or

is

contaminated

by

outlier

points

Local

-LRB-

general

-RRB-

flow

estimation

being

used

-LRB-

Section

-RRB-

this

system

of

equations

can

be

solved

at

reasonable

computational

cost

However

for

general

flow

computation

there

may

be

thousands

of

spline

control

variables

-LRB-

e.g.

for

a

image

with

m

we

have

ss

parameters

-RRB-

In

this

case

iterative

sparse

matrix

techniques

have

to

be

used

to

solve

the

above

system

of

equations

In

our

current

implementation

we

use

preconditioned

gradient

descent

to

update

our

flow

estimates

Du

B

g

-LRB-

-RRB-

where

B

A

I

and

A

block

diag

-LRB-

A

-RRB-

is

the

set

of

block

diagonal

matrices

used

in

-LRB-

-RRB-

In

this

simplest

version

the

update

rule

is

very

close

to

that

used

by

-LSB-

Lucas

-RSB-

and

others

with

the

following

differences

the

equations

for

computing

the

g

and

A

are

different

-LRB-

based

on

spline

interpolation

-RRB-

an

additional

diagonal

term

is

added

for

there

is

a

step

size

The

step

size

is

necessary

because

we

are

ignoring

the

off-block-diagonal

terms

in

A

which

can

be

quite

significant

An

optimal

value

for

can

be

computed

at

each

iteration

by

minimizing

DE

-LRB-

d

-RRB-

ss

Ad

dTg

i.e.

by

setting

-LRB-

d

g

-RRB-

-LRB-

dT

Ad

-RRB-

The

denominator

can

be

computed

without

explicitly

computing

A

by

noting

that

dTAd

X

i

-LRB-

Gxi

ffiui

Gyi

ffivi

-RRB-

where

ffiui

X

j

wij

ffi

uj

ffivi

X

j

wij

ffi

vj

and

the

-LRB-

ffi

uj

ffi

vj

-RRB-

are

the

components

of

d.

To

handle

larger

displacements

we

run

our

algorithm

in

a

coarse-to-fine

-LRB-

hierarchical

-RRB-

fashion

A

Gaussian

image

pyramid

is

first

computed

using

an

iterated

filter

-LSB-

Burt

and

Adelson

fill

in

prevents

the

application

of

direct

sparse

matrix

solvers

-LSB-

Terzopoulos

Szeliski

-RSB-

vector

g

B

is

called

the

preconditioned

residual

vector

For

preconditioned

conjugate

gradient

descent

the

direction

vector

d

is

set

to

g.

Bayesian

justification

can

be

found

in

-LSB-

Simoncelli

et

al.

-RSB-

and

additional

possible

local

weightings

in

-LSB-

Lucas

p.

-RSB-

Global

-LRB-

planar

-RRB-

flow

estimation

-RSB-

We

then

run

the

algorithm

on

one

of

the

smaller

pyramid

levels

and

use

the

resulting

flow

estimates

to

initialize

the

next

finer

level

-LRB-

using

bilinear

interpolation

and

doubling

the

displacement

magnitudes

-RRB-

Figure

shows

a

block

diagram

of

the

processing

stages

involved

in

our

spline-based

image

registration

algorithm

Figure

shows

an

example

of

the

flow

estimates

produced

by

our

technique

The

input

image

is

pixels

and

the

flow

is

displayed

on

a

grid

We

used

pixel

spline

patches

and

levels

in

the

pyramid

with

iterations

at

each

level

The

flow

estimates

are

very

good

in

the

textured

areas

corresponding

to

the

Rubik

cube

the

stationary

boxes

and

the

turntable

edges

Flow

vectors

in

the

uniform

intensity

areas

-LRB-

e.g.

table

and

turntable

tops

-RRB-

are

fairly

arbitrary

This

example

uses

no

regularization

beyond

that

imposed

by

the

spline

patches

nor

does

it

threshold

flow

vectors

according

to

certainty

For

a

more

detailed

analysis

see

Section

Global

-LRB-

planar

-RRB-

flow

estimation

In

many

applications

e.g.

in

the

registration

of

pieces

of

a

flat

scene

when

the

distance

between

the

camera

and

the

scene

is

large

-LSB-

Bergen

et

al.

-RSB-

or

when

performing

a

coarse

registration

of

slices

in

a

volumetric

data

set

-LSB-

Carlbom

et

al.

-RSB-

a

single

global

description

of

the

motion

model

may

suffice

A

simple

example

of

such

a

global

motion

is

an

affine

flow

-LSB-

Koenderink

and

van

Doorn

Rehg

and

Witkin

Bergen

et

al.

-RSB-

u

-LRB-

x

y

-RRB-

-LRB-

-RRB-

x

v

-LRB-

x

y

-RRB-

-LRB-

-RRB-

y

-LRB-

-RRB-

The

parameters

m

-LRB-

-RRB-

T

are

called

the

global

motion

parameters

Models

with

fewer

degrees

of

freedom

such

as

pure

translation

translation

and

rotation

or

translation

plus

rotation

and

scale

-LRB-

similarity

transform

-RRB-

are

also

possible

but

they

will

not

be

studied

in

this

paper

To

compute

the

global

motion

estimate

we

take

a

two

step

approach

First

we

define

the

spline

control

vertices

uj

-LRB-

uj

vj

-RRB-

T

in

terms

of

the

global

motion

parameters

uj

xj

yj

xj

yj

xj

yj

Tjm

xj

-LRB-

-RRB-

Second

we

define

the

flow

at

each

pixel

using

our

usual

spline

interpolation

Note

that

for

affine

Global

-LRB-

planar

-RRB-

flow

estimation

First

Image

Second

Image

Control

vertices

Warping

-LRB-

resampling

-RRB-

Spline

Interpolation

Difference

Spatial

gradient

Flow

gradient

Flow

Hessian

Flow

step

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

xi

yi

-RRB-

-LRB-

xi

yi

-RRB-

-LRB-

xi

ui

yi

vi

-RRB-

s

ei

-LRB-

Gxi

Gyi

-RRB-

s

g

f

-LRB-

guj

gvj

-RRB-

g

A

fauvjk

g

u

f

-LRB-

uj

vj

-RRB-

g

soeoe

-LRB-

ui

vi

-RRB-

Du

i

Figure

Block

diagram

of

spline-based

image

registration

The

numbers

in

the

lower

right

corner

of

each

processing

box

refer

to

the

associated

equation

numbers

in

the

paper

Global

-LRB-

planar

-RRB-

flow

estimation

local

flow

-LSB-

p

rubik/s

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Example

of

general

flow

computation

-LRB-

or

simpler

-RRB-

flow

this

gives

the

correct

flow

at

each

pixel

if

linear

or

bilinear

interpolants

are

used

For

affine

-LRB-

or

simpler

-RRB-

flow

it

is

therefore

possible

to

use

only

a

single

spline

patch

Why

use

this

two-step

procedure

then

First

this

approach

will

work

better

when

we

generalize

our

motion

model

to

projective

transformations

-LRB-

see

below

-RRB-

Second

there

are

computational

savings

in

only

storing

the

wij

for

smaller

patches

Lastly

we

can

obtain

a

better

estimate

of

the

Hessian

at

m

at

lower

computational

cost

as

we

discuss

below

To

apply

Levenberg-Marquardt

as

before

we

need

to

compute

both

the

gradient

of

the

cost

function

with

respect

to

m

and

the

Hessian

Computing

the

gradient

is

straightforward

gm

@E

@m

X

j

uj

@m

@E

uj

X

j

TTj

gj

-LRB-

-RRB-

where

gj

-LRB-

guj

gvj

-RRB-

T

The

Hessian

matrix

can

be

computed

in

a

similar

fashion

Am

@mT

@m

ss

X

jk

TTj

AjkTk

-LRB-

-RRB-

where

the

Ajk

are

the

submatrices

of

A.

definitions

of

uj

and

vj

would

have

to

be

adjusted

if

biquadratic

splines

are

being

used

affine

patches

can

also

be

used

to

track

features

-LSB-

Rehg

and

Witkin

-RSB-

Our

general

flow

estimator

-LRB-

Section

-RRB-

performs

a

similar

function

except

in

parallel

and

with

overlapping

windows

Global

-LRB-

planar

-RRB-

flow

estimation

We

can

approximate

the

Hessian

matrix

even

further

by

neglecting

the

off-diagonal

Ajk

matrices

This

is

equivalent

to

modeling

the

flow

estimate

at

each

control

vertex

as

being

independent

of

other

vertex

estimates

When

the

spline

patches

are

sufficiently

large

and

contain

sufficient

texture

this

turns

out

to

be

a

reasonable

approximation

To

compute

the

optimal

step

size

we

let

d

Tdm

-LRB-

-RRB-

where

T

is

the

concatenation

of

all

the

Tj

matrices

and

then

set

-LRB-

d

g

-RRB-

-LRB-

dT

Ad

-RRB-

as

before

Figure

shows

a

block

diagram

of

the

processing

stages

involved

in

the

global

motion

estimation

algorithms

presented

in

this

and

the

next

section

Examples

of

our

global

affine

flow

estimator

applied

to

two

different

motion

sequences

can

be

seen

in

Figures

and

These

two

sequences

were

generated

synthetically

from

a

real

base

image

with

Figure

being

pure

translational

motion

and

Figure

being

divergent

motion

-LSB-

Barron

et

al.

-RSB-

As

expected

the

motion

is

recovered

extremely

well

in

this

case

-LRB-

see

Section

for

quantitative

results

-RRB-

A

more

interesting

case

in

general

is

that

of

a

planar

surface

in

motion

viewed

through

a

pinhole

camera

This

motion

can

be

described

as

a

projective

transformation

of

the

plane

u

-LRB-

x

y

-RRB-

x

v

-LRB-

x

y

-RRB-

y

-LRB-

-RRB-

Our

projective

requires

parameters

per

frame

which

is

the

same

number

as

the

quadratic

flow

field

used

in

-LSB-

Bergen

et

al.

-RSB-

However

our

formulation

allows

for

arbitrarily

large

displacements

whereas

-LSB-

Bergen

et

al.

-RSB-

is

based

on

instantaneous

-LRB-

infinitesimal

-RRB-

motion

Our

formulation

also

does

not

require

the

camera

to

be

calibrated

and

allows

the

internal

camera

parameters

-LRB-

e.g.

zoom

-RRB-

to

vary

over

time

The

price

we

pay

is

that

the

motion

field

is

no

longer

a

linear

function

of

the

global

motion

parameters

To

compute

the

gradient

and

the

Hessian

we

proceed

as

before

We

use

the

equations

uj

xj

yj

xj

yj

xj

its

full

generality

we

should

have

an

instead

of

a

in

the

denominator

The

situation

occurs

only

for

camera

rotations

Global

-LRB-

planar

-RRB-

flow

estimation

Control

vertices

Global

motion

parameters

Depth

estimates

Image

warping

differencing

and

gradient

computation

-LRB-

Figure

-RRB-

Motion

model

Motion

step

Depth

step

Motion

gradient

Depth

gradient

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

u

f

-LRB-

uj

vj

-RRB-

g

s

A

fauvjk

g

s

g

f

-LRB-

guj

gvj

-RRB-

g

s

m

soe

Dm

i

gm

Am

z

f

zj

g

soe

Dz

i

gz

Az

Figure

Block

diagram

of

global

motion

estimation

The

numbers

in

the

lower

right

corner

of

each

processing

box

refer

to

the

associated

equation

numbers

in

the

paper

Global

-LRB-

planar

-RRB-

flow

estimation

affine

flow

-LSB-

treet/c

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Example

of

affine

flow

computation

translation

affine

-LSB-

treed/c

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Example

of

affine

flow

computation

divergence

-LRB-

zoom

-RRB-

Mixed

global

and

local

-LRB-

rigid

-RRB-

flow

estimation

vj

xj

yj

xj

yj

yj

-LRB-

-RRB-

to

compute

the

spline

control

vertices

and

use

the

B-spline

interpolants

to

compute

the

flow

at

each

pixel

This

flow

is

not

exactly

equivalent

to

the

true

projective

flow

defined

in

-LRB-

-RRB-

since

the

latter

involves

a

division

at

each

pixel

However

the

error

will

be

small

if

the

patches

are

small

and/or

the

perspective

distortion

-LRB-

-RRB-

is

small

To

compute

the

gradient

we

note

that

ffi

uj

Dj

xj

yj

xjNuj

Dj

yj

Nuj

Dj

xj

yj

xjNvj

Dj

yj

Nvj

Dj

ffim

Tj

ffim

-LRB-

-RRB-

where

Nuj

xj

yj

and

Nvj

xj

yj

are

the

current

numerators

of

-LRB-

-RRB-

Dj

xj

yj

is

the

current

denominator

and

m

-LRB-

-RRB-

T

With

this

modification

to

the

affine

case

we

can

proceed

as

before

applying

-LRB-

-RRB-

-LRB-

-RRB-

to

compute

the

global

gradient

Hessian

and

the

stepsize

Thus

we

see

that

even

though

the

problem

is

no

longer

linear

the

modifications

involve

simply

using

an

extra

division

per

spline

control

vertex

Figure

shows

an

image

which

has

been

perspectively

distorted

and

the

accompanying

recovered

flow

field

The

motion

is

that

of

a

plane

rotating

around

its

y

axis

and

moving

slightly

forward

viewed

with

a

wide

angle

lens

Mixed

global

and

local

-LRB-

rigid

-RRB-

flow

estimation

A

special

case

of

optic

flow

computation

which

occurs

often

in

practice

is

that

of

rigid

motion

i.e.

when

the

camera

moves

through

a

static

scene

or

a

single

object

moves

rigidly

in

front

of

a

camera

Commonly

used

techniques

-LRB-

direct

methods

-RRB-

based

on

estimating

the

instantaneous

camera

egomotion

-LRB-

R

-LRB-

-RRB-

t

-RRB-

and

a

camera-centered

depth

Z

-LRB-

x

y

-RRB-

are

given

in

-LSB-

Horn

and

Weldon

Jr.

Hanna

Bergen

et

al.

-RSB-

This

has

the

disadvantage

of

only

being

valid

for

small

motions

of

requiring

a

calibrated

camera

and

of

sensitivity

problems

with

the

depth

estimates

Our

approach

is

based

on

a

projective

formulation

of

structure

from

motion

-LSB-

Hartley

et

al.

Faugeras

Mohr

et

al.

Szeliski

and

Kang

-RSB-

u

-LRB-

x

y

-RRB-

z

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

x

-LSB-

Bergen

et

al.

-RSB-

mitigate

the

Z

sensitivity

problem

by

estimating

Z

-LRB-

x

y

-RRB-

instead

Mixed

global

and

local

-LRB-

rigid

-RRB-

flow

estimation

projective

flow

-LSB-

treed/c

pp

treed-warp

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Example

of

projective

motion

estimation

v

-LRB-

x

y

-RRB-

z

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

y

-LRB-

-RRB-

This

formulation

is

valid

for

any

pinhole

camera

model

even

with

time

varying

internal

camera

parameters

The

local

shape

estimates

z

-LRB-

x

y

-RRB-

are

projective

depth

estimates

i.e.

the

-LRB-

x

y

z

-RRB-

coordinates

are

related

to

the

true

Euclidean

coordinates

-LRB-

X

Y

Z

-RRB-

through

some

projective

transformation

-LRB-

collineation

-RRB-

which

can

given

enough

views

be

recovered

from

the

projective

motion

estimates

-LSB-

Szeliski

-RSB-

Compared

to

the

usual

rigid

motion

formulation

we

have

to

estimate

more

global

parameters

-LRB-

instead

of

-RRB-

for

the

global

motion

so

we

might

be

concerned

with

an

increased

uncertainty

in

these

parameters

However

we

do

not

require

our

camera

to

be

calibrated

or

to

have

fixed

internal

parameters

We

can

also

deal

with

arbitrarily

large

displacements

and

non-smooth

motion

Furthermore

situations

in

which

either

the

global

motion

or

local

shape

estimates

are

poorly

recovered

-LRB-

e.g.

planar

scenes

pure

rotation

-RRB-

do

not

cause

any

problems

for

our

technique

A

special

case

of

the

rigid

motion

problem

is

when

we

are

given

a

pair

of

calibrated

images

is

an

ambiguity

in

the

shape

recovered

i.e.

we

can

add

any

plane

equation

to

the

z

coordinates

ax

by

cz

d

and

still

have

a

valid

solution

after

adjusting

the

ml

This

ambiguity

is

not

a

problem

for

gradient

descent

techniques

Mixed

global

and

local

-LRB-

rigid

-RRB-

flow

estimation

-LRB-

stereo

-RRB-

or

a

set

of

weakly

calibrated

images

-LRB-

only

the

epipoles

in

the

images

are

known

-LSB-

Hartley

et

al.

Faugeras

-RSB-

-RRB-

In

either

case

we

can

compute

the

ml

resulting

in

a

simple

estimation

problem

in

z

-LRB-

x

y

-RRB-

We

can

then

either

proceed

to

resample

-LRB-

rectify

-RRB-

the

images

so

that

corresponding

pixels

lie

along

scanlines

-LSB-

Hartley

and

Gupta

-RSB-

or

we

can

directly

work

with

the

original

images

proceeding

as

below

but

omitting

the

m

update

step

To

compute

the

global

and

local

flow

estimates

we

combine

several

of

the

approaches

developed

previously

in

the

paper

First

we

compute

the

flows

at

the

control

vertices

by

evaluating

-LRB-

-RRB-

at

the

vertex

locations

f

-LRB-

xj

yj

-RRB-

g

We

compute

the

gradients

and

Hessian

with

respect

to

the

global

motion

parameters

as

before

with

Tj

Dj

xj

yj

xjNuj

Dj

yj

Nuj

Dj

zj

zjNuj

Dj

xj

yj

xjNvj

Dj

yj

Nvj

Dj

zj

zjNvj

Dj

-LRB-

-RRB-

Nuj

xj

yj

zj

Nvj

xj

yj

zj

and

Dj

xj

yj

zj

The

derivatives

with

respect

to

the

depth

estimates

zj

are

gzj

@E

zj

@E

uj

uj

zj

@E

vj

uj

zj

guj

Dj

Dj

gvj

Dj

Dj

-LRB-

-RRB-

The

Hessian

matrix

Az

for

the

z

f

zjg

parameters

has

components

azjk

-LRB-

puv

z

j

-RRB-

TAjkpuv

z

k

with

puv

z

j

-LRB-

pu

z

j

pv

z

j

-RRB-

T

-LRB-

uj

zj

vj

zj

-RRB-

T

-LRB-

-RRB-

There

are

two

ways

at

this

point

to

estimate

the

m

and

z

parameters

We

can

take

simultaneous

steps

in

Dm

and

Dz

or

we

can

alternate

steps

in

Dm

and

Dz

The

former

approach

is

the

one

we

adopted

in

-LSB-

Szeliski

-RSB-

since

the

full

Hessian

matrix

had

already

been

computed

thus

enabling

a

regular

Levenberg-Marquardt

step

and

this

proved

to

have

faster

convergence

In

this

work

we

adopt

the

latter

approach

since

it

proved

to

be

simpler

to

implement

We

plan

to

compare

both

approaches

in

future

work

The

performance

of

our

rigid

motion

estimation

algorithm

on

a

sample

image

sequence

is

shown

in

Figure

As

can

be

seen

the

overall

direction

of

motion

-LRB-

the

epipolar

geometry

-RRB-

has

been

recovered

well

and

the

motion

estimates

look

reasonable

The

computed

depth

map

is

shown

in

grayscale

The

lower

right

flow

field

shows

the

local

-LRB-

general

flow

-RRB-

model

applied

to

the

same

image

pair

initialized

the

m

vector

in

this

example

to

a

horizontal

epipolar

geometry

If

unknown

the

epipolar

geometry

can

be

recovered

from

a

general

motion

field

-LSB-

Faugeras

Hartley

et

al.

Hartley

and

Gupta

Szeliski

-RSB-

Mixed

global

and

local

-LRB-

rigid

-RRB-

flow

estimation

rigid

flow

s

-LSB-

y

sri/s

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

-LRB-

a

-RRB-

-LRB-

b

-RRB-

local

flow

-LSB-

sri/s

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

Example

of

projective

-LRB-

rigid

-RRB-

motion

estimation

-LRB-

a

-RRB-

intensity

image

-LRB-

b

-RRB-

constrained

rigid

flow

-LRB-

c

-RRB-

recovered

depth

map

-LRB-

d

-RRB-

unconstrained

local

flow

-LRB-

for

comparison

-RRB-

Multiframe

flow

estimation

Multiframe

flow

estimation

Many

current

optical

flow

techniques

use

more

than

two

images

to

arrive

at

local

estimates

of

flow

This

is

particularly

true

of

spatio-temporal

filtering

approaches

-LSB-

Adelson

and

Bergen

Heeger

Fleet

and

Jepson

-RSB-

For

example

the

implementation

of

-LSB-

Fleet

and

Jepson

-RSB-

described

in

-LSB-

Barron

et

al.

-RSB-

uses

images

per

estimate

Stereo

matching

techniques

have

also

successfully

used

multiple

images

-LSB-

Bolles

et

al.

Matthies

et

al.

Okutomi

and

Kanade

-RSB-

Using

large

numbers

of

images

not

only

improves

the

accuracy

of

the

estimates

through

noise

averaging

but

it

can

also

disambiguate

between

possible

matches

-LSB-

Okutomi

and

Kanade

-RSB-

The

extension

of

our

local

global

and

mixed

motion

models

to

multiple

frames

is

straightforward

For

local

flow

we

assume

that

displacements

between

successive

images

and

a

base

image

are

known

scalar

multiples

of

each

other

ut

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

and

vt

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

-LRB-

-RRB-

i.e.

that

we

have

linear

flow

-LRB-

no

acceleration

-RRB-

We

then

minimize

the

overall

cost

function

E

-LRB-

fui

vig

-RRB-

X

t

X

i

-LSB-

It

-LRB-

xi

stui

yi

stvi

-RRB-

-LRB-

xi

yi

-RRB-

-RSB-

-LRB-

-RRB-

This

approach

is

similar

to

the

sum

of

sum

of

squared-distance

-LRB-

SSSD

-RRB-

algorithm

of

-LSB-

Okutomi

and

Kanade

-RSB-

except

that

we

represent

the

motion

with

a

subsampled

set

of

spline

coefficients

eliminating

the

need

for

overlapping

correlation

windows

The

modifications

to

the

flow

estimation

algorithm

are

minor

and

obvious

For

example

the

gradient

with

respect

to

the

local

flow

estimate

uj

in

-LRB-

-RRB-

becomes

guj

X

t

st

X

i

etiGxtiwij

with

eti

and

Gxti

being

the

same

as

ei

and

Gxi

with

replaced

by

It

Given

a

block

of

images

estimating

two

sets

of

flows

one

registered

with

the

first

image

and

another

registered

with

the

last

would

allow

us

to

do

bidirectional

prediction

for

motion-compensated

video

coding

-LSB-

Le

Gall

-RSB-

Examples

of

the

improvements

in

accuracy

due

to

multiframe

estimation

are

given

in

Section

the

most

common

case

a

uniform

temporal

sampling

-LRB-

st

t

-RRB-

is

assumed

but

this

is

not

strictly

necessary

-LSB-

Okutomi

and

Kanade

-RSB-

Experimental

results

For

global

motion

estimation

we

can

either

assume

that

the

motion

estimates

mt

are

related

by

a

known

transform

-LRB-

e.g.

uniform

camera

velocity

-RRB-

or

we

can

assume

an

independent

motion

estimate

for

each

frame

The

latter

situation

seems

more

useful

especially

in

multiframe

image

mosaicing

applications

The

motion

estimation

problem

in

this

case

decomposes

into

a

set

of

independent

global

motion

estimation

sub-problems

The

multiframe

global/local

motion

estimation

problem

is

more

interesting

Here

we

can

assume

that

the

global

motion

parameters

for

each

frame

mt

are

independent

but

that

the

local

shape

parameters

zj

do

not

vary

over

time

This

is

the

situation

when

we

analyze

multiple

arbitrary

views

of

a

rigid

scene

e.g.

in

the

multiframe

uncalibrated

stereo

problem

The

modifications

to

the

estimation

algorithm

are

also

straightforward

The

gradients

and

Hessian

with

respect

to

the

global

motion

parameters

mt

are

the

same

as

before

except

that

the

denominator

Dtj

is

now

different

for

each

frame

-LRB-

since

it

is

a

function

of

mt

-RRB-

The

derivatives

with

respect

to

the

depth

estimates

zj

are

computed

by

summing

over

all

frames

gzj

X

t

pu

z

tj

gutj

pv

z

tj

gvtj

-LRB-

-RRB-

where

the

pu

z

tj

and

pv

z

tj

-LRB-

which

depend

on

mt

-RRB-

and

the

gutj

and

gvtj

-LRB-

which

depend

on

It

-RRB-

are

different

for

each

frame

Note

that

we

can

no

longer

get

away

with

a

single

temporally

invariant

flow

field

gradient

-LRB-

guj

guj

-RRB-

-LRB-

another

way

to

see

this

is

that

the

epipolar

lines

in

each

image

can

be

arbitrary

-RRB-

Experimental

results

In

this

section

we

demonstrate

the

performance

of

our

algorithms

on

the

standard

motion

sequences

analyzed

in

-LSB-

Barron

et

al.

-RSB-

Some

of

the

images

in

these

sequences

have

already

been

shown

in

Figures

The

remaining

images

are

shown

in

Figures

We

follow

the

organization

of

-LSB-

Barron

et

al.

-RSB-

presenting

quantitative

results

on

synthetically

generated

sequences

first

followed

by

qualitative

results

on

real

motion

sequences

Tables

give

the

quantitative

results

of

our

algorithms

In

these

tables

the

top

two

rows

are

copied

from

-LSB-

Barron

et

al.

-RSB-

The

errors

are

reported

as

in

-LSB-

Barron

et

al.

-RSB-

i.e.

by

converting

flow

measurements

into

unit

vector

in

and

taking

the

angle

between

them

The

density

is

the

percentage

of

flow

estimates

reported

to

have

reliable

flow

estimates

The

computation

times

on

a

DEC

Model

AXP

for

these

algorithms

range

from

second

for

the

Experimental

results

Technique

Average

Error

Standard

Deviation

Density

Lucas

and

Kanade

-LRB-

no

thresholding

-RRB-

ffi

ffi

Fleet

and

Jepson

-LRB-

o

-RRB-

ffi

ffi

local

flow

-LRB-

n

s

L

b

-RRB-

ffi

ffi

local

flow

-LRB-

n

s

L

b

-RRB-

ffi

ffi

local

flow

-LRB-

n

s

L

b

-RRB-

ffi

ffi

local

flow

-LRB-

n

s

L

b

-RRB-

ffi

ffi

affine

flow

-LRB-

s

L

b

-RRB-

ffi

ffi

affine

flow

-LRB-

s

L

b

-RRB-

ffi

ffi

Table

Summary

of

Sinusoid

results

Sinusoid

image

-LRB-

single

level

iterations

-RRB-

to

seconds

for

the

Nasa

Sequence

-LRB-

three

levels

rigid

flow

iterations

per

level

-RRB-

From

the

nine

algorithms

in

-LSB-

Barron

et

al.

-RSB-

we

have

chosen

to

show

the

Lucas

and

Kanade

results

since

their

algorithm

most

closely

matches

ours

and

generally

gives

good

results

and

the

Fleet

and

Jepson

algorithm

since

it

generally

gave

the

best

results

The

most

salient

difference

between

our

-LRB-

local

-RRB-

algorithm

and

Lucas

and

Kanade

is

that

we

use

a

spline

representation

which

removes

the

need

for

overlapping

correlation

windows

and

is

therefore

much

more

computationally

efficient

The

biggest

difference

with

Fleet

and

Jepson

is

that

they

use

the

whole

image

sequence

-LRB-

frames

-RRB-

whereas

we

normally

use

only

two

-LRB-

multiframe

results

are

shown

in

Table

-RRB-

As

with

many

motion

estimation

algorithms

our

algorithms

require

the

selection

of

some

relevant

parameters

The

most

important

of

these

are

n

-LSB-

-RSB-

the

number

of

frames

s

-LSB-

-RSB-

the

step

between

frames

i.e.

consecutive

frames

every

other

frame

m

-LSB-

-RSB-

the

size

of

the

patch

-LRB-

width

and

height

pixels

per

patch

-RRB-

L

-LSB-

-RSB-

the

number

of

coarse-to-fine

levels

b

-LSB-

-RSB-

the

amount

of

initial

blurring

-LRB-

of

iterations

of

a

box

filter

-RRB-

Unless

mentioned

otherwise

we

used

the

default

values

shown

in

brackets

above

for

the

results

in

Tables

Bilinear

interpolation

was

used

for

the

flow

fields

The

simplest

motions

to

analyze

are

two

constant-translation

sequences

Sinusoid

and

Square

Experimental

results

Figure

Sinusoid

and

Square

sample

images

-LRB-

Figure

-RRB-

The

translations

in

these

sequences

are

-LRB-

-RRB-

and

-LRB-

-RRB-

pixels

per

frame

respectively

Our

local

flow

estimates

for

the

sinusoid

sequence

are

very

good

using

only

two

frames

-LRB-

Table

-RRB-

and

beat

all

other

algorithms

when

or

more

frames

are

used

For

this

sequence

we

use

a

single

level

and

no

blurring

and

take

a

-LRB-

s

-RRB-

frame

step

for

better

results

To

help

overcome

local

minima

for

the

multiframe

-LRB-

n

-RRB-

sequences

we

solve

a

series

of

easier

subproblems

-LSB-

Xu

et

al.

-RSB-

We

first

estimate

two-frame

motion

then

use

the

resulting

estimate

to

initialize

a

three-frame

estimator

etc.

Without

this

modification

performance

on

longer

-LRB-

e.g.

n

-RRB-

sequences

would

start

to

degrade

because

of

local

minima

The

global

affine

model

motion

estimator

performs

well

For

the

translating

square

-LRB-

Table

-RRB-

our

results

are

not

as

good

because

of

the

aperture

problem

but

with

additional

regularization

we

still

outperform

all

of

the

nine

algorithms

studied

in

-LSB-

Barron

et

al.

-RSB-

To

produce

the

sparse

flow

estimates

-LRB-

density

-RRB-

we

set

a

threshold

Te

on

the

minimum

eigenvalues

of

the

local

Hessian

matrices

Ajj

interpolated

over

the

whole

grid

-LRB-

this

selects

areas

where

both

components

of

the

motion

estimate

are

well

determined

-RRB-

The

affine

-LRB-

global

-RRB-

flow

for

the

square

sequence

works

extremely

well

outperforming

all

other

techniques

by

a

large

margin

The

sequences

Translating

Tree

and

Diverging

Tree

were

generated

using

a

real

image

-LRB-

Figure

Experimental

results

Technique

Average

Error

Standard

Deviation

Density

Lucas

and

Kanade

-LRB-

-RRB-

ffi

ffi

Fleet

and

Jepson

-LRB-

o

-RRB-

ffi

ffi

local

flow

-LRB-

Te

-RRB-

ffi

ffi

local

flow

-LRB-

s

Te

-RRB-

ffi

ffi

local

flow

-LRB-

s

Te

-RRB-

ffi

ffi

local

flow

-LRB-

s

-RRB-

ffi

ffi

affine

flow

ffi

ffi

Table

Summary

of

Square

results

Technique

Average

Error

Standard

Deviation

Density

Lucas

and

Kanade

-LRB-

-RRB-

ffi

ffi

Fleet

and

Jepson

-LRB-

o

-RRB-

ffi

ffi

local

flow

-LRB-

n

-RRB-

ffi

ffi

local

flow

-LRB-

n

-RRB-

ffi

ffi

local

flow

-LRB-

n

-RRB-

ffi

ffi

local

flow

-LRB-

n

-RRB-

ffi

ffi

affine

flow

ffi

ffi

Table

Summary

of

Translating

Tree

results

Experimental

results

Technique

Average

Error

Standard

Deviation

Density

Lucas

and

Kanade

-LRB-

-RRB-

ffi

ffi

Fleet

and

Jepson

-LRB-

o

-RRB-

ffi

ffi

local

flow

-LRB-

s

L

-RRB-

ffi

ffi

local

flow

-LRB-

s

L

-RRB-

ffi

ffi

affine

flow

ffi

ffi

Table

Summary

of

Diverging

Tree

results

Technique

Average

Error

Standard

Deviation

Density

Lucas

and

Kanade

-LRB-

-RRB-

ffi

ffi

Fleet

and

Jepson

-LRB-

o

-RRB-

ffi

ffi

local

flow

-LRB-

s

Te

-RRB-

ffi

ffi

local

flow

-LRB-

s

Te

-RRB-

ffi

ffi

local

flow

cropped

-LRB-

s

-RRB-

ffi

ffi

rigid

flow

cropped

-LRB-

s

-RRB-

ffi

ffi

Table

Summary

of

Yosemite

results

-RRB-

and

synthetic

-LRB-

global

-RRB-

motion

Our

results

on

the

translating

motion

sequence

-LRB-

Table

-RRB-

are

as

good

as

any

other

technique

for

the

local

algorithm

-LRB-

note

the

difference

in

density

between

our

results

and

the

previous

ones

-RRB-

and

outperform

all

techniques

for

the

affine

motion

model

even

though

we

are

just

using

two

frames

from

the

sequence

The

results

on

the

diverging

tree

sequence

are

good

for

the

local

flow

but

not

as

good

for

the

affine

flow

These

results

are

comparable

or

better

than

the

other

techniques

in

-LSB-

Barron

et

al.

-RSB-

which

produce

density

The

final

motion

sequence

for

which

quantitative

results

are

available

is

Yosemite

-LRB-

Figure

and

Table

-RRB-

The

images

in

this

sequence

were

generated

by

Lynn

Quam

using

his

texture

mapping

algorithm

applied

to

an

aerial

photograph

registered

with

a

digital

terrain

model

There

is

significant

occlusion

and

temporal

aliasing

and

the

fractal

clouds

move

independently

from

the

terrain

Our

results

on

this

more

realistic

sequence

are

better

than

any

of

the

techniques

in

-LSB-

Barron

et

al.

-RSB-

even

though

we

again

only

use

two

images

As

expected

the

quality

of

the

results

depends

on

the

threshold

Te

used

to

produce

sparse

flow

estimates

i.e.

there

is

a

tradeoff

between

Experimental

results

local

flow

-LSB-

yos/s

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Yosemite

sample

image

and

flow

-LRB-

unthresholded

-RRB-

the

density

of

the

estimates

and

their

quality

We

also

ran

our

algorithm

on

just

the

lower

-LRB-

out

of

-RRB-

rows

of

the

images

sequence

The

dense

-LRB-

unthresholded

-RRB-

estimates

are

comparable

to

the

thresholded

full-frame

estimates

Unfortunately

the

results

using

the

rigid

motion

model

were

slightly

worse

To

conclude

our

experimental

section

we

show

results

on

some

real

motion

sequences

for

which

no

ground

truth

data

is

available

The

SRI

Trees

results

have

already

been

presented

in

Figure

for

both

rigid

and

local

-LRB-

general

-RRB-

flow

Figure

shows

the

NASA

Sequence

in

which

the

camera

moves

forward

in

a

rigid

scene

-LRB-

there

is

significant

aliasing

-RRB-

The

motion

estimates

look

quite

reasonable

as

does

the

associated

depth

map

-LRB-

not

shown

-RRB-

Figure

shows

the

sparse

flow

computed

for

the

Rubik

Cube

sequence

-LRB-

the

dense

flows

were

shown

in

Figure

-RRB-

The

areas

with

texture

and/or

corners

produce

the

most

reliable

flow

estimates

Finally

the

results

on

the

Hamburg

Taxi

are

shown

in

Figure

where

the

independent

motion

of

the

three

moving

cars

can

be

clearly

distinguished

Overall

these

results

are

comparable

or

better

than

those

shown

in

-LSB-

Barron

et

al.

-RSB-

Much

work

remains

to

be

done

in

the

experimental

evaluation

of

our

algorithms

In

addition

to

systematically

studying

the

effects

of

the

parameters

n

s

m

L

and

b

-LRB-

introduced

previously

-RRB-

we

plan

to

study

the

effects

of

different

spline

interpolation

functions

the

effects

of

different

preconditioners

and

the

usefulness

of

using

conjugate

gradient

descent

this

sequence

and

for

the

Yosemite

sequence

we

initialized

the

m

vector

to

a

forward

looming

motion

Experimental

results

s.

pp

dev/null

-RSB-

rigid

flow

-LSB-

p

y

nasa/s

pp

nasa/s

pp

dev/null

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

NASA

Sequence

rigid

flow

pp

-RSB-

sparse

local

flow

-LSB-

p

rubik/s

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Rubik

Cube

sequence

sparse

flow

Discussion

local

flow

level

-LSB-

taxi/s

pp

taxi/s

pp

-RSB-

Image

size

x

Subsampled

by

scaled

by

Figure

Hamburg

Taxi

sequence

dense

local

flow

Discussion

The

spline-based

motion

estimation

algorithms

introduced

in

this

paper

are

a

hybrid

of

local

optic

flow

algorithms

and

global

motion

estimators

utilizing

the

best

features

of

both

approaches

Like

other

local

methods

we

can

produce

detailed

local

flow

estimates

which

perform

well

in

the

presence

of

independently

moving

objects

and

large

depth

variations

Unlike

correlationbased

methods

however

we

do

not

assume

a

local

translational

model

in

each

correlation

window

Instead

the

pixel

motion

within

each

of

our

patches

can

model

affine

or

even

more

complex

motions

-LRB-

e.g.

bilinear

interpolation

of

the

four

spline

control

vertices

can

provide

an

approximation

to

local

projective

flow

-RRB-

This

is

especially

important

when

we

analyze

extended

motion

sequences

where

local

intensity

patterns

can

deform

significantly

Our

technique

can

be

viewed

as

a

generalization

of

affine

patch

trackers

-LSB-

Rehg

and

Witkin

Shi

and

Tomasi

-RSB-

where

the

patch

corners

are

stitched

together

over

the

whole

image

Another

major

difference

between

our

spline-based

approach

and

correlation-based

approaches

is

in

computational

efficiency

Each

pixel

in

our

approach

only

contributes

its

error

to

the

spline

control

vertices

influencing

its

displacement

whereas

in

correlation-based

approaches

each

pixel

contributes

to

overlapping

windows

Furthermore

operations

such

as

inverting

the

local

Hessian

or

computing

the

contribution

to

a

global

model

only

occur

at

the

spline

control

vertices

thereby

providing

an

O

-LRB-

-RRB-

speedup

over

correlation-based

techniques

For

typically-sized

patches

-LRB-

m

-RRB-

this

can

be

significant

The

price

we

pay

for

this

efficiency

is

a

slight

decrease

in

the

Discussion

resolution

of

the

computed

flow

field

especially

when

compared

to

locally

adaptive

widows

-LSB-

Okutomi

and

Kanade

-RSB-

-LRB-

which

are

extremely

computationally

demanding

-RRB-

However

since

window-based

approaches

produce

highly

correlated

estimates

anyway

we

do

not

expect

this

difference

to

be

significant

Compared

to

spatio-temporal

filtering

approaches

we

see

a

similar

improvement

in

computational

efficiency

Separable

filters

can

reduce

the

complexity

of

computing

the

required

local

features

from

O

-LRB-

-RRB-

to

O

-LRB-

m

-RRB-

but

these

operations

must

still

be

performed

at

each

pixel

Furthermore

a

large

number

of

differently

tuned

filters

are

normally

used

Since

the

final

estimates

are

highly

correlated

anyway

it

just

makes

more

computational

sense

to

perform

the

calculations

on

a

sparser

grid

as

we

do

Because

our

spline-based

motion

representation

already

has

a

smoothness

constraint

built

in

regularization

which

requires

many

iterations

to

propagate

local

constraints

is

not

usually

necessary

If

we

desire

longer-range

smoothness

constraints

regularization

can

easily

be

added

to

our

framework

Having

fewer

free

variables

in

our

estimation

framework

leads

to

faster

convergence

when

iteration

is

necessary

to

propagate

such

constraints

Turning

to

global

motion

estimation

our

motion

model

for

planar

surface

flow

can

handle

arbitrarily

large

motions

and

displacements

unlike

the

instantaneous

model

of

-LSB-

Bergen

et

al.

-RSB-

We

see

this

as

an

advantage

in

many

situations

e.g.

in

compositing

multiple

views

of

planar

surfaces

-LSB-

Szeliski

-RSB-

Furthermore

our

approach

does

not

require

the

camera

to

be

calibrated

and

can

handle

temporally-varying

internal

camera

parameters

While

our

flow

field

is

not

linear

in

the

unknown

parameters

this

is

not

significant

since

the

overall

problem

is

non-linear

and

requires

iteration

Our

mixed

global/local

-LRB-

rigid

body

-RRB-

model

shares

similar

advantages

over

previously

developed

direct

methods

it

does

not

require

camera

calibration

and

can

handle

time-varying

camera

parameters

and

arbitrary

camera

displacements

Furthermore

experimental

evidence

from

some

related

structure

from

motion

research

-LSB-

Szeliski

-RSB-

suggests

that

our

projective

formulation

of

structure

and

motion

converges

more

quickly

than

traditional

Euclidean

formulations

Our

experimental

results

suggest

that

our

techniques

are

competitive

in

quality

with

the

best

currently

available

motion

estimators

examined

in

-LSB-

Barron

et

al.

-RSB-

especially

when

additional

regularization

is

used

A

more

complete

experimental

evaluation

remain

to

be

done

Future

work

and

Conclusions

Future

work

and

Conclusions

To

improve

the

performance

of

our

algorithm

on

difficult

scenes

with

repetitive

textures

we

are

planning

to

add

local

search

i.e.

to

evaluate

several

possible

displacements

instead

of

just

relying

on

gradient

descent

-LSB-

Anandan

Singh

-RSB-

We

also

plan

to

study

hierarchical

basis

functions

as

an

alternative

to

coarse-to-fine

estimation

-LSB-

Szeliski

-RSB-

This

approach

has

proven

to

be

very

effective

in

other

vision

problems

such

as

surface

reconstruction

and

shape

from

shading

where

smoothness

or

consistency

constraints

need

to

be

propagated

over

large

distances

-LSB-

Szeliski

-RSB-

It

is

unclear

however

if

this

is

a

significant

problem

in

motion

estimation

especially

with

richly

textured

scenes

Finally

we

plan

to

address

the

problems

of

discontinuities

and

occlusions

-LSB-

Geiger

et

al.

-RSB-

which

must

be

resolved

for

any

motion

analysis

system

to

be

truly

useful

In

terms

of

applications

we

are

currently

using

our

global

flow

estimator

to

register

multiple

images

e.g.

to

align

successive

microscope

slice

images

or

to

composite

pieces

of

flat

scenes

such

as

whiteboards

seen

with

a

video

camera

-LSB-

Szeliski

-RSB-

We

plan

to

use

our

local/global

model

to

extract

projective

scene

geometry

from

multiple

images

We

would

also

like

to

study

the

performance

of

our

local

motion

estimator

in

extended

motion

sequences

as

a

parallel

feature

tracker

i.e.

by

using

only

estimates

with

high

local

confidence

Finally

we

would

like

to

test

our

spline-based

motion

estimates

as

predictors

for

motion-compensated

video

coding

as

an

alternative

to

block-structured

predictors

such

as

MPEG

To

summarize

spline-based

image

registration

combines

the

best

features

of

local

motion

models

and

global

-LRB-

parametric

-RRB-

motion

models

The

size

of

the

spline

patches

and

the

order

of

spline

interpolation

can

be

used

to

vary

smoothly

between

these

two

extremes

The

resulting

algorithm

is

more

computationally

efficient

than

correlation-based

or

spatio-temporal

filter-based

techniques

while

providing

estimates

of

comparable

quality

Purely

global

and

mixed

local/global

estimators

have

also

been

developed

based

on

this

representation

for

those

situations

where

a

more

specific

motion

model

can

be

used

References

-LSB-

Adelson

and

Bergen

-RSB-

E.

H.

Adelson

and

J.

R.

Bergen

Spatiotemporal

energy

models

for

the

perception

of

motion

Journal

of

the

Optical

Society

of

America

A

-LRB-

-RRB-

February

Future

work

and

Conclusions

-LSB-

Amit

-RSB-

Y.

Amit

A

non-linear

variational

problem

for

image

matching

unpublished

manuscript

-LRB-

from

Newton

Institute

-RRB-

-LSB-

Anandan

-RSB-

P.

Anandan

A

computational

framework

and

an

algorithm

for

the

measurement

of

visual

motion

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Bajcsy

and

Broit

-RSB-

R.

Bajcsy

and

C.

Broit

Matching

of

deformed

images

In

Sixth

International

Conference

on

Pattern

Recognition

-LRB-

ICPR

-RRB-

pages

IEEE

Computer

Society

Press

Munich

Germany

October

-LSB-

Bajcsy

and

Kovacic

-RSB-

R.

Bajcsy

and

S.

Kovacic

Multiresolution

elastic

matching

Computer

Vision

Graphics

and

Image

Processing

-LSB-

Barnard

and

Fischler

-RSB-

S.

T.

Barnard

and

M.

A.

Fischler

Computational

stereo

Computing

Surveys

-LRB-

-RRB-

December

-LSB-

Barron

et

al.

-RSB-

J.

L.

Barron

D.

J.

Fleet

and

S.

S.

Beauchemin

Performance

of

optical

flow

techniques

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Beier

and

Neely

-RSB-

T.

Beier

and

S.

Neely

Feature-based

image

metamorphosis

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Bergen

et

al.

-RSB-

J.

R.

Bergen

P.

Anandan

K.

J.

Hanna

and

R.

Hingorani

Hierarchical

modelbased

motion

estimation

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Santa

Margherita

Liguere

Italy

May

-LSB-

Blake

et

al.

-RSB-

A.

Blake

R.

Curwen

and

A.

Zisserman

A

framework

for

spatio-temporal

control

in

the

tracking

of

visual

contour

International

Journal

of

Computer

Vision

-LRB-

-RRB-

October

-LSB-

Bolles

et

al.

-RSB-

R.

C.

Bolles

H.

H.

Baker

and

D.

H.

Marimont

Epipolar-plane

image

analysis

An

approach

to

determining

structure

from

motion

International

Journal

of

Computer

Vision

-LSB-

Brown

-RSB-

L.

G.

Brown

A

survey

of

image

registration

techniques

Computing

Surveys

-LRB-

-RRB-

December

-LSB-

Burr

-RSB-

D.

J.

Burr

A

dynamic

model

for

image

registration

Computer

Graphics

and

Image

Processing

-LRB-

-RRB-

February

-LSB-

Burt

and

Adelson

-RSB-

P.

J.

Burt

and

E.

H.

Adelson

The

Laplacian

pyramid

as

a

compact

image

code

IEEE

Transactions

on

Communications

-LRB-

-RRB-

April

Future

work

and

Conclusions

-LSB-

Carlbom

et

al.

-RSB-

I.

Carlbom

D.

Terzopoulos

and

K.

M.

Harris

Reconstructing

and

visualizing

models

of

neuronal

dendrites

In

N.

M.

Patrikalakis

editor

Scientific

Visualization

of

Physical

Phenomena

pages

Springer-Verlag

New

York

-LSB-

Dhond

and

Aggarwal

-RSB-

U.

R.

Dhond

and

J.

K.

Aggarwal

Structure

from

stereo?a

review

IEEE

Transactions

on

Systems

Man

and

Cybernetics

-LRB-

-RRB-

November/December

-LSB-

Dreschler

and

Nagel

-RSB-

L.

Dreschler

and

H.-H

Nagel

Volumetric

model

and

trajectory

of

a

moving

car

derived

from

monocular

tv

frame

sequences

of

a

stree

scene

Computer

Graphics

and

Image

Processing

-LSB-

Enkelmann

-RSB-

W.

Enkelmann

Investigations

of

multigrid

algorithms

for

estimation

of

optical

flow

fields

in

image

sequences

Computer

Vision

Graphics

and

Image

Processing

-LSB-

Faugeras

-RSB-

O.

D.

Faugeras

What

can

be

seen

in

three

dimensions

with

an

uncalibrated

stereo

rig

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Santa

Margherita

Liguere

Italy

May

-LSB-

Fleet

and

Jepson

-RSB-

D.

Fleet

and

A.

Jepson

Computation

of

component

image

velocity

from

local

phase

information

International

Journal

of

Computer

Vision

-LSB-

Geiger

et

al.

-RSB-

D.

Geiger

B.

Ladendorf

and

A.

Yuille

Occlusions

and

binocular

stereo

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

SpringerVerlag

Santa

Margherita

Liguere

Italy

May

-LSB-

Gennert

-RSB-

M.

A.

Gennert

Brightness-based

stereo

matching

In

Second

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Tampa

Florida

December

-LSB-

Goshtasby

-RSB-

A.

Goshtasby

Piecewise

linear

mapping

functions

for

image

registration

Pattern

Recognition

-LRB-

-RRB-

-LSB-

Goshtasby

-RSB-

A.

Goshtasby

Image

registration

by

local

approximation

methods

Image

and

Vision

Computing

-LRB-

-RRB-

November

-LSB-

Hanna

-RSB-

K.

J.

Hanna

Direct

multi-resolution

estimation

of

ego-motion

and

structure

from

motion

In

IEEE

Workshop

on

Visual

Motion

pages

IEEE

Computer

Society

Press

Princeton

New

Jersey

October

Future

work

and

Conclusions

-LSB-

Hartley

and

Gupta

-RSB-

R.

Hartley

and

R.

Gupta

Computing

matched-epipolar

projections

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

New

York

New

York

June

-LSB-

Hartley

et

al.

-RSB-

R.

Hartley

R.

Gupta

and

T.

Chang

Stereo

from

uncalibrated

cameras

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Press

Champaign

Illinois

June

-LSB-

Heeger

-RSB-

D.

J.

Heeger

Optical

flow

from

spatiotemporal

filters

In

First

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

London

England

June

-LSB-

Hildreth

-RSB-

E.

C.

Hildreth

Computing

the

velocity

field

along

contours

In

N.

I.

Badler

and

J.

K.

Tsotsos

editors

Motion

Representation

and

Perception

pages

NorthHolland

New

York

New

York

-LSB-

Horn

and

Weldon

Jr.

-RSB-

B.

K.

P.

Horn

and

E.

J

Weldon

Jr.

Direct

methods

for

recovering

motion

International

Journal

of

Computer

Vision

-LRB-

-RRB-

-LSB-

Horn

and

Schunck

-RSB-

B.

K.

P.

Horn

and

B.

G.

Schunck

Determining

optical

flow

Artificial

Intelligence

-LSB-

Kass

et

al.

-RSB-

M.

Kass

A.

Witkin

and

D.

Terzopoulos

Snakes

Active

contour

models

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Koenderink

and

van

Doorn

-RSB-

J.

J.

Koenderink

and

A.

J.

van

Doorn

Affine

structure

from

motion

Journal

of

the

Optical

Society

of

America

A

-LSB-

Le

Gall

-RSB-

D.

Le

Gall

MPEG

A

video

compression

standard

for

multimedia

applications

Communications

of

the

ACM

-LRB-

-RRB-

April

-LSB-

Lucas

-RSB-

B.

D.

Lucas

Generalized

Image

Matching

by

the

Method

of

Differences

PhD

thesis

Carnegie

Mellon

University

July

-LSB-

Lucas

and

Kanade

-RSB-

B.

D.

Lucas

and

T.

Kanade

An

iterative

image

registration

technique

with

an

application

in

stereo

vision

In

Seventh

International

Joint

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

pages

Vancouver

-LSB-

Manmatha

and

Oliensis

-RSB-

R.

Manmatha

and

J.

Oliensis

Measuring

the

affine

transform

I

Scale

and

Rotation

Technical

Report

University

of

Massachussets

Amherst

Massachussets

Future

work

and

Conclusions

-LSB-

Matthies

et

al.

-RSB-

L.

H.

Matthies

R.

Szeliski

and

T.

Kanade

Kalman

filter-based

algorithms

for

estimating

depth

from

image

sequences

International

Journal

of

Computer

Vision

-LSB-

Menet

et

al.

-RSB-

S.

Menet

P.

Saint-Marc

and

G.

Medioni

B-snakes

implementation

and

applications

to

stereo

In

Image

Understanding

Workshop

pages

Morgan

Kaufmann

Publishers

Pittsburgh

Pennsylvania

September

-LSB-

Mohr

et

al.

-RSB-

R.

Mohr

L.

Veillon

and

L.

Quan

Relative

reconstruction

using

multiple

uncalibrated

images

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

New

York

New

York

June

-LSB-

Nagel

-RSB-

H.-H

Nagel

On

the

estimation

of

optical

flow

Relations

between

different

approaches

and

some

new

results

Artificial

Intelligence

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

locally

adaptive

window

for

signal

matching

International

Journal

of

Computer

Vision

-LRB-

-RRB-

April

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

multiple

baseline

stereo

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

April

-LSB-

Poggio

et

al.

-RSB-

T.

Poggio

V.

Torre

and

C.

Koch

Computational

vision

and

regularization

theory

Nature

-LRB-

-RRB-

September

-LSB-

Press

et

al.

-RSB-

W.

H.

Press

B.

P.

Flannery

S.

A.

Teukolsky

and

W.

T.

Vetterling

Numerical

Recipes

in

C

The

Art

of

Scientific

Computing

Cambridge

University

Press

Cambridge

England

second

edition

-LSB-

Quam

-RSB-

L.

H.

Quam

Hierarchical

warp

stereo

In

Image

Understanding

Workshop

pages

Science

Applications

International

Corporation

New

Orleans

Louisiana

December

-LSB-

Rehg

and

Witkin

-RSB-

J.

Rehg

and

A.

Witkin

Visual

tracking

with

deformation

models

In

IEEE

International

Conference

on

Robotics

and

Automation

pages

IEEE

Computer

Society

Press

Sacramento

California

April

-LSB-

Sethi

and

Jain

-RSB-

I.

K.

Sethi

and

R.

Jain

Finding

trajectories

of

feature

points

in

a

monocular

image

sequence

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

PAMI

-LRB-

-RRB-

January

-LSB-

Shi

and

Tomasi

-RSB-

J.

Shi

and

C.

Tomasi

Good

features

to

track

In

IEEE

Computer

Society

Future

work

and

Conclusions

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

Seattle

Washington

June

-LSB-

Simoncelli

et

al.

-RSB-

E.

P.

Simoncelli

E.

H.

Adelson

and

D.

J.

Heeger

Probability

distributions

of

optic

flow

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Press

Maui

Hawaii

June

-LSB-

Singh

-RSB-

A.

Singh

An

estimation-theoretic

framework

for

image-flow

computation

In

Third

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Osaka

Japan

December

-LSB-

Szeliski

-RSB-

R.

Szeliski

Bayesian

Modeling

of

Uncertainty

in

Low-Level

Vision

Kluwer

Academic

Publishers

Boston

Massachusetts

-LSB-

Szeliski

-RSB-

R.

Szeliski

Fast

surface

interpolation

using

hierarchical

basis

functions

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

June

-LSB-

Szeliski

-RSB-

R.

Szeliski

Fast

shape

from

shading

CVGIP

Image

Understanding

-LRB-

-RRB-

March

-LSB-

Szeliski

-RSB-

R.

Szeliski

Image

Mosaicing

for

Tele-Reality

Applications

Technical

Report

Digital

Equipment

Corporation

Cambridge

Research

Lab

June

-LSB-

Szeliski

-RSB-

R.

Szeliski

A

least

squares

approach

to

affine

and

projective

structure

and

motion

recovery

-LRB-

in

preparation

-RRB-

-LSB-

Szeliski

and

Kang

-RSB-

R.

Szeliski

and

S.

B.

Kang

Recovering

shape

and

motion

from

image

streams

using

nonlinear

least

squares

Journal

of

Visual

Communication

and

Image

Representation

-LRB-

-RRB-

March

-LSB-

Terzopoulos

-RSB-

D.

Terzopoulos

Regularization

of

inverse

visual

problems

involving

discontinuities

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

July

-LSB-

Tomasi

and

Kanade

-RSB-

C.

Tomasi

and

T.

Kanade

Shape

and

motion

from

image

streams

under

orthography

A

factorization

method

International

Journal

of

Computer

Vision

-LRB-

-RRB-

November

-LSB-

Witkin

et

al.

-RSB-

A.

Witkin

D.

Terzopoulos

and

M.

Kass

Signal

matching

through

scale

space

International

Journal

of

Computer

Vision

Future

work

and

Conclusions

-LSB-

Wolberg

-RSB-

G.

Wolberg

Digital

Image

Warping

IEEE

Computer

Society

Press

Los

Alamitos

California

-LSB-

Xu

et

al.

-RSB-

G.

Xu

S.

Tsuji

and

M.

Asada

A

motion

stereo

method

based

on

coarse-to-fine

control

strategy

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

PAMI

-LRB-

-RRB-

March

-LSB-

Zheng

and

Chellappa

-RSB-

Q.

Zheng

and

R.

Chellappa

Automatic

Feature

Point

Extraction

and

Tracking

in

Image

Sequences

for

Arbitrary

Camera

Motion

Technical

Report

Computer

Vision

Laboratory

Center

for

Automation

Research

University

of

Maryland

June

global motion models
motion analysis
multiframe image analysis
optical flow
splines
structure from motion

direct motion estimation
hierarchical image registration