Motion

Estimation

with

Quadtree

Splines

Richard

Szeliski

and

Heung-Yeung

Shum

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

March

Digital

Equipment

Corporation

has

four

research

facilities

the

Systems

Research

Center

and

the

Western

Research

Laboratory

both

in

Palo

Alto

California

the

Paris

Research

Laboratory

in

Paris

and

the

Cambridge

Research

Laboratory

in

Cambridge

Massachusetts

The

Cambridge

laboratory

became

operational

in

and

is

located

at

One

Kendall

Square

near

MIT

CRL

engages

in

computing

research

to

extend

the

state

of

the

computing

art

in

areas

likely

to

be

important

to

Digital

and

its

customers

in

future

years

CRL

's

main

focus

is

applications

technology

that

is

the

creation

of

knowledge

and

tools

useful

for

the

preparation

of

important

classes

of

applications

CRL

Technical

Reports

can

be

ordered

by

electronic

mail

To

receive

instructions

send

a

mes

sage

to

one

of

the

following

addresses

with

the

word

help

in

the

Subject

line

On

Digital

's

EASYnet

CRL

TECHREPORTS

On

the

Internet

techreports@crl.dec.com

This

work

may

not

be

copied

or

reproduced

for

any

commercial

purpose

Permission

to

copy

without

payment

is

granted

for

non?profit

educational

and

research

purposes

provided

all

such

copies

include

a

notice

that

such

copy

ing

is

by

permission

of

the

Cambridge

Research

Lab

of

Digital

Equipment

Corporation

an

acknowledgment

of

the

authors

to

the

work

and

all

applicable

portions

of

the

copyright

notice

The

Digital

logo

is

a

trademark

of

Digital

Equipment

Corporation

Cambridge

Research

Laboratory

One

Kendall

Square

Cambridge

Massachusetts

TM

Motion

Estimation

with

Quadtree

Splines

Richard

Szeliski

and

Heung-Yeung

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

March

Abstract

This

paper

presents

a

motion

estimation

algorithm

based

on

a

new

multiresolution

representation

the

quadtree

spline

This

representation

describes

the

motion

field

as

a

collection

of

smoothly

connected

patches

of

varying

size

where

the

patch

size

is

automatically

adapted

to

the

complexity

of

the

underlying

motion

The

topology

of

the

patches

is

determined

by

a

quadtree

data

structure

and

both

split

and

merge

techniques

are

developed

for

estimating

this

spatial

subdivision

The

quadtree

spline

is

implemented

using

another

novel

representation

the

adaptive

hierarchical

basis

spline

and

combines

the

advantages

of

adaptively-sized

correlation

windows

with

the

speedups

obtained

with

hierarchical

basis

preconditioners

Results

are

presented

on

some

standard

motion

sequences

c

Digital

Equipment

Corporation

All

rights

reserved

Robotics

Institute

Carnegie

Mellon

University

Pittsburgh

PA

Contents

i

Contents

Introduction

Previous

work

General

problem

formulation

Spline-based

flow

estimation

Function

minimization

Hierarchical

basis

splines

Quadtree

-LRB-

adaptive

resolution

-RRB-

splines

Subdivision

strategy

A

Bayesian

interpretation

Experimental

results

Extensions

Discussion

and

Conclusions

ii

LIST

OF

TABLES

List

of

Figures

Displacement

spline

the

spline

control

vertices

f

-LRB-

uj

vj

-RRB-

g

are

shown

as

circles

-LRB-

ffi

-RRB-

and

the

pixel

displacements

f

-LRB-

ui

vi

-RRB-

g

are

shown

as

pluses

-LRB-

-RRB-

Example

of

general

flow

computation

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

-LRB-

d

-RRB-

flow

estimates

for

m

and

Multiresolution

pyramid

Hierarchical

basis

preconditioned

conjugate

gradient

algorithm

Square

sample

image

and

convergence

plot

Quadtree

associated

with

spline

function

and

potential

cracks

in

quadtree

spline

Quadtree

spline

motion

estimation

-LRB-

Two

Discs

-LRB-

SRI

Trees

-RRB-

sequence

-RRB-

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

true

flow

-LRB-

c

-RRB-

split

technique

-LRB-

d

-RRB-

merge

technique

Flow

error

ku

u?k

and

residual

normal

flow

kuNi

k

for

Two

Discs

-LRB-

SRI

Trees

-RRB-

sequence

Note

how

most

of

the

errors

are

concentrated

near

the

motion

disconti

nuities

and

especially

the

disoccluded

region

in

the

center

Hamburg

Taxi

sequence

estimated

quadtree

and

estimated

flow

merging

s

patches

in

a

pyramid

Flower

Garden

sequence

estimated

quadtree

and

estimated

flow

merging

s

patches

in

a

pyramid

Michael

Otte

's

sequence

estimated

quadtree

and

estimated

flow

merging

s

patches

in

a

pyramid

List

of

Tables

Summary

of

Two

Discs

-LRB-

SRI

Trees

-RRB-

results

Introduction

Introduction

One

of

the

fundamental

tradeoffs

in

designing

motion

estimation

and

stereo

matching

algorithms

is

selecting

the

size

of

the

windows

or

filters

to

be

used

in

comparing

portions

of

corresponding

images

Using

larger

windows

leads

to

better

noise

immunity

through

averaging

and

can

also

disambiguate

potential

matches

in

areas

of

weak

texture

or

potential

aperture

problems

However

larger

windows

fail

where

they

straddle

motion

or

depth

discontinuities

or

in

general

where

the

motion

or

disparity

varies

significantly

within

the

window

Many

techniques

have

been

devised

to

deal

with

this

problem

e.g.

using

adaptively-sized

windows

in

stereo

matching

In

this

paper

we

present

a

technique

for

recursively

subdividing

an

image

into

square

patches

of

varying

size

and

then

matching

these

patches

to

subsequent

frames

in

a

way

which

preserves

inter-patch

motion

continuity

Our

technique

is

an

extension

of

the

spline-based

image

registration

technique

presented

in

-LSB-

Szeliski

and

Coughlan

-RSB-

and

thus

has

the

same

advantages

when

compared

to

correlation-based

approaches

i.e.

lower

computational

cost

and

the

ability

to

handle

large

image

deformations

As

a

first

step

we

show

how

using

hierarchical

basis

splines

instead

of

regular

splines

can

lead

to

faster

convergence

and

qualitatively

perform

a

smoothing

function

similar

to

regularization

Then

we

show

how

selectively

setting

certain

nodes

in

the

hierarchical

basis

to

zero

leads

to

an

adaptive

hierarchical

basis

We

can

use

this

idea

to

build

a

spline

defined

over

a

quadtree

domain

i.e.

a

quadtree

spline

To

determine

the

size

of

the

patches

in

our

adaptive

basis

i.e.

the

shape

of

the

quadtree

we

develop

both

split

and

merge

techniques

based

on

the

residual

errors

in

the

current

optical

flow

estimates

While

this

paper

deals

primarily

with

motion

estimation

-LRB-

also

known

as

image

registration

or

optical

flow

computation

-RRB-

the

techniques

developed

here

can

equally

well

be

applied

to

stereo

matching

In

our

framework

we

view

stereo

as

a

special

case

of

motion

estimation

where

the

epipolar

geometry

-LRB-

corresponding

lines

-RRB-

are

known

thus

reducing

a

two-dimensional

search

space

at

each

pixel

to

a

one-dimensional

space

Our

techniques

can

also

be

used

as

part

of

a

direct

method

which

simultaneously

solves

for

projective

depth

and

camera

motion

-LSB-

Szeliski

and

Coughlan

-RSB-

The

adaptive

hierarchical

basis

splines

developed

in

this

paper

are

equivalent

to

adaptively

subdividing

global

parametric

motion

regions

while

maintaining

continuity

between

adjacent

patches

We

can

therefore

implement

a

continuum

of

motion

models

ranging

from

a

single

global

-LRB-

e.g.

affine

-RRB-

motion

all

the

way

to

a

completely

general

local

motion

as

warranted

by

the

data

in

a

given

image

sequence

By

examining

the

local

certainty

in

the

flow

computation

we

can

also

use

our

Previous

work

algorithm

as

a

parallel

feature

tracker

for

very

long

motion

sequences

where

image

deformations

may

be

significant

-LSB-

Szeliski

et

al.

-RSB-

The

motion

estimation

algorithms

developed

in

this

paper

can

be

used

in

a

number

of

applications

Examples

include

motion

compensation

for

video

compression

the

extraction

of

scene

geometry

and

camera

motion

robot

navigation

and

the

registration

of

multiple

images

e.g.

for

medical

applications

Feature

tracking

algorithms

based

on

our

techniques

can

be

used

in

human

interface

applications

such

as

gaze

tracking

or

expression

detection

in

addition

to

classical

robotics

applications

The

remainder

of

the

paper

is

structured

as

follows

Section

presents

a

review

of

relevant

previous

work

Section

gives

the

general

problem

formulations

for

image

registration

Section

reviews

the

spline-based

motion

estimation

algorithm

Section

shows

how

hierarchical

basis

functions

can

be

used

to

accelerate

and

regularize

spline-based

flow

estimation

Section

presents

our

novel

quadtree

splines

and

discusses

how

their

shape

can

be

estimated

using

both

split

and

merge

techniques

Section

discusses

the

relationship

of

adaptive

hierarchical

basis

splines

to

multiscale

Markov

Random

Fields

Section

presents

experimental

results

based

on

some

commonly

used

motion

test

sequences

We

close

with

a

comparison

of

our

approach

to

previous

algorithms

and

a

discussion

of

future

work

Previous

work

Motion

estimation

has

long

been

one

of

the

most

actively

studied

areas

of

computer

vision

and

image

processing

-LSB-

Aggarwal

and

Nandhakumar

Brown

-RSB-

Motion

estimation

algorithms

include

optical

flow

-LRB-

general

motion

-RRB-

estimators

global

parametric

motion

estimators

constrained

motion

estimators

-LRB-

direct

methods

-RRB-

stereo

and

multiframe

stereo

hierarchical

-LRB-

coarse-tofine

-RRB-

methods

feature

trackers

and

feature-based

registration

techniques

We

will

use

this

rough

taxonomy

to

briefly

review

previous

work

while

recognizing

that

these

algorithms

overlap

and

that

many

algorithms

use

ideas

from

several

of

these

categories

The

general

motion

estimation

problem

is

often

called

optical

flow

recovery

-LSB-

Hornand

Schunck

-RSB-

This

involves

estimating

an

independent

displacement

vector

for

each

pixel

in

an

image

Approaches

to

this

problem

include

gradient-based

approaches

based

on

the

brightness

constraint

-LSB-

Horn

and

Schunck

Lucas

and

Kanade

Nagel

-RSB-

correlation-based

techniques

such

as

the

sum

of

squared

differences

-LRB-

SSD

-RRB-

-LSB-

Anandan

-RSB-

spatio-temporal

filtering

-LSB-

Adelson

Previous

work

and

Bergen

Heeger

Fleet

and

Jepson

Weber

and

Malik

-RSB-

and

regularization

-LSB-

Horn

and

Schunck

Hildreth

Poggio

et

al.

-RSB-

Nagel

-LSB-

-RSB-

Anandan

-LSB-

-RSB-

and

Otte

and

Nagel

-LSB-

-RSB-

provide

comparisons

and

derive

relations

between

different

techniques

while

Barron

et

al.

-LSB-

-RSB-

provide

some

numerical

comparisons

Global

motion

estimators

-LSB-

Lucas

Bergen

et

al.

-RSB-

use

a

simple

flow

field

model

parameterized

by

a

small

number

of

unknown

variables

Examples

of

global

motion

models

include

affine

and

quadratic

flow

fields

In

the

taxonomy

of

Bergen

et

al.

-LSB-

-RSB-

these

fields

are

called

parametric

motion

models

since

they

can

be

used

locally

as

well

-LRB-

e.g.

affine

flow

can

be

estimated

at

every

pixel

-RRB-

The

spline-based

flow

fields

we

describe

in

the

next

section

can

be

viewed

as

local

parametric

models

since

the

flow

within

each

spline

patch

is

defined

by

a

small

number

of

control

vertices

Global

methods

are

most

useful

when

the

scene

has

a

particularly

simple

form

e.g.

when

the

scene

is

planar

These

methods

can

be

extended

to

more

complex

scenes

however

by

using

a

collection

of

global

motion

models

For

example

each

pixel

can

be

associated

with

one

of

several

global

motion

hypotheses

resulting

in

a

layered

motion

model

-LSB-

Wang

and

Adelson

Jepson

and

Black

Etoh

and

Shirai

Bober

and

Kittler

-RSB-

Alternatively

a

single

image

can

be

recursively

subdivided

into

smaller

parametric

motion

patches

based

on

estimates

of

the

current

residual

error

in

the

flow

estimate

-LSB-

M?uller

et

al.

-RSB-

Our

approach

is

similar

to

this

latter

work

except

that

it

preserves

inter-patch

motion

continuity

and

uses

both

split

and

merge

techniques

Stereo

matching

-LSB-

Barnard

and

Fischler

Quam

Dhond

and

Aggarwal

-RSB-

is

traditionally

considered

as

a

separate

sub-discipline

within

computer

vision

-LRB-

and

of

course

photogrammetry

-RRB-

but

there

are

strong

connections

between

it

and

motion

estimation

Stereo

can

be

viewed

as

a

simplified

version

of

constrained

motion

estimation

where

the

epipolar

geometry

is

given

so

that

each

flow

vector

is

constrained

to

lie

along

a

known

line

While

stereo

is

traditionally

performed

on

pairs

of

images

more

recent

algorithms

use

sequences

of

images

-LRB-

multiframe

stereo

or

motion

stereo

-RRB-

-LSB-

Bolles

et

al.

Matthies

et

al.

Okutomi

and

Kanade

-RSB-

The

idea

of

using

adaptive

window

sizes

in

stereo

-LSB-

Okutomi

and

Kanade

Okutomi

and

Kanade

-RSB-

is

similar

in

spirit

to

the

idea

used

in

this

paper

although

their

algorithm

has

a

much

higher

computational

complexity

Hierarchical

-LRB-

coarse-to-fine

-RRB-

matching

algorithms

have

a

long

history

of

use

both

in

stereo

matching

-LSB-

Quam

Witkin

et

al.

-RSB-

and

in

motion

estimation

-LSB-

Enkelmann

Anandan

Singh

Bergen

et

al.

-RSB-

Hierarchical

algorithms

first

solve

the

matching

problem

on

General

problem

formulation

smaller

lower-resolution

images

and

then

use

these

to

initialize

higher-resolution

estimates

Their

advantages

include

both

increased

computation

efficiency

and

the

ability

to

find

better

solutions

by

escaping

from

local

minima

The

algorithm

presented

in

this

paper

is

also

related

to

patch-based

feature

trackers

-LSB-

Lucas

and

Kanade

Rehg

and

Witkin

Tomasi

and

Kanade

-RSB-

It

differs

from

these

previous

approaches

in

that

we

use

patches

of

varying

size

we

completely

tile

the

image

with

patches

and

we

have

no

motion

discontinuities

across

patch

boundaries

Our

motion

estimator

can

be

used

as

a

parallel

adaptive

feature

tracker

by

selecting

spline

control

vertices

with

low

uncertainty

in

both

motion

components

-LSB-

Szeliski

et

al.

-RSB-

General

problem

formulation

The

general

motion

estimation

problem

can

be

formulated

as

follows

We

are

given

a

sequence

of

images

It

-LRB-

x

y

-RRB-

which

we

assume

were

formed

by

locally

displacing

a

reference

image

I

-LRB-

x

y

-RRB-

with

horizontal

and

vertical

displacement

ut

-LRB-

x

y

-RRB-

and

vt

-LRB-

x

y

-RRB-

i.e.

It

-LRB-

x

ut

y

vt

-RRB-

I

-LRB-

x

y

-RRB-

-LRB-

-RRB-

Each

individual

image

is

assumed

to

be

corrupted

with

uniform

white

Gaussian

noise

We

also

ignore

possible

occlusions

-LRB-

foldovers

-RRB-

in

the

warped

images

Given

such

a

sequence

of

images

we

wish

to

simultaneously

recover

the

displacement

fields

-LRB-

ut

vt

-RRB-

and

the

reference

image

I

-LRB-

x

y

-RRB-

The

maximum

likelihood

solution

to

this

problem

is

well

known

and

consists

of

minimizing

the

squared

error

X

t

Z

Z

-LSB-

It

-LRB-

x

ut

y

vt

-RRB-

I

-LRB-

x

y

-RRB-

-RSB-

dy

-LRB-

-RRB-

In

practice

we

are

usually

given

a

set

of

discretely

sampled

images

so

we

replace

the

above

integrals

with

summations

over

the

set

of

pixels

f

-LRB-

xi

yi

-RRB-

g

If

the

displacement

fields

ut

and

vt

at

different

times

are

independent

of

each

other

and

the

reference

intensity

image

I

-LRB-

x

y

-RRB-

is

assumed

to

be

known

the

above

minimization

problem

decomposes

into

a

set

of

independent

minimizations

one

for

each

frame

For

now

we

will

assume

that

this

is

the

case

and

only

study

the

two

frame

problem

which

can

be

rewritten

as

E

-LRB-

fui

vig

-RRB-

X

i

-LSB-

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

xi

yi

-RRB-

-RSB-

-LRB-

-RRB-

will

use

the

terms

displacement

field

flow

field

and

motion

estimate

interchangeably

General

problem

formulation

This

equation

is

called

the

sum

of

squared

differences

-LRB-

SSD

-RRB-

formula

-LSB-

Anandan

-RSB-

Expanding

in

a

first

order

Taylor

series

expansion

in

-LRB-

ui

vi

-RRB-

yields

the

the

image

brightness

constraint

-LSB-

Horn

and

Schunck

-RSB-

E

-LRB-

fui

vig

-RRB-

ss

X

i

-LSB-

I

Ixui

Iyvi

-RSB-

where

I

and

-LRB-

Ix

Iy

-RRB-

is

the

intensity

gradient

The

squared

pixel

error

function

-LRB-

-RRB-

is

by

no

means

the

only

possible

optimization

criterion

For

example

it

can

be

generalized

to

account

for

photometric

variation

-LRB-

global

brightness

and

contrast

changes

-RRB-

using

-LRB-

fui

vig

-RRB-

X

i

-LSB-

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

xi

yi

-RRB-

b

-RSB-

where

b

and

c

are

the

-LRB-

per-frame

-RRB-

brightness

and

contrast

correction

terms

Both

of

these

parameters

can

be

estimated

concurrently

with

the

flow

field

at

little

additional

cost

Their

inclusion

is

most

useful

in

situations

where

the

photometry

can

change

between

successive

views

-LRB-

e.g.

when

the

images

are

not

acquired

concurrently

-RRB-

Another

way

to

generalize

the

criterion

is

to

replace

the

squaring

function

with

a

non-quadratic

penalty

function

which

results

in

a

robust

motion

estimator

which

can

reject

outlier

measurements

-LSB-

Black

and

Anandan

Bober

and

Kittler

Black

and

Rangarajan

-RSB-

Another

possibility

is

to

weight

each

squared

error

term

with

a

factor

proportional

to

where

and

are

the

variances

of

the

image

and

derivative

noise

which

can

compensate

for

noise

in

the

image

derivative

computation

-LSB-

Simoncelli

et

al.

-RSB-

To

further

increase

noise

immunity

the

intensity

images

used

in

-LRB-

-RRB-

can

be

replaced

by

filtered

images

-LSB-

Burt

and

Adelson

-RSB-

The

above

minimization

problem

typically

has

many

local

minima

Several

techniques

are

commonly

used

to

find

a

more

globally

optimal

estimate

For

example

the

SSD

algorithm

performs

the

summation

at

each

pixel

over

an

m

m

window

-LRB-

typically

-RRB-

-LSB-

Anandan

-RSB-

More

recent

variations

use

adaptive

windows

-LSB-

Okutomi

and

Kanade

-RSB-

and

multiple

frames

-LSB-

Okutomi

and

Kanade

-RSB-

Regularization-based

algorithms

add

smoothness

constraints

on

the

u

and

v

fields

to

obtain

good

solutions

-LSB-

Horn

and

Schunck

Hildreth

Poggio

et

al.

-RSB-

Finally

multiscale

or

hierarchical

-LRB-

coarse-to-fine

-RRB-

techniques

are

often

used

to

speed

the

search

for

the

optimum

displacement

estimate

and

to

avoid

local

minima

Spline-based

flow

estimation

e

e

e

e

-LRB-

uj

vj

-RRB-

-LRB-

ui

vi

-RRB-

Figure

Displacement

spline

the

spline

control

vertices

f

-LRB-

uj

vj

-RRB-

g

are

shown

as

circles

-LRB-

ffi

-RRB-

and

the

pixel

displacements

f

-LRB-

ui

vi

-RRB-

g

are

shown

as

pluses

-LRB-

-RRB-

The

choice

of

representation

for

the

-LRB-

u

v

-RRB-

field

also

strongly

influences

the

performance

of

the

motion

estimation

algorithm

The

most

commonly

made

choice

is

to

assign

an

independent

estimate

at

each

pixel

-LRB-

ui

vi

-RRB-

but

global

motion

descriptors

are

also

possible

-LSB-

Lucas

Bergen

et

al.

Szeliski

and

Coughlan

-RSB-

One

can

observe

however

that

motion

estimates

at

individual

pixels

are

never

truly

independent

Both

local

correlation

windows

-LRB-

as

in

SSD

-RRB-

and

global

smoothness

constraints

aggregate

information

from

neighboring

pixels

The

resulting

displacement

estimates

are

therefore

highly

correlated

While

it

is

possible

to

analyze

the

correlations

induced

by

overlapping

windows

-LSB-

Matthies

et

al.

-RSB-

and

regularization

-LSB-

Szeliski

-RSB-

the

procedures

are

cumbersome

and

rarely

used

For

these

reasons

we

have

chosen

in

our

work

to

represent

the

motion

field

as

a

spline

which

is

a

representation

which

falls

in

between

per-pixel

motion

estimates

and

purely

global

motion

estimates

Spline-based

flow

estimation

Our

approach

is

to

represent

the

displacements

fields

u

-LRB-

x

y

-RRB-

and

v

-LRB-

x

y

-RRB-

as

two-dimensional

splines

controlled

by

a

smaller

number

of

displacement

estimates

uj

and

vj

which

lie

on

a

coarser

spline

control

grid

-LRB-

Figure

-RRB-

The

value

for

the

displacement

at

a

pixel

i

can

be

written

as

u

-LRB-

xi

yi

-RRB-

X

j

uj

Bj

-LRB-

xi

yi

-RRB-

or

ui

X

j

uj

wij

-LRB-

-RRB-

Function

minimization

where

the

Bj

-LRB-

x

y

-RRB-

are

called

the

basis

functions

and

are

only

non-zero

over

a

small

interval

-LRB-

finite

support

-RRB-

We

call

the

wij

Bj

-LRB-

xi

yi

-RRB-

weights

to

emphasize

that

the

-LRB-

ui

vi

-RRB-

are

known

linear

combinations

of

the

-LRB-

uj

vj

-RRB-

In

our

current

implementation

the

basis

functions

are

spatially

shifted

versions

of

each

other

i.e.

Bj

-LRB-

x

y

-RRB-

B

-LRB-

x

xj

y

yj

-RRB-

We

have

studied

five

different

interpolation

functions

-LRB-

-RRB-

block

-LRB-

-RRB-

linear

on

squares

-LRB-

-RRB-

linear

on

triangles

-LRB-

-RRB-

bilinear

and

-LRB-

-RRB-

biquadratic

-LSB-

Szeliski

and

Coughlan

-RSB-

In

practice

we

most

often

use

the

bilinear

bases

We

also

impose

the

condition

that

the

spline

control

grid

is

a

regular

subsampling

of

the

pixel

grid

xj

mxi

yj

myi

so

that

each

set

of

m?m

pixels

corresponds

to

a

single

spline

patch

Function

minimization

To

recover

the

local

spline-based

flow

parameters

we

need

to

minimize

the

cost

function

-LRB-

-RRB-

with

respect

to

the

f

uj

vjg

We

do

this

using

a

variant

of

the

Levenberg-Marquardt

iterative

non-linear

minimization

technique

-LSB-

Press

et

al.

-RSB-

First

we

compute

the

gradient

of

E

in

-LRB-

-RRB-

with

respect

to

each

of

the

parameters

uj

and

vj

guj

@E

uj

X

i

eiGxi

wij

gvj

@E

vj

X

i

eiGyi

wij

-LRB-

-RRB-

where

ei

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

xi

yi

-RRB-

-LRB-

-RRB-

is

the

intensity

error

at

pixel

i

-LRB-

Gxi

Gyi

-RRB-

-LRB-

xi

ui

yi

vi

-RRB-

-LRB-

-RRB-

is

the

intensity

gradient

of

at

the

displaced

position

for

pixel

i

and

the

wij

are

the

sampled

values

of

the

spline

basis

function

-LRB-

-RRB-

Algorithmically

we

compute

the

above

gradients

by

first

forming

the

displacement

vector

for

each

pixel

-LRB-

ui

vi

-RRB-

using

-LRB-

-RRB-

then

computing

the

resampled

intensity

and

gradient

values

of

at

-LRB-

-RRB-

-LRB-

xi

ui

yi

vi

-RRB-

computing

ei

and

finally

incrementing

the

guj

and

gvj

values

of

all

control

vertices

affecting

that

pixel

-LSB-

Szeliski

and

Coughlan

-RSB-

the

remainder

of

the

paper

we

will

use

indices

i

for

pixels

and

j

for

spline

control

vertices

Spline-based

flow

estimation

For

the

Levenberg-Marquardt

algorithm

we

also

require

the

approximate

Hessian

matrix

A

where

the

second-derivative

terms

are

left

out

The

matrix

A

contains

entries

of

the

form

auujk

X

i

@ei

uj

@ei

uk

X

i

wijwik

-LRB-

Gxi

-RRB-

auvjk

avujk

X

i

@ei

uj

@ei

vk

X

i

wijwikGxi

Gyi

-LRB-

-RRB-

avvjk

X

i

@ei

vj

@ei

vk

X

i

wijwik

-LRB-

Gyi

-RRB-

The

entries

of

A

can

be

computed

at

the

same

time

as

the

energy

gradients

The

Levenberg-Marquardt

algorithm

proceeds

by

computing

an

increment

u

to

the

current

displacement

estimate

u

which

satisfies

-LRB-

A

I

-RRB-

u

g

-LRB-

-RRB-

where

u

is

the

vector

of

concatenated

displacement

estimates

f

uj

vjg

g

is

the

vector

of

concatenated

energy

gradients

fguj

gvj

g

and

is

a

stabilization

factor

which

varies

over

time

-LSB-

Press

et

al.

-RSB-

To

solve

this

large

sparse

system

of

linear

equations

we

use

preconditioned

gradient

descent

u

B

g

-LRB-

-RRB-

where

B

A

I

and

A

block

diag

-LRB-

A

-RRB-

is

the

set

of

block

diagonal

matrices

defined

in

-LRB-

-RRB-

with

j

k

and

g

B

is

called

the

preconditioned

residual

vector

An

optimal

value

for

can

be

computed

at

each

iteration

by

minimizing

E

-LRB-

d

-RRB-

ss

dTg

i.e.

by

setting

-LRB-

d

g

-RRB-

-LRB-

dT

Ad

-RRB-

where

d

g

is

the

direction

vector

for

the

current

step

See

-LSB-

Szeliski

and

Coughlan

-RSB-

for

more

details

on

our

algorithm

implementation

To

handle

larger

displacements

we

run

our

algorithm

in

a

coarse-to-fine

-LRB-

hierarchical

-RRB-

fashion

A

Gaussian

image

pyramid

is

first

computed

using

an

iterated

filter

-LSB-

Burt

and

Adelson

-RSB-

We

then

run

the

algorithm

on

one

of

the

smaller

pyramid

levels

and

use

the

resulting

flow

estimates

to

initialize

the

next

finer

level

-LRB-

using

bilinear

interpolation

and

doubling

the

displacement

magnitudes

-RRB-

means

adjusting

the

descent

direction

to

accelerate

the

convergence

e.g.

by

pre-multiplying

it

by

an

approximate

inverse

of

A

-LSB-

Axelsson

and

Barker

Press

et

al.

-RSB-

Function

minimization

ull

Dresults/rubik

rubik/s

pp

dev/null

Image

size

x

Subsampled

by

scaled

by

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Dresults/rubik

rubik/s

pp

dev/null

Image

size

x

Subsampled

by

scaled

by

ll

Dresults/rubik

rubik/s

pp

dev/null

Image

size

x

Subsampled

by

scaled

by

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

Example

of

general

flow

computation

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

-LRB-

d

-RRB-

flow

estimates

for

m

and

Hierarchical

basis

splines

Figure

shows

an

example

of

the

flow

estimates

produced

by

our

technique

The

input

image

is

pixels

and

the

flow

is

displayed

on

a

grid

We

show

the

results

of

using

a

level

pyramid

iterations

at

each

level

and

with

three

different

patch

sizes

m

m

and

m

As

we

can

see

using

patches

that

are

too

large

result

in

flow

estimates

which

are

too

smooth

while

using

patches

that

are

too

small

result

in

noisy

estimates

-LRB-

This

latter

problem

could

potentially

be

fixed

by

adding

regularization

but

at

the

cost

of

increased

iterations

-RRB-

To

overcome

this

problem

we

need

a

technique

which

automatically

selects

the

best

patch

size

in

each

region

of

the

image

This

is

the

idea

we

will

develop

in

the

next

two

sections

Hierarchical

basis

splines

Regularized

problems

often

require

many

iterations

to

propagate

information

from

regions

with

high

certainty

-LRB-

textures

or

edges

-RRB-

to

regions

with

little

information

-LRB-

uniform

intensities

-RRB-

Several

techniques

have

been

developed

to

overcome

this

problem

Coarse-to-fine

techniques

-LSB-

Quam

Anandan

-RSB-

can

help

but

often

do

n't

converge

as

quickly

to

the

optimal

solution

as

multigrid

techniques

-LSB-

Terzopoulos

-RSB-

Conjugate

gradient

descent

can

also

be

used

especially

for

non-linear

problems

such

as

shape-from-shading

-LSB-

Simchony

et

al.

-RSB-

Perhaps

the

most

effective

technique

is

a

combination

of

conjugate

gradient

descent

with

hierarchical

basis

functions

-LSB-

Yserentant

-RSB-

which

has

been

applied

both

to

interpolation

problems

in

stereo

matching

-LSB-

Szeliski

-RSB-

and

to

shape-from-shading

-LSB-

Szeliski

-RSB-

Hierarchical

basis

functions

are

based

on

using

a

pyramidal

representation

for

the

data

-LSB-

Burt

and

Adelson

-RSB-

where

the

number

of

nodes

in

the

pyramid

is

equal

to

the

original

number

of

nodes

at

the

finest

level

-LRB-

Figure

-RRB-

To

convert

from

the

hierarchical

basis

representation

to

the

usual

finelevel

representation

-LRB-

which

is

called

the

nodal

basis

representation

-LSB-

Yserentant

-RSB-

-RRB-

we

start

at

the

coarsest

-LRB-

smallest

-RRB-

level

of

the

pyramid

and

interpolate

the

values

at

this

level

thus

doubling

the

resolution

These

interpolated

values

are

then

added

to

the

hierarchical

representation

values

at

the

next

lower

level

and

the

process

is

repeated

until

the

nodal

representation

is

obtained

This

process

can

be

written

algorithmically

as

procedure

S

for

l

L

down

to

basis

splines

are

therefore

a

degenerate

-LRB-

non-orthogonal

-RRB-

form

of

wavelets

-LSB-

Mallat

-RSB-

with

extremely

compact

support

and

inverses

Hierarchical

basis

splines

EE

EE

EE

EE

EE

EE

EE

EE

EE

EE

EE

EE

SS

SS

SS

SS

SS

SS

SS

SS

fine

medium

coarse

l

l

l

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

s

s

s

s

s

s

s

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

s

s

s

s

s

s

s

s

s

s

s

s

s

s

s

s

s

s

s

Figure

Multiresolution

pyramid

The

multiple

resolution

levels

are

a

schematic

representation

of

the

hierarchical

basis

spline

The

circles

indicate

the

nodes

in

the

hierarchical

basis

Filled

circles

-LRB-

ffl

-RRB-

are

free

variables

in

the

quadtree

spline

-LRB-

Section

-RRB-

while

open

circles

-LRB-

ffi

-RRB-

must

be

zero

-LRB-

see

Figure

-RRB-

for

j

Ml

ulj

ulj

P

wjk

ul

k

end

S

In

this

procedure

each

node

is

assigned

to

one

of

the

level

collections

Ml

-LRB-

the

circles

in

Figure

-RRB-

Each

node

also

has

a

number

of

parent

nodes

Nj

on

the

next

coarser

level

that

contribute

to

its

value

during

the

interpolation

process

The

wjk

are

the

weighting

functions

that

depend

on

the

particular

choice

of

interpolation

function

For

the

examples

shown

in

this

paper

we

use

bilinear

interpolation

since

previous

experiments

suggest

that

this

is

a

reasonable

choice

for

the

interpolator

-LSB-

Szeliski

-RSB-

We

can

write

the

above

process

algebraically

as

u

S

u

SL

u

-LRB-

-RRB-

Hierarchical

basis

splines

with

-LRB-

Sl

-RRB-

jk

if

j

k

wjk

if

j

Ml

and

k

Nj

otherwise

and

u

is

the

hierarchical

basis

representation

Using

a

hierarchical

basis

representation

for

the

flow

field

is

equivalent

to

using

SST

as

a

preconditioner

i.e.

g

SSTg

-LSB-

Axelsson

and

Barker

Szeliski

-RSB-

The

transformation

SST

can

be

used

as

a

preconditioner

because

the

influence

of

hierarchical

bases

at

coarser

levels

-LRB-

which

are

obtained

from

the

ST

operation

-RRB-

are

propagated

to

the

nodal

basis

at

the

fine

level

through

the

S

operation

To

evaluate

ST

i.e.

to

convert

from

the

nodal

basis

representation

to

the

hierarchical

basis

representation

we

use

the

procedure

procedure

ST

for

l

to

L

for

k

Ml

ul

k

ul

k

P

j

wjk

ulj

end

ST

When

combining

hierarchical

basis

preconditioning

with

the

block

diagonal

preconditioning

in

-LRB-

-RRB-

we

have

several

choices

We

can

apply

the

block

diagonal

preconditioning

first

g

SSTB

or

second

g

B

or

we

can

interleave

the

two

preconditioners

g

SB

or

g

B?T

SST

Bg

where

B

B

The

latter

two

operations

correspond

to

well-defined

preconditioners

-LRB-

i.e.

optimization

under

a

change

of

basis

-RRB-

while

the

first

two

are

easier

to

implement

In

our

current

work

we

use

the

first

form

i.e.

we

apply

block

preconditioning

first

and

then

use

sweep

up

and

then

down

the

hierarchical

basis

pyramid

to

smooth

the

residual

In

future

work

we

plan

to

develop

optimal

combinations

of

block

diagonal

and

hierarchical

basis

preconditioning

To

summarize

our

algorithm

-LRB-

Figure

-RRB-

we

keep

both

the

hierarchical

and

nodal

representations

and

map

between

the

two

as

required

For

accumulating

the

distances

and

gradients

required

in

-LRB-

-RRB-

we

compute

the

image

flows

and

the

derivatives

with

respect

to

the

parameters

in

the

nodal

basis

We

then

use

the

hierarchical

basis

to

smooth

the

residual

vector

g

before

selecting

a

new

conjugate

direction

and

computing

the

optimal

step

size

Using

this

technique

not

only

makes

the

convergence

faster

but

also

propagates

local

corrections

over

the

whole

domain

which

tends

to

smooth

the

resulting

flow

significantly

To

demonstrate

the

performance

improvements

available

with

hierarchical

basis

functions

we

use

as

our

example

the

Square

sequence

which

is

part

of

the

data

set

used

by

Barron

et

al.

-LSB-

-RSB-

Hierarchical

basis

splines

d

gn

rE

-LRB-

u

-RRB-

y

gn

SZSTB

n

gn

gn

gn

gn

dn

gn

ndn

n

dn

gn

dTn

Adn

un

un

ndn

increment

n

loop

to

y

S

mapping

from

hierarchical

to

nodal

basis

B

blockdiag

-LRB-

A

-RRB-

Z

matrix

for

quadtree

spline

basis

-LRB-

Section

-RRB-

Figure

Hierarchical

basis

preconditioned

conjugate

gradient

algorithm

RMS

flow

error

iteration

Square

convergence

-LRB-

gradient

descent

-RRB-

L

l

L

l

L

l

L

l

H

l

H

l

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

convergence

plot

-LRB-

error

vs.

iteration

number

-RRB-

Figure

Square

sample

image

and

convergence

plot

Quadtree

-LRB-

adaptive

resolution

-RRB-

splines

Figure

shows

one

image

in

the

sequence

while

Figure

shows

the

convergence

rates

for

regular

gradient

descent

-LRB-

L

-RRB-

coarse-to-fine

estimation

-LRB-

L

-RRB-

and

preconditioning

with

hierarchical

basis

functions

-LRB-

H

-RRB-

with

different

amounts

of

regularization

-LRB-

-RRB-

As

we

can

see

from

these

results

adding

more

regularization

results

in

a

more

accurate

solution

-LRB-

this

is

because

the

true

flow

is

a

single

constant

value

-RRB-

using

coarse

to

fine

is

quicker

than

single-level

relaxation

and

hierarchical

basis

preconditioning

is

faster

than

coarse-to-fine

relaxation

It

is

interesting

to

note

that

using

hierarchical

basis

functions

even

without

regularization

quickly

smooths

out

the

solution

and

outperforms

coarse-to-fine

without

regularization

Quadtree

-LRB-

adaptive

resolution

-RRB-

splines

While

hierarchical

basis

splines

can

help

accelerate

an

estimation

algorithm

or

even

to

add

extra

smoothness

to

the

solution

they

do

not

in

themselves

solve

the

problem

of

having

adaptively-sized

patches

For

this

we

will

use

the

idea

of

quadtree

splines

i.e.

splines

defined

on

a

quadtree

domain

A

quadtree

is

a

representation

built

by

recursively

subdividing

rectangles

into

four

pieces

-LRB-

Figure

-RRB-

-LSB-

Samet

-RSB-

The

basic

concept

of

a

quadtree

spline

is

to

define

a

continuous

function

over

a

quadtree

domain

by

interpolating

numeric

values

at

the

corners

of

each

spline

leaf

cell

-LRB-

square

-RRB-

However

because

cells

are

non-uniformly

subdivided

cracks

or

first-order

discontinuities

in

the

interpolated

function

will

arise

-LRB-

Figure

-RRB-

unless

a

crack-filling

strategy

is

used

-LSB-

Samet

-RSB-

The

simplest

strategy

is

to

simply

replace

the

values

at

the

nodes

along

a

crack

edge

-LRB-

the

white

circles

in

Figure

-RRB-

with

the

average

values

of

its

two

parent

nodes

along

the

edge

This

is

the

strategy

we

used

in

developing

octree

splines

for

the

representation

of

multi-resolution

distance

maps

in

pose

estimation

problems

-LSB-

Lavall?ee

et

al.

-RSB-

When

the

problem

is

one

of

iteratively

estimating

the

values

on

the

nodes

in

the

quadtree

spline

enforcing

the

crack-filling

rule

becomes

more

complicated

A

useful

strategy

which

we

developed

for

estimating

displacement

fields

in

elastic

medical

image

registration

-LSB-

Szeliski

and

Lavall?ee

-RSB-

is

to

use

a

hierarchical

basis

and

to

selectively

zero

out

nodes

in

this

basis

Observe

that

if

in

Figures

and

we

set

the

values

of

the

open

circles

-LRB-

ffi

-RRB-

to

zero

in

the

hierarchical

basis

and

then

re-compute

the

nodal

basis

using

S

the

resulting

spline

has

the

desired

continuity

i.e.

nodes

along

longer

edges

are

the

averages

of

their

parents

The

formulation

of

the

quadtree

spline

in

terms

of

an

adaptive

hierarchical

basis

i.e.

a

basis

in

which

some

nodes

are

set

to

zero

has

several

advantages

First

it

is

very

easy

to

implement

Quadtree

-LRB-

adaptive

resolution

-RRB-

splines

t

t

t

t

t

t

t

t

t

t

t

t

t

t

d

t

t

t

t

d

t

t

t

t

t

d

t

t

t

t

d

d

d

td

d

d

d

dt

t

t

d

d

dt

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Quadtree

associated

with

spline

function

and

potential

cracks

in

quadtree

spline

-LRB-

a

-RRB-

the

nodes

with

filled

circles

-LRB-

ffl

-RRB-

are

free

variables

in

the

associated

hierarchical

basis

whereas

the

open

circles

-LRB-

ffi

-RRB-

-LRB-

and

also

the

nodes

not

drawn

-RRB-

must

be

zero

-LRB-

in

the

nodal

basis

these

nodes

are

interpolated

from

their

ancestors

-RRB-

-LRB-

b

-RRB-

potential

cracks

in

a

simpler

quadtree

spline

are

shown

as

shaded

areas

simply

requiring

a

selective

zeroing

step

between

the

ST

and

S

operations

-LRB-

algebraically

we

write

g

SZSTg

where

Z

is

a

diagonal

matrix

with

's

and

's

on

the

diagonal?see

Figure

-RRB-

Second

it

generalizes

to

splines

of

arbitrary

order

e.g.

we

can

build

a

quadtree

spline

based

on

quadratic

B-splines

using

adaptive

hierarchical

basis

functions

However

for

higher-order

splines

even

more

nodes

have

to

be

zeroed

in

order

to

ensure

that

finer

level

splines

do

not

affect

nearby

coarser

-LRB-

undivided

-RRB-

cells

Third

as

we

will

discuss

in

the

next

section

the

adaptive

hierarchical

basis

idea

is

even

more

general

than

the

quadtree

spline

and

corresponds

to

a

specific

kind

of

multi-resolution

prior

model

The

quadtree

spline

as

described

here

ensures

that

the

function

within

any

leaf

cell

-LRB-

square

domain

-RRB-

has

a

simple

form

-LRB-

single

polynomial

description

no

spurious

ripples

-RRB-

An

alternative

way

of

interpreting

the

quadtree

in

Figure

is

that

it

specifies

the

minimum

degree

of

complexity

in

each

cell

i.e.

that

each

square

is

guaranteed

to

have

its

full

degrees

of

freedom

-LRB-

e.g.

all

corners

have

independent

values

in

the

bilinear

case

-RRB-

In

this

latter

interpretation

the

open

circles

in

the

hierarchical

basis

are

not

zeroed

and

only

the

circles

actually

not

drawn

in

Figure

are

zeroed

In

this

approach

large

squares

can

have

arbitrarily-detailed

ripples

inside

their

domain

resulting

the

Z

matrix

changes

we

also

have

to

re-compute

the

quadtree

spline

using

u

SZS

The

S

procedure

is

similar

to

ST

but

now

uj

uj

wjk

uk

Quadtree

-LRB-

adaptive

resolution

-RRB-

splines

from

fine-level

basis

functions

near

the

square

's

boundaries

To

date

we

have

not

investigated

this

alternative

possibility

Subdivision

strategy

The

quadtree

spline

provides

a

convenient

way

to

use

adaptively-sized

patches

for

motion

estimation

while

maintaining

inter-patch

continuity

The

question

remains

how

to

actually

determine

the

topology

of

the

patches

i.e.

which

patches

get

subdivided

and

which

ones

remain

large

Ideally

we

would

like

each

patch

to

cover

a

region

of

the

image

within

which

the

parametric

motion

model

is

valid

In

a

real-world

situation

this

may

correspond

to

planar

surface

patches

undergoing

rigid

motion

with

a

small

amount

of

perspective

distortion

-LRB-

bilinear

flow

is

then

very

close

to

projective

flow

-RRB-

However

usually

we

are

not

a

priori

given

the

required

segmentation

of

the

image

Instead

we

must

deduce

such

a

segmentation

based

on

the

adequacy

of

the

flow

model

within

each

patch

The

fundamental

tool

we

will

use

here

is

the

concept

of

residual

flow

-LSB-

Irani

et

al.

-RSB-

recently

used

by

M?uller

et

al.

-LSB-

-RSB-

to

subdivide

affine

motion

patches

-LRB-

which

they

call

tiles

-RRB-

The

residual

flow

is

the

per-pixel

estimate

of

flow

required

to

register

the

two

images

in

addition

to

the

flow

currently

being

modeled

by

the

parametric

motion

model

At

a

single

pixel

only

the

normal

flow

can

be

estimated

uNi

-LRB-

eiGxi

eiGyi

-RRB-

k

-LRB-

Gxi

Gyi

-RRB-

k

ffl

-LRB-

-RRB-

where

the

intensity

error

ei

and

the

gradient

-LRB-

Gxi

Gyi

-RRB-

are

given

in

-LRB-

-RRB-

This

measure

is

different

from

that

used

in

-LSB-

Irani

et

al.

M?uller

et

al.

-RSB-

who

sum

the

numerator

and

denominator

in

-LRB-

-RRB-

over

a

small

neighborhood

around

each

pixel

To

decide

whether

to

split

a

spline

patch

into

four

smaller

patches

we

sum

the

magnitude

of

the

residual

normal

flow

kuNi

k

over

all

the

pixels

in

the

patch

and

compare

it

to

a

threshold

u.

Patches

where

the

motion

model

is

adequate

should

fall

below

this

threshold

while

patches

which

have

multiple

motions

should

be

above

Starting

with

the

whole

image

we

subdivide

recursively

until

either

the

p-norm

residual

falls

below

an

acceptable

value

or

the

smallest

patch

size

considered

-LRB-

typically

pixels

wide

-RRB-

is

reached

Figures

c

show

an

example

of

a

quadtree

spline

motion

estimate

produced

with

this

splitting

technique

for

a

simple

synthetic

example

in

which

two

central

disks

are

independently

moving

we

use

a

p?norm

-LRB-

P

i

kuNi

kp

-RRB-

=p

which

can

model

a

max

operation

as

p

Subdivision

strategy

True

flow

for

two

disc

image

sequence

-LRB-

true

flow

-RRB-

Image

size

x

Subsampled

by

scaled

by

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

Quadtree

spline

motion

estimation

-LRB-

Two

Discs

-LRB-

SRI

Trees

-RRB-

sequence

-RRB-

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

true

flow

-LRB-

c

-RRB-

split

technique

-LRB-

d

-RRB-

merge

technique

A

Bayesian

interpretation

against

a

textured

background

The

quadtree

boundaries

are

warped

to

show

the

extent

of

the

estimated

image

motion

-LRB-

up

and

left

for

the

top

disc

down

and

right

for

the

bottom

disc

-RRB-

Note

how

the

subdivision

occurs

mostly

at

the

object

boundaries

as

would

be

expected

The

most

visible

error

-LRB-

near

the

upper

right

edge

of

the

lower

disc

-RRB-

occurs

in

an

area

of

little

image

contrast

and

where

the

motion

is

mostly

parallel

to

the

region

contour

An

alternative

to

the

iterative

splitting

strategy

is

to

start

with

small

patches

and

to

then

merge

adjacent

patches

with

compatible

motion

estimates

into

larger

patches

-LRB-

within

the

constraints

of

allowable

quadtree

topologies

-RRB-

To

test

if

a

larger

patch

has

consistent

flow

we

compare

the

four

values

along

the

edge

of

the

patch

and

the

value

at

the

center

with

the

average

values

interpolated

from

the

four

corner

cells

-LRB-

look

at

the

lower

left

quadrant

of

Figure

to

visualize

this

-RRB-

The

relative

difference

between

the

estimated

and

interpolated

values

d

k

uj

ujk

q

k

d

where

uj

is

the

interpolated

value

must

be

below

a

threshold

d

-LRB-

typically

-RRB-

for

all

five

nodes

before

the

four

constituent

patches

are

allowed

to

be

merged

into

a

larger

patch

Notice

that

the

quantity

uj

uj

is

exactly

the

value

of

the

hierarchical

basis

function

at

a

node

-LRB-

at

least

for

bilinear

splines

-RRB-

so

we

are

in

effect

converting

small

hierarchical

basis

values

close

to

be

exactly

zero

-LRB-

this

has

a

Bayesian

interpretation

as

we

will

discuss

in

the

next

section

-RRB-

Note

also

that

this

consistency

criterion

may

fail

in

regions

of

little

texture

where

the

flow

estimates

are

initially

unreliable

unless

regularization

is

applied

to

make

these

flow

fields

more

smooth

Figure

shows

an

example

of

a

quadtree

spline

motion

estimate

produced

with

this

merging

technique

The

results

are

qualitatively

quite

similar

to

the

results

obtained

with

the

split

technique

A

Bayesian

interpretation

The

connection

between

energy-based

or

regularized

low-level

vision

problems

and

Bayesian

estimation

formulations

is

well

known

-LSB-

Kimeldorf

and

Wahba

Marroquin

et

al.

Szeliski

-RSB-

In

a

nutshell

it

can

be

shown

that

the

energy

or

cost

function

being

minimized

can

be

converted

into

a

probability

distribution

over

the

unknowns

using

a

Gibbs

or

Boltzmann

distribution

and

that

finding

the

minimum

energy

solution

is

equivalent

to

maximum

a

posteriori

-LRB-

MAP

-RRB-

estimation

The

Bayesian

model

nicely

decomposes

the

energy

function

into

a

measurement

model

A

Bayesian

interpretation

-LRB-

typically

the

squared

error

terms

between

the

measurements

and

their

predicted

values

-RRB-

and

a

prior

model

-LRB-

which

usually

corresponds

to

the

stabilizer

or

smoothing

term

-RRB-

i.e.

E

-LRB-

u

-RRB-

Ed

-LRB-

u

d

-RRB-

Ep

-LRB-

u

-RRB-

p

-LRB-

u

-RRB-

e?E

-LRB-

u

-RRB-

e?Ed

-LRB-

u

;d

-RRB-

e?Ep

-LRB-

u

-RRB-

p

-LRB-

u

d

-RRB-

p

-LRB-

u

-RRB-

-LRB-

-RRB-

It

then

becomes

straightforward

to

make

use

of

robust

statistical

models

by

simply

modifying

the

appropriate

energy

terms

-LSB-

Black

and

Anandan

Black

and

Rangarajan

-RSB-

The

basic

spline-based

flow

model

introduced

in

-LSB-

Szeliski

and

Coughlan

-RSB-

is

already

a

valid

prior

model

since

it

restricts

the

family

of

functions

to

the

smooth

set

of

tensor-product

splines

In

most

cases

a

small

amount

of

intensity

variation

inside

each

spline

patch

is

sufficient

to

ensure

that

a

unique

well-behaved

solution

exists

However

just

to

be

on

the

safe

side

it

is

easy

to

add

a

small

amount

of

regularization

with

quadratic

penalty

terms

on

the

uj

's

and

their

finite

differences

Hierarchical

basis

splines

as

well

as

other

multilevel

representations

such

as

overcomplete

pyramids

can

be

viewed

as

multiresolution

priors

-LSB-

Szeliski

and

Terzopoulos

-RSB-

There

are

two

basic

approaches

to

specifying

such

a

prior

The

first

which

we

use

in

our

current

work

is

to

simply

view

the

hierarchical

basis

as

a

preconditioner

and

to

define

the

prior

model

over

the

usual

nodal

basis

-LSB-

Szeliski

-RSB-

The

alternative

is

to

define

the

prior

model

directly

on

the

hierarchical

basis

usually

assuming

that

each

basis

element

is

statistically

independent

from

the

others

-LRB-

i.e.

that

the

covariance

matrix

is

diagonal

-RRB-

-LSB-

Szeliski

and

Terzopoulos

Pentland

-RSB-

An

extreme

example

of

this

is

the

scale-recursive

multiscale

Markov

Random

Fields

introduced

in

-LSB-

Chin

et

al.

-RSB-

whose

special

structure

makes

it

possible

to

recover

the

field

in

a

single

sweep

through

the

pyramid

Unfortunately

their

technique

is

based

on

a

piecewise-constant

model

of

flow

which

results

in

recovered

fields

that

have

excessive

blockiness

-LSB-

Luettgen

et

al.

-RSB-

Within

this

framework

adaptive

hierarchical

basis

splines

can

be

viewed

as

having

a

more

complex

multiresolution

prior

where

each

hierarchical

node

has

a

non-zero

prior

probability

of

being

exactly

zero

The

split

and

merge

algorithms

can

be

viewed

as

simple

heuristic

techniques

designed

to

recover

the

underlying

motion

field

and

to

decide

which

nodes

are

actually

zero

More

sophisticated

techniques

to

solve

this

problem

would

include

simulated

annealing

-LSB-

Marroquin

et

al.

-RSB-

and

mean-field

annealing

-LSB-

Geiger

and

Girosi

-RSB-

Quadtree

splines

have

an

even

more

complicated

prior

model

since

the

existence

of

zeros

at

certain

levels

in

the

pyramid

implies

zeros

at

lower

levels

as

well

as

zeros

at

some

neighboring

nodes

-LRB-

depending

on

the

exact

interpretation

of

the

quadtree

spline

-RRB-

We

will

not

pursue

these

model

further

in

this

paper

and

leave

their

investigation

to

future

work

Experimental

results

Technique

Pixel

Error

Std

Dev

Avg

Ang

Error

Std

Dev

Density

regular

spline

-LRB-

s

-RRB-

ffi

ffi

regular

spline

-LRB-

s

-RRB-

ffi

ffi

regular

spline

-LRB-

s

-RRB-

ffi

ffi

quadtree

spline

-LRB-

merge

s

-RRB-

ffi

ffi

quadtree

spline

-LRB-

split

s

-RRB-

ffi

ffi

Table

Summary

of

Two

Discs

-LRB-

SRI

Trees

-RRB-

results

Experimental

results

To

investigate

the

performance

of

our

quadtree

spline-based

motion

estimator

we

use

the

synthetically

generated

Two

Discs

-LRB-

SRI

Trees

-RRB-

sequence

shown

in

Figure

for

which

we

know

the

true

motion

-LRB-

Figure

-RRB-

The

results

of

our

spline-based

motion

estimator

for

various

choices

of

window

size

s

as

well

as

the

results

with

both

the

split

and

merge

techniques

are

shown

in

Table

The

experiments

show

that

the

optimal

fixed

window

size

is

s

and

that

both

split

and

merge

techniques

provide

slightly

better

results

The

relatively

small

difference

is

error

between

the

various

techniques

is

due

to

most

of

the

error

being

concentrated

in

the

regions

where

occlusions

occur

-LRB-

Figure

-RRB-

Adding

an

occlusion

detection

process

to

our

algorithm

should

help

reduce

the

errors

in

these

regions

We

also

tested

our

algorithm

on

some

of

the

standard

motion

sequences

used

in

other

recent

motion

estimation

papers

-LSB-

Barron

et

al.

Wang

and

Adelson

Otte

and

Nagel

-RSB-

The

results

on

the

Hamburg

Taxi

sequence

are

shown

in

Figure

where

the

independent

motion

of

the

three

moving

cars

can

be

clearly

distinguished

Notice

that

the

algorithm

was

also

able

to

pick

out

the

small

region

of

the

moving

pedestrian

near

the

upper

left

corner

The

result

on

the

Flower

Garden

sequence

are

shown

in

Figure

Here

the

trunk

of

the

tree

is

clearly

segmented

from

the

rest

of

the

scene

The

top

of

the

flower

garden

on

the

other

hand

is

not

clearly

segmented

from

the

house

and

sky

since

it

appears

that

the

continuous

motion

field

represented

by

the

splines

is

an

adequate

description

The

final

sequence

which

we

studied

is

the

table

of

marble

blocks

acquired

by

Michael

Otte

the

global

motion

estimates

used

in

-LSB-

Wang

and

Adelson

-RSB-

we

do

not

require

that

the

motion

be

a

combination

of

a

few

global

affine

motions

Experimental

results

Figure

Flow

error

ku?u?k

and

residual

normal

flow

kuNi

k

for

Two

Discs

-LRB-

SRI

Trees

-RRB-

sequence

Note

how

most

of

the

errors

are

concentrated

near

the

motion

discontinuities

and

especially

the

disoccluded

region

in

the

center

axi

m.pp

taxi/s

pp

p

Dresults/taxi

m

Oresults/taxi

m.pp

taxi/s

pp

Image

size

x

Subsampled

by

scaled

by

Figure

Hamburg

Taxi

sequence

estimated

quadtree

and

estimated

flow

merging

s

patches

in

a

pyramid

Extensions

m

Oresults/flower

m.pp

flower/quarter/c

ppm

egrep

scale

p

Dresults/flower

m

Oresults/flower

m.pp

flower/quarter/c

ppm

egrep

scale

Image

size

x

Subsampled

by

scaled

by

Figure

Flower

Garden

sequence

estimated

quadtree

and

estimated

flow

merging

s

patches

in

a

pyramid

-LSB-

Otte

and

Nagel

-RSB-

In

this

scene

the

camera

is

moving

forward

and

left

while

all

of

the

blocks

are

stationary

except

for

the

short

central

block

which

is

independently

moving

to

the

left

The

quadtree

segmentation

of

the

motion

field

has

separated

out

the

tall

block

in

the

foreground

and

the

independently

moving

block

but

has

not

separated

the

other

blocks

from

the

table

or

the

checkered

background

Changing

the

thresholds

on

the

merge

algorithm

could

be

used

to

achieve

a

greater

segmentation

but

this

does

not

appear

to

be

necessary

to

adequately

model

the

motion

field

Extensions

We

are

currently

extending

the

algorithm

described

in

this

paper

in

a

number

of

directions

which

include

better

multiframe

flow

estimation

parallel

feature

tracking

and

local

search

When

given

more

than

two

frames

we

must

assume

a

model

of

motion

coherency

across

frames

to

take

advantage

of

the

additional

information

available

The

simplest

assumption

is

that

of

linear

flow

i.e.

that

displacements

between

successive

images

and

a

base

image

are

known

scalar

multiples

of

each

other

ut

Flow

estimation

can

then

be

formulated

by

summing

the

intensity

differences

between

the

base

frame

and

all

other

frames

-LSB-

Szeliski

and

Coughlan

-RSB-

which

is

similar

to

the

sum

of

sum

of

squared-distance

-LRB-

SSSD

-RRB-

algorithm

of

-LSB-

Okutomi

and

Kanade

-RSB-

We

have

found

that

in

practice

this

works

well

although

it

is

often

necessary

to

bootstrap

the

motion

the

most

common

case

e.g.

for

spatio-temporal

filtering

a

uniform

temporal

sampling

-LRB-

st

t

-RRB-

is

assumed

but

this

is

not

strictly

necessary

Extensions

m.pp

tcorrect/correct

otte/half/tr

ppm

egrep

std.dev

p

m

m.pp

tcorrect/correct

otte/half/tr

ppm

egrep

std.dev

Image

size

x

Subsampled

by

scaled

by

Figure

Michael

Otte

's

sequence

estimated

quadtree

and

estimated

flow

merging

s

patches

in

a

pyramid

estimate

by

first

computing

motion

estimates

with

fewer

frames

-LRB-

this

is

because

gradient

descent

gets

trapped

in

local

minima

when

the

inter-frame

displacements

become

large

-RRB-

When

the

motion

is

not

linear

i.e.

we

have

a

non-zero

acceleration

we

can

not

perform

a

single

batch

optimization

Instead

we

can

compute

a

separate

flow

field

between

each

pair

of

images

using

the

previous

flow

as

an

initial

guess

Alternatively

we

can

compute

the

motion

between

a

base

image

and

each

successive

image

using

a

linear

predictor

ut

ut

-LRB-

ut

ut

-RRB-

This

latter

approach

is

useful

if

we

are

trying

to

track

feature

points

without

the

problem

of

drift

-LRB-

accumulated

error

-RRB-

which

can

occur

if

we

just

use

inter-frame

flows

The

linearly

predicted

multiframe

motion

estimator

forms

the

basis

of

our

parallel

extended

image

sequence

feature

tracker

-LSB-

Szeliski

et

al.

-RSB-

To

separate

locations

in

the

image

where

features

are

being

tracked

reliably

from

uninformative

or

confusing

regions

we

use

a

combination

of

the

local

Hessian

estimate

-LRB-

-RRB-

and

the

local

intensity

error

within

each

spline

patch

This

is

similar

to

Shi

and

Tomasi

's

tracker

-LSB-

Shi

and

Tomasi

-RSB-

except

that

we

use

bilinear

patches

stitched

together

by

the

spline

motion

model

which

yields

better

stability

than

isolated

affine

patches

To

deal

with

the

local

minima

which

can

trap

our

gradient

descent

technique

we

are

also

adding

an

exhaustive

search

component

to

our

algorithm

At

the

beginning

of

each

set

of

iterations

e.g.

Discussion

and

Conclusions

after

inter-level

transfers

in

the

coarse

to

fine

algorithm

or

after

splitting

in

the

quadtree

spline

estimator

we

search

around

the

current

-LRB-

u

v

-RRB-

estimate

by

trying

a

discrete

set

of

nearby

-LRB-

u

v

-RRB-

values

-LRB-

as

in

SSD

algorithms

-LSB-

Anandan

-RSB-

-RRB-

However

because

we

must

maintain

spline

continuity

we

can

not

make

the

selection

of

best

motion

estimate

for

each

patch

independently

Instead

we

average

the

motion

estimates

of

neighboring

patches

to

determine

the

motion

of

each

spline

control

vertex

In

future

work

we

plan

to

extend

our

algorithm

to

handle

occlusions

in

order

to

improve

the

accuracy

of

the

flow

estimates

The

first

part

which

is

simpler

to

implement

is

to

simply

detect

foldovers

i.e.

when

one

region

occludes

another

due

to

faster

motion

and

to

disable

error

contributions

from

the

occluded

background

The

second

part

would

be

to

add

an

explicit

occlusion

model

which

is

not

as

straightforward

because

our

splines

are

currently

continuous

In

other

work

we

would

also

like

to

study

the

suitability

of

our

method

as

a

robust

way

to

bootstrap

layered

motion

models

We

also

plan

to

test

our

technique

on

standard

stereo

problems

Discussion

and

Conclusions

The

quadtree-spline

motion

algorithm

we

have

developed

provides

a

novel

way

of

computing

an

accurate

motion

estimate

while

performing

an

initial

segmentation

of

the

motion

field

Our

approach

optimizes

the

same

stability

versus

detail

tradeoff

as

adaptively-sized

correlation

windows

without

incurring

the

large

computational

cost

of

overlapping

windows

and

trial-and-error

window

size

adjustment

Compared

to

the

recursively

split

affine

patch

tracker

of

-LSB-

M?uller

et

al.

-RSB-

our

technique

provides

a

higher

level

of

continuity

in

the

motion

field

which

leads

to

more

accurate

motion

estimates

The

general

framework

of

quadtree

splines

and

hierarchical

basis

functions

is

equally

applicable

to

other

computer

vision

problems

such

as

surface

interpolation

as

well

as

computer

graphics

and

numerical

relaxation

problems

It

has

already

been

applied

successfully

to

the

elastic

registration

of

medical

images

-LSB-

Szeliski

and

Lavall?ee

-RSB-

and

we

plan

to

extend

our

approach

to

other

applications

References

-LSB-

Adelson

and

Bergen

-RSB-

E.

H.

Adelson

and

J.

R.

Bergen

Spatiotemporal

energy

models

for

Discussion

and

Conclusions

the

perception

of

motion

Journal

of

the

Optical

Society

of

America

A

-LRB-

-RRB-

February

-LSB-

Aggarwal

and

Nandhakumar

-RSB-

J.

K.

Aggarwal

and

N.

Nandhakumar

On

the

computation

of

motion

from

sequences

of

images?a

review

Proceedings

of

the

IEEE

-LRB-

-RRB-

August

-LSB-

Anandan

-RSB-

P.

Anandan

A

computational

framework

and

an

algorithm

for

the

measurement

of

visual

motion

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Axelsson

and

Barker

-RSB-

O.

Axelsson

and

V.

A.

Barker

Finite

Element

Solution

of

Boundary

Value

Problems

Theory

and

Computation

Academic

Press

Inc.

Orlando

Florida

-LSB-

Barnard

and

Fischler

-RSB-

S.

T.

Barnard

and

M.

A.

Fischler

Computational

stereo

Computing

Surveys

-LRB-

-RRB-

December

-LSB-

Barron

et

al.

-RSB-

J.

L.

Barron

D.

J.

Fleet

and

S.

S.

Beauchemin

Performance

of

optical

flow

techniques

International

Journal

of

Computer

Vision

-LRB-

-RRB-

January

-LSB-

Bergen

et

al.

-RSB-

J.

R.

Bergen

P.

Anandan

K.

J.

Hanna

and

R.

Hingorani

Hierarchical

model-based

motion

estimation

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Santa

Margherita

Liguere

Italy

May

-LSB-

Black

and

Anandan

-RSB-

M.

J.

Black

and

P.

Anandan

A

framework

for

the

robust

estimation

of

optic

flow

In

Fourth

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Berlin

Germany

May

-LSB-

Black

and

Rangarajan

-RSB-

M.

J.

Black

and

A.

Rangarajan

The

outlier

process

Unifying

line

processes

and

robust

statistics

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Seattle

Washington

June

-LSB-

Bober

and

Kittler

-RSB-

M.

Bober

and

J.

Kittler

Estimation

of

complex

multimodal

motion

An

approach

based

on

robust

statistics

and

hough

transform

In

British

Machine

Vision

Conference

pages

BMVA

Press

-LSB-

Bolles

et

al.

-RSB-

R.

C.

Bolles

H.

H.

Baker

and

D.

H.

Marimont

Epipolar-plane

image

analysis

An

approach

to

determining

structure

from

motion

International

Journal

of

Computer

Vision

-LSB-

Brown

-RSB-

L.

G.

Brown

A

survey

of

image

registration

techniques

Computing

Surveys

-LRB-

-RRB-

December

Discussion

and

Conclusions

-LSB-

Burt

and

Adelson

-RSB-

P.

J.

Burt

and

E.

H.

Adelson

The

Laplacian

pyramid

as

a

compact

image

code

IEEE

Transactions

on

Communications

-LRB-

-RRB-

April

-LSB-

Chin

et

al.

-RSB-

T.

M.

Chin

M.

R.

Luettgen

W.

C.

Karl

and

A.

S.

Willsky

An

estimationtheoretic

perspective

on

image

processing

and

the

calculation

of

optic

flow

In

Advances

in

Image

Sequence

Processing

Kluwer

March

-LSB-

Dhond

and

Aggarwal

-RSB-

U.

R.

Dhond

and

J.

K.

Aggarwal

Structure

from

stereo?a

review

IEEE

Transactions

on

Systems

Man

and

Cybernetics

-LRB-

-RRB-

November/December

-LSB-

Enkelmann

-RSB-

W.

Enkelmann

Investigations

of

multigrid

algorithms

for

estimation

of

optical

flow

fields

in

image

sequences

Computer

Vision

Graphics

and

Image

Processing

-LSB-

Etoh

and

Shirai

-RSB-

M.

Etoh

and

Y.

Shirai

Segmentation

and

motion

estimation

by

region

fragments

In

Fourth

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Berlin

Germany

May

-LSB-

Fleet

and

Jepson

-RSB-

D.

Fleet

and

A.

Jepson

Computation

of

component

image

velocity

from

local

phase

information

International

Journal

of

Computer

Vision

-LSB-

Geiger

and

Girosi

-RSB-

D.

Geiger

and

F.

Girosi

Mean

field

theory

for

surface

reconstruction

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

May

-LSB-

Heeger

-RSB-

D.

J.

Heeger

Optical

flow

from

spatiotemporal

filters

In

First

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

London

England

June

-LSB-

Hildreth

-RSB-

E.

C.

Hildreth

Computing

the

velocity

field

along

contours

In

N.

I.

Badler

and

J.

K.

Tsotsos

editors

Motion

Representation

and

Perception

pages

NorthHolland

New

York

New

York

-LSB-

Horn

and

Schunck

-RSB-

B.

K.

P.

Horn

and

B.

G.

Schunck

Determining

optical

flow

Artificial

Intelligence

-LSB-

Irani

et

al.

-RSB-

M.

Irani

B.

Rousso

and

S.

Peleg

Detecting

and

tracking

multiple

moving

objects

using

temporal

integration

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Santa

Margherita

Liguere

Italy

May

Discussion

and

Conclusions

-LSB-

Jepson

and

Black

-RSB-

A.

Jepson

and

M.

J.

Black

Mixture

models

for

optical

flow

computation

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

New

York

New

York

June

-LSB-

Kimeldorf

and

Wahba

-RSB-

G.

Kimeldorf

and

G.

Wahba

A

correspondence

between

Bayesian

estimation

on

stochastic

processes

and

smoothing

by

splines

The

Annals

of

Mathematical

Statistics

-LRB-

-RRB-

-LSB-

Lavall?ee

et

al.

-RSB-

S.

Lavall?ee

R.

Szeliski

and

L.

Brunie

Matching

smooth

surfaces

with

their

projections

using

distance

maps

In

SPIE

Vol

Geometric

Methods

in

Computer

Vision

pages

Society

of

Photo-Optical

Instrumentation

Engineers

San

Diego

July

-LSB-

Lucas

-RSB-

B.

D.

Lucas

Generalized

Image

Matching

by

the

Method

of

Differences

PhD

thesis

Carnegie

Mellon

University

July

-LSB-

Lucas

and

Kanade

-RSB-

B.

D.

Lucas

and

T.

Kanade

An

iterative

image

registration

technique

with

an

application

in

stereo

vision

In

Seventh

International

Joint

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

pages

Vancouver

-LSB-

Luettgen

et

al.

-RSB-

M.

R.

Luettgen

W.

C.

Karl

and

A.

S.

Willsky

Efficient

multiscale

regularization

with

applications

to

the

computation

of

optical

flow

IEEE

Transactions

on

Image

Processing

-LRB-

-RRB-

January

-LSB-

Mallat

-RSB-

S.

G.

Mallat

A

theory

for

multiresolution

signal

decomposition

the

wavelet

representation

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

PAMI

-LRB-

-RRB-

July

-LSB-

Marroquin

et

al.

-RSB-

J.

Marroquin

S.

Mitter

and

T.

Poggio

Probabilistic

solution

of

illposed

problems

in

computational

vision

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

March

-LSB-

Matthies

et

al.

-RSB-

L.

H.

Matthies

R.

Szeliski

and

T.

Kanade

Kalman

filter-based

algorithms

for

estimating

depth

fromimage

sequences

International

Journal

of

Computer

Vision

-LSB-

M?uller

et

al.

-RSB-

J.

R.

M?uller

P.

Anandan

and

J.

R.

Bergen

Adaptive-complexity

registration

of

images

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Seattle

Washington

June

Discussion

and

Conclusions

-LSB-

Nagel

-RSB-

H.-H

Nagel

On

the

estimation

of

optical

flow

Relations

between

different

approaches

and

some

new

results

Artificial

Intelligence

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

locally

adaptive

window

for

signal

matching

International

Journal

of

Computer

Vision

-LRB-

-RRB-

April

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

multiple

baseline

stereo

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

April

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

stereo

matching

algorithm

with

an

adaptive

window

Theory

and

experiment

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

September

-LSB-

Otte

and

Nagel

-RSB-

M.

Otte

and

H.-H

Nagel

Optical

flow

estimation

advances

and

comparisons

In

Third

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

SpringerVerlag

Stockholm

Sweden

May

-LSB-

Pentland

-RSB-

A.

P.

Pentland

Interpolation

using

wavelet

bases

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

April

-LSB-

Poggio

et

al.

-RSB-

T.

Poggio

V.

Torre

and

C.

Koch

Computational

vision

and

regularization

theory

Nature

-LRB-

-RRB-

September

-LSB-

Press

et

al.

-RSB-

W.

H.

Press

B.

P.

Flannery

S.

A.

Teukolsky

and

W.

T.

Vetterling

Numerical

Recipes

in

C

The

Art

of

Scientific

Computing

Cambridge

University

Press

Cambridge

England

second

edition

-LSB-

Quam

-RSB-

L.

H.

Quam

Hierarchical

warp

stereo

In

Image

Understanding

Workshop

pages

Science

Applications

International

Corporation

New

Orleans

Louisiana

December

-LSB-

Rehg

and

Witkin

-RSB-

J.

Rehg

and

A.

Witkin

Visual

tracking

with

deformation

models

In

IEEE

International

Conference

on

Robotics

and

Automation

pages

IEEE

Computer

Society

Press

Sacramento

California

April

-LSB-

Samet

-RSB-

H.

Samet

The

Design

and

Analysis

of

Spatial

Data

Structures

Addison-Wesley

Reading

Massachusetts

-LSB-

Shi

and

Tomasi

-RSB-

J.

Shi

and

C.

Tomasi

Good

features

to

track

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Seattle

Washington

June

Discussion

and

Conclusions

-LSB-

Simchony

et

al.

-RSB-

T.

Simchony

R.

Chellappa

and

Z.

Lichtenstein

Pyramid

implementation

of

optimal-step

conjugate-search

algorithms

for

some

low-level

vision

problems

IEEE

Transactions

on

Systems

Man

and

Cybernetics

-LRB-

-RRB-

November/December

-LSB-

Simoncelli

et

al.

-RSB-

E.

P.

Simoncelli

E.

H.

Adelson

and

D.

J.

Heeger

Probability

distributions

of

optic

flow

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Press

Maui

Hawaii

June

-LSB-

Singh

-RSB-

A.

Singh

An

estimation-theoretic

framework

for

image-flow

computation

In

Third

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Osaka

Japan

December

-LSB-

Szeliski

-RSB-

R.

Szeliski

Bayesian

Modeling

of

Uncertainty

in

Low-Level

Vision

Kluwer

Academic

Publishers

Boston

Massachusetts

-LSB-

Szeliski

-RSB-

R.

Szeliski

Fast

surface

interpolation

using

hierarchical

basis

functions

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

June

-LSB-

Szeliski

-RSB-

R.

Szeliski

Fast

shape

from

shading

CVGIP

Image

Understanding

-LRB-

-RRB-

March

-LSB-

Szeliski

and

Coughlan

-RSB-

R.

Szeliski

and

J.

Coughlan

Hierarchical

spline-based

image

registration

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Seattle

Washington

June

-LSB-

Szeliski

and

Lavall?ee

-RSB-

R.

Szeliski

and

S.

Lavall?ee

Matching

anatomical

surfaces

with

non-rigid

deformations

using

octree-splines

In

IEEE

Workshop

on

Biomedical

Image

Analysis

pages

IEEE

Computer

Society

Seattle

Washington

June

-LSB-

Szeliski

and

Terzopoulos

-RSB-

R.

Szeliski

and

D.

Terzopoulos

Parallel

multigrid

algorithms

and

computer

vision

applications

In

Fourth

Copper

Mountain

Conference

on

Multigrid

Methods

pages

Society

for

Industrial

and

Applied

Mathematics

Copper

Mountain

Colorado

April

-LSB-

Szeliski

et

al.

-RSB-

R.

Szeliski

S.

B.

Kang

and

H.-Y

Shum

A

Parallel

Feature

Tracker

for

Extended

Image

Sequences

Technical

Report

Digital

Equipment

Corporation

Cambridge

Research

Lab

April

-LSB-

Terzopoulos

-RSB-

D.

Terzopoulos

Image

analysis

using

multigrid

relaxation

methods

IEEE

Discussion

and

Conclusions

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

March

-LSB-

Tomasi

and

Kanade

-RSB-

C.

Tomasi

and

T.

Kanade

Shape

and

motion

from

image

streams

under

orthography

A

factorization

method

International

Journal

of

Computer

Vision

-LRB-

-RRB-

November

-LSB-

Wang

and

Adelson

-RSB-

J.

Y.

A.

Wang

and

E.

H.

Adelson

Layered

representation

for

motion

analysis

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

New

York

New

York

June

-LSB-

Weber

and

Malik

-RSB-

J.

Weber

and

J.

Malik

Robust

computation

of

optical

flow

in

a

multiscale

differential

framework

In

Fourth

International

Conference

on

Computer

Vision

-LRB-

ICCV

-RRB-

pages

IEEE

Computer

Society

Press

Berlin

Germany

May

-LSB-

Witkin

et

al.

-RSB-

A.

Witkin

D.

Terzopoulos

and

M.

Kass

Signal

matching

through

scale

space

International

Journal

of

Computer

Vision

-LSB-

Yserentant

-RSB-

H.

Yserentant

On

the

multi-level

splitting

of

finite

element

spaces

Numerische

Mathematik

Acknowledgements

We

would

like

to

thank

John

Wang

and

Michael

Otte

for

providing

us

with

the

motion

sequences

used

in

their

research

-LSB-

Wang

and

Adelson

Otte

and

Nagel

-RSB-

motion analysis
optical flow
quadtree splines
splines

adaptive subdivision
hierarchical image registration
multiframe image analysis