Image

Mosaicing

for

Tele-Reality

Applications

Richard

Szeliski

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

May

Digital

Equipment

Corporation

has

four

research

facilities

the

Systems

Research

Center

and

the

Western

Research

Laboratory

both

in

Palo

Alto

California

the

Paris

Research

Laboratory

in

Paris

and

the

Cambridge

Research

Laboratory

in

Cambridge

Massachusetts

The

Cambridge

laboratory

became

operational

in

and

is

located

at

One

Kendall

Square

near

MIT

CRL

engages

in

computing

research

to

extend

the

state

of

the

computing

art

in

areas

likely

to

be

important

to

Digital

and

its

customers

in

future

years

CRL

's

main

focus

is

applications

technology

that

is

the

creation

of

knowledge

and

tools

useful

for

the

preparation

of

important

classes

of

applications

CRL

Technical

Reports

can

be

ordered

by

electronic

mail

To

receive

instructions

send

a

mes

sage

to

one

of

the

following

addresses

with

the

word

help

in

the

Subject

line

On

Digital

's

EASYnet

CRL

TECHREPORTS

On

the

Internet

techreports@crl.dec.com

This

work

may

not

be

copied

or

reproduced

for

any

commercial

purpose

Permission

to

copy

without

payment

is

granted

for

non?profit

educational

and

research

purposes

provided

all

such

copies

include

a

notice

that

such

copy

ing

is

by

permission

of

the

Cambridge

Research

Lab

of

Digital

Equipment

Corporation

an

acknowledgment

of

the

authors

to

the

work

and

all

applicable

portions

of

the

copyright

notice

The

Digital

logo

is

a

trademark

of

Digital

Equipment

Corporation

Cambridge

Research

Laboratory

One

Kendall

Square

Cambridge

Massachusetts

TM

Image

Mosaicing

for

Tele-Reality

Applications

Richard

Szeliski

Digital

Equipment

Corporation

Cambridge

Research

Lab

CRL

May

Abstract

While

a

large

number

of

virtual

reality

applications

such

as

fluid

flow

analysis

and

molecular

modeling

deal

with

simulated

data

many

newer

applications

attempt

to

recreate

true

reality

as

convincingly

as

possible

Building

detailed

models

for

such

applications

which

we

call

tele-reality

is

a

major

bottleneck

holding

back

their

deployment

In

this

paper

we

present

techniques

for

automatically

deriving

realistic

scenes

and

texture-mapped

models

from

video

sequences

which

can

help

overcome

this

bottleneck

The

fundamental

technique

we

use

is

image

mosaicing

i.e.

the

automatic

alignment

of

multiple

images

into

larger

aggregates

which

are

then

used

to

represent

portions

of

a

scene

We

begin

with

the

easiest

problems

those

of

flat

scene

and

panoramic

scene

mosaicing

and

progress

to

more

complicated

scenes

culminating

in

full

models

We

also

present

a

number

of

novel

applications

based

on

tele-reality

technology

c

Digital

Equipment

Corporation

All

rights

reserved

Contents

i

Contents

Introduction

Related

work

Basic

imaging

equations

Planar

image

mosaicing

Local

image

registration

Global

image

registration

Results

Panoramic

image

mosaicing

Results

Scenes

with

arbitrary

depth

Formulation

Results

Full

model

recovery

Applications

Planar

mosaicing

applications

model

building

applications

End-user

applications

Discussion

Conclusions

A

projective

transformations

ii

LIST

OF

FIGURES

List

of

Figures

Rigid

affine

and

projective

transformations

Whiteboard

image

mosaic

example

Panoramic

image

mosaic

example

-LRB-

bookshelf

and

cluttered

desk

-RRB-

Depth

recovery

example

table

with

stacks

of

papers

Depth

recovery

example

outdoor

scene

with

trees

model

recovery

example

Introduction

Introduction

Virtual

reality

is

currently

creating

a

lot

of

excitement

and

interest

in

the

computer

graphics

community

Typical

virtual

reality

systems

use

immersive

technologies

such

as

head-mounted

or

stereo

displays

and

data

gloves

The

intent

is

to

convince

users

that

they

are

interacting

with

an

alternate

physical

world

and

also

often

to

give

insight

into

computer

simulations

e.g.

for

fluid

flow

analysis

or

molecular

modeling

-LSB-

Earnshaw

et

al.

-RSB-

Realism

and

speed

of

rendering

are

therefore

important

issues

Another

class

of

virtual

reality

applications

attempts

to

recreate

true

reality

as

convincingly

as

possible

Examples

of

such

applications

include

flight

simulators

-LRB-

which

were

among

the

earliest

uses

of

virtual

reality

-RRB-

various

interactive

multi-player

games

and

medical

simulators

and

visualization

tools

Since

the

reality

being

recreated

is

usually

distant

we

use

the

term

telereality

in

this

paper

to

refer

to

such

virtual

reality

scenarios

based

on

real

imagery

Tele-reality

applications

which

could

be

built

using

the

techniques

described

in

this

paper

include

scanning

a

whiteboard

in

your

office

to

obtain

a

high-resolution

image

walkthroughs

of

existing

buildings

for

re-modeling

or

selling

participating

in

a

virtual

classroom

and

browsing

through

your

local

supermarket

aisles

from

home

-LRB-

see

Section

-RRB-

While

most

focus

in

virtual

reality

today

is

on

input/output

devices

and

the

quality

and

speed

of

rendering

perhaps

the

biggest

bottleneck

standing

in

the

way

of

widespread

tele-reality

applications

is

the

slow

and

tedious

model-building

process

-LSB-

Adam

-RSB-

This

also

affects

many

other

areas

of

computer

graphics

such

as

computer

animation

special

effects

and

CAD

For

small

objects

say

laser-based

scanners

are

a

good

solution

These

scanners

provide

registered

depth

and

colored

texture

maps

usually

in

either

raster

or

cylindrical

coordinates

-LSB-

Cyberware

Laboratory

Inc

Rioux

and

Bird

-RSB-

They

have

been

used

extensively

in

the

entertainment

industry

for

special

effects

and

computer

animation

Unfortunately

laser-based

scanners

are

fairly

expensive

limited

in

resolution

-LRB-

typically

pixels

-RRB-

and

most

importantly

limited

in

range

-LRB-

e.g.

they

can

not

be

used

to

scan

in

a

building

-RRB-

The

image-based

ranging

techniques

which

we

develop

in

this

paper

have

the

potential

to

overcome

these

limitations

Imagine

walking

through

an

environment

such

as

a

building

interior

and

filming

a

video

sequence

of

what

you

see

By

registering

and

compositing

the

images

in

the

term

telepresence

is

also

often

used

especially

for

applications

such

as

tele-medicine

where

two-way

interactivity

-LRB-

remote

operation

-RRB-

is

important

-LSB-

Earnshaw

et

al.

-RSB-

Related

work

video

together

into

large

mosaics

of

the

scene

image-based

ranging

can

achieve

an

essentially

unlimited

resolution

Since

the

images

can

be

acquired

using

any

optical

technology

-LRB-

from

microscopy

through

hand-held

videocams

to

satellite

photography

-RRB-

the

range

or

scale

of

the

scene

being

reconstructed

is

not

an

issue

Finally

as

desktop

video

becomes

ubiquitous

in

our

computing

environments?initially

for

videoconferencing

and

later

as

an

advanced

user

interface

tool

-LSB-

Gold

Rehg

and

Kanade

Blake

and

Isard

-RSB-

image-based

scene

and

model

acquisition

will

become

accessible

to

all

users

In

this

paper

we

develop

novel

techniques

for

extracting

large

textures

and

models

from

image

sequences

based

on

image

registration

and

compositing

techniques

and

also

present

some

potential

applications

After

a

review

of

related

work

-LRB-

Section

-RRB-

and

of

the

basic

image

formation

equations

-LRB-

Section

-RRB-

we

present

our

technique

for

registering

pieces

of

a

flat

-LRB-

planar

-RRB-

scene

which

is

the

simplest

interesting

image

mosaicing

problem

-LRB-

Section

-RRB-

We

then

show

how

the

same

technique

can

be

used

to

mosaic

panoramic

scenes

obtained

by

rotating

the

camera

around

its

center

of

projection

-LRB-

Section

-RRB-

Section

discusses

how

to

recover

depth

in

scenes

Section

discusses

the

most

general

and

difficult

problem

that

of

building

full

models

from

multiple

images

Section

presents

some

novel

applications

of

the

tele-reality

technology

developed

in

this

paper

Finally

we

close

with

a

comparison

of

our

work

with

previous

approaches

and

a

discussion

of

the

significance

of

our

results

Related

work

While

models

acquired

with

laser-based

scanners

are

now

commonly

used

in

computer

animations

and

for

special

effects

models

derived

directly

from

still

or

video

images

are

still

a

rarity

One

example

of

physically-based

models

derived

from

images

are

the

dancing

vegetables

in

the

Cooking

with

Kurt

video

-LSB-

Terzopoulos

and

Witkin

-RSB-

Another

more

recent

example

is

the

model

of

a

building

extracted

from

a

video

sequences

in

-LSB-

Azarbayejani

et

al.

-RSB-

which

was

then

composited

with

a

teapot

Somewhat

related

are

graphics

techniques

which

extract

camera

position

information

from

images

such

as

-LSB-

Gleicher

and

Witkin

-RSB-

traditional

use

of

the

term

image

compositing

-LSB-

Porter

and

Duff

Foley

et

al.

-RSB-

is

for

blending

images

which

are

already

registered

We

use

the

term

image

mosaicing

in

this

paper

for

our

work

to

avoid

confusion

with

this

previous

work

although

compositing

-LRB-

as

in

composite

sketches

-RRB-

also

seems

appropriate

Basic

imaging

equations

The

extraction

of

geometric

information

from

multiple

images

has

long

been

one

of

the

central

problems

in

computer

vision

-LSB-

Ballard

and

Brown

Horn

-RSB-

and

photogrammetry

-LSB-

Moffitt

and

Mikhail

-RSB-

However

many

of

the

techniques

used

are

based

on

feature

extraction

produce

sparse

descriptions

of

shape

and

often

require

manual

assistance

Certain

techniques

such

as

stereo

produce

depth

or

elevation

maps

which

are

inadequate

to

model

true

objects

In

computer

vision

the

recovered

geometry

is

normally

used

either

for

object

recognition

or

for

grasping

and

navigation

-LRB-

robotics

-RRB-

Relatively

little

attention

has

been

paid

to

building

models

registered

with

intensity

-LRB-

texture

maps

-RRB-

which

are

necessary

for

computer

graphics

and

virtual

reality

applications

-LRB-

but

see

-LSB-

Mann

-RSB-

for

recent

work

which

addresses

some

of

the

same

problems

as

this

paper

-RRB-

In

computer

graphics

compositing

multiple

image

streams

together

to

create

larger

format

-LRB-

Omnimax

-RRB-

images

is

discussed

in

-LSB-

Greene

and

Heckbert

-RSB-

However

in

this

application

the

relative

position

of

the

cameras

was

known

in

advance

The

registration

techniques

developed

in

this

paper

are

related

to

image

warping

-LSB-

Wolberg

-RSB-

since

once

the

images

are

registered

they

can

be

warped

into

a

common

reference

frame

before

being

composited

While

most

current

techniques

require

the

manual

specification

of

feature

correspondences

some

recent

techniques

-LSB-

Beymer

et

al.

-RSB-

as

well

as

the

techniques

developed

in

this

paper

can

be

used

to

automate

this

process

Combinations

of

local

image

warping

and

compositing

are

now

commonly

used

for

special

effects

under

the

general

rubric

of

morphing

-LSB-

Beier

and

Neely

-RSB-

Recently

morphing

based

on

z-buffer

depths

and

camera

motion

has

been

applied

to

view

interpolation

as

a

quick

alternative

to

full

rendering

-LSB-

Chen

and

Williams

-RSB-

The

techniques

we

develop

in

Section

can

be

used

to

compute

the

depth

maps

necessary

for

this

approach

directly

from

real-world

image

sequences

Basic

imaging

equations

Many

of

the

ideas

in

this

paper

have

their

simplest

expression

using

projective

geometry

-LSB-

Semple

and

Kneebone

-RSB-

However

rather

than

relying

on

these

results

we

will

use

more

standard

methods

and

notations

from

computer

graphics

-LSB-

Foley

et

al.

-RSB-

and

prove

whatever

simple

can

view

the

image

registration

task

as

an

inverse

warping

problem

since

we

are

given

two

images

and

asked

to

recover

the

unknown

warping

function

Basic

imaging

equations

Figure

Rigid

affine

and

projective

transformations

results

we

require

in

Appendix

A.

Throughout

we

will

use

homogeneous

coordinates

to

represent

points

i.e.

we

denote

points

in

the

image

plane

as

-LRB-

x

y

w

-RRB-

with

-LRB-

x

w

y

w

-RRB-

being

the

corresponding

Cartesian

coordinates

-LSB-

Foley

et

al.

-RSB-

Similarly

points

with

homogeneous

coordinates

-LRB-

x

y

z

w

-RRB-

have

Cartesian

coordinates

-LRB-

x

w

y

w

z

w

-RRB-

Using

homogeneous

coordinates

we

can

describe

the

class

of

planar

transformations

using

matrix

multiplication

x

y

w

or

-LRB-

-RRB-

The

simplest

transformations

in

this

general

class

are

pure

translations

followed

by

translations

and

rotations

-LRB-

rigid

transforms

-RRB-

plus

scaling

-LRB-

similarity

transforms

-RRB-

affine

transforms

and

full

projective

transforms

Figure

shows

a

square

and

possible

rigid

affine

and

projective

transformations

Rigid

and

affine

transformations

have

the

following

forms

for

the

matrix

cos

sin

tx

sin

cos

ty

with

and

degrees

of

freedom

respectively

while

projective

transformations

have

a

general

matrix

with

degrees

of

freedom

In

we

have

the

same

hierarchy

of

transformations

with

rigid

-LRB-

degrees

of

freedom

-RRB-

similarity

-LRB-

dof

-RRB-

affine

-LRB-

dof

-RRB-

and

full

projective

-LRB-

dof

-RRB-

transforms

The

matrices

in

this

matrices

are

equivalent

if

they

are

scalar

multiples

of

each

other

We

remove

this

redundancy

by

setting

Planar

image

mosaicing

case

are

Of

particular

interest

are

the

rigid

-LRB-

Euclidean

-RRB-

transformation

E

R

t

T

-LRB-

-RRB-

where

R

is

a

orthonormal

rotation

matrix

and

t

is

a

translation

vector

and

the

viewing

matrix

V

h

V

i

f

-LRB-

-RRB-

which

projects

points

through

the

origin

onto

a

projection

plane

a

distance

f

along

the

z

axis

-LSB-

Foley

et

al.

-RSB-

The

V

matrix

can

be

a

general

matrix

in

the

case

where

the

internal

camera

parameters

are

unknown

The

last

column

of

V

is

always

zero

for

central

projection

The

combined

equations

projecting

a

world

coordinate

p

-LRB-

x

y

z

w

-RRB-

onto

a

screen

location

u

-LRB-

-RRB-

can

thus

be

written

as

u

VEp

Mcamp

-LRB-

-RRB-

where

Mcam

is

a

camera

matrix

This

equation

is

valid

even

if

the

camera

calibration

parameters

and/or

the

camera

orientation

are

unknown

Planar

image

mosaicing

The

simplest

possible

set

of

images

to

mosaic

are

pieces

of

a

planar

scene

such

as

a

document

whiteboard

or

flat

desktop

Imagine

that

we

have

a

camera

fixed

directly

over

our

desk

As

we

slide

a

document

under

the

camera

different

portions

of

the

document

are

visible

Any

two

such

pieces

are

related

to

each

other

by

a

translation

and

a

rotation

-LRB-

rigid

transformation

-RRB-

Now

imagine

that

we

are

scanning

a

whiteboard

with

a

hand-held

video

camera

The

class

of

transformations

relating

two

pieces

of

the

board

is

more

general

in

this

case

It

is

easy

to

see

that

this

class

is

the

family

of

projective

transformations

-LRB-

just

imagine

how

a

square

or

grid

in

one

image

can

appear

in

another

-RRB-

These

transformations

can

be

computed

without

any

knowledge

of

the

internal

camera

calibration

parameters

-LRB-

such

as

focal

length

or

optical

center

-RRB-

or

of

the

relative

camera

motion

between

frames

The

fact

that

projective

transformations

capture

all

such

possible

mappings

is

a

basic

result

of

projective

geometry

-LRB-

see

Appendix

A.

-RRB-

Planar

image

mosaicing

Given

this

knowledge

how

do

we

go

about

computing

the

transformations

relating

the

various

scene

pieces

so

that

we

can

paste

them

together

A

variety

of

techniques

are

possible

some

more

automated

than

others

For

example

we

could

manually

identify

four

or

more

corresponding

points

between

the

two

views

which

is

enough

information

to

solve

for

the

eight

unknowns

in

the

projective

transformation

Or

we

could

iteratively

adjust

the

relative

positions

of

input

images

using

either

a

blink

comparator

or

transparency

-LSB-

Carlbom

et

al.

-RSB-

These

kinds

of

manual

approaches

are

too

tedious

to

be

useful

for

large

tele-reality

applications

Local

image

registration

The

approach

we

use

in

this

paper

is

to

directly

minimize

the

discrepancy

in

intensities

between

pairs

of

images

after

applying

the

transformation

we

are

recovering

This

has

the

advantage

of

not

requiring

any

easily

identifiable

feature

points

and

of

being

statistically

optimal

once

we

are

in

the

vicinity

of

the

true

solution

-LSB-

Szeliski

and

Coughlan

-RSB-

More

concretely

suppose

we

describe

our

transformations

as

-LRB-

-RRB-

Our

technique

minimizes

the

sum

of

the

squared

intensity

errors

E

X

i

-LSB-

I

-LRB-

-RRB-

I

-LRB-

xi

yi

-RRB-

-RSB-

X

i

-LRB-

-RRB-

over

all

corresponding

pairs

of

pixels

i

which

are

inside

both

images

I

-LRB-

x

y

-RRB-

and

I

-LRB-

-RRB-

-LRB-

pixels

which

are

mapped

outside

image

boundaries

do

not

contribute

-RRB-

Once

we

have

found

the

best

transformation

we

can

warp

image

I

into

the

reference

frame

of

I

using

M

-LSB-

Wolberg

-RSB-

and

then

blend

the

two

images

together

To

reduce

visible

artifacts

we

weight

images

being

blended

together

more

heavily

towards

the

center

using

a

bilinear

weighting

function

To

perform

the

minimization

we

use

the

Levenberg-Marquardt

iterative

non-linear

minimization

algorithm

-LSB-

Press

et

al.

-RSB-

This

algorithm

requires

the

computation

of

the

partial

derivatives

of

ei

with

respect

to

the

unknown

motion

parameters

These

are

straightforward

to

compute

i.e.

@ei

xi

Di

@I

@ei

yi

Di

@I

@I

-LRB-

-RRB-

Local

image

registration

where

Di

is

the

denominator

in

-LRB-

-RRB-

and

-LRB-

@I

@I

-RRB-

is

the

image

intensity

gradient

of

I

at

-LRB-

-RRB-

From

these

partials

the

Levenberg-Marquardt

algorithm

computes

an

approximate

Hessian

matrix

A

and

the

weighted

gradient

vector

b

with

components

akl

X

i

@ei

@mk

@ei

@ml

bk

X

i

ei

@ei

@mk

-LRB-

-RRB-

and

then

updates

the

motion

parameter

estimate

m

by

an

amount

Dm

-LRB-

A

I

-RRB-

where

is

a

time-varying

stabilization

parameter

-LSB-

Press

et

al.

-RSB-

The

advantage

of

using

LevenbergMarquardt

over

straightforward

gradient

descent

is

that

it

converges

in

fewer

iterations

The

complete

registration

algorithm

thus

consists

of

the

following

steps

For

each

pixel

i

at

location

-LRB-

xi

yi

-RRB-

-LRB-

a

-RRB-

compute

its

corresponding

position

in

the

other

image

-LRB-

-RRB-

using

-LRB-

-RRB-

-LRB-

b

-RRB-

compute

the

error

in

intensity

between

the

corresponding

pixels

ei

I

-LRB-

-RRB-

I

-LRB-

xi

yi

-RRB-

and

the

intensity

gradient

-LRB-

@I

@I

-RRB-

-LRB-

c

-RRB-

compute

the

partial

derivative

of

ei

w.r.t.

the

mk

using

@ei

@mk

@I

@mk

@I

@mk

as

in

-LRB-

-RRB-

-LRB-

d

-RRB-

add

the

pixel

's

contribution

to

A

and

b

as

in

-LRB-

-RRB-

Solve

the

system

of

equations

-LRB-

A

I

-RRB-

Dm

b

and

update

the

motion

estimate

m

-LRB-

t

-RRB-

m

-LRB-

t

-RRB-

Dm

Check

that

the

error

in

-LRB-

-RRB-

has

decreased

if

not

increment

-LRB-

as

described

in

-LSB-

Press

et

al.

-RSB-

-RRB-

and

compute

a

new

Dm

Continue

iterating

until

the

error

is

below

a

threshold

or

a

fixed

number

of

steps

has

been

completed

The

steps

in

this

algorithm

are

similar

to

the

operations

performed

when

warping

images

-LSB-

Wolberg

Beier

and

Neely

-RSB-

with

additional

operations

for

correcting

the

current

warping

parameters

based

on

the

local

intensity

error

and

its

gradients

-LRB-

similar

to

the

process

in

-LSB-

Gleicher

and

Witkin

-RSB-

-RRB-

For

more

details

on

the

exact

implementation

see

-LSB-

Szeliski

and

Coughlan

-RSB-

Planar

image

mosaicing

Global

image

registration

Unfortunately

both

gradient

descent

and

Levenberg-Marquardt

only

find

locally

optimal

solutions

If

the

motion

between

successive

frames

is

large

we

must

use

a

different

strategy

to

find

the

best

registration

We

have

implemented

two

different

techniques

for

dealing

with

this

problem

The

first

technique

which

is

commonly

used

in

computer

vision

is

hierarchical

matching

which

first

registers

smaller

subsampled

versions

of

the

images

where

the

apparent

motion

is

smaller

-LSB-

Quam

Witkin

et

al.

Bergen

et

al.

-RSB-

Motion

estimates

from

these

smaller

coarser

levels

are

then

used

to

initialize

motion

estimates

at

finer

levels

thereby

avoiding

the

local

minimum

problem

-LRB-

see

-LSB-

Szeliski

and

Coughlan

-RSB-

for

details

-RRB-

While

this

technique

is

not

guaranteed

to

find

the

correct

registration

we

have

observed

empirically

that

it

works

well

when

the

initial

misregistration

is

only

a

few

pixels

-LRB-

the

exact

domain

of

convergence

depends

on

the

intensity

pattern

in

the

image

-RRB-

For

larger

displacements

we

use

phase

correlation

-LSB-

Kuglin

and

Hines

Brown

-RSB-

This

technique

estimates

the

translation

between

a

pair

of

images

by

taking

Fourier

Transforms

of

each

image

computing

the

phase

difference

at

each

frequency

performing

an

inverse

Fourier

Transform

and

searching

for

a

peak

in

the

magnitude

image

In

our

experiments

this

technique

has

proven

to

work

remarkably

well

providing

good

initial

guesses

for

image

pairs

which

overlap

by

as

little

as

The

technique

will

not

work

however

if

the

inter-frame

motion

is

not

mostly

translational

-LRB-

large

rotations

or

zooms

-RRB-

but

this

does

not

occur

in

practice

with

our

current

image

sequences

Results

To

demonstrate

the

performance

of

our

algorithm

we

digitized

an

image

sequence

with

a

camera

panning

over

a

whiteboard

Figure

shows

the

final

mosaic

of

the

whiteboard

with

the

locations

of

the

constituent

images

in

the

mosaic

shown

as

black

outlines

Figure

is

a

portion

of

the

final

mosaic

without

these

outlines

This

mosaic

is

pixels

based

on

compositing

NTSC

-LRB-

-RRB-

resolution

images

To

compute

this

mosaic

we

developed

an

interactive

image

mosaicing

tool

which

lets

the

user

coarsely

position

successive

frames

relative

to

each

other

The

tool

also

has

an

automatic

mosaicing

option

which

computes

the

initial

rough

placement

of

each

image

with

respect

to

the

previous

one

Panoramic

image

mosaicing

using

phase

correlation

Our

algorithm

then

refines

the

location

of

each

image

by

minimizing

-LRB-

-RRB-

using

the

current

mosaic

thus

far

as

I

-LRB-

x

y

-RRB-

and

the

input

frame

being

adjusted

as

I

-LRB-

-RRB-

The

images

in

Figure

were

automatically

composited

without

user

intervention

using

the

middle

frame

-LRB-

center

of

the

image

-RRB-

as

the

anchor

image

-LRB-

no

deformations

-RRB-

As

we

can

see

our

technique

works

well

on

this

example

Section

discusses

some

imaging

artifacts

which

can

cause

difficulties

Panoramic

image

mosaicing

Another

image

mosaicing

problem

which

turns

out

to

be

equally

simple

to

solve

is

panoramic

image

mosaicing

In

this

scenario

we

rotate

a

camera

around

its

optical

center

in

order

to

create

a

full

viewing

sphere

of

images

-LRB-

in

computer

graphics

this

representation

is

sometimes

called

an

environment

map

-LSB-

Greene

-RSB-

-RRB-

This

is

similar

to

the

action

of

panoramic

still

photographic

cameras

where

the

rotation

of

a

camera

on

top

of

a

tripod

is

mechanically

coordinated

with

the

film

transport

the

anamorphic

lens

used

in

-LSB-

Lippman

-RSB-

and

to

cinema

in

the

round

movie

theaters

-LSB-

Greene

and

Heckbert

-RSB-

In

our

case

however

we

can

mosaic

multiple

images

of

arbitrary

detail

and

resolution

and

we

need

not

know

the

camera

motion

Examples

of

applications

include

constructing

true

scenic

panoramas

-LRB-

say

of

the

view

at

the

rim

of

the

Grand

Canyon

-RRB-

or

limited

virtual

environments

-LRB-

recreating

a

meeting

room

or

office

as

seen

from

one

location

-RRB-

It

turns

out

that

images

taken

from

the

same

viewpoint

-LRB-

stationary

optical

center

-RRB-

are

related

by

projective

transformations

just

as

in

the

planar

scene

case

-LRB-

see

Appendix

A

for

a

quick

proof

-RRB-

Intuitively

we

can

not

tell

the

relative

depth

of

points

in

the

scene

as

we

rotate

-LRB-

there

is

no

motion

parallax

-RRB-

so

they

may

as

well

be

located

on

any

plane

-LRB-

in

projective

geometry

we

could

say

they

lie

on

the

plane

at

infinity

w

-RRB-

For

some

applications

this

property

is

usually

viewed

as

a

deficiency

i.e.

we

can

not

recover

scene

depth

from

rotational

camera

motion

For

image

mosaicing

this

is

actually

an

advantage

since

we

just

want

to

composite

a

large

scene

and

be

able

to

look

around

from

a

static

viewpoint

More

formally

the

transformation

denoted

by

is

related

to

the

viewing

matrices

V

and

and

the

inter-view

rotation

R

by

V

-LRB-

-RRB-

-LRB-

see

Appendix

A.

-RRB-

In

the

case

of

a

calibrated

camera

-LRB-

known

center

of

projection

-RRB-

this

has

a

Panoramic

image

mosaicing

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Whiteboard

image

mosaic

example

-LRB-

a

-RRB-

mosaic

with

component

locations

shown

as

black

outlines

-LRB-

b

-RRB-

portion

of

mosaic

in

greater

detail

Results

particularly

simple

form

f

f

f

-LRB-

-RRB-

where

f

and

f

are

the

scaled

focal

lengths

in

the

two

views

and

rkl

are

the

entries

in

the

rotation

matrix

R.

In

this

case

we

only

have

to

recover

five

independent

parameters

-LRB-

or

three

if

the

f

values

are

known

-RRB-

instead

of

the

usual

eight

How

do

we

represent

a

panoramic

scene

composited

using

our

techniques

One

approach

is

to

divide

the

viewing

sphere

into

several

large

potentially

overlapping

regions

and

to

represent

each

region

with

a

plane

onto

which

we

paste

the

images

-LSB-

Greene

-RSB-

Another

approach

is

to

compute

the

relative

position

of

each

frame

relative

to

some

base

frame

and

to

then

re-compute

an

arbitrary

view

on

the

fly

from

all

visible

pieces

given

a

particular

view

direction

R

and

zoom

factor

f

In

our

current

implementation

we

adopt

the

former

strategy

since

it

is

a

simple

extension

of

the

planar

case

Results

Figure

shows

a

mosaic

of

a

bookshelf

and

cluttered

desk

which

is

obviously

a

non-planar

scene

The

images

were

obtained

by

tilting

and

panning

a

video

camera

mounted

on

a

tripod

without

any

special

steps

taken

to

ensure

that

the

rotation

was

around

the

true

center

of

projection

The

complete

scene

is

registered

quite

well

Scenes

with

arbitrary

depth

Mosaicing

flat

scenes

or

panoramic

scenes

may

be

interesting

for

certain

tele-reality

applications

e.g.

transmitting

whiteboard

or

document

contents

or

viewing

an

outdoor

panorama

However

for

most

applications

we

must

recover

the

depth

associated

with

the

scene

to

give

the

illusion

of

a

environment

e.g.

through

nearby

view

synthesis

-LSB-

Chen

and

Williams

-RSB-

Two

possible

approaches

are

to

model

the

scene

as

piecewise-planar

or

to

recover

dense

depth

maps

large

focal

lengths

and

small

rotations

resembles

a

pure

translation

matrix

Scenes

with

arbitrary

depth

-LRB-

a

-RRB-

-LRB-

c

-RRB-

Figure

Panoramic

image

mosaic

example

-LRB-

bookshelf

and

cluttered

desk

-RRB-

-LRB-

a

-RRB-

mosaic

with

component

locations

shown

as

black

outlines

-LRB-

b

-RRB-

portion

of

mosaic

in

greater

detail

Formulation

The

first

approach

is

to

assume

that

the

scene

is

piecewise-planar

as

is

the

case

with

many

man-made

environments

such

as

building

exteriors

and

office

interiors

The

image

mosaicing

technique

developed

in

Section

can

then

be

applied

to

each

of

the

planar

regions

in

the

image

The

segmentation

of

each

image

into

its

planar

components

can

either

be

done

interactively

-LRB-

e.g.

by

drawing

the

polygonal

outline

of

each

region

to

be

registered

-RRB-

or

automatically

by

associating

each

pixel

with

one

of

several

global

motion

hypotheses

-LSB-

Jepson

and

Black

Wang

and

Adelson

-RSB-

Once

the

independent

planar

pieces

have

been

composited

we

could

in

principle

recover

the

relative

geometry

of

the

various

planes

and

the

camera

motion

-LRB-

see

Appendix

A.

-RRB-

However

rather

than

pursuing

this

approach

in

this

paper

we

will

purse

the

second

more

general

solution

which

is

to

recover

a

full

depth

map

i.e.

to

infer

the

missing

z

component

associated

with

each

pixel

in

a

given

image

sequence

When

the

camera

motion

is

known

the

problem

of

depth

map

recovery

is

called

stereo

reconstruction

-LRB-

or

multi-frame

stereo

if

more

than

two

views

are

used

-LSB-

Matthies

et

al.

Okutomi

and

Kanade

-RSB-

-RRB-

This

problem

has

been

extensively

studied

in

photogrammetry

and

computer

vision

-LSB-

Moffitt

and

Mikhail

Barnard

and

Fischler

Horn

Dhond

and

Aggarwal

-RSB-

When

the

camera

motion

is

unknown

we

have

the

more

difficult

structure

from

motion

problem

-LSB-

Horn

Weng

et

al.

Azarbayejani

et

al.

Szeliski

and

Kang

-RSB-

In

this

section

we

present

our

solution

to

this

latter

problem

based

on

recovering

projective

depth

which

is

particularly

simple

and

robust

and

fits

in

well

with

the

material

already

developed

in

this

paper

Readers

interested

in

alternative

techniques

should

consult

the

above

references

Formulation

To

formulate

the

projective

structure

from

motion

recovery

problem

we

note

that

the

coordinates

corresponding

to

a

pixel

u

with

projective

depth

w

in

some

other

frame

can

be

written

as

V

w

w?t

-LRB-

-RRB-

where

V

E

R

and

t

are

defined

in

-LRB-

-RRB-

and

and

t

are

the

computed

planar

projection

matrix

and

epipole

direction

-LRB-

see

-LRB-

-RRB-

in

Appendix

A.

-RRB-

To

recover

the

parameters

in

and

t

for

each

frame

along

with

the

depth

values

w

-LRB-

which

are

the

same

for

all

frames

-RRB-

we

use

the

same

Levenberg-Marquardt

algorithm

as

before

Full

model

recovery

In

more

detail

we

write

the

projection

equation

as

-LRB-

-RRB-

We

compute

the

partial

derivatives

of

and

w.r.t.

the

mk

and

tk

-LRB-

which

we

concatenate

into

the

motion

vector

m

-RRB-

as

before

in

-LRB-

-RRB-

We

similarly

compute

the

partials

of

and

with

respect

to

wi

i.e.

@wi

@wi

-LRB-

-RRB-

where

Di

is

the

denominator

in

-LRB-

-RRB-

To

estimate

the

unknown

parameters

we

alternate

iterations

of

the

Levenberg-Marquardt

algorithm

over

the

motion

parameters

in

m

and

the

depth

parameters

fwig

using

the

partial

derivatives

defined

above

to

compute

the

approximate

Hessian

matrices

A

and

the

weighted

error

vectors

b

as

in

-LRB-

-RRB-

In

our

current

implementation

in

order

to

reduce

the

total

number

of

parameters

being

estimated

we

represent

the

depth

map

using

a

tensor-product

spline

and

only

recover

the

depth

estimates

at

the

spline

control

vertices

-LRB-

the

complete

depth

map

is

available

by

interpolation

-RRB-

-LSB-

Szeliski

and

Coughlan

-RSB-

Results

Figure

shows

an

example

of

using

our

projective

depth

recovery

algorithm

The

image

sequence

was

taken

by

moving

the

camera

up

and

over

the

scene

of

a

table

with

stacks

of

papers

-LRB-

Figure

-RRB-

The

resulting

depth-map

is

shown

in

Figure

as

intensity-coded

range

values

Figures

d

show

the

original

intensity

image

texture

mapped

onto

the

surface

-LRB-

an

example

of

view

extrapolation

-RRB-

and

Figures

d

show

a

set

of

grid-lines

overlayed

on

the

recovered

surface

The

shape

is

recovered

reasonably

well

in

areas

where

there

is

sufficient

texture

-LRB-

uniform

intensity

areas

give

no

depth

cues

-RRB-

and

the

extrapolated

view

looks

reasonable

Figure

shows

results

from

another

sequence

this

time

from

an

outdoor

scene

of

some

trees

Again

the

geometry

of

the

scene

is

recovered

reasonably

well

as

indicated

in

the

depth

map

in

Figure

Full

model

recovery

Recovering

depth

maps

may

be

adequate

for

many

tele-reality

applications

e.g.

scanning

library

bookshelves

or

supermarket

aisles

but

for

other

applications

e.g.

when

we

need

to

see

the

back

Full

model

recovery

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

e

-RRB-

-LRB-

f

-RRB-

Figure

Depth

recovery

example

table

with

stacks

of

papers

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

intensity-coded

depth

map

-LRB-

dark

is

farther

back

-RRB-

-LRB-

c?d

-RRB-

texture-mapped

surface

seen

from

novel

viewpoints

-LRB-

e?f

-RRB-

gridded

surface

Full

model

recovery

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Depth

recovery

example

outdoor

scene

with

trees

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

intensity-coded

depth

map

-LRB-

dark

is

farther

back

-RRB-

side

of

an

object

full

models

may

be

required

This

is

the

most

difficult

image

mosaicing

problem

since

not

only

do

we

have

to

recover

depth

but

we

also

have

to

merge

-LRB-

register

and

composite

-RRB-

multiple

depth

maps

and

represent

objects

given

no

a

priori

knowledge

about

their

rough

shape

or

topology

For

simple

topologies

and

shapes

deformable

physically-based

models

can

do

a

good

job

of

recovering

the

unknown

geometry

-LSB-

Terzopoulos

et

al.

Pentland

and

Sclaroff

-RSB-

For

general

topologies

the

problem

is

more

difficult

Since

many

current

shape

recovery

techniques

produce

incomplete

or

sparse

geometric

descriptions

a

general

surface

interpolation

technique

may

be

required

to

generate

a

smooth

continuous

surface

-LSB-

Hoppe

et

al.

Szeliski

and

Tonnesen

-RSB-

There

exist

many

techniques

for

extracting

shape

from

multiple

views

For

example

we

can

recover

a

volumetric

description

from

the

binary

silhouettes

of

an

object

against

its

background

-LSB-

Szeliski

-RSB-

compute

local

optic

flow

-LRB-

pixel

motion

-RRB-

estimates

and

convert

these

into

sparse

point

estimates

-LSB-

Szeliski

-RSB-

or

track

the

occluding

contours

of

an

object

to

generate

space

curves

-LSB-

Szeliski

and

Weiss

-RSB-

A

complete

survey

of

shape

extraction

techniques

is

beyond

the

scope

of

this

paper

Instead

we

present

the

results

of

two

of

our

previously

developed

algorithms

applied

to

an

image

sequence

of

a

cup

rotating

on

a

calibrated

turntable

-LRB-

Figure

-RRB-

Full

model

recovery

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

model

recovery

example

-LRB-

a

-RRB-

input

image

-LRB-

b

-RRB-

octree

recovered

from

silhouettes

-LRB-

c

-RRB-

edges

-LRB-

d

-RRB-

extremal

contours

Applications

Our

first

technique

converts

each

image

into

a

binary

silhouette

by

differencing

the

image

with

an

empty

background

image

Each

cube

in

the

octree

volumetric

model

is

then

projected

-LRB-

using

the

known

camera

position

-RRB-

into

the

silhouette

and

cubes

that

fall

outside

the

silhouette

are

culled

Cubes

which

fall

partially

into

the

silhouette

are

marked

for

later

subdivision

and

the

process

is

repeated

after

each

complete

revolution

at

successively

finer

resolutions

-LSB-

Szeliski

-RSB-

The

resulting

octree

is

shown

in

Figure

Our

second

technique

extracts

silhouette

-LRB-

extremal

-RRB-

edges

from

the

image

sequence

and

uses

a

multi-frame

stereo

algorithm

to

reconstruct

their

position

-LRB-

internal

albedo

edges

are

also

extracted

and

reconstructed

-RRB-

-LSB-

Szeliski

and

Weiss

-RSB-

Figure

shows

one

of

the

edge

images

used

by

the

algorithm

and

Figure

shown

the

collection

of

curves

estimated

by

the

algorithm

These

examples

demonstrate

some

of

our

algorithms

for

reconstructing

an

isolated

object

undergoing

known

motion

Similar

techniques

can

be

used

to

solve

the

more

general

scene

recovery

problem

where

the

camera

motion

is

unknown

Methods

for

determining

the

motion

include

the

projective

motion

algorithm

presented

in

the

previous

section

as

well

as

techniques

described

in

-LSB-

Horn

Weng

et

al.

Azarbayejani

et

al.

-RSB-

Compared

to

the

and

depth-map

models

developed

in

Sections

the

recovery

of

models

is

more

difficult

and

less

robust

but

holds

the

promise

of

more

realistic

and

more

general

object

and

scene

models

Applications

Given

automated

techniques

for

building

and

scenes

from

video

sequences

what

can

we

do

with

these

models

In

this

section

we

describe

a

number

of

potential

applications

including

whiteboard

and

document

scanning

model

acquisition

for

inverse

CAD

model

acquisition

for

computer

animation

and

special

effects

supermarket

shopping

at

home

interactive

walkthroughs

of

historical

buildings

and

live

tele-reality

-LRB-

telepresence

-RRB-

applications

Planar

mosaicing

applications

The

most

straightforward

application

is

scanning

whiteboards

or

blackboards

as

an

aid

to

videoconferencing

or

as

an

easy

way

to

capture

ideas

Scanning

can

produce

images

of

much

greater

resolution

than

single

wide-angle

lens

shots

This

facility

could

be

combined

with

a

video

projection

system

to

create

a

virtual

whiteboard

similar

to

the

DigitalDesk

proposed

by

Wellner

-LSB-

Wellner

model

building

applications

-RSB-

-LRB-

see

-LSB-

Gold

-RSB-

for

other

examples

of

ubiquitous

computing

and

augmented

reality

-RRB-

The

techniques

developed

in

this

paper

enable

any

video

camera

attached

to

a

computer

to

be

used

In

specialized

situations

e.g.

in

classrooms

a

computer-controlled

camera

could

be

used

to

perform

the

scanning

removing

the

need

for

automatic

registration

of

the

images

Another

obvious

application

of

this

technology

is

for

document

scanning

Hand-held

scanners

currently

perform

this

function

quite

well

However

since

they

are

based

on

linear

CCDs

they

are

subject

to

skewing

problems

in

each

strip

of

the

scanned

image

which

must

be

corrected

manually

Using

video

images

as

input

removes

some

of

these

problems

A

final

global

skew

may

still

be

needed

to

make

the

document

square

but

this

can

be

performed

automatically

by

detecting

document

edges

or

internal

horizontal

and

vertical

edges

-LRB-

e.g.

column

borders

-RRB-

model

building

applications

The

model

building

technology

while

more

difficult

to

automate

reliably

has

even

greater

potential

For

example

we

could

construct

a

fax

which

would

scan

objects

at

one

end

using

a

video

camera

and

either

mechanical

or

user-controlled

motion

and

a

graphics

display

at

the

other

end

-LSB-

Carlbom

et

al.

-RSB-

This

fax

could

be

used

to

transmit

models

to

a

remote

site

for

viewing

and

design

revision

or

to

input

rough

CAD

models

for

reverse

engineering

or

clay

mock-up

applications

Of

course

this

technology

could

also

be

combined

with

stereolithography

or

numerically

controlled

milling

to

make

this

a

true

solid

fax

-LSB-

Bresenham

-RSB-

Rapid

model

building

is

also

critical

in

the

computer

animation

and

special

effects

industries

Currently

it

can

take

days

or

weeks

to

build

by

hand

each

computer

model

used

in

a

feature-length

film

The

ability

to

build

such

models

rapidly

from

real

objects

or

physical

mock-ups

would

be

of

great

utility

especially

when

freed

from

the

range

and

resolution

limitations

of

active

rangefinding

systems

The

techniques

we

use

for

computing

geometry

could

also

be

modified

to

automatically

track

the

motions

of

objects

or

actors

in

a

scene

-LRB-

this

is

called

motion

capture

and

is

currently

done

using

specially

placed

markers

-RRB-

End-user

applications

A

more

mass-market

application

is

home

shopping

applied

not

to

single

objects

-LRB-

either

demonstrated

by

a

model

or

available

in

a

catalog

-RRB-

but

to

complete

stores

such

as

your

local

supermarket

This

Applications

has

the

advantage

of

having

a

familiar

look

and

organization

and

enables

the

pre-planning

of

your

next

shopping

trip

The

images

of

the

aisles

-LRB-

with

their

current

contents

-RRB-

can

be

digitized

by

rolling

a

video

camera

through

the

store

More

detailed

geometric

models

of

individual

items

can

be

acquired

either

by

a

model

building

process

or

in

the

future

directly

from

the

manufacturer

The

shopper

can

then

stroll

down

the

aisles

pick

out

individual

items

and

look

at

their

ingredients

and

prices

A

similar

scenario

is

possible

with

other

buildings

Architectural

walkthroughs

have

long

been

a

part

of

computer

graphics

-LSB-

Greenberg

-RSB-

but

only

for

buildings

currently

under

design

Walkthrough

of

existing

building

can

have

a

number

of

applications

For

example

interactive

walkthroughs

of

your

home

built

by

walking

a

video

camera

through

the

rooms

and

mosaicing

the

image

sequences

could

be

used

for

selling

your

house

-LRB-

an

extension

of

existing

still-image

based

systems

-RRB-

or

for

re-modeling

or

renovating

its

design

Walkthroughs

of

historic

building

-LRB-

e.g.

palaces

or

museums

-RRB-

can

be

used

for

educational

and

entertainment

purposes

A

museum

scenario

might

include

the

ability

to

look

at

individual

objects

such

as

sculptures

and

to

bring

up

related

information

in

a

hypertext

system

-LSB-

Miller

et

al.

-RSB-

Walkthroughs

-LRB-

or

fly-throughs

-RRB-

can

also

be

done

in

general

-LRB-

e.g.

outdoor

-RRB-

scenes

An

early

example

of

this

was

the

Movie-Maps

project

-LSB-

Lippman

-RSB-

which

was

based

on

videodisc

technology

Since

this

system

was

based

on

choosing

from

a

number

of

pre-stored

motion

sequences

the

range

of

possible

viewpoints

-LRB-

and

detail

-RRB-

was

limited

The

basic

Movie-Maps

idea

can

be

extended

by

building

true

scenes

from

the

input

motion

sequences

Such

tele-reality

models

have

the

potential

of

being

of

extremely

high

complexity

and

will

require

solving

problems

in

representation

partial

models

-LSB-

Chen

and

Williams

-RSB-

and

switching

between

different

levels

of

resolution

-LSB-

Funkhouser

and

S?equin

-RSB-

The

ultimate

in

tele-reality

systems

is

dynamic

tele-reality

-LRB-

sometimes

called

telepresence

-RRB-

which

composites

video

from

multiple

source

in

real-time

to

create

the

illusion

of

being

in

a

dynamic

-LRB-

and

perhaps

reactive

-RRB-

environment

An

example

of

such

an

application

might

be

to

view

a

version

of

a

concert

with

control

over

the

camera

shots

even

being

able

to

see

the

concert

from

the

musicians

point

of

view

Other

examples

might

be

to

participate

or

consult

in

a

surgery

from

a

remote

location

-LRB-

tele-medicine

-RRB-

or

to

remotely

participate

in

a

virtual

classroom

Building

such

dynamic

models

at

frame

rates

is

beyond

the

processing

power

of

today

's

highperformance

superscalar

workstations

but

it

could

be

achieved

using

a

collection

of

such

machines

Discussion

-LRB-

or

special-purpose

stereo

hardware

-RRB-

In

such

applications

we

need

to

use

multiple

cameras

since

in

moving

scenes

we

can

not

rely

on

the

motion

parallax

from

a

single

camera

to

give

us

reliable

shape

information

Discussion

Image

mosaicing

provides

a

powerful

new

way

of

creating

the

detailed

models

and

scenes

needed

for

tele-reality

applications

By

registering

multiple

images

together

we

can

create

scenes

of

extremely

high

resolution

and

at

the

same

time

recover

partial

or

full

geometric

information

While

we

use

techniques

from

computer

vision

to

perform

the

registration

our

focus

is

different

traditional

vision

techniques

are

designed

for

inspection

recognition

and

robot

control

while

our

techniques

are

designed

to

produce

realistic

models

for

computer

graphics

and

virtual

reality

applications

The

approach

we

use

namely

direct

minimization

of

intensity

differences

between

warped

images

has

a

number

of

advantages

over

more

traditional

vision

techniques

which

are

based

on

tracking

features

from

frame

to

frame

-LSB-

Tomasi

and

Kanade

Azarbayejani

et

al.

Szeliski

and

Kang

-RSB-

Our

techniques

produce

dense

estimates

of

shape

work

in

highly

textured

areas

where

features

may

not

be

reliably

observed

and

make

statistically

optimal

use

of

all

the

information

-LSB-

Szeliski

and

Coughlan

-RSB-

Our

approach

is

similar

to

-LSB-

Bergen

et

al.

-RSB-

who

also

use

intensity

differences

However

we

use

full

projective

models

of

motion

instead

of

instantaneous

quadratic

flow

fields

and

we

also

use

a

projective

formulation

of

structure

from

motion

which

eliminates

the

need

for

calibrated

cameras

It

is

also

very

similar

to

-LSB-

Mann

-RSB-

who

also

use

planar

projective

motion

estimates

Furthermore

-LSB-

Mann

and

Picard

-RSB-

have

also

demonstrated

superresolution

results

i.e.

the

ability

to

obtain

higher

resolution

images

by

simply

jittering

the

camera

rather

than

panning

-LSB-

Irani

and

Peleg

-RSB-

While

our

techniques

have

worked

well

in

the

scenes

in

which

we

have

tried

them

we

must

be

cautious

about

their

general

applicability

The

intensity-based

techniques

we

use

are

sensitive

to

image

intensity

variation

such

as

those

caused

by

video

camera

gain

control

and

vignetting

-LRB-

darkening

of

the

corners

at

wide

lens

apertures

-RRB-

working

with

band-pass

filtered

images

can

remove

most

of

these

problems

All

vision

techniques

are

also

sensitive

to

geometric

distortions

-LRB-

deviations

from

the

pinhole

model

-RRB-

in

the

optics

so

careful

calibration

is

necessary

for

optimal

Conclusions

accuracy

-LRB-

the

results

in

this

paper

were

obtained

with

uncalibrated

cameras

-RRB-

The

depth

extraction

techniques

we

use

rely

on

the

presence

of

texture

in

the

image

In

areas

of

sufficient

texture

it

is

still

possible

for

the

registration/matching

algorithm

to

compute

an

erroneous

depth

estimate

-LRB-

such

gross

errors

are

less

common

in

active

rangefinders

-RRB-

Where

texture

is

absent

interpolation

must

be

used

and

this

can

lead

to

erroneous

-LRB-

hallucinated

-RRB-

depth

estimates

Other

visual

cues

such

as

occluding

contours

or

shading

-LSB-

Horn

-RSB-

can

be

used

to

mitigate

these

problems

in

some

cases

but

a

general-purpose

reliable

vision-based

ranging

system

remains

very

much

an

open

research

problem

The

size

and

structure

of

the

models

produced

for

tele-reality

applications

have

interesting

implications

for

specialized

graphics

hardware

and

software

interfaces

Current

graphics

hardware

stores

the

intensity

images

-LRB-

texture

maps

-RRB-

separately

from

the

geometry

which

is

stored

as

polygons

usually

spanning

many

pixels

A

more

appropriate

architecture

might

be

registered

multi-resolution

color

images

and

depth

maps

View

interpolation

rather

than

rendering

could

then

be

used

to

synthesize

novel

views

-LSB-

Chen

and

Williams

-RSB-

These

kinds

of

architectures

and

algorithms

point

towards

a

tighter

coupling

between

computer

graphics

and

image

processing

-LSB-

Pickering

-RSB-

Conclusions

In

this

paper

we

have

presented

a

hierarchy

of

scene

models

ranging

from

planar

and

panoramic

mosaics

through

full

object

models

and

developed

a

collection

of

associated

scene

recovery

algorithms

The

creation

of

realistic

high-resolution

scenes

from

video

imagery

opens

up

many

new

applications

for

tele-reality

technology

These

include

office

applications

such

as

whiteboard

document

and

bookshelf

scanning

simulated

meeting

spaces

engineering

applications

such

as

reverse

engineering

and

collaborative

design

and

computer

graphics

applications

such

as

model

building

More

importantly

this

technology

enables

mass

market

applications

such

as

home

shopping

education

-LRB-

the

virtual

classroom

-RRB-

and

entertainment

-LRB-

virtual

travel

-RRB-

Ultimately

as

processing

speeds

and

reconstruction

algorithms

improve

further

we

will

see

dynamic

real-time

scene

and

model

recovery

being

used

to

provide

an

even

more

exciting

range

of

telepresence

and

tele-reality

applications

Conclusions

Acknowledgements

Bob

Thomas

introduced

me

to

the

term

tele-reality

which

he

used

to

describe

the

live

concert

telepresence

scenario

James

Coughlan

developed

the

image

registration

code

David

Tonnesen

Demetri

Terzopoulos

Sing

Bing

Kang

and

Richard

Weiss

were

collaborators

on

some

of

the

reconstruction

algorithms

described

in

this

paper

William

Hsu

helped

me

produce

the

results

in

this

paper

Ingrid

Carlbom

and

Gudrun

Klinker

provided

useful

comments

on

the

manuscript

References

-LSB-

Cyberware

Laboratory

Inc

-RSB-

Cyberware

Laboratory

Inc.

Scanner

with

color

digitizer

Monterey

-LSB-

Adam

-RSB-

J.

A.

Adam

Virtual

reality

is

for

real

Spectrum

October

-LSB-

Azarbayejani

et

al.

-RSB-

A.

Azarbayejani

B.

Horowitz

and

A.

Pentland

Recursive

estimation

of

structure

and

motion

using

relative

orientation

constraints

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

New

York

New

York

June

-LSB-

Ballard

and

Brown

-RSB-

D.

H.

Ballard

and

C.

M.

Brown

Computer

Vision

Prentice-Hall

Englewood

Cliffs

New

Jersey

-LSB-

Barnard

and

Fischler

-RSB-

S.

T.

Barnard

and

M.

A.

Fischler

Computational

stereo

Computing

Surveys

-LRB-

-RRB-

December

-LSB-

Beier

and

Neely

-RSB-

T.

Beier

and

S.

Neely

Feature-based

image

metamorphosis

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Bergen

et

al.

-RSB-

J.

R.

Bergen

P.

Anandan

K.

J.

Hanna

and

R.

Hingorani

Hierarchical

modelbased

motion

estimation

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Santa

Margherita

Liguere

Italy

May

-LSB-

Beymer

et

al.

-RSB-

D.

Beymer

A.

Shashua

and

T.

Poggio

Example

Based

Image

Analysis

and

Synthesis

A.

I.

Memo

Massachusetts

Institute

of

Technology

November

-LSB-

Blake

and

Isard

-RSB-

A.

Blake

and

M.

Isard

position

attitude

and

shape

input

using

video

tracking

of

hands

and

lips

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

July

Conclusions

-LSB-

Bresenham

-RSB-

J.

Bresenham

Real

virtuality

Stereolithography

rapid

prototyping

in

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

August

-LSB-

Brown

-RSB-

L.

G.

Brown

A

survey

of

image

registration

techniques

Computing

Surveys

-LRB-

-RRB-

December

-LSB-

Carlbom

et

al.

-RSB-

I.

Carlbom

and

others

Modeling

and

analysis

of

empirical

data

in

collaborative

environments

Communications

of

the

ACM

-LRB-

-RRB-

April

-LSB-

Carlbom

et

al.

-RSB-

I.

Carlbom

D.

Terzopoulos

and

K.

M.

Harris

Reconstructing

and

visualizing

models

of

neuronal

dendrites

In

N.

M.

Patrikalakis

editor

Scientific

Visualization

of

Physical

Phenomena

pages

Springer-Verlag

New

York

-LSB-

Chen

and

Williams

-RSB-

S.

Chen

and

L.

Williams

View

interpolation

for

image

synthesis

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

August

-LSB-

Dhond

and

Aggarwal

-RSB-

U.

R.

Dhond

and

J.

K.

Aggarwal

Structure

from

stereo?a

review

IEEE

Transactions

on

Systems

Man

and

Cybernetics

-LRB-

-RRB-

November/December

-LSB-

Earnshaw

et

al.

-RSB-

R.

A.

Earnshaw

M.

A.

Gigante

and

H.

Jones

editors

Virtual

Reality

Systems

Academic

Press

London

-LSB-

Faugeras

-RSB-

O.

D.

Faugeras

What

can

be

seen

in

three

dimensions

with

an

uncalibrated

stereo

rig

In

Second

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Santa

Margherita

Liguere

Italy

May

-LSB-

Foley

et

al.

-RSB-

J.

D.

Foley

A.

van

Dam

S.

K.

Feiner

and

J.

F.

Hughes

Computer

Graphics

Principles

and

Practice

Addison-Wesley

Reading

MA

edition

-LSB-

Funkhouser

and

S?equin

-RSB-

T.

A.

Funkhouser

and

C.

H.

S?equin

Adaptive

display

algorithm

for

interactive

frame

rates

during

visualization

of

complex

virtual

environments

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

August

-LSB-

Gleicher

and

Witkin

-RSB-

M.

Gleicher

and

A.

Witkin

Through-the-lens

camera

control

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Gold

-RSB-

R.

Gold

Ubiquitous

computing

and

augmented

reality

Computer

Graphics

-LRB-

SIG

GRAPH

-RRB-

August

-LSB-

Greenberg

-RSB-

D.

P.

Greenberg

Computer

graphics

in

architecture

Scientific

American

May

Conclusions

-LSB-

Greene

-RSB-

N.

Greene

Environment

mapping

and

other

applications

of

world

projections

IEEE

Computer

Graphics

and

Applications

-LRB-

-RRB-

November

-LSB-

Greene

and

Heckbert

-RSB-

N.

Greene

and

P.

Heckbert

Creating

raster

Omnimax

images

from

multiple

perspective

views

using

the

elliptical

weighted

average

filter

IEEE

Computer

Graphics

and

Applications

-LRB-

-RRB-

June

-LSB-

Hartley

-RSB-

R.

I.

Hartley

Euclidean

reconstruction

from

uncalibrated

views

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

Seattle

Washington

June

-LSB-

Hoppe

et

al.

-RSB-

W.

Hoppe

and

others

Surface

reconstruction

from

unorganized

points

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Horn

-RSB-

B.

K.

P.

Horn

Robot

Vision

MIT

Press

Cambridge

Massachusetts

-LSB-

Irani

and

Peleg

-RSB-

M.

Irani

and

S.

Peleg

Improving

resolution

by

image

registration

Graphical

Models

and

Image

Processing

-LRB-

-RRB-

May

-LSB-

Jepson

and

Black

-RSB-

A.

Jepson

and

M.

J.

Black

Mixture

models

for

optical

flow

computation

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

New

York

New

York

June

-LSB-

Kuglin

and

Hines

-RSB-

C.

D.

Kuglin

and

D.

C.

Hines

The

phase

correlation

image

alignment

method

In

IEEE

Conference

on

Cybernetics

and

Society

pages

New

York

September

-LSB-

Lippman

-RSB-

A.

Lippman

Movie

maps

An

application

of

the

optical

videodisc

to

computer

graphics

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Mann

-RSB-

S.

Mann

Compositing

multiple

pictures

of

the

same

scene

Generalized

largedisplacement

motion

In

Proceedings

of

the

Annual

IS&T

Conference

The

Society

of

Imaging

Science

and

Technology

Cambridge

Massachusetts

May

-LSB-

Mann

and

Picard

-RSB-

S.

Mann

and

R.

W.

Picard

The

projective

group

model

for

multiframe

resolution

enhancement

In

First

IEEE

International

Conference

on

Image

Processing

-LRB-

ICIP

-RRB-

-LRB-

submitted

-RRB-

-LSB-

Matthies

et

al.

-RSB-

L.

H.

Matthies

R.

Szeliski

and

T.

Kanade

Kalman

filter-based

algorithms

for

estimating

depth

from

image

sequences

International

Journal

of

Computer

Vision

Conclusions

-LSB-

Miller

et

al.

-RSB-

G.

Miller

and

others

The

virtual

museum

Interactive

navigation

of

a

multimedia

database

Journal

of

Visualization

and

Computer

Animation

-LRB-

-RRB-

-LSB-

Moffitt

and

Mikhail

-RSB-

F.

H.

Moffitt

and

E.

M.

Mikhail

Photogrammetry

Harper

Row

New

York

edition

-LSB-

Okutomi

and

Kanade

-RSB-

M.

Okutomi

and

T.

Kanade

A

multiple

baseline

stereo

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

April

-LSB-

Pentland

and

Sclaroff

-RSB-

A.

Pentland

and

S.

Sclaroff

Closed-form

solutions

for

physicallybased

shape

modeling

and

recognition

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

July

-LSB-

Pickering

-RSB-

W.

R.

Pickering

Merging

graphics

and

imaging

applications

and

issues

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

August

-LSB-

Porter

and

Duff

-RSB-

T.

Porter

and

T.

Duff

Compositing

digital

images

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Press

et

al.

-RSB-

W.

H.

Press

B.

P.

Flannery

S.

A.

Teukolsky

and

W.

T.

Vetterling

Numerical

Recipes

in

C

The

Art

of

Scientific

Computing

Cambridge

University

Press

Cambridge

England

second

edition

-LSB-

Quam

-RSB-

L.

H.

Quam

Hierarchical

warp

stereo

In

Image

Understanding

Workshop

pages

Science

Applications

International

Corporation

New

Orleans

Louisiana

December

-LSB-

Rehg

and

Kanade

-RSB-

J.

Rehg

and

T.

Kanade

Visual

tracking

of

high

dof

articulated

structures

an

application

to

human

hand

tracking

In

Third

European

Conference

on

Computer

Vision

-LRB-

ECCV

-RRB-

pages

Springer-Verlag

Stockholm

Sweden

May

-LSB-

Rioux

and

Bird

-RSB-

M.

Rioux

and

T.

Bird

White

laser

synced

scan

IEEE

Computer

Graphics

and

Applications

-LRB-

-RRB-

May

-LSB-

Semple

and

Kneebone

-RSB-

J.

G.

Semple

and

G.

T.

Kneebone

Algebraic

Projective

Geometry

Clarendon

Press

Oxford

England

-LSB-

Szeliski

-RSB-

R.

Szeliski

Shape

from

rotation

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

IEEE

Computer

Society

Press

Maui

Hawaii

June

Conclusions

-LSB-

Szeliski

-RSB-

R.

Szeliski

Rapid

octree

construction

from

image

sequences

CVGIP

Image

Understanding

-LRB-

-RRB-

July

-LSB-

Szeliski

-RSB-

R.

Szeliski

A

least

squares

approach

to

affine

and

projective

structure

and

motion

recovery

In

preparation

-LSB-

Szeliski

and

Coughlan

-RSB-

R.

Szeliski

and

J.

Coughlan

Hierarchical

spline-based

image

registration

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

Seattle

Washington

June

-LSB-

Szeliski

and

Kang

-RSB-

R.

Szeliski

and

S.

B.

Kang

Recovering

shape

and

motion

from

image

streams

using

nonlinear

least

squares

Journal

of

Visual

Communication

and

Image

Representation

-LRB-

-RRB-

March

-LSB-

Szeliski

and

Tonnesen

-RSB-

R.

Szeliski

and

D.

Tonnesen

Surface

modeling

with

oriented

particle

systems

Computer

Graphics

-LRB-

SIGGRAPH

-RRB-

-LRB-

-RRB-

July

-LSB-

Szeliski

and

Weiss

-RSB-

R.

Szeliski

and

R.

Weiss

Robust

shape

recovery

from

occluding

contours

using

a

linear

smoother

In

Image

Understanding

Workshop

pages

Morgan

Kaufmann

Publishers

Washington

D.

C.

April

A

longer

version

is

available

as

CRL

TR

-LSB-

Terzopoulos

and

Witkin

-RSB-

D.

Terzopoulos

and

A.

Witkin

Physically-based

models

with

rigid

and

deformable

components

IEEE

Computer

Graphics

and

Applications

-LRB-

-RRB-

-LSB-

Terzopoulos

et

al.

-RSB-

D.

Terzopoulos

A.

Witkin

and

M.

Kass

Symmetry-seeking

models

and

object

reconstruction

International

Journal

of

Computer

Vision

-LRB-

-RRB-

October

-LSB-

Tomasi

and

Kanade

-RSB-

C.

Tomasi

and

T.

Kanade

Shape

and

motion

from

image

streams

under

orthography

A

factorization

method

International

Journal

of

Computer

Vision

-LRB-

-RRB-

November

-LSB-

Wang

and

Adelson

-RSB-

J.

Y.

A.

Wang

and

E.

H.

Adelson

Layered

representation

for

motion

analysis

In

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

-LRB-

CVPR

-RRB-

pages

New

York

New

York

June

-LSB-

Wellner

-RSB-

P.

Wellner

Interacting

with

paper

on

the

DigitalDesk

Communications

of

the

ACM

-LRB-

-RRB-

July

A

projective

transformations

-LSB-

Weng

et

al.

-RSB-

J.

Weng

T.

S.

Huang

and

N.

Ahuja

Motion

and

Structure

from

Image

Sequences

Springer-Verlag

Berlin

-LSB-

Witkin

et

al.

-RSB-

A.

Witkin

D.

Terzopoulos

and

M.

Kass

Signal

matching

through

scale

space

International

Journal

of

Computer

Vision

-LSB-

Wolberg

-RSB-

G.

Wolberg

Digital

Image

Warping

IEEE

Computer

Society

Press

Los

Alamitos

California

A

projective

transformations

Planar

scene

views

are

related

by

projective

transformations

This

is

true

in

the

most

general

case

i.e.

for

any

to

mapping

u

Mcamp

-LRB-

-RRB-

where

p

is

a

world

coordinate

u

is

the

screen

location

and

Mcam

is

a

camera

matrix

defined

in

-LRB-

-RRB-

Since

Mcam

is

of

rank

we

have

p

M?u

sm

-LRB-

-RRB-

where

M

is

a

left

inverse

of

Mcam

and

m

is

in

the

space

of

Mcam

i.e.

Mcamm

The

equation

of

a

plane

in

world

coordinates

can

be

written

as

ax

by

cz

d

or

n

p

-LRB-

-RRB-

from

which

we

can

conclude

that

nTM?u

sn

m

or

s

-LRB-

nTM?u

-RRB-

-LRB-

n

m

-RRB-

-LRB-

-RRB-

and

hence

p

-LRB-

I

-LRB-

n

m

-RRB-

-RRB-

M?u

Mu

-LRB-

-RRB-

From

any

other

viewpoint

we

have

Mu

-LRB-

-RRB-

which

is

a

planar

projective

transformation

A

projective

transformations

For

the

remainder

of

this

appendix

it

is

convenient

to

assume

that

the

world

coordinate

system

coincides

with

the

first

camera

position

i.e.

E

I

and

Mcam

V

h

V

i

and

therefore

M

V

and

m

-LRB-

-RRB-

T

in

-LRB-

-RRB-

An

image

point

u

then

corresponds

to

a

world

coordinate

p

p

V

w

-LRB-

-RRB-

where

w

is

the

unknown

fourth

coordinate

of

p

-LRB-

called

projective

depth

-RRB-

which

controls

how

far

the

point

is

from

the

origin

For

panoramic

image

mosaicing

we

rotate

the

point

p

around

the

camera

optical

center

to

obtain

R

T

R

V

w

-LRB-

-RRB-

with

a

corresponding

screen

coordinate

V

-LRB-

-RRB-

Thus

the

mapping

between

the

two

screen

coordinate

systems

can

be

described

by

a

projective

transformation

For

piecewise-planar

scenes

the

point

p

will

lie

on

some

plane

k

whose

equation

we

can

write

as

nk

p

-LRB-

nk

-RRB-

p

Then

-LRB-

-RRB-

reduces

to

pk

I

nTk

V

-LRB-

-RRB-

as

the

position

on

plane

k

corresponding

to

the

screen

position

u

In

a

second

camera

position

the

world

coordinates

have

undergone

a

Euclidean

transformation

Epk

with

rotation

R

and

translation

t

The

new

screen

coordinates

are

therefore

-LRB-

R

tnTk

-RRB-

V

Mku

-LRB-

-RRB-

Thus

we

see

that

projective

motion

matrices

for

the

various

planar

pieces

are

interrelated

and

are

in

fact

determined

by

the

plane

equations

-LRB-

nk

-RRB-

the

rigid

motion

-LRB-

R

and

t

-RRB-

and

the

viewing

matrices

-LRB-

V

and

-RRB-

For

methods

for

recovering

these

unknown

parameters

from

the

Mk

see

-LSB-

Faugeras

Hartley

-RSB-

exclude

planes

passing

through

the

first

camera

center

in

this

formulation

A

projective

transformations

For

general

depth

recovery

each

image

point

u

corresponds

to

a

point

p

with

an

unknown

projective

depth

w

as

given

in

-LRB-

-RRB-

In

some

other

frame

whose

relative

orientation

to

the

first

frame

is

denoted

by

E

we

have

V

w

w?t

-LRB-

-RRB-

Thus

the

motion

between

the

two

frames

can

be

described

by

our

familiar

planar

projective

motion

plus

an

amount

of

motion

proportional

to

the

projective

depth

along

the

direction

t

-LRB-

which

is

called

the

epipole

-RRB-

-LSB-

Faugeras

-RSB-

Note

that

the

values

of

t

and

w

are

not

unique

there

is

a

scale

ambiguity

between

w

and

t

and

multiples

of

t

can

be

added

to

with

the

appropriate

adjustments

to

w.

To

remove

these

ambiguities

we

must

enforce

the

rigidity

constraints

V

-LSB-

Szeliski

-RSB-

image compositing
image registration
motion estimation
panoramic scenes
structure from motion
virtual reality

3-d scene recovery
image mosaics
planar scenes
tele-reality