Free

Form

Deformation

with

Scattered

Data

Interpolation

Methods

Detlef

Ruprecht

and

Heinrich

M?uller

Abstract

Free

form

deformation

has

applications

in

computer

animation

in

data

acquisition

and

in

computer

aided

geometric

design

One

approach

is

to

deform

the

space

in

order

to

induce

a

deformation

of

an

embedded

object

A

new

method

of

this

class

is

proposed

which

is

based

on

scattered

data

interpolation

The

usefulness

of

two

selected

classes

of

scattered

data

interpolation

methods

in

this

context

is

analyzed

with

emphasis

on

the

two-dimensional

case

It

turns

out

that

suitable

scattered

data

methods

do

exist

and

that

the

new

approach

should

compete

well

with

more

complex

computationally

expensive

physically

based

models

Published

in

G.

Farin

H.

Hagen

and

H.

Noltemeier

-LRB-

eds

-RRB-

Geometric

Modelling

-LRB-

Computing

Suppl

-RRB-

pages

-LCB-

Springer

Verlag

Wien

Free

Form

Deformation

Suppose

we

are

given

an

image

of

two-dimensional

geometric

objects

The

task

is

to

distort

the

image

irregularly

In

the

physical

world

this

goal

can

be

reached

by

drawing

the

image

on

an

elastic

skin

The

elastic

skin

is

deformed

by

moving

some

of

its

points

into

a

new

position

This

causes

a

deformation

of

the

image

too

In

a

computer

the

elastic

skin

the

image

drawn

on

it

and

the

input

of

the

points

to

be

moved

can

be

easily

modeled

by

an

interactive

program

The

user

interface

of

such

a

program

is

shown

in

fig.

Besides

some

menus

it

consists

of

two

neighboring

rectangular

regions

The

left

region

shows

the

image

before

transformation

in

the

example

a

digitized

cross

section

of

a

blossom

If

a

second

image

is

known

which

is

to

be

used

as

a

target

for

the

deformation

it

can

be

displayed

in

the

right

region

The

example

in

fig.

shows

another

cross

section

of

the

blossom

at

a

later

stage

of

evolution

By

consecutively

choosing

a

point

in

the

left

and

a

point

in

the

right

region

pairs

of

an

original

point

and

its

image

under

the

deformation

to

be

carried

out

are

defined

This

displacement

of

control

points

is

displayed

by

dashed

lines

The

calculation

of

the

deformation

is

initiated

Institut

f?ur

Informatik

Universit?at

Freiburg

Rheinstr

-LCB-

Freiburg

i.Br

E-mail

-LSB-

ruprechtjmueller]@informatik.uni-freiburg.de

by

selecting

a

corresponding

menu

item

The

resulting

deformed

picture

is

then

displayed

in

the

right

region

overlaid

over

the

second

image

if

one

is

present

as

shown

in

fig.

The

images

deformed

in

this

paper

consist

of

polygonal

chains

The

deformed

picture

is

obtained

by

calculating

the

images

under

deformation

of

points

on

the

polygonal

chains

and

then

connecting

them

by

line

segments

in

the

same

way

as

in

the

original

picture

It

is

not

too

difficult

to

deform

raster

images

too

A

possibility

is

to

use

a

look

-LCB-

up

table

with

positions

of

pixels

of

the

initial

picture

as

it

was

done

in

-LSB-

Wolberg

Boult

-RSB-

An

immediate

application

of

image

deformation

is

in

two-dimensional

computer

animation

A

given

drawing

can

be

animated

by

continuous

transformation

Usually

the

animation

will

not

be

designed

by

extrapolating

an

initial

image

but

by

interpolating

between

two

given

keyframes

In

applying

the

technique

of

deformation

interpolation

can

be

performed

approximately

by

deforming

the

first

keyframe

so

that

it

coincides

with

the

second

as

well

as

possible

as

it

was

done

in

fig

The

intermediate

frames

required

for

animation

may

then

be

obtained

by

linear

interpolation

between

the

first

keyframe

and

its

deformation

Let

p

be

a

point

in

the

first

keyframe

f

the

function

describing

the

deformation

Then

the

location

of

p

at

time

t

t

is

-LRB-

t

-RRB-

p

t

f

-LRB-

p

-RRB-

The

initial

motivation

for

the

work

reported

on

here

was

a

problem

of

geometric

design

A

widely

used

technique

of

solid

modeling

is

the

reconstruction

of

the

surface

of

a

solid

from

cross

sections

Reconstruction

from

cross

sections

finds

its

application

in

particular

in

the

acquisition

of

existing

solid

structures

by

techniques

of

tomography

Tomography

is

used

in

medicine

but

also

in

microscopy

or

geology

One

of

the

most

powerful

approaches

of

reconstruction

is

that

of

-LSB-

Boissonnat

-RSB-

This

approach

is

based

on

nearest-neighbor

heuristics

and

is

thus

able

to

interconnect

even

multiple

contours

including

branches

in

a

usually

quite

likely

way

However

troubles

arise

if

the

cross

sections

are

highly

dissimilar

The

blossoms

of

fig.

are

an

example

if

the

time

axis

is

taken

as

the

third

spatial

axis

Free

form

deformation

allows

a

distortion

of

the

cross

sections

with

only

a

few

assignments

so

that

Boissonnat

's

algorithm

may

find

a

reconstruction

between

the

deformed

cross

sections

The

actual

reconstruction

is

obtained

by

back-transformation

which

can

simply

be

achieved

by

replacing

the

deformed

coordinates

by

the

original

ones

In

this

manner

a

good

reconstruction

from

the

original

cross

sections

can

often

be

achieved

Another

problem

with

techniques

of

tomography

is

that

their

physical

properties

often

cause

distortions

of

the

cross

sections

during

catching

-LSB-

Buche

Camillerapp

-RSB-

For

their

correction

free

form

deformation

can

be

used

as

well

A

last

example

of

an

application

is

the

design

of

free

form

solids

in

computer

aided

geometric

design

by

deforming

initially

simple

solids

into

complex

shapes

-LSB-

Sederberg

Parry

-RSB-

The

crucial

question

not

addressed

yet

is

the

calculation

of

the

deformation

One

possibility

is

the

correct

simulation

of

a

physical

model

for

example

an

elastic

skin

of

rubber

Such

models

have

found

application

in

e.g.

face

modeling

-LSB-

Guenther

-RSB-

These

approaches

seem

to

be

expensive

In

the

following

a

more

simple

method

is

proposed

The

problem

of

deformation

is

interpreted

as

a

problem

of

interpolation

It

turns

out

that

deformation

is

closely

related

to

the

problem

of

scattered

data

interpolation

This

opens

the

possibility

to

refer

to

well-known

techniques

in

this

area

-LSB-

Franke

Alfeld

-RSB-

From

the

high

number

of

scattered

data

methods

we

have

chosen

two

types

for

our

investigations

the

classic

Shepard

approach

with

some

modifications

and

interpolation

with

radial

basis

functions

in

particular

Hardy

's

multiquadrics

The

experimental

investigations

show

that

with

this

approach

good

results

can

be

achieved

as

long

as

the

Figure

Deformation

of

a

cross

section

of

a

blossom

The

upper

picture

shows

two

given

states

of

evolution

of

the

blossom

and

an

assignment

for

deformation

In

the

right

region

of

the

picture

below

the

original

second

state

and

the

deformation

of

the

first

state

calculated

from

the

assignment

are

displayed

simultaneously

dissimilarity

between

the

original

image

and

that

endeavored

by

deformation

is

not

too

high

A

first

impression

can

be

got

from

Fig

in

which

interpolation

with

multiquadrics

is

used

Despite

the

low

number

of

assignments

the

similarity

of

both

pictures

is

convincing

Bigger

deformations

can

be

performed

by

decomposition

into

a

sequence

of

smaller

deformations

The

remainder

of

this

contribution

is

organized

as

follows

In

sect.

the

relation

between

the

problem

of

deformation

and

the

problem

of

scattered

data

interpolation

is

established

The

scattered

data

methods

and

their

behavior

for

the

problem

of

deformation

are

the

subject

of

sect.

and

Piecewise

deformation

is

discussed

in

sect

A

comparison

between

the

two

approaches

of

scattered

data

interpolation

as

well

as

with

other

methods

of

deformation

known

in

literature

is

performed

in

sect.

Deformation

and

Scattered

Data

Interpolation

The

problem

of

deformation

can

be

formulated

as

follows

Deformation

Problem

Input

n

pairs

-LRB-

pi

qi

-RRB-

of

control

points

pi

qi

IRd

i

n.

Output

An

at

least

continuous

function

f

IRd

IRd

with

f

-LRB-

pi

-RRB-

qi

For

the

image

deformation

problem

we

have

d

Besides

continuity

other

properties

of

the

function

f

are

desirable

which

concern

the

visual

pleasantness

and

which

are

hard

to

describe

in

an

exact

manner

These

properties

will

become

recognizable

in

the

following

discussion

of

concrete

propositions

The

problem

of

scattered

data

interpolation

is

to

find

a

real

valued

multivariate

function

interpolating

a

finite

set

of

irregularly

located

data

points

Scattered

Data

Interpolation

Problem

Input

n

data

points

-LRB-

xi

yi

-RRB-

xi

IRd

yi

IR

i

n.

Output

An

at

least

continuous

function

f

IRd

IR

interpolating

the

given

data

points

i.e.

f

-LRB-

xi

-RRB-

yi

i

n.

Scattered

data

methods

can

now

be

simply

used

by

interpolating

each

component

fj

IRd

IR

j

d

of

f

separately

The

data

points

to

be

used

are

-LRB-

pi

qi

j

-RRB-

qi

j

the

j-th

component

of

qi

Shepard

Interpolation

The

Shepard

approach

of

scattered

data

interpolation

-LSB-

Shepard

-RSB-

uses

a

weighted

average

of

the

data

values

at

the

data

points

with

weights

dependent

on

the

distance

of

the

observed

point

from

the

given

data

points

f

-LRB-

x

-RRB-

nX

i

wi

-LRB-

x

-RRB-

yi

-LRB-

-RRB-

wi

IRn

IR

is

the

weight

function

and

yi

IR

the

data

value

at

the

data

point

xi

IRd

The

weight

functions

satisfy

the

conditions

wi

-LRB-

x

-RRB-

i

n

nX

i

wi

-LRB-

x

-RRB-

and

-LRB-

-RRB-

wi

-LRB-

xj

-RRB-

ffiij

i.e.

wi

-LRB-

xi

-RRB-

and

wi

-LRB-

xj

-RRB-

j

i

i

j

n

-LRB-

-RRB-

These

conditions

guarantee

the

property

of

interpolation

In

-LSB-

Shepard

-RSB-

the

following

weight

function

was

proposed

wi

-LRB-

x

-RRB-

oei

-LRB-

x

-RRB-

Pnj

oej

-LRB-

x

-RRB-

with

-LRB-

-RRB-

oei

-LRB-

x

-RRB-

-LRB-

di

-LRB-

x

-RRB-

-RRB-

di

-LRB-

x

-RRB-

the

distance

between

x

and

xi

-LRB-

-RRB-

The

smoothness

is

determined

by

the

exponent

assures

the

continuity

of

the

first

derivative

its

value

disappears

at

the

data

points

The

Shepard

interpolation

in

this

form

shows

an

unfavorable

behavior

when

applied

to

the

problem

of

deformation

Using

the

weighted

average

of

the

given

control

points

causes

all

other

points

to

be

shifted

towards

the

geometric

center

of

the

displaced

control

points

Fig.

shows

the

example

of

a

square

grid

whose

four

corners

are

fixed

No

further

assignments

were

given

apart

from

these

four

pairs

of

identical

points

The

original

grid

is

shown

with

dashed

lines

the

deformation

with

solid

lines

The

created

image

is

considerably

distorted

which

means

that

a

simple

Shepard

approach

will

usually

not

be

well

suited

for

the

application

to

deformation

a

b

Figure

Deformation

of

a

square

grid

with

fixed

corners

-LRB-

a

-RRB-

simple

Shepard

approach

-LRB-

b

-RRB-

radial

basis

functions

with

Hardy

's

multiquadrics

without

polynomial

Local

interpolants

Several

authors

among

them

-LSB-

Gordon

Wixom

Franke

Lancaster

Salkauskas

Farwig

-RSB-

have

proposed

modified

approaches

which

replace

the

value

yi

in

Shepard

's

formula

by

a

local

interpolant

fi

-LRB-

x

-RRB-

IRn

IR

f

-LRB-

x

-RRB-

nX

i

wi

-LRB-

x

-RRB-

fi

-LRB-

x

-RRB-

fi

-LRB-

xi

-RRB-

yi

i

n

-LRB-

-RRB-

-LSB-

Franke

-RSB-

proposed

to

compute

the

local

interpolant

fi

-LRB-

x

-RRB-

by

minimizing

the

squared

error

of

the

mapping

of

all

other

control

points

xj

with

fi

weighted

with

a

coefficient

wij

to

attenuate

the

influence

of

distant

points

The

corresponding

error

function

Ei

-LRB-

f

-RRB-

is

Ei

-LRB-

f

-RRB-

nX

j

j

i

wij

kf

-LRB-

xj

-RRB-

-LRB-

-RRB-

The

optimum

local

interpolant

is

obtained

by

minimizing

Ei

-LRB-

f

-RRB-

The

coefficients

wij

should

depend

on

the

distance

between

xi

and

xj

in

a

manner

similar

to

wi

-LRB-

x

-RRB-

but

it

is

not

possible

to

use

wi

because

of

-LRB-

-RRB-

A

simple

possibility

is

to

use

wij

oei

-LRB-

xj

-RRB-

This

approach

has

been

used

in

our

examples

on

local

interpolants

The

application

of

-LRB-

-RRB-

to

the

problem

of

deformation

leads

to

f

-LRB-

p

-RRB-

nX

i

wi

-LRB-

p

-RRB-

fi

-LRB-

p

-RRB-

fi

-LRB-

pi

-RRB-

qi

i

n

-LRB-

-RRB-

fi

IRd

IRd

are

the

local

interpolants

They

can

be

rewritten

as

fi

-LRB-

p

-RRB-

qi

Di

-LRB-

p

pi

-RRB-

with

Di

IRd

IRd

Di

-LRB-

-RRB-

-LRB-

-RRB-

A

simple

possibility

is

to

choose

the

Di

as

linear

transformations

Then

Di

can

be

represented

by

a

d?d

-LCB-

matrix

-LRB-

di

kl

-RRB-

k

l

d

In

the

two-dimensional

case

the

error

function

corresponding

to

a

linear

transformation

D

is

Ei

-LRB-

D

-RRB-

nX

j

j

i

wij

kqi

-LRB-

pj

pi

-RRB-

-LRB-

-RRB-

NX

j

j

i

wij

-LSB-

-LRB-

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

qi

qj

-RRB-

-LRB-

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

qi

qj

-RRB-

-RSB-

-LRB-

-RRB-

The

minimum

of

the

error

function

is

obtained

with

the

partial

derivatives

with

respect

to

the

dkl

k

l

For

@Ei

-LRB-

D

-RRB-

for

example

we

have

@Ei

-LRB-

D

-RRB-

nX

j

j

i

wij

-LSB-

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-LRB-

qi

qj

-RRB-

-LRB-

pj

pi

-RRB-

-RSB-

-LRB-

-RRB-

Hence

the

derivatives

are

linear

functions

of

the

di

kl

Their

second

derivatives

show

that

the

zeros

of

the

first

derivative

are

indeed

a

minimum

Thus

a

linear

system

of

four

equations

with

four

unknowns

is

obtained

The

system

has

a

unique

solution

if

its

determinant

is

not

equal

to

zero

The

solutions

are

deti

-LRB-

nX

j

j

i

wij

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

nX

j

j

i

wij

-LRB-

pj

pi

-RRB-

-RRB-

h

nX

j

j

i

wij

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-LRB-

-RRB-

di

deti

h

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

i

-LRB-

-RRB-

di

deti

h

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

i

-LRB-

-RRB-

di

deti

h

-LRB-

Xwi

-LRB-

pj

-RRB-

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

i

-LRB-

-RRB-

di

deti

h

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

Xwij

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

X

wij

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

i

-LRB-

-RRB-

If

deti

a

simpler

local

interpolant

with

di

di

and

di

di

is

used

which

allows

uniform

scaling

and

rotation

but

no

shearing

The

di

kl

are

again

calculated

by

minimizing

Ei

-LRB-

D

-RRB-

with

the

result

di

Xwij

-LRB-

-LRB-

pj

pi

-RRB-

-LRB-

pj

pi

-RRB-

-RRB-

-LRB-

-RRB-

di

di

di

Xwij

-LRB-

-LRB-

pj

pi

-RRB-

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-LRB-

qj

qi

-RRB-

-RRB-

-LRB-

-RRB-

di

di

di

Xwij

-LRB-

-LRB-

pj

pi

-RRB-

-LRB-

qj

qi

-RRB-

-LRB-

pj

pi

-RRB-

-LRB-

qj

qi

-RRB-

-RRB-

-LRB-

-RRB-

If

even

this

can

not

be

computed

Di

is

set

to

an

identical

transformation

With

a

linear

local

interpolant

the

interpolation

has

linear

precision

i.e.

if

the

requested

displacements

can

be

satisfied

by

a

linear

transformation

this

is

what

will

be

obtained

For

example

a

deformation

as

in

fig.

where

all

control

points

retain

their

initial

position

will

exactly

reproduce

the

initial

image

The

examples

in

figs.

-LCB-

demonstrate

the

effect

of

the

deformation

To

make

comparison

easier

various

examples

of

the

deformation

of

a

square

grid

are

shown

Examples

of

modeling

applications

are

also

given

The

same

set

of

displacement

assignments

for

the

control

points

is

used

throughout

each

figure

For

clarity

the

original

image

and

the

displacements

are

only

shown

in

the

first

example

in

each

figure

Fig.

shows

a

square

grid

with

fixed

corners

within

which

a

square

has

been

turned

clockwise

by

The

original

grid

is

displayed

with

dashed

lines

and

the

displacement

assignments

with

solid

lines

Fig.

shows

the

result

of

Shepard

deformation

with

The

waviness

of

the

edges

of

the

inner

square

comes

from

the

influence

of

the

fixed

corner

points

of

the

grid

Increasing

reduces

this

waviness

as

the

influence

of

the

corner

points

is

reduced

Similarly

the

grid

edges

are

pulled

along

by

the

rotation

of

the

inner

square

As

some

points

along

the

grid

edges

are

actually

closer

to

the

inner

square

than

to

the

corner

points

increasing

does

not

help

here

Fig.

shows

the

deformation

with

Fig.

and

show

the

same

square

grid

deformed

into

a

side-view

of

a

head

Fig.

uses

fig.

uses

The

method

reacts

to

the

input

of

displacements

in

an

intuitive

and

straightforward

manner

Note

the

differences

caused

by

the

different

range

of

influence

of

the

control

points

Fig.

shows

a

straight

bar

which

is

deformed

into

a

telephone

handset

with

The

produced

image

is

not

very

satisfying

The

observed

unevenness

near

the

data

points

is

typical

for

the

Shepard

approach

Therefore

it

is

better

suited

for

applications

where

some

jaggedness

is

desirable

-LCB-

as

in

the

head

shown

in

fig.

-LCB-

than

for

applications

needing

smooth

deformations

as

is

common

in

Computer

Aided

Geometric

Design

Higher

order

polynomials

can

be

used

as

well

-LSB-

Franke

Nielson

-RSB-

recommend

the

use

of

second

order

polynomials

This

does

not

appear

to

be

necessary

in

the

case

of

deformation

Also

higher

order

polynomials

tend

to

introduce

oscillations

which

may

be

acceptable

in

data

interpolation

but

undesirable

for

deformation

a

b

c

d

e

f

Figure

Rotating

an

embedded

-LCB-

square

by

within

a

square

grid

with

fixed

corners

-LRB-

a

-RRB-

Original

grid

with

displacements

-LRB-

b

-RRB-

Shepard

with

local

interpolation

-LRB-

-RRB-

-LRB-

c

-RRB-

Shepard

with

local

interpolation

-LRB-

-RRB-

-LRB-

d

-RRB-

Shepard

with

weight

functions

with

limited

influence

-LRB-

r

-RRB-

-LRB-

e

-RRB-

Hardy

's

multiquadrics

-LRB-

-RRB-

-LRB-

f

-RRB-

Hardy

's

multiquadrics

with

limited

influence

-LRB-

r

-RRB-

Relaxing

the

Interpolation

Condition

The

smoothness

of

the

deformation

can

be

improved

by

adding

a

damping

term

c

to

the

weight

function

-LRB-

-RRB-

i.e.

oei

-LRB-

x

-RRB-

-LRB-

di

-LRB-

x

-RRB-

c

-RRB-

-LRB-

-RRB-

With

this

weight

function

exact

interpolation

is

not

provided

but

with

c

sufficiently

small

the

approximation

follows

the

intention

specified

in

the

control

point

displacements

quite

well

Fig.

demonstrates

this

for

a

value

of

c

This

image

is

noticeable

smoother

than

the

images

in

fig.

and

b

An

individual

ci

could

be

specified

for

each

data

point

so

that

the

user

can

specify

the

desired

precision

for

each

control

point

This

however

would

make

the

input

of

displacements

more

cumbersome

With

c

strict

adherence

to

the

local

interpolant

can

be

enforced

for

di

-LRB-

x

-RRB-

c

This

might

be

useful

if

special

conditions

e.g.

conservation

of

area

or

angles

have

to

be

met

in

the

vicinity

of

the

control

points

a

b

c

d

e

f

Figure

Deformation

of

a

square

grid

into

a

head

-LRB-

a

-RRB-

Shepard

with

local

interpolation

-LRB-

-RRB-

-LRB-

b

-RRB-

Shepard

with

local

interpolation

-LRB-

-RRB-

-LRB-

c

-RRB-

Shepard

with

damping

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Shepard

with

limited

influence

-LRB-

r

-RRB-

-LRB-

e

-RRB-

Hardy

's

Multiquadrics

-LRB-

-RRB-

-LRB-

f

-RRB-

Hardy

's

Multiquadrics

with

limited

influence

-LRB-

-RRB-

a

b

c

d

Figure

Deformation

of

a

straight

bar

to

a

telephone

handset

-LRB-

a

-RRB-

Shepard

with

local

interpolation

-LRB-

b

-RRB-

Shepard

with

limited

influence

-LRB-

r

-RRB-

-LRB-

c

-RRB-

Hardy

's

Multiquadrics

-LRB-

d

-RRB-

Radial

basis

function

d

-LRB-

x

-RRB-

-LRB-

x

-RRB-

Locally

Bounded

Weight

Functions

If

only

part

of

an

image

is

to

be

deformed

it

is

desirable

to

limit

the

sphere

of

influence

of

the

control

points

so

that

the

deformation

of

a

point

is

only

influenced

by

control

points

at

a

distance

di

-LRB-

x

-RRB-

ri

-LSB-

Franke

Nielson

-RSB-

proposed

the

following

weight

function

oei

-LRB-

x

-RRB-

n

ri

di

-LRB-

x

-RRB-

ridi

-LRB-

x

-RRB-

for

di

-LRB-

x

-RRB-

ri

di

-LRB-

x

-RRB-

ri

-LRB-

-RRB-

In

order

to

ensure

that

there

is

at

least

one

non-zero

weight

for

all

points

in

the

source

image

a

global

function

-LRB-

x

-RRB-

and

a

corresponding

weight

-LRB-

x

-RRB-

should

be

added

to

-LRB-

-RRB-

and

-LRB-

-RRB-

so

that

f

-LRB-

p

-RRB-

nX

i

wi

-LRB-

p

-RRB-

fi

-LRB-

p

-RRB-

with

fi

-LRB-

pi

-RRB-

qi

i

n

and

-LRB-

-RRB-

wi

-LRB-

x

-RRB-

oei

-LRB-

x

-RRB-

Pnj

oej

-LRB-

x

-RRB-

-LRB-

-RRB-

In

the

case

of

localized

deformations

identical

transformations

-LRB-

x

-RRB-

x

with

a

constant

user

-LCB-

selectable

perform

quite

well

Fig.

shows

the

result

for

the

rotated

square

with

r

and

Areas

further

than

grid

units

away

from

the

rotated

square

are

completely

unaffected

by

the

deformation

Fig.

shows

the

grid

deformed

into

a

head

with

r

and

The

results

obtained

here

are

unsatisfying

which

is

probably

due

to

the

sometimes

sudden

change

of

spheres

of

influence

from

one

control

point

to

another

Fig.

shows

the

telephone

handset

produced

from

a

straight

bar

with

r

and

Here

the

limitation

of

the

influence

of

the

control

point

has

quite

a

positive

effect

on

the

deformation

although

the

result

is

still

not

perfect

-LSB-

Franke

Nielson

-RSB-

alternatively

also

propose

weight

functions

bounded

by

a

triangulation

of

the

control

points

This

shows

that

there

is

an

almost

unlimited

number

of

variations

on

weight

functions

to

suit

special

purposes

Radial

Basis

Functions

A

further

approach

to

scattered

data

interpolation

is

to

construct

the

interpolant

as

a

linear

combination

of

basis

functions

and

then

to

determine

the

coefficients

of

the

basis

functions

f

-LRB-

x

-RRB-

nX

i

i

R

-LRB-

di

-LRB-

x

-RRB-

-RRB-

pm

-LRB-

x

-RRB-

-LRB-

-RRB-

The

values

of

the

basis

function

R

usually

depend

only

on

the

distance

from

the

data

point

and

are

thus

called

radial

pm

-LRB-

x

-RRB-

is

a

polynomial

of

degree

m

It

assures

a

certain

degree

m

of

polynomial

precision

The

coefficients

i

are

calculated

by

putting

the

data

points

into

-LRB-

-RRB-

and

solving

the

resulting

linear

system

of

equations

Well-known

radial

basis

functions

are

Hardy

's

multiquadrics

-LSB-

Hardy

-RSB-

R

-LRB-

d

-RRB-

-LRB-

-RRB-

with

r

and

-LRB-

-RRB-

Hardy

suggests

to

set

pm

identical

to

This

implies

a

polynomial

precision

of

For

the

exponent

Hardy

proposes

has

also

been

used

but

is

generally

less

stable

r

can

be

chosen

arbitrarily

It

determines

the

smoothness

of

interpolation

at

the

given

data

points

-LSB-

Eck

-RSB-

proposes

to

use

individual

values

ri

for

each

data

point

pi

computed

from

the

distance

to

the

nearest

neighbor

i.e.

ri

j

di

-LRB-

xj

-RRB-

-LRB-

-RRB-

This

causes

the

deformation

to

be

softer

when

data

points

are

widely

spaced

and

stronger

when

they

are

closer

together

This

approach

has

been

used

in

our

examples

unless

otherwise

noted

The

application

of

radial

basis

functions

to

the

problem

of

deformation

gives

f

-LRB-

p

-RRB-

nX

i

i

R

-LRB-

di

-LRB-

p

-RRB-

-RRB-

pm

-LRB-

p

-RRB-

-LRB-

-RRB-

Now

R

IR

IRd

is

a

vector

valued

function

of

the

distance

and

pm

IRd

IRd

is

a

function

with

polynomial

components

of

degree

m

It

turns

out

that

Hardy

's

multiquadrics

approach

without

a

polynomial

term

pm

does

not

yield

satisfactory

results

as

it

produces

distortions

even

with

the

simple

transformation

of

a

square

grid

with

fixed

corners

as

shown

in

fig.

This

behavior

is

caused

by

the

non-linearity

of

the

basis

functions

Introducing

a

linear

pm

i.e.

m

as

a

global

approximation

with

coefficients

computed

by

the

least

squares

method

overcomes

the

problem

With

this

the

example

yields

an

identical

transformation

as

desired

Fig.

shows

a

rotated

square

inside

a

grid

with

The

deformation

is

much

smoother

than

that

achieved

with

Shepard

methods

Fig.

shows

a

head

created

from

a

square

grid

with

multiquadrics

The

smoothness

of

the

results

obtained

by

this

method

is

obviously

not

very

well

suited

for

this

sort

of

application

Fig.

however

shows

that

the

telephone

handset

created

from

a

straight

bar

looks

very

good

with

multiquadrics

With

a

pm

as

introduced

above

it

is

possible

to

use

locally

bounded

basis

functions

which

disappear

for

di

-LRB-

x

-RRB-

ri

This

way

radial

basis

functions

can

be

used

for

local

deformations

For

these

applications

the

identical

transformation

pm

-LRB-

p

-RRB-

p

is

an

adequate

polynomial

approximation

The

effect

of

multiquadrics

with

a

negative

exponent

is

quite

localized

even

though

they

are

not

strictly

bounded

basis

functions

Fig.

shows

the

rotated

square

for

r

and

The

locality

of

the

deformation

is

well

demonstrated

Fig.

shows

multiquadrics

with

applied

to

the

deformation

of

a

grid

into

a

head

The

result

is

better

than

with

although

for

this

type

of

artistic

modeling

the

results

of

Shepard

's

methods

are

still

superior

Other

radial

basis

functions

have

been

proposed

-LSB-

Duchon

-RSB-

which

allow

a

physical

interpretation

as

a

minimal

thin

plate

functional

R

-LRB-

d

-RRB-

and

-LRB-

-RRB-

R

-LRB-

d

-RRB-

log

d

-LRB-

-RRB-

In

our

experiments

deformations

obtained

from

these

basis

functions

were

consistently

less

satisfactory

than

those

from

multiquadrics

Fig.

shows

an

example

with

the

telephone

handset

for

the

radial

basis

function

-LRB-

-RRB-

Iterated

Deformation

The

experiments

show

that

in

the

case

of

small

deformations

good

results

can

be

achieved

with

many

approaches

For

bigger

deformations

however

undesired

foldover

can

occur

in

the

deformed

picture

-LRB-

fig

-RRB-

These

are

caused

by

the

fact

that

contrary

to

the

scattered

data

problem

the

image

space

is

mapped

onto

itself

The

influence

of

the

control

points

on

a

transformed

point

should

therefore

be

dependent

not

only

on

the

initial

relative

position

but

also

on

the

trajectory

along

which

a

control

point

is

moved

to

its

new

location

Moving

the

control

points

can

be

simulated

by

decomposing

the

deformation

into

several

smaller

deformations

which

are

executed

sequentially

When

choosing

a

linear

trajectory

the

deformation

with

pairs

of

control

points

-LRB-

pi

qi

-RRB-

is

linearly

subdivided

into

m

deformations

-LRB-

pi

k

qi

k

-RRB-

qi

k

pi

k

k

m

with

pi

k

xi

k

m

-LRB-

qi

pi

-RRB-

k

m

-LRB-

-RRB-

Fig.

and

b

show

a

square

grid

with

an

embedded

square

rotated

by

with

a

non-iterated

transformation

and

with

four

iterations

respectively

For

the

transformations

multiquadrics

are

used

The

foldover

in

the

non-iterated

transformation

is

clearly

visible

Note

that

iteration

is

not

used

in

any

of

the

other

examples

The

deformation

in

these

examples

is

not

strong

enough

to

make

this

necessary

a

b

Figure

rotation

of

a

square

embedded

in

a

square

grid

by

-LRB-

a

-RRB-

without

iteration

-LRB-

b

-RRB-

with

four

steps

of

iteration

Splitting

a

deformation

into

a

series

of

iterated

transformations

is

also

helpful

when

it

is

necessary

to

reverse

the

deformation

This

can

be

done

by

deforming

the

result

with

the

reversed

control

points

-LRB-

qi

pi

-RRB-

To

reproduce

the

original

image

well

the

weights

wi

or

the

i

for

radial

basis

functions

should

not

change

much

between

successive

iteration

steps

i.e.

wi

k

-LRB-

pj

k

-RRB-

ss

wi

k

-LRB-

pj

k

-RRB-

This

can

always

be

enforced

by

using

a

sufficiently

large

m.

For

keyframe

animation

it

appears

to

be

useful

to

use

a

non-linear

division

of

the

deformation

to

avoid

discontinuities

of

the

movement

at

the

keyframes

Discussion

An

approach

for

the

deformation

of

geometric

objects

has

been

presented

The

deformation

is

controlled

by

pairs

of

points

which

can

be

chosen

arbitrarily

From

these

points

a

function

of

deformation

is

calculated

by

methods

derived

from

scattered

data

interpolation

The

function

is

defined

on

the

whole

surrounding

space

and

thus

is

in

particular

applied

to

any

objects

embedded

in

this

space

The

given

points

are

mapped

exactly

while

the

other

points

follow

in

a

natural

manner

dependent

on

the

method

applied

Two

classes

of

approaches

for

scattered

data

interpolation

were

investigated

the

Shepard

methods

and

the

method

of

radial

basis

functions

Shepard

methods

have

the

advantage

of

more

intuitive

parameters

with

the

consequence

that

the

behavior

of

the

deformation

can

be

easily

tailored

to

meet

specific

needs

They

tend

to

produce

a

somewhat

jagged

deformation

which

is

often

useful

in

artistic

applications

but

is

undesirable

if

smooth

deformation

is

sought

Radial

basis

function

methods

on

the

other

hand

give

very

smooth

and

visually

pleasant

results

as

usually

desired

in

CAGD

applications

and

are

more

robust

with

respect

to

parameters

Modifying

them

to

suit

special

purposes

is

somewhat

less

intuitive

however

Both

approaches

are

easily

modified

to

give

local

deformations

Both

behave

gracefully

outside

the

convex

hull

of

the

control

points

In

-LSB-

Sederberg

Parry

-RSB-

an

approach

was

described

which

also

calculates

deformations

controlled

by

a

finite

number

of

movable

points

The

points

control

the

deformation

by

Bernstein

polynomials

In

contrast

to

our

approach

the

control

points

have

to

be

arranged

on

a

regular

grid

Furthermore

their

influence

is

less

immediate

so

that

the

result

of

the

deformation

is

harder

to

estimate

Global

mathematical

mappings

like

e.g.

those

used

in

-LSB-

Buche

Camillerap

-RSB-

can

only

be

used

for

small

deformations

or

for

deformations

with

a

simple

mathematical

description

In

summary

the

approach

presented

in

this

contribution

has

the

following

advantages

ffl

The

control

points

can

be

chosen

arbitrarily

ffl

The

deformation

is

performed

exactly

at

the

control

points

ffl

It

is

easy

to

control

the

global

influence

of

local

deformations

ffl

The

methods

are

easily

tailored

to

meet

special

requirements

There

are

other

classes

of

methods

of

scattered

data

interpolation

not

investigated

here

One

example

are

the

FEM

approaches

based

on

triangulation

of

the

given

data

points

FEM

interpolation

methods

were

not

taken

into

consideration

since

they

can

be

carried

out

well

in

but

become

computationally

expensive

and

hard

to

implement

in

higher

dimensions

whereas

the

methods

presented

here

seem

easily

extensible

The

latter

has

still

to

be

done

in

particular

in

space

Each

method

of

interpolation

behaves

like

a

deformable

material

with

very

own

properties

These

properties

are

more

or

less

natural

to

the

user

and

the

given

problem

The

usefulness

depends

on

the

experience

of

the

user

who

will

typically

be

well

acquainted

with

concrete

physical

materials

say

like

an

elastic

skin

of

rubber

If

a

method

of

interpolation

behaves

quite

different

the

user

may

have

troubles

and

may

need

some

time

to

gain

experience

with

the

new

material

Then

however

this

non-conventional

method

might

be

even

better

suited

for

solving

the

given

problem

of

deformation

than

any

real

material

may

be

Nevertheless

a

task

of

further

work

is

to

design

and

implement

methods

of

deformation

derived

from

physical

laws

References

-LSB-

-RSB-

P.

Alfeld

Scattered

data

interpolation

in

three

or

more

variables

In

T.

Lyche

and

L.L.

Schumaker

editors

Mathematical

Methods

in

CAGD

pages

-LCB-

Academic

Press

-LSB-

-RSB-

J.-D

Boissonat

Shape

reconstruction

from

planar

cross

sections

Computer

Vision

Graphics

and

Image

Processing

-LCB-

-LSB-

-RSB-

P.

Buche

and

J.

Camillerapp

Serial

cuttings

matching

An

application

to

muscle

fiber

characterization

In

Proceedings

EuroGraphics

pages

-LCB-

-LSB-

-RSB-

J.

Duchon

Splines

minimizing

rotation

invariant

semi-norms

in

Sobolev

spaces

In

W.

Schempp

and

K.

Zeller

editors

Constructive

Theory

of

Functions

of

Several

Variables

Lecture

Notes

in

Math

Vol

pages

-LCB-

Springer

Verlag

-LSB-

-RSB-

M.

Eck

Interpolationsmethoden

zur

Rekonstruktion

von

achen

aus

ebenen

Schnittfolgen

CAD

und

Computergraphik

-LRB-

-RRB-

-LCB-

February

-LSB-

-RSB-

R.

Farwig

Rate

of

convergence

of

Shepard

's

global

interpolation

formula

Mathematics

of

Computation

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

R.

Franke

Scattered

data

interpolation

Tests

of

some

methods

Mathematics

of

Computation

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

R.

Franke

and

G.

Nielson

Smooth

interpolation

of

large

sets

of

scattered

data

Int

Journal

for

Numerical

Methods

in

Engineering

-LCB-

-LSB-

-RSB-

W.J.

Gordon

and

J.A.

Wixom

Shepard

's

method

of

metric

interpolation

to

bivariate

and

multivariate

interpolation

Mathematics

of

Computation

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

B.

Guenther

A

system

for

simulating

human

facial

expression

In

N.

MagnenatThalmann

and

D.

Thalmann

editors

State-of-the-art

in

Computer

Animation

pages

-LCB-

Springer

Verlag

-LSB-

-RSB-

R.L.

Hardy

Multiquadric

equations

of

topography

and

other

irregular

surfaces

Journal

of

Geophysical

Research

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

P.

Lancaster

and

K.

Salkauskas

Surfaces

generated

by

moving

least

squares

methods

Mathematics

of

Computation

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

T.W.

Sederberg

and

S.R.

Parry

Free-form

deformation

of

solid

geometric

models

Computer

Graphics

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

D.

Shepard

A

two-dimensional

interpolation

function

for

irregularly

spaced

data

In

Proc

Nat

Conf

ACM

pages

-LCB-

-LSB-

-RSB-

G.

Wolberg

and

T.E.

Boult

Separable

image

warping

with

spatial

lookup

tables

Computer

Graphics

-LRB-

-RRB-

-LCB-

O O B O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I I O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O B I I O O O O O O O O B O O O O O O O O O O B O B I I O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

computer animation
deformation
geometric modelling
scattered data interpolation


CERC Technical Report Series Technical Memoranda CERC-TR-TM-95-006 MEDIAWARE : ON MULTIMEDIA SYNCHRONIZATION Yahya Y. Al-Salqan March 1995 Concurrent Engineering Research Center West Virginia University P.O. Box 6506 , Morgantown , WV 26506 Proceedings of the Second IEEE International Conference on Multimedia Computing and Systems . Washington , DC : IEEE Computer Society Press , May 1995 , Pp. 150-157 . 1 MediaWare : On Multimedia Synchronization Yahya Y. Al-Salqany , Carl K. Changz , and Y. V. -LRB- Ramana -RRB- Reddyy yConcurrent Engineering Research Center West Virginia University Morgantown , WV 26506 E-mail : alsalqan@cerc.wvu.edu zEECS Department -LRB- M/C 154 -RRB- University of Illinois at Chicago Chicago , IL 60607 Abstract Temporal relationships are inevitable among multimedia objects . Synchronization of multiple data streams in time has thus been recognized as a crucial requirement for multimedia applications . In this paper we study the synchronization requirements among distributed multimedia objects . We describe a temporal model that deals with inter-stream and intra-stream synchronization requirements in live media objects . Both the blocking policy and the restricted blocking policy will be presented . In addition , our model handles the accurate and uncertain temporal requirements which may exist in the teleorchestration of fuzzy scenarios . The fuzzy scenarios that contain imprecise synchronization constraints , such as unknown object presentation starting or ending points , can be transformed into more accurate temporal specifications . The paper provides several examples to show our approach . 1 Introduction Distributed multimedia applications are growing rapidly in a variety of fields , including education , entertainment , weather forecasting , medical diagnosis , etc. . Multimedia objects include images , voice , full motion video , text and numeric data . Due to the diverse and distributed nature of multimedia objects , an application object can be composed of different but related components . For real-time playout of such inter-related streams , a synchronization technique is essential . Multimedia composition can be specified by a scenario representing the temporal and spatial relationships among the involved media objects . The composition and presentation of multimedia objects were discussed in a distributed multimedia information system . Two major problems involved in the media composition were investigated , namely : the specification and modeling of the required temporal relationships , and the implementation of the temporal specifications by scheduling the involved objects -LSB- LKG94 -RSB- . The composition of multimedia teleorchestration -LRB- remote multimedia presentation -RRB- may involve imprecise and relative temporal relationships among media objects . A transformation technique to convert such imprecise scenarios into more accurate ones is therefore needed . In this paper , we present a modeling mechanism to specify the distributed multimedia synchronization and temporal relationships . Our approach deals with both accurate and fuzzy scenarios . Synchronization , in general , can be classified into two types : intra-stream and inter-stream . Intra-stream synchronization is mainly concerned with delivering every media object in time to meet the intended playout deadline . Intra-stream is dealing with the low-level synchronization of an individual data object originated by any media source . Inter-stream synchronization , however , maintains the temporal relationships between different but related media streams , such as the lip-synch between video and audio objects . Petri Nets have been used for modeling and simulating concurrent systems since their introduction by C. A. Petri in the 1960s -LSB- Mur92 , GL81 , Jen81 , Rei83 , BGV91 , Jen91 -RSB- . For the purpose of defining a better formalization for distributed multimedia computing , concurrent object-oriented , concurrent engineering , heterogeneous systems and open system architecture , we introduced the Interoperable Petri Nets -LRB- IPNs -RRB- -LSB- AS94 -RSB- . IPNs extends the High-Level Petri Nets by adding a port and a port interface construct at the 3 graphical notation level , and by introducing the interoperability notion at the operational level . In our work , we introduce MediaWare as a distributed multimedia computing environment and formalism . The MediaWare architecture includes seven layers : the application layer on top ; the Quality-of-Connection layer , which includes the Quality-of-Service , Synchronization specifications , and User Interaction ; the Interoperable Virtual Connection Protocol -LRB- IVCP -RRB- layer ; the Object Database layer ; the Object Request Broker -LRB- ORB -RRB- layer , the Interface Definition Language -LRB- IDL -RRB- mapping ; and , finally , the Object Implementation layer . One of MediaWare 's major features is its ability to support the dynamic creations of distributed multimedia scenarios , which is the focus of this paper . This paper is structured as follows . Section 2 reviews related work . Section 3 informally introduces the Interoperable Petri Net and discusses the IPN 's firing rules1 . Section 4 presents the IPN modeling of the inter-stream and intra-stream synchronization schemes . Both the blocking and restricted blocking policies will be modeled . Finally , Section 5 concludes the paper . 2 Related Work The complexity of multimedia applications has grown significantly during the past several years . To tame such complexity , formal approaches for the specification , design , and analysis of multimedia applications are therefore highly desirable . However , few such models have been adequately defined to date . A model should be flexible and rich enough to address several multimedia issues , especially the distributed nature of multimedia resources and synchronization among their playout . Multimedia information systems are characterized by the need to compose data of multiple types and formats for presentation , communication , or storage . The need to handle synchronization among media streams is especially important in distributed applications and environments . Temporal relationships are inevitable among multimedia objects . In particular , the synchronization of multiple data streams in time has been recognized as a significant requirement of multimedia applications -LSB- GBC88 , GC91 , 1For more details and a formal definition of IPN , refer to -LSB- AS94 -RSB- . 4 LG91 -RSB- . Recently literature suggests several methods to address this problem at the low -LRB- bitways -RRB- level -LSB- LBea90 , LG91 , Nic90 , Ste90 -RSB- . Several authors have considered formal methods for multimedia synchronization -LSB- LG91 , QWG93 , LG90 , PR93 -RSB- . In -LSB- LG91 , QWG93 , LG90 -RSB- , the Object Composition Petri Net -LRB- OCPN -RRB- and its extension -LRB- XOCPN -RRB- are discussed . An example of an OCPN specification for multimedia data is shown in Figure 1 , which depicts a video clip , an audio clip and some text data ordered in time as illustrated . Firing the transition t1 puts a token into the ` Video ' and ` Audio ' places . Each of these places is assigned a duration o , which specifies the concurrent playout of the video and audio objects for o time units . After o units have elapsed , transition t2 fires to start the text object playout . Pinitial t1 Video Audio t2 Text t3 Pfinal t t Figure 1 : An OCPN Example : playing audio and video clips followed by text . Figure 1 can be translated into the XOCPN as shown in Figure 2 . In Figure 2 , places R -LRB- Resource Setup -RRB- , L -LRB- Resource Release -RRB- , and IPP -LRB- Interstream Pacing Point -RRB- represent control objects ; and the SIU places represent Synchronization Interval Units . The playout duration of video and audio object , since they are synchronous , are divided equally into SIUs . SIUs are used for intra-stream synchronization , whereas IPP s are used for interstream synchronization . The XOCPN -LRB- and its predecessor OCPN -RRB- does not capture the distributed nature of multimedia applications and devices . For instance , in order to allocate -LRB- release -RRB- a resource , all that XOCPN does is introducing an R -LRB- L -RRB- place . Resource allocation in distributed multimedia applications is much more complex , however . An additional disadvantage of the XOCPN model is that the resultant playout of the synchronized streams are dependent upon each other . This 5 Pinitialt1 t2 Text Pfinal R R SIU1 SIU2 IPP IPP SIU3 ... L L SIU3SIU2SIU1 ... R SIU1 ... Video Audio Video resource audior resouce Figure 2 : XOCPN transformation from OCPN . does not correspond to the real-world situation . For example , a video camera captures a video stream and a microphone captures a voice stream , although the resultant playout should be synchronized , the camera and the microphone devices , are completely independent and they capture and produce data concurrently . Our model will address this problem satisfactorily . Interoperability is a distinguished feature of distributed multimedia applications . Generally speaking , two or more objects/agents/applications are interoperable if they can cooperate , regardless of their underlying technology , to accomplish a joint task2 . In order to overcome the limitations of the XOCPN model , we introduce the Interoperable Petri Net -LRB- IPN -RRB- , which preserves the interoperability characteristics of multimedia applications . We use the IPN model in the formal specifications and analysis of multimedia objects and synchronization . 3 Interoperable Petri Nets Heldman in -LSB- Hel92 -RSB- pointed out that , \ As we integrate communications and computers , as we layer information networks ' services , we are indeed forming a global village '' . In this section , we propose an integration mechanism and a formalism for distributed computational agents . Our model , the Inter - 2This is our own definition of interoperability . 6 operable Petri Net -LRB- IPN -RRB- , extends the High-Level Petri Net -LRB- HLPN -RRB- model by introducing port and port interface constructs at the graphical notation level and by introducing the interoperability feature at the operational level . Interoperability is achieved through the Interoperable Virtual Connection Protocol -LRB- IVCP -RRB- 3 . Our paper reveals the power of the IPN to model synchronization among distributed multimedia objects . An IPN model , informally , consists of processing element , ports and ports ' interfaces . The processing element is represented by HLPN ; ports can input or output ; and ports interfaces are the visible part of the IPN module where data can be consumed or produced . In order for two -LRB- or more -RRB- IPNs to interoperate , a virtual connection must be established first between the source output port interface and the destination input port interface . There are two different types of possible virtual connections associated with two different types of ports : ffl A connection may pass messages \ colored tokens '' from transition -LRB- s -RRB- to place -LRB- s -RRB- . In this case , the transition is the output port in the source IPN module and the place is the input port of the target IPN module . ffl A connection may pass color tokens from place -LRB- s -RRB- to transition -LRB- s -RRB- . In this case , the place is the output port in the source IPN module , whereas the transition is the input port in the target IPN module . In IPNs , the distinction between these two types of connections is made clear by the corresponding interfaces icons4 . We introduce some of the IPN icons for port interfaces in Figure 3 . A place seeker is used as an interface for a transition output port , while a place receptor is used as an interface for a place input port . On the other hand , a transition seeker is used as an interface for a place output port , while a transition receptor is used as an interface for a transition input port . The IPN net in Figure 3 , for example , describes a simple representation of a microphone and a speaker devices . In the initial marking , there is one colored token , < T = audio ; BW = 100 ; D = mic > , at place A . The IPN in Figure 3 consists of four parts : ffl The net structure . 3Refer to -LSB- AS94 -RSB- for a full description of IVCP . 4Some of them are similar to the icons in -LSB- Neu93 -RSB- . 7 Microphone P < audio ,100 , mic > A -LRB- y -RRB- -LRB- y -RRB- t1 -LRB- y , loc -RRB- -LRB- T , BW , D , Loc -RRB- -LSB- Loc = loc. , T = audio -RSB- -LSB- D = mic , T = audio -RSB- Speaker Color TType T = -LCB- audio -RCB- Color BWType BW = -LSB- 80 ... 100 -RSB- Color DType D = -LCB- mic , file -RCB- Color EType E = -LCB- e -RCB- Color LocType Loc = -LCB- machine addresses -RCB- Color PType P = Product TType * BWType * DType Var x , y = P -LRB- a -RRB- -LRB- T , BW , D , Loc -RRB- Speaker -LRB- x -RRB- -LSB- Loc = mymach. , T = audio -RSB- t2 P C B <e> -LRB- x -RRB- -LRB- x -RRB- -LRB- x -RRB- Speaker Color TType T = -LCB- audio -RCB- Color BWType BW = -LSB- 70 ... 120 -RSB- Color DType D = -LCB- speaker , mic , file -RCB- Color EType E = -LCB- e -RCB- Color LocType Loc = -LCB- machine addresses -RCB- Color PType P = Product TType * BWType * DType Var x , y = P -LRB- b -RRB- -LRB- b -RRB- Figure 3 : Two IPN modules , a Microphone -LRB- a -RRB- and a Speaker -LRB- b -RRB- . ffl The declarations . ffl The net inscription . ffl The port interfaces . The net structure is a bipartite , directed graph with two kinds of nodes , places and transitions . In Figure 3 the net structures , enclosed in the shaded boxes , represent the IPN processing elements . Specifically , Figure 3 -LRB- a -RRB- represents the microphone behavior specification , and -LRB- b -RRB- represents the speaker counterpart . The declarations declare the types of color sets and variables . They are analogous to the type declarations in programming languages -LSB- Jen91 -RSB- . Each place has a color type associated with it , and a token must be of the same type -LRB- color set -RRB- as the place in order to reside in it . In this example , color TType T is a set of device type -LRB- audio -RRB- . Color PType P is a product of color types TType , BWType -LRB- bandwidth of range type -RRB- , and DType -LRB- enumerating device kind -RRB- . Finally , we present an uncolored token by e of type EType . 8 Therefore , in order for a token to reside in place A , it must have a token with a color value of < audio ; 80 < = BW > = 100 ; micjf ile > . The net inscriptions are attached to places , transitions , arcs , port interfaces , or IPN modules . In Figure 3 , places , such as A , have three different types of inscription : names -LRB- e.g. A -RRB- , color sets -LRB- e.g. P -RRB- , and initialization expression -LRB- e.g. < TType = audio ; BW = 100 ; DType = mic > -RRB- . Transitions , such as t1 , have two kinds of inscriptions : names -LRB- e.g. t1 -RRB- and guards -LRB- e.g. -LSB- D = mic ; T = audio -RSB- -RRB- . Unlike a place name , the port interface name has a semantic use . We can regard it as a function , a procedure , or a method name . The IPN module has one kind of inscription , its name -LRB- e.g. Microphone -RRB- . A guard is a boolean expression5 that must be evaluated to true in order for a transition to fire or a port interface to be promoted to an excitation state . For example , the component D in the variable y must equal to mic as well as T must equal to audio in order for transition t1 to fire . As a convention , guards which always evaluate to true are omitted from the net . The arc label may contain variables -LRB- such as Y -RRB- , constants -LRB- such as loc -RRB- , and functions and operations that are defined in the declarations . Each port has a corresponding port interface -LRB- e.g. place seeker attached to transition t1 -RRB- . A transition -LRB- place -RRB- output port has a place -LRB- transition -RRB- seeker as a port interface . On the other hand , a transition -LRB- place -RRB- input port has a transition -LRB- place -RRB- provider as a port interface . 3.1 IPN 's Firing Rules Before any transition can be enabled in IPN -LRB- similar to HLPN -RRB- , all variables in the arc label function must be bound to colors of the corresponding types . According to this binding , the guard must also be evaluated to true . Then , the transition can be fired thus removing tokens from its input place -LRB- s -RRB- and adding tokens to its output place -LRB- s -RRB- . The number of removed and added tokens is determined by the value of the corresponding arc label function -LRB- evaluated with respect to the binding on hand -RRB- . If a transition is an output port of an IPN module -LRB- as shown in Figure 3 -LRB- a -RRB- -RRB- , its firing produces colored tokens which are added to its port interface . If the interface is at its normal state , it will be promoted into the excitation state . 5Enclosed in square brackets -LRB- -LSB- -RSB- -RRB- . 9 If a place is an output port of an IPN module , after receiving a token , the arc label function of the arc connects the port to its interface -LRB- transition seeker -RRB- which will then be evaluated and bound to the colors of the corresponding types . According to the binding , the port interface will be promoted into the excitation state only if the guard is evaluated to true . Since we use IPNs to model multimedia synchronization and synchronization deals with the temporal behavior of related streams , we include the notion of time into IPNs -LRB- with the same conventions of timed Petri nets -RRB- . With timed IPNs , we are free to choose the granularity of synchronization by assigning time durations to places . The firing rules of a timed IPN are similar to the rules of timed Petri nets : 1 . A transition , ti , fires immediately when each of its input places connected through regular arcs contains an unlocked number of tokens matching the corresponding arc labels , and these tokens evaluate the transition 's guard to true . If ti is incident on an inhibitor arc6 , the place incident on that arc must be empty in order for ti to fire . 2 . Upon firing , the transition , ti , removes tokens from each of its input places and adds tokens to each of its output places according to arc labels . 3 . After receiving a token , a place , pi , remains in the active state for the interval assigned to it which is specified by a duration , Tj . During Tj , the token is locked . When a place , pi , becomes inactive -LRB- Tj expires -RRB- the token becomes unlocked . We use the IPN model to formalize distributed multimedia applications and environments . Henceforth , we refer to the distributed multimedia environment and its formalism as MediaWare . MediaWare includes the formalization of multimedia synchronization , multimedia quality-of-service , and user interactions . This article is devoted to formalize multimedia synchronization and temporal relations among media objects . 6An arc with a circle instead of arrow in its end . 10 4 Modeling Multimedia Synchronization Schemes Current multimedia systems use different synchronization schemes to combine heterogeneous data streams . In this section , we discuss the most popular schemes and model them using IPNs . In particular , we consider the interand intra-stream synchronization schemes . For each scheme , we examine the blocking -LRB- idle-waiting -RRB- and restricted blocking synchronization policies , and show that these policies can be sufficiently and naturally modeled using IPNs . 4.1 Intra-Stream Synchronization Intra-stream synchronization is concerned with delivering each stream object on time to meet its playout deadline -LSB- QWG93 -RSB- . More specifically , for isonchronous streams , the purpose of intra-stream synchronization is to maintain the synchronization requirements of isonchronous data . Thus , some synchronization schemes -LRB- at source or destination -RRB- must be invoked to maintain an acceptable level of playout standards . This becomes very crucial in distributed environments , where unexpected synchronization anomalies can exist . As described in -LSB- QWG93 -RSB- , the two types of intra-stream synchronization policies are Blocking -LRB- idle-waiting -RRB- Policy and Restricted Blocking Policy . In the blocking policy , the playout of slow stream -LRB- i.e. a stream that does not arrive at the expected time -RRB- , is delayed until that stream arrives . This semantics can be specified using IPNs as shown in Figure 4 . Figure 4 is a representation of capturing data streams in input devices such as a camera device or a microphone device . In order for this model to start its intended operation , a triggering token is required at place A . The triggering token is passed to the model through the input port interface -LRB- place provider -RRB- . A camera source device capturing 30 frames per second is modeled , since the camera produces an isonchronous streams . Assessing a duration -LRB- o = 130 -RRB- second to place A will divide the input stream into frames of 130 second each . Capturing a frame is modeled by firing the source transition -LRB- ts -RRB- . When the triggering token is placed into place A , this token becomes unlocked after 130 second . Assuming that transition ts has been fired -LRB- camera device is on -RRB- , this puts a token in place B which enables transition tc . When transition tc fires , one token -LRB- frame -RRB- is put in the playout state -LRB- at 11 tc Playout A B -LRB- x -RRB- -LRB- x -RRB- ts t = 1/30 t = 1/30 Figure 4 : IPN model of intra-stream blocking synchronization . the output port interface -RRB- , and another token is put back in place A . After 130 second , the token in place A is unlocked . If transition ts does not fire -LRB- does not capture the second frame ; no tokens will be in place B -RRB- , therefore , transition tc can not fire . In other words , the playout of the second frame is blocked until that frame comes in , during which the transition tc is idlewaiting . The blocking policy could be inefficient in some multimedia applications , since it may lead to indefinite waiting or deadlock . Therefore , there is a need for less restricted policy to handle the intra-stream synchronization . In the Restricted Blocking Policy , the playout process is blocked for a pre-specified period of time only -LRB- i.e. , timeout -RRB- . After the timeout expires , it may play out the most recent sub-stream object -LRB- such as the last frame in a camera case -RRB- . This policy is modeled using IPNs as shown in Figure 5 . In Figure 5 , place S represents a buffer to hold the source data \ frames '' , place A is a control place , place B is a buffer which holds the last played out token , and place C is a common place to collect the data for delivery . In addition , transition ts produces the source frames , and transitions tc1 and tc2 put the needed frames for playout . When transition ts fires , it adds a token to place S . This enables transition tc1 . When transition tc1 fires , it indicates that a new frame has been captured and is ready for playout . Firing tc1 adds a token to place C , and another one to place B . The token in place C enables the output port transition , tn1 . When tn1 fires , it moves the token to its interface for playout . After a time unit of 130 second has elapsed , the 12 tc1 Playout A S -LRB- x -RRB- -LRB- x -RRB- ts t = 1/30 t = 1/30 B t = 1/30 d = e C tn1 tn2 tc2 Figure 5 : IPN model of intra-stream restricted blocking synchronization . 13 tokens in places A and B will be unlocked . If there are no new tokens in place S , transition tc2 will fire and put the most recent frame into place C for playout . Firing tc2 when place S is empty is guaranteed by introducing the inhibitor arc between the transition tc2 and the place S. Transition tn2 is a sink transition which is introduced to prevent buffer overflow . Note that transition tn2 is assigned a delay of ffl > 0 , whereas all other transitions are immediate -LRB- ffl = 0 -RRB- . Therefore , if transition ts continues to produce stream objects with no delay , the previous frame will be destroyed after the delay ffl . The delay ffl gives tc2 a higher priority than tn2 . Figures 4 and 5 show that our IPN model reflects the distributed nature of multimedia applications and devices . Frames are modeled by putting tokens at the output port interface -LRB- a place seeker in this case -RRB- . Whenever the output port interface receives a token , it will be promoted from the normal into the excitation state , which activates the IVCP -LSB- AS94 -RSB- . The IVCP then provides the appropriate virtual path to send the source data from the source device -LRB- camera -RRB- to the destination -LRB- a display in this case -RRB- . Since the source and the destination could be geographically distributed , we present the source as an autonomous IPN module . The distributed nature was not reflected in any previous formalization techniques . 4.2 Inter-Stream Synchronization Inter-stream synchronization is needed to meet the temporal requirements among related streams . The inter-stream synchronization problem can be addressed by pacing the playout of related multiple streams with respect to each other so that the skew among them does not exceed a tolerable limit . We addressed the inter-stream synchronization issue by weaver objects depicted in Figure 6 below . Figure 6 mixes the data streams from the two objects , the MicrophoneDevice object and the CameraDevice object . These two objects produce data streams . According to the IVCP , there will be a virtual connection with pi8 through the virtual connection V C1 , and another virtual connection with pi9 through V C2 . Through the virtual connections , V C1 and V C2 , tokens will be passed into pi8 and pi9 , respectively . The port interfaces pi8 and pi9 pass the token into places A and B , respectively . Consequently , transition t3 can fire , which produces the mixed data streams into port interface pi10 for transmission to a destination . Figure 6 depicts the inter-stream blocking synchronization policy , where the faster stream needs 14 to wait -LRB- blocked -RRB- for the slower one7 . MicrophoneDevice CameraDevice t3 pi8 pi10 WeaverDevice pi9 VC1 VC2 A B Figure 6 : Inter-stream synchronization : blocking policy . Note that the IPN modeling of the inter-stream restricted blocking policy is similar to its counterpart -LRB- intra-stream restricted blocking policy -RRB- , and is omitted . Also , an un-weaver object is required to unweave the data at the destination , where the weaved -LRB- synchronized -RRB- data need to be played out . This guarantees that the end-to-end playout of related data streams will be synchronized . Above , we discussed the requirements and formalism of synchronized live multimedia objects . In -LSB- AS94 -RSB- we describe various temporal and synchronization relations that may exist between any teleorchestrated multimedia objects . Through the discussion , we show that our MediaWare is rich enough to support \ absolute '' and \ fuzzy '' temporal relations among media objects . Such detail were omitted from this paper because of space limitations . 7The IPN inscription details were omitted from Figure 6 for simplicity . 15 5 Conclusion In this paper , we have introduced the Interoperable Petri Nets -LRB- IPNs -RRB- for formal modeling and specification of the synchronization requirements and temporal relationships among distributed multimedia objects . The inter-stream and intra-stream synchronization schemes have been modeled using IPNs . Both the blocking and restricted blocking policies have been addressed . All the temporal interval relations were discussed and have been modeled8 . We believe that our model is capable of specifying absolute and fuzzy scenarios , and capable of transforming the imprecise specifications into more accurate ones . In addition to synchronization , we use IPNs to model several distributed multimedia issues , such as quality-of-service , user interactions , and interoperability among media objects . Acknowledgment The authors would like to express their deepest gratitude to Ms. Mary Carriger and Ms. Christina Godlewski for their valuable help . References -LSB- AS94 -RSB- Yahya Y. Al-Salqan . MediaWare : A Distributed Multimedia Computing Environment and Formalism Based on Interoperable Petri Net . Ph.D. . Thesis , University of Illinois at Chicago , 1994 . -LSB- BGV91 -RSB- W. Brauer , R. Gold , and W. Vogler . A survey of behaviour and equivalence preserving refinement of Petri nets . Lecture Notes in Computer Science : Advances in Petri nets 90 , pages 1 -LCB- 46 , 1991 . -LSB- GBC88 -RSB- A Ghafoor , P. Berra , and R. Chen . A distributed multimedia database system . Proceedings of IEEE Workshop on future trends in distributed computing systems in 1990s , pages 461 -LCB- 469 , September 1988 . 8Refer to -LSB- AS94 -RSB- for the full modeling of all the temporal relations . 16 -LSB- GC91 -RSB- J. Gemmel and S. Christodoulakis . Principles of delay sensitive multimedia data retrieval . Proceedings of first IEEE international conference on multimedia information systems , pages 147 -LCB- 158 , January 1991 . -LSB- GL81 -RSB- H.J. Genrich and K. Lautenbach . System Modeling with HighLevel Petri Nets . Theoretical Computer Science , 13:109 -LCB- 136 , 1981 . -LSB- Hel92 -RSB- R.K. Heldman . Global Telecommunications - Layered Networks Layered Services . McGraw Hill , 1992 . -LSB- Jen81 -RSB- K. Jensen . Colored Petri Nets and invariant-method . Theoretical Computer Science , 14:317 -LCB- 336 , 1981 . -LSB- Jen91 -RSB- K. Jensen . Coloured Petri nets : A high level language for system design and analysis . Lecture Notes in Computer Science : Advances in Petri nets 90 , pages 342 -LCB- 416 , 1991 . -LSB- LBea90 -RSB- W.H. Leung , T. J. Baumgartner , and Y. H. Hwang et al. . A software architecture for workstations supporting multimedia conferencing in packet switching networks . IEEE Journal on selected areas in communications , 8 -LRB- 3 -RRB- :380 -LCB- 390 , April 1990 . -LSB- LG90 -RSB- Thomas D. C. Little and Arif Ghafoor . Synchronization and storage model for multimedia objects . IEEE Journal on selected areas in communications , 8 -LRB- 3 -RRB- :413 -LCB- 427 , April 1990 . -LSB- LG91 -RSB- Thomas D. C. Little and Arif Ghafoor . Multimedia synchronization protocols for broadband integrated services . IEEE Journal on selected areas in communications , 9 -LRB- 9 -RRB- :1368 -LCB- 1381 , December 1991 . -LSB- LKG94 -RSB- L. Li , A. Karmouch , and N.D. Georganas . Multimedia Teleorchestra with Independent sources : Part 1 - Temporal Modeling of Collaborative Multimedia Scenarios . Multimeda Systems , pages 143 -LCB- 153 , February 1994 . -LSB- Mur92 -RSB- Tadao Murata . Lecture Notes for EECS 564 . University of Illinois at Chicago-Department of Electrical Engineering and Computer Science , Chicago , IL , 1992 . 17 -LSB- Neu93 -RSB- Phillip D. Neumiller . Optimal Object Oriented Design of Data Networking Software using an Actor Augmented Booch Methodology . Master Thesis , University of Illinois at Chicago , 1993 . -LSB- Nic90 -RSB- Cosmos Nicolaou . An architecture of real-time multimedia communication systems . IEEE Journal on selected areas in communications , 8 -LRB- 3 -RRB- :391 -LCB- 400 , April 1990 . -LSB- PR93 -RSB- B. Prabhakaran and S.V Raghavan . Synchronization models for multimedia presentation with user participation . Proceedings of first ACM conference on multimedia , pages 157 -LCB- 166 , August 1993 . -LSB- QWG93 -RSB- Naveed U. Qazi , Miae Woo , and Arif Ghafoor . A synchronization and communication model for distributed multimedia objects . Proceedings of first ACM conference on multimedia , pages 147 -LCB- 156 , August 1993 . -LSB- Rei83 -RSB- W. Reisig . Petri nets with individual tokens . InformatikFachberichte , 66 -LRB- 21 -RRB- :229 -LCB- 249 , 1983 . -LSB- Ste90 -RSB- Ralf Steinmetz . Synchronization properties in multimedia systems . IEEE Journal on selected areas in communications , 8 -LRB- 3 -RRB- :401 -LCB- 412 , April 1990 . 18

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

interoperable petri nets
ipn
scenario
teleorchestration
temporal relations