Adaptive

algorithms

vs.

Time

Warp

An

analytical

comparison

Sudhir

Srinivasan

Paul

F.

Reynolds

Jr.

Computer

Science

Report

No

April

Adaptive

algorithms

vs.

Time

Warp

An

analytical

comparison

Sudhir

Srinivasan

Paul

F.

Reynolds

Jr.

Department

of

Computer

Science

Olsson

Hall

University

of

Virginia

Charlottesville

VA

Phone

-LRB-804-RRB- 982-2200

Fax

-LRB-804-RRB- 982-2214

Email

-LCB-

pfr

-RCB-

@uvacs

cs.Virginia.EDU

Abstract

Adaptive

synchronization

algorithms

have

been

proposed

to

improve

upon

purely

conservative

and

purely

optimisitic

algorithms

Experimental

studies

have

have

indeed

provided

encouraging

results

In

the

spirit

of

previous

analyses

we

present

the

first

known

analytical

comparison

of

adaptively

optimisitic

algorithms

with

the

Time

Warp

protocol

We

define

a

class

of

adaptive

protocols

the

asynchronous

adaptive

waiting

protocols

-LRB-

AAWP?s

-RRB-

and

identify

several

practical

protocols

that

belong

to

this

class

We

show

that

Time

Warp

can

outperform

an

AAWP

arbitrarily

We

describe

NPSI

adaptive

protocols

a

sub-class

of

AAWP?s

and

specify

a

member

of

this

sub-class

the

Elastic

Time

Algorithm

We

show

that

this

algorithm

can

outperform

Time

Warp

arbitrarily

Submitted

to

Winter

Simulation

Conference

Introduction

Historically

research

in

synchronization

algorithms

-LRB-

protocols

-RRB-

for

parallel

discrete

event

simulation

-LRB-

PDES

-RRB-

has

followed

two

tracks

conservative

and

optimistic

-LSB-

-RSB-

A

significant

result

of

this

research

is

that

neither

approach

seems

to

be

universally

efficient

As

defined

in

-LSB-

-RSB-

adaptive

protocols

are

those

that

modify

their

behavior

dynamically

in

response

to

changes

in

the

state

of

the

simulation

Several

adaptive

protocols

have

been

proposed

and

implemented

recently

with

encouraging

results

However

there

are

no

analytical

studies

comparing

the

performance

of

adaptive

protocols

with

that

of

traditional

protocols

While

experiments

have

shown

that

adaptively

optimistic

protocols

improve

on

the

performance

of

the

purely

optimistic

Time

Warp

protocol

-LSB-

-RSB-

in

general

it

is

interesting

to

question

whether

they

can

perform

worse

than

Time

Warp

We

present

a

comparison

of

a

general

class

of

adaptive

protocols

called

the

asynchronous

adaptive

waiting

protocols

-LRB-

AAWP?s

-RRB-

with

Time

Warp

in

the

vein

of

the

worst-case

comparison

between

the

Chandy-Misra

protocol

-LSB-

-RSB-

and

Time

Warp

in

-LSB-

-RSB-

We

show

that

it

is

possible

for

Time

Warp

and

AAWP?s

to

outperform

each

other

arbitrarily

Thus

while

intuition

suggests

that

adaptive

protocols

should

enhance

performance

in

general

our

analysis

indicates

that

they

must

be

designed

carefully

since

incorrect

adaptive

decisions

can

lead

to

arbitrarily

worse

performance

than

Time

Warp

We

assume

familiarity

with

the

common

approach

to

PDES

-LSB-

-RSB-

namely

the

partitioning

of

a

discrete

event

simulation

into

components

called

logical

processes

-LRB-

LP?s

-RRB-

Each

LP

is

itself

a

sequential

discrete

event

simulator

The

LP?s

must

execute

events

whether

generated

internally

or

scheduled

by

other

LP?s

without

violating

causality

constraints

-LRB-

effectively

-RRB-

Typically

this

is

the

responsibility

of

a

protocol

A

conservative

protocol

is

one

in

which

an

LP

executes

an

event

only

after

determining

that

it

is

safe

to

do

so

-LRB-

i.e.

no

other

event

with

a

smaller

timestamp

will

be

scheduled

later

-RRB-

Thus

it

is

possible

for

an

LP

to

remain

blocked

for

some

period

of

time

while

there

is

insufficient

information

to

proceed

with

the

next

scheduled

event

An

optimistic

algorithm

takes

the

opposite

approach

in

that

LP?s

execute

events

without

the

guarantee

of

safety

and

repair

their

execution

if

and

when

an

error

is

detected

typically

using

a

checkpoint

and

rollback

approach

In

-LSB-

-RSB-

aggressiveness

is

defined

to

be

the

property

by

which

an

LP

processes

events

conditionally

-LRB-

i.e.

without

a

guarantee

of

safety

-RRB-

and

risk

is

defined

as

passing

messages

that

are

based

on

aggressive

or

inaccurate

processing

Conservative

protocols

are

non-aggressive

and

without

risk

while

optimistic

protocols

are

completely

aggressive

and

with

risk

Therefore

these

two

classes

of

protocols

represent

the

extremities

of

a

spectrum

of

possibilities

Both

conservative

and

optimistic

strategies

have

their

pros

and

cons

The

main

advantage

of

conservative

protocols

is

that

they

do

not

have

any

checkpointing

and

rollback

overheads

However

processes

may

be

blocked

due

to

insufficient

information

when

in

fact

it

would

be

safe

for

them

to

proceed

This

artificial

blocking

-LSB-

-RSB-

introduces

lost-opportunity

cost

-LSB-

-RSB-

On

the

other

hand

optimistic

protocols

do

not

incur

lostopportunity

costs

since

processes

never

block

The

main

hazard

of

optimistic

protocols

is

that

the

checkpointing

and

rollback

costs

may

degrade

performance

severely

The

gap

between

these

two

extremes

may

be

bridged

either

by

adding

optimism

to

conservative

protocols

or

by

limiting

the

optimism

of

purely

optimistic

protocols

The

inherently

dynamic

nature

of

simulations

-LSB-

-RSB-

suggests

that

regardless

of

the

approach

to

combine

the

conservative

and

optimistic

strategies

the

hybrid

scheme

must

also

be

dynamic

Adaptive

protocols

are

those

that

change

the

bindings

of

one

or

more

of

their

design

variables

dynamically

-LSB-

-RSB-

Adaptive

protocols

that

modify

their

aggressiveness

and

risk

-LRB-

collectively

called

optimism

-RRB-

dynamically

seem

the

most

likely

to

perform

well

consistently

Henceforth

we

will

refer

to

such

protocols

simply

as

adaptive

protocols

Several

protocols

have

been

proposed

that

either

limit

optimism

in

Time

Warp

or

add

optimism

to

conservative

protocols

-LSB-

-RSB-

Most

of

these

have

been

shown

to

improve

the

performance

of

the

protocols

from

which

which

they

are

derived

Since

all

of

these

results

are

experimental

it

is

interesting

and

important

to

question

whether

adaptive

protocols

can

perform

worse

than

traditional

protocols

Our

analysis

shows

they

can

In

the

next

section

we

define

the

AAWP

class

of

adaptive

protocols

and

identify

several

members

of

this

class

We

then

show

by

example

that

Time

Warp

can

outperform

an

AAWP

arbitrarily

Next

we

describe

a

framework

for

a

family

of

adaptive

protocols

called

NPSI

adaptive

protocols

-LRB-

which

are

also

AAWP?s

-RRB-

Finally

we

describe

a

specific

NPSI

adaptive

protocol

ETA

based

on

this

framework

and

show

that

ETA

can

outperform

Time

Warp

by

a

factor

proportional

to

the

amount

of

logical

time

simulated

Assumptions

The

reader

is

referred

to

-LSB-

-RSB-

for

details

of

Time

Warp

and

related

concepts

We

assume

the

following

for

all

protocols

in

this

paper

Each

LP

is

located

on

its

own

processor

The

protocols

employ

aggressive

cancellation

and

aggressive

rollback

The

following

assumptions

define

asynchronous

adaptive

waiting

protocols

-LRB-

AAWP?s

-RRB-

the

class

of

adaptive

protocols

to

which

our

analysis

applies

These

protocols

are

based

on

Time

Warp

and

control

optimism

by

introducing

delays

between

event

executions

i

-RRB-

The

simulation

loop

of

an

LP

is

as

follows

Note

we

do

not

make

any

assumptions

about

the

particulars

of

the

AAWP

such

as

criteria

for

deciding

if

an

LP

should

wait

on

a

given

iteration

or

how

long

it

should

wait

We

only

assume

that

such

waiting

occurs

between

event

executions

if

at

all

An

LP

aborts

its

waiting

if

it

receives

a

message

that

will

cause

it

to

roll

back

A

separate

delay

scheme

may

be

used

to

control

risk

ii

-RRB-

The

waiting

at

each

LP

is

asynchronous

with

respect

to

the

waiting

at

other

LP?s

iii

-RRB-

The

actions

of

an

LP

that

are

relevant

to

the

analysis

are

event

execution

rollback

and

adaptive

waiting

We

ignore

overheads

such

as

state

saving

receiving

messages

global

virtual

time

-LRB-

GVT

-RRB-

computation

and

fossil

collection

for

simplicity

the

analysis

can

be

extended

to

include

these

as

well

iv

-RRB-

The

AAWP

does

not

increase

the

capability

of

the

LP?s

to

guess

computation

An

increase

in

the

guessing

power

of

an

LP

could

compensate

for

any

deleterious

effects

of

the

adaptive

control

Since

an

LP?s

guessing

capability

depends

entirely

on

the

application

being

simulated

it

is

reasonable

to

assume

the

AAWP

can

not

increase

this

capability

v

-RRB-

Since

adaptive

waiting

is

expected

to

reduce

rollback

costs

the

depth

-LRB-

and

therefore

cost

-RRB-

of

each

rollback

is

assumed

to

be

bounded

by

a

constant

for

the

AAWP

If

the

rollback

costs

are

not

While

not

done

Process

event

Wait

for

time

d

Rollback

if

necessary

Process

received

messages

Save

state

Collect

fossils

endwhile

bounded

for

the

AAWP

our

claim

that

an

AAWP

can

perform

arbitrarily

worse

than

Time

Warp

can

be

shown

trivially

vi

-RRB-

Since

the

delay

time

d

can

be

controlled

directly

by

the

AAWP

it

is

assumed

to

be

bounded

by

a

constant

Once

again

if

this

were

not

true

our

claim

that

an

AAWP

can

perform

arbitrarily

worse

than

Time

Warp

can

be

shown

trivially

Several

protocols

that

have

been

proposed

and

implemented

belong

to

the

class

of

AAWP?s

In

the

penalty

based

throttling

scheme

of

-LSB-

-RSB-

an

LP

that

has

been

rolling

back

excessively

is

made

to

block

for

some

period

of

time

This

decision

is

made

independently

of

other

LP?s

Similarly

in

Adaptive

Time

Warp

-LSB-

-RSB-

an

LP

may

decide

to

block

after

executing

an

event

based

on

local

history

and

statistical

estimation

In

-LSB-

-RSB-

LP?s

estimate

each

others

logical

clock

values

and

block

if

their

clock

value

differs

largely

from

that

of

another

LP

In

-LSB-

-RSB-

a

real-time

blocking

window

is

computed

each

time

an

LP

executes

an

event

and

the

LP

blocks

for

an

amount

of

time

equal

to

this

window

-LRB-

which

may

be

zero

-RRB-

Similarly

in

-LSB-

-RSB-

an

LP

blocks

probabilistically

for

some

amount

of

time

after

each

event

execution

The

new

class

of

adaptive

protocols

we

have

described

in

-LSB-

-RSB-

which

we

call

NPSI

adaptive

protocols

also

satisfy

the

AAWP

assumptions

Windowing

algorithms

in

which

the

windows

are

computed

individually

for

different

LP?s

-LRB-

such

as

Unified

Distributed

Simulation

system

-LSB-

-RSB-

and

Breathing

Time

Warp

-LSB-

-RSB-

-RRB-

are

AAWP?s

as

well

since

the

LP?s

wait

when

they

reach

the

ceilings

of

their

independent

windows

Note

global

windowing

algorithms

do

not

fit

the

AAWP

model

since

the

global

window

forces

all

LP?s

to

synchronize

before

any

of

them

can

proceed

Protocols

in

which

optimism

is

limited

based

on

spatial

clustering

-LSB-

-RSB-

also

satisfy

our

assumptions

since

waiting

may

be

required

at

cluster

boundaries

where

events

are

exchanged

conservatively

and

the

clusters

operate

asynchronously

Time

Warp

outperforms

AAWP?s

We

show

by

example

that

an

AAWP

can

take

arbitrarily

longer

than

Time

Warp

to

complete

a

simulation

The

intuition

behind

the

example

is

this

on

the

one

hand

it

is

possible

for

Time

Warp

simulations

to

execute

very

efficiently

with

few

rollbacks

on

the

other

hand

it

is

also

possible

for

a

Time

Warp

simulation

to

generate

many

false

events

and

consequent

rollbacks

which

can

degrade

its

performance

severely

-LSB-

-RSB-

Errors

in

adaptive

decisions

regarding

when

and

how

long

to

wait

can

cause

a

Time

Warp

execution

to

move

from

the

former

category

to

the

latter

In

the

example

here

we

show

a

situation

where

the

AAWP

induces

a

false

rollback

chain

that

delays

the

committing

of

an

event

-LRB-

relative

to

the

Time

Warp

execution

-RRB-

by

at

least

an

amount

of

time

proportional

to

the

length

of

the

rollback

chain

By

arguing

that

this

rollback

chain

can

be

arbitrarily

long

we

show

that

the

committing

of

an

event

can

be

delayed

arbitrarily

Consider

the

Time

Warp

execution

shown

in

Figure

The

x-axis

denotes

advance

of

wall-clock

time

while

the

y-axis

denotes

the

different

LP?s

The

numbers

below

the

events

are

their

respective

timestamps

A

dashed

arrow

indicates

the

causal

dependence

of

an

event

on

a

message

-LRB-

i.e.

the

event

at

the

head

of

the

arrow

was

scheduled

by

the

arrival

of

the

message

at

the

tail

of

the

arrow

-RRB-

The

two

important

events

to

note

in

this

execution

are

-LRB-

i

-RRB-

the

event

with

timestamp

at

which

is

causally

dependent

on

a

message

from

that

arrives

just

in

time

to

be

executed

by

and

-LRB-

ii

-RRB-

the

event

with

timestamp

at

which

is

the

one

whose

committing

execution

will

be

delayed

due

to

an

erroneous

waiting

decision

Figure

shows

an

execution

of

the

same

simulation

as

in

Figure

using

an

AAWP

For

simplicity

we

assume

the

initial

conditions

for

the

two

executions

are

the

same

except

for

one

difference

both

and

wait

for

some

time

at

the

beginning

of

the

portion

of

the

148 152 158

156 150 156

140 165 170 175

Wall

clock

time

Figure

Time

Warp

execution

Event

execution

committed

Event

execution

uncommitted

Rollback

Adaptive

waiting

Positive

message

transmission

Causality

Antimessage

149 155 157 160

148 152 155 158

156 150 155 156

140 165 170 175

O

-LRB-

length

of

false

rollback

chain

-RRB-

Wall

clock

time

Figure

False

rollback

chain

with

cycle

due

to

incorrect

waiting

MF

ED

ED

execution

depicted

Due

to

what

will

turn

out

to

be

an

error

in

the

decision

process

delays

longer

than

As

a

result

is

ready

to

execute

an

event

before

the

message

from

arrives

and

causes

the

event

with

timestamp

to

be

scheduled

executes

its

next

scheduled

event

-LRB-

with

timestamp

-RRB-

and

sends

a

message

to

Since

we

assume

aggressive

rollback

this

event

is

a

false

one

as

it

is

executed

out

of

order

When

the

message

from

arrives

later

this

false

event

is

rolled

back

and

an

antimessage

is

sent

to

However

the

message

sent

to

by

the

false

event

has

already

intiated

a

chain

of

false

events

The

antimessage

starts

a

chain

of

rollbacks

and

antimessages

that

follows

close

behind

the

chain

of

false

events

Regarding

these

two

chains

we

observe

the

following

i

-RRB-

It

is

possible

that

the

rollback

chain

catches

up

with

the

false

event

chain

immediately

and

thus

terminates

both

chains

However

the

case

that

is

relevant

to

this

discussion

is

the

one

shown

in

Figure

where

the

rollback

chain

does

not

catch

up

with

the

event

chain

until

the

latter

reaches

and

initiates

an

unnecessary

rollback

This

scenario

is

feasible

-LSB-

-RSB-

ii

-RRB-

The

chains

by

themselves

may

be

harmless

the

problem

arises

from

the

fact

that

the

final

false

message

that

arrives

at

-LRB-

marked

MF

in

Figure

-RRB-

has

timestamp

which

is

smaller

than

the

timestamp

of

the

second

event

at

-LRB-

marked

ED

in

Figure

-RRB-

Thus

MF

rolls

back

the

first

execution

of

ED

even

though

that

execution

was

correct

and

could

have

been

committed

ED

is

reexecuted

after

the

false

rollback

completes

Therefore

the

committing

execution

ED

is

delayed

by

at

least

an

amount

of

time

proportional

to

the

length

of

the

false

rollback

chain

iii

-RRB-

Each

of

the

two

chains

has

a

cycle

in

it

involving

and

While

we

have

shown

only

one

iteration

of

this

cycle

it

is

possible

to

have

an

arbitrary

number

of

iterations

i.e.

the

chains

could

revisit

these

LP?s

an

arbitrary

number

of

times

before

reaching

iv

-RRB-

For

the

false

event

chain

to

delay

the

committing

execution

of

ED

the

timestamp

of

MF

must

be

smaller

than

that

of

ED

Since

we

assume

a

correct

implementation

of

the

underlying

Time

Warp

protocol

and

the

application

it

follows

that

logical

time

must

advance

eventually

as

we

traverse

the

false

event

chain

Therefore

an

arbitrarily

long

chain

would

require

that

the

timestamp

of

ED

also

be

arbitrarily

large

-LRB-

we

disregard

pathological

situations

such

as

when

the

false

chain

makes

diminishing

advances

in

logical

time

such

that

the

timestamp

of

MF

is

bounded

such

situations

reflect

a

flaw

in

the

implementation

the

physical

system

being

simulated

can

not

make

diminishing

progress

in

real

time

-RRB-

The

larger

the

timestamp

of

ED

the

less

likely

that

its

first

execution

is

on

the

critical

path

of

the

simulation

since

other

LP?s

are

farther

behind

in

logical

time

Thus

while

it

is

possible

for

the

false

chains

to

be

arbitrarily

long

the

probability

that

these

chains

are

damaging

to

the

simulation

decreases

with

the

length

of

the

chain

However

the

probability

is

not

zero

we

can

imagine

a

simulation

where

ED

marks

the

transition

to

a

new

phase

of

simulated

time

i.e.

the

entire

simulation

makes

a

jump

in

logical

time

to

the

next

phase

of

activity

If

so

the

delaying

of

ED

could

cause

the

entire

phase

transition

to

be

delayed

v

-RRB-

It

is

possible

to

have

very

long

false

chains

which

do

not

span

much

logical

time

as

shown

in

Figure

Here

logical

time

does

not

increase

in

an

iteration

of

the

cycle

in

the

false

chain

it

increases

only

across

iterations

Thus

the

timestamp

of

MF

is

small

but

the

chains

are

long

vi

-RRB-

Finally

even

if

the

lengths

of

the

chains

are

bounded

-LRB-

by

some

means

-RRB-

it

is

possible

to

have

an

arbitrary

number

of

instances

of

such

chains

in

the

course

of

a

single

simulation

run

In

summary

we

have

shown

by

example

that

the

committing

execution

of

an

event

may

be

delayed

-LRB-

relative

to

the

Time

Warp

execution

-RRB-

by

an

arbitrary

amount

of

time

due

to

false

events

and

rollbacks

created

by

errors

in

the

waiting

decision

Lazy

cancellation

Since

the

events

in

the

rollback

chain

are

false

events

-LRB-

i.e.

they

should

not

have

been

generated

-RRB-

they

will

have

to

be

rolled

back

even

under

lazy

cancellation

Thus

the

example

holds

under

lazy

cancellation

as

well

In

fact

the

problem

is

aggravated

to

a

certain

extent

with

lazy

cancellation

because

the

initial

antimessage

generated

by

the

rollback

of

the

event

with

timestamp

at

will

be

delayed

until

crosses

time

after

executing

the

event

with

timestamp

Thus

the

physical

time

lag

between

the

two

chains

may

be

longer

allowing

the

false

event

chain

to

propagate

farther

Lazy

rollback

The

commitment

of

the

first

execution

of

ED

in

the

Time

Warp

execution

implies

that

the

event

with

timestamp

at

must

be

a

false

one

even

with

lazy

rollback

Thus

the

two

chains

will

be

generated

in

this

case

also

However

ED

may

not

be

rolled

back

when

MF

arrives

This

can

happen

due

to

one

of

the

following

-LRB-

i

-RRB-

MF

has

no

impact

on

the

state

in

which

ED

was

executed

or

-LRB-

ii

-RRB-

the

rollback

chain

arrives

at

while

executes

the

event

scheduled

by

MF

so

that

when

the

execution

is

complete

-LRB-

or

is

preempted

assuming

event

preemption

-RRB-

it

is

cancelled

immediately

and

the

operating

state

for

ED

is

not

affected

However

even

if

the

final

execution

of

ED

is

not

delayed

the

advance

of

GVT

beyond

will

be

delayed

by

at

least

the

amount

of

time

it

takes

for

the

chains

to

disappear

Since

GVT

is

the

commitment

horizon

for

events

-LSB-

-RSB-

the

committing

of

ED

can

be

delayed

arbitrarily

even

with

lazy

rollback

Pragmatic

issues

We

have

identified

a

general

class

of

adaptive

protocols

the

AAWP?s

and

shown

that

it

is

possible

for

Time

Warp

to

outperform

them

by

an

arbitrary

amount

Since

several

practical

protocols

belong

to

the

AAWP

class

this

counter-intuitive

result

suggests

that

AAWP?s

must

be

designed

with

care

Correspondingly

while

we

have

developed

several

very

effective

NPSI

adaptive

protocols

-LSB-

-RSB-

we

have

also

discovered

some

that

perform

no

better

than

Time

Warp

and

some

that

perform

worse

It

is

possible

to

modify

protocols

so

as

to

avoid

the

scenario

described

earlier

However

it

is

not

clear

whether

doing

so

will

be

beneficial

in

the

general

case

Moreover

since

there

may

be

other

scenarios

similar

to

the

one

we

have

described

establishing

a

property

of

AAWP?s

that

avoids

this

scenario

can

not

guarantee

that

Time

Warp

will

not

outperform

AAWP?s

by

more

than

a

constant

factor

NPSI

adaptive

protocols

We

describe

a

design

framework

for

a

new

class

of

adaptive

protocols

This

framework

will

be

used

in

section

to

define

a

specific

adaptive

protocol

that

outperforms

Time

Warp

by

a

factor

proportional

to

the

length

of

the

simulation

run

We

call

this

new

class

near-perfect

state

information

-LRB-

NPSI

-RRB-

adaptive

protocols

because

these

protocols

assume

the

availability

of

near-perfect

information

at

each

LP

about

the

state

of

the

system

at

little

or

no

cost

to

the

simulation

The

adaptive

waiting

decisions

of

LP?s

are

based

on

this

NPSI

In

practice

such

information

can

be

disseminated

using

a

high-speed

reduction

network

-LSB-

-RSB-

at

almost

no

cost

to

the

simulation

Our

studies

-LSB-

-RSB-

have

shown

that

such

a

network

can

disseminate

critical

information

to

the

LP?s

at

latencies

Global

virtual

time

is

the

minimum

of

all

logical

clocks

and

timestamps

of

any

messages

in

transit

-LSB-

-RSB-

that

are

two

or

three

orders

of

magnitude

smaller

than

typical

event

execution

times

-LRB-

i.e

microseconds

versus

milliseconds

-RRB-

We

have

designed

and

implemented

NPSI

protocols

over

a

prototype

reduction

network

with

very

encouraging

results

A

more

detailed

discussion

on

the

rationale

behind

NPSI

protocols

their

design

and

performance

can

be

found

in

-LSB-

-RSB-

There

are

two

phases

in

the

design

of

NPSI

adaptive

protocols

identifying

the

information

that

must

be

collected

dynamically

and

on

which

the

decision

to

limit

optimism

is

to

be

based

designing

the

mechanism

that

translates

the

collected

information

into

control

over

an

LP?s

aggressiveness

and

risk

The

framework

depicted

in

Figure

separates

these

phases

by

introducing

a

quantity

we

call

error

potential

-LRB-

EPi

-RRB-

associated

with

each

LPi

EPi

is

an

estimate

of

the

need

for

LPi

to

dcrease

its

optimism

The

mapping

translates

the

relevant

NPSI

to

a

value

of

EPi

The

NPSI

adaptive

protocol

keeps

EPi

up-to-date

for

each

LPi

as

the

simulation

progresses

by

evaluating

at

high

frequency

using

state

information

it

receives

from

the

feedback

system

dynamically

reflects

new

values

of

EPi

in

the

event

execution

and

communication

rates

Different

NPSI

adaptive

protocols

may

be

constructed

by

designing

the

mappings

and

Note

these

mappings

only

specify

if

an

LP

should

wait

and

how

long

it

should

wait

the

general

structure

of

NPSI

protocols

conforms

to

the

AAWP

model

in

section

AAWP?s

outperform

Time

Warp

We

show

by

example

that

Time

Warp

can

take

arbitrarily

longer

than

an

AAWP

to

complete

a

simulation

Our

approach

is

to

describe

the

execution

of

a

simulation

using

both

Time

Warp

and

a

specific

AAWP

We

show

that

Time

Warp

takes

an

amount

of

time

that

is

quadratic

in

the

amount

of

logical

time

simulated

while

the

AAWP

takes

linear

time

Since

the

difference

in

completion

times

is

not

bounded

by

a

Figure

Framework

for

NPSI

adaptive

protocols

Error

Potential

Event

processing

and

message

sending

System

State

constant

for

a

given

simulation

the

AAWP

outperforms

Time

Warp

arbitrarily

Physical

system

The

system

we

consider

for

simulation

was

described

previously

in

-LSB-

-RSB-

as

an

example

of

echoing

in

Time

Warp

We

refer

to

it

as

EchoSystem

It

consists

of

three

physical

processes

-LRB-

PP?s

-RRB-

A

B

and

C

with

the

communication

topology

shown

in

Figure

Upon

receiving

a

message

from

PPA

PPB

processes

it

and

sends

a

message

to

PPA

and

vice-versa

If

no

message

is

received

from

the

other

both

of

them

prepare

a

message

to

send

to

PPC

If

a

message

arrives

when

one

is

being

created

or

sent

to

PPC

that

sending

is

aborted

and

the

new

message

is

processed

Sending

and

receiving

of

messages

takes

no

time

Processing

a

message

from

PPA

-LRB-

PPB

-RRB-

takes

u

real

time

units

on

PPB

-LRB-

PPA

-RRB-

Preparation

of

a

message

to

PPC

takes

real

time

units

Suppose

at

time

PPA

receives

the

first

message

from

PPB

Then

it

may

be

verified

that

the

only

message

traffic

that

occurs

in

this

system

is

between

PPA

and

PPB

at

intervals

of

u

real

time

units

The

idle

periods

between

intervals

are

insufficient

to

build

a

message

to

PPC

Note

real

time

in

the

physical

system

corresponds

to

logical

time

in

the

simulator

For

the

simulator

we

assume

the

following

processing

a

message

between

PPA

and

PPB

takes

one

unit

of

wall-clock

time

-LRB-

including

sending

the

follow-on

message

-RRB-

preparing

a

message

to

PPC

takes

one

time

unit

and

sending

of

an

antimessage

also

takes

one

time

unit

LP?s

advance

their

simulation

clocks

to

the

timestamp

of

the

next

event

after

executing

that

event

We

assume

this

to

simplify

the

proofs

the

theorems

can

be

proven

even

if

it

is

assumed

that

logical

clocks

are

advanced

before

commencing

event

execution

LPC

is

assumed

to

perform

its

work

fast

enough

so

that

its

actions

are

irrelevant

to

the

discussions

and

proofs

sX

denotes

the

logical

clock

value

of

LPX

PPA

PPB

PPC

Figure

Physical

system

for

echoing

Time

Warp

execution

The

Time

Warp

execution

of

this

simulation

is

shown

in

Figure

The

x-axis

represents

wall-clock

time

The

numbers

-LRB-

in

multiples

of

u

-RRB-

at

the

junctions

of

the

LP

time

lines

and

the

unit

time

intervals

indicate

the

logical

time

to

which

the

LP?s

have

simulated

-LRB-

i.e.

the

logical

clock

value

of

the

LP

-RRB-

A

solid

arrow

indicates

a

message

transmission

while

a

dashed

arrow

indicates

an

antimessage

transmission

The

bold

lines

at

various

points

on

the

time

lines

of

LPA

and

LPB

indicate

rollback

From

the

picture

the

echoing

is

evident

immediately

in

the

fact

that

the

two

LP?s

roll

back

alternately

with

increasing

amplitudes

THEOREM

A

Time

Warp

execution

of

EchoSystem

takes

n

-LRB-

n

-RRB-

wall-clock

time

units

to

simulate

nu

units

of

logical

time

PROOF

The

theorem

is

proved

by

induction

on

the

amount

of

wall-clock

time

required

for

GVT

to

advance

from

logical

time

-LRB-

-RRB-

u

to

nu

Induction

hypothesis

It

takes

n

units

of

wall-clock

time

for

GVT

to

advance

from

logical

time

-LRB-

-RRB-

u

to

nu

Base

case

n

From

Figure

we

see

that

in

the

first

wall-clock

time

interval

-LRB-

i.e.

wall-clock

time

-LSB-

-RRB-

-RRB-

LPA

advances

sA

to

u

and

LPB

advances

sB

to

Thus

GVT

has

advanced

from

to

u

in

one

wall-clock

time

unit

and

the

hypothesis

holds

Induction

step

Assume

the

hypothesis

holds

for

the

logical

time

window

-LSB-

-LRB-

-RRB-

u

nu

-RRB-

Recall

that

the

message

traffic

in

the

physical

system

consists

only

of

the

single

message

being

exchanged

by

LPA

and

LPB

Thus

in

any

correct

simulation

of

this

system

GVT

advances

by

u

logical

time

units

each

time

an

LP

receives

this

message

processes

it

and

sends

it

back

During

this

process

the

actions

of

other

LP?s

can

not

affect

GVT

Without

loss

of

generality

assume

LPA

takes

n

wall-clock

time

units

to

advance

GVT

from

-LRB-

-RRB-

u

to

nu

-LRB-

by

induction

hypothesis

-RRB-

In

these

n

wallclock

time

units

LPB

will

send

n

messages

to

LPC

Thus

at

the

end

of

-LSB-

-LRB-

-RRB-

u

nu

-RRB-

LPA

has

just

sent

a

message

to

LPB

with

timestamp

nu

and

LPB

has

sent

n

false

messages

to

LPC

Therefore

LPB

takes

n

wall

clock

time

units

to

send

the

n

antimessages

to

LPC

and

one

wall-clock

time

unit

to

process

the

message

advance

sB

to

-LRB-

n

-RRB-

u

and

send

the

message

back

to

LPA

with

timestamp

-LRB-

n

-RRB-

u

At

this

point

GVT

will

have

advanced

to

logical

time

-LRB-

n

-RRB-

u

requiring

n

wall-clock

time

units

to

do

so

Thus

the

total

wall-clock

time

required

to

simulate

up

to

logical

time

nu

is

n

We

have

thus

shown

that

Time

Warp

takes

O

-LRB-

-RRB-

wallclock

time

to

simulate

the

specified

physical

system

where

n

is

a

measure

of

the

logical

time

span

of

the

simulation

run

AAWP

execution

We

describe

a

specific

NPSI

adaptive

protocol

based

on

the

design

framework

described

in

section

The

error

potential

-LRB-

EPi

-RRB-

of

each

LPi

is

given

by

the

following

where

GVTi

is

the

value

of

GVT

made

available

to

LPi

by

the

feedback

system

is

a

function

that

maps

EPi

into

a

wall-clock

time

delay

di

given

by

i

i

n

n

n

-LRB-

-RRB-

EPi

si

GVTi

where

MaxEPi

is

the

maximum

value

of

EPi

observed

thus

far

After

executing

an

event

LPi

re-computes

di

and

waits

for

di

units

of

wall-clock

time

before

proceeding

to

the

next

event

If

a

message

is

received

during

a

wait

period

that

will

cause

a

rollback

the

LP

aborts

the

waiting

and

proceeds

to

roll

back

Since

this

algorithm

is

a

variant

of

the

elastic

time

algorithm

-LRB-

ETA

-RRB-

described

in

-LSB-

-RSB-

we

will

refer

to

it

by

the

same

name

Figure

shows

the

execution

of

the

simulation

using

the

protocol

described

above

The

shaded

lines

represent

waiting

due

to

the

adaptive

protocol

It

is

evident

from

the

diagram

that

the

echoing

observed

under

Time

Warp

-LRB-

Figure

-RRB-

has

been

avoided

since

each

rollback

is

of

only

unit

length

di

EP

i

MaxEP

i

EPi

EPi

u

Figure

Echoing

in

Time

Warp

LPA

LPC

LPB

u

Wall-clock

time

Figure

Avoiding

echoing

with

adaptive

aggressiveness

LPA

LPC

LPB

u

u

EPA

MaxEPA

dA

EPB

MaxEPB

dB

Wall-clock

time

The

discussion

in

section

justifies

the

following

assumption

NPSI

Assumption

A

change

in

an

LP?s

logical

clock

value

is

reflected

in

the

values

of

GVT

visible

to

the

different

LP?s

in

a

fraction

of

the

time

it

takes

for

an

LP

to

execute

an

event

For

example

we

may

assume

this

latency

is

equal

to

wall-clock

time

units

since

an

LP

takes

one

wall-clock

time

unit

to

execute

an

event

THEOREM

An

execution

of

EchoSystem

using

ETA

takes

wall-clock

time

units

to

simulate

nu

units

of

logical

time

PROOF

The

proof

consists

of

induction

on

the

amount

of

wall-clock

time

required

for

GVT

to

advance

from

logical

time

-LRB-

-RRB-

u

to

nu

Induction

hypothesis

For

n

-LRB-

i

-RRB-

it

takes

units

of

wall-clock

time

for

GVT

to

advance

from

logical

time

-LRB-

-RRB-

u

to

nu

-LRB-

ii

-RRB-

MaxEPA

and

-LRB-

iii

-RRB-

MaxEPB

Base

case

n

Referring

to

Figure

at

wall-clock

time

LPA

is

at

logical

time

u

and

has

sent

a

message

to

LPB

while

LPB

is

at

logical

time

and

has

sent

a

false

message

to

LPC

Thus

GVT

u

at

wall-clock

time

The

message

from

LPA

causes

LPB

to

roll

back

to

logical

time

u

and

send

an

antimessage

to

LPC

requiring

one

wall-clock

time

unit

In

the

wall-clock

time

interval

-LSB-

-RRB-

LPB

processes

the

message

advances

sB

to

and

sends

a

message

with

timestamp

back

to

LPA

In

the

wall-clock

time

period

-LSB-

-RRB-

LPA

advances

sA

to

builds

a

message

and

sends

it

to

LPC

Since

LPB

rolls

sB

back

to

u

just

after

wall-clock

time

by

the

NPSI

assumption

at

wall-clock

time

we

have

sA

GVTA

u

and

therefore

EPA

Since

MaxEPA

was

initially

MaxEPA

giving

dA

Thus

LPA

waits

during

the

wall-clock

time

period

-LSB-

-RRB-

Consequently

at

wall-clock

time

GVT

MaxEPA

and

MaxEPB

Induction

step

Assume

the

hypothesis

holds

for

the

logical

time

window

-LSB-

-LRB-

-RRB-

u

nu

-RRB-

As

in

the

proof

of

Theorem

the

argument

is

made

that

in

any

correct

simulation

of

the

specified

physical

system

-LRB-

i

-RRB-

GVT

advances

by

u

logical

time

units

each

time

an

LP

receives

a

message

processes

it

and

sends

it

back

and

-LRB-

ii

-RRB-

the

actions

of

other

LP?s

during

this

period

can

not

influence

this

GVT

advance

Without

loss

of

generality

assume

LPA

takes

wall-clock

time

units

-LRB-

by

hypothesis

-RRB-

say

-LSB-

i

-RRB-

and

-LSB-

i

i

-RRB-

to

advance

GVT

from

-LRB-

-RRB-

u

to

nu

and

send

a

message

with

timestamp

nu

to

LPB

In

-LSB-

i

-RRB-

LPB

advances

sB

from

-LRB-

-RRB-

u

to

-LRB-

n

-RRB-

u

and

sends

a

message

to

LPC

Since

LPA

sent

a

message

with

timestamp

nu

to

LPB

at

the

end

of

-LSB-

i

i

-RRB-

sA

must

be

-LRB-

-RRB-

u

at

the

end

of

-LSB-

i

-RRB-

Further

LPA

received

a

message

from

LPB

at

the

beginning

of

-LSB-

i

-RRB-

-LRB-

because

LPA

advances

GVT

-RRB-

This

implies

LPA

could

not

have

built

and

sent

a

message

to

LPC

during

-LSB-

i

-RRB-

Therefore

LPA

must

have

rolled

back

during

-LSB-

i

-RRB-

This

means

sA

must

have

been

rolled

back

to

-LRB-

-RRB-

u

at

the

beginning

of

-LSB-

i

-RRB-

By

the

NPSI

assumption

at

the

end

of

-LSB-

i

-RRB-

GVTB

-LRB-

-RRB-

u.

Thus

sB

-LRB-

n

-RRB-

u

EPB

MaxEPB

-LRB-

by

hypothesis

-RRB-

and

dB

Consequently

LPB

waits

during

-LSB-

i

i

-RRB-

At

the

end

of

-LSB-

i

i

-RRB-

GVT

nu

LPA

has

sent

a

message

with

timestamp

nu

to

LPB

and

LPB

has

sent

one

false

message

to

LPC

The

message

from

LPA

at

the

end

of

-LSB-

i

i

-RRB-

causes

LPB

to

roll

back

and

send

one

antimessage

to

LPC

-LRB-

requiring

one

wall-clock

time

unit

-RRB-

LPB

uses

another

wall-clock

time

unit

to

process

the

message

advance

sB

to

-LRB-

n

-RRB-

u

and

send

a

message

with

timestamp

-LRB-

n

-RRB-

u

to

LPA

Thus

after

two

wall

clock

time

units

GVT

is

advanced

from

nu

to

-LRB-

n

-RRB-

u

Since

LPB

controls

the

advance

of

GVT

in

these

two

wall-clock

time

units

EPB

and

consequently

MaxEPB

The

actions

of

LPA

in

this

period

are

exactly

the

same

as

those

of

LPB

in

the

interval

-LSB-

-LRB-

-RRB-

u

-LRB-

i

-RRB-

u

-RRB-

described

earlier

It

follows

that

MaxEPA

also

equals

The

total

wall-clock

time

required

to

simulate

nu

units

of

logical

time

is

therefore

given

by

the

sum

of

the

wall-clock

times

required

to

simulate

the

windows

-LSB-

u

-RRB-

and

-LSB-

u

nu

-RRB-

n

We

have

thus

shown

that

ETA

takes

O

-LRB-

n

-RRB-

wallclock

time

to

simulate

EchoSystem

where

n

is

a

measure

of

the

logical

time

span

of

the

simulation

run

The

corresponding

completion

time

with

Time

Warp

is

O

-LRB-

-RRB-

Since

the

difference

in

completion

times

is

not

bounded

by

a

constant

for

a

given

simulation

the

adaptive

protocol

outperforms

Time

Warp

arbitrarily

Pragmatic

issues

We

have

described

a

specific

AAWP

the

elastic

time

algorithm

-LRB-

ETA

-RRB-

and

shown

that

it

can

outperform

Time

Warp

arbitrarily

While

the

simulation

for

which

this

phenomenon

is

observed

is

somewhat

contrived

i

n

n

ETA

is

not

In

-LSB-

-RSB-

we

describe

an

experiment

on

a

four-processor

Time

Warp

implementation

using

a

workload

very

similar

to

EchoSystem

The

experimental

set-up

included

a

prototype

high-speed

reduction

network

which

was

used

to

provide

near-perfect

state

information

to

implement

ETA

In

conformance

with

our

analysis

here

we

observed

that

the

speedup

of

ETA

over

Time

Warp

increased

with

the

logical

time

span

of

the

simulation

i.e.

the

larger

the

maximum

simulated

time

the

larger

the

speedup

Summary

The

lack

of

consistent

performance

with

the

two

traditional

approaches

to

synchronization

in

parallel

discrete

event

simulations

-LRB-

conservative

and

optimistic

-RRB-

has

led

to

a

number

of

hybrid

approaches

Many

of

these

have

indeed

demonstrated

better

performance

under

test

cases

An

adaptive

protocol

one

that

modifies

itself

in

response

to

changes

in

the

simulation

appears

to

be

the

most

likely

to

perform

well

with

a

wide

range

of

simulations

However

there

have

been

no

analytical

studies

comparing

the

performance

of

adaptive

protocols

with

that

of

traditional

protocols

We

present

the

first

known

analytical

comparison

of

adaptive

protocols

with

Time

Warp

We

demonstrate

that

it

is

possible

for

Time

Warp

to

arbitrarily

outperform

a

class

of

adaptive

protocols

we

call

asynchronous

adaptive

wait

protocols

-LRB-

AAWP?s

-RRB-

Protocols

in

this

class

control

aggressiveness

and

risk

by

introducing

independently

controlled

delays

at

the

LP?s

This

class

is

general

enough

to

include

many

practical

protocols

Conversely

we

describe

a

member

of

a

new

class

of

adaptive

protocols

called

NPSI

adaptive

protocols

-LRB-

which

are

a

subset

of

AAWP?s

-RRB-

and

present

an

example

in

which

this

protocol

outperforms

Time

Warp

arbitrarily

Thus

while

adaptive

limiting

of

optimism

appears

to

enhance

performance

in

practice

our

study

shows

that

care

must

be

taken

in

the

design

of

AAWP?s

since

incorrect

adaptive

decisions

can

lead

to

arbitrarily

worse

performance

than

Time

Warp

Acknowledgments

We

are

grateful

to

Bronis

de

Supinski

for

proof

reading

the

paper

and

for

his

insightful

comments

We

thank

Craig

Williams

for

her

suggestions

This

work

was

supported

by

Mystech

Inc.

-LRB-

Academic

Affiliates

Program

-RRB-

References

-LSB-

-RSB-

Ball

D.

and

Hoyt

S.

The

adaptive

TimeWarp

concurrency

control

algorithm

Proceedings

of

the

SCS

Multiconference

on

Distributed

Simulation

January

-LSB-

-RSB-

Distributed

Simulation

A

case

study

in

the

design

and

verification

of

distributed

programs

IEEE

Transactions

on

Software

Engineering

Vol

No.

-LRB-

September

-RRB-

-LSB-

-RSB-

Ferscha

A.

and

Tripathi

S.K.

Parallel

and

distributed

simulation

of

discrete

event

systems

University

of

Maryland

at

College

Park

Technical

Report

number

August

-LSB-

-RSB-

Fujimoto

R.M.

Parallel

discrete

event

simulation

Communications

of

the

ACM

Vol

No.

October

-LSB-

-RSB-

Gimarc

R.L.

Distributed

simulation

using

hierarchical

rollback

Proceedings

of

the

Winter

Simulation

Conference

-LSB-

-RSB-

Hamnes

D.O.

and

Tripathi

A.

Evaluation

of

a

local

adaptive

protocol

for

distributed

discrete

event

simulation

Proceedings

of

the

International

Conference

on

Parallel

Processing

August

Vol

III

-LSB-

-RSB-

Jefferson

D.R.

Virtual

time

ACM

Transactions

on

Programming

Languages

and

Systems

Vol

No.

July

-LSB-

-RSB-

Lipton

R.J.

and

Mizell

D.W.

Time

Warp

vs.

Chandy-Misra

A

worst-case

comparison

Proceedings

of

the

SCS

Multiconference

on

Distributed

Simulation

January

-LSB-

-RSB-

Lubachevsky

B.

Weiss

A.

and

Shwartz

A.

Rollback

sometimes

works

if

filtered

Proceedings

of

the

Winter

Simulation

Conference

December

-LSB-

-RSB-

Lubachevsky

B.

Weiss

A.

and

Shwartz

A.

An

analysis

of

rollback-based

simulation

ACM

Transactions

on

Modeling

and

Computer

Simulation

Vol

No.

April

-LSB-

-RSB-

Madisetti

V.K.

Randomized

algorithms

for

self-synchronization

Private

communication

-LSB-

-RSB-

McAffer

J.

A

unified

distributed

simulation

system

Proceedings

of

the

Winter

Simulation

Conference

-LSB-

-RSB-

Nicol

D.M.

and

Reynolds

P.F.

Jr.

Optimal

dynamic

remapping

of

parallel

computations

IEEE

Transactions

on

Computers

Vol

No.

February

-LSB-

-RSB-

Pancerella

C.M.

and

Reynolds

P.F.

Jr.

Disseminating

critical

target-specific

synchronization

information

in

parallel

discrete

event

simulations

Proceedings

of

the

Workshop

on

Parallel

and

Distributed

Simulation

May

-LSB-

-RSB-

Rajaei

H.

Ayani

R.

and

Thorelli

L-E.

The

Local

Time

Warp

approach

to

parallel

simulation

Proceedings

of

the

Workshop

on

Parallel

and

Distributed

Simulation

May

-LSB-

-RSB-

Reiher

P.L.

and

Jefferson

D.

Limitation

of

optimism

in

the

Time

Warp

operating

system

Proceedings

of

the

Winter

Simulation

Conference

December

-LSB-

-RSB-

Reynolds

P.F.

Jr.

A

shared

resource

algorithm

for

distributed

simulation

Proceedings

of

the

Annual

Symposium

on

Computer

Architecture

April

-LSB-

-RSB-

Reynolds

P.F.

Jr.

A

spectrum

of

options

for

parallel

simulation

Proceedings

of

the

Winter

Simulation

Conference

December

-LSB-

-RSB-

Srinivasan

S.

and

Reynolds

P.F.

Jr.

Noninterfering

GVT

computation

via

asynchronous

global

reductions

Proceedings

of

the

Winter

Simulation

Conference

December

-LRB-

available

by

anonymous

FTP

from

-RRB-

-LSB-

-RSB-

Srinivasan

S.

and

Reynolds

P.F.

Jr.

NPSI

adaptive

synchronization

algorithms

for

PDES

Technical

report

number

Department

of

Computer

Science

University

of

Virginia

November

-LRB-

submitted

to

the

Winter

Simulation

Conference

available

by

anonymous

FTP

from

-RRB-

-LSB-

-RSB-

Steinman

J.S.

Breathing

Time

Warp

Proceedings

of

the

Workshop

on

Parallel

and

Distributed

Simulation

May

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O B I O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O

adaptive protocols
elastic time algorithm
parallel simulation
time warp

limited optimism
modeling methodology
reduction networks