Proceedings

of

the

Annual

Simulation

Symposium

-LRB-

to

appear

-RRB-

A

Simulation

Study

of

Replication

Control

Protocols

Using

Volatile

Witnesses

Perry

K.

Sloope

Jehan-Franc

ois

Pa?ris

Darrell

D.E.

Long

Department

of

Computer

Science

Computer

and

Information

Sciences

University

of

Houston

University

of

California

Houston

TX

Santa

Cruz

CA

Abstract

Voting

protocols

guarantee

the

consistency

of

replicated

data

objects

by

disallowing

all

access

requests

that

can

not

gather

a

sufficient

quorum

of

replicas

The

performance

of

voting

protocols

can

be

greatly

enhanced

by

adding

to

the

replicas

small

independent

entities

that

hold

no

data

but

can

attest

to

the

state

of

the

replicated

data

object

It

has

been

recently

proposed

to

store

these

witnesses

in

volatile

storage

Volatile

witnesses

repond

faster

to

write

requests

than

those

stored

in

stable

storage

They

can

also

be

more

easily

regenerated

as

many

local

area

networks

contain

a

majority

of

diskless

sites

We

present

a

simulation

study

of

the

availability

afforded

by

two

voting

protocols

using

volatile

witnesses

and

investigate

the

impact

of

access

rates

network

topology

and

witness

placement

on

the

availability

of

the

replicated

data

Introduction

Many

distributed

systems

maintain

multiple

copies

of

some

critical

data

either

as

protection

against

equipment

failures

or

to

improve

read

access

times

Managing

replicated

data

presents

a

difficult

challenge

as

site

failures

and

network

malfunctions

are

likely

to

result

in

inconsistent

updates

Special

replication

control

protocols

have

been

proposed

to

prevent

this

occurrence

These

replication

control

protocols

define

a

new

abstraction

that

has

the

same

semantics

as

an

unreplicated

instance

of

the

object

being

replicated

while

providing

a

higher

resiliency

Replication

control

protocols

vary

greatly

in

their

complexity

the

number

of

messages

sent

and

the

level

of

protection

they

provide

against

site

failures

and

network

malfunctions

As

a

result

the

evaluation

of

their

performance

has

become

an

area

of

great

practical

interest

In

most

cases

the

most

important

aspect

of

this

performance

is

the

availability

of

the

replicated

data

object

managed

by

the

protocol

The

availability

of

a

replicated

data

object

represents

the

steady-state

probability

that

the

object

is

available

at

any

given

moment

Several

techniques

have

been

used

to

evaluate

the

availability

of

replicated

data

Combinatorial

models

are

very

simple

to

use

-LSB-

-RSB-

but

can

not

represent

complex

recovery

modes

like

those

found

in

voting

protocols

with

witnesses

-LSB-

-RSB-

and

dynamic

voting

protocols

-LSB-

-RSB-

Stochastic

models

have

been

extensively

used

to

study

replication

protocols

-LSB-

-RSB-

but

suffer

from

two

important

limitations

First

stochastic

models

quickly

become

intractable

unless

all

failure

and

repair

processes

have

exponential

distributions

Second

stochastic

models

do

not

describe

communication

failures

well

since

the

number

of

distinct

states

in

a

model

increases

exponentially

with

the

number

of

failure

modes

being

considered

Discrete

event

simulation

does

not

suffer

from

these

limitations

Simulation

models

allow

for

the

relaxation

of

most

assumptions

that

are

required

for

stochastic

models

They

can

also

represent

systems

with

communication

failures

For

all

of

its

advantages

simulation

has

one

major

disadvantage

it

provides

only

numerical

results

This

makes

it

more

difficult

to

predict

how

the

modeled

system

would

behave

when

some

of

its

parameters

are

modified

because

the

simulation

must

be

run

again

when

a

parameter

is

changed

We

present

a

simulation

study

of

the

performance

of

two

new

voting

protocols

that

use

regenerable

volatile

witnesses

Witnesses

-LSB-

-RSB-

are

small

entities

that

keep

track

of

the

state

of

a

replicated

data

object

but

hold

no

data

They

are

used

by

replication

control

protocols

to

reduce

the

number

of

replicas

required

to

achieve

a

given

level

of

availability

-LSB-

-RSB-

There

have

been

recent

proposals

to

store

these

witnesses

in

volatile

storage

-LSB-

-RSB-

and

to

regenerate

witnesses

that

fail

instead

of

waiting

for

them

to

recover

-LSB-

-RSB-

Our

study

completes

the

stochastic

analyses

found

in

-LSB-

-RSB-

and

-LSB-

-RSB-

by

providing

availability

data

on

configurations

of

replicated

data

objects

that

could

not

be

analyzed

using

the

stochastic

approach

The

remainder

of

this

paper

is

organized

as

follows

Section

two

introduces

voting

protocols

with

volatile

witnesses

Section

three

discusses

our

simulation

model

and

section

four

presents

our

results

Finally

our

conclusions

are

presented

in

section

five

Voting

protocols

with

volatile

witnesses

Majority

consensus

voting

-LRB-

MCV

-RRB-

-LSB-

-RSB-

ensures

the

consistency

of

replicated

data

objects

by

requiring

that

all

read

and

write

requests

collect

the

votes

of

a

majority

of

replicas

MCV

offers

the

important

advantage

of

operating

correctly

in

the

presence

of

network

partitions

This

robustness

comes

at

a

price

MCV

requires

at

least

three

replicas

to

guarantee

full

access

to

the

data

in

the

presence

of

a

single

site

failure

Voting

with

witnesses

-LSB-

-RSB-

reduces

the

number

of

replicas

necessary

to

achieve

a

given

level

of

fault-tolerance

by

replacing

some

replicas

with

witnesses

that

hold

no

data

but

can

attest

to

the

state

of

the

replicated

data

object

Storage

requirements

can

be

significantly

reduced

since

two

replicas

and

one

witness

provide

almost

the

same

availability

as

three

complete

replicas

while

saving

of

the

storage

cost

Volatile

witnesses

are

witnesses

that

are

stored

in

volatile

memory

and

so

can

be

placed

on

diskless

sites

Volatile

witnesses

have

one

major

limitation

they

can

not

recover

on

their

own

from

a

site

failure

and

will

need

to

read

the

current

state

of

the

replicated

data

object

from

a

valid

quorum

of

witnesses

and

replicas

The

recovery

process

associated

with

each

site

that

holds

one

or

more

volatile

witnesses

thus

needs

to

put

its

recovering

volatile

witnesses

in

a

state

of

temporary

amnesia

and

to

prevent

them

from

participating

in

quorums

until

they

can

read

the

current

status

of

the

replicated

data

object

Because

of

their

very

small

size

new

volatile

witnesses

can

be

created

inexpensively

and

stored

on

any

site

in

the

network

Voting

with

regenerable

volatile

witnesses

-LSB-

-RSB-

takes

advantage

of

this

property

and

replaces

failed

witnesses

instead

of

waiting

for

the

failed

site

to

recover

This

technique

provides

protection

similar

to

additional

witnesses

but

at

a

much

lower

cost

In

this

study

we

investigate

two

replication

control

protocols

using

regenerable

volatile

witnesses

The

first

protocol

is

a

majority

consensus

voting

protocol

where

some

of

the

replicas

are

replaced

by

regenerable

volatile

witnesses

Regeneration

is

attempted

every

time

the

protocol

detects

that

one

or

more

witnesses

does

not

respond

to

a

quorum

call

Hence

frequent

accesses

ensure

a

full

complement

of

witnesses

and

improve

the

availability

of

the

replicated

object

Accomplishing

a

regeneration

requires

a

majority

of

the

replicas

and

witnesses

Once

a

quorum

has

been

formed

spare

sites

are

selected

and

new

volatile

witnesses

are

created

Inconsistencies

could

occur

if

the

replaced

witnesses

could

vote

again

To

prevent

this

we

add

a

generation

number

-LSB-

-RSB-

to

all

voting

entities

This

generation

number

is

incremented

each

time

regeneration

occurs

When

a

quorum

call

is

made

the

maximum

generation

number

is

calculated

Witnesses

that

do

not

have

the

maximum

generation

number

are

excluded

from

the

vote

A

quorum

is

present

if

either

a

majority

of

the

votes

were

collected

and

at

least

one

current

replica

was

present

or

all

replicas

responded

to

the

quorum

call

The

second

condition

is

necessary

to

guarantee

that

a

quorum

can

be

found

in

the

event

that

all

witnesses

have

failed

and

the

replicas

do

not

constitute

a

majority

quorum

The

second

protocol

we

investigated

is

the

optimistic

dynamic

voting

protocol

that

was

proposed

by

Pa?ris

and

Long

in

-LSB-

-RSB-

Unlike

MCV

and

other

static

voting

protocols

dynamic

voting

protocols

exclude

from

quorum

computations

all

replicas

and

witnesses

that

have

become

unreachable

until

they

again

become

reachable

and

can

be

reintegrated

into

the

current

majority

-LSB-

-RSB-

The

optimistic

dynamic

voting

protocol

uses

sets

of

site

names

called

partition

sets

to

record

the

identity

of

the

replicas

and

witnesses

that

belong

to

the

current

majority

These

partition

sets

are

maintained

at

each

site

holding

a

replica

or

a

witness

The

partition

set

for

a

site

contains

the

sets

of

replicas

and

witnesses

that

participated

in

the

last

successful

operation

that

included

that

site

These

partition

sets

are

recalculated

whenever

the

data

are

accessed

or

a

site

recovers

from

a

failure

In

addition

to

these

partition

sets

each

site

holding

a

replica

or

a

witness

maintains

a

version

number

and

an

operation

number

Version

numbers

are

incremented

every

time

the

replicated

data

is

modified

while

operation

numbers

are

incremented

every

time

the

partion

sets

are

modified

This

is

similar

to

the

information

required

by

the

original

optimistic

dynamic

voting

protocol

-LSB-

-RSB-

Unlike

the

MCV

protocol

the

ODV

protocol

uses

a

two-tier

voting

strategy

to

minimize

the

impact

of

witness

failures

on

the

availability

of

the

replicated

data

objects

A

quorum

is

present

if

any

of

the

following

four

conditions

holds

The

set

of

responding

replicas

constitutes

a

majority

of

replicas

in

the

previous

quorum

The

set

of

responding

replicas

contains

exactly

half

of

the

replicas

from

the

previous

quorum

and

a

majority

of

the

current

witnesses

also

respond

The

set

of

responding

replicas

contains

exactly

half

of

the

replicas

from

the

previous

quorum

and

exactly

half

of

the

current

witnesses

respond

In

this

case

a

quorum

exists

if

the

highest

ranking

witnesses

in

some

total

ordering

is

present

There

were

no

witnesses

present

in

the

previous

quorum

exactly

one

half

of

the

replicas

in

the

previous

quorum

have

responded

and

the

set

of

responding

replica

includes

the

highest

ranking

replica

in

some

total

ordering

of

the

previous

quorum

As

in

the

MCV

protocol

regeneration

is

attempted

every

time

the

protocol

detects

that

one

or

more

witnesses

do

not

respond

to

a

quorum

call

Generation

numbers

are

not

required

as

witnesses

that

do

not

respond

to

a

quorum

call

are

automatically

expelled

from

the

current

majority

The

simulation

model

Most

studies

of

replicated

data

availability

have

depended

on

probabilistic

models

to

evaluate

the

availability

of

replica

control

protocols

These

models

do

not

generally

consider

the

effect

of

network

partitioning

because

of

the

enormous

complexity

that

would

be

involved

As

a

result

the

data

that

they

present

are

for

ideal

environments

that

are

unlikely

to

exist

under

actual

conditions

We

use

discrete

event

simulation

to

observe

the

behavior

of

two

replica

control

protocols

under

more

realistic

conditions

We

also

develop

a

heuristic

for

witness

placement

that

improves

availability

by

considering

the

effect

that

network

partitioning

will

have

on

the

protocols

Many

parameters

can

effect

the

availability

of

replicated

data

Not

all

sites

in

a

network

will

be

equally

reliable

The

mean

time

to

fail

-LRB-

MTTF

-RRB-

and

mean

time

to

repair

-LRB-

MTTR

-RRB-

will

vary

among

the

sites

as

will

the

types

of

failures

A

failure

due

to

software

errors

will

involve

a

simple

reboot

of

the

machine

taking

only

minutes

while

a

hardware

failure

may

result

in

a

site

being

down

for

hours

or

even

days

Some

sites

may

be

taken

down

periodically

for

maintenance

while

others

may

never

be

maintained

until

they

fail

Periodic

installation

of

new

software

may

result

in

one

or

more

sites

being

removed

from

general

use

for

a

few

hours

Simulation

models

that

consider

these

factors

have

been

developed

In

-LSB-

-RSB-

a

simulation

study

of

voting

protocols

based

on

a

network

existing

then

at

the

UCSD

computer

science

department

was

presented

Our

simulation

models

are

equally

capable

of

modeling

all

of

these

possible

failure

modes

and

have

the

additional

capability

to

simulate

arbitrary

network

configurations

-LSB-

-RSB-

The

modeled

networks

are

assumed

to

be

collections

of

carrier-sense

segments

or

token

rings

linked

together

by

gateway

sites

or

repeaters

Site

failures

are

assumed

to

be

fail-stop

-LSB-

-RSB-

ceasing

operations

immediately

if

not

operating

correctly

The

network

may

be

partitioned

if

a

gateway

site

or

repeater

fails

but

sites

on

the

same

carrier-sense

segment

or

token

ring

always

remain

capable

of

communicating

with

each

other

Messages

can

be

lost

due

to

gateway

failures

but

messages

that

reach

their

destination

are

always

delivered

intact

in

the

order

they

were

sent

All

sites

are

capable

of

storing

a

replicated

data

object

and

manipulating

the

data

Users

are

assumed

to

be

able

to

access

the

replicated

data

object

from

any

point

in

the

network

and

are

modeled

as

a

single

process

Hence

the

replicated

data

are

assumed

to

be

available

as

long

as

they

can

be

accessed

from

at

least

one

site

in

the

network

Jajodia

and

Mutchler

have

recently

introduced

another

definition

of

data

availability

which

we

will

call

user

availability

-LSB-

-RSB-

They

define

the

availability

of

replicated

data

as

the

average

fraction

of

the

replicas

that

remain

accessible

at

equilibrium

User

availability

is

a

good

index

of

the

performance

of

consistency

protocols

distributed

over

long-haul

networks

If

a

replicated

object

is

not

accessible

from

a

local

site

access

at

a

remote

site

is

not

possible

due

to

the

sheer

distances

involved

It

is

a

less

accurate

measure

of

the

performance

of

protocols

distributed

over

local-area

networks

since

it

is

highly

likely

that

users

unable

to

reach

one

server

will

be

able

to

connect

to

another

one

General

organization

The

models

for

the

two

replication

control

policies

were

programmed

in

SIMSCRIPT

II

and

run

on

a

Sun

The

process

approach

was

selected

since

it

can

describe

the

network

sites

and

the

user

as

independent

processes

Information

on

the

mean

availability

and

unavailability

of

the

replicated

data

were

collected

using

standard

SIM

SCRIPT

II

statistical

routines

All

simulations

were

started

with

all

sites

operating

and

a

time-to-steady-state

interval

of

days

these

were

considered

acceptable

by

noting

the

time

spent

in

each

state

and

by

examining

graphs

of

the

measured

values

in

the

early

history

of

the

simulations

The

simulations

were

run

long

enough

to

establish

reasonable

confidence

intervals

for

all

replicated

data

object

unavailabilities

To

analyze

the

output

data

batch-means

analysis

was

chosen

for

its

simplicity

and

its

general

applicability

-LSB-

-RSB-

After

the

initial

bias

was

removed

by

ignoring

the

statistics

gathered

during

the

time-to-steady-state

interval

a

very

long

simulation

run

was

divided

into

time

segments

of

equal

size

called

batches

Pairs

of

batches

were

then

merged

until

the

resulting

batches

could

be

considered

statistically

independent

of

each

other

The

final

batch

size

was

chosen

using

a

heuristic

developed

by

Law

and

Carson

-LSB-

-RSB-

After

finding

a

batch

size

m

where

the

batches

have

a

correlation

of

or

less

the

batches

of

size

could

be

considered

uncorrelated

Site

processes

There

is

a

separate

site

process

for

each

physical

site

in

the

network

and

each

site

has

a

unique

identity

number

Each

site

works

for

an

exponentially

distributed

period

of

time

based

on

the

MTTF

Upon

failure

the

site

activates

a

procedure

that

determines

if

the

failure

of

the

site

changes

the

availability

status

of

the

replicated

data

object

The

site

then

waits

an

exponentially

distributed

period

of

time

based

on

the

MTTR

When

this

period

expires

it

activates

a

recovery

procedure

The

recovery

procedure

varies

slightly

depending

on

the

protocol

being

modeled

but

all

operate

in

a

similar

manner

It

first

determines

if

the

recovery

of

the

site

changes

the

availability

status

of

the

replicated

data

object

Afterwards

if

the

site

does

not

contain

a

replica

it

terminates

Otherwise

it

performs

whatever

recovery

process

is

required

by

the

protocol

This

generally

involves

an

attempt

at

gathering

a

quorum

in

order

to

update

the

replica

If

a

quorum

can

not

be

gathered

it

waits

a

short

period

of

time

and

tries

again

The

user

process

There

is

one

user

process

that

is

capable

of

accessing

the

replicated

data

object

from

any

site

in

the

network

The

primary

purpose

of

this

routine

is

to

periodically

activate

the

access

routines

This

is

critical

for

dynamic

voting

protocols

since

the

availability

they

provide

is

dependent

on

the

frequency

of

the

accesses

The

user

process

also

calculates

the

percentage

of

accesses

that

were

successfully

completed

An

access

is

considered

successful

if

it

can

be

performed

from

at

least

one

site

in

the

network

For

the

sake

of

brevity

we

present

only

the

total

unavailability

and

not

the

percentage

of

successful

accesses

The

user

process

waits

an

exponentially

distributed

period

of

time

between

accesses

It

then

decides

whether

to

perform

a

read

or

write

operation

The

choice

is

made

randomly

such

that

a

read

to

write

ratio

of

is

achieved

Experimental

studies

of

access

patterns

have

shown

this

to

be

a

reasonable

assumption

-LSB-

-RSB-

The

user

process

then

activates

the

read

or

write

routine

which

in

turn

activates

the

appropriate

routines

to

effect

the

operation

for

the

modeled

protocol

If

the

access

fails

the

process

is

repeated

on

the

remaining

sites

with

replicas

until

it

is

successful

or

all

replicas

are

exhausted

The

access

times

used

for

the

models

are

discussed

in

the

following

sections

The

access

routines

These

vary

depending

on

the

protocol

being

modeled

but

each

operates

in

the

manner

described

for

the

protocol

Here

we

will

describe

the

general

form

for

accessing

common

to

all

of

them

When

the

read

or

write

routines

are

activated

they

are

given

the

ID

of

the

site

from

which

the

access

is

to

be

made

The

routine

-LRB-

s

-RRB-

for

the

modeled

protocol

that

attempt

to

gather

the

quorum

of

sites

needed

for

an

access

are

activated

and

given

the

ID

of

the

originating

site

These

routines

in

turn

activate

a

routine

that

determines

the

set

of

sites

that

can

communicate

with

the

originating

site

The

access

routine

then

determines

if

this

set

of

sites

constitutes

a

quorum

for

the

modeled

protocol

A

critical

part

of

the

model

is

determining

if

two

sites

can

communicate

Since

all

the

protocols

rely

on

communications

between

sites

to

determine

the

status

of

the

replicated

data

a

fast

simple

means

was

needed

to

determine

communication

links

For

sites

on

the

same

carrier-sense

network

the

solution

is

simple

If

any

two

sites

are

up

and

running

it

can

be

assumed

that

they

can

communicate

For

sites

not

on

the

same

network

segment

that

assumption

can

not

be

made

since

they

may

be

separated

by

one

or

more

gateway

sites

or

repeaters

A

solution

to

this

was

found

by

viewing

the

network

as

a

tree

structure

whose

nodes

consist

of

the

different

network

segments

and

their

respective

sites

One

segment

is

chosen

as

the

root

Communications

are

determined

by

traversing

the

tree

between

two

sites

The

tree

structure

is

conceptual

and

is

represented

by

two

arrays

whose

indices

represent

the

identity

number

of

each

site

The

first

of

these

two

arrays

indicates

if

a

site

is

currently

operating

or

is

down

due

to

failure

The

second

array

indicates

if

a

site

is

connected

to

a

higher

level

node

by

a

gateway

site

If

a

site

is

in

the

root

segment

then

its

entry

in

the

array

is

Otherwise

it

contains

the

identity

number

of

the

gateway

site

Given

the

identity

numbers

of

two

sites

the

communications

routine

determines

if

they

can

communicate

It

begins

by

checking

the

operating

status

of

one

of

the

sites

If

the

site

is

down

the

routine

exits

with

failure

Otherwise

using

the

identity

of

the

given

site

it

retrieves

the

identity

of

the

gateway

site

from

the

second

array

It

then

checks

the

status

of

the

gateway

site

and

retrieves

the

identity

of

its

gateway

site

The

process

continues

until

the

gateway

identity

retrieved

is

that

of

the

root

node

If

Figure

Network

Configuration

at

any

time

the

status

of

one

of

the

sites

checked

is

not

operational

the

routine

terminates

with

failure

The

entire

process

is

repeated

for

the

second

site

that

was

passed

to

the

communications

routine

If

this

process

reaches

the

root

node

for

both

sites

without

encountering

an

inoperative

site

then

the

two

sites

can

communicate

Simulation

results

Instead

of

simulating

a

specific

network

topology

as

in

-LSB-

-RSB-

we

decided

instead

to

consider

a

generic

configuration

that

was

likely

to

be

encountered

in

many

local

area

networks

This

network

configuration

is

shown

in

Figure

It

consists

of

a

backbone

segment

with

seven

sites

to

which

three

smaller

segments

are

linked

through

gateways

To

simplify

the

analysis

of

our

simulation

results

we

decided

to

run

all

our

simulations

assuming

that

all

sites

had

identical

mean

times

to

fail

and

mean

times

to

repair

If

individual

site

attributes

had

been

considered

it

would

have

been

difficult

to

determine

if

the

availability

was

being

affected

by

partitioning

or

by

some

other

factor

such

as

a

site

failing

frequently

for

short

periods

of

time

We

assumed

therefore

that

all

sites

have

a

MTTF

of

days

and

a

MTTR

of

one

day

which

effects

a

site

availability

of

This

is

consistent

with

measurements

of

site

reliability

made

using

the

Internet

-LSB-

-RSB-

The

nine

configurations

considered

in

our

study

are

enumerated

in

Table

The

first

three

configurations

consist

of

three

replicas

Configuration

A

does

not

allow

for

partitions

B

allows

for

three

different

partitions

and

C

allows

for

two

partitions

The

following

three

configurations

consist

of

four

replicas

Configuration

D

does

not

allow

for

partitioning

E

has

four

partitions

and

F

allows

for

three

partitions

The

remaining

configurations

are

for

five

replicas

and

allow

for

the

same

amount

of

partitioning

as

for

the

preceding

configuration

For

MCV

with

four

voting

entities

one

of

the

replicas

was

given

two

votes

in

order

to

avoid

ties

Previous

studies

of

replication

control

protocols

that

regenerate

replicas

or

witnesses

-LSB-

-RSB-

only

considered

environments

where

network

partitions

are

excluded

As

a

result

these

studies

did

not

treat

in

depth

the

issue

of

spare

site

selection

In

a

study

of

regenerating

full

replicas

-LSB-

-RSB-

one

of

the

authors

suggested

that

the

site

with

the

best

fault

tolerance

characteristics

should

be

selected

While

that

will

enhance

availability

it

does

not

address

fully

the

effects

of

further

partitioning

The

newly

regenerated

replica

may

soon

be

isolated

due

to

the

failure

of

a

gateway

site

causing

the

data

to

become

unavailable

We

decided

on

a

simple

heuristic

that

attempts

to

address

this

problem

newly

regenerated

witnesses

are

to

be

placed

on

the

LAN

segment

that

contains

the

largest

number

of

replicas

in

the

current

majority

partition

For

ODV

we

add

the

following

tie-breaking

rule

in

the

event

of

a

tie

newly

regenerated

witnesses

are

to

be

placed

on

the

LAN

segment

with

the

highest

ordered

replica

These

two

rules

attempt

to

place

the

new

witness

in

a

location

where

it

is

least

likely

to

be

separated

from

the

majority

of

replicas

by

network

partition

The

second

rule

is

required

because

of

the

manner

in

which

ODV

determines

quorums

In

a

quorum

call

if

exactly

one-half

of

the

replicas

respond

the

quorum

is

successful

if

the

highest

ordered

replica

is

present

By

placing

a

witness

in

that

partition

a

quorum

will

still

be

possible

even

if

less

than

half

the

replicas

are

available

Since

the

witnesses

are

volatile

they

can

be

placed

on

diskless

sites

Also

the

witnesses

are

never

actually

accessed

by

the

user

so

their

location

is

not

as

dependent

on

data

usage

patterns

as

are

the

actual

replicas

Since

all

sites

were

assumed

to

have

identical

failure

characteristics

we

did

not

consider

placing

witnesses

on

sites

with

the

best

fault

tolerance

characteristics

If

an

available

site

can

not

be

found

on

the

LAN

segment

with

the

most

replicas

then

an

attempt

is

made

on

the

LAN

segment

containing

the

next

most

populated

group

of

replicas

If

a

site

can

not

be

found

on

any

LAN

segments

with

replicas

then

a

site

is

chosen

randomly

In

this

manner

the

heuristic

attempts

to

predict

the

occurrence

of

a

future

current

partition

This

concentrates

the

voting

entities

in

an

environment

where

they

can

not

be

separated

as

easily

by

partitioning

The

performance

of

protocols

using

volatile

witnesses

is

dependent

on

the

access

rate

to

the

replicated

data

As

the

access

rate

increases

failed

witnesses

are

detected

and

regenerated

more

rapidly

and

the

performance

of

the

protocols

improve

To

study

this

effect

we

ran

simulations

for

all

configurations

at

four

different

access

rates

Table

Replica

Placement

Included

Sites

Configuration

LAN

LAN

LAN

LAN

A

B

C

D

E

F

G

H

I

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiicc

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

one

access

per

week

one

per

day

one

per

hour

and

at

a

continuous

access

rate

Tables

to

summarize

the

results

of

the

simulations

for

all

configurations

Data

unavailabilities

were

measured

instead

of

data

availabilities

as

they

show

more

clearly

differences

among

the

protocols

The

column

labeled

Sites

with

Witnesses

indicates

the

placement

of

the

witnesses

Unavailabilities

for

optimistic

dynamic

voting

and

majority

consensus

voting

are

in

the

columns

labeled

ODV

and

MCV

respectively

Without

witnesses

ODV

outperformed

MCV

in

all

cases

for

access

rates

of

one

per

hour

and

constant

accessing

often

performing

twice

as

well

as

MCV

This

was

expected

since

it

had

been

predicted

in

prior

studies

-LSB-

-RSB-

For

the

slower

access

rates

ODV

performed

about

as

well

as

MCV

for

configurations

using

three

replicas

and

performed

better

than

MCV

when

using

more

than

three

replicas

The

reason

for

this

is

that

system

state

information

used

by

ODV

is

updated

during

an

access

operation

and

when

a

replica

recovers

from

a

failure

At

lower

access

rates

the

updating

of

this

information

becomes

more

dependent

on

recovering

sites

than

at

high

access

rates

As

the

number

of

replicas

are

increased

updates

due

to

recovering

sites

also

are

increased

and

the

protocol

operates

more

efficiently

To

evaluate

the

performance

of

ODV

using

witnesses

we

first

consider

its

performance

at

a

constant

access

rate

For

all

cases

ODV

outperformed

MCV

When

one

replica

was

replaced

by

a

witness

the

unavailability

provided

by

ODV

was

decreased

in

each

case

In

some

cases

it

performed

twice

as

well

as

ODV

without

witnesses

This

demonstrates

that

storage

costs

can

be

decreased

and

availability

improved

by

the

replacement

of

a

replica

with

a

regenerable

witness

Alternately

the

availability

provided

by

a

given

number

of

replicas

can

be

improved

by

the

addition

of

a

witness

When

two

replicas

were

replaced

by

witnesses

the

availability

provided

was

less

than

that

without

witnesses

or

with

one

witness

The

reason

for

this

is

that

a

quorum

can

not

be

gathered

if

all

replicas

fail

within

the

same

period

of

time

The

likelihood

of

this

increases

as

the

number

of

replicas

decrease

and

the

number

of

witnesses

increase

A

threshold

is

reached

at

which

point

the

probability

of

all

replicas

failing

simultaneously

is

greater

than

the

availability

provided

by

the

protocol

when

only

replicas

are

used

That

threshold

is

reached

at

two

witnesses

with

ODV

for

the

configurations

we

considered

Availability

can

be

increased

by

the

addition

of

witnesses

without

replacing

replicas

but

a

similar

threshold

is

reached

at

which

increasing

the

number

of

witnesses

does

not

increase

availability

For

instance

configuration

D

with

one

witness

can

be

considered

as

three

replicas

with

one

witness

added

to

increase

availability

This

can

be

compared

to

configuration

A

which

utilizes

three

replicas

and

no

witnesses

In

this

case

configuration

D

with

one

witness

provides

times

less

unavailability

than

does

configuration

A.

Similarly

configuration

G

with

two

witnesses

is

equivalent

to

using

three

replicas

with

the

addition

of

two

witnesses

to

increase

availability

However

when

compared

to

D

with

one

witness

no

increase

in

availability

is

observed

This

is

because

the

optimum

availability

that

can

be

achieved

with

three

replicas

for

this

protocol

is

attained

by

the

addition

of

one

witness

For

MCV

at

a

constant

access

rate

replacing

one

replica

with

a

witness

resulted

in

availabilities

about

the

same

or

better

than

MCV

with

only

replicas

When

two

replicas

were

replaced

the

availability

was

the

same

or

better

than

when

just

one

replica

was

replaced

For

five

replicas

replacing

one

or

two

replicas

resulted

in

improved

availability

for

each

case

The

addition

of

two

witnesses

to

a

given

number

of

replicas

provided

better

Table

Data

Unavailabilities

for

Continuous

Access

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

Configuration

Sites

with

MCV

ODV

Witnesses

-LRB-

Confidence

Intervals

-RRB-

-LRB-

Confidence

Intervals

-RRB-

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

A

B

C

D

E

F

G

H

I

iiiiiiiiiiiiiiiiiiiiiiiiiiiiicc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

availability

than

just

adding

one

witness

for

the

configurations

studied

This

can

be

seen

by

observing

the

unavailabilities

provided

by

configurations

A

and

D

with

one

witness

and

G

with

two

witnesses

With

an

even

number

of

replicas

MCV

provides

about

the

same

availability

as

it

does

with

one

less

replica

even

when

one

replica

is

given

an

extra

vote

to

break

ties

In

this

case

the

availability

can

be

greatly

enhanced

by

the

addition

of

one

witness

to

provide

an

odd

number

of

votes

For

example

configuration

G

with

one

witness

is

equivalent

to

D

with

the

addition

of

a

witness

and

it

provides

more

than

eight

times

less

unavailability

At

an

access

rate

of

one

per

hour

both

ODV

and

MCV

experienced

a

slight

increase

in

unavailability

but

in

general

the

results

were

similar

to

those

at

a

constant

access

rate

As

was

expected

at

access

rates

of

one

per

day

and

one

per

week

the

performance

of

the

protocols

using

witnesses

were

worse

than

when

witnesses

were

not

used

As

was

discussed

earlier

the

performance

of

protocols

using

regenerable

volatile

witnesses

is

dependent

on

the

access

rate

to

the

data

When

the

time

between

accesses

is

greater

than

the

MTTR

of

the

failed

site

the

detection

of

failed

witnesses

is

too

slow

to

effect

an

increase

in

availability

over

that

when

witnesses

are

not

used

It

can

also

be

seen

that

the

difference

in

performance

between

ODV

and

MCV

using

witnesses

starts

to

narrow

as

the

access

rate

decreases

This

is

most

apparent

for

the

one

per

week

access

rate

although

ODV

still

outperforms

MCV

in

most

cases

Conclusions

We

have

presented

in

this

paper

a

comparative

simulation

study

of

two

voting

protocols

with

volatile

witnesses

We

found

that

the

dynamic

voting

protocol

with

regenerable

volatile

witnesses

outperformed

majority

consensus

voting

and

majority

consensus

voting

with

volatile

witnesses

in

most

cases

ODV

with

regenerable

volatile

witnesses

can

provide

better

data

availability

than

MCV

while

using

less

replicas

We

also

found

that

the

location

of

the

witnesses

had

an

impact

on

the

protocol

performance

More

work

is

needed

to

get

a

better

grasp

of

this

impact

and

develop

better

topology-dependent

witness

placements

heuristics

It

might

be

that

the

performance

of

ODV

with

regenerable

volatile

witnesses

can

be

Table

Data

Unavailabilities

for

One

Access

per

Hour

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

Configuration

Sites

with

MCV

ODV

Witnesses

-LRB-

Confidence

Intervals

-RRB-

-LRB-

Confidence

Intervals

-RRB-

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

A

B

C

D

E

F

G

H

I

iiiiiiiiiiiiiiiiiiiiiiiiiiiiicc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

improved

by

restricting

regeneration

to

the

LAN

segment

on

which

the

witness

was

originally

spawned

We

also

found

that

static

MCV

with

volatile

witnesses

can

provide

comparable

availability

to

MCV

while

using

less

replicas

Here

too

the

choice

of

the

location

of

the

witnesses

significantly

affects

data

availability

Acknowledgements

Darrell

D.

E.

Long

was

supported

in

part

by

the

National

Science

Foundation

under

Grant

NSF

and

by

the

Institute

for

Scientific

Computing

Research

at

Lawrence

Livermore

National

Laboratory

The

simulation

results

were

obtained

with

the

aid

of

SIMSCRIPT

II

a

simulation

language

developed

and

supported

by

CACI

Products

Company

of

La

Jolla

CA

We

are

grateful

to

Robb

Mills

for

providing

us

with

the

code

used

to

compute

batch

averages

References

-LSB-

-RSB-

M.

G.

Baker

J.

H.

Hartman

M.

D.

Kupfer

K.

W.

Shirriff

and

J.

K.

Ousterhout

Measurements

of

a

Distributed

File

System

Proc

ACM

Symposium

on

Operating

System

Principles

-LRB-

-RRB-

pp.

-LSB-

-RSB-

D.

Davcev

and

W.

A.

Burkhard

Consistency

and

Recovery

Control

for

Replicated

Files

Proc

ACM

Symposium

on

Operating

System

Principles

-LRB-

-RRB-

pp.

-LSB-

-RSB-

D.

K.

Gifford

Weighted

Voting

for

Replicated

Data

Proc

ACM

Symposium

on

Operating

System

Principles

-LRB-

-RRB-

pp.

-LSB-

-RSB-

A.

Hisgen

A.

Birrell

T.

Mann

M.

Schroeder

and

G.

Swart

Availability

and

Consistency

Tradeoffs

in

the

Echo

Distributed

File

System

Proc

Workshop

on

Workstation

Operating

Systems

-LRB-

-RRB-

pp.

Table

Data

Unavailabilities

for

One

Access

per

Day

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

Configuration

Sites

with

MCV

ODV

Witnesses

-LRB-

Confidence

Intervals

-RRB-

-LRB-

Confidence

Intervals

-RRB-

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

A

B

C

D

E

F

G

H

I

iiiiiiiiiiiiiiiiiiiiiiiiiiiiicc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

-LSB-

-RSB-

S.

Jajodia

and

D.

Mutchler

Dynamic

Voting

Algorithms

for

Maintaining

the

Consistency

of

a

Replicated

Database

ACM

Transactions

on

Database

Systems

Vol

No.

-LRB-

-RRB-

pp.

-LSB-

-RSB-

A.

Law

and

J.

Carson

A

Sequential

Procedure

for

Determining

the

Length

of

a

Steady-State

Simulation

Operations

Research

Vol

-LRB-

-RRB-

pp.

-LSB-

-RSB-

A.

Law

and

R.

Mills

Statistical

Analysis

of

Simulation

Output

Data

Using

SIMSCRIPT

II

Los

Angeles

CACI

Inc.

-LRB-

-RRB-

-LSB-

-RSB-

B.

Liskov

S.

Ghemawat

R.

Gruber

P.

Johnson

L.

Shrira

and

M.

Williams

Replication

in

the

Harp

File

System

Proc

ACM

Symposium

on

Operating

Systems

Principles

-LRB-

-RRB-

pp.

-LSB-

-RSB-

D.D.E.

Long

J.L.

Carroll

and

C.J.

Park

A

Study

of

the

Reliability

of

Internet

Sites

Proc

Symposium

on

Reliable

Distributed

Systems

-LRB-

-RRB-

pp.

-LSB-

-RSB-

D.D.E.

Long

The

Management

of

Replicated

Data

in

a

Distributed

System

Ph.D.

dissertation

Department

of

CSE

University

of

California

San

Diego

-LRB-

-RRB-

-LSB-

-RSB-

D.D.E.

Long

and

J.-F

Pa?ris

A

Realistic

Evaluation

of

Optimistic

Dynamic

Voting

Proc

Symposium

on

Reliable

Distributed

Systems

-LRB-

-RRB-

pp.

-LSB-

-RSB-

J.D.

Noe

and

A.

Andreassian

Effectiveness

of

Replication

in

Distributed

Computing

Networks

Proc

International

Conference

on

Data

Engineering

-LRB-

-RRB-

pp.

-LSB-

-RSB-

J.

Ousterhout

H.

Da

Costa

D.

Harrison

J.

Kunze

M.

Kupfer

and

J.

Thompson

A

Trace-Driven

Analysis

of

the

UNIX

BSD

File

System

Proc

Symposium

on

Operating

System

Principles

-LRB-

-RRB-

pp.

Table

Data

Unavailabilities

for

One

Access

per

Week

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

Configuration

Sites

with

MCV

ODV

Witnesses

-LRB-

Confidence

Intervals

-RRB-

-LRB-

Confidence

Intervals

-RRB-

iiiiiiiiiiiiiiiiiiiiiiiiiiiii

A

B

C

D

E

F

G

H

I

iiiiiiiiiiiiiiiiiiiiiiiiiiiiicc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

cc

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

c

-LSB-

-RSB-

J.-F

Pa?ris

D.D.E.

Long

and

A.

Glockner

A

Realistic

Evaluation

of

Consistency

Algorithms

for

Replicated

Files

Proceedings

Annual

Simulation

Symposium

-LRB-

-RRB-

pp.

-LSB-

-RSB-

J.-F

Pa?ris

and

D.D.E.

Long

Voting

with

Regenerable

Volatile

Witnesses

Proc

International

Conference

on

Data

Engineering

-LRB-

-RRB-

pp.

-LSB-

-RSB-

J.-F

Pa?ris

Voting

with

Witnesses

A

Consistency

Scheme

for

Replicated

Files

Proc

International

Conference

on

Distributed

Computing

Systems

-LRB-

-RRB-

pp.

-LSB-

-RSB-

J.-F

Pa?ris

Efficient

Management

of

Replicated

Data

Proc

International

Conference

on

Database

Theory

Lecture

Notes

in

Computer

Science

Springer

Verlag

-LRB-

-RRB-

pp.

-LSB-

-RSB-

J.-F

Pa?ris

Efficient

Voting

Protocols

with

Witnesses

Proc

International

Conference

on

Database

Theory

Lecture

Notes

in

Computer

Science

Springer

Verlag

-LRB-

-RRB-

pp.

-LSB-

-RSB-

C.

Pu

J.D.

Noe

and

A.B.

Proudfoot

Regeneration

of

Replicated

Objects

A

Technique

and

its

Eden

Implementation

IEEE

Transactions

on

Software

Engineering

Vol

No.

-LRB-

-RRB-

pp.

-LSB-

-RSB-

R.

D.

Schlichting

and

F.

B.

Schneider

Fail

Stop

Processors

An

Approach

to

Designing

FaultTolerant

Computing

Systems

ACM

Transactions

on

Computer

Systems

Vol

-LRB-

-RRB-

-LSB-

-RSB-

P.

K.

Sloope

A

Realistic

Evaluation

of

the

Effect

of

Network

Partitioning

on

the

Management

of

Replicated

Data

M.S.

Thesis

Department

of

Computer

Science

University

of

Houston

-LRB-

-RRB-

O O O O O O O O O O O O O O O B I I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I I O O O B O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B I I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O B O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O B O O O O B O O O B O O O B O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

discrete event simulation
replication control protocols
voting
witnesses