USING

GROUP

COMMUNICATION

TO

IMPLEMENT

A

FAULT-TOLERANT

DIRECTORY

SERVICE

M.

Frans

Kaashoek

Andrew

S.

Tanenbaum

Kees

Verstoep

Email

kaashoek@cs.vu.nl,

ast@cs.vu.nl,

and

versto@cs.vu.nl

Dept.

of

Mathematics

and

Computer

Science

Vrije

Universiteit

Amsterdam

The

Netherlands

ABSTRACT

Group

communication

is

an

important

paradigm

for

building

distributed

applications

This

paper

discusses

a

fault-tolerant

distributed

directory

service

based

on

group

communication

and

compares

it

with

the

previous

design

and

implementation

based

on

remote

procedure

call

The

group

directory

service

uses

an

active

replication

scheme

and

when

triplicated

can

handle

lookup

operations

per

second

and

update

operations

per

second

-LRB-

using

nonvolatile

RAM

-RRB-

This

performance

is

better

than

the

performance

for

the

RPC

implementation

and

it

is

even

better

than

the

performance

for

directory

operations

under

SunOS

which

does

not

provide

any

fault

tolerance

at

all

The

paper

concludes

that

the

implementation

using

group

communication

is

simpler

and

has

better

performance

than

the

one

based

on

remote

procedure

call

supporting

the

claim

that

a

distributed

operating

system

should

provide

both

remote

procedure

call

and

group

communication

Introduction

Many

distributed

operating

systems

support

only

point-to-point

communication

-LSB-

Rashid

Ousterhout

et

al.

Pike

et

al.

-RSB-

Amoeba

-LSB-

Mullender

et

al.

Tanenbaum

et

al.

-RSB-

on

the

other

hand

is

a

distributed

operating

system

that

provides

both

point-to-point

communication

and

communication

-LRB-

group

communication

-RRB-

This

paper

discusses

the

design

and

implementation

of

a

faulttolerant

distributed

directory

service

using

reliable

and

totally-ordered

group

communication

and

com

pares

it

to

the

previous

implementation

based

on

remote

procedure

call

-LRB-

RPC

-RRB-

-LSB-

Birrell

and

Nelson

-RSB-

Although

the

group

directory

service

is

better

-LRB-

it

also

tolerates

network

partitions

-RRB-

we

conclude

that

its

design

and

implementation

are

simpler

and

that

it

has

better

performance

The

directory

service

is

an

example

application

that

tolerates

faults

by

using

active

replication

Our

results

extend

in

principle

to

other

services

that

are

based

on

active

replication

Based

on

our

experience

with

the

directory

service

and

run-time

systems

for

distributed

parallel

programming

-LSB-

Tanenbaum

et

al.

-RSB-

we

claim

that

a

distributed

operating

system

should

provide

support

for

both

remote

procedure

call

and

totally-ordered

group

communication

The

Amoeba

directory

service

has

a

long

history

The

version

in

use

for

the

last

two

years

is

based

on

RPC

It

consists

of

two

servers

each

running

on

a

separate

machine

-LSB-

Van

Renesse

-RSB-

Directory

operations

that

do

not

modify

a

directory

can

be

executed

by

either

of

the

two

servers

without

any

communication

with

the

other

server

Directory

operations

that

modify

a

directory

result

in

communication

between

the

two

servers

When

an

update

request

comes

in

at

one

server

it

performs

an

RPC

with

the

other

server

informing

it

of

the

intended

update

If

the

other

server

is

not

busy

performing

a

conflicting

operation

it

stores

the

intentions

on

disk

and

sends

an

OK

message

back

On

receiving

the

reply

the

original

server

performs

the

update

and

sends

a

reply

back

to

the

client

To

avoid

the

costs

of

immediately

creating

two

copies

on

separate

disks

of

a

directory

that

is

updated

the

directory

service

uses

lazy

replication

The

server

that

processes

the

client

request

creates

a

copy

on

its

local

disk

but

the

second

copy

at

the

other

server

is

created

later

in

the

background

As

the

directory

service

is

only

duplicated

it

can

not

guarantee

consistency

in

the

presence

of

network

partitions

This

paper

discusses

an

alternative

design

and

implementation

of

the

directory

service

using

Amoeba

's

primitives

for

group

communication

-LRB-

see

Fig.

-RRB-

These

primitives

guarantee

that

all

processes

constituting

a

group

see

each

event

in

the

same

total

order

Random

mixtures

where

some

members

see

first

a

message

from

A

and

then

a

message

from

B

while

other

members

see

them

in

the

reverse

order

are

guaranteed

not

to

happen

A

programmer

can

request

by

specifying

a

resilience

degree

r

that

even

in

the

face

of

r

processor

failures

each

surviving

member

will

see

all

messages

in

the

same

order

For

example

if

a

programmer

specifies

r

for

a

group

with

members

the

system

will

guarantee

that

even

in

the

face

of

processor

failures

the

remaining

members

will

receive

each

message

in

a

total

order

By

setting

r

the

programmer

can

trade

performance

against

fault

tolerance

The

protocols

that

Amoeba

uses

to

implement

the

group

communication

primitives

have

been

described

in

an

earlier

publication

-LSB-

Kaashoek

and

Tanenbaum

-RSB-

The

design

of

the

directory

service

depends

on

the

total

ordering

of

events

but

this

does

not

restrict

our

work

to

Amoeba

alone

Other

systems

such

as

Isis

-LSB-

Birman

and

Joseph

-RSB-

Psync

-LSB-

Peterson

et

al.

-RSB-

extensions

to

the

V

-LSB-

Cheriton

and

Zwaenepoel

-RSB-

system

-LSB-

Navaratnam

et

al.

-RSB-

and

-LSB-

Barrett

et

al.

-RSB-

provide

primitives

with

similar

semantics

Primitive

Description

CreateGroup

Create

a

new

group

JoinGroup

Make

a

process

member

of

a

group

LeaveGroup

Leave

a

group

SendToGroup

Send

a

message

to

all

members

of

a

group

ReceiveFromGroup

Receive

the

next

message

in

sequence

ResetGroup

Rebuild

group

after

a

failure

GetInfoGroup

Obtain

information

about

a

group

from

the

kernel

Fig.

Amoeba

's

system

calls

for

group

communication

The

outline

of

the

rest

of

the

paper

is

as

follows

In

Section

we

will

describe

the

functionality

required

from

the

directory

service

and

its

role

in

the

Amoeba

system

In

Section

we

will

discuss

in

detail

the

design

and

implementation

using

group

communication

In

Section

we

will

compare

the

directory

service

using

group

communication

with

the

directory

service

using

RPC

and

give

performance

measurements

for

both

In

the

same

section

we

will

also

compare

the

performance

of

directory

operations

under

Amoeba

with

similar

operations

under

SunOS

and

compare

the

performance

of

the

group

directory

service

with

the

same

service

using

nonvolatile

RAM

-LRB-

NVRAM

-RRB-

In

Section

we

will

discuss

the

results

of

the

comparisons

and

describe

related

work

In

Section

we

will

draw

our

conclusions

A

Fault-Tolerant

Directory

Service

The

directory

service

is

a

vital

service

in

the

Amoeba

distributed

operating

system

-LSB-

Van

Renesse

-RSB-

It

provides

among

other

things

a

mapping

from

ASCII

names

to

capabilities

A

capability

in

Amoeba

identifies

and

protects

an

object

-LRB-

e.g.

a

file

-RRB-

A

capability

is

string

consisting

of

parts

-RRB-

a

port

that

identifies

the

service

-RRB-

an

object

number

that

identifies

an

object

at

the

service

specified

by

the

port

-RRB-

a

right

field

that

specifies

which

operations

the

holder

of

the

capability

may

perform

-RRB-

a

check

field

that

determines

if

the

capability

is

valid

or

not

The

set

of

capabilities

a

user

possesses

determines

which

objects

he

can

access

The

directory

service

allows

the

users

to

store

these

capabilities

under

ASCII

names

to

make

life

easier

for

them

The

layout

of

an

example

directory

with

six

entries

is

shown

simplified

in

This

directory

has

one

row

for

each

of

the

six

file

names

stored

in

it

The

directory

also

has

three

columns

each

one

representing

a

different

protection

domain

For

example

the

first

column

might

store

capabilities

for

the

owner

-LRB-

with

all

the

rights

bits

on

-RRB-

the

second

might

store

capabilities

for

members

of

the

owner

's

group

-LRB-

with

some

of

the

rights

bits

turned

off

-RRB-

and

the

third

might

store

capabilities

for

everyone

else

-LRB-

with

only

the

read

bit

turned

on

-RRB-

When

the

owner

of

a

directory

gives

away

a

capability

for

it

the

capability

is

really

a

capability

for

a

single

column

not

for

the

directory

as

a

whole

When

giving

a

directory

capability

to

an

unrelated

person

the

owner

could

give

a

capability

for

the

third

column

The

recipient

of

this

capability

would

have

no

access

to

the

more

powerful

capabilities

in

the

first

two

columns

Column

Column

Column

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Capability

Fig.

Layout

of

an

example

directory

The

directory

service

supports

the

operations

shown

in

Figure

There

are

operations

to

manipulate

directories

to

manipulate

a

single

row

-LRB-

i.e.

a

tuple

consisting

of

a

string

and

a

capability

-RRB-

of

a

directory

and

to

manipulate

a

set

of

rows

One

of

the

most

important

things

to

know

about

the

directory

service

is

the

frequency

of

the

read

operations

-LRB-

e.g.

list

directory

-RRB-

and

write

operations

-LRB-

e.g.

delete

directory

-RRB-

because

these

numbers

influence

the

design

Measurements

over

three

weeks

show

that

of

all

directory

operations

are

reads

Therefore

both

the

RPC

directory

service

and

the

group

directory

service

optimize

read

operations

-LRB-

i.e.

operations

that

do

not

modify

a

directory

-RRB-

In

this

paper

we

focus

on

the

design

and

implementation

of

the

directory

service

and

not

on

the

reasons

why

this

interface

was

chosen

This

has

been

discussed

by

Van

Renesse

-LSB-

Van

Renesse

-RSB-

For

other

papers

discussing

the

design

of

a

naming

service

we

refer

the

reader

to

-LSB-

Curtis

and

Wittie

Schroeder

et

al.

Lampson

Cheriton

and

Mann

-RSB-

The

directory

service

must

be

highly

reliable

and

highly

available

Users

rely

on

the

directory

service

to

store

capabilities

without

losing

them

and

users

must

always

be

able

to

access

their

capabilities

To

fulfill

these

demands

the

directory

service

replicates

-LRB-

name

capability

-RRB-

pairs

on

multiple

machines

Operation

Description

Create

dir

Create

a

new

directory

Delete

dir

Delete

a

directory

List

dir

List

a

directory

Append

row

Add

a

new

row

to

a

directory

Chmod

row

Change

protection

Delete

row

Delete

row

of

a

directory

Lookup

set

Lookup

the

capabilities

in

a

set

of

rows

Replace

set

Replace

the

capabilities

in

a

set

of

rows

Fig.

Operations

supported

by

the

directory

service

each

with

its

own

disk

If

one

of

the

machines

is

unavailable

one

of

the

other

machines

will

be

able

to

reply

to

a

user

's

request

If

one

of

the

disks

becomes

unreadable

one

of

the

other

disks

can

be

used

to

reply

to

a

user

's

request

The

key

problem

is

to

keep

the

replicas

of

a

name-capability

pair

consistent

in

an

efficient

way

An

update

to

a

directory

must

be

performed

quickly

because

otherwise

many

applications

will

run

less

efficiently

We

require

that

the

directory

service

maintains

one-copy

serializability

-LSB-

Bernstein

et

al.

-RSB-

The

execution

of

operations

on

the

directory

service

must

be

equivalent

to

a

serial

execution

of

the

operations

on

a

nonreplicated

directory

service

To

achieve

this

goal

each

operation

of

the

directory

service

is

executed

indivisibly

The

directory

service

does

not

support

indivisible

execution

of

a

set

of

operations

as

this

requires

atomic

transactions

-LSB-

Gray

Lampson

-RSB-

It

also

does

not

support

failure-free

operations

for

clients

as

this

requires

updating

a

log

file

on

each

operation

We

feel

that

these

semantics

are

too

expensive

to

support

and

moreover

are

seldomly

necessary

Both

designs

of

the

directory

service

assume

clean

failures

A

processor

works

or

does

not

work

-LRB-

i.e.

fail-stop

failures

-RRB-

and

it

does

not

send

malicious

or

contradictory

messages

-LRB-

i.e.

it

does

not

exhibit

Byzantine

failures

-RRB-

The

RPC

implementation

also

assumes

that

network

partitions

will

not

happen

the

group

implementation

however

guarantees

consistency

even

in

the

case

of

clean

network

partitions

-LRB-

e.g.

any

two

processors

in

the

same

partition

can

communicate

and

any

two

processors

in

different

partitions

can

not

communicate

-RRB-

-LSB-

Davidson

et

al.

-RSB-

Stronger

failure

semantics

could

have

been

implemented

using

techniques

as

described

in

-LSB-

Cristian

Barrett

et

al.

Hariri

et

al.

Shrivastava

et

al.

-RSB-

Again

we

feel

that

these

stronger

semantics

are

too

expensive

to

support

and

moreover

are

overkill

for

an

application

like

the

directory

service

In

the

rest

of

this

paper

we

assume

that

the

following

basic

requirements

for

a

fault-tolerant

directory

service

are

met

Each

directory

server

should

be

located

on

a

separate

electrical

group

-LRB-

with

its

own

fuse

-RRB-

and

all

the

directory

servers

should

be

connected

by

multiple

redundant

networks

Because

the

Amoeba

communication

primitives

are

implemented

on

top

of

a

network

protocol

-LRB-

FLIP

-RRB-

-LSB-

Kaashoek

et

al.

-RSB-

the

latter

requirement

can

be

fulfilled

Although

it

could

run

on

multiple

networks

without

a

single

software

modification

the

current

implementation

runs

on

a

single

network

Directory

Service

Based

on

Group

Communication

Unlike

the

RPC

directory

service

the

group

implementation

is

triplicated

-LRB-

though

four

or

more

replicas

are

also

possible

without

changing

the

protocol

-RRB-

and

uses

active

replication

Also

it

allows

network

partitions

To

keep

the

copies

consistent

it

uses

a

modified

version

of

the

read-one

write-all

policy

called

accessible

copies

-LSB-

El

Abbadi

et

al.

-RSB-

Recovery

is

based

on

the

protocol

described

by

Skeen

-LSB-

Skeen

-RSB-

In

this

section

we

will

describe

in

detail

the

algorithms

used

in

the

implementation

of

the

group

directory

service

The

organization

of

the

group

directory

service

is

depicted

in

Figure

The

directory

service

is

currently

built

out

of

three

directory

servers

three

Bullet

servers

-LRB-

Amoeba

's

file

server

-RRB-

and

three

disk

servers

A

Bullet

server

and

a

disk

server

share

one

disk

Each

directory

server

only

uses

one

Bullet

server

and

one

disk

server

Each

directory

server

stores

one

copy

of

each

directory

in

a

separate

Bullet

file

The

actual

implementation

of

the

directory

service

is

slightly

more

complicated

each

server

caches

recently

used

directories

in

RAM

When

a

request

is

received

a

directory

server

looks

first

in

its

cache

to

see

if

the

directory

is

stored

there

If

so

it

can

reply

immediately

without

going

to

the

Bullet

server

If

not

the

directory

server

performs

an

RPC

with

the

Bullet

server

to

get

the

requested

directory

and

stores

it

in

its

cache

for

subsequent

requests

The

cache

is

there

to

avoid

the

overhead

of

performing

an

additional

RPC

with

the

Bullet

server

The

administrative

data

are

stored

on

a

raw

disk

partition

of

n

fixed-length

blocks

Block

zero

contains

global

information

about

the

directory

service

and

blocks

to

n

contain

the

capabilities

of

the

Bullet

files

storing

the

contents

of

a

directory

Block

the

commit

block

for

the

group

directory

service

is

shown

in

Figure

The

configuration

vector

is

a

bit

vector

indexed

by

server

number

If

server

for

example

is

down

bit

in

the

vector

is

set

to

It

describes

the

last

configuration

with

a

majority

of

which

the

server

was

a

member

Each

time

an

update

operation

is

performed

a

sequence

number

stored

with

the

directory

is

in

Disk

Bullet

Dir

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Disk

Bullet

Dir

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Disk

Bullet

Dir

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Directory

service

group

Fig.

Organization

of

the

directory

service

based

on

group

communication

-LRB-

a

-RRB-

Administrative

data

-LRB-

b

-RRB-

Directories

-LRB-

c

-RRB-

Files

up

up

up

Sequence

number

Recovering

Configuration

vector

Fig.

Layout

of

the

commit

block

creased

During

recovery

the

sequence

number

is

computed

by

taking

the

maximum

of

all

the

sequence

numbers

stored

with

the

directory

files

and

the

sequence

number

stored

in

the

commit

block

At

first

sight

it

may

seem

strange

that

a

sequence

number

is

also

stored

in

the

commit

block

but

this

is

needed

for

the

following

case

When

a

directory

is

deleted

the

reference

to

the

Bullet

file

containing

the

directory

and

the

sequence

number

is

deleted

but

the

server

must

record

somewhere

that

it

performed

an

update

The

sequence

number

in

the

commit

block

is

used

for

this

case

It

is

only

updated

when

a

directory

is

deleted

The

recovering

field

is

needed

to

keep

track

whether

a

server

crashed

during

recovery

If

this

field

is

set

the

server

knows

that

it

crashed

during

recovery

In

this

case

it

sets

the

sequence

number

to

zero

because

its

state

is

inconsistent

It

may

have

recent

versions

of

some

directories

and

old

versions

of

other

directories

The

sequence

number

is

set

to

zero

to

ensure

that

other

servers

will

not

try

to

update

their

directories

from

a

server

whose

state

is

inconsistent

Blocks

to

n

contain

a

table

of

capabilities

indexed

by

object

number

which

is

part

of

the

capability

that

identifies

a

directory

The

capability

in

the

object

table

points

to

a

Bullet

file

that

stores

the

directory

a

random

number

for

access

protection

and

the

sequence

number

of

the

last

change

Unlike

the

RPC

implementation

the

sequence

number

for

the

last

change

is

also

with

the

directory

In

the

group

implementation

if

a

directory

is

changed

only

the

file

has

to

be

written

while

in

the

RPC

implementation

both

a

Bullet

file

and

the

commit

block

have

to

be

written

In

this

way

the

group

implementation

saves

one

disk

operation

-LRB-

writing

the

commit

block

-RRB-

though

the

RPC

implementation

could

have

used

the

same

optimization

Default

Operation

The

protocol

to

keep

the

directories

consistent

and

to

perform

concurrency

control

is

given

in

Figure

A

server

in

the

group

directory

service

consists

of

several

threads

the

server

threads

and

one

group

thread

The

server

threads

are

waiting

for

a

request

from

a

client

The

group

thread

is

waiting

for

an

internal

message

sent

to

the

group

There

can

be

multiple

server

threads

but

there

is

only

one

group

thread

A

server

thread

that

receives

a

request

and

initiates

a

directory

operation

is

called

the

initiator

The

initiator

first

checks

if

the

current

group

has

a

majority

-LRB-

i.e.

at

least

two

of

the

three

servers

must

be

up

-RRB-

If

not

the

request

is

refused

otherwise

the

request

is

processed

The

reason

why

even

a

read

request

requires

a

majority

is

that

the

network

might

become

partitioned

Consider

the

following

situation

Two

servers

and

a

client

are

on

one

side

of

the

network

partition

and

the

client

deletes

the

directory

foo

This

update

will

be

performed

because

the

two

servers

have

a

majority

Now

assume

that

the

two

servers

crash

and

that

the

network

partition

is

repaired

If

the

client

asks

the

remaining

server

to

list

the

directory

foo

it

would

get

the

contents

of

a

directory

that

it

deleted

successfully

earlier

Therefore

read

requests

are

refused

if

the

group

of

servers

does

not

have

a

majority

-LRB-

There

is

an

escape

for

system

administrators

in

case

two

servers

lose

their

data

forever

due

to

for

example

head

crashes

-RRB-

As

in

the

RPC

implementation

read

operations

can

be

handled

by

any

server

without

the

need

for

communication

between

the

servers

When

a

read

request

is

received

the

initiator

checks

if

the

kernel

has

any

messages

buffered

using

GetInfoGroup

If

so

it

blocks

to

give

the

group

thread

a

chance

to

process

the

buffered

messages

before

performing

a

read

operation

the

initiator

has

to

be

sure

that

it

has

performed

all

preceeding

write

operations

If

a

client

for

example

deletes

a

directory

and

then

tries

to

read

it

back

it

has

to

receive

an

error

even

if

the

client

requests

were

processed

at

different

directory

servers

As

messages

are

sent

using

a

resilience

degree

r

it

is

sufficient

to

check

if

there

are

any

messages

buffered

on

arrival

of

the

read

request

Once

these

buffered

messages

are

processed

the

initiator

can

perform

the

read

request

as

it

can

be

percent

sure

that

it

has

seen

all

preceeding

update

operations

Write

operations

require

communication

among

the

servers

First

the

initiator

generates

a

new

check

field

because

all

the

servers

must

use

the

same

check

field

when

creating

a

new

directory

The

initiator

broadcasts

the

request

to

the

group

using

the

primitive

SendToGroup

and

blocks

until

the

group

thread

received

and

executed

the

request

Once

it

is

unblocked

it

sends

the

result

of

the

request

back

to

the

client

Initiator

if

-LRB-

NumberOfServers

-RRB-

return

failure

majority

if

-LRB-

-LRB-

request

-RRB-

-RRB-

-LCB-

read

request

GetInfoGroup

-LRB-

-RRB-

check

for

buffered

messages

buffered

buffered

-LRB-

-RRB-

wait

until

seqno

-RCB-

else

-LCB-

write

request

generate

check

field

for

new

directory

SendToGroup

-LRB-

request

check

field

me

-RRB-

wait

until

group

thread

has

received

and

executed

the

request

-RCB-

send

reply

to

client

Group

thread

if

-LRB-

group

failure

-RRB-

-LCB-

did

a

server

fail

rebuild

majority

of

group

call

ResetGroup

if

-LRB-

group

rebuild

failed

-RRB-

enter

recovery

GetInfoGroup

-LRB-

-RRB-

get

info

about

rebuilt

group

write

commit

block

update

configuration

vector

try

again

start

receiving

again

-RCB-

else

-LCB-

create

directory

on

Bullet

file

use

supplied

check

field

update

cache

update

object

table

write

changed

object

table

to

disk

commit

increase

-LRB-

seqno

-RRB-

if

-LRB-

sender

me

-RRB-

wakeup

initiator

remove

old

bullet

files

-RCB-

Fig.

Protocol

to

ensure

consistency

of

the

copies

of

a

directory

The

group

thread

is

continuously

waiting

for

a

message

sent

to

the

group

-LRB-

i.e.

it

is

blocked

in

ReceiveFromGroup

-RRB-

If

ReceiveFromGroup

returns

the

group

thread

first

checks

if

the

call

to

Re

ceiveFromGroup

returned

successfully

If

not

one

of

the

servers

must

have

crashed

In

this

case

the

remaining

directory

servers

run

the

recovery

protocol

described

in

the

next

section

If

ReceiveFromGroup

returns

successfully

the

server

creates

the

new

directories

on

its

Bullet

server

updates

its

cache

updates

its

object

table

and

writes

the

changed

entry

in

the

object

table

to

its

disk

As

soon

as

one

server

writes

the

new

entry

to

disk

the

operation

is

committed

If

no

server

fails

each

server

will

receive

all

requests

and

service

all

requests

in

the

same

order

and

therefore

all

the

copies

of

the

directories

stay

consistent

There

might

be

a

small

delay

but

eventually

each

server

will

receive

all

messages

When

the

client

's

RPC

returns

successfully

the

user

knows

that

one

new

copy

of

the

directory

is

stored

on

disk

and

that

at

least

two

other

servers

have

received

the

request

and

stored

the

new

directory

on

disk

too

or

will

do

so

shortly

If

one

server

fails

the

client

can

still

access

his

directories

Let

us

analyze

the

cost

of

a

directory

operation

in

terms

of

communication

cost

and

disk

operations

As

in

the

RPC

implementation

read

operations

do

not

involve

communication

or

disk

operations

-LRB-

if

the

requested

directory

is

in

the

cache

-RRB-

Write

operations

require

one

group

message

sent

with

r

a

Bullet

operation

to

store

the

new

directory

and

one

disk

operation

to

store

the

changed

entry

in

the

object

table

Compared

to

the

RPC

implementation

the

number

of

disk

operations

is

smaller

The

RPC

implementation

requires

an

additional

disk

operation

to

store

an

intention

list

with

updates

The

number

of

messages

in

the

group

service

however

is

higher

A

SendToGroup

with

r

requires

messages

whereas

an

RPC

in

Amoeba

requires

only

messages

The

cost

of

sending

a

message

however

is

an

order

of

magnitude

less

than

the

cost

of

performing

a

disk

operation

Thus

roughly

the

performance

of

the

group

implementation

is

better

than

the

performance

of

the

RPC

implementation

while

providing

more

fault

tolerance

and

a

higher

availability

This

analysis

is

not

completely

fair

however

If

the

RPC

implementation

like

the

group

implementation

had

stored

the

sequence

number

with

the

directory

files

and

thereby

avoided

one

disk

write

for

the

commit

block

the

performance

of

the

RPC

implementation

would

have

been

better

as

it

would

send

fewer

messages

On

the

other

hand

if

the

RPC

service

had

been

triplicated

it

would

have

been

slower

than

the

group

service

because

then

it

would

have

sent

more

messages

-LRB-

RPCs

against

one

SendToGroup

-RRB-

Recovery

Protocol

A

server

starts

executing

the

recovery

protocol

when

it

is

a

member

of

a

group

that

forms

a

minority

or

when

it

comes

up

after

having

been

down

The

protocol

for

recovery

of

the

group

service

is

more

complicated

than

the

protocol

for

the

RPC

service

because

more

servers

are

involved

Consider

the

following

sequence

of

events

in

a

group

of

three

servers

that

is

up

and

running

Server

crashes

Servers

and

rebuild

the

group

so

their

configuration

vectors

have

the

value

-LRB-

and

are

up

is

down

-RRB-

Now

both

and

also

crash

When

server

comes

up

again

its

vector

reads

but

on

its

own

it

can

not

form

a

group

To

execute

a

client

request

a

majority

of

the

servers

must

be

up

otherwise

copies

of

a

directory

could

become

inconsistent

for

example

in

the

case

of

a

network

partition

If

server

also

comes

up

its

vector

reads

At

first

sight

it

may

appear

that

and

can

form

a

group

as

together

they

form

a

majority

However

this

is

not

sufficient

Server

who

is

still

down

may

have

performed

the

latest

update

To

see

this

consider

the

following

sequence

of

events

just

before

and

crashed

A

client

request

is

received

by

it

successfully

sends

it

to

and

Now

both

and

have

the

message

buffered

It

can

happen

that

crashes

before

processing

the

message

while

crashes

after

processing

the

message

In

this

case

server

has

the

latest

version

of

the

directories

and

thus

and

can

not

form

a

new

group

and

start

accepting

requests

Now

assume

that

server

comes

up

instead

of

server

The

configuration

vector

of

both

server

and

read

From

this

information

they

can

conclude

that

crashed

before

and

did

Furthermore

no

update

can

have

been

performed

after

or

crashed

because

there

was

no

majority

Server

and

together

are

therefore

sure

that

one

of

them

has

the

latest

version

of

the

directories

Thus

they

can

recover

without

server

and

use

the

sequence

number

to

determine

who

actually

has

the

latest

version

In

general

two

conditions

have

to

be

met

to

recover

The

new

group

must

have

a

majority

to

avoid

inconsistencies

during

network

partitions

The

new

group

must

contain

the

set

of

servers

that

possibly

performed

the

latest

update

It

is

the

latter

requirement

that

makes

recovery

of

the

group

service

complicated

During

recovery

the

servers

need

an

algorithm

to

determine

which

are

the

servers

that

failed

last

Such

an

algorithm

exists

it

is

due

to

Skeen

-LSB-

Skeen

-RSB-

and

it

works

as

follows

Each

server

keeps

a

mourned

set

of

servers

that

crashed

before

it

When

a

server

starts

recovering

it

sets

the

new

group

to

only

itself

Then

it

exchanges

with

all

other

alive

servers

its

mourned

set

Each

time

it

receives

a

new

mourned

set

it

adds

the

servers

in

the

received

mourned

set

to

its

own

mourned

set

Furthermore

it

puts

the

server

with

whom

it

exchanged

the

mourned

set

in

the

new

group

The

algorithm

terminates

when

all

servers

minus

the

mourned

set

are

a

subset

of

the

new

group

Figure

gives

the

complete

recovery

protocol

When

a

server

enters

recovery

mode

it

first

tries

to

join

the

group

If

this

fails

it

assumes

that

the

group

is

not

created

yet

and

it

creates

the

group

If

after

a

certain

waiting

period

an

insufficient

number

of

members

joined

the

group

the

server

leaves

the

group

and

starts

all

over

again

It

may

have

happened

that

two

servers

recreated

the

group

-LRB-

e.g.

two

servers

on

each

side

of

a

network

partition

-RRB-

and

that

they

both

can

not

acquire

a

majority

of

the

members

Recovery

join

server

group

or

create

it

while

-LRB-

minority

timeout

-RRB-

-LCB-

wait

for

some

time

GetInfoGroup

-LRB-

-RRB-

-RCB-

if

-LRB-

minority

-RRB-

try

again

start

over

again

newgroup

-LSB-

me

-RSB-

initialize

new

group

vector

SequenceNo

-LSB-

me

-RSB-

Sequence

number

initialize

sequence

number

vector

initialize

mourned

vector

from

configuration

vector

for

-LRB-

all

members

in

group

-RRB-

-LCB-

exchange

info

with

each

server

exchange

info

with

server

s

exchange

mourned

set

and

sequence

no

if

-LRB-

success

-RRB-

-LCB-

RPC

succeeded

newgroup

-LSB-

s

-RSB-

add

server

to

new

group

SequenceNo

-LSB-

s

-RSB-

Sequence

number

mourned

set

received

mourned

set

take

union

-RCB-

-RCB-

last

all

servers

mourned

set

the

servers

that

performed

the

last

update

if

-LRB-

last

is

not

subset

of

new

group

-RRB-

try

again

s

HighestNumber

-LRB-

SequenceNo

-RRB-

determine

server

with

highest

sequence

no

get

copies

of

latest

version

of

directories

from

s

if

-LRB-

success

-RRB-

try

again

succeeded

in

getting

copies

write

commit

block

store

configuration

vector

enter

normal

operation

recovery

is

done

Fig.

Recovery

protocol

for

group

directory

service

Once

a

server

has

created

or

joined

a

group

that

contains

a

majority

of

all

directory

servers

it

executes

Skeen

's

algorithm

to

determine

the

set

of

servers

that

crashed

last

the

last

set

If

this

set

is

not

a

subset

of

the

new

group

the

server

starts

all

over

again

waiting

for

servers

from

the

last

set

to

join

the

group

If

the

last

set

is

a

subset

of

the

new

group

the

new

group

has

the

most

recent

version

of

the

directories

The

server

determines

who

in

the

group

has

them

and

gets

them

Once

it

is

up-to-date

it

writes

the

new

configuration

to

disk

and

enters

normal

operation

The

recovery

protocol

can

be

improved

Skeen

's

algorithm

assumes

that

network

partitions

do

not

occur

To

make

his

algorithm

work

for

our

assumption

that

network

partitions

can

happen

we

forced

the

servers

that

have

a

minority

to

fail

Now

the

recovery

protocol

will

fail

in

certain

cases

in

which

it

is

actually

possible

to

recover

Consider

the

following

sequence

of

events

Server

and

are

up

server

crashes

server

and

form

a

new

group

server

crashes

Now

as

we

want

to

tolerate

network

partitions

correctly

we

forced

server

to

fail

However

this

is

too

strict

If

server

stays

alive

and

server

is

restarted

server

and

can

form

a

new

group

because

server

must

have

performed

all

the

updates

that

server

could

have

performed

The

rule

in

general

is

that

two

servers

can

recover

if

the

server

that

did

not

fail

has

a

higher

sequence

number

as

in

this

case

it

is

certain

that

the

new

member

has

not

formed

a

group

with

the

-LRB-

now

-RRB-

unavailable

member

in

the

meantime

We

will

incorporate

this

improvement

in

our

directory

service

in

the

near

future

Experimental

Comparison

Both

the

RPC

and

group

service

are

operational

The

RPC

service

has

been

in

daily

use

for

over

two

years

The

group

directory

service

has

been

used

in

an

experimental

environment

for

several

months

and

will

shortly

replace

the

RPC

version

Both

directory

services

run

on

the

same

hardware

machines

comparable

to

a

connected

by

Mbit/s

Ethernet

When

a

message

is

sent

using

SendToGroup

the

Amoeba

kernel

uses

the

Ethernet

multicast

capability

to

send

the

message

in

one

packet

to

all

members

in

the

specified

group

The

Bullet

servers

run

on

and

are

equipped

with

Wren

IV

SCSI

disks

Performance

Experiments

with

Single

Client

We

have

measured

the

failure-free

performance

of

three

kinds

of

operations

The

results

are

shown

in

Figure

The

first

experiment

measures

the

time

to

append

a

new

-LRB-

name

capability

-RRB-

pair

to

a

directory

and

delete

it

subsequently

-LRB-

e.g.

appending

and

deleting

a

name

for

a

temporary

file

-RRB-

The

second

experiment

measures

the

time

to

create

a

file

register

its

capability

with

the

directory

service

look

up

the

name

read

the

file

back

from

the

file

service

and

delete

the

name

from

the

directory

service

This

corresponds

to

the

use

of

a

temporary

file

that

is

the

output

of

the

first

phase

of

a

compiler

and

then

is

used

as

an

input

file

for

the

second

phase

Thus

the

first

experiment

measures

only

the

directory

service

while

the

second

experiment

measures

both

the

directory

and

file

service

The

third

experiment

measures

the

performance

of

the

directory

server

for

read

operations

Sun

Group

Operation

Group

RPC

NFS

NVRAM

-LRB-

copies

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Append-delete

Tmp

file

215 277 111

Directory

lookup

Fig.

Performance

of

three

kinds

of

directory

operations

for

three

different

Amoeba

implementations

and

for

one

UNIX

implementation

All

times

are

in

msec

For

the

append-delete

test

and

for

the

tmp

file

test

the

implementation

using

group

communication

is

slightly

more

efficient

than

the

one

using

RPC

Thus

although

the

group

directory

service

is

triplicated

and

the

RPC

implementation

is

only

duplicated

the

group

directory

service

is

more

efficient

The

reason

is

that

the

RPC

implementation

uses

one

additional

disk

operation

to

store

intentions

and

the

new

sequence

number

For

read

operations

the

performance

of

all

implementations

is

the

same

Read

operations

do

not

involve

any

disk

operations

as

all

implementations

cache

recently

used

directories

in

RAM

and

involve

only

one

server

For

comparison

reasons

we

ran

the

same

experiments

using

Sun

NFS

the

results

are

listed

in

the

third

column

The

measurements

were

run

on

and

the

file

used

was

located

in

usr/tmp

NFS

does

not

provide

any

fault

tolerance

or

consistency

-LRB-

e.g.

if

another

client

has

cached

the

directory

this

copy

will

not

be

updated

consistently

when

the

original

is

changed

-RRB-

Compared

to

NFS

providing

high

reliability

and

availability

costs

a

factor

of

in

performance

for

the

append-delete

test

and

in

performance

for

the

tmp

file

test

The

dominant

cost

in

providing

a

fault-tolerant

directory

service

is

the

cost

for

doing

the

disk

operations

Therefore

we

have

implemented

a

third

version

of

the

directory

service

which

does

not

perform

any

disk

operations

in

the

critical

path

Instead

of

directly

storing

modified

directories

on

disk

this

implementation

stores

the

modifications

to

a

directory

in

a

NonVolatile

RAM

-LRB-

NVRAM

-RRB-

When

the

server

is

idle

or

the

NVRAM

is

full

it

applies

the

modifications

logged

in

NVRAM

to

the

directories

stored

on

disk

Because

NVRAM

is

a

reliable

medium

this

implementation

provides

the

same

degree

of

fault

tolerance

as

the

other

implementations

while

the

performance

is

better

A

similar

optimization

has

been

used

in

-LSB-

Daniels

et

al.

Liskov

et

al.

Hariri

et

al.

-RSB-

Using

NVRAM

some

sequences

of

directory

operations

do

not

require

any

disk

operations

at

all

Consider

the

use

of

tmp

A

file

written

in

tmp

is

often

deleted

shortly

after

it

is

used

If

the

append

operation

is

still

logged

in

NVRAM

when

the

delete

is

performed

then

both

the

append

and

the

delete

modifications

to

tmp

can

be

removed

from

NVRAM

without

executing

any

disk

operations

at

all

We

have

implemented

and

measured

a

version

of

the

directory

service

that

uses

NVRAM

Using

group

communication

and

NVRAM

the

performance

improvements

for

the

experiments

are

enormous

-LRB-

see

fourth

column

in

Fig.

-RRB-

This

implementation

is

and

times

more

efficient

than

the

pure

group

implementation

The

implementation

based

on

NVRAM

is

even

faster

than

Sun

NFS

which

provides

less

fault

tolerance

and

has

a

lower

availability

If

the

RPC

service

would

have

been

implemented

with

NVRAM

one

could

expect

similar

performance

improvements

Performance

Experiments

with

Multiple

Clients

To

determine

the

performance

of

the

directory

services

for

multiple

clients

we

ran

three

additional

experiments

The

first

experiment

measures

the

throughput

for

read

operations

and

its

results

are

depicted

in

Figure

The

graph

shows

the

total

number

of

directory

lookup

operations

that

were

processed

by

a

directory

service

for

a

varying

number

of

clients

A

rough

estimate

of

the

maximum

number

of

lookup

operations

that

can

be

processed

per

second

can

be

easily

computed

The

time

needed

by

a

server

to

process

a

read

operation

is

roughly

equal

to

msec

-LRB-

the

time

for

a

lookup

operation

minus

the

time

to

perform

an

RPC

with

the

server

-RRB-

The

maximum

number

of

read

operation

per

server

is

therefore

per

second

Thus

the

upper

bound

on

read

operations

for

the

group

service

is

per

second

and

for

the

RPC

directory

service

it

is

per

second

Neither

server

achieves

the

upper

bounds

because

the

client

requests

are

not

evenly

distributed

among

the

servers

The

first

time

a

client

performs

a

RPC

with

some

service

its

kernel

locates

the

service

by

sending

a

broadcast

message

containing

the

port

p

for

the

requested

service

Every

server

that

listens

to

the

port

p

answers

with

an

RPC

HEREIS

message

The

client

's

kernel

stores

the

network

address

for

each

server

that

answers

in

its

port

cache

and

sends

the

request

to

the

first

server

that

replied

If

at

some

point

one

of

the

servers

is

busy

and

is

not

listening

to

p

when

a

request

comes

in

its

kernel

sends

a

NOTHERE

back

to

the

client

's

kernel

The

client

's

kernel

removes

the

server

's

network

address

from

its

port

cache

and

selects

another

server

from

its

port

cache

or

locates

the

service

again

if

its

port

cache

does

not

contain

an

alternative

This

heuristic

for

choosing

a

server

is

not

optimal

Some

clients

may

pick

the

same

server

while

another

server

is

idle

From

the

graph

one

can

see

this

happen

This

possibility

was

also

reflected

in

our

measurements

The

numbers

depicted

are

averages

over

a

large

number

of

runs

but

the

standard

devia

Throughput

-LRB-

#lookup

sec

-RRB-

Number

of

clients

d

d

d

d

d

d

d

d

d

d

Group

service

Group

service

NVRAM

RPC

service

Fig.

Throughput

for

read

operations

tion

is

high

In

some

runs

the

standard

deviation

was

almost

operations

per

second

The

heuristic

however

is

good

enough

that

with

higher

load

the

clients

requests

are

evenly

distributed

among

the

servers

One

can

also

conclude

from

the

graph

that

the

RPC

directory

service

can

support

fewer

clients

than

the

group

service

The

RPC

directory

service

gets

overloaded

with

requests

per

second

whereas

the

group

service

gets

overloaded

with

requests

per

second

Figure

shows

the

throughput

for

the

append-delete

test

This

experiment

measures

the

maximum

number

of

pairs

of

append-delete

operations

that

the

service

can

support

per

second

Again

an

upper

bound

can

be

easily

estimated

for

each

service

Processing

a

pair

of

append-delete

operations

takes

roughly

msec

in

the

group

NVRAM

service

msec

for

the

group

service

and

for

the

RPC

service

As

write

operations

can

not

be

performed

in

parallel

the

upper

bounds

per

service

are

and

All

three

implementations

reach

the

upper

bound

Throughput

-LRB-

#append

delete/sec

-RRB-

Number

of

clients

d

d

d

d

d

d

d

d

d

d

Group

service

Group

service

NVRAM

RPC

service

Fig.

Throughput

for

pairs

of

append-delete

operations

As

append

and

delete

operations

are

both

write

operations

the

actual

write

throughput

is

twice

as

high

Discussion

and

Comparison

To

make

a

fair

comparison

between

the

group

directory

service

and

the

RPC

directory

service

is

hardly

possible

as

both

services

assume

different

failure

modes

The

RPC

service

is

duplicated

and

does

not

provide

consistency

in

the

face

of

network

partitions

whereas

the

group

service

is

triplicated

and

does

provide

consistency

in

the

face

of

network

partitions

Furthermore

the

RPC

implementation

employs

lazy

replication

whereas

the

group

implementation

employs

active

replication

resulting

in

a

higher

degree

of

reliability

and

availability

for

the

group

directory

service

After

the

RPC

directory

service

has

performed

an

update

on

a

directory

the

new

directory

is

stored

on

only

one

Bullet

file

If

the

Bullet

server

storing

this

file

crashes

before

the

second

replica

is

generated

the

directory

will

become

unavailable

In

the

group

directory

service

this

can

not

happen

because

the

service

creates

all

replicas

at

about

the

same

time

before

the

client

is

told

that

the

update

has

succeeded

Although

the

group

service

provides

a

higher

reliability

and

availability

its

protocols

for

normal

operation

-LRB-

no

failures

-RRB-

are

as

simple

as

the

RPC

protocols

The

group

recovery

protocols

are

more

com

plex

but

this

is

due

to

the

fact

that

the

group

service

is

built

out

of

three

servers

instead

of

out

of

two

A

three

server

implementation

of

the

RPC

service

would

require

a

similar

protocol

for

recovery

as

used

in

the

group

service

The

performance

of

the

group

directory

service

is

better

than

the

RPC

directory

service

This

is

however

mainly

due

to

the

fact

that

the

group

implementation

avoids

one

disk

write

The

RPC

directory

service

could

have

been

implemented

in

such

a

way

that

this

implementation

also

avoids

this

additional

disk

write

Such

an

implementation

is

likely

to

have

the

same

performance

as

the

group

implementation

On

the

other

hand

if

the

RPC

directory

service

had

been

triplicated

it

would

have

had

to

implement

two-phase

locking

-LSB-

Eswaran

et

al.

-RSB-

resulting

in

higher

communication

overhead

compared

with

the

group

implementation

Summarizing

the

group

directory

service

is

hard

to

compare

with

the

RPC

directory

service

due

to

differences

in

the

failure

semantics

and

differences

in

the

implementations

that

are

not

related

to

using

group

communication

or

using

RPC

However

we

believe

that

the

comparison

gives

enough

insight

in

all

the

design

choices

and

implementation

issues

that

a

RPC

directory

service

with

the

same

specification

as

the

group

directory

service

will

be

more

complicated

and

have

a

worse

performance

than

a

group

implementation

There

is

an

extensive

literature

on

designing

and

building

fault-tolerant

systems

covering

both

practical

and

theoretical

research

It

is

outside

the

scope

of

this

paper

however

to

review

all

this

work

Instead

we

focus

on

a

number

of

actual

implementations

of

fault-tolerant

file

and

directory

services

in

systems

similar

to

Amoeba

Marzullo

and

Schmuck

describe

a

fault-tolerant

implementation

of

Sun

's

Network

File

System

-LRB-

NFS

-RRB-

using

the

Isis

toolkit

-LSB-

Marzullo

and

Schmuck

-RSB-

As

the

authors

did

not

want

to

change

either

the

client

side

or

the

server

side

of

NFS

they

introduced

an

extra

level

of

indirection

Client

processes

do

not

talk

directly

with

the

file

service

but

go

through

an

intermediate

process

called

an

agent

The

agents

hide

from

the

clients

that

the

file

service

is

replicated

and

use

internally

one

of

Isis

's

broadcast

primitives

to

keep

their

state

consistent

The

agents

update

the

replicas

of

a

file

using

regular

Sun

RPC

because

to

employ

broadcast

would

have

meant

changing

the

file

servers

to

use

Isis

Harp

is

another

approach

to

increase

the

fault

tolerance

of

NFS

-LSB-

Liskov

et

al.

-RSB-

Unlike

Marzullo

and

Schmuck

the

authors

of

Harp

decided

to

change

the

file

server

to

avoid

an

extra

level

of

indirection

Harp

is

based

on

a

primary

copy

protocol

-LSB-

Alsberg

and

Day

-RSB-

Clients

communicate

with

one

designated

server

called

the

primary

to

perform

operations

The

other

servers

are

termed

secondaries

On

a

write

operation

the

primary

first

sends

the

results

to

secondaries

before

sending

a

reply

to

the

client

All

servers

store

the

result

in

NVRAM

and

copy

the

result

lazily

to

disk

to

improve

performance

If

the

primary

crashes

the

secondaries

elect

a

new

primary

Another

fault-tolerant

file

system

is

Coda

-LSB-

Satyanarayanan

-RSB-

Coda

replicates

files

at

the

server

side

and

also

caches

files

at

the

client

side

The

clients

cache

whole

files

so

even

if

all

servers

fail

the

clients

are

able

to

continue

working

with

the

cached

files

If

client

and

servers

are

connected

callbacks

are

used

to

keep

the

caches

of

the

clients

and

servers

consistent

The

servers

themselves

use

active

replication

and

an

optimistic

variant

of

the

read-one

write-all

policy

to

keep

replicas

consistent

The

implementation

is

based

on

a

parallel

RPC

mechanism

that

exploits

the

multicast

capability

of

a

network

Another

approach

to

a

fault-tolerant

distributed

file

system

is

Echo

-LSB-

Hisgen

et

al.

-RSB-

Like

Harp

Echo

uses

a

primary

copy

scheme

Unlike

Harp

it

does

not

perform

replication

at

the

file

level

but

at

the

level

of

an

array

of

disk

blocks

One

of

the

reasons

for

doing

so

it

that

Echo

uses

multi-ported

disks

which

can

be

accessed

by

multiple

servers

The

multi-ported

disks

and

replication

at

the

level

of

disk

blocks

allow

a

primary

to

continue

working

even

if

all

secondaries

have

failed

The

primary

can

directly

write

to

all

disks

without

having

to

go

through

the

secondaries

A

fault-tolerant

directory

service

is

described

by

Mishra

Peterson

and

Schlichting

-LSB-

Mishra

et

al.

-RSB-

This

directory

service

also

uses

active

replication

but

it

is

based

on

the

assumption

that

operations

are

idempotent

It

uses

Psync

's

protocols

to

enforce

an

ordering

on

the

messages

sent

by

the

servers

-LSB-

Peterson

et

al.

-RSB-

To

enhance

concurrency

of

the

directory

service

operations

the

directory

service

uses

the

partial

ordering

and

the

property

that

some

operations

are

commutative

-LRB-

e.g.

list

directory

and

lookup

entry

-RRB-

To

be

able

to

recover

the

servers

checkpoint

their

state

to

nonvolatile

storage

Using

the

checkpoint

and

the

partial

order

among

messages

the

service

can

reconstruct

the

state

before

a

failure

Daniels

and

Spector

describe

an

algorithm

designed

for

replicated

directories

-LSB-

Daniels

and

Spector

Bloch

et

al.

-RSB-

Their

algorithm

is

based

on

Gifford

's

weighted

voting

-LSB-

Gifford

-RSB-

The

algorithm

exploits

that

many

operations

on

a

single

directory

entry

can

be

performed

in

parallel

if

the

operations

access

different

entries

Simulations

done

by

the

authors

show

that

the

additional

cost

for

their

algorithm

is

low

while

it

provides

better

performance

Conclusion

We

have

tried

to

support

the

claim

that

a

distributed

system

should

not

only

support

RPC

but

group

communication

as

well

Group

communication

allows

simpler

and

more

efficient

implementations

of

a

large

class

of

distributed

applications

As

an

example

to

demonstrate

the

claim

we

looked

in

detail

at

the

design

and

implementation

of

a

fault-tolerant

directory

service

Although

a

completely

fair

comparison

is

not

possible

due

to

differences

in

failure

semantics

and

differences

in

the

implementations

unrelated

to

using

group

communication

and

RPC

we

nevertheless

claim

that

the

directory

service

using

group

communication

is

not

only

easier

to

implement

but

also

more

efficient

Another

important

conclusion

of

our

study

is

that

disk

operations

are

the

major

performance

bottleneck

in

providing

fault

tolerance

By

using

a

relatively

new

but

already

wide-spread

technology

NVRAM

the

performance

of

the

directory

service

for

update

operations

improves

by

an

order

of

magnitude

The

group

directory

service

allows

for

lookup

operations

per

second

and

update

operations

per

second

Acknowledgements

Henri

Bal

Leendert

van

Doorn

Elmootazbellah

Nabil

Elnozahy

Robbert

van

Renesse

Mark

Wood

and

Willy

Zwaenepoel

provided

comments

on

drafts

of

this

paper

References

Alsberg

P.

and

Day

J.

A

Principle

for

Resilient

Sharing

of

Distributed

Resources

Proc

Second

International

Conference

on

Software

Engineering

pp.

Oct.

Barrett

P.

A.

Hilborne

A.

M.

Verissimo

P.

Rodrigues

L.

Bond

P.

G.

Seaton

D.

T.

and

Speirs

N.

A.

The

Extra

Performance

Architecture

-LRB-

XPA

-RRB-

Proc

International

Symposium

on

Fault-Tolerant

Computing

pp.

Newcastle

UK

June

Bernstein

P.

A.

Hadzilacos

V.

and

Goodman

N.

Concurrency

Control

and

Recovery

in

Database

Systems

Addison-Wesley

Reading

MA

Birman

K.

P.

and

Joseph

T.

A.

Exploiting

Virtual

Synchrony

in

Distributed

Systems

Proc

Eleventh

ACM

Symposium

on

Operating

Systems

Principles

pp.

Austin

TX

Nov.

Birrell

A.

D.

and

Nelson

B.

J.

Implementing

Remote

Procedure

Calls

ACM

Trans

Comp

Syst

Vol

No.

pp.

Feb.

Bloch

J.

J.

Daniels

D.

S.

and

Spector

A.

Z.

A

Weighted

Voting

Algorithm

for

Replicated

Directories

Journal

of

the

ACM

Vol

No.

pp.

Oct.

Cheriton

D.

R.

and

Mann

T.

P.

Decentralizing

a

Global

Naming

Service

for

Improved

Performance

and

Fault

Tolerance

ACM

Trans

Comp

Syst.

Vol

No.

pp.

May

Cheriton

D.

R.

and

Zwaenepoel

W.

Distributed

Process

Groups

in

the

V

kernel

ACM

Trans

Comp

Syst.

Vol

No.

pp.

May

Cristian

F.

Understanding

Fault-Tolerant

Distributed

Systems

Commun

ACM

Vol

No.

pp.

Feb.

Curtis

R.

and

Wittie

L.

Global

Naming

in

Distributed

Systems

IEEE

Software

pp.

July

Daniels

D.

S.

and

Spector

A.

Z.

An

Algorithm

for

Replicated

Directories

Proc

Second

Annual

Symposium

on

Principles

of

Distributed

Computing

pp.

Montreal

Canada

Aug.

Daniels

D.

S.

Spector

A.

Z.

and

Thompson

D.

S.

Distributed

Logging

for

Transaction

Processing

Proc

ACM

SIGMOD

Annual

Conference

pp.

San

Francisco

CA

May

Davidson

S.

B.

Garcia-Molina

H.

and

Skeen

D.

Consistency

in

Partitioned

Networks

ACM

Computing

Surveys

Vol

No.

pp.

Sep.

El

Abbadi

A.

Skeen

D.

and

Cristian

F.

An

Efficient

Fault-Tolerant

Algorithm

for

Replicated

Data

Management

Proc

Fifth

Symposium

on

Principles

of

Database

Systems

pp.

Portland

OR

Mar.

Eswaran

K.

P.

Gray

J.

N.

Lorie

R.

A.

and

Traiger

I.

L.

The

Notion

of

Consistency

and

Predicate

Locks

in

a

Database

System

Commun

ACM

Vol

No.

pp.

Nov.

Gifford

D.

K.

Weighted

Voting

for

Replicated

Data

Proc

Seventh

ACM

Symposium

on

Operating

System

Principles

pp.

Pacific

Grove

CA

Oct.

Gray

J.

N.

Notes

on

Database

Operating

Systems

Vol

pp

in

Operating

Systems

An

Advanced

Course

Lecture

Notes

in

Computer

Science

Springer

Verlag

New

York

Hariri

S.

Choudhary

A.

and

Sarikaya

B.

Architectural

Support

for

Designing

Fault-Tolerant

Open

Distributed

Systems

IEEE

Computer

Vol

No

pp.

June

Hisgen

A.

Birrell

A.

D.

Jerian

C.

Mann

T.

Schroeder

M.

and

Swart

C.

Granularity

and

Semantic

Level

of

Replication

in

the

Echo

Distributed

File

System

IEEE

TCOS

Newsletter

Vol

No.

pp

Kaashoek

M.

F.

and

Tanenbaum

A.

S.

Group

Communication

in

the

Amoeba

Distributed

Operating

System

Proc

Eleventh

International

Conference

on

Distributed

Computing

Systems

pp.

Arlington

TX

May

Kaashoek

M.

F.

Van

Renesse

R.

Van

Staveren

H.

and

Tanenbaum

A.

S.

FLIP

an

Internetwork

Protocol

for

Supporting

Distributed

Systems

ACM

Trans

Comp

Syst.

-LRB-

accepted

for

publication

-RRB-

Lampson

B.

W.

Atomic

Transactions

pp.

in

Distributed

Systems

Architecture

and

Implementation

Lecture

and

Notes

in

Computer

Science

Springer

Verlag

Berlin

Lampson

B.

W.

Designing

a

Global

Name

Service

Proc

Fifth

Annual

Symposium

on

Principles

of

Distributed

Computing

pp.

Calgary

Canada

Aug.

Liskov

B.

Ghemawat

S.

Gruber

R.

Johnson

P.

Shrira

L.

and

Williams

M.

Replication

in

the

Harp

File

System

Proc

Thirteenth

Symposium

on

Operating

System

Principles

pp.

Pacific

Grove

CA

Oct.

Marzullo

K.

and

Schmuck

F.

Supplying

High

Availability

with

a

Standard

Network

File

System

Proc

Eighth

International

Conference

on

Distributed

Computing

Systems

pp.

San

Jose

CA

June

Mishra

S.

Peterson

L.

L.

and

Schlichting

R.

D.

Implementing

Fault-Tolerant

Replicated

Objects

Using

Psync

Proc

Eighth

Symposium

on

Reliable

Distributed

Systems

Seattle

WA

Oct.

Mullender

S.

J.

Van

Rossum

G.

Tanenbaum

A.

S.

Van

Renesse

R.

and

Van

Staveren

H.

Amoeba

A

Distributed

Operating

System

for

the

IEEE

Computer

Vol

No.

pp.

May

Navaratnam

S.

Chanson

S.

and

Neufeld

G.

Reliable

Group

Communication

in

Distributed

Systems

Proc

Eighth

International

Conference

on

Distributed

Computing

Systems

pp.

San

Jose

CA

June

Ousterhout

J.

K.

Cherenson

A.

R.

Douglis

F.

Nelson

M.

N.

and

Welch

B.

B.

The

Sprite

Network

Operating

System

IEEE

Computer

pp.

Feb.

Peterson

L.

L.

Buchholtz

N.

C.

and

Schlichting

R.

D.

Preserving

and

Using

Context

Information

in

IPC

ACM

Trans

Comp

Syst.

Vol

No.

pp.

Aug.

Pike

R.

Presto

D.

Thompson

K.

and

Trickey

H.

Plan

from

Bell

Labs

EKUUG

Summer

pp.

London

July

Rashid

R.

F.

Threads

of

a

New

System

Unix

Review

Vol

No.

p.

Aug.

Satyanarayanan

M.

Scalable

Secure

and

Highly

Available

Distributed

File

Access

IEEE

Computer

Vol

No.

pp

May

Schroeder

M.

D.

Birrell

A.

D.

and

Needham

R.

M.

Experience

with

Grapevine

The

Growth

of

a

Distributed

System

ACM

Trans

Comp

Syst.

Vol

No.

pp.

Feb.

Shrivastava

S.

K.

Ezhilchelvan

P.

D.

Speirs

N.

A.

Tao

S.

and

Tully

A.

Principal

Features

of

the

VOLTAN

Family

of

Reliable

Node

Architectures

for

Distributed

Systems

IEEE

Trans

on

Computers

Vol

No.

pp.

May

Skeen

D.

Determining

the

Last

Process

to

Fail

ACM

Trans

Comp

Syst.

Vol

No.

pp.

Feb.

Tanenbaum

A.

S.

Kaashoek

M.

F.

and

Bal

H.

E.

Parallel

Programming

Using

Shared

Objects

and

Broadcasting

IEEE

Computer

Vol

pp.

Aug.

Tanenbaum

A.

S.

Van

Renesse

R.

Van

Staveren

H.

Sharp

G.

Mullender

S.

J.

Jansen

A.

and

Van

Rossum

G.

Experiences

with

the

Amoeba

Distributed

Operating

System

Commun

ACM

Vol

No

pp.

Dec.

Van

Renesse

R.

The

Functional

Processing

Model

Ph.D.

Thesis

Vrije

Universiteit

Amsterdam

amoeba
directory service
distributed applications
distributed systems
group communication
multicast
operating