Comprehensive

Low-overhead

Process

Recovery

Based

on

Quasi-synchronous

Checkpointing

D.

Manivannan

and

M.

Singhal

Distributed

Systems

Research

Group

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

E-mail

fmanivann,singhalg@cis.ohio-state.edu

Abstract

In

this

paper

we

propose

a

low-overhead

recovery

algorithm

based

on

a

quasisynchronous

checkpointing

algorithm

The

checkpointing

algorithm

preserves

process

autonomy

by

allowing

them

to

take

checkpoints

asynchronously

and

uses

communicationinduced

checkpoint

coordination

for

the

progression

of

the

recovery

line

which

helps

bound

rollback

propagation

during

a

recovery

Thus

it

has

the

easeness

and

low

overhead

of

asynchronous

checkpointing

and

the

recovery

time

advantages

of

synchronous

checkpointing

There

is

no

extra

message

overhead

involved

during

checkpointing

and

the

additional

checkpointing

overhead

is

nominal

The

checkpointing

algorithm

ensures

the

existence

of

a

recovery

line

consistent

with

the

latest

checkpoint

of

any

process

all

the

time

The

recovery

algorithm

exploits

this

feature

to

restore

the

system

to

a

state

consistent

with

the

latest

checkpoint

of

a

failed

process

in

the

event

of

a

failure

The

recovery

algorithm

has

no

domino

effect

and

a

failed

process

needs

only

to

rollback

to

its

latest

checkpoint

and

request

the

other

processes

to

roll

back

to

a

consistent

checkpoint

It

uses

selective

pessimistic

message

logging

at

the

receiver

end

to

handle

the

messages

lost

due

to

rollback

Neither

the

recovery

algorithm

nor

the

checkpointing

algorithm

requires

the

channels

to

be

FIFO

We

do

not

use

vector

timestamps

for

determining

dependency

between

checkpoints

which

result

in

high

message

overhead

during

failure-free

operation

if

the

number

of

processes

is

large

Introduction

A

distributed

system

consists

of

N

sequential

processes

denoted

by

PN

The

concurrent

execution

of

all

the

processes

on

a

network

of

processors

is

called

a

distributed

computation

The

processes

do

not

share

a

global

memory

or

a

global

physical

clock

Message

passing

is

the

only

way

for

processes

to

communicate

with

one

another

The

computation

is

asynchronous

each

process

evolves

at

his

own

speed

and

messages

are

exchanged

through

communication

channels

whose

transmission

delays

are

finite

but

arbitrary

We

assume

that

messages

are

not

lost

altered

or

spuriously

introduced

No

assumption

is

made

about

the

FIFO

nature

of

the

channels

Processes

are

fail-stop

All

failures

are

detected

immediately

and

result

in

halting

failed

processes

and

initiating

recovery

action

-LSB-

-RSB-

A

process

can

be

inactive

due

to

failure

for

an

arbitrarily

long

but

finite

time

In

a

distributed

system

the

states

of

processes

depend

on

one

another

due

to

inter-process

communication

So

when

a

process

P

rolls

back

after

a

failure

the

processes

that

have

states

directly

or

transitively

dependent

on

P

's

state

are

forced

to

roll

back

The

use

of

checkpoints

on

a

stable

storage

and

rollback-recovery

protocols

are

well

established

techniques

for

dealing

with

process

failures

in

a

distributed

system

When

a

failure

occurs

a

rollback

protocol

uses

the

checkpoints

and

message

logs

to

restore

the

system

to

a

consistent

global

state

-LSB-

-RSB-

By

a

consistent

global

state

we

mean

that

if

the

receipt

operation

of

a

message

has

been

recorded

in

the

state

of

some

process

then

the

send

operation

of

that

message

must

have

been

recorded

also

In

the

literature

several

checkpointing

schemes

have

been

proposed

for

distributed

systems

They

can

be

broadly

classified

into

two

categories

-LCB-

asynchronous

and

synchronous

In

asynchronous

checkpointing

-LSB-

-RSB-

processes

take

checkpoints

periodically

without

any

coordination

with

others

To

recover

from

a

failure

a

process

communicates

with

other

processes

to

determine

if

their

local

states

are

causally

related

If

they

are

processes

that

received

messages

which

are

responsible

for

causal

dependencies

roll

back

to

eliminate

these

causal

dependencies

This

process

is

repeated

until

the

local

states

of

all

the

processes

are

free

from

causal

dependencies

This

approach

allows

maximum

process

autonomy

and

has

low

checkpointing

overhead

However

this

approach

may

suffer

from

the

domino

effect

in

which

the

processes

roll

back

repeatedly

while

determining

a

consistent

set

of

checkpoints

To

reduce

domino

effect

Kim

et

al.

-LSB-

-RSB-

Venkatesh

et

al.

-LSB-

-RSB-

and

Xu

et

al.

-LSB-

-RSB-

use

the

dependency

tracking

and

insert

checkpoints

before

processing

a

new

message

that

introduces

dependency

Message

logging

-LSB-

-RSB-

and

message

reordering

-LSB-

-RSB-

have

been

suggested

in

the

literature

to

cope

with

the

domino

effect

Asynchronous

checkpointing

requires

multiple

checkpoints

to

be

stored

at

each

process

Thus

storage

requirement

may

be

large

This

problem

can

be

solved

by

periodically

establishing

a

recovery

and

deleting

the

checkpoints

that

precede

the

checkpoints

in

the

recovery

line

In

synchronous

checkpointing

schemes

domino-free

recovery

is

achieved

by

sacrificing

process

autonomy

and

incurring

extra

message

overhead

during

checkpointing

In

this

approach

processes

synchronize

their

checkpointing

activities

so

that

a

globally

consistent

set

of

checkpoints

is

always

maintained

in

the

system

-LSB-

-RSB-

The

storage

requirement

for

the

checkpoints

is

minimum

because

each

process

keeps

only

one

checkpoint

in

the

stable

storage

at

any

given

time

Process

execution

may

have

to

be

suspended

during

the

checkpointing

coordination

as

in

-LSB-

-RSB-

resulting

in

performance

degradation

globally

consistent

set

of

checkpoints

Paper

Objectives

In

this

paper

we

first

present

a

quasi-synchronous

checkpointing

algorithm

which

has

the

easeness

and

low

overhead

of

asynchronous

checkpointing

and

the

recovery

time

advantages

of

synchronous

checkpointing

It

ensures

the

existence

of

a

recovery

line

consistent

with

the

latest

checkpoint

of

any

process

all

the

time

We

then

present

a

low-overhead

recovery

algorithm

based

on

the

quasi-synchronous

checkpointing

algorithm

The

algorithm

is

fully

asynchronous

i.e

a

failed

process

needs

only

to

roll

back

to

its

latest

checkpoint

and

inform

other

processes

about

the

rollback

it

can

resume

normal

computation

without

waiting

for

other

processes

to

roll

back

to

a

consistent

sate

Messages

are

selectively

logged

at

the

receiver

end

to

handle

various

types

of

messages

that

arise

due

to

rollback

this

selective

message

logging

reduces

message

logging

overhead

during

the

normal

operation

considerably

There

is

no

extra

message

transfered

during

the

normal

computation

The

rest

of

the

paper

is

organized

as

follows

In

the

next

section

we

present

the

quasisynchronous

checkpointing

algorithm

In

Section

we

present

a

basic

recovery

algorithm

which

rolls

back

the

processes

to

a

set

of

globally

consistent

checkpoints

in

the

event

of

a

failure

In

Section

we

extend

the

basic

recovery

algorithm

to

a

comprehensive

recovery

algorithm

which

restores

the

system

to

a

consistent

state

in

the

event

of

a

failure

In

Section

we

analyze

the

overhead

involved

in

the

checkpointing

and

the

recovery

algorithms

In

Section

we

compare

our

algorithms

with

the

existing

related

work

Section

concludes

the

paper

The

Quasi-synchronous

Checkpointing

Algorithm

Activity

of

each

process

is

modeled

by

a

sequence

of

events

-LRB-

i.e.

executed

actions

-RRB-

Occurrence

of

an

event

generally

causes

a

change

of

the

local

state

of

the

process

The

local

state

of

a

process

saved

in

stable

storage

is

called

a

checkpoint

of

the

process

Each

process

takes

checkpoints

independently

Processes

also

take

checkpoints

as

a

result

of

the

reception

of

some

messages

Independently

taken

checkpoints

are

called

basic

checkpoints

and

those

triggered

by

message

receptions

are

called

forced

checkpoints

The

forced

checkpoints

help

in

advancing

the

recovery

line

Definitions

and

Notations

Each

checkpoint

of

a

process

is

assigned

a

unique

sequence

number

The

sequence

number

assigned

to

a

checkpoint

C

is

denoted

by

C

sn

The

sequence

numbers

of

the

checkpoints

of

a

process

increase

monotonically

Each

message

is

piggybacked

with

the

sequence

number

of

the

latest

checkpoint

of

the

process

sending

the

message

The

sequence

number

piggybacked

with

message

M

is

denoted

by

M

sn

The

checkpoint

with

sequence

number

m

of

process

Pi

is

denoted

by

Ci

m

The

send

and

receive

events

of

a

message

M

are

denoted

respectively

by

send

-LRB-

M

-RRB-

and

receive

-LRB-

M

-RRB-

We

say

send

-LRB-

M

-RRB-

Ci

m

if

message

M

was

sent

by

process

Pi

before

taking

the

checkpoint

Ci

m.

Also

we

say

receive

-LRB-

M

-RRB-

Ci

m

if

message

M

was

received

and

processed

by

Pi

before

taking

the

checkpoint

Ci

m.

Definition

A

set

S

CN

mNg

of

N

checkpoints

one

from

each

process

is

said

to

be

a

global

if

for

any

message

M

and

for

any

integer

i

i

N

receive

-LRB-

M

-RRB-

Ci

mi

=-RRB-

send

-LRB-

M

-RRB-

Cj

mj

for

some

j

j

N

In

the

checkpointing

algorithm

each

process

has

two

variables

sni

and

nexti

The

value

of

the

variable

sni

denotes

the

sequence

number

of

the

latest

checkpoint

taken

by

process

Pi

The

value

of

nexti

denotes

the

sequence

number

to

be

assigned

to

the

next

basic

checkpoint

that

Pi

will

take

The

variable

nexti

is

incremented

by

Pi

every

x

time

units

where

x

is

called

a

global

snapshot

or

a

recovery

line

the

smallest

of

the

checkpoint

interval

times

of

all

processes

The

main

purpose

of

nexti

is

to

keep

the

sequence

numbers

of

the

latest

checkpoints

of

the

processes

close

to

each

other

as

we

will

see

later

this

helps

in

the

progression

of

the

recovery

line

Now

we

present

the

quasi-synchronous

checkpointing

algorithm

formally

The

Quasi-synchronous

Checkpointing

Algorithm

Data

Structures

at

Process

Pi

sni

integer

-LRB-

-RRB-

fsequence

number

of

the

latest

checkpoint

initialized

to

This

is

updated

every

time

a

new

checkpoint

is

taken.g

nexti

integer

-LRB-

-RRB-

fsequence

number

to

be

assigned

to

the

next

basic

checkpoint

initialized

to

When

it

is

time

for

process

Pi

to

increment

nexti

nexti

nexti

fnexti

incremented

at

periodic

time

intervalsg

When

it

is

time

for

process

Pi

to

take

a

basic

checkpoint

If

nexti

sni

then

fskips

taking

a

basic

checkpoint

if

nexti

sni

-LRB-

i.e.

if

it

had

Take

checkpoint

C

already

taken

a

forced

checkpoint

C

with

C

sn

nexti

-RRB-

g

C

sn

nexti

f

Assigns

nexti

as

the

sequence

number

for

the

checkpoint

taken

g

sni

C

sn

When

process

Pi

sends

a

message

M

M

sn

sni

fsequence

number

of

the

current

checkpoint

piggybacked

with

message

Mg

send

-LRB-

M

-RRB-

When

Process

Pj

Receives

a

message

M

from

process

Pi

If

M

sn

snj

then

Take

checkpoint

C

C

sn

M

sn

snj

C

sn

Process

the

message

Now

we

explain

the

Quasi-synchronous

checkpointing

algorithm

-LRB-

hereafter

referred

to

as

QSA

-RRB-

with

an

example

A

system

consisting

of

three

processes

and

is

shown

in

Figure

The

basic

checkpoints

are

shown

in

the

figure

as

-LSB-

and

the

forced

checkpoints

are

shown

as

-LSB-

The

sequence

numbers

assigned

to

checkpoints

are

also

shown

in

the

figure

Here

each

process

Pi

increments

its

variable

nexti

every

x

time

units

Process

takes

a

basic

checkpoint

every

x

time

units

takes

a

basic

checkpoint

every

x

time

units

and

takes

a

basic

checkpoint

-LSB-

-LSB-

-LSB-

-LSB-

PPP

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

M

M

-LSB-

-LSB-

-LSB-

M

M

-LSB-

-LSB-

Figure

Example

illustrating

the

QSA

every

x

time

units

Message

forces

to

take

a

forced

checkpoint

with

sequence

number

before

processing

As

a

result

skips

taking

a

basic

checkpoint

with

sequence

number

Message

forces

process

to

take

a

forced

checkpoint

with

sequence

number

before

processing

because

sn

and

-LRB-

-RRB-

while

receiving

the

message

Similarly

message

forces

to

take

a

checkpoint

before

processing

the

message

and

forces

to

take

a

checkpoint

before

processing

the

message

However

does

not

force

the

receiving

process

to

take

a

checkpoint

before

processing

it

Note

that

there

may

be

gaps

in

the

sequence

numbers

assigned

to

checkpoints

For

example

process

does

not

have

checkpoints

with

sequence

numbers

and

Basic

Recovery

Algorithm

In

this

section

we

present

a

basic

recovery

algorithm

based

on

the

QSA

presented

in

Section

The

basic

recovery

algorithm

rolls

back

the

processes

to

a

global

checkpoint

in

the

event

of

the

failure

of

a

process

We

assume

that

if

a

process

fails

no

other

process

fails

until

all

the

processes

are

rolled

back

to

a

global

checkpoint

Now

we

present

the

basic

recovery

algorithm

-LRB-

hereafter

called

the

BRA

-RRB-

The

Basic

Recovery

Algorithm

When

process

Pi

fails

Roll

back

to

the

latest

checkpoint

C

send

roll

back

to

-LRB-

C

sn

-RRB-

to

all

the

other

processes

Process

Pj

on

receiving

roll

back

to

-LRB-

n

-RRB-

message

If

snj

n

then

Find

the

earliest

checkpoint

C

of

Pj

such

that

C

sn

n

Roll

back

to

C

snj

C

sn

Discard

all

the

checkpoints

beyond

C

Else

fIn

this

case

the

process

does

not

rollback

at

allg

Take

a

checkpoint

C

C

sn

n

snj

C

sn

An

Explanation

of

the

Basic

Recovery

Algorithm

The

BRA

works

as

follows

When

a

process

Pi

fails

it

rolls

back

to

its

latest

checkpoint

and

sends

roll

back

to

-LRB-

n

-RRB-

message

to

all

the

other

processes

where

n

is

the

sequence

number

of

the

latest

checkpoint

of

Pi

On

receiving

this

message

a

process

Pj

rolls

back

to

the

earliest

checkpoint

whose

sequence

number

is

n

if

there

is

no

such

checkpoint

-LRB-

i.e.

all

the

existing

checkpoints

of

Pj

have

sequence

numbers

n

-RRB-

then

it

takes

a

checkpoint

and

assigns

n

as

the

sequence

number

for

the

checkpoint

taken

So

we

can

assume

for

simplicity

that

all

the

processes

-LRB-

including

the

ones

that

just

take

a

checkpoint

-RRB-

roll

back

to

the

earliest

checkpoint

whose

sequence

number

is

n

We

prove

below

that

the

checkpoints

to

which

the

processes

roll

back

as

a

result

of

receiving

the

roll

back

to

message

indeed

form

a

global

checkpoint

The

basic

recovery

algorithm

is

asynchronous

in

other

words

after

rolling

back

a

process

can

proceed

normally

without

waiting

for

other

processes

to

rollback

For

example

in

Figure

if

process

fails

it

will

rollback

to

its

latest

checkpoint

and

send

roll

back

to

-LRB-

-RRB-

message

to

and

Upon

receiving

this

message

will

rollback

to

checkpoint

since

is

the

earliest

checkpoint

of

whose

sequence

number

is

However

since

does

not

have

a

checkpoint

whose

sequence

number

is

it

will

not

rollback

but

will

take

a

checkpoint

with

sequence

number

Correctness

of

the

Basic

Recovery

Algorithm

In

this

section

we

prove

that

the

basic

recovery

algorithm

rolls

back

all

the

processes

to

a

global

checkpoint

when

a

process

fails

As

mentioned

earlier

when

a

process

fails

and

initiates

recovery

no

other

process

fails

until

the

recovery

is

complete

Before

proving

the

correctness

of

the

recovery

algorithm

we

make

the

following

observations

from

the

checkpointing

and

recovery

algorithms

presented

above

Observation

If

Pj

rolls

back

to

the

checkpoint

Cj

mj

upon

receiving

roll

back

to

-LRB-

n

-RRB-

message

from

Pi

-LRB-

j

N

-RRB-

then

mj

n

all

checkpoints

taken

by

Pj

prior

to

checkpoint

Cj

mj

have

sequence

numbers

less

than

n.

Observation

For

any

message

M

sent

by

Pj

-LRB-

j

N

-RRB-

send

-LRB-

M

-RRB-

Cj

mj

-LRB-

-RRB-

M

sn

mj

Observation

For

any

message

M

received

by

Pj

-LRB-

j

N

-RRB-

receive

-LRB-

M

-RRB-

Cj

mj

=-RRB-

M

sn

mj

Note

that

the

converse

is

not

true

Observation

For

any

j

Pj

processes

a

received

message

M

only

after

it

had

taken

a

checkpoint

with

sequence

number

M

sn

Theorem

Assume

that

upon

receiving

roll

back

to

-LRB-

n

-RRB-

message

from

process

Pi

process

Pj

rolls

back

to

its

checkpoint

Cj

mj

-LRB-

j

N

-RRB-

Then

the

set

S

CN

mNg

is

a

global

checkpoint

Here

mi

n.

Proof

From

Observation

we

have

j

N

mj

n

-LRB-

-RRB-

Suppose

S

is

not

a

global

checkpoint

Then

there

exists

a

message

M

sent

by

some

process

Pj

to

some

process

Pk

such

that

send

-LRB-

M

-RRB-

Cj

mj

but

receive

-LRB-

M

-RRB-

Ck

mk

So

M

sn

mj

from

Observation

-LRB-

-RRB-

M

sn

mk

from

Observation

-LRB-

-RRB-

From

equations

and

we

get

n

mj

M

sn

mk

-LRB-

-RRB-

From

Observation

all

the

checkpoints

taken

by

process

Pk

prior

to

checkpoint

Ck

mk

have

sequence

numbers

less

than

n

Since

M

sn

n

Pk

must

have

processed

M

only

after

a

checkpoint

with

sequence

number

n

had

been

taken

-LRB-

from

Observation

-RRB-

Since

receive

-LRB-

M

-RRB-

Ck

mk

message

M

must

have

been

received

by

process

Pk

before

the

checkpoint

Ck

mk

was

taken

Hence

there

exists

a

checkpoint

Ck

of

Pk

that

was

taken

before

Ck

mk

such

that

n

This

contradicts

the

fact

that

all

the

checkpoints

taken

by

process

Pk

prior

to

checkpoint

Ck

mk

have

sequence

numbers

less

than

n.

Hence

our

assumption

that

S

is

not

a

global

checkpoint

is

incorrect

Hence

the

theorem

A

Comprehensive

Recovery

Algorithm

Under

the

basic

recovery

algorithm

when

a

process

fails

all

the

processes

roll

back

to

a

global

checkpoint

by

Theorem

However

this

may

not

restore

the

system

to

a

consistent

state

Restoring

the

system

to

a

consistent

state

involves

handling

the

messages

received/sent

before

and

after

the

rollback

appropriately

Generally

message

logging

along

with

checkpointing

is

used

to

handle

the

messages

appropriately

Message

logging

schemes

can

be

broadly

classified

into

two

categories

-LCB-

optimistic

and

pessimistic

In

pessimistic

message

logging

messages

received

by

a

process

are

stored

in

the

stable

storage

before

they

are

processed

This

ensures

that

messages

that

have

been

processed

are

available

in

the

stable

storage

when

required

during

recovery

Since

all

the

messages

received

by

a

process

are

stored

first

in

the

stable

storage

handling

lost

messages

becomes

simple

in

the

event

of

failure

recovery

The

drawback

of

pessimistic

message

logging

however

is

that

it

causes

performance

degradation

since

each

message

received

by

a

process

must

be

stored

in

the

stable

storage

before

processing

On

the

other

hand

in

optimistic

message

logging

as

messages

are

received

they

are

logged

to

volatile

storage

and

processed

they

are

periodically

flushed

to

a

stable

log

This

allows

the

system

to

use

its

idle

time

for

volatile

to

stable

log

flushes

resulting

in

more

efficient

use

of

the

processor

However

since

messages

stored

in

the

volatile

storage

are

lost

during

process

failure

when

a

process

rolls

back

to

a

checkpoint

due

to

a

failure

some

of

the

messages

may

not

be

available

in

the

stable

log

for

replay

Our

comprehensive

recovery

algorithm

uses

neither

pessimistic

nor

optimistic

message

logging

It

uses

selective

pessimistic

message

logging

where

only

those

messages

that

would

probably

need

to

be

replayed

if

the

process

rolls

back

to

the

checkpoint

that

precedes

the

message

reception

are

logged

into

the

message

log

associated

with

that

checkpoint

With

each

checkpoint

C

there

is

an

associated

message

log

denoted

as

C

mesglog

A

message

received

after

checkpoint

C

is

logged

into

C

mesglog

only

if

it

is

likely

to

be

replayed

if

the

process

rolls

back

to

C

in

the

future

Before

presenting

the

comprehensive

recovery

algorithm

we

classify

the

messages

that

need

to

be

handled

during

recovery

into

different

types

Classifying

messages

into

various

types

will

help

clarify

the

message

management

issues

involved

in

recovery

A

Message

Classification

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

PPPP

X

fails

recovery

line

Figure

Different

Types

of

Messages

We

use

Figure

to

help

classify

different

types

of

messages

that

need

to

be

handled

during

a

recovery

Let

us

assume

that

process

in

Figure

fails

at

X

and

rolls

back

to

its

latest

checkpoint

As

a

result

processes

and

roll

back

to

the

checkpoints

and

respectively

under

the

BRA

The

recovery

line

corresponding

to

this

failure

is

shown

in

the

figure

by

a

zigzag

line

Lost

messages

These

are

messages

whose

send

events

are

not

undone

but

whose

receive

events

are

undone

Such

messages

arise

when

a

process

rolls

back

to

a

checkpoint

prior

to

the

reception

of

the

message

while

the

sender

does

not

roll

back

to

a

checkpoint

prior

to

the

send

event

of

the

message

In

Figure

message

is

a

lost

message

Delayed

Messages

These

are

messages

whose

send

events

are

not

undone

and

whose

receive

events

are

not

recorded

because

either

the

message

was

received

while

the

receiving

process

was

down

or

the

message

was

received

after

the

receiving

process

had

recovered

For

example

message

in

Figure

is

a

delayed

message

Orphan

messages

Messages

whose

send

has

been

undone

but

whose

receive

has

not

been

undone

Orphan

messages

do

not

arise

if

processes

roll

back

to

a

global

checkpoint

So

under

BRA

orphan

messages

do

not

arise

Duplicate

Messages

Duplicate

messages

arise

due

to

replaying

of

messages

from

the

log

during

recovery

For

example

in

Figure

after

rolling

back

to

if

replays

the

message

from

its

log

then

will

be

a

duplicate

message

because

will

be

resent

by

after

its

rollback

Now

we

extend

the

BRA

to

a

comprehensive

recovery

algorithm

which

handles

all

the

above

types

of

messages

appropriately

and

restores

the

system

to

a

consistent

state

in

the

event

of

a

failure

Since

processes

roll

back

to

a

global

checkpoint

under

BRA

orphan

messages

do

not

arise

Messages

received

while

a

process

is

down

are

buffered

and

are

delivered

to

the

process

after

recovery

This

can

be

achieved

with

a

front-end

processor

or

message

spooling

-LSB-

-RSB-

Thus

we

need

only

handle

the

following

types

of

messages

delayed

messages

that

are

received

after

a

failed

process

recovers

from

failure

messages

lost

because

of

rollback

and

duplicate

messages

Restoring

the

system

to

a

consistent

state

involves

-LRB-

i

-RRB-

restoring

the

processes

to

a

global

checkpoint

and

-LRB-

ii

-RRB-

handling

the

different

types

of

messages

appropriately

Restoring

the

Processes

to

a

Global

Checkpoint

Each

recovery

has

an

associated

incarnation

number

and

a

recovery

line

number

Whenever

a

process

fails

it

initiates

recovery

with

a

new

incarnation

number

and

a

new

recovery

line

number

The

recovery

line

number

and

the

incarnation

number

determine

the

recovery

ini

tiated

by

a

process

uniquely

Each

process

Pi

keeps

the

current

incarnation

number

and

the

recovery

line

number

in

the

variables

inci

and

rec

linei

respectively

The

values

of

all

these

variables

are

kept

in

the

stable

storage

so

that

they

are

available

in

the

event

of

process

failure

Initially

i

inci

and

rec

linei

When

a

process

Pi

fails

and

restarts

it

rolls

back

to

its

latest

checkpoint

C

increments

inci

sets

rec

linei

C

sn

-LRB-

sni

-RRB-

and

sends

the

message

rollback

-LRB-

inci

rec

linei

-RRB-

to

all

other

processes

Upon

receiving

this

message

process

Pj

sets

incj

inci

and

rec

linej

rec

linei

and

rolls

back

to

its

earliest

checkpoint

whose

sequence

number

is

rec

linei

if

Pj

does

not

have

such

a

checkpoint

-LRB-

i.e

all

the

checkpoints

of

Pj

have

sequence

numbers

less

than

rec

linei

-RRB-

then

it

does

not

roll

back

but

takes

a

new

checkpoint

with

sequence

number

rec

linei

Thus

all

the

processes

roll

back

to

their

earliest

checkpoint

whose

sequence

number

is

rec

linei

The

checkpoints

to

which

the

processes

roll

back

in

response

to

the

message

rollback

-LRB-

inci

rec

linei

-RRB-

form

a

global

checkpoint

Si

-LRB-

from

Theorem

-RRB-

where

Si

fCk

mk

j

Ck

mk

is

the

earliest

checkpoint

of

Pk

whose

sequence

number

is

rec

lineig

Note

that

Si

is

uniquely

determined

by

rec

linei

So

we

call

rec

linei

the

recovery

line

number

of

the

rollback

Note

that

rec

linei

is

basically

the

sequence

number

of

the

latest

checkpoint

of

the

failed

process

Pi

In

general

if

Si

is

the

recovery

line

established

by

the

message

rollback

-LRB-

inci

rec

linei

-RRB-

then

the

checkpoints

that

lie

to

the

left

of

Si

have

sequence

numbers

less

than

rec

linei

and

those

that

lie

to

the

right

will

have

sequence

numbers

greater

than

rec

linei

This

nice

property

of

the

recovery

line

number

helps

in

determining

the

lost

delayed

and

duplicate

messages

In

Section

we

explain

how

this

property

is

exploited

in

our

algorithm

to

handle

the

various

types

of

messages

during

recovery

For

example

in

Figure

if

process

fails

at

the

point

X

then

it

rolls

back

to

its

latest

checkpoint

and

sends

rollback

-LRB-

-RRB-

message

to

all

other

processes

In

response

to

this

message

processes

and

roll

back

to

the

checkpoints

and

respectively

The

recovery

line

for

this

rollback

is

shown

by

the

zigzag

line

in

the

figure

The

recovery

line

number

for

this

recovery

is

Note

that

the

sequence

numbers

of

all

the

checkpoints

that

lie

to

the

left

of

this

recovery

line

are

less

than

and

the

sequence

numbers

of

all

the

checkpoints

that

lie

to

the

right

are

greater

than

Handling

Messages

In

addition

to

the

sequence

number

of

the

latest

checkpoint

each

message

is

piggybacked

with

the

current

recovery

line

number

and

the

current

incarnation

number

The

incarnation

number

and

the

recovery

line

number

piggybacked

with

a

message

M

are

denoted

by

M

inc

and

M

rec

line

respectively

We

selectively

log

the

received

messages

to

handle

messages

lost

during

rollback

This

selective

message

logging

helps

reducing

the

overhead

involved

in

message

logging

substantially

First

we

explain

how

messages

lost

due

to

rollback

are

determined

and

replayed

and

also

how

duplicate

messages

are

handled

Handling

Lost

and

Duplicate

Messages

When

a

process

Pj

receives

the

message

rollback

-LRB-

inci

rec

linei

-RRB-

from

a

process

Pi

it

finds

its

earliest

checkpoint

C

whose

sequence

number

is

rec

linei

combines

the

message

logs

associated

with

the

checkpoints

that

succeed

C

and

assigns

that

as

C

mesglog

in

other

words

all

the

messages

that

were

logged

after

the

checkpoint

C

form

the

message

log

of

C.

Then

it

restores

the

checkpoint

C

and

replays

from

C

mesglog

only

those

messages

whose

send

will

not

be

undone

in

other

words

a

process

replays

only

those

messages

that

originated

to

the

left

of

the

the

current

recovery

line

and

delivered

to

the

right

of

the

current

recovery

line

This

message

replaying

strategy

takes

care

of

the

lost

messages

and

eliminates

duplicate

messages

This

is

because

lost

messages

are

precisely

those

messages

which

were

sent

from

the

left

of

the

recovery

line

and

received

to

the

right

of

the

recovery

line

Since

only

lost

messages

are

replayed

duplicate

messages

duplicate

messages

do

not

arise

due

to

replay

More

formally

Rule

for

handling

lost

and

duplicate

messages

After

a

process

Pj

rolls

back

to

a

checkpoint

C

it

replays

a

message

M

from

it

message

log

if

and

only

if

it

was

received

after

the

checkpoint

C

and

M

sn

rec

linej

-LSB-

-LSB-

-LSB-

-LSB-

PPP

P

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

M

M

M

M

M

X

inc

-LSB-

M

M

Messages

sent

after

the

rollback

Messages

sent

before

the

rollback

Fails

Figure

Handling

of

messages

during

recovery

In

Figure

if

process

fails

at

the

point

X

it

rolls

back

to

its

latest

checkpoint

increments

to

sets

rec

and

sends

rollback

-LRB-

-RRB-

message

to

all

other

processes

When

receives

this

rollback

message

it

sets

rec

and

rolls

back

to

since

is

the

earliest

checkpoint

of

whose

sequence

number

is

After

rolling

back

to

replays

the

messages

and

but

does

not

replay

this

is

because

the

send

of

has

not

been

undone

by

the

send

of

has

not

been

undone

by

while

the

send

of

was

undone

by

due

to

its

rollback

to

the

checkpoint

In

the

figure

messages

sent

before

the

rollback

are

shown

by

solid

arrows

and

messages

sent

after

the

rollback

are

shown

by

broken

arrows

So

M

inc

for

all

the

messages

shown

by

solid

arrows

and

M

inc

for

all

the

messages

shown

by

broken

arrows

Next

we

discuss

how

delayed

messages

are

handled

and

also

discuss

how

messages

are

logged

selectively

To

answer

these

questions

it

is

sufficient

to

answer

the

more

general

question

of

how

a

received

message

is

handled

Handling

Delayed

Messages

and

Message

Logging

Suppose

process

Pk

receives

a

message

M

from

process

Pj

The

following

three

cases

arise

Case

-LRB-

i

-RRB-

M

was

sent

in

a

previous

incarnation

-LRB-

i.e.

M

inc

inck

-RRB-

This

means

that

Pj

was

not

aware

of

the

current

recovery

at

the

time

of

sending

the

message

M

When

Pj

comes

to

know

of

the

current

recovery

it

will

rollback

to

its

earliest

checkpoint

whose

sequence

number

is

rec

linek

So

Pj

will

rollback

prior

to

the

send

of

M

iff

M

sn

rec

linek

Hence

M

should

be

processed

if

and

only

if

M

sn

rec

linek

otherwise

it

should

be

discarded

since

its

send

will

be

undone

Thus

M

is

a

delayed

message

in

this

case

iff

M

sn

rec

linek

If

M

sn

rec

linek

then

M

is

logged

into

Cur

chkptk

mesglog

before

being

processed

because

if

Pk

has

to

rollback

in

the

future

to

the

current

checkpoint

due

to

the

failure

of

some

process

and

if

the

recovery

line

number

for

that

rollback

is

greater

than

M

sn

then

the

send

of

M

will

not

be

undone

but

its

receive

will

be

undone

and

hence

M

will

have

to

be

replayed

So

the

rule

for

determining

a

delayed

message

is

as

follows

Rule

for

determining

delayed

messages

A

message

M

received

by

a

process

Pk

is

a

delayed

message

iff

M

inc

inck

and

M

sn

rec

linek

For

example

in

Figure

is

logged

and

then

processed

by

because

sn

rec

and

inc

whereas

is

discarded

by

because

sn

rec

and

inc

Case

-LRB-

ii

-RRB-

M

was

sent

in

the

current

incarnation

-LRB-

i.e.

M

inc

inck

-RRB-

In

this

case

if

M

sn

snk

then

M

is

logged

before

being

processed

this

is

because

if

Pk

needs

to

rollback

to

the

current

checkpoint

M

will

have

to

be

replayed

if

the

recovery

line

number

of

that

rollback

would

be

M

sn

If

M

sn

snk

then

M

is

processes

after

Pj

takes

a

checkpoint

with

sequence

number

M

sn

and

M

is

not

logged

If

M

sn

snk

the

message

is

processed

without

logging

or

taking

a

new

checkpoint

From

Case

-LRB-

i

-RRB-

and

Case

-LRB-

ii

-RRB-

we

have

the

following

message

logging

rule

Message

Logging

Rule

When

a

process

Pk

receives

a

message

M

it

logs

the

message

before

processing

it

iff

-LRB-

M

inc

inck

and

M

sn

rec

linek

-RRB-

or

-LRB-

M

inc

inck

and

M

sn

snk

-RRB-

For

example

in

Figure

message

was

sent

by

process

after

it

had

recovered

and

it

was

received

by

after

had

rolled

back

In

that

case

inc

and

sn

has

to

be

logged

before

being

processed

because

if

has

to

roll

back

to

checkpoint

due

to

the

failure

of

some

process

in

the

future

then

will

have

to

replay

if

the

recovery

line

number

of

that

rollback

is

-LRB-

i.e.

or

-RRB-

Case

-LRB-

iii

-RRB-

M

was

sent

in

a

future

incarnation

-LRB-

i.e.

M

inc

inck

-RRB-

In

this

case

Pk

comes

to

know

of

a

recovery

initiated

by

a

process

through

the

message

M

So

it

sets

rec

linek

M

rec

line

and

inck

M

inc

and

rolls

back

to

its

earliest

checkpoint

with

sequence

number

M

rec

line

After

rolling

back

since

M

inc

inck

the

message

is

handled

as

in

Case

-LRB-

ii

-RRB-

In

the

figure

would

be

an

example

of

such

a

message

if

it

was

received

by

before

receiving

the

message

rollback

-LRB-

-RRB-

from

process

Salient

Features

of

the

Recovery

Algorithm

Following

are

the

important

features

of

the

recovery

algorithm

ffl

Messages

are

only

selectively

logged

So

message

logging

overhead

is

low

ffl

With

each

message

only

three

integers

are

piggybacked

No

additional

control

messages

are

exchanged

during

failure-free

operation

Thus

during

the

failure-free

operation

the

message

overhead

is

very

low

ffl

When

a

process

fails

and

sends

the

rollback

message

all

other

processes

know

exactly

to

which

checkpoint

they

have

to

rollback

and

all

the

processes

know

exactly

what

messages

need

to

be

replayed

to

cope

with

messages

lost

due

to

rollback

No

communication

among

sites

is

needed

rollback

message

sent

by

the

failed

process

is

the

only

control

message

exchanged

during

the

recovery

initiated

by

a

failed

process

ffl

Recovery

is

fully

asynchronous

No

process

needs

to

wait

for

other

processes

to

rollback

No

explicit

synchronization

is

required

during

the

recovery

After

all

the

processes

roll

back

the

system

is

restored

to

a

consistent

state

ffl

Delayed

duplicate

and

lost

messages

are

handled

easily

Handling

all

these

types

of

messages

do

not

require

any

explicit

synchronization

Thus

the

overhead

involved

during

the

recovery

time

is

very

low

ffl

A

failed

process

only

needs

to

roll

back

to

its

latest

checkpoint

There

always

exists

a

global

checkpoint

which

contains

this

checkpoint

Now

we

present

the

comprehensive

algorithm

formally

Formal

Description

of

the

Algorithm

We

first

summarize

the

notations

used

C

sn

Sequence

number

of

the

checkpoint

C.

C

mesglog

Message

log

associated

with

checkpoint

C.

M

sn

Sequence

number

piggybacked

with

the

message

M

M

inc

Incarnation

number

piggybacked

with

the

message

M

M

rec

line

Recovery

line

number

piggybacked

with

the

message

M

Cur

chkpti

The

current

checkpoint

-LRB-

i.e.

the

latest

checkpoint

-RRB-

of

process

Pi

All

the

messages

are

application

messages

except

the

rollback

message

which

is

a

control

message

sent

by

a

failed

process

Comprehensive

Recovery

Algorithm

Initialization

of

Data

Structures

at

Pi

sni

integer

-LRB-

-RRB-

fsequence

number

of

current

checkpoint

initialized

to

g

nexti

integer

-LRB-

-RRB-

fsequence

number

to

be

assigned

to

the

next

basic

checkpoint

initialized

to

inci

integer

-LRB-

-RRB-

fsequence

number

of

current

incarnation

initialized

to

g

rec

linei

integer

-LRB-

-RRB-

flatest

recovery

line

number

initialized

to

g

When

it

is

time

for

process

Pi

to

increment

nexti

nexti

nexti

fnexti

incremented

at

periodic

time

intervalsg

When

it

is

time

for

process

Pi

to

take

a

basic

checkpoint

if

nexti

sni

then

fskips

taking

a

basic

checkpoint

if

nexti

sni

-LRB-

i.e.

if

it

had

Take

checkpoint

C

already

taken

a

forced

checkpoint

C

with

C

sn

nexti

-RRB-

g

C

sn

nexti

sni

C

sn

When

process

Pi

sends

a

message

M

M

sn

sni

fsequence

number

of

the

current

checkpoint

piggybacked

with

message

Mg

M

rec

line

rec

linei

fcurrent

recovery

line

number

piggybacked

with

message

Mg

M

inc

inci

fcurrent

incarnation

number

piggybacked

with

message

Mg

send

-LRB-

M

-RRB-

When

process

Pi

receives

a

message

M

if

-LRB-

M

inc

inci

-RRB-

then

if

-LRB-

M

sn

rec

linei

-RRB-

then

fM

is

a

delayed

messageg

log

the

message

M

in

to

Cur

chkpti

mesglog

Process

the

message

M

else

Discard

the

message

M

fthis

message

will

be

resentg

else

if

-LRB-

M

inc

inci

-RRB-

then

if

-LRB-

M

sn

sni

-RRB-

then

fNote

that

in

this

case

M

sn

rec

lineig

log

M

in

to

Cur

chkpti

mesglog

else

if

-LRB-

M

sn

sni

-RRB-

then

Take

checkpoint

C

C

sn

M

sn

sni

C

sn

Process

M

else

if

-LRB-

M

inc

inci

-RRB-

then

rec

linei

M

rec

line

inci

M

inc

Roll

Back

-LRB-

i

-RRB-

fProcedure

callg

Replay

logged

messages

-LRB-

i

-RRB-

fProcedure

callg

Process

the

message

M

continue

as

normal

Recovery

initiated

by

process

Pi

after

failure

Restore

Cur

chkpti

inci

inci

rec

linei

sni

send

rollback

-LRB-

inci

rec

linei

-RRB-

to

all

other

processes

Replay

logged

messages

-LRB-

i

-RRB-

fProcedure

callg

continue

as

normal

Process

Pi

upon

receiving

rollback

-LRB-

incj

rec

linej

-RRB-

from

process

Pj

if

-LRB-

inc

j

inc

i

-RRB-

then

fif

-LRB-

inc

i

inc

j

-RRB-

Pi

is

aware

of

the

recovery

with

incarnation

inci

incj

inc

j

through

a

message

sent

by

a

process

that

already

rolled

backg

rec

linei

rec

linej

Roll

back

-LRB-

i

-RRB-

fProcedure

callg

Replay

logged

messages

-LRB-

i

-RRB-

fProcedure

callg

continue

as

normal

else

Ignore

the

rollback

message

Procedure

Roll

back

-LRB-

i

integer

-RRB-

if

rec

linei

sni

then

f

In

this

case

no

need

to

roll

back

but

takes

a

new

checkpoint

Take

checkpoint

C

that

becomes

part

of

the

recovery

lineg

C

sn

rec

linei

sni

C

sn

else

Find

the

earliest

checkpoint

C

such

that

C

sn

rec

linei

sni

C

sn

Cur

chkpti

C

Cur

chkpti

mesglog

sn

rec

linei

-LRB-

C

mesglog

-RRB-

Restore

Cur

chkpti

Delete

all

the

checkpoints

beyond

Cur

chkpti

from

the

stable

storage

Procedure

Replay

logged

messages

-LRB-

i

integer

-RRB-

for

each

message

M

Cur

chkpti

mesglog

do

if

M

sn

rec

linei

then

Replay

the

message

M

else

Delete

M

from

Cur

chkpti

mesglog

Correctness

of

the

Algorithm

First

we

summarize

the

following

observations

Observation

After

a

process

Pi

rolls

back

to

checkpoint

C

it

replays

a

message

M

from

it

message

log

if

and

only

if

it

was

received

after

the

checkpoint

C

and

M

sn

rec

linei

Observation

A

message

M

received

by

a

process

Pi

is

a

delayed

message

iff

M

inc

inci

and

M

sn

rec

linei

Observation

A

message

received

and

processed

by

Pi

after

the

current

checkpoint

C

is

logged

into

C

mesglog

if

and

only

if

either

-LRB-

M

inc

inci

and

M

sn

rec

linei

-RRB-

or

-LRB-

M

inc

inci

and

M

sn

sni

-RRB-

Also

Pi

discards

a

received

message

M

iff

-LRB-

M

inc

inci

and

M

sn

rec

linei

-RRB-

Since

rec

linei

sni

C

sn

it

follows

that

a

message

M

processed

after

a

checkpoint

C

is

logged

into

C

mesglog

iff

M

sn

C

sn

Observation

A

process

Pi

rolls

back

to

a

checkpoint

in

one

of

the

following

three

situations

Pi

fails

Pi

receives

a

rollback

message

which

carries

a

incarnation

number

inci

Pi

receives

a

message

M

with

M

inc

inci

-LRB-

In

this

case

M

was

sent

by

a

process

which

rolled

back

-RRB-

Lemma

When

a

process

Pi

rolls

back

to

a

checkpoint

C

a

message

M

that

was

received

after

C

is

a

lost

message

if

and

only

if

M

sn

rec

linei

Proof

If

Pi

rolls

back

to

C

C

is

the

earliest

checkpoint

of

Pi

such

that

C

sn

rec

linei

and

all

the

other

processes

also

roll

back

to

their

earliest

checkpoint

whose

sequence

number

is

rec

linei

So

the

send

event

of

M

will

not

be

undone

by

its

sender

iff

M

sn

rec

linei

Hence

a

message

M

that

was

received

after

C

is

a

lost

message

iff

M

sn

rec

linei

Since

messages

are

logged

selectively

we

need

to

prove

that

all

the

messages

lost

due

to

a

rollback

will

be

available

in

the

stable

storage

for

replay

This

is

proved

in

the

following

lemma

Lemma

When

a

process

Pi

rolls

back

to

a

checkpoint

C

lost

messages

are

available

in

stable

storage

for

replay

Proof

When

process

Pi

rolls

back

to

a

checkpoint

C

each

of

the

other

processes

also

rolls

back

to

its

earliest

checkpoint

whose

sequence

number

is

rec

linei

So

if

process

Pi

rolls

back

to

a

checkpoint

C

a

message

M

that

was

received

after

C

will

not

be

resent

by

its

sender

if

it

was

sent

prior

to

the

current

recovery

line

-LRB-

i.e.

if

M

sn

rec

linei

-RRB-

Since

rec

linei

C

sn

from

Observation

it

follows

that

all

lost

messages

would

be

in

the

stable

storage

Lemma

When

a

process

Pi

rolls

back

to

a

checkpoint

C

it

replays

a

message

M

from

its

stable

storage

if

and

only

if

M

is

a

lost

message

Proof

From

Observation

after

Pi

rolls

back

to

C

it

replays

a

message

M

from

its

stable

storage

if

and

only

if

M

was

received

after

checkpoint

C

and

M

sn

rec

linei

Thus

it

is

enough

to

prove

that

a

message

received

after

C

is

a

lost

message

if

and

only

if

it

was

received

after

the

checkpoint

C

was

taken

and

M

sn

rec

linei

This

follows

from

Lemma

Lemma

The

comprehensive

recovery

algorithm

handles

the

delayed

messages

appropriately

Proof

Note

that

a

message

M

received

by

a

process

Pi

is

a

delayed

message

if

and

only

if

M

inc

inci

and

M

sn

rec

linei

from

Observation

It

is

also

clear

from

the

algorithm

that

Pi

processes

a

message

M

with

M

inc

inci

if

and

only

if

M

sn

rec

linei

otherwise

it

discards

it

Thus

delayed

messages

are

handled

appropriately

Theorem

If

process

Pi

fails

and

initiates

recovery

by

sending

rollback

-LRB-

inci

rec

linei

-RRB-

message

the

system

is

restored

to

a

consistent

state

after

all

the

processes

roll

back

in

response

to

this

message

Proof

A

process

knows

about

this

rollback

message

either

after

receiving

it

directly

or

after

receiving

a

message

sent

by

a

process

that

already

received

this

rollback

message

from

Observation

In

any

case

as

a

result

of

this

rollback

message

all

the

processes

roll

back

to

their

earliest

checkpoint

whose

sequence

number

is

rec

linei

By

Theorem

the

checkpoints

to

which

the

processes

rollback

form

a

global

checkpoint

A

process

that

rolls

back

handles

lost

and

delayed

messages

appropriately

by

Lemmas

and

Duplicate

messages

do

not

arise

because

only

lost

messages

are

replayed

after

a

process

rolls

back

by

Lemma

Thus

the

system

is

restored

to

a

consistent

state

after

all

the

processes

rollback

Handling

Failures

due

to

non-fail-stop

Processes

An

important

characteristic

of

our

recovery

algorithm

is

that

it

tolerates

failures

that

go

undetected

before

taking

checkpoints

Such

errors

may

corrupt

the

checkpoints

of

one

or

more

processes

In

other

words

if

processes

are

not

fail-stop

a

failed

process

may

generate

spurious

messages

However

if

a

process

finds

at

a

later

stage

that

an

error

has

occurred

before

taking

a

certain

checkpoint

it

can

rollback

to

a

checkpoint

prior

to

that

error

event

and

request

the

other

processes

to

rollback

to

a

global

checkpoint

consistent

with

that

This

is

one

advantage

of

keeping

multiple

checkpoints

for

each

process

in

stable

storage

This

illustrated

in

Figure

In

Figure

if

process

notices

that

an

error

had

occurred

at

the

point

marked

by

X

then

it

rolls

back

to

and

sends

rollback

request

to

all

the

processes

The

recovery

line

determined

by

this

rollback

is

shown

in

the

figure

The

recovery

algorithm

remains

the

same

for

handling

such

failures

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

-LSB-

A

G

H

-LSB-

-LSB-

-LSB-

-LSB-

C

-LSB-

F

-LSB-

P

PPP

X

failed

Figure

Handling

Failures

that

go

undetected

immediately

An

Overhead

Analysis

In

checkpointing

algorithms

generally

three

kinds

of

overheads

exist

First

is

the

extra

control

messages

required

for

checkpoint

coordination

second

is

the

control

information

sent

on

computation

messages

and

the

third

is

the

number

of

checkpoints

that

need

to

be

taken

-LRB-

i.e.

the

checkpointing

overhead

-RRB-

During

the

normal

operation

each

computation

message

is

piggybacked

with

three

integers

There

is

no

other

message

overhead

So

we

only

analyze

the

checkpointing

overhead

involved

The

checkpointing

overhead

depends

on

the

run-time

communication

pattern

as

well

as

the

basic

checkpointing

pattern

Let

basicnum

denote

the

total

number

of

checkpoints

the

processes

would

take

for

a

basic

checkpointing

pattern

let

quasinum

denote

the

total

number

of

checkpoints

the

processes

would

take

for

the

same

computation

and

basic

checkpointing

pattern

when

the

QSA

is

used

We

define

the

induction

ratio

R

for

a

computation

as

R

quasinum

basicnum

We

use

R

to

analyze

the

checkpointing

overhead

of

the

QSA

One

would

desire

R

to

be

as

close

to

as

possible

If

R

then

there

is

no

extra

checkpointing

overhead

induced

by

the

QSA

If

R

then

there

is

additional

checkpointing

overhead

We

analyze

the

checkpointing

overhead

under

two

given

basic

checkpointing

patterns

-LRB-

i

-RRB-

Each

process

takes

basic

checkpoints

at

fixed

time

intervals

periodically

the

time

interval

being

same

for

all

processes

-LRB-

ii

-RRB-

Each

process

takes

basic

checkpoints

at

its

own

pace

The

results

of

the

analysis

in

these

two

cases

are

given

in

Assertions

and

respectively

below

The

proofs

of

these

assertions

can

be

found

in

-LSB-

-RSB-

Assertion

Assume

that

under

a

basic

checkpointing

pattern

each

process

takes

a

basic

checkpoint

at

the

end

of

every

x

time

units

and

the

local

clocks

of

the

processes

can

drift

by

at

most

ffi

where

ffi

x

Then

the

QSA

has

no

additional

checkpointing

overhead

for

this

basic

checkpointing

pattern

Thus

R

and

the

algorithm

ensures

that

the

recovery

line

stays

at

the

end

of

the

computation

No

process

needs

to

roll

back

to

a

distance

of

more

than

one

checkpoint

when

a

process

fails

and

initiates

recovery

Assertion

If

processes

take

basic

checkpoints

at

their

own

pace

then

the

induction

ratio

R

dQe

where

Q

is

the

maximum

of

the

ratios

of

the

lengths

of

the

basic

checkpoint

intervals

of

any

two

processes

Moreover

no

process

needs

to

rollback

to

a

distance

more

than

dQe

checkpoints

In

the

event

of

failure

the

recovery

algorithm

requires

the

failed

process

send

a

rollback

message

to

all

the

other

processes

No

other

explicit

synchronization

is

required

Only

selected

messages

are

logged

into

stable

storage

during

the

normal

operation

This

reduces

message

logging

overhead

to

a

great

extent

Since

the

QSA

ensures

the

existence

of

a

recovery

line

consistent

with

the

latest

checkpoint

failed

process

starts

from

the

latest

checkpoint

From

Assertion

a

failed

process

may

force

other

processes

to

roll

back

to

a

maximum

distance

of

Q

checkpoints

where

Q

is

the

maximum

ratio

of

the

checkpoint

intervals

of

all

the

processes

in

the

worst

case

However

if

the

sequence

number

of

the

latest

checkpoint

of

the

failed

process

is

greater

than

the

sequence

numbers

of

the

latest

checkpoints

of

all

the

other

processes

then

no

process

other

than

the

failed

process

needs

to

roll

back

they

only

need

to

take

a

checkpoint

Comparison

With

Existing

Work

The

checkpointing

algorithms

proposed

in

-LSB-

-RSB-

have

a

two-phase

structure

This

causes

processes

to

suspend

the

normal

computation

for

making

checkpoint

decisions

which

may

greatly

increase

the

overhead

during

normal

computation

The

QSA

does

not

cause

any

such

overhead

and

avoids

domino

effect

completely

during

recovery

In

Acharya

et

al.

's

-LSB-

-RSB-

asynchronous

checkpointing

algorithm

for

mobile

computing

systems

a

process

takes

a

checkpoint

whenever

a

message

reception

is

preceded

by

a

message

transmission

This

might

force

the

processes

to

take

as

many

checkpoints

as

the

number

of

messages

if

the

message

reception

and

transmission

are

interleaved

resulting

in

high

checkpointing

overhead

Wang

et

al.

-LSB-

-RSB-

proposed

lazy

checkpoint

coordination

for

bounding

rollback

propagation

Like

our

approach

their

technique

requires

the

checkpoint

number

being

piggybacked

on

the

computation

messages

so

that

the

receiving

processes

can

take

an

extra

checkpoint

when

required

However

their

approach

has

higher

checkpointing

overhead

since

processes

do

not

skip

basic

checkpoints

if

they

have

taken

forced

checkpoints

Many

of

the

existing

recovery

algorithms

use

vector

timestamps

-LSB-

-RSB-

to

track

dependency

between

checkpoints

and

events

Vector

timestamps

generally

result

in

high

message

overhead

during

failure-free

operation

The

optimistic

recovery

algorithm

proposed

by

Strom

and

Yemini

-LSB-

-RSB-

suffers

from

domino

effect

The

recovery

protocols

based

on

vector

clock

proposed

by

Peterson

and

Kearns

-LSB-

-RSB-

is

synchronous

and

tolerates

single

process

failure

It

requires

the

channels

to

be

FIFO

Recovery

proposed

by

Sistla

and

Welch

-LSB-

-RSB-

is

synchronous

for

single

process

failure

requires

the

channels

to

be

FIFO

and

uses

vector

timestamps

Smith

and

Johnson

-LSB-

-RSB-

proposed

an

asynchronous

recovery

algorithm

for

multiple

process

failures

however

the

size

of

the

vector

timestamp

is

O

-LRB-

f

-RRB-

where

f

is

the

maximum

number

of

failures

of

any

single

process

Johnson

and

Zwaenepoel

-LSB-

-RSB-

proposed

a

centralized

protocol

to

optimistically

recover

the

maximum

recoverable

state

Our

recovery

algorithm

does

not

require

vector

timestamps

Channels

need

not

be

FIFO

Recovery

is

fully

asynchronous

Recovery

requires

only

one

rollback

message

to

be

sent

to

the

other

processes

Maximum

number

of

rollbacks

of

a

process

per

failure

is

When

a

process

whose

latest

checkpoint

has

maximum

sequence

number

fails

it

will

not

force

any

other

process

to

roll

back

If

the

process

whose

latest

checkpoint

has

lowest

sequence

number

fails

it

will

force

the

other

processes

to

roll

back

to

a

distance

of

at

most

Q

checkpoints

where

Q

is

the

maximum

of

the

ratios

of

the

checkpoint

interval

lengths

of

any

two

processes

Conclusion

We

presented

a

novel

quasi-synchronous

checkpointing

algorithm

and

a

comprehensive

recovery

algorithm

based

on

it

The

checkpointing

algorithm

has

the

easeness

of

asynchronous

checkpointing

and

the

recovery

time

advantages

of

synchronous

checkpointing

Communicationinduced

checkpointing

coordination

guarantees

the

progression

of

recovery

line

The

algorithm

has

no

additional

control

message

overhead

and

has

nominal

checkpointing

overhead

It

also

guarantees

the

existence

of

a

recovery

line

consistent

with

the

latest

checkpoint

of

any

process

all

the

time

The

recovery

algorithm

exploits

this

property

of

the

checkpointing

algorithm

to

restore

the

system

to

a

consistent

state

asynchronously

in

the

case

of

a

process

failure

Unlike

other

existing

recovery

algorithms

our

recovery

algorithm

does

not

use

vector

timestamps

for

tracking

dependency

between

checkpoints

The

overhead

involved

in

the

recovery

is

very

low

since

messages

are

logged

and

replayed

selectively

and

there

is

no

explicit

synchronization

overhead

involved

during

recovery

References

-LSB-

-RSB-

A.

Acharya

B.

R.

Badrinath

and

T.

Imielinski

Checkpointing

Distributed

Applications

on

Mobile

Computing

Systems

Technical

report

Department

of

Computer

Science

Rutgers

University

-LSB-

-RSB-

R.

Attar

P.

A.

Bernstein

and

N.

Goodman

Site

Initialization

Recovery

and

Backup

in

a

Distributed

Database

System

IEEE

Trans

on

Software

Engineering

pages

-LCB-

November

-LSB-

-RSB-

B.

Bhargava

and

P.

Leu

Concurrent

Robust

Checkpointing

and

Recovery

in

Distributed

Systems

In

Proc

of

IEEE

Int

Conf

Data

Eng.

pages

-LCB-

February

-LSB-

-RSB-

B.

Bhargava

and

S.

R.

Lian

Independent

Checkpointing

and

Concurrent

Rollback

for

Recovery

in

Distributed

Systems

-LCB-

An

Optimistic

Approach

In

Proc

IEEE

Symp

Reliable

Distributed

Syst.

pages

-LCB-

-LSB-

-RSB-

Luis

Moura

e

Silva

and

Jou?ao

Gabriel

Silva

Global

Checkpointing

for

Distributed

Programs

In

Proc

Symp

Reliable

Distributed

Systems

pages

-LCB-

-LSB-

-RSB-

D.

B.

Johnson

and

W.

Zwaenepoel

Recovery

in

Distributed

Systems

Using

Optimistic

Message

Logging

and

Checkpointing

Journal

of

Algorithms

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

T.

T-Y

Juang

and

S.

Venkatesan

Crash

Recovery

with

Little

Overhead

In

Proc

of

International

Conf

on

Distributed

Comput

Syst.

pages

-LCB-

-LSB-

-RSB-

Junguk

L.

Kim

and

Taesoon

Park

An

Efficient

Protocol

for

Checkpointing

recovery

in

Distributed

Systems

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

August

-LSB-

-RSB-

K.

H

Kim

A

Scheme

for

Coordinated

Execution

of

Independently

Designed

Recoverable

Distributed

Processes

In

Proc

of

IEEE

Symp

Fault-Tolerant

Comput.

pages

-LCB-

June

-LSB-

-RSB-

R.

Koo

and

S.

Toueg

Checkpointing

and

Roll-back

Recovery

for

Distributed

Systems

IEEE

Trans

on

Software

Eng.

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

K.Tsuruoka

A.

Kaneko

and

Y.

Nishihara

Dynamic

Recovery

Schemes

for

Distributed

Process

In

Proceedings

of

IEEE

Symp

on

Reliability

in

Distributed

Software

and

Database

Systems

pages

-LCB-

-LSB-

-RSB-

T.

H.

Lai

and

T.

H.

Yang

On

Distributed

Snapshots

Information

Processing

Letters

-LCB-

May

-LSB-

-RSB-

K.

Li

J.

F.

Naughton

and

J.

S.

Plank

Checkpointing

Multicomputer

Applications

In

Proc

Symp

on

Reliable

Distributed

Systems

pages

-LCB-

-LSB-

-RSB-

D.

Manivannan

and

M.

Singhal

A

Quasi-synchronous

Algorithm

for

Checkpointing

in

Distributed

Systems

Technical

Report

The

Ohio

State

University

Department

of

Computer

and

Information

Science

-LSB-

-RSB-

S.

L.

Peterson

and

Phil

Kearns

Rollback

Based

on

Vector

Time

In

Proc

Symp

on

Reliable

Distributed

Systems

pages

-LCB-

-LSB-

-RSB-

Golden

G.

Richard

Techniques

for

Process

Recovery

in

Message

Passing

and

Distributed

Shared

Memory

Systems

PhD

thesis

The

Ohio

State

University

-LSB-

-RSB-

David

B.

Johnson

Sean

W.

Smith

and

J.

D.

Tygar

Completely

Asynchronous

Optimistic

recovery

with

Minimal

Rollbacks

In

Proc

of

International

Symposium

on

FaultTolerant

Computing

pages

-LCB-

IEEE

-LSB-

-RSB-

A.

P.

Sistla

and

J.

L.

Welch

Efficient

Distributed

Recovery

Using

Message

Logging

In

Proc

of

ACM

Symp

Principles

Distributed

Comput.

pages

-LCB-

August

-LSB-

-RSB-

R.

E.

Strom

and

S.

Yemini

Optimistic

Recovery

in

Distributed

Systems

ACM

Trans

Comput

Syst.

-LRB-

-RRB-

-LCB-

August

-LSB-

-RSB-

K.

Venkatesh

and

T.

Radhakrishnan

Optimal

Checkpointing

and

Local

Encoding

for

Domino-free

Rollback

Recovery

Information

Processing

Letters

-LCB-

July

-LSB-

-RSB-

Y.

M.

Wang

and

W.

K.

Fuchs

Lazy

checkpoint

Coordination

for

Bounding

Rollback

Propagation

Technical

Report

Coordinated

Science

Laboratory

University

of

Illinois

at

Urbana-Champaign

-LSB-

-RSB-

Y.

M.

Wang

and

W.

K.

Fuchs

Optimistic

Message

Logging

for

Independent

Checkpointing

in

Message

Passing

Systems

In

Proc

Symp

on

Reliable

Distributed

Systems

pages

-LCB-

October

-LSB-

-RSB-

Y.

M.

Wang

and

W.

K.

Fuchs

Scheduling

Message

Processing

for

Reducing

Rollback

Propagation

In

Proc

IEEE

Fault-Tolerant

Computing

Symposium

pages

-LCB-

July

-LSB-

-RSB-

Y.

M.

Wang

Y.

Huang

and

W.

K.

Fuchs

Progressive

Retry

for

Software

Recovery

in

Distributed

Systems

In

Proc

IEEE

Fault-Tolerant

Computing

Symposium

pages

-LCB-

June

-LSB-

-RSB-

Jian

Xu

and

Robert

H.

B.

Netzer

Adaptive

Independent

Checkpointing

for

Reducing

Rollback

Propagation

In

IEEE

Symposium

on

Parallel

and

Distributed

Processing

December