Finding

Consistent

Global

Checkpoints

in

a

Distributed

Computation

D.

Manivannan

Robert

H.

B.

Netzery

Mukesh

Singhalz

Abstract

Finding

consistent

global

checkpoints

of

a

distributed

computation

is

important

for

analyzing

testing

or

verifying

properties

of

these

computations

In

this

paper

we

present

a

theoretical

foundation

for

finding

consistent

global

checkpoints

Given

an

arbitrary

set

S

of

local

checkpoints

we

prove

exactly

which

sets

of

other

local

checkpoints

can

be

combined

with

S

to

build

consistent

global

checkpoints

and

we

present

an

algorithm

for

finding

all

such

global

checkpoints

The

minimal

and

maximal

consistent

global

checkpoints

are

presented

as

special

cases

The

results

are

based

on

the

notion

of

zigzag

paths

introduced

by

Netzer

and

Xu

-LSB-

-RSB-

We

also

present

a

method

for

finding

zigzag

paths

using

the

rollback-dependency

graph

introduced

by

Wang

-LSB-

-RSB-

Introduction

Consistency

of

global

states

is

a

recurring

theme

in

distributed

systems

-LSB-

-RSB-

A

global

state

is

a

set

of

individual

process

states

one

per

process

that

represents

a

snapshot

at

some

instant

of

each

process

execution

When

global

states

are

periodically

recorded

or

analyzed

during

execution

they

are

called

global

checkpoints

Global

checkpoints

have

applications

in

many

problems

such

as

transparent

failure

recovery

-LSB-

-RSB-

distributed

debugging

-LSB-

-RSB-

monitoring

distributed

events

-LSB-

-RSB-

setting

distributed

breakpoints

-LSB-

-RSB-

protocol

specification

and

verification

-LSB-

-RSB-

and

others

-LSB-

-RSB-

In

this

paper

we

explore

a

type

of

global

checkpoint

that

is

said

to

be

consistent

meaning

that

it

is

a

snapshot

of

process

states

that

actually

occurred

simultaneously

during

the

execution

or

had

the

potential

of

occurring

simultaneously

-LSB-

-RSB-

Checkpoints

are

consistent

when

no

causal

dependency

exists

between

any

of

its

local

process

states

-LSB-

-RSB-

-LRB-

for

brevity

we

use

the

term

consistent

checkpoint

when

the

context

makes

it

clear

that

a

global

checkpoint

is

implied

-RRB-

Finding

consistent

checkpoints

is

a

natural

part

of

many

problems

involving

global

states

For

example

in

failure

recovery

the

system

state

can

be

restored

from

a

saved

consistent

checkpoint

and

then

restarted

-LSB-

-RSB-

The

problem

we

address

is

to

determine

how

individual

local

checkpoints

can

be

combined

with

those

from

other

processes

to

form

global

checkpoints

that

are

consistent

A

solution

to

this

problem

forms

the

basis

for

many

Current

address

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

-LRB-

email

manivann@cis.ohio-state.edu

-RRB-

yCurrent

address

Department

of

Computer

Science

Brown

University

Box

Providence

RI

-LRB-

email

rn@cs.brown.edu

-RRB-

zCurrent

address

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

-LRB-

email

singhal@cis.ohio-state.edu

-RRB-

Copyright

D.

Manivannan

Robert

H.B.

Netzer

Mukesh

Singhal

algorithms

and

protocols

that

must

record

on-the-fly

only

consistent

checkpoints

or

determine

post-mortem

which

global

checkpoints

are

consistent

To

introduce

the

problem

let

us

consider

how

to

build

a

consistent

checkpoint

from

some

arbitrary

set

S

of

local

checkpoints

from

some

-LRB-

but

not

all

-RRB-

processes

First

since

a

global

checkpoint

contains

one

local

checkpoint

from

each

process

we

must

select

one

candidate

checkpoint

from

each

process

not

represented

in

S

and

combine

these

candidates

with

S

to

form

the

global

checkpoint

However

these

candidates

must

be

selected

carefully

to

ensure

the

resulting

global

checkpoint

is

consistent

Figure

illustrates

the

basic

problem

with

an

example

three-process

execution

-LRB-

the

X

's

indicate

local

checkpoints

-RRB-

Given

two

local

checkpoints

such

as

A

and

C

if

they

are

to

belong

to

the

same

consistent

checkpoint

consistency

requires

that

neither

causally

affect

the

other

-LRB-

in

the

sense

that

no

sequence

of

messages

is

sent

from

after

A

and

received

before

C

-RRB-

this

condition

is

necessary

for

the

checkpoints

to

be

consistent

In

Figure

a

path

of

messages

exists

from

A

to

C

so

any

global

checkpoint

constructed

from

them

will

never

be

consistent

since

such

a

checkpoint

would

always

have

a

causal

path

between

two

of

its

local

checkpoints

-LRB-

A

and

C

-RRB-

In

contrast

checkpoints

B

and

C

have

no

causal

path

between

them

-LRB-

they

are

mutually

unordered

-RRB-

and

indeed

they

can

be

combined

with

the

second

checkpoint

of

to

form

a

consistent

checkpoint

as

shown

by

the

dashed

line

Checkpoints

lying

on

the

dashed

line

form

a

consistent

checkpoint

because

no

message

path

exists

from

any

checkpoint

on

the

line

to

any

other

on

the

line

Figure

Example

execution

B

and

C

can

be

used

to

build

a

consistent

checkpoint

-LRB-

dashed

line

-RRB-

but

A

and

C

or

B

and

D

can

not

From

this

observation

one

might

be

tempted

to

conclude

that

any

set

of

mutually

unordered

local

checkpoints

can

always

be

used

to

build

a

consistent

checkpoint

However

this

is

not

true

being

unordered

is

not

always

sufficient

to

ensure

consistency

In

Figure

checkpoints

B

and

D

are

also

unordered

but

they

can

not

be

combined

in

a

consistent

way

There

is

no

checkpoint

in

that

can

be

combined

with

both

B

and

D

while

maintaining

consistency

Because

of

message

D

is

not

consistent

with

the

second

checkpoint

of

-LRB-

or

any

earlier

checkpoint

in

-RRB-

since

those

checkpoints

have

a

causal

path

to

D.

Similarly

because

of

message

B

is

not

consistent

with

any

later

checkpoints

in

since

B

has

a

causal

path

to

those

checkpoints

Thus

no

checkpoint

in

can

be

used

to

build

a

consistent

checkpoint

containing

both

B

and

D

In

this

paper

we

explore

these

issues

in

detail

proving

precisely

which

local

checkpoints

can

be

used

in

conjunction

with

a

given

set

of

checkpoints

S

to

build

consistent

checkpoints

This

problem

was

first

considered

by

Netzer

and

Xu

-LSB-

-RSB-

who

proved

the

conditions

necessary

and

sufficient

for

some

consistent

checkpoint

to

be

built

from

S

but

they

did

not

define

the

set

of

possible

consistent

checkpoints

or

present

an

algorithm

to

construct

them

Building

on

Netzer

and

Xu

's

results

Wang

-LSB-

-RSB-

then

presented

algorithms

for

computing

from

S

the

minimum

and

maximum

consistent

checkpoints

that

can

be

constructed

and

showed

their

application

to

several

problems

We

further

build

on

this

past

work

by

analyzing

the

set

of

all

consistent

checkpoints

that

can

be

built

from

S

We

prove

exactly

which

sets

of

local

checkpoints

from

each

process

can

be

combined

with

those

in

S

and

still

retain

consistency

We

also

present

an

algorithm

that

enumerates

all

such

consistent

checkpoints

These

results

provide

a

deeper

understanding

of

what

constitutes

consistency

and

can

be

applied

to

applications

that

use

consistent

checkpoints

The

rest

of

the

paper

is

organized

as

follows

In

Section

we

present

the

system

model

and

discuss

related

work

in

Section

In

Sections

and

we

present

our

theoretical

foundation

for

reasoning

about

consistency

and

give

an

algorithm

for

finding

all

the

consistent

checkpoints

that

contain

S

We

also

characterize

the

location

of

the

maximal

and

minimal

consistent

checkpoints

and

discuss

a

graph

-LRB-

the

rollback-dependency

graph

-RRB-

introduced

by

Wang

-LSB-

-RSB-

that

can

be

used

in

the

construction

of

consistent

checkpoints

Section

concludes

the

paper

Model

The

distributed

computation

we

consider

consists

of

N

spatially

separated

sequential

processes

denoted

by

PN

The

processes

do

not

share

a

common

memory

or

a

common

clock

Message

passing

is

the

only

way

for

processes

to

communicate

with

one

another

The

computation

is

asynchronous

each

process

progresses

at

its

own

speed

and

messages

are

exchanged

through

reliable

communication

channels

whose

transmission

delays

are

finite

but

arbitrary

Communication

channels

are

FIFO

Execution

of

a

process

is

modeled

by

three

types

of

events

the

send

event

of

a

message

the

receive

event

of

a

message

and

a

checkpoint

event

The

states

of

processes

depend

on

one

another

due

to

interprocess

communication

Lamport

's

happened

before

relation

-LSB-

-RSB-

on

events

HB

is

defined

as

the

transitive

closure

of

the

union

of

two

other

relations

HB

-LRB-

XO

-LSB-

M

-RRB-

The

XO

relation

captures

the

order

in

which

local

events

of

a

process

are

executed

The

ith

event

of

any

process

Pp

-LRB-

denoted

ep

i

-RRB-

always

executes

before

the

-LRB-

i

-RRB-

st

event

ep

i

XO

ep

i

The

M

relation

shows

the

relation

between

the

send

and

receive

events

of

the

same

message

if

a

is

the

send

event

of

a

message

and

b

is

the

corresponding

receive

event

of

the

same

message

then

a

M

b

If

a

is

ordered

before

b

by

HB

-LRB-

i.e.

a

HB

b

-RRB-

we

say

that

a

causal

path

exists

from

a

to

b

If

neither

a

nor

b

happens

before

the

other

we

say

that

they

are

unordered

A

local

checkpoint

of

a

process

is

a

recorded

state

of

the

process

A

set

of

local

checkpoints

one

from

each

process

is

called

a

consistent

global

checkpoint

-LRB-

or

just

consistent

checkpoint

-RRB-

iff

of

them

happened

before

the

other

We

assume

that

processes

take

checkpoints

asynchronously

Each

checkpoint

taken

by

a

process

is

assigned

a

unique

sequence

number

The

ith

-LRB-

i

-RRB-

checkpoint

of

process

Pp

is

assigned

the

sequence

number

i

and

is

denoted

by

Cp

i

We

assume

that

each

process

takes

an

initial

checkpoint

before

execution

begins

and

a

virtual

checkpoint

after

execution

ends

Checkpoints

are

sometimes

denoted

by

the

letters

A

B

or

C

for

clarity

The

ith

checkpoint

interval

of

process

Pp

is

all

the

computation

performed

between

its

-LRB-

i

-RRB-

th

and

ith

checkpoints

-LRB-

and

includes

the

-LRB-

i

-RRB-

th

checkpoint

but

not

ith

-RRB-

Background

and

Related

Work

The

definition

of

consistency

states

that

for

a

set

S

of

local

checkpoints

to

be

a

consistent

global

checkpoint

S

must

contain

one

local

checkpoint

from

each

process

and

for

any

two

checkpoints

A

B

S

A

and

B

must

be

unordered

As

discussed

above

from

this

observation

one

might

conclude

that

if

we

consider

a

smaller

set

S

containing

local

checkpoints

from

some

processes

but

not

others

having

all

local

checkpoints

in

S

be

mutually

unordered

is

also

sufficient

to

ensure

that

a

consistent

checkpoint

containing

S

can

be

built

However

as

shown

in

Figure

the

subtle

nature

of

consistency

makes

this

conclusion

false

Netzer

and

Xu

-LSB-

-RSB-

originally

addressed

this

subtlety

and

proved

the

necessary

and

sufficient

conditions

that

S

must

meet

to

be

able

to

construct

a

consistent

checkpoint

containing

S

Our

results

are

based

on

this

past

work

discussed

next

The

basic

subtlety

is

that

although

no

causal

path

may

exist

between

any

two

checkpoints

in

S

this

alone

is

insufficient

to

ensure

that

local

checkpoints

from

processes

not

represented

in

S

can

be

added

to

S

to

form

a

global

state

G

that

is

consistent

The

absence

of

causal

paths

does

not

fully

capture

whether

S

has

the

ability

to

become

a

consistent

checkpoint

To

determine

exactly

when

S

has

this

ability

Netzer

and

Xu

-LSB-

-RSB-

define

a

generalization

of

causal

paths

called

zigzag

paths

which

we

call

Z-paths

for

brevity

They

prove

that

it

is

the

absence

of

Z-paths

between

checkpoints

in

S

which

guarantees

that

S

can

be

extended

to

a

consistent

checkpoint

Since

Z-paths

capture

the

exact

conditions

for

consistency

they

are

a

powerful

notion

for

reasoning

about

problems

that

involve

consistent

checkpoints

-LSB-

-RSB-

Definition

A

Z-path

exists

from

Cp

i

to

Cq

j

iff

p

q

and

i

j

-LRB-

i.e.

one

checkpoint

precedes

the

other

in

the

same

process

-RRB-

or

there

exist

messages

mn

-LRB-

n

-RRB-

such

that

-LRB-

a

-RRB-

is

sent

by

process

Pp

after

Cp

i

-LRB-

b

-RRB-

if

mk

-LRB-

k

n

-RRB-

is

received

by

Pr

then

mk

is

sent

by

Pr

in

the

same

or

later

checkpoint

interval

-LRB-

although

mk

may

be

sent

before

or

after

mk

is

received

-RRB-

and

-LRB-

c

-RRB-

mn

is

received

by

Pq

before

Cq

j

A

checkpoint

C

is

said

to

be

in

a

Z-cycle

iff

there

exists

a

Z-path

from

C

to

itself

A

Z-path

between

two

checkpoints

A

and

B

is

like

a

causal

path

both

are

sequences

of

messages

that

start

after

A

and

end

before

B

and

both

define

a

transitive

relation

but

the

differences

between

them

are

important

A

causal

path

exists

from

A

to

B

iff

there

is

a

chain

of

messages

starting

after

A

and

ending

before

B

with

each

message

sent

after

the

previous

one

in

the

chain

is

received

Such

a

chain

is

also

a

Z-path

but

a

and

Xu

's

definition

only

contains

the

second

clause

but

it

is

convenient

to

also

define

a

Z-path

to

exist

from

A

to

B

if

A

and

B

belong

to

the

same

process

and

A

precedes

B.

Z-path

is

also

allowed

to

have

any

message

in

the

chain

be

sent

before

the

previous

one

is

received

as

long

as

the

send

and

receive

are

in

the

same

checkpoint

interval

Thus

a

causal

path

is

always

a

Z-path

but

a

Z-path

may

not

be

a

causal

path

Figure

illustrates

this

difference

There

is

a

Z-path

from

A

to

B

because

message

is

sent

after

A

is

sent

in

the

same

checkpoint

interval

as

is

received

and

is

received

before

B

-LRB-

the

path

forms

a

zigzag

shape

hence

the

name

-RRB-

This

Z-path

is

not

a

causal

path

Another

difference

stems

from

Z-paths

not

always

defining

a

partial

order

A

Z-path

can

exist

from

a

checkpoint

back

to

itself

-LRB-

a

Z-cycle

-RRB-

In

contrast

causal

paths

never

form

cycles

In

Figure

a

Z-cycle

exists

involving

checkpoint

C.

Message

is

sent

after

C

is

sent

in

the

same

interval

in

which

is

received

and

is

received

before

C

completing

the

Z-path

from

C

to

itself

-LRB-

the

Z-cycle

-RRB-

Figure

Z-paths

-LRB-

a

-RRB-

Z-path

from

A

to

B

and

Z-cycle

involving

C

-LRB-

b

-RRB-

Any

cut

through

A

and

B

is

inconsistent

as

well

as

any

cut

involving

C

-LRB-

all

the

dashed

lines

are

inconsistent

-RRB-

To

see

the

significance

of

Z-paths

between

checkpoints

consider

the

dashed

lines

in

Figure

which

show

the

global

states

that

can

include

A

and

B

or

C

The

Z-path

from

A

to

B

forces

any

global

state

-LRB-

or

cut

-RRB-

passing

through

A

and

B

to

be

inconsistent

since

the

cut

must

cross

either

or

Cutting

through

means

that

A

is

ordered

before

the

checkpoint

taken

after

is

received

cutting

through

means

that

the

checkpoint

taken

before

is

received

is

ordered

before

B

In

either

case

the

existence

of

the

Z-path

renders

it

impossible

to

construct

a

global

checkpoint

through

both

A

and

B

that

contains

only

mutually

unordered

checkpoints

Similarly

any

cut

passing

through

C

is

inconsistent

since

it

must

cut

across

either

or

and

as

above

the

message

being

cut

always

renders

the

global

checkpoint

inconsistent

To

formally

reason

about

Z-paths

we

use

the

following

notation

motivated

by

Wang

-LSB-

-RSB-

Definition

Let

A

B

be

individual

checkpoints

and

R

S

be

sets

of

checkpoints

Let

be

a

relation

defined

over

checkpoints

and

sets

of

checkpoints

such

that

A

B

iff

a

Z-path

exists

from

A

to

B

A

S

iff

a

Z-path

exists

from

A

to

some

member

of

S

S

A

iff

a

Z-path

exists

from

some

member

of

S

to

A

and

R

S

iff

a

Z-path

exists

from

some

member

of

R

to

some

member

of

S.

Using

this

notation

the

results

of

Netzer

and

Xu

are

easily

described

Their

basic

result

is

that

no

Z-path

exists

between

two

checkpoints

exactly

when

they

can

be

used

to

build

a

consistent

checkpoint

Proofs

of

this

result

-LRB-

and

several

corollaries

-RRB-

appear

in

Netzer

and

Xu

-LSB-

-RSB-

Note

that

S

S

implies

the

checkpoints

in

S

are

all

from

different

processes

Theorem

A

set

of

checkpoints

S

can

be

extended

to

a

consistent

global

checkpoint

if

and

only

if

S

S.

Corollary

A

set

of

checkpoints

S

is

a

consistent

global

checkpoint

if

and

only

if

S

S

and

jSj

N

where

N

is

the

number

of

processes

Corollary

A

checkpoint

C

can

be

part

of

a

consistent

global

checkpoint

if

and

only

if

it

is

not

involved

in

a

Z-cycle

Finding

Consistent

Global

Checkpoints

Although

Netzer

and

Xu

proved

the

exact

conditions

under

which

a

set

of

checkpoints

S

can

be

used

to

build

a

consistent

checkpoint

they

did

not

discuss

in

detail

how

to

actually

construct

the

consistent

checkpoints

Our

main

results

concern

this

issue

Wang

-LSB-

-RSB-

presents

algorithms

for

finding

certain

consistent

checkpoints

that

contain

S

but

he

focuses

only

on

the

so-called

minimal

and

maximal

checkpoints

-LRB-

discussed

below

-RRB-

He

uses

a

graph

called

the

rollback-dependency

graph

-LRB-

or

R-graph

-RRB-

and

presents

several

applications

of

minimal

and

maximal

checkpoints

We

extend

these

results

next

by

discussing

exactly

which

consistent

checkpoints

can

be

built

from

S

showing

the

minimal

and

maximal

checkpoints

as

special

cases

We

also

present

an

algorithm

to

enumerate

these

consistent

checkpoints

In

Section

we

discuss

the

R-graph

proving

that

it

represents

exactly

the

Z-paths

of

an

execution

The

R-graph

is

thus

useful

in

implementing

consistent

checkpoint

construction

since

it

shows

between

which

checkpoints

Z-paths

exist

Extending

S

to

a

Consistent

Checkpoint

Given

a

set

S

of

checkpoints

such

that

S

S

we

first

analyze

what

other

checkpoints

can

be

combined

with

S

to

build

a

consistent

global

checkpoint

There

are

three

important

observations

First

of

the

checkpoints

that

have

a

Z-path

to

or

from

any

of

the

checkpoints

in

S

can

be

used

because

by

Theorem

no

checkpoints

between

which

a

Z-path

exists

can

ever

be

part

of

a

consistent

checkpoint

Thus

only

those

checkpoints

that

have

no

Z-paths

to

or

from

any

of

the

checkpoints

in

S

are

candidates

We

call

the

set

of

all

such

candidates

the

Z-cone

of

S.

Similarly

we

call

the

set

of

all

checkpoints

that

have

no

causal

path

to

or

from

any

checkpoint

in

S

the

C-cone

of

S.

The

Z-cone

and

C-cone

help

us

reason

about

orderings

and

consistency

Since

causal

paths

are

always

Z-paths

the

Z-cone

of

S

is

a

subset

of

the

C-cone

of

S

as

shown

in

Figure

for

some

arbitrary

S.

Note

that

if

a

Z-path

exists

from

checkpoint

Cq

j

in

process

Pq

to

a

checkpoint

in

S

then

a

Z-path

also

exists

from

every

checkpoint

in

Pq

preceding

Cq

j

to

the

same

checkpoint

in

S

-LRB-

because

Z-paths

are

transitive

-RRB-

similarly

causal

paths

are

transitive

as

well

terms

are

inspired

by

the

so-called

light

cone

of

an

event

e

which

is

the

set

of

all

events

with

causal

paths

from

e

-LRB-

i.e.

events

in

e

's

future

-RRB-

-LSB-

-RSB-

Although

the

light

cone

of

e

contains

events

ordered

after

e

we

define

the

Z-cone

and

C-cone

of

S

to

be

those

events

with

no

zigzag

or

causal

ordering

respectively

to

or

from

any

member

of

S.

SZ-unordered

with

SCausally

unordered

with

SZ-paths

to

SZ-paths

from

SCausal

paths

to

SCausal

paths

from

S

-LRB-

C-cone

-RRB-

-LRB-

Z-cone

-RRB-

Edge

of

C-coneEdge

of

C-coneEdges

of

Z-cone

Figure

The

Z-cone

and

the

C-cone

associated

with

a

set

of

checkpoints

S

The

second

observation

is

that

although

candidates

for

building

a

consistent

checkpoint

from

S

must

lie

in

the

Z-cone

of

S

not

all

checkpoints

in

the

Z-cone

are

usable

If

a

checkpoint

in

the

Z-cone

is

involved

in

a

Z-cycle

then

it

can

not

be

part

of

a

consistent

checkpoint

by

Corollary

We

prove

below

in

Lemma

that

if

we

remove

from

consideration

all

checkpoints

in

the

Z-cone

that

are

in

Z-cycles

then

the

remaining

checkpoints

are

exactly

those

useful

for

building

a

consistent

checkpoint

from

S

That

is

each

such

checkpoint

can

be

combined

with

S

to

build

some

consistent

checkpoint

Definition

Let

S

be

a

set

of

checkpoints

such

that

S

S.

Then

for

each

process

Pq

the

set

Squseful

is

defined

as

Squseful

fCq

i

j

-LRB-

S

Cq

i

-RRB-

-LRB-

Cq

i

S

-RRB-

-LRB-

Cq

i

Cq

i

-RRB-

g

In

addition

we

define

Suseful

-LSB-

q

Squseful

Lemma

Let

S

be

a

set

of

checkpoints

such

that

S

S.

Let

Cq

i

be

any

checkpoint

of

process

Pq

such

that

Cq

i

S.

Then

S

-LSB-

fCq

ig

can

be

extended

to

a

consistent

checkpoint

if

and

only

if

Cq

i

Suseful

Proof

Suppose

S

-LSB-

fCq

ig

can

be

extended

to

a

consistent

checkpoint

Then

by

Theorem

-LRB-

S

-LSB-

fCq

ig

-RRB-

-LRB-

S

-LSB-

fCq

ig

-RRB-

This

implies

-LRB-

S

Cq

i

Cq

i

S

and

Cq

i

Cq

i

-RRB-

which

implies

Cq

i

Squseful

Suseful

Conversely

suppose

Cq

i

Suseful

Then

Cq

i

Squseful

since

Cq

i

is

a

checkpoint

of

process

Pq

Hence

S

Cq

i

Cq

i

S

and

Cq

i

Cq

i.

Moreover

S

S

and

hence

S

-LSB-

fCq

ig

S

-LSB-

fCq

ig

So

by

Theorem

S

-LSB-

fCq

ig

can

be

extended

to

a

consistent

checkpoint

Lemma

states

that

if

we

are

given

a

set

S

such

that

S

S

we

are

guaranteed

that

any

single

checkpoint

from

Suseful

can

belong

to

some

consistent

global

checkpoint

that

also

contains

S.

However

our

final

observation

is

that

if

we

attempt

to

build

a

consistent

checkpoint

from

S

by

choosing

a

subset

T

of

checkpoints

from

Suseful

to

combine

with

S

we

have

no

guarantee

that

the

checkpoints

in

T

have

no

Z-paths

among

them

In

other

words

simply

because

all

the

checkpoints

in

Suseful

are

in

the

Z-cone

of

S

-LRB-

and

thus

have

no

Z-paths

to

or

from

any

checkpoint

in

S

-RRB-

Z-paths

may

still

exist

between

members

of

Suseful

Therefore

we

have

one

final

constraint

we

must

place

on

the

set

T

we

choose

from

Suseful

to

build

a

consistent

checkpoint

from

S

the

checkpoints

in

T

must

have

no

Z-paths

among

them

Furthermore

since

S

S

by

Theorem

at

least

one

such

T

must

exist

Theorem

Let

S

be

a

set

of

checkpoints

such

that

S

S

and

let

T

be

any

set

of

checkpoints

such

that

S

T

Then

S

-LSB-

T

is

a

consistent

global

checkpoint

if

and

only

if

T

Suseful

T

T

and

jS

-LSB-

T

j

N

Proof

Suppose

S

-LSB-

T

is

a

consistent

global

checkpoint

Let

C

T

S

-LSB-

fCg

can

be

extended

to

a

consistent

global

checkpoint

since

S

-LSB-

T

is

one

such

extension

Hence

C

Suseful

by

Lemma

Thus

every

checkpoint

in

T

belongs

to

Suseful

Hence

T

Suseful

Since

S

-LSB-

T

is

a

consistent

global

checkpoint

-LRB-

S

-LSB-

T

-RRB-

-LRB-

S

-LSB-

T

-RRB-

by

Theorem

In

particular

T

T

By

definition

the

number

of

checkpoints

in

any

consistent

global

checkpoint

is

N

so

jS

-LSB-

T

j

N

Conversely

suppose

S

is

a

set

of

checkpoints

such

that

S

S

and

T

is

any

set

of

checkpoints

disjoint

from

S

satisfying

the

three

conditions

Since

T

Suseful

for

any

two

checkpoints

C

and

D

such

that

C

T

and

D

S

C

D

and

D

C

hold

Hence

T

S

and

S

T

Moreover

since

S

S

and

T

T

it

follows

that

-LRB-

S

-LSB-

T

-RRB-

-LRB-

S

-LSB-

T

-RRB-

Since

jS

-LSB-

T

j

N

S

-LSB-

T

is

a

consistent

global

checkpoint

by

Corollary

Minimal

and

Maximal

Consistent

Global

Checkpoints

Theorem

shows

us

exactly

which

sets

of

checkpoints

can

be

combined

with

S

to

form

consistent

checkpoints

Two

interesting

special

cases

we

discuss

next

are

the

minimal

and

maximal

consistent

checkpoints

that

contain

S.

Intuitively

these

are

the

earliest

and

latest

consistent

checkpoints

that

can

be

constructed

that

contain

S

The

following

formal

definitions

are

based

on

those

by

Wang

-LSB-

-RSB-

Definition

Let

S

be

any

set

of

checkpoints

such

that

S

S.

Let

M

S

-LSB-

T

be

a

consistent

global

checkpoint

where

T

Cpk

ikg

and

T

S

Then

M

is

the

maximal

consistent

checkpoint

containing

S

iff

for

any

consistent

checkpoint

M

S

-LSB-

T

containing

S

where

T

Cpk

jkg

we

have

n

k.

in

jn

M

is

the

minimal

consistent

checkpoint

containing

S

iff

for

any

consistent

checkpoint

M

S

-LSB-

T

containing

S

where

T

Cpk

jkg

we

have

n

k.

in

jn

Wang

-LSB-

-RSB-

showed

that

the

minimal

-LRB-

maximal

-RRB-

consistent

checkpoints

containing

S

are

those

formed

from

by

choosing

from

each

process

not

represented

in

S

the

earliest

-LRB-

latest

-RRB-

checkpoint

that

has

no

Z-path

to

or

from

any

member

of

S.

Viewed

in

terms

of

the

Z-cone

of

S

these

checkpoints

form

what

we

might

call

the

leading

and

trailing

edges

of

the

Z-cone

as

illustrated

in

Figure

SFirst

ckps

with

noz-paths

to

or

from

SZ-coneSz-paths

to

or

from

SZ-coneLast

ckps

with

no

-LRB-

leading

edge

-RRB-

-LRB-

trailing

edge

-RRB-

Figure

The

minimal

and

the

maximal

consistent

checkpoints

containing

a

target

set

S

This

observation

implies

that

the

Z-cone

possesses

some

interesting

properties

First

the

leading

and

trailing

edges

always

exist

This

follows

from

Theorem

which

states

that

at

least

one

set

T

must

exist

that

can

be

used

to

extend

S

to

a

consistent

checkpoint

In

the

degenerate

case

the

leading

and

trailing

edges

contain

exactly

the

same

checkpoints

and

only

one

such

T

exists

In

the

general

case

however

the

leading

and

trailing

edges

are

distinct

Second

the

checkpoints

making

up

the

leading

-LRB-

and

trailing

-RRB-

edge

never

have

Z-paths

between

them

-LRB-

including

any

Z-cycles

-RRB-

Since

the

Z-cone

is

defined

to

be

those

checkpoints

with

no

Z-paths

to

or

from

S

it

then

follows

that

the

checkpoints

containing

the

leading

and

trailing

edges

-LRB-

i.e.

the

minimal

and

maximal

checkpoints

-RRB-

are

always

consistent

These

properties

are

embodied

in

the

following

theorems

The

existence

of

the

minimal

and

maximal

consistent

checkpoints

has

been

proven

before

-LRB-

by

Wang

-LSB-

-RSB-

and

others

-RRB-

but

not

directly

in

terms

of

the

Z-cone

which

we

have

shown

to

be

the

basis

of

consistency

Netzer

and

Xu

-LSB-

-RSB-

construct

the

minimal

checkpoint

in

one

of

their

proofs

but

never

discuss

its

properties

per

se

Theorem

Let

S

be

a

set

of

checkpoints

such

that

S

S

and

let

Smax

fCq

j

j

Cq

j

Squseful

k

j

Cq

k

Squsefulg

Then

Smax

is

the

maximal

consistent

checkpoint

containing

S.

Proof

If

Cp

i

S

then

Cp

i

is

the

only

checkpoint

that

belongs

to

Spuseful

and

hence

Cp

i

Smax

Thus

every

checkpoint

in

S

belongs

to

Smax

and

hence

S

Smax

Since

Squseful

jSmaxj

N

Let

Smax

S

-LSB-

T

for

some

T

such

that

S

T

Note

that

T

is

determined

by

S

uniquely

If

jSj

N

then

T

and

S

itself

is

a

consistent

checkpoint

by

Corollary

and

hence

it

follows

that

Smax

S

is

the

maximal

consistent

checkpoint

containing

S.

So

we

assume

jSj

N

First

we

prove

that

S

-LSB-

T

is

a

consistent

checkpoint

To

prove

that

S

-LSB-

T

is

a

consistent

checkpoint

it

is

sufficient

to

prove

that

all

the

conditions

stated

in

Theorem

are

satisfied

T

Suseful

since

T

Smax

Suseful

Claim

T

T

Proof

of

Claim

Suppose

T

T

Since

T

Suseful

C

C

C

T

Hence

there

exist

distinct

checkpoints

Cp

i

Cq

j

T

such

that

Cp

i

Cq

j

Since

Cq

j

Suseful

S

-LSB-

fCq

jg

can

be

extended

to

a

consistent

checkpoint

by

Lemma

Since

Cp

i

Cq

j

Cp

k

Cq

j

k

i.

Also

since

Cp

i

T

-LRB-

Cp

k

Suseful

-RRB-

k

i

which

implies

-LRB-

-LRB-

Cp

k

Cp

k

-RRB-

-LRB-

Cp

k

S

-RRB-

-LRB-

S

Cp

k

-RRB-

-RRB-

k

i.

Thus

-LRB-

Cp

k

Cp

k

-RRB-

-LRB-

Cp

k

-LRB-

S

-LSB-

fCq

jg

-RRB-

-RRB-

-LRB-

-LRB-

S

-LSB-

fCq

jg

-RRB-

Cp

k

-RRB-

and

hence

-LRB-

S

-LSB-

fCq

jg

-RRB-

puseful

which

means

Pp

contains

no

checkpoint

that

can

be

used

to

extend

S

-LSB-

fCq

jg

to

a

consistent

checkpoint

This

is

a

contradiction

to

Lemma

Hence

our

assumption

that

T

T

is

incorrect

and

hence

T

T

Clearly

jS

-LSB-

T

j

N

Thus

T

satisfies

all

the

conditions

of

Theorem

and

hence

S

-LSB-

T

Smax

is

a

consistent

checkpoint

Next

we

prove

that

Smax

S

-LSB-

T

is

indeed

the

maximal

consistent

checkpoint

containing

S.

Let

T

Cpk

ikg

Suppose

T

Cpk

jkg

is

a

set

of

checkpoints

such

that

S

-LSB-

T

is

a

consistent

checkpoint

It

is

sufficient

to

prove

that

jn

in

n

k.

Suppose

not

then

jn

in

for

some

n.

Then

for

this

n

Cpn

jn

Spn

useful

since

Cpn

in

Smax

and

jn

in

Hence

Cpn

jn

can

not

belong

to

any

consistent

checkpoint

containing

S

by

Lemma

This

is

a

contradiction

to

the

fact

that

S

-LSB-

T

is

a

consistent

checkpoint

containing

S.

Hence

jn

in

n

k

and

Smax

S

-LSB-

T

is

the

maximal

consistent

checkpoint

containing

S.

Theorem

Let

S

be

a

set

of

checkpoints

such

that

S

S

and

let

Smin

fCq

j

j

Cq

j

Squseful

k

j

Cq

k

Squsefulg

Then

Smin

is

the

minimal

consistent

checkpoint

containing

S.

Proof

If

Cp

i

S

then

Cp

i

is

the

only

checkpoint

that

belongs

to

Spuseful

and

hence

Cp

i

Smin

Thus

every

checkpoint

in

S

belongs

to

Smin

and

hence

S

Smin

Since

Squseful

jSminj

N

Let

Smin

S

-LSB-

T

for

some

T

such

that

S

T

Note

that

T

is

determined

by

S

uniquely

If

jSj

N

then

T

and

S

itself

is

a

consistent

checkpoint

by

Corollary

and

hence

it

follows

that

Smin

S

is

the

minimal

consistent

checkpoint

containing

S.

So

we

assume

jSj

N

First

we

prove

that

S

-LSB-

T

is

a

consistent

checkpoint

To

prove

that

S

-LSB-

T

is

a

consistent

checkpoint

it

is

sufficient

to

prove

that

all

the

conditions

stated

in

Theorem

are

satisfied

T

Suseful

since

T

Smin

Suseful

Claim

T

T

Proof

of

Claim

Suppose

T

T

Since

T

Suseful

C

C

C

T

Hence

there

exist

distinct

checkpoints

Cp

i

Cq

j

T

such

that

Cp

i

Cq

j

Cp

i

Spuseful

and

Cq

j

Squseful

from

the

definition

of

T

Since

Cp

i

Cq

j

Cp

i

Cq

k

k

j

This

implies

Cp

i

C

C

Squseful

since

Cq

j

is

the

first

checkpoint

in

Squseful

Since

Cp

i

T

Suseful

S

-LSB-

fCp

ig

can

be

extended

to

a

consistent

checkpoint

by

Lemma

Since

S

-LRB-

S

-LSB-

fCp

ig

-RRB-

-LRB-

S

-LSB-

fCp

ig

-RRB-

quseful

Squseful

Moreover

since

Cp

i

C

C

Squseful

it

follows

that

Cp

i

-LRB-

S

-LSB-

fCp

ig

-RRB-

quseful

which

is

a

contradiction

since

there

can

be

no

Z-path

from

Cp

i

to

any

checkpoint

in

-LRB-

S

-LSB-

fCp

ig

-RRB-

quseful

by

the

definition

of

the

set

-LRB-

S

-LSB-

fCp

ig

-RRB-

quseful

Hence

T

T

Clearly

jS

-LSB-

T

j

N

Hence

S

-LSB-

T

Smin

is

a

consistent

checkpoint

Next

we

prove

that

Smin

S

-LSB-

T

is

indeed

the

minimal

consistent

checkpoint

containing

S.

Let

T

Cpk

ikg

Suppose

T

Cpk

jkg

is

a

set

of

checkpoints

such

that

S

-LSB-

T

is

a

consistent

checkpoint

It

is

sufficient

to

prove

that

jn

in

n

k.

Suppose

not

then

jn

in

for

some

n.

Then

for

this

n

Cpn

jn

Spn

useful

since

Cpn

in

Smin

and

jn

in

Hence

Cpn

jn

can

not

belong

to

any

consistent

checkpoint

containing

S

by

Lemma

This

contradicts

the

fact

that

S

-LSB-

T

is

a

consistent

checkpoint

Hence

jn

in

n

k

and

Smin

is

the

minimal

consistent

checkpoint

containing

S.

Algorithm

for

Enumerating

Consistent

Checkpoints

Above

we

showed

exactly

which

checkpoints

can

be

used

to

extend

S

to

a

consistent

checkpoint

Our

next

main

result

is

an

algorithm

to

enumerate

all

such

consistent

checkpoints

Algorithms

have

been

presented

in

the

literature

for

computing

various

sets

of

consistent

checkpoints

such

as

for

global

predicate

detection

-LSB-

-RSB-

and

recovery

-LSB-

-RSB-

but

explicitly

compute

all

the

consistent

checkpoints

that

include

a

given

set

S.

Wang

's

work

-LSB-

-RSB-

shows

how

to

compute

the

maximal

and

the

minimal

consistent

checkpoints

that

contain

S

Our

algorithm

is

not

targeted

toward

a

specific

application

as

it

simply

computes

the

set

of

all

consistent

checkpoints

containing

S

but

it

illustrates

the

use

of

our

theoretical

results

in

determining

consistent

checkpoints

Our

algorithm

is

novel

in

that

it

restricts

its

selection

of

checkpoints

to

those

within

the

Z-cone

of

S

and

it

checks

for

the

presence

of

Z-cycles

within

the

Z-cone

In

the

next

section

we

show

how

to

perform

this

detection

using

a

graph

discussed

originally

by

Wang

-LSB-

-RSB-

from

which

Z-cones

and

Z-paths

can

be

detected

Our

algorithm

is

shown

in

Figure

The

function

ComputeAllCgs

-LRB-

S

-RRB-

returns

the

set

of

all

consistent

checkpoints

that

contain

S

The

crux

of

our

algorithm

is

the

function

ComputeAllCgsF

rom

-LRB-

T

P

rocSet

-RRB-

which

extends

a

set

of

checkpoints

T

in

all

possible

consistent

ways

but

uses

checkpoints

only

from

processes

in

the

set

P

rocSet

After

verifying

that

S

S

ComputeAllCgs

simply

calls

ComputeAllCgsF

rom

passing

a

P

rocSet

consisting

of

the

processes

not

represented

in

S

-LRB-

lines

-LCB-

-RRB-

The

resulting

consistent

checkpoints

are

collected

in

the

global

variable

G

which

is

returned

-LRB-

line

-RRB-

It

is

worth

nothing

that

if

S

the

algorithm

computes

all

of

the

consistent

checkpoints

that

exist

in

the

execution

The

recursive

function

ComputeAllCgsF

rom

-LRB-

T

P

rocSet

-RRB-

works

by

choosing

any

process

from

P

rocSet

say

Pq

and

iterating

through

all

checkpoints

C

in

T

quseful

Recall

that

Lemma

states

that

each

such

checkpoint

extends

T

part-way

toward

a

consistent

checkpoint

This

means

that

T

-LSB-

C

can

itself

be

further

extended

eventually

arriving

at

a

consistent

checkpoint

Since

this

further

extension

is

simply

another

instance

of

constructing

all

consistent

checkpoints

that

contain

checkpoints

from

a

given

set

we

make

a

recursive

call

-LRB-

line

-RRB-

passing

T

-LSB-

C

and

a

P

rocSet

from

which

process

Pq

is

removed

The

recursion

eventually

terminates

when

the

passed

set

contains

checkpoints

from

all

processes

-LRB-

i.e.

P

rocSet

is

empty

-RRB-

ComputeAllCgs

-LRB-

S

-RRB-

f

let

G

if

S

S

then

let

AllP

rocs

be

the

set

of

all

processes

not

represented

in

S

ComputeAllCgsF

rom

-LRB-

S

AllP

rocs

-RRB-

return

G

g

ComputeAllCgsF

rom

-LRB-

T

P

rocSet

-RRB-

f

if

-LRB-

P

rocSet

;-RRB-

then

G

G

-LSB-

fTg

else

let

Pq

be

any

process

in

P

rocSet

for

each

checkpoint

C

T

quseful

do

ComputeAllCgsF

rom

-LRB-

T

-LSB-

fCg

P

rocSet

n

fPqg

-RRB-

g

Figure

Algorithm

for

computing

all

consistent

checkpoints

containing

S

In

this

case

T

is

finally

a

global

checkpoint

as

it

contains

one

checkpoint

from

every

process

and

it

is

added

to

G

-LRB-

line

-RRB-

When

the

algorithm

terminates

all

candidates

in

Suseful

have

been

used

in

extending

S

so

G

contains

every

possible

consistent

checkpoint

that

contains

S

The

following

theorem

argues

correctness

Theorem

Let

S

be

a

set

of

checkpoints

and

G

be

the

set

returned

by

ComputeAllCgs

-LRB-

S

-RRB-

If

S

S

then

T

G

if

and

only

if

T

is

a

consistent

checkpoint

containing

S

That

is

G

contains

exactly

the

consistent

checkpoints

that

contain

S.

Proof

Claim

-LRB-

-LRB-

T

T

-RRB-

-LRB-

P

rocSet

contains

all

the

processes

not

represented

in

T

-RRB-

-LRB-

S

T

-RRB-

-RRB-

is

an

invariant

during

the

execution

of

the

algorithm

where

T

and

P

rocSet

are

the

parameters

being

passed

to

ComputeAllCgsF

rom

Proof

of

Claim

It

is

sufficient

to

prove

that

the

invariant

holds

before

every

call

to

ComputeAllCgsF

rom

It

holds

before

the

first

call

because

at

that

time

T

S

and

the

set

AllP

rocs

contains

all

the

processes

not

represented

in

S.

Note

that

if

T

T

and

P

rocSet

then

for

each

Pq

P

rocSet

T

quseful

this

is

because

T

quseful

contains

precisely

those

checkpoints

of

Pq

that

can

be

combined

with

T

to

extend

T

to

a

consistent

checkpoint

by

Lemma

and

each

such

process

should

have

at

least

one

such

checkpoint

that

can

be

combined

with

T

to

extend

it

to

a

consistent

checkpoint

by

Theorem

also

for

each

checkpoint

C

T

quseful

T

-LSB-

fCg

T

-LSB-

fCg

by

Lemma

and

P

rocsetnPq

contains

all

the

processes

not

represented

in

T

-LSB-

fCg

Thus

if

-LRB-

-LRB-

T

T

-RRB-

-LRB-

P

rocSet

contains

all

the

processes

not

represented

in

T

-RRB-

-LRB-

S

T

-RRB-

-RRB-

holds

before

the

call

to

ComputeAllCgsF

rom

then

it

also

holds

after

the

call

to

ComputeAllCgsF

rom

and

hence

is

an

invariant

It

follows

from

this

invariant

and

Corollary

that

for

all

T

G

T

is

a

consistent

checkpoint

containing

S

since

T

is

added

to

the

set

G

only

if

P

rocset

Conversely

suppose

T

is

a

consistent

checkpoint

containing

S.

Let

T

n

S

Cpn

ing

and

Tj

S

-LSB-

Cpj

ij

g

-LRB-

j

n

-RRB-

and

From

Lemma

j

n

Cpj

ij

-LRB-

S

-LSB-

Tj

-RRB-

pjuseful

Thus

T

would

have

been

included

in

the

set

G.

Hence

G

consists

of

precisely

those

consistent

checkpoints

that

contain

S.

Finding

Z-paths

in

a

Distributed

Computation

Tracking

Z-paths

on

the

fly

seems

to

be

a

difficult

and

remains

an

open

problem

In

this

section

we

describe

a

method

for

determining

the

existence

of

Z-paths

between

checkpoints

in

a

distributed

computation

that

has

terminated

or

stopped

execution

using

the

rollback-dependency

graph

-LRB-

R-graph

-RRB-

introduced

by

Wang

-LSB-

-RSB-

First

we

present

the

definition

of

an

R-graph

Definition

The

rollback-dependency

graph

of

a

distributed

computation

is

a

directed

graph

G

-LRB-

V

E

-RRB-

where

the

vertices

V

are

the

checkpoints

of

the

distributed

computation

and

an

edge

-LRB-

Cp

i

Cq

j

-RRB-

from

the

checkpoint

Cp

i

to

the

checkpoint

Cq

j

belongs

to

E

if

p

q

and

j

i

or

p

q

and

a

message

m

sent

from

the

ith

checkpoint

interval

of

Pp

is

received

by

Pq

in

its

jth

checkpoint

interval

-LRB-

i

j

-RRB-

Construction

of

the

R-graph

When

a

process

Pp

sends

a

message

m

in

its

ith

checkpoint

interval

it

piggybacks

the

pair

-LRB-

p

i

-RRB-

with

the

message

When

the

receiver

Pq

receives

m

in

its

jth

checkpoint

interval

it

records

the

existence

of

an

edge

from

Cp

i

to

Cq

j

When

a

process

wants

to

construct

the

R-graph

for

finding

Z-paths

between

checkpoints

it

broadcasts

a

request

message

to

collect

the

existing

direct

dependencies

from

all

other

processes

and

constructs

the

complete

R-graph

We

assume

that

each

process

stops

execution

after

it

sends

reply

to

the

request

so

that

additional

dependencies

between

checkpoints

are

not

formed

while

the

R-graph

is

being

constructed

For

each

process

a

volatile

checkpoint

is

added

the

volatile

checkpoint

represents

the

volatile

state

of

the

process

-LSB-

-RSB-

Figure

A

distributed

computation

Example

of

an

R-graph

Figure

shows

the

R-graph

of

the

computation

in

Figure

In

Figure

and

represent

the

volatile

checkpoints

the

checkpoints

representing

the

last

state

the

process

attained

before

terminating

Volatile

checkpoints

Figure

The

R-graph

of

the

computation

in

Figure

We

denote

the

fact

that

there

is

a

path

from

C

to

D

in

the

R-graph

by

C

rd

D

-LRB-

it

only

means

that

there

is

a

path

it

does

not

however

specify

any

particular

path

-RRB-

For

example

in

Figure

rd

When

we

need

to

specify

a

particular

path

we

give

the

sequence

of

checkpoints

that

constitute

the

path

For

example

-LRB-

-RRB-

is

a

path

from

to

and

-LRB-

-RRB-

is

also

a

path

from

to

The

following

theorem

establishes

the

correspondence

between

the

paths

in

the

R-graph

and

the

Z-paths

between

checkpoints

This

correspondence

is

very

useful

in

determining

whether

or

not

a

Z-path

exists

between

two

given

checkpoints

Theorem

Let

G

-LRB-

V

E

-RRB-

be

the

R-graph

of

a

distributed

computation

Then

for

any

two

checkpoints

Cp

i

and

Cq

j

Cp

i

Cq

j

if

and

only

if

p

q

and

i

j

or

Cp

i

rd

Cq

j

in

G

-LRB-

note

that

in

this

case

p

could

still

be

equal

to

q

-RRB-

Proof

-LRB-

-LRB-

-RRB-

Sufficiency

If

p

q

and

i

j

then

clearly

Cp

i

Cq

j

So

we

assume

that

:-LRB-

-LRB-

p

q

-RRB-

-LRB-

i

j

-RRB-

-RRB-

holds

and

Cp

i

rd

Cq

j

in

G.

Let

-LRB-

Cp

i

Cpm

im

Cq

j

-RRB-

be

a

path

of

minimum

length

from

Cp

i

to

Cq

j

in

the

graph

G.

To

prove

Cp

i

Cq

j

we

use

induction

on

m

the

length

of

the

minimal

path

from

Cp

i

to

Cq

j

in

G.

Let

m

In

this

case

we

need

to

prove

that

if

-LRB-

Cp

i

Cq

j

-RRB-

E

then

Cp

i

Cq

j

which

follows

from

the

definition

of

R-graph

Now

let

m

Assume

that

for

any

two

checkpoints

C

and

D

if

there

exists

a

path

of

length

k

-LRB-

k

m

-RRB-

from

C

to

D

in

the

graph

G

then

there

exists

a

Z-path

from

C

to

D.

By

induction

hypothesis

since

-LRB-

Cp

i

Cpm

im

-RRB-

is

a

path

of

length

m

from

Cp

i

to

Cpm

im

in

G

Cp

i

Cpm

im

Case

-LRB-

i

-RRB-

pm

q

In

this

case

Cpm

im

precedes

Cq

j

in

the

same

process

Pq

and

hence

Cpm

im

Cq

j

Since

is

a

transitive

relation

it

follows

that

Cp

i

Cq

j

Case

-LRB-

ii

-RRB-

pm

q

In

this

case

a

message

M

sent

from

the

-LRB-

im

-RRB-

th

checkpoint

interval

of

Ppm

was

received

by

Pq

in

its

jth

checkpoint

interval

It

is

clear

that

the

messages

that

caused

the

Z-path

from

Cp

i

to

Cpm

im

together

with

the

message

M

constitute

a

Z-path

from

Cp

i

to

Cq

j

and

hence

Cp

i

Cq

j

-LRB-

-RRB-

-RRB-

Necessity

Conversely

suppose

Cp

i

Cq

j

Assume

that

:-LRB-

-LRB-

p

q

-RRB-

-LRB-

i

j

-RRB-

-RRB-

holds

Then

there

exists

a

sequence

of

messages

Mn

-LRB-

n

-RRB-

satisfying

the

conditions

given

in

Definition

Now

we

want

to

prove

that

Cp

i

rd

Cq

j

in

G.

To

prove

this

we

use

induction

on

n

the

length

of

the

sequence

of

messages

that

constitute

the

Z-path

If

n

then

was

sent

after

Cp

i

and

received

before

Cq

j

This

implies

that

there

exist

integers

and

i

and

j

such

that

was

sent

in

the

th

checkpoint

interval

of

Pp

and

was

received

by

Pq

in

its

th

checkpoint

interval

Thus

-LRB-

Cp

Cq

-RRB-

E

Since

i

i

Thus

i

j

and

-LRB-

Cp

Cq

-RRB-

E

implies

that

Cp

i

rd

Cq

j

Now

let

n

We

assume

that

the

result

is

true

for

all

Z-paths

consisting

of

fewer

than

n

messages

Suppose

the

message

sequence

Mn

constitutes

the

Z-path

from

Cp

i

to

Cq

j

Assume

that

Mn

constitutes

a

Z-path

from

Cp

i

to

Cr

k

for

some

r

and

k.

By

induction

hypothesis

Cp

i

rd

Cr

k.

Note

that

Mn

must

have

been

sent

by

Pr

either

in

its

kth

checkpoint

interval

or

some

later

checkpoint

interval

Case

-LRB-

i

-RRB-

Message

Mn

is

sent

by

Pr

in

its

kth

checkpoint

interval

In

this

case

-LRB-

Cr

k

Cq

j

-RRB-

E

Since

Cp

i

rd

Cr

k

it

follows

that

Cp

i

rd

Cq

j

Case

-LRB-

ii

-RRB-

Message

Mn

is

sent

by

Pr

in

its

lth

checkpoint

interval

for

some

l

k

In

this

case

Cr

k

rd

Cr

l

and

-LRB-

Cr

l

Cq

j

-RRB-

E

Since

Cp

i

rd

Cr

k

it

follows

that

Cp

i

rd

Cq

j

This

proves

the

theorem

The

following

Corollary

gives

the

necessary

and

sufficient

conditions

for

a

set

of

local

checkpoints

to

be

part

of

a

consistent

checkpoint

in

terms

of

paths

in

the

R-graph

Corollary

Let

G

-LRB-

V

E

-RRB-

be

the

R-graph

of

a

distributed

computation

and

let

S

be

any

set

of

checkpoints

Then

the

following

three

statements

are

equivalent

S

can

be

extended

to

a

consistent

checkpoint

S

S.

Snext

rd

S

in

G

where

Snext

fCp

i

j

Cp

i

Sg

Proof

-LRB-

-RRB-

follows

from

Theorem

-LRB-

-RRB-

follows

from

Theorem

Conclusion

In

this

paper

we

presented

a

theoretical

foundation

for

determining

consistent

checkpoints

in

a

distributed

computation

run

It

is

based

on

the

notion

of

zigzag

paths

introduced

by

Netzer

and

Xu

-LSB-

-RSB-

Zigzag

paths

help

in

identifying

all

the

checkpoints

that

are

useful

for

constructing

consistent

checkpoints

We

also

presented

a

characterization

of

the

maximal

and

the

minimal

consistent

checkpoints

containing

a

given

set

of

local

checkpoints

and

also

demonstrated

how

it

can

be

used

for

rolling

back

the

processes

to

the

maximal

consistent

checkpoint

in

the

event

of

a

process

failure

Based

on

the

theoretical

foundation

we

also

presented

an

algorithm

for

finding

all

the

consistent

checkpoints

containing

a

given

set

S

of

local

checkpoints

if

we

take

S

then

the

algorithm

gives

the

set

of

all

the

consistent

checkpoints

of

a

distributed

computation

run

We

established

the

correspondence

between

the

Z-paths

and

the

paths

in

the

R-graph

which

helps

finding

the

existence

of

Z-paths

between

checkpoints

References

-LSB-

-RSB-

R.

Baldoni

J.

M.

Helary

A.

Mostefaoui

and

M.

Raynal

Characterizing

consistent

checkpoints

in

large-scale

ditributed

systems

IEEE

International

Conference

on

Future

Trends

of

Distributed

Computing

Systems

August

-LSB-

-RSB-

R.

Baldoni

J.

M.

Helary

A.

Mostefaoui

and

M.

Raynal

Consistent

checkpoints

in

message

passing

distributed

systems

Rapporte

de

Recherche

INRIA

-LRB-

France

-RRB-

n.

May

-LSB-

-RSB-

R.

Baldoni

J.

M.

Helary

A.

Mostefaoui

and

M.

Raynal

On

modeling

consistent

checkpoints

and

the

domino

effect

in

distributed

systems

Rapporte

de

Recherche

INRIA

-LRB-

France

-RRB-

n.

May

-LSB-

-RSB-

R.

Baldoni

J.

M.

Helary

and

M.

Raynal

Mutually

consistent

recording

in

asynchronous

computation

ACM

symposium

on

Principle

of

Distributed

Computing

May

-LSB-

-RSB-

K.

Mani

Chandy

and

Leslie

Lamport

Distributed

Snapshots

Determining

Global

States

of

Distributed

Systems

ACM

Trans

of

Comp

Sys.

-LRB-

-RRB-

February

-LSB-

-RSB-

R.

Cooper

and

K.

Marzullo

Consistent

Detection

of

Global

Predicates

In

Proceedings

of

ACM/ONR

Workshop

on

Parallel

and

Distributed

Debugging

pages

-LCB-

May

-LSB-

-RSB-

E.

Fromentin

N.

Plouzeau

and

M.

Raynal

An

Introduction

to

the

Analysis

and

Debug

of

Distributed

Computations

In

Proceedings

of

IEEE

International

Conference

on

Algorithms

and

Architectures

for

Parallel

Processing

Brisbane

Australia

pages

-LCB-

April

-LSB-

-RSB-

K.

Geihs

and

M.

Seifert

Automated

Validation

of

a

Co-operation

Protocol

for

Distributed

Systems

In

Proc

Int

Conf

Distributed

Comp

Systems

pages

-LCB-

-LSB-

-RSB-

R.

Koo

and

S.

Toueg

Checkpointing

and

Roll-back

Recovery

for

Distributed

Systems

IEEE

Trans

on

Software

Eng.

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

Ajay

D.

Kshemkalyani

Michel

Raynal

and

Mukesh

Singhal

An

Introduction

to

Snapshot

Algorithms

in

Distributed

Computing

Distributed

Systems

Engineering

Journal

-LRB-

-RRB-

-LCB-

December

-LSB-

-RSB-

L.

Lamport

Time

Clocks

and

ordering

of

events

in

distributed

systems

Communications

of

the

ACM.

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

Leslie

Lamport

The

Mutual

Exclusion

Problem

Part

I

A

Theory

of

Interprocess

Communication

JACM

-LRB-

-RRB-

-LCB-

April

-LSB-

-RSB-

B.

Miller

and

J.

Choi

Breakpoints

and

Halting

in

Distributed

Programs

In

Proceedings

of

the

International

Conference

on

Distributed

Comput

Systems

pages

-LCB-

-LSB-

-RSB-

Robert

H.B.

Netzer

and

Jian

Xu

Necessary

and

Sufficient

Conditions

for

Consistent

Global

Snapshots

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

February

-LSB-

-RSB-

M.

Spezialetti

and

P.

Kearns

Simultanious

Regions

A

Framework

for

the

Consistent

Monitoring

of

Distributed

Systems

In

Proceedings

of

the

International

Conference

on

Distributed

Computing

Systems

pages

-LCB-

-LSB-

-RSB-

Yi-Min

Wang

Consistent

global

checkpoints

that

contain

a

given

set

of

local

checkpoints

In

AT&T

Technical

Report

-LSB-

-RSB-

Yi-Min

Wang

Maximum

and

Minimum

Consistent

Global

Checkpoints

and

their

Applications

In

Proceedings

of

the

IEEE

Symposium

on

Reliable

Distributed

Systems

pages

-LCB-

Bad

Neuenahr

Germany

September

causality
consistent global checkpoint collection
distributed checkpointing
fault tolerance