To

appear

Annual

ACM

Symposium

on

Parallel

Algorithm

and

Architecture

The

Verification

of

Cache

Coherence

Protocols

Technical

Report

Fong

Pong

and

Michel

Dubois

Department

of

Electrical

Engineering

Systems

University

of

Southern

California

Los

Angeles

CA

dubois@paris.usc.edu

pong@paris.usc.edu

-LRB-213-RRB- 740-9130

November

Abstract

In

this

paper

we

introduce

a

verification

technique

for

cache

coherence

protocols

at

the

behavior

level

Protocols

are

specified

by

a

Finite

State

Machine

-LRB-

FSM

-RRB-

model

The

global

state

space

is

the

Cartesian

product

of

an

arbitrary

number

of

individual

cache

state

spaces

and

is

symbolically

expanded

A

global

FSM

characterizing

the

protocol

behavior

is

built

and

protocol

verification

becomes

equivalent

to

finding

whether

or

not

the

global

FSM

may

enter

erroneous

states

State

expansion

only

takes

a

few

steps

contrary

to

current

approaches

The

verification

procedure

is

applied

to

the

verification

of

five

existing

protocols

The

Verification

of

Cache

Coherence

Protocols

Abstract

In

this

paper

we

introduce

a

verification

technique

for

cache

coherence

protocols

at

the

behavior

level

Protocols

are

specified

by

a

Finite

State

Machine

-LRB-

FSM

-RRB-

model

The

global

state

space

is

the

Cartesian

product

of

an

arbitrary

number

of

individual

cache

state

spaces

and

is

symbolically

expanded

A

global

FSM

characterizing

the

protocol

behavior

is

built

and

protocol

verification

becomes

equivalent

to

finding

whether

or

not

the

global

FSM

may

enter

erroneous

states

State

expansion

only

takes

a

few

steps

contrary

to

current

approaches

The

verification

procedure

is

applied

to

the

verification

of

five

existing

protocols

Introduction

In

a

shared-memory

multiprocessor

private

caches

are

needed

to

reduce

the

effects

of

memory

access

latency

and

contention

Whereas

private

caches

significantly

improve

system

performance

they

introduce

the

cache

coherence

problem

Multiple

cached

copies

of

the

same

memory

word

must

be

consistent

at

any

time

A

cache

coherence

protocol

ensures

that

changes

made

to

shared

memory

locations

by

any

processor

are

visible

to

all

other

processors

A

cache

coherence

protocol

is

a

set

of

rules

coordinating

communicating

entities

-LRB-

usually

cache

and

memory

controllers

-RRB-

to

enforce

consistency

among

multiple

data

copies

Several

protocols

-LSB-

-RSB-

have

been

proposed

described

and

implemented

however

they

have

never

been

formally

validated

The

simplicity

of

these

protocol

the

lack

of

verification

tools

and

the

complexity

of

current

formal

validation

procedures

may

explain

this

state

of

affair

Informal

techniques

for

protocol

verification

are

based

on

time-consuming

error-prone

testing

procedures

by

engineers

and

require

a

great

deal

of

ingenuity

As

the

complexity

of

protocols

grows

it

becomes

extremely

difficult

to

verify

protocols

by

simply

relying

on

human

reasoning

In

a

broad

sense

the

goal

of

validation

is

to

verify

that

a

protocol

satisfies

its

specification

and

possesses

the

required

invariant

properties

Validation

activities

including

simulation

studies

state

reachability

analysis

and

logical

reasoning

should

exist

at

all

phases

of

design

and

implementation

Simulations

are

conceptually

simple

but

suffer

from

incompleteness

since

a

random

test

sequence

must

be

run

indefinitely

to

enter

all

reach

able

states

It

is

also

very

unlikely

that

validation

procedures

based

on

trace-driven

simulations

can

detect

most

design

errors

A

protocol

passing

the

test

is

only

shown

to

be

correct

with

respect

to

the

particular

simulation

runs

and

hence

only

partial

correctness

is

demonstrated

Early

work

by

Baer

and

Girault

-LSB-

-RSB-

introduced

a

Petri

Net

model

of

cache

protocols

This

model

comprehensively

specifies

the

underlying

hardware

structure

The

Petri

Net

model

is

valuable

in

capturing

the

synchronization

between

communicating

hardware

entities

and

hence

it

is

an

important

methodology

for

mapping

protocol

designs

to

actual

implementations

The

construction

of

a

Petri

Net

model

as

shown

in

-LSB-

-RSB-

is

difficult

to

automate

and

is

very

complex

even

though

the

protocol

is

simple

Moreover

the

final

step

of

verification

is

a

lengthy

verbose

analysis

rather

than

a

formal

validation

Reachability

analysis

is

primarily

based

on

exploring

exhaustively

all

the

possible

interactions

between

entities

interacting

in

the

protocol

The

system

can

be

characterized

by

its

state

From

a

given

state

the

exploration

of

all

possible

interactions

between

entities

leads

to

a

number

of

new

states

States

in

which

the

protocol

fails

to

preserve

expected

correctness

properties

are

classified

as

erroneous

states

otherwise

states

are

permissible

If

any

erroneous

state

is

reachable

the

protocol

is

incorrect

The

major

difficulty

of

this

technique

is

the

state

space

explosion

problem

-LSB-

-RSB-

normally

the

state

exploration

complexity

blows

out

rapidly

with

the

growing

number

and

complexity

of

entities

involved

in

the

protocol

Reachability

analysis

has

been

widely

adopted

for

the

automated

verification

of

communication

protocols

because

violations

of

correctness

conditions

such

as

unspecified

reception

-LRB-

state

receiving

unspecified

messages

-RRB-

and

deadlock

-LRB-

state

without

exits

-RRB-

are

directly

tied

to

the

structure

of

the

reachability

graph

-LSB-

-RSB-

However

in

order

to

validate

a

cache

coherence

protocol

the

values

of

data

copies

must

also

be

tracked

during

the

generation

of

reachable

states

State

models

are

adequate

for

state-oriented

transition

aspects

but

they

do

not

capture

aspects

associated

with

data

values

In

-LSB-

-RSB-

Rudolf

and

Segal

presented

a

proof

of

a

snooping

protocol

by

enumerating

the

various

scenarios

of

reads

and

writes

Each

cache

is

considered

as

a

finite

state

automaton

and

a

product

machine

is

a

collection

of

n

finite

state

automata

Nanda

and

Bhuyan

-LSB-

-RSB-

presented

a

similar

approach

which

based

on

the

composition

of

communicating

finite

state

machines

and

on

state

enumeration

In

an

enumeration

approach

a

large

number

of

redundant

states

are

visited

and

expanded

during

the

state

expansion

procedure

-LSB-

-RSB-

Enumerating

states

for

complex

protocols

faces

the

state

space

explosion

problem

Another

technique

for

validating

protocols

relies

on

logical

proofs

-LSB-

-RSB-

This

approach

can

validate

a

full

range

of

properties

Ideally

any

property

which

can

be

formulated

in

logic

can

be

verified

but

proof

and

formulation

of

assertions

reflecting

the

desired

correctness

properties

are

often

error-prone

and

need

considerable

ingenuity

In

some

studies

requiring

great

efforts

correctness

conditions

are

still

incorrectly

and/or

incompletely

formulated

More

importantly

this

approach

can

not

deal

with

state-oriented

transitions

An

approach

combining

the

advantages

of

reachability

analysis

and

of

logical

proof

has

recently

been

applied

to

the

verification

of

communication

protocols

-LSB-

-RSB-

Reachability

analysis

based

on

state

models

with

augmented

variables

and

processing

routines

is

used

to

expand

major

states

while

logic

properties

are

formulated

and

proved

over

associated

context

variables

This

intermediate

approach

is

suited

to

the

verification

of

cache

coherence

protocols

on

one

hand

the

coherence

activities

are

mainly

reflected

by

the

state

changes

of

caches

which

suggests

the

reachability

analysis

on

the

other

hand

the

modeling

of

data

aspects

can

be

dealt

with

by

augmenting

the

state

description

with

context

variables

Recent

work

has

focused

on

the

state

space

explosion

problem

Instead

of

enumerating

the

state

space

a

symbolic

expansion

of

the

state

space

is

advocated

in

-LSB-

-RSB-

A

complex

system

often

exhibits

a

great

deal

of

regularity

Ideally

a

structure

which

exploits

this

regularity

is

able

to

compress

the

representation

of

a

set

of

states

by

extracting

their

common

features

For

example

the

regular

expression

-LCB-

A

B

C

-RCB-

-LCB-

A

-RCB-

represents

the

set

of

all

strings

containing

an

arbitrary

number

of

replications

of

-LCB-

A

B

C

-RCB-

and

ending

with

A

This

symbolic

form

provides

a

concise

representation

for

the

enumeration

of

an

infinite

set

An

n-ary

relation

R

with

respect

to

sets

An

is

any

subset

of

We

can

characterize

a

relation

R

by

a

boolean

function

such

that

if

and

only

if

the

particular

tuple

with

satisfies

the

relation

R

In

this

way

-LSB-

-RSB-

the

enumeration

of

the

state

space

is

replaced

by

a

search

for

sets

of

relations

whose

union

is

the

global

state

space

Whether

or

not

a

state

is

a

member

of

the

set

of

states

represented

by

R

is

indicated

by

the

value

of

f

corresponding

to

the

state

This

paper

introduces

a

new

methodology

for

validating

cache

coherence

protocols

at

the

early

design

stage

Our

method

is

based

on

reachability

analysis

The

system

state

is

the

composition

of

individual

cache

states

as

in

-LSB-

-RSB-

but

the

system

state

space

is

symbolically

expanded

and

represented

Abstracted

from

the

details

of

their

hardware

An

f

An

true

f

a

l

se

-LCB-

-RCB-

fi

f

an

-LRB-

-RRB-

t

rue

an

-LRB-

-RRB-

ai

A

i

implementation

protocols

are

specified

by

finite

state

automata

which

characterize

the

behavior

of

the

caches

Relying

on

the

symmetry

of

the

system

we

derive

equivalence

relations

among

states

we

then

group

all

caches

with

equivalent

states

into

classes

Each

class

is

represented

separately

and

its

representation

includes

a

set

of

caches

with

equivalent

states

This

state

representation

drastically

reduces

the

complexity

of

the

state

expansion

procedure

A

global

state

graph

which

facilitates

the

interpretation

of

the

protocol

behavior

is

reported

at

the

end

of

the

procedure

The

global

state

graph

is

useful

not

only

to

verify

data

consistency

but

also

to

demonstrate

the

similarities

and

disparities

among

protocols

The

first

part

of

this

paper

provides

a

protocol

model

with

its

fundamental

definitions

Equivalence

relations

leading

to

the

state

representation

by

classes

are

then

introduced

followed

by

the

symbolic

construction

of

the

global

state

graph

Finally

the

methodology

is

applied

to

several

protocols

of

moderate

complexity

described

in

-LSB-

-RSB-

Protocol

Model

Finite

State

Automaton

In

this

paper

we

employ

a

simple

finite

state

machine

-LRB-

FSM

-RRB-

model

to

specify

the

protocols

The

protocol

model

has

the

following

assumptions

Each

protocol

transition

is

atomic

that

is

the

time

required

for

the

change

of

states

of

all

caches

is

zero

Only

the

interactions

between

caches

and

main

memory

are

modeled

Representing

a

cache

coherence

protocol

by

an

FSM

model

is

natural

from

the

perspective

of

protocol

designers

In

the

past

the

FSM

models

have

been

extensively

used

to

describe

and

specify

cache

coherence

protocols

at

a

logical

level

Without

loss

of

generality

formal

definitions

of

the

protocol

model

are

as

follows

Definition

-LRB-

FSM

Model

-RRB-

A

deterministic

finite

state

machine

modeling

a

cache

coherence

protocol

has

structure

M

-LRB-

Q

S

F

d

-RRB-

where

Q

is

a

finite

set

of

state

symbols

S

is

the

set

of

operations

causing

state

transitions

F

is

a

characteristic

function

defined

for

each

state

and

which

can

be

and

d

defines

the

state

transition

functions

F

Q

S

Qfi

Finally

the

finite

state

machine

must

be

strongly

connected

that

is

starting

from

any

given

state

there

exist

at

least

one

path

leading

to

all

other

states

Definition

-LRB-

Global

or

System

State

-RRB-

With

respect

to

a

particular

memory

location

the

global

or

system

state

is

defined

as

the

composition

of

each

individual

cache

In

several

protocols

the

next

state

of

a

cache

depends

only

on

its

current

state

and

on

the

type

of

request

issued

by

the

local

processor

for

these

protocols

the

characteristic

functions

is

The

characteristic

function

F

is

introduced

to

include

protocols

whose

transitions

depend

on

the

global

state

and

not

just

on

the

state

of

the

local

cache

For

example

in

the

Illinois

protocol

-LSB-

-RSB-

a

read

miss

in

a

cache

may

bring

in

the

desired

cache

block

either

in

state

Valid-Exclusive

or

Shared

depending

on

the

presence

of

the

block

in

other

caches

Other

protocols

exhibiting

this

property

are

write-broadcast

protocols

such

as

the

Dragon

and

the

Firefly

We

can

define

a

relation

and

its

characteristic

-LRB-

sharingdetection

-RRB-

function

over

all

caches

states

from

the

perspective

of

cache

Ci

as

and

We

consider

that

the

state

invalid

includes

the

cases

where

the

block

is

in

cache

and

has

been

invalidated

or

where

the

block

is

not

present

in

cache

Note

that

there

is

one

sharing-detection

function

per

cache

If

there

is

only

one

copy

in

caches

then

the

function

returns

for

the

cache

with

the

valid

copy

and

for

all

other

caches

In

fact

the

sharingdetection

function

as

introduced

here

provides

a

more

formal

framework

to

model

protocols

than

the

framework

partially

specified

in

natural

language

given

in

-LSB-

-RSB-

Although

F

may

be

generalized

to

any

function

of

the

global

state

even

unrelated

to

sharing

we

limit

ourselves

to

functions

used

in

existing

protocols

that

is

F

is

either

or

is

the

sharingdetection

function

Throughout

this

paper

we

track

the

state

of

a

single

block

For

simplicity

we

use

the

terms

cache

state

and

memory

state

to

signify

the

state

of

block

copies

in

a

cache

or

in

memory

f

i

Cn

-LRB-

-RRB-

t

rue

i

f

C

j

Ci

s.t.

s

t

ate

Cj

-LRB-

-RRB-

i

nva

l

id

f

a

l

se

otherwise

-LRB-

F

f

fn

-LRB-

-RRB-

The

FSM

model

specifies

the

protocol

behavior

in

terms

of

global

state

transitions

and

is

only

the

support

for

the

protocol

verification

To

some

extent

each

cache

state

carries

some

semantic

interpretation

For

example

in

the

Illinois

protocol

a

cache

block

in

the

Dirty

state

means

that

the

local

copy

has

been

modified

and

that

main

memory

has

an

obsolete

data

copy

whereas

a

cache

block

in

the

Shared

state

indicates

that

the

local

copy

is

potentially

shared

with

other

caches

and

that

all

cached

copies

must

be

identical

with

the

main

memory

copy

As

a

result

if

different

caches

are

in

states

Dirty

and

Shared

in

the

same

global

state

then

a

contradiction

in

the

interpretation

of

cache

states

occurs

Similarly

several

caches

in

the

Dirty

state

signal

another

contradiction

This

suggests

a

primary

verification

procedure

consisting

of

searching

all

reachable

global

states

and

proving

that

all

reached

states

are

permissible

in

the

sense

that

individual

cache

states

are

compatible

-LSB-

-RSB-

The

problem

of

searching

the

global

state

space

is

therefore

converted

into

the

problem

of

finding

an

efficient

model

for

the

global

FSM

Model

for

Data

Consistency

A

cache

coherence

protocol

must

support

correct

execution

of

a

program

on

a

multiprocessor

system

In

general

there

are

two

distinct

requirements

The

ordering

of

accesses

must

conform

to

a

well

defined

consistency

model

and

the

parallel

code

must

be

written

correctly

for

this

model

The

memory

system

must

be

generally

coherent

i.e.

if

we

assume

that

all

accesses

are

atomic

the

cache

protocol

must

always

return

the

latest

value

on

each

load

We

formulate

this

latter

condition

within

the

framework

of

the

reachability

expansion

as

follows

Definition

-LRB-

Data

Consistency

-RRB-

With

respect

to

a

particular

memory

location

the

protocol

preserves

data

consistency

if

and

only

if

the

following

condition

is

always

true

during

the

reachability

analysis

the

family

of

global

states

originated

from

G

including

G

itself

consistently

observe

the

value

written

by

a

STORE

transition

t

which

brings

a

global

state

G

to

G

or

the

value

written

by

STORE

transitions

after

t

That

is

states

reached

by

expanding

G

are

not

allowed

to

access

the

old

value

defined

before

t.

Global

states

which

violate

this

condition

will

be

referred

to

as

erroneous

states

If

any

erroneous

state

is

reachable

the

protocol

is

incorrect

FIGURE

The

Illinois

Transition

Diagram

from

Perspective

of

Cache

Ci

The

Illinois

Protocol

Figure

shows

the

state

transition

diagram

of

the

Illinois

protocol

We

will

use

this

protocol

as

a

running

example

throughout

the

paper

The

Illinois

protocol

distinguishes

private

and

non-actively

shared

blocks

from

actively

shared

blocks

such

that

invalidations

for

write

hits

on

private

and

non-actively

shared

blocks

can

be

avoided

On

a

read

miss

a

block

is

loaded

in

states

Valid-Exclusive

or

Shared

depending

on

the

value

of

the

sharingdetection

function

f

There

are

four

states

for

cached

blocks

Invalid

Valid-Exclusive

-LRB-

not

modified

only

copy

in

caches

-RRB-

Shared

-LRB-

not

modified

possibly

copies

in

other

caches

-RRB-

and

Dirty

-LRB-

modified

only

copy

in

caches

-RRB-

We

can

summarize

the

Illinois

protocol

as

the

following

State

Symbols

Q

-LCB-

Invalid

Valid-Exclusive

Shared

Dirty

-RCB-

Operations

S

-LCB-

R

W

Rep

-RCB-

which

stand

for

read

write

and

replacement

Cache

algorithm

from

the

perspective

of

cache

Ci

Read

Hit

No

coherence

actions

need

to

be

taken

Read

Miss.

If

cache

Cj

has

a

dirty

copy

Cj

will

supply

the

missing

block

and

update

the

main

memory

at

the

same

time

both

Ci

and

Cj

change

their

local

state

to

Shared

Otherwise

Ci

gets

the

missed

block

from

other

caches

if

any

copies

are

cached

and

all

caches

with

a

copy

of

the

block

set

their

local

state

to

Shared

If

there

is

no

cached

copy

Ci

gets

the

block

from

main

memory

and

receives

an

exclusive

copy

-LRB-

state

Valid-Exclusive

-RRB-

Invalid

Dirty

V-Ex

W

-LRB-

j

-RRB-

Rep

-LRB-

i

-RRB-

Shared

R

-LRB-

j

-RRB-

f

i

L

R

-LRB-

i

-RRB-

W

-LRB-

i

-RRB-

fi

L

R

-LRB-

i

-RRB-

W

-LRB-

j

-RRB-

Rep

-LRB-

i

-RRB-

R

-LRB-

j

-RRB-

W

-LRB-

j

-RRB-

Rep

-LRB-

i

-RRB-

W

-LRB-

i

-RRB-

R

-LRB-

j

-RRB-

W

-LRB-

j

-RRB-

Rep

-LRB-

j

-RRB-

R

-LRB-

i

-RRB-

R

-LRB-

j

-RRB-

Rep

-LRB-

j

-RRB-

R

-LRB-

i

-RRB-

W

-LRB-

i

-RRB-

W

-LRB-

i

-RRB-

Write

Hit

If

the

block

is

Dirty

no

action

is

taken

If

the

write

is

Valid-Exclusive

its

state

changes

to

Dirty

Otherwise

all

other

caches

with

a

copy

must

invalidate

their

local

copies

and

Ci

receives

a

copy

in

state

Dirty

Write

Miss.

Like

a

read

miss

except

that

all

other

caches

with

a

copy

invalidate

it

and

the

block

is

loaded

in

the

Dirty

state

Replacement

Only

blocks

in

state

Dirty

need

to

be

written

back

to

main

memory

Augmenting

the

Global

State

with

Context

Variables

The

definition

of

global

state

is

primarily

based

on

the

cache

block

state

However

to

verify

the

protocol

we

need

to

augment

it

with

attributes

characterizing

the

value

of

data

Definition

-LRB-

Augmented

Global

State

-RRB-

In

a

system

with

n

caches

the

global

state

of

a

block

G

-LRB-

qn

-RRB-

qi

Q

is

augmented

by

M

-LRB-

mn

-RRB-

where

mi

denotes

a

set

of

context

variables

which

characterize

the

block

from

the

point

of

view

of

cache

i

The

global

state

is

augmented

in

order

to

strengthen

the

FSM

model

by

the

introduction

of

auxiliary

variables

M.

By

a

state

transition

-LRB-

G

M

-RRB-

fit

-LRB-

G

M

-RRB-

we

actually

mean

that

-LRB-

G

fit

G

-RRB-

and

the

discrepancy

between

M

and

M

reflects

the

environmental

changes

and

in

particular

the

data

aspects

such

as

data

transfers

and

values

Thus

the

reachability

graph

is

constructed

over

all

reached

global

states

G

and

associated

Ms

are

considered

attributes

of

these

states

The

augmented

context

variables

in

definition

allows

the

caches

to

customize

their

views

of

the

system

in

a

flexible

way

Caches

may

have

different

views

of

their

local

copies

and

of

the

memory

copy

because

of

the

buffering

of

stores

For

protocols

in

this

paper

all

caches

will

have

the

same

view

of

the

memory

block

because

the

effects

of

a

write

operation

is

instantaneous

in

these

protocols

-LRB-

we

have

assumed

atomic

accesses

throughout

this

paper

-RRB-

Therefore

a

single

global

context

variable

can

be

used

to

represent

the

state

of

the

memory

copy

The

more

general

notation

has

nonetheless

been

introduced

here

so

that

it

will

applicable

to

more

complex

protocols

later

on

To

formally

model

the

Illinois

protocol

is

relatively

easy

because

the

state

of

all

the

words

in

the

block

are

the

same

as

the

state

of

the

block

containing

it

Therefore

we

can

work

on

the

block

state

or

word

state

equivalently

The

formal

model

associates

each

cache

Ci

with

an

auxiliary

variable

cdatai

and

the

memory

with

auxiliary

variable

mdata

to

keep

track

of

consistency

between

memory

and

cached

copies

Selecting

a

proper

abstraction

level

cdata

takes

values

from

domain

-LCB-

nodata

fresh

obsolete

-RCB-

and

mdata

from

domain

-LCB-

fresh

obsolete

-RCB-

Initially

all

caches

are

in

the

Invalid

state

without

data

copies

-LRB-

cdatai

nodata

for

all

i

-RRB-

and

memory

has

the

fresh

copy

noted

as

mdata

fresh

The

value

assignments

to

these

variables

during

the

state

expansion

conform

to

the

protocol

algorithm

Data

inconsistency

occurs

when

a

processor

can

access

its

local

copy

indicated

as

an

obsolete

copy

With

this

notation

the

data

transfer

aspects

of

the

Illinois

cache

protocol

from

the

perspective

of

Ci

can

be

specified

as

follows

Read

Miss.

if

-LRB-

there

exist

Cj

in

Dirty

state

-RRB-

-LRB-

mdata

cdataj

-RRB-

update

memory

-LRB-

cdatai

mdata

-RRB-

get

data

from

memory

else

if

-LRB-

there

is

no

cached

copy

-RRB-

-LRB-

cdatai

mdata

-RRB-

get

data

from

memory

else

arbitrarily

choose

Cj

with

a

copy

-LRB-

cdatai

cdataj

-RRB-

Write

Hit

if

-LRB-

Ci

has

a

dirty

copy

-RRB-

no

action

is

taken

else

j

cdataj

nodata

invalidated

cdatai

fresh

mdata

obsolete

Write

Miss.

if

-LRB-

there

exist

Cj

in

Dirty

state

-RRB-

cdatai

cdataj

must

be

a

fresh

copy

else

if

-LRB-

there

exist

Cj

with

a

copy

-RRB-

cdatai

cdataj

must

observe

fresh

copy

else

no

cached

copy

cdatai

mdata

must

be

fresh

at

this

stage

j

cdataj

nodata

invalidated

cdatai

fresh

mdata

obsolete

Replacement

if

-LRB-

Ci

has

a

dirty

copy

-RRB-

mdata

cdatai

cdatai

nodata

Reachability

Analysis

State

Space

Expansion

Since

the

reachability

graph

is

constructed

over

all

global

states

and

value

assignments

to

auxiliary

variables

M

are

irrelevant

to

state

transitions

the

following

Sections

concentrate

on

the

transitions

between

global

states

alone

Exhaustive

Enumeration

of

the

State

Space

To

completely

verify

the

protocol

at

the

finite

automaton

level

all

states

and

transitions

must

be

exhaustively

simulated

Conventionally

an

exhaustive

search

algorithm

as

shown

in

Figure

is

used

to

explore

the

system

state

space

In

this

algorithm

a

working

list

of

newly

produced

states

and

an

history

list

keeping

track

of

visited

states

are

maintained

At

each

step

all

states

directly

reachable

from

the

current

state

are

generated

and

inserted

in

the

working

list

if

necessary

Redundant

states

are

pruned

by

checking

the

new

state

against

the

visited

states

Since

the

state

space

is

enumerated

explicitly

the

number

of

caches

must

be

exactly

defined

As

a

result

the

state

space

must

be

finite

because

the

number

of

state

symbols

and

cache

events

are

also

finite

Let?s

have

a

system

with

n

caches

Q

m

state

symbols

and

S

k

cache

events

The

maximum

number

of

state

in

the

system

state

space

is

-LRB-

m

-RRB-

n

states

However

the

number

of

state

visited

in

the

expansion

process

is

far

more

than

-LRB-

m

-RRB-

n

states

For

each

state

in

the

working

list

we

must

generate

all

its

directly

reachable

states

although

some

of

them

may

have

been

visited

previously

Without

any

pruning

effort

we

need

at

least

approximately

nk

-LRB-

m

-RRB-

n

state

visits

to

complete

the

expansion

process

If

the

connectivity

information

faithfully

showing

the

path

leading

to

a

particular

state

from

a

given

state

is

stored

the

problem

of

limited

memory

capacity

becomes

appar

Algorithm

exhaustive

search

W

list

of

working

states

H

list

of

visited

states

while

-LRB-

W

is

not

empty

-RRB-

do

begin

get

current

state

A

from

W

and

put

A

in

H.

for

all

A

A

is

a

successor

state

of

A.

if

-LRB-

A

W?H

-RRB-

then

add

A

to

W.

end

FIGURE

Algorithm

for

exhaustive

search

ent

It

is

clear

that

the

state

space

grows

exponentially

with

the

complexity

of

the

protocol

and

the

number

of

entities

in

the

validation

model

A

quantitative

analysis

of

this

technique

for

verifying

small

to

medium-size

protocols

is

presented

in

-LSB-

-RSB-

Pruning

the

State

Space

by

Counting

Equivalence

To

keep

the

state

space

manageable

pruning

of

redundant

states

is

necessary

Two

system

states

-LRB-

qn

-RRB-

and

-LRB-

sn

-RRB-

are

strictly

equivalent

if

and

only

if

qi

si

qi

si

Q

for

all

i

n

This

strict

equivalence

relation

is

certainly

too

conservative

As

we

mentioned

before

all

cache

entities

behavior

is

characterized

by

a

common

FSM

with

deterministic

transition

functions

All

n

permutations

of

a

state

-LRB-

qn

-RRB-

are

equivalent

in

the

validation

process

because

the

order

of

the

tuple

is

not

important

For

example

in

the

validation

of

a

system

with

three

caches

the

tuples

-LRB-

shared

shared

invalid

-RRB-

and

-LRB-

shared

invalid

shared

-RRB-

should

represent

equivalent

states

This

equivalence

based

on

symmetry

arguments

suggests

a

concrete

way

to

represent

a

set

of

equivalent

states

by

the

number

of

caches

in

each

state

A

state

for

a

system

with

n

caches

can

be

represented

as

where

ki

denotes

the

number

of

caches

in

state

qi

Q

and

Definition

-LRB-

Counting

Equivalence

-RRB-

Two

system

states

and

are

equivalent

if

ki

li

for

all

i

The

relaxed

equivalence

of

Definition

is

a

first

step

in

the

right

direction

Equivalence

classes

can

be

further

broadened

Symbolic

Expansion

The

conventional

expansion

algorithm

has

three

efficiency

problems

the

timeconsuming

operation

of

comparing

states

the

efficiency

of

convergence

-LRB-

termination

-RRB-

of

the

expansion

process

and

the

large

consumption

of

memory

resources

Searching

through

the

history

and

working

lists

against

the

new

state

becomes

intolerable

for

large

state

spaces

The

number

of

visited

states

that

can

be

maintained

is

also

limited

by

the

memory

size

Another

technical

problem

is

the

number

of

caches

to

model

in

a

validation

procedure

It

is

not

clear

at

first

that

a

protocol

correct

for

a

system

with

n

caches

would

also

be

correct

for

a

system

with

n

caches

n

n

The

verification

procedure

should

deal

with

any

n

in

order

to

validate

the

protocol

for

any

system

Pi

Q

q

iki

Si

Q

k

i

n

Pi

Q

qik

i

Pi

Q

q

ili

Composite

State

To

some

extent

the

number

of

caches

in

a

particular

cache

state

plays

an

important

role

in

judging

whether

or

not

a

system

state

is

permissible

For

example

several

caches

in

the

Dirty

state

signals

data

inconsistency

Similarly

if

a

cache

is

in

Shared

state

the

local

copy

is

clean

and

perhaps

present

in

other

caches

In

theory

an

infinite

number

of

caches

could

have

clean

copies

without

affecting

the

protocol

correctness

In

all

these

cases

the

actual

number

of

copies

is

really

unimportant

What

is

critical

in

all

protocols

that

we

are

aware

of

is

whether

there

are

or

several

copies

in

a

given

These

possibilities

can

be

represented

by

the

following

set

of

repetition

operators

Definition

-LRB-

Repetition

Operator

-RRB-

The

Singleton

-LRB-

-RRB-

states

that

there

is

one

and

only

one

cache

in

state

q

Q

This

operator

can

be

omitted

The

Positive

or

Plus-operator

-LRB-

q

-RRB-

indicates

that

at

least

one

cache

is

in

state

q

Q.

The

Star-operator

-LRB-

q

-RRB-

extends

the

plus-operator

by

including

the

case

of

instance

q

means

that

or

some

caches

are

in

state

q

Q

In

a

composite

system

state

caches

in

the

same

state

are

grouped

into

a

cache

state

class

and

specified

by

one

of

the

above

repetition

operators

For

example

a

composite

state

-LRB-

-RRB-

indicates

that

there

are

exactly

one

cache

in

state

one

or

more

caches

in

state

and

or

some

caches

in

state

Formally

the

definition

for

a

composite

state

is

as

follows

Definition

-LRB-

Composite

State

-RRB-

A

composite

state

represents

the

composition

of

cache

states

in

a

system

with

an

arbitrary

number

of

cache

entities

It

is

constructed

over

cache

state

classes

of

the

form

where

n

Q

and

ri

-LSB-

-RSB-

The

representation

of

a

composite

state

carries

all

the

information

needed

to

verify

the

protocol

For

example

in

the

Illinois

protocol

there

are

two

possible

sources

of

data

inconsistencies.The

first

possibility

is

that

cache

-LRB-

s

-RRB-

in

the

Shared

state

coexist

with

a

cache

in

the

Dirty

state

and

the

second

possibility

is

that

more

than

one

cache

are

in

the

Dirty

state

It

is

clear

that

the

first

case

can

be

formulated

by

the

cache

state

classes

appearing

in

the

composite

state

-LRB-

e.g.

the

composite

state

-LRB-

Dirty

Shared

-RRB-

-RRB-

The

second

case

is

cov

Note

that

if

a

protocol

was

ever

invented

in

which

two

dirty

copies

are

permissible

the

methodology

is

still

applicable

provided

that

we

add

this

new

possibility

to

the

list

of

repetition

operator

The

operator

means

instance

and

is

added

for

completeness

q

nr

n

-LRB-

-RRB-

ered

by

the

repetition

operator

over

state

classes

with

multiple

Dirty

copies

such

as

-LRB-

Dirty

-RRB-

-LRB-

Dirty

-RRB-

The

definition

of

composite

state

expands

state

equivalence

relations

beyond

the

strict

counting

equivalence

of

definition

Information

Ordering

and

Pruning

Repetition

operators

can

be

ordered

by

the

possible

states

they

specify

The

resulting

order

is

and

we

also

write

q

q

where

q

Q.

q

reveals

that

one

cache

is

in

state

q

-LRB-

which

is

always

permissible

-RRB-

or

that

multiple

caches

are

in

state

q

-LRB-

which

may

indicate

a

data

inconsistency

condition

-RRB-

The

star

operator

adds

the

possibility

of

instance

Therefore

we

say

that

is

weaker

than

if

where

q

Q

and

-LSB-

-RSB-

The

instance

can

be

ordered

with

respect

to

i.e.

Definition

-LRB-

Structural

Covering

-RRB-

We

say

that

composite

state

structurally

covers

composite

state

or

if

where

are

repetition

operators

Definition

-LRB-

Containment

-RRB-

We

say

that

composite

state

contains

composite

state

or

F

if

and

F

-LRB-

-RRB-

F

-LRB-

-RRB-

where

F

is

a

characteristic

function

defined

in

the

FSM

model

of

Section

The

definition

of

structural

covering

extends

the

order

between

repetition

operators

to

composite

states

Furthermore

a

composite

state

contains

if

is

structurally

covered

by

and

if

and

have

the

same

value

of

the

characteristic

function

F

An

interesting

consequence

of

containment

is

that

if

F

and

if

is

not

permissible

then

is

also

not

permissible

could

thus

be

discarded

during

the

verification

process

provided

we

keep

We

will

show

that

the

expansion

process

is

a

monotonic

operator

on

the

set

of

composite

states

S

that

is

if

F

then

t

-LRB-

-RRB-

F

t

-LRB-

-RRB-

where

t

is

an

operator

representing

a

cache

event

As

the

expansion

process

progresses

new

composite

states

are

created

A

new

state

is

discarded

if

it

is

contained

in

a

visited

state

Similarly

all

visited

states

con

q

r

i.e.

fi

tained

in

a

new

state

are

discarded

At

the

end

of

the

expansion

process

all

visited

states

are

essential

states

Definition

-LRB-

Essential

State

-RRB-

Composite

state

S

is

essential

if

and

only

if

there

does

not

exist

a

composite

state

S

such

that

S

F

S

As

a

result

the

state

space

at

the

end

of

the

expansion

process

is

simply

decomposed

into

several

families

-LRB-

which

may

be

overlapping

-RRB-

represented

by

essential

composite

states

Rules

and

Algorithm

for

the

Expansion

Process

We

need

now

to

define

the

set

of

operations

applied

to

construct

a

composite

state

in

the

state

generation

process

In

the

following

the

slash

symbol

signifies

or

selection

Aggregation

-LRB-

a

-RRB-

-LRB-

qr

-RRB-

qr

-LRB-

b

-RRB-

-LRB-

q

-RRB-

q

-LRB-

c

-RRB-

-LRB-

q

qr

-RRB-

q

Aggregation

rules

are

equivalence

rules

between

composite

states

obtained

by

merging

cache

states

Coincident

Transition

fit

where

r

-LSB-

-RSB-

and

t

is

an

observed

transition

originated

by

cache

-LCB-

C

state

-LRB-

C

-RRB-

-RCB-

It

states

that

all

caches

in

state

change

state

coincidentally

because

of

a

transition

originated

by

another

cache

not

in

state

One-step

Transition

fit

and

fit

-LRB-

-RRB-

where

t

is

a

transition

originated

by

cache

-LCB-

C

state

-LRB-

C

-RRB-

-RCB-

The

next

state

of

C

is

and

all

other

caches

in

the

state

class

change

state

to

For

example

one

cache

modifying

its

local

copy

in

the

Shared

state

changes

to

the

Dirty

state

and

the

copies

in

all

other

caches

in

the

same

Shared

state

are

invalidated

N-steps

Transitions

This

rule

specifies

the

repetitive

application

of

the

same

transition

N

times

where

N

is

an

arbitrary

positive

integer

-LRB-

a

-RRB-

-LRB-

Q

-RRB-

F

fit

-LRB-

Q

-RRB-

F

fit

-LRB-

Q

-RRB-

F

fit

fit

-LRB-

Q

-RRB-

F

where

t

is

a

transition

originated

by

cache

-LCB-

C

state

-LRB-

C

-RRB-

-RCB-

and

fit

It

states

that

the

same

transition

t

can

be

applied

infinitely

many

times

as

long

as

there

are

caches

in

state

and

every

generated

state

has

the

same

value

defined

over

the

characteristic

function

F.

Every

application

of

the

transition

brings

down

the

number

of

caches

in

state

by

one

and

increases

the

number

of

caches

in

state

The

transition

t

has

no

effect

on

other

caches

denoted

as

Q

in

the

tuple

One

trivial

example

is

a

series

consecutive

replacements

another

example

is

the

case

where

Q

is

Invalid

and

is

Shared

following

consecutive

read

misses

-LRB-

b

-RRB-

-LRB-

Q

-RRB-

F

fit

-LRB-

Q

-RRB-

F

fit

fit

-LRB-

Q

-RRB-

F

fit

fit

-LRB-

Q

-RRB-

F

The

same

interpretation

is

made

as

in

-LRB-

a

-RRB-

except

that

the

derivation

ends

with

a

composite

state

of

different

value

F

This

rule

is

used

to

model

the

effect

of

the

sharing-detection

function

introduced

in

Section

During

the

state

expansion

process

the

next

state

is

produced

by

stimulating

the

current

state

by

exploring

all

possible

cache

transactions

and

by

repeatedly

applying

the

above

rules

Before

we

formalize

the

algorithm

for

symbolic

state

expansion

and

protocol

verification

we

first

prove

as

promised

the

monotonicity

of

the

expansion

process

Lemma

The

immediate

successor

originated

from

state

is

structurally

covered

by

originated

from

state

if

rj

rj

for

all

j

i

the

same

cache

event

t

S

is

applied

to

and

and

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Proof

The

proof

is

direct

First

of

all

conforming

to

the

condition

F

-LRB-

-RRB-

F

-LRB-

-RRB-

caches

in

the

same

state

in

and

will

get

to

the

same

next

cache

state

under

common

cache

event

t.

Let?s

ignore

caches

in

state

qi

first

that

is

-LCB-

qi

-RCB-

-LCB-

qir

-RCB-

Since

both

states

are

exactly

the

same

any

transition

t

will

bring

them

to

the

same

state

Next

we

augment

and

by

adding

the

effect

of

applying

t

on

caches

in

state

qi

in

and

The

only

change

is

the

number

of

caches

in

the

particular

state

-LCB-

qj

qi

fit

qj

-RCB-

It

is

obvious

that

-LCB-

qi

fit

qj

-RCB-

-LCB-

fit

-RCB-

i.e.

-LCB-

qi

-RCB-

-LCB-

-RCB-

results

in

-LCB-

qj

-RCB-

-LCB-

-RCB-

after

the

transition

We

can

thus

conclude

that

Lemma

The

claim

holds

if

F

that

is

rj

rj

for

all

j

and

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Proof

The

result

extends

the

conclusion

of

lemma

and

the

proof

is

similar

q

i

r

i

q

ii

qi

ri

qn

rn

-LRB-

-RRB-

r

qi

ri

q

i

r

i

qn

r

n

-LRB-

-RRB-

The

result

of

lemma

indicates

that

if

F

the

immediately

reachable

state

of

is

structurally

covered

-LRB-

-RRB-

by

the

immediately

reachable

state

of

To

demonstrate

the

monotonicity

we

still

need

to

show

that

and

have

the

same

characteristic

functions

that

is

F

-LRB-

-RRB-

F

-LRB-

-RRB-

We

prove

this

in

Appendix

Corollary

If

F

is

and

then

for

every

reachable

from

there

exists

reachable

from

such

that

Proof

Because

F

is

the

transition

functions

depend

only

on

local

cache

state

and

intended

operations

By

definition

the

relation

of

containment

is

characterized

by

the

relation

of

structural

covering

alone

As

a

result

the

claim

is

just

a

recursive

induction

from

lemma

Corollary

demonstrates

that

protocols

whose

behavior

does

not

depend

on

any

characteristic

function

exhibits

the

monotonous

property

of

our

symbolic

expansion

process

Therefore

during

the

expansion

process

can

be

discarded

provided

we

can

demonstrate

this

monotonous

property

because

all

successors

originated

from

can

be

generated

by

expanding

the

family

originated

from

Corollary

Protocols

employing

the

sharing-detection

function

also

exhibit

the

monotonicity

of

expansion

If

F

is

the

sharing-detection

function

and

then

for

every

reachable

from

there

exists

reachable

from

such

that

F

Proof

Direct

consequence

of

the

proof

in

Appendix

A.

The

preceding

results

suggest

a

very

efficient

expansion

process

to

obtain

essential

states

as

shown

in

Figure

Two

lists

are

used

to

keep

non-expanded

and

visited

states

At

each

step

a

new

state

is

produced

by

expanding

the

current

state

and

then

a

pruning

process

based

on

the

realization

of

monotonicity

removes

contained

states

The

final

output

reported

in

list

H

is

the

set

of

essential

states

All

possible

states

are

included

in

the

reported

essential

states

as

we

now

show

Theorem

The

essential

composite

states

generated

by

the

proposed

algorithm

are

complete

They

symbolically

characterize

all

states

which

can

be

produced

by

an

exhaustive

expansion

process

such

as

the

algorithm

depicted

in

Figure

Proof

The

number

of

caches

needs

to

be

explicitly

specified

in

the

validation

model

for

an

enumeration

approach

while

the

symbolic

approach

employs

canonical

form

to

characterize

states

Consider

states

u

v

-LRB-

derived

from

u

by

transition

t

-RRB-

in

the

enumera

tion

approach

and

composite

states

s

t

-LRB-

derived

from

s

by

transition

t

-RRB-

in

the

symbolic

form

and

s

symbolically

characterize

u.

t

also

characterizes

v

because

the

same

transition

functions

are

applied

and

the

information

is

accumulated

during

the

generation

of

composite

states

Verification

of

the

Illinois

Protocol

We

have

coded

the

algorithm

of

Figure

and

in

this

section

we

take

as

example

the

Illinois

protocol

to

demonstrate

the

symbolic

expansion

algorithm

The

other

four

protocols

described

in

-LSB-

-RSB-

are

verified

in

Appendix

B

We

start

the

expansion

process

by

an

initial

state

-LRB-

Invalid

-RRB-

in

which

no

cache

has

a

block

copy

After

a

small

number

of

expansion

steps

the

essential

states

and

the

global

state

transition

diagram

are

reported

The

global

transition

diagram

built

upon

the

symbolic

essential

states

highlights

our

tech

Algorithm

essential

states

generation

W

list

of

working

composite

states

H

list

of

visited

composite

states

-LRB-

output

essential

states

-RRB-

while

-LRB-

W

is

not

empty

-RRB-

do

begin

get

current

state

A

from

W.

for

all

cache

state

class

v

A

for

all

applicable

operations

t

on

v

A

fit

A

for

any

state

P

W

and

Q

H

if

-LRB-

A

F

P

or

A

F

Q

or

A

F

A

-RRB-

then

discard

A

else

begin

remove

P

from

W

if

P

F

A

remove

Q

from

H

if

Q

F

A

add

A

to

W.

if

-LRB-

A

F

A

-RRB-

then

discard

A

and

terminate

all

FOR

loops

starting

a

new

run

end

end

end

end

end

FIGURE

Algorithm

for

generating

essential

states

nique

and

facilitates

the

verification

of

the

data

consistency

and

the

demonstration

of

similarities

and

disparities

between

protocols

FIGURE

The

Illinois

Global

Transition

Diagram

After

state

visits

-LRB-

see

Appendix

A.

-RRB-

five

essential

states

-LRB-

Invalid

-RRB-

-LRB-

V-Ex

Invalid

-RRB-

-LRB-

Dirty

Invalid

-RRB-

-LRB-

Shared

Invalid

-RRB-

and

-LRB-

Shared

Invalid

-RRB-

are

reported

and

the

global

transition

diagram

is

shown

in

Figure

where

R

W

and

Z

stand

for

read

write

and

replacement

respectively

An

optional

subscript

identifies

the

state

of

the

cache

originating

the

transition

A

superscript

n

means

that

the

transition

is

the

result

of

an

N-steps

transitions

The

values

of

the

sharing-detection

function

and

of

the

auxiliary

variables

showing

the

status

of

the

cached

data

copies

and

of

the

memory

copy

are

listed

It

is

clear

that

the

data

consistency

requirement

is

satisfied

because

processors

always

have

the

most

recent

data

values

if

a

valid

copy

is

in

their

cache

as

shown

in

the

table

in

Figure

We

need

to

distinguish

between

states

-LRB-

Shared

Invalid

-RRB-

and

-LRB-

Shared

Invalid

-RRB-

because

of

different

observations

of

data

sharing

from

the

perspective

of

caches

in

state

class

Shared

is

structurally

covered

by

but

is

not

contained

in

By

the

definition

of

the

sharing-detection

function

in

Section

the

boolean

values

returned

are

Rninvalid

W

Znshared

Zv-ex

Zdirty

W

-LRB-

v-ex

invalid

-RRB-

-LRB-

shared

invalid

-RRB-

-LRB-

dirty

invalid

-RRB-

-LRB-

invalid

-RRB-

R

W

W

-LRB-

shared

invalid

-RRB-

Rninvalid

Rninvalid

Zshared

-LRB-

Invalid

-RRB-

-LRB-

false

-RRB-

-LRB-

nodata

-RRB-

fresh

-LRB-

V-Ex

Invalid

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

Dirty

Invalid

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

obsolete

-LRB-

Shared

Invalid

-RRB-

-LRB-

true

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

Shared

Invalid

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

state

sharing

-LRB-

F

-RRB-

cdata

mdata

-LRB-

true

true

-RRB-

-LRB-

false

true

-RRB-

for

and

respectively

The

state

-LRB-

Invalid

-RRB-

is

also

an

essential

state

and

is

not

contained

by

another

state

for

the

same

reason

The

definition

of

the

plus

-LRB-

-RRB-

operator

needs

some

explanation

when

the

sharingdetection

function

F

is

employed

In

fact

Shared

in

state

-LRB-

Shared

Invalid

-RRB-

with

F

-LRB-

-RRB-

-LRB-

true

true

-RRB-

denotes

that

there

are

at

least

two

caches

in

the

Shared

state

when

is

first

constructed

This

does

not

violate

the

original

definition

of

the

plus

operator

because

the

additional

information

is

carried

by

the

value

of

F

associated

with

the

composite

state

We

use

the

additional

information

carried

by

the

value

of

F

to

distinguish

between

and

Certainly

we

could

explicitly

express

the

effect

of

F

by

defining

another

operator

that

denotes

two

or

more

caches

in

a

particular

cache

state

However

more

expansion

rules

would

be

introduced

and

more

essential

states

would

be

generated

On

the

other

hand

the

original

definition

of

the

plus

operator

provides

a

general

notation

and

the

characteristic

function

F

shows

the

subtle

difference

States

and

are

also

reported

for

the

Firefly

protocol

-LRB-

see

Appendix

B

-RRB-

however

these

two

protocols

do

have

different

behavior

Following

a

write

in

the

block

in

the

Shared

state

in

-LRB-

Shared

Invalid

-RRB-

the

system

moves

into

state

-LRB-

Dirty

Invalid

-RRB-

for

the

Illinois

protocol

and

-LRB-

Exclusive

Invalid

-RRB-

for

the

Firefly

protocol

The

Firefly

protocol

employs

a

write

broadcast

scheme

to

keep

copies

consistent

as

long

as

multiple

data

copies

exist

that

is

the

detection

of

data

sharing

is

done

whether

or

not

the

block

is

present

in

the

cache

On

the

other

hand

the

Illinois

protocol

needs

the

sharing

information

only

when

a

read

miss

occur

i.e.

at

a

time

when

the

block

is

not

present

in

the

cache

So

the

sharing-detection

function

for

the

Illinois

protocol

could

be

changed

to

rather

than

the

function

introduced

in

Section

This

would

further

albeit

marginally

simplify

the

global

state

diagram

of

the

Illinois

protocol

because

the

state

-LRB-

Shared

Invalid

-RRB-

would

then

be

contained

by

-LRB-

Shared

Invalid

-RRB-

after

the

refinement

Conclusion

In

this

paper

we

have

introduced

a

simple

method

for

validating

cache

coherence

protocols

at

the

behavior

level

By

exploiting

equivalence

relations

among

global

states

we

can

symbolically

represent

and

generate

the

system

state

space

rather

then

enumerate

it

The

reduction

in

complexity

of

the

verification

procedure

over

existing

approaches

is

so

drastic

that

we

can

contemplate

efficient

verification

of

much

more

complex

protocols

f

Cn

-LRB-

-RRB-

true

if

Ci

s.t.

sta

t

e

C

i

-LRB-

-RRB-

inva

l

i

d

f

alse

otherwise

-LRB-

with

large

number

of

cache

states

such

as

relaxed

consistency

protocols

protocols

with

locked

states

and

protocols

for

hierarchically

organized

machines

We

did

not

find

any

consistency

problem

in

the

five

protocols

that

we

have

have

examined

This

was

somewhat

expected

The

protocols

are

relatively

simple

and

have

been

time-tested

-LRB-

ie

we

would

know

by

now

if

there

was

a

problem

-RRB-

Extension

of

our

work

beyond

more

complex

protocols

could

be

the

definition

of

a

formal

specification

language

capable

of

describing

both

the

protocol

behavior

and

the

processes

implementing

it

This

formal

specification

model

should

facilitate

greater

automatisation

of

the

verification

activities

which

would

reduce

the

possibility

of

errors

It

is

also

possible

to

extend

the

model

and

the

terminology

to

include

more

details

of

the

protocol

implementation

having

to

do

with

ordering

and

consistency

models

References

-LSB-

-RSB-

J.

Archibald

and

J.-L

Baer

Cache

Coherence

Protocols

Evaluation

Using

a

Multiprocessor

Simulation

Model

ACM

Trans

on

Computer

Systems

Vol

Nov.

pp.

-LSB-

-RSB-

J.-L

Baer

and

C.

Girault

A

Petri

Net

Model

for

a

Solution

to

the

Cache

Coherence

Problem

Proc

of

the

Conf

on

Supercomputing

Systems

pp.

-LSB-

-RSB-

G.V.

Bochmann

Combining

Assertions

and

States

For

the

Validation

of

Process

Communication

Proc

of

the

IFIP

-LSB-

-RSB-

G.V.

Bochmann

and

J.

Gecsei

A

Unified

Method

for

the

Specification

and

Verification

of

Protocols

Proc

of

the

IFIP

Toronto

-LSB-

-RSB-

G.V.

Bochmann

and

C.A.

Sunshine

Formal

Methods

in

Communication

Protocol

Design

IEEE

Transactions

on

Communications

Vol

No.

Apr.

pp.

-LSB-

-RSB-

L.M.

Censier

and

P.

Feautrier

A

new

solution

to

coherence

problems

in

multicache

systems

IEEE

Trans

on

Comp.

Dec.

pp.

-LSB-

-RSB-

Y.

Kakuda

Y.

Wakahara

and

M.

Norigoe

An

Acyclic

Expansion

Algorithm

for

Fast

Protocol

Verification

IEEE

Trans

on

Software

Engineering

Vol

No.

Aug.

pp.

-LSB-

-RSB-

G.J.

Holzmann

Algorithms

for

Automated

Protocol

Verification

AT

T

Technical

Journal

Jan.

Feb.

-LSB-

-RSB-

R.H.

Katz

S.J.

Egger

and

et

al.

Implementing

a

Cache

Consistency

Protocol

Proceedings

of

the

International

Symposium

on

Computer

Architecture

-LSB-

-RSB-

K.L.

McMillan

and

J.

Schwalbe

Formal

Verification

of

the

Gigamax

Cache

Consistency

Protocol

Proc

of

the

ISSM

Int?l

Conf

on

Parallel

and

Distributed

Computing

Oct.

-LSB-

-RSB-

K.

Tarnay

Protocol

Specification

and

Testing

Plenum

Press

New

York

-LSB-

-RSB-

P.

Zafiropulo

C.H.

West

H.Rudin

et

al.

Towards

Analyzing

and

Synthesizing

Protocols

IEEE

Transactions

on

Communications

Vol

No.

Apr.

pp.

-LSB-

-RSB-

Brent

T.

Hailpern

Verifying

Concurrent

Processes

Using

Temporal

Logic

Lecture

Notes

in

Computer

Science

Springer-Verlag

New

York

-LSB-

-RSB-

E.

M.

Clarke

E.

A.

Emerson

and

A.

P.

Sistla

Automatic

Verification

of

Finite-State

Concurrent

Systems

using

Temporal

Logic

Specifications

ACM

Trans

on

Programming

Languages

and

Systems

Vol

No.

April

-LSB-

-RSB-

L.

Rudolf

and

Z.

Segall

Dynamic

Decentralized

Cache

Schemes

for

MIMD

Parallel

Processors

Proceedings

of

the

International

Symposium

on

Computer

Architecture

June

pp.

-LSB-

-RSB-

Ashwini

K.

Nanda

and

Laxmi

N.

Bhuyan

A

Formal

Specification

and

Verification

Technique

for

Cache

Coherence

Protocols

Proceedings

of

the

International

Conference

on

Parallel

Processing

pp

Appendix

A

A.

Proof

of

Corollary

The

proof

is

not

formal

but

is

based

on

the

enumeration

of

all

possible

scenarios

The

sketch

of

the

proof

is

as

follows

Proof

By

lemma

and

lemma

we

already

know

that

must

structurally

cover

that

is

It

further

remains

to

show

that

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Consider

the

value

of

F

returned

by

a

composite

state

there

are

three

possibilities

-LRB-

false

false

false

-RRB-

no

cached

copy

exists

and

the

only

possible

composite

state

is

-LRB-

Invalid

-RRB-

-LRB-

true

true

true

false

-RRB-

one

and

only

one

cached

copy

exists

The

composite

state

must

have

the

form

as

-LRB-

Invalid

q

-RRB-

where

q

-LRB-

Q

-LCB-

Invalid

-RCB-

-RRB-

-LRB-

true

true

true

-RRB-

there

are

several

valid

copies

in

the

caches

The

composite

state

has

the

structure

where

qi

-LRB-

Q

-LCB-

Invalid

-RCB-

-RRB-

for

all

i

and

there

must

exist

either

a

qi

with

ri

or

arbitrary

qi

qj

with

ri

rj

or

any

combination

of

above

Since

F

and

return

the

same

values

of

F

either

or

Case

F

-LRB-

-RRB-

F

-LRB-

-RRB-

In

this

case

and

are

identical

and

their

successive

states

and

are

also

identical

which

implies

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Case

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Either

and

are

identical

or

-LRB-

Invalid

q

-RRB-

is

contained

by

-LRB-

Invalid

q

-RRB-

In

both

cases

the

same

transition

t

results

in

and

having

the

same

value

of

F

This

could

be

shown

by

contradiction

Assume

that

F

-LRB-

-RRB-

and

F

-LRB-

-RRB-

This

can

happen

only

if

t

causes

-LRB-

q

fit

Invalid

Invalid

fit

Invalid

-RRB-

in

and

t

has

different

effects

on

which

is

impossible

because

F

Similar

contradictions

could

be

shown

to

refute

the

other

two

cases

say

F

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

and

thus

we

could

conclude

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Inva

l

i

d

q

r

qm

rm

-LRB-

-RRB-

Case

F

-LRB-

-RRB-

F

-LRB-

-RRB-

Let?s

consider

the

values

of

F

returned

by

the

generated

composite

state

-LRB-

a

-RRB-

F

-LRB-

-RRB-

This

can

happen

if

the

transition

t

is

repeatedly

applied

to

on

caches

in

state

q

with

-LRB-

q

fit

Invalid

-RRB-

and

has

the

structure

-LRB-

Invalid

q

-RRB-

should

have

the

same

structure

and

result

in

F

-LRB-

-RRB-

because

the

transition

t

removes

all

cached

copies

in

and

it

will

have

the

same

effect

when

applied

to

and

thus

F

-LRB-

-RRB-

-LRB-

b

-RRB-

F

-LRB-

-RRB-

This

can

happen

if

a

transition

t

is

applied

to

and

removes

all

cached

copies

except

in

the

cache

which

originates

t.

Because

F

t

must

have

the

same

effects

when

applied

to

and

hence

F

-LRB-

-RRB-

-LRB-

c

-RRB-

F

-LRB-

-RRB-

In

this

case

carries

the

information

that

two

or

more

than

two

cached

copies

exist

Since

we

know

that

or

is

structurally

covered

by

two

or

more

than

two

cached

copies

exist

in

that

is

F

-LRB-

-RRB-

From

the

above

we

can

conclude

that

F

-LRB-

-RRB-

F

-LRB-

-RRB-

in

all

cases

A.

Expansion

Steps

for

The

Illinois

Protocol

The

intermediate

steps

for

exploring

the

Illinois

state

space

are

listed

below

-LRB-

Inv

-RRB-

fi

Winv

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Inv

-RRB-

fi

Rinv

fi

-LRB-

V-Ex

Inv

-RRB-

-LRB-

Dirty

Inv

-RRB-

fiRepdirty

fi

-LRB-

Inv

-RRB-

-LRB-

Dirty

Inv

-RRB-

fiWdirty

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Dirty

Inv

-RRB-

fiRdirty

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Dirty

Inv

-RRB-

fiWinv

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Dirty

Inv

-RRB-

fiRninv

fi

-LRB-

Shared

Inv

-RRB-

-LRB-

V-Ex

Inv

-RRB-

fi

Repv-ex

fi

-LRB-

Inv

-RRB-

-LRB-

V-Ex

Inv

-RRB-

fi

Wv-ex

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

V-Ex

Inv

-RRB-

fi

Rv-ex

fi

-LRB-

V-Ex

Inv

-RRB-

-LRB-

V-Ex

Inv

-RRB-

fi

Winv

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

V-Ex

Inv

-RRB-

fi

Rninv

fi

-LRB-

Shared

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiRepnsharedfi

-LRB-

Shared

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiWshared

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiRshared

fi

-LRB-

Shared

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiWinv

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiRninv

fi

-LRB-

Shared

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiRepsharedfi

-LRB-

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiWshared

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiRshared

fi

-LRB-

Shared

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiWinv

fi

-LRB-

Dirty

Inv

-RRB-

-LRB-

Shared

Inv

-RRB-

fiRninv

fi

-LRB-

Shared

Inv

-RRB-

Appendix

B

The

Verification

of

all

Protocols

B.

The

Write-Once

Protocol

The

Write-Once

protocol

is

mainly

characterized

by

the

introduction

of

the

Reserved

cache

state

A

first

time

write

to

a

clean

and

potentially

shared

block

is

a

writethrough

to

memory

and

it

updates

the

main

memory

copy

as

well

as

the

local

copy

The

local

copy

becomes

Reserved

which

indicates

an

exclusive

copy

in

the

system

and

saves

following

write

invalidations

The

result

of

applying

our

algorithm

to

the

write-once

protocol

is

shown

in

Figure

A

processor

always

accesses

fresh

data

and

therefore

data

consistency

is

maintained

FIGURE

The

Write-Once

Protocol

after

State

Visits

Rinvalid

W

Znvalid

Zreserve

Zdirty

Winvalid

-LRB-

reserved

invalid

-RRB-

-LRB-

valid

invalid

-RRB-

-LRB-

dirty

invalid

-RRB-

-LRB-

valid

invalid

-RRB-

Wvalid

Winvalid

Wvalid

Rinvalid

Rninvalid

-LRB-

valid

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

reserved

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

valid

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

dirty

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

obsolete

state

cdata

mdata

B.

The

Berkeley

Protocol

The

Berkeley

protocol

introduced

the

notion

of

ownership

On

a

cache

miss

the

required

data

is

always

supplied

by

the

owner

who

has

the

latest

data

In

the

Berkeley

protocol

caches

in

Dirty

or

Shared-Dirty

states

are

owner

otherwise

the

main

memory

is

the

owner

It

is

a

write-invalidate

protocol

and

the

memory

copy

is

updated

using

a

write-back

policy

when

a

copy

in

states

Dirty

or

Shared-Dirty

are

victimized.The

global

state

transition

diagram

is

shown

as

Figure

A

block

can

be

in

state

Shared-Dirty

in

only

one

cache

but

it

may

also

be

present

in

state

Valid

in

other

caches

FIGURE

The

Berkeley

Protocol

after

State

Visits

Znvalid

Znvalid

Zdirty

-LRB-

share-dirty

valid

invalid

-RRB-

-LRB-

valid

invalid

-RRB-

-LRB-

dirty

invalid

-RRB-

-LRB-

valid

invalid

-RRB-

W

W

Rninvalid

Rninvalid

W

-LRB-

share-dirty

valid

invalid

-RRB-

W

Zshare-dirty

Zshare-dirty

Rninvalid

-LRB-

valid

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

valid

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

shared-dirty

valid

invalid

-RRB-

-LRB-

fresh

fresh

nodata

-RRB-

obsolete

-LRB-

dirty

invalid

-RRB-

-LRB-

fresh

nodata

-RRB-

obsolete

state

cdata

mdata

-LRB-

shared-dirty

valid

invalid

-RRB-

-LRB-

fresh

fresh

nodata

-RRB-

obsolete

B.

The

Firefly

Protocol

Figure

shows

the

global

state

transition

diagram

for

the

Firefly

protocol

The

Firefly

protocol

is

a

write-broadcast

with

dynamic

detection

of

sharing

As

long

as

there

exists

more

than

one

cached

copy

writes

are

broadcast

to

memory

and

to

other

caches

The

Exclusive

state

indicates

that

no

other

cache

has

a

copy

and

that

writes

need

no

longer

be

broadcasted

A

write-back

occurs

when

a

dirty

copy

is

selected

for

replacement

FIGURE

The

Firefly

Protocol

after

State

Visits

B.

The

Dragon

Protocol

In

Fig.

the

output

of

our

algorithm

is

shown

for

the

Dragon

protocol

The

Dragon

protocol

is

a

write-broadcast

as

the

Firefly

protocol

One

difference

is

the

updates

Zshared

Znshared

-LRB-

dirty

invalid

-RRB-

-LRB-

shared

invalid

-RRB-

-LRB-

shared

invalid

-RRB-

-LRB-

invalid

-RRB-

Wshared

RWninvalid

R

W

Wexclusive

Zexclusive

Zdirty

RWninvalid

RWninvalid

-LRB-

exclusive

invalid

-RRB-

-LRB-

invalid

-RRB-

-LRB-

false

-RRB-

-LRB-

nodata

-RRB-

fresh

-LRB-

shared

invalid

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

exclusive

invalid

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

shared

invalid

-RRB-

-LRB-

true

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

dirty

invalid

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

obsolete

state

sharing

-LRB-

F

-RRB-

cdata

mdata

to

shared

blocks

are

not

immediately

reflected

at

main

memory

The

difference

is

the

existence

of

the

SharedDirty

state

The

cache

which

performed

the

latest

write

to

the

shared

block

is

in

the

SharedDirty

state

and

is

responsible

for

supplying

the

block

on

misses

in

remote

caches

and

for

updating

main

memory

on

replacement

A

SharedDirty

copy

is

not

consistent

with

the

main

memory

copy

which

is

apparent

from

the

table

in

Fig.

FIGURE

The

Dragon

Protocol

after

State

Visits

-LRB-

Dirty

Inv

-RRB-

-LRB-

V-Ex

Inv

-RRB-

-LRB-

Inv

-RRB-

WShrdD

WV-Ex

ZV-Ex

ZDirty

RWnInv

-LRB-

ShrdC

inv

-RRB-

-LRB-

ShrdD

Inv

-RRB-

-LRB-

ShrdD

ShrdC

Inv

-RRB-

-LRB-

ShrdC

Inv

-RRB-

RInv

WInv

ZShrdD

ZShrdC

WShrdC

WnInv

ZnShrdC

RWnInv

WnInv

WnInv

WnShrdC

ZShrdD

RnInv

ZnShrdC

RnInv

-LRB-

Inv

-RRB-

-LRB-

false

-RRB-

-LRB-

nodata

-RRB-

fresh

-LRB-

V-Ex

Inv

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

Dirty

Inv

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

obsolete

-LRB-

ShrdC

Inv

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

ShrdD

Inv

-RRB-

-LRB-

false

true

-RRB-

-LRB-

fresh

nodata

-RRB-

obsolete

state

sharing

-LRB-

F

-RRB-

cdata

mdata

-LRB-

ShrdC

Inv

-RRB-

-LRB-

true

true

-RRB-

-LRB-

fresh

nodata

-RRB-

fresh

-LRB-

ShrdD

ShrdC

Inv

-RRB-

-LRB-

true

true

true

-RRB-

-LRB-

fresh

fresh

nodata

-RRB-

obsolete

Inv

Invalid

V-Ex

Valid-Exclusive

ShrdC

SharedClean

ShrdD

SharedDirty

cache coherence protocol
finite state machine
formal verification

symbolic expansion and shared-memory multiprocessor