To

Appear

in

the

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

Detecting

Data

Races

on

Weak

Memory

Systems

Sarita

V.

Adve

Mark

D.

Hill

Barton

P.

Miller

Robert

H.B.

Netzer

Computer

Sciences

Department

University

of

Wisconsin

Madison

Wisconsin

sarita@cs.wisc.edu

ABSTRACT

For

shared-memory

systems

the

most

commonly

assumed

programmer

's

model

of

memory

is

sequential

consistency

The

weaker

models

of

weak

ordering

release

consistency

with

sequentially

consistent

synchronization

operations

and

provide

higher

performance

by

guaranteeing

sequential

consistency

to

only

a

restricted

class

of

programs

mainly

programs

that

do

not

exhibit

data

races

To

allow

programmers

to

use

the

intuition

and

algorithms

already

developed

for

sequentially

consistent

systems

it

is

important

to

determine

when

a

program

written

for

a

weak

system

exhibits

no

data

races

In

this

paper

we

investigate

the

extension

of

dynamic

data

race

detection

techniques

developed

for

sequentially

consistent

systems

to

weak

systems

A

potential

problem

is

that

in

the

presence

of

a

data

race

weak

systems

fail

to

guarantee

sequential

consistency

and

therefore

dynamic

techniques

may

not

give

meaningful

results

However

we

reason

that

in

practice

a

weak

system

will

preserve

sequential

consistency

at

least

until

the

first

data

races

since

it

can

not

predict

if

a

data

race

will

occur

We

formalize

this

condition

and

show

that

it

allows

data

races

to

be

dynamically

detected

Further

since

this

condition

is

already

obeyed

by

all

proposed

implementations

of

weak

systems

the

full

performance

of

weak

systems

can

be

exploited

hhhhhhhhhhhhhhhhhh

Research

supported

in

part

by

National

Science

Foundation

grants

and

Office

of

Naval

Research

grant

and

external

research

grants

from

A.

T.

T.

Bell

Laboratories

Cray

Research

Digital

Equipment

Corporation

and

Texas

Instruments

Introduction

The

programmer

's

model

of

common

sharedmemory

multiprocessor

systems

is

a

collection

of

sequential

processors

and

a

specification

of

the

semantics

of

shared

memory

Intuitively

a

read

of

shared

memory

should

return

the

value

of

the

last

write

to

the

same

address

A

specification

of

shared

memory

semantics

called

a

memory

model

or

memory

consistency

model

is

necessary

to

determine

exactly

which

writes

could

be

last

The

most

commonly

and

often

implicitly

assumed

memory

model

is

sequential

consistency

-LSB-

-RSB-

a

direct

extension

of

the

way

memory

is

viewed

in

a

multiprogrammed

uniprocessor

An

execution

-LRB-

of

a

program

-RRB-

on

a

shared-memory

multiprocessor

is

sequentially

consistent

if

reads

return

values

such

that

all

memory

operations

appear

to

have

occurred

in

a

sequential

order

where

the

order

of

operations

of

an

individual

processor

is

as

specified

by

its

program

A

system

provides

sequential

consistency

if

it

only

allows

sequentially

consistent

executions

Implementing

sequential

consistency

in

shared-memory

multiprocessors

however

restricts

the

use

of

many

performance

enhancing

features

of

uniprocessor

systems

such

as

outof-order

instruction

issue

and

completion

write

buffers

and

lockup-free

caches

-LSB-

-RSB-

For

higher

performance

researchers

have

proposed

alternative

memory

models

-LSB-

-RSB-

These

models

however

do

not

provide

sequential

consistency

to

all

programs

To

allow

programmers

to

use

the

intuition

and

algorithms

already

developed

for

sequential

consistency

it

is

important

to

characterize

the

programs

for

which

the

models

do

provide

sequential

consistency

The

models

of

weak

ordering

-LRB-

WO

-RRB-

-LSB-

-RSB-

release

consistency

with

sequentially

consistent

synchronization

operations

-LRB-

RCsc

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

-LSB-

-RSB-

and

-LRB-

-RRB-

-LSB-

-RSB-

are

similar

in

the

respect

that

they

provide

sequential

consistency

to

all

programs

that

exhibit

no

data

races

in

any

execution

on

sequentially

consistent

hardware

-LRB-

i.e.

to

data-race-free

programs

-RRB-

Informally

two

memory

operations

in

an

execution

form

a

data

race

if

at

least

one

of

them

is

a

data

operation

-LRB-

as

opposed

to

a

synchronization

operation

-RRB-

at

least

one

of

them

is

a

write

-LRB-

as

opposed

to

a

read

-RRB-

both

access

the

same

memory

location

and

they

are

not

ordered

by

intervening

synchronization

operations

We

would

like

to

determine

when

programs

for

the

above

models

are

data-race-free

so

programmers

can

assume

sequential

consistency

when

designing

and

debugging

their

programs

Henceforth

the

systems

specified

by

WO

RCsc

and

will

collectively

be

referred

to

as

the

weak

systems

and

an

execution

on

a

weak

system

will

be

referred

to

as

a

weak

execution

The

problem

of

detecting

data

races

is

not

unique

to

weak

systems

Even

on

sequentially

consistent

systems

the

presence

of

data

races

makes

it

hard

to

reason

about

a

program

and

is

usually

considered

to

be

a

bug

Much

current

research

in

parallel

programming

has

therefore

been

devoted

to

detecting

data

races

in

programs

written

for

sequentially

consistent

systems

Static

techniques

perform

a

compile-time

analysis

of

the

program

text

to

detect

a

superset

of

all

possible

data

races

that

could

potentially

occur

in

all

possible

sequentially

consistent

executions

of

the

program

-LSB-

-RSB-

In

general

static

analysis

must

be

conservative

and

slow

because

detecting

data

races

is

undecidable

for

arbitrary

programs

-LSB-

-RSB-

and

is

NP-complete

for

even

very

restricted

classes

of

programs

-LRB-

e.g.

those

containing

no

branches

-RRB-

-LSB-

-RSB-

Dynamic

techniques

on

the

other

hand

use

a

tracing

mechanism

to

detect

whether

a

particular

sequentially

consistent

execution

of

a

program

actually

exhibited

a

data

race

-LSB-

-RSB-

While

dynamic

techniques

provide

precise

information

about

a

single

execution

they

provide

little

information

about

other

executions

a

serious

disadvantage

For

these

reasons

the

general

consensus

among

researchers

investigating

data

race

detection

is

that

tools

should

support

both

static

and

dynamic

techniques

in

a

complementary

fashion

-LSB-

-RSB-

Rather

than

start

from

scratch

we

seek

to

apply

the

data

race

detection

techniques

from

sequentially

consistent

systems

to

weak

systems

Static

techniques

can

be

applied

to

programs

for

weak

systems

unchanged

because

they

do

not

rely

on

executing

the

program

Dynamic

techniques

however

depend

on

executing

a

program

If

a

program

has

no

data

races

then

all

executions

of

it

on

a

weak

system

will

be

sequentially

consistent

and

the

dynamic

techniques

will

correctly

conclude

that

no

data

races

occurred

If

a

program

has

data

races

on

the

other

hand

an

execution

of

it

on

a

weak

system

may

not

be

sequentially

consistent

Applying

the

dynamic

techniques

to

such

an

execution

may

produce

unpredictable

results

In

this

paper

we

develop

a

hardware

condition

for

weak

systems

that

allows

data

races

to

be

dynamically

detected

The

key

observation

we

use

is

that

practical

implementations

of

weak

systems

give

sequential

consistency

at

least

until

the

first

data

races

-LRB-

those

not

affected

by

others

-RRB-

because

they

can

not

predict

that

a

data

race

will

occur

We

formalize

this

observation

with

a

condition

that

all

executions

on

weak

systems

have

a

sequentially

consistent

prefix

that

extends

to

the

first

data

races

-LRB-

or

the

end

of

the

execution

-RRB-

With

this

condition

we

show

how

to

use

a

dynamic

approach

that

either

-LRB-

a

-RRB-

correctly

determines

no

data

races

occurred

and

concludes

that

the

execution

was

sequentially

consistent

or

-LRB-

b

-RRB-

identifies

the

first

data

races

that

could

have

occurred

in

a

sequentially

consistent

execution

with

approximately

the

same

accuracy

as

on

a

sequentially

consistent

system

Furthermore

since

all

implementations

of

WO

and

RCsc

and

all

proposed

implementations

of

and

already

exhibit

the

required

sequentially

consistent

prefix

we

argue

that

the

new

condition

is

obeyed

for

free

in

practice

Alternatively

one

could

use

dynamic

techniques

on

weak

systems

by

having

weak

systems

support

a

slower

sequentially

consistent

mode

that

is

used

for

debugging

The

results

of

this

paper

however

show

that

such

a

slower

mode

is

not

necessary

for

detecting

data

races

Furthermore

we

believe

that

our

results

also

allow

other

debugging

tools

for

sequentially

consistent

systems

to

be

used

unchanged

on

weak

systems

If

there

are

no

data

races

in

the

execution

then

the

execution

is

sequentially

consistent

and

other

debugging

tools

can

be

directly

applied

If

there

are

data

races

then

the

presence

of

the

sequentially

consistent

prefix

allows

the

tools

to

be

applied

to

the

part

of

the

execution

that

contains

the

first

bugs

of

the

program

The

rest

of

the

paper

is

organized

as

follows

Section

formalizes

the

notion

of

a

data

race

and

briefly

reviews

the

weak

models

Section

discusses

the

potential

problems

in

applying

the

current

dynamic

techniques

to

weak

systems

develops

the

hardware

condition

to

overcome

these

problems

and

explains

why

all

proposed

implementations

of

the

weak

systems

already

obey

this

condition

Section

shows

how

data

races

can

be

dynamically

detected

on

a

system

obeying

our

condition

Section

discusses

how

the

limitations

of

our

dynamic

technique

on

weak

hardware

are

similar

to

those

that

occur

on

sequentially

consistent

hardware

Section

concludes

the

paper

Data

Races

and

Weak

Models

In

this

section

we

formalize

the

notion

of

a

data

race

on

sequentially

consistent

systems

and

then

briefly

review

the

weak

models

Formalizing

Data

Races

We

first

clarify

some

terminology

used

throughout

the

paper

The

term

program

refers

to

the

program

text

-LRB-

a

set

of

machine

instructions

-RRB-

and

the

input

data

A

machine

instruction

involves

zero

or

more

memory

operations

each

memory

operation

either

reads

a

memory

location

-LRB-

a

read

operation

-RRB-

or

modifies

a

memory

location

-LRB-

a

write

operation

-RRB-

Further

an

operation

is

uniquely

identified

by

the

location

it

accesses

and

the

part

of

the

program

in

which

it

is

specified

the

value

it

reads

or

writes

is

not

considered

For

each

of

its

executions

a

program

defines

a

partial

order

on

the

memory

operations

of

that

execution

called

the

pro-

gram

order

-LRB-

denoted

by

po

-RRB-

Finally

two

memory

operations

conflict

if

they

access

the

same

location

and

at

least

one

of

them

is

a

write

operation

Memory

operations

can

be

partitioned

into

two

groups

synchronization

operations

that

are

used

to

order

events

and

the

more

frequent

data

operations

that

read

and

write

data

Often

different

instructions

are

used

for

the

two

groups

We

will

call

an

operation

a

synchronization

operation

if

it

is

recognized

by

the

hardware

as

meant

for

synchronization

For

example

a

synchronization

operation

could

be

a

read

or

a

write

operation

by

a

special

instruction

such

as

a

Test

Set

or

it

could

be

an

ordinary

read

or

a

write

operation

to

a

special

location

known

to

the

hardware

Operations

that

are

not

recognized

as

synchronization

by

the

hardware

will

be

called

data

operations

For

a

formal

definition

of

a

data

race

we

define

a

or

relation

for

every

sequentially

consistent

execution

of

a

program

Our

definition

is

closely

related

to

the

happened-before

relation

defined

by

Lam

port

-LSB-

-RSB-

for

message-passing

systems

The

relation

for

a

sequentially

consistent

execution

is

a

partial

order

on

its

memory

operations

Two

operations

initiated

by

the

same

processor

are

ordered

by

according

to

program

order

Two

operations

initiated

by

different

pro-

cessors

are

ordered

by

only

if

there

exist

intervening

synchronization

operations

Further

with

the

synchronization

operations

that

are

allowed

to

be

used

for

ordering

there

should

be

associated

certain

semantics

We

encapsulate

these

semantics

by

using

the

classification

of

release

and

acquire

synchronization

operations

proposed

in

-LSB-

-RSB-

and

a

pairing

relation

as

follows

Definition

Two

synchronization

operations

s

and

s

are

paired

iff

the

following

are

satisfied

-LRB-

-RRB-

Synchronization

operation

s

is

a

write

and

can

be

used

by

a

processor

to

communicate

the

completion

of

all

its

previous

memory

operations

-LRB-

in

program

order

-RRB-

to

a

processor

that

subsequently

executes

operation

s

Such

a

write

is

called

a

release

operation

-LRB-

-RRB-

Synchronization

operation

s

is

a

read

and

can

be

used

by

a

processor

to

conclude

the

completion

of

previous

operations

of

a

processor

that

executed

an

s

earlier

in

the

execution

Such

a

read

is

called

an

acquire

operation

-LRB-

-RRB-

Operations

s

and

s

access

the

same

location

and

s

returns

the

value

written

by

s

For

every

system

the

classification

of

synchronization

operations

into

release

and

acquire

operations

that

can

potentially

be

paired

can

be

determined

by

the

semantics

that

are

associated

with

the

operations

For

example

in

a

system

that

provides

the

Unset

and

Test

Set

instructions

a

write

due

to

an

Unset

is

a

release

that

could

be

paired

with

a

read

due

to

a

Test

Set

which

is

an

acquire

However

the

write

due

to

a

Test

Set

is

not

a

release

since

it

is

not

meant

to

be

used

to

communicate

the

completion

of

previous

memory

operations

of

its

processor

Memory

operations

initiated

by

different

processors

are

ordered

by

only

if

paired

release

and

acquire

operations

occur

between

them

For

a

more

formal

definition

we

use

the

program

order

-LRB-

po

-RRB-

relation

and

the

-LRB-

-RRB-

relation

defined

on

the

synchronization

operations

of

a

sequentially

consistent

execution

as

follows

Definition

s

s

iff

s

is

a

release

operation

and

s

is

a

paired

acquire

operation

Definition

The

relation

for

a

sequential

ly

consistent

execution

is

defined

as

-LRB-

po

-RRB-

the

irreflexive

transitive

closure

of

po

and

The

definitions

of

a

data

race

a

data-race-free

execution

and

a

data-race-free

program

follow

Definition

Two

memory

operations

x

and

y

in

a

sequentially

consistent

execution

form

a

race

x

y

iff

x

and

y

conflict

and

they

are

not

ordered

by

the

relation

of

the

execution

The

race

x

y

is

a

data

race

iff

at

least

one

of

x

or

y

is

a

data

operation

A

sequentially

consistent

execution

is

data-race-free

iff

of

its

operations

form

a

data

race

A

program

is

data-race-free

iff

all

its

sequentially

consistent

executions

are

data-race-free

Figures

and

respectively

show

sequentially

consistent

executions

with

and

without

data

races

The

Pi

's

denote

processors

op

-LRB-

x

-RRB-

denotes

a

memory

operation

on

the

location

x.

Operations

denoted

by

Read

and

Write

are

data

operations

and

the

remaining

are

synchronization

operations

A

write

operation

due

to

an

Unset

can

be

paired

with

a

read

operation

due

to

a

Test

Set

The

execution

of

Figure

is

not

data-race-free

since

the

conflicting

write

and

read

data

operations

of

and

respectively

are

not

or

dered

by

The

execution

of

Figure

-LRB-

b

-RRB-

is

datarace-free

because

all

conflicting

data

operations

are

ordered

by

and

no

synchronization

operation

conflicts

with

a

data

operation

-LRB-

a

-RRB-

Write

-LRB-

x

-RRB-

Read

-LRB-

x

-RRB-

po

po

po

Read

-LRB-

y

-RRB-

Write

-LRB-

x

-RRB-

Read

-LRB-

x

-RRB-

po

po

Test

Set

-LRB-

s

-RRB-

Unset

-LRB-

s

-RRB-

po

po

Read

-LRB-

y

-RRB-

-LRB-

b

-RRB-

Unset

-LRB-

s

-RRB-

Write

-LRB-

y

-RRB-

Write

-LRB-

y

-RRB-

Figure

Executions

-LRB-

a

-RRB-

with

and

-LRB-

b

-RRB-

without

data

races

Weak

Memory

Models

We

now

briefly

review

and

compare

the

weak

memory

models

that

we

consider

in

this

paper

For

a

datarace-free

program

between

two

conflicting

data

operations

of

different

processors

there

are

explicit

hardware

recognizable

synchronization

operations

This

allows

most

of

the

actions

to

ensure

sequential

consistency

to

be

delayed

from

the

data

operation

to

the

subsequent

synchronization

operation

leading

to

higher

performance

All

four

weak

models

considered

in

this

paper

are

based

on

this

general

principle

For

example

consider

the

data-race-free

execution

in

Figure

A

conventional

implementation

of

sequential

consistency

would

stall

on

every

memory

operation

until

its

completion

Specifically

would

stall

on

each

of

its

writes

However

the

weak

models

only

require

that

either

the

writes

of

are

complete

before

the

Unset

is

issued

or

before

the

Test

Set

of

succeeds

Either

approach

guarantees

sequential

consistency

for

the

execution

in

Figure

To

see

how

sequential

consistency

may

be

violated

in

a

weak

system

in

the

presence

of

data

races

consider

the

execution

in

Figure

In

this

execution

it

is

possible

that

the

new

value

of

y

written

by

is

propagated

to

before

the

new

value

of

x.

Thus

it

is

possible

for

to

read

the

new

value

for

y

but

the

old

value

for

x

thereby

violating

sequential

consistency

We

now

discuss

how

the

four

weak

models

differ

from

each

other

The

model

of

WO

-LSB-

-RSB-

is

specified

as

a

set

of

explicit

conditions

on

hardware

It

has

been

shown

that

these

conditions

ensure

sequential

consistency

for

all

data-race-free

programs

-LSB-

-RSB-

The

model

of

RCsc

-LSB-

-RSB-

is

also

specified

as

a

set

of

conditions

on

hardware

The

main

difference

between

the

conditions

of

WO

and

RCsc

is

that

the

latter

exploit

the

distinction

between

acquire

and

release

synchronization

operations

The

hardware

conditions

of

RCsc

are

accompanied

by

a

formalization

of

the

programs

for

which

they

ensure

sequential

consistency

Such

programs

have

been

called

properly

labeled

with

respect

to

sequential

consistency

and

are

the

same

as

data-race-free

programs

-LSB-

-RSB-

There

do

exist

some

programs

that

exhibit

certain

restricted

kinds

of

data

races

for

which

WO

and

RCsc

hardware

ensure

sequential

consistency

-LSB-

-RSB-

These

programs

however

are

few

and

are

not

formally

characterized

Hence

for

all

practical

purposes

we

assume

that

for

sequential

consistency

to

be

guaranteed

by

these

models

programs

should

be

data-race-free

-LSB-

-RSB-

and

-LSB-

-RSB-

are

defined

to

include

all

hardware

that

guarantees

sequential

consistency

to

programs

that

are

data-racefree

differs

from

in

that

it

does

not

distinguish

between

acquire

and

release

operations

A

Hardware

Condition

for

Dynamically

Detecting

Data

Races

In

this

section

we

first

describe

the

problems

that

could

potentially

limit

the

use

of

current

dynamic

data

race

detection

techniques

on

weak

systems

We

then

give

a

hardware

condition

that

addresses

these

limitations

and

explain

why

this

condition

is

already

obeyed

by

all

proposed

implementations

of

the

weak

systems

Problems

in

Applying

Dynamic

Techniques

to

Weak

Systems

Existing

dynamic

data

race

detection

techniques

for

sequentially

consistent

systems

-LSB-

-RSB-

instrument

the

program

to

record

information

about

the

memory

operations

of

its

exe

cution

This

information

allows

the

relation

to

be

constructed

thereby

allowing

the

detection

of

data

races

There

are

two

approaches

for

recording

and

using

this

information

The

post-mortem

techniques

generate

trace

files

containing

the

order

of

all

synchronization

operations

to

the

same

location

and

the

memory

locations

accessed

between

two

synchronization

operations

in

a

given

process

These

trace

files

are

analyzed

after

the

execution

the

ord

ering

of

the

synchronization

operations

allows

the

relation

of

the

execution

to

be

constructed

from

which

the

relation

can

be

constructed

The

pairs

of

conflicting

operations

-LRB-

at

least

one

of

which

is

data

-RRB-

that

are

not

or

dered

by

the

relation

are

reported

as

data

races

The

on-the-fly

techniques

do

not

produce

explicit

trace

files

but

buffer

partial

trace

information

in

memory

and

detect

data

races

as

they

occur

during

the

execution

The

difficulty

in

applying

the

above

techniques

on

weak

systems

is

that

the

weak

execution

may

not

be

sequentially

consistent

-LRB-

if

the

program

is

not

data-racefree

-RRB-

For

such

an

execution

we

first

need

to

formalize

the

hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

Enqueue

-LRB-

addr

-RRB-

addr

Dequeue

-LRB-

-RRB-

compute

addr

of

region

else

on

which

to

work

write

-LRB-

Q

-RRB-

Unset

-LRB-

s

-RRB-

read

-LRB-

Q

-RRB-

work

on

region

Unset

-LRB-

s

-RRB-

-LRB-

b

-RRB-

-LRB-

a

-RRB-

Sequentially

consistent

data

races

Non-sequentially

consistent

data

races

work

on

region

addr

addr

-LCB-

missing

Test

Set

-RCB-

-LCB-

missing

Test

Set

-RCB-

write

-LRB-

-RRB-

write

-LRB-

-RRB-

write

-LRB-

-RRB-

write

-LRB-

-RRB-

read

-LRB-

-RRB-

write

-LRB-

-RRB-

Unset

-LRB-

S

-RRB-

Unset

-LRB-

S

-RRB-

QEmpty

False

if

-LRB-

QEmpty

False

-RRB-

then

write

-LRB-

QEmpty

-RRB-

read

-LRB-

QEmpty

-RRB-

End

of

SCP

SCP

is

defined

in

Section

Figure

-LRB-

a

-RRB-

Program

fragment

and

-LRB-

b

-RRB-

example

graph

showing

data

races

and

SCP

hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

notion

of

a

data

race

This

can

be

done

in

a

manner

analogous

to

that

for

sequentially

consistent

executions

by

defining

the

and

relations

Note

that

since

in

general

the

synchronization

operations

of

a

weak

system

are

not

constrained

to

be

executed

in

a

sequentially

con

sistent

manner

the

relation

and

hence

the

relation

may

contain

cycles

and

hence

not

be

partial

orders

Nevertheless

the

current

dynamic

techniques

for

sequentially

consistent

executions

can

still

be

applied

to

find

the

data

races

of

a

weak

execution

Although

data

races

of

a

weak

execution

can

be

easily

detected

with

the

current

techniques

there

are

two

potential

problems

that

need

to

be

resolved

for

the

reported

data

races

to

be

meaningful

The

first

problem

is

that

on

arbitrary

weak

hardware

it

is

theoretically

possible

for

an

execution

to

not

exhibit

data

races

and

yet

not

be

sequentially

consistent

Fortunately

as

we

shall

see

later

the

implementations

of

the

weak

models

proposed

to

date

do

not

exhibit

this

problem

The

second

problem

is

that

data

races

that

may

be

reported

from

weak

executions

may

never

occur

on

any

sequentially

consistent

execution

of

the

program

This

could

easily

occur

on

the

current

weak

systems

Figure

illustrates

a

program

fragment

and

one

weak

execution

in

which

such

non-sequentially

consistent

data

races

occur

In

this

program

processor

places

the

starting

address

of

a

region

for

processor

to

work

on

in

a

shared

queue

and

resets

the

QEmpty

flag

Processor

checks

to

determine

if

work

is

available

and

dequeues

an

address

if

the

queue

is

not

empty

Processor

works

independently

on

some

part

of

the

address

space

Since

and

both

access

the

shared

queue

operations

manipulating

the

queue

are

enclosed

in

critical

sections

implemented

with

the

Test

Set

and

Unset

instructions

However

due

to

an

oversight

the

Test

Set

instructions

were

omitted

The

program

is

therefore

not

data-race-free

A

sequentially

consistent

execution

of

this

program

will

exhibit

data

races

between

the

accesses

to

the

queue

and

the

variable

QEmpty

Because

the

program

is

not

data-race-free

sequential

consistency

can

be

violated

in

an

execution

on

a

weak

system

One

such

execution

is

shown

in

Figure

where

op

-LRB-

x

a

-RRB-

represents

a

read

or

a

write

memory

operation

to

location

x

that

respectively

returns

or

stores

the

value

a

In

this

execution

although

processor

finds

QEmpty

to

be

reset

it

does

not

read

the

new

value

enqueued

by

Instead

it

reads

an

old

value

in

this

case

The

region

that

processor

starts

working

on

now

overlaps

with

the

region

accessed

by

This

gives

rise

to

many

data

races

between

the

operations

of

and

as

shown

On

a

sequentially

consistent

system

could

never

have

returned

the

value

and

hence

these

races

would

never

have

occurred

Nevertheless

naively

using

the

dynamic

techniques

would

report

all

of

these

data

races

For

debugging

programs

for

weak

memory

systems

we

are

only

interested

in

detecting

data

races

that

would

also

occur

in

a

sequentially

consistent

execution

of

the

program

Further

the

motivation

of

detecting

data

races

was

to

allow

programmers

to

reason

in

terms

of

sequential

consistency

Therefore

reporting

data

races

that

can

not

occur

in

a

sequentially

consistent

execution

can

be

confusing

to

the

programmer

and

can

complicate

the

task

of

debugging

In

the

next

sub-section

we

develop

a

hardware

condition

that

addresses

this

problem

A

Hardware

Condition

for

Dynamic

Data

Race

Detection

We

develop

a

hardware

condition

such

that

for

any

execution

on

a

system

that

obeys

this

condition

a

set

of

data

races

that

would

also

occur

in

some

sequentially

consistent

execution

can

be

identified

In

Section

we

will

show

how

dynamic

detection

can

be

used

to

report

this

set

Intuitively

our

condition

requires

that

an

implementation

guarantee

sequential

consistency

until

a

data

race

actually

occurs

in

the

execution

Further

once

a

data

race

occurs

sequential

consistency

should

be

violated

only

in

parts

of

the

execution

that

are

affected

by

the

data

race

This

will

ensure

that

every

execution

has

a

sequentially

consistent

prefix

that

contains

the

first

data

races

-LRB-

those

not

affected

by

others

-RRB-

of

the

execution

Dynamic

techniques

can

then

potentially

be

applied

to

this

sequentially

consistent

prefix

to

report

the

first

races

We

introduce

the

following

terminology

to

formalize

our

condition

Definition

A

prefix

of

an

execution

E

is

a

subset

of

the

memory

operations

of

E

such

that

if

y

is

in

the

prefix

and

x

y

in

E

then

x

is

in

the

prefix

Definition

A

prefix

of

an

execution

E

of

a

program

P

is

a

sequentially

consistent

prefix

or

SCP

of

E

iff

-LRB-

-RRB-

it

is

also

the

prefix

of

a

sequentially

consistent

execution

Eseq

of

program

P

and

-LRB-

-RRB-

if

x

and

y

are

in

the

prefix

then

x

y

is

a

data

race

in

E

iff

x

y

is

also

a

data

race

in

Eseq

An

example

of

an

SCP

is

shown

in

Figure

Thus

for

an

execution

E

of

any

program

the

operations

of

a

processor

in

its

SCP

are

also

the

initial

operations

of

the

processor

in

some

sequentially

consistent

execution

Eseq

of

the

program

Further

a

data

race

involving

operations

in

the

SCP

occurs

in

E

iff

it

also

occurs

in

Eseq

This

implies

that

the

set

of

data

races

that

have

their

operations

in

a

particular

SCP

is

a

valid

set

of

sequentially

consistent

data

races

to

report

To

enable

the

identification

of

data

races

in

an

SCP

we

propose

a

condition

that

in

an

execution

either

a

data

race

has

its

operations

in

a

specific

SCP

of

the

execution

or

the

data

race

is

affected

by

another

data

race

with

operations

in

the

SCP

where

affected

is

defined

as

follows

Definition

A

race

x

y

affects

a

memory

operation

z

written

x

y

A

z

iff

-LRB-

-RRB-

z

is

the

same

memory

operation

as

x

or

y

or

-LRB-

-RRB-

x

z

or

y

z

or

-LRB-

-RRB-

there

exists

a

race

x

y

such

that

x

y

A

z

and

either

x

y

A

x

or

x

y

A

y

A

race

x

y

affects

another

race

x

y

written

x

y

A

x

y

iff

x

y

A

x

or

x

y

A

y

This

implies

that

data

races

that

are

not

affected

by

any

other

data

race

-LRB-

intuitively

the

first

data

races

-RRB-

should

always

be

in

an

SCP

of

the

execution

i.e.

they

should

also

occur

in

a

sequentially

consistent

execution

Thus

the

data

races

that

are

not

affected

by

any

others

constitute

a

valid

set

of

data

races

that

can

be

reported

The

formal

hardware

condition

that

we

propose

is

as

follows

We

say

that

a

race

x

y

occurs

in

an

SCP

if

the

operations

x

and

y

are

in

the

SCP

Condition

For

any

execution

E

of

a

program

P

-LRB-

-RRB-

if

there

are

no

data

races

in

E

then

E

is

a

sequentially

consistent

execution

of

program

P

and

-LRB-

-RRB-

there

exists

an

SCP

of

E

such

that

a

data

race

in

E

either

occurs

in

the

SCP

or

is

affected

by

another

data

race

that

occurs

in

the

SCP

Condition

-LRB-

-RRB-

ensures

that

if

no

data

races

are

reported

then

the

programmer

can

safely

assume

that

the

hardware

is

sequentially

consistent

overcoming

the

first

problem

cited

in

the

previous

section

Condition

-LRB-

-RRB-

ensures

that

if

a

data

race

is

detected

in

E

then

there

is

a

data

race

in

E

that

affects

this

data

race

that

also

occurs

in

some

sequentially

consistent

execution

of

the

program

Thus

the

set

of

data

races

that

are

not

affected

by

any

other

data

race

in

E

form

a

valid

reportable

set

of

data

races

that

also

occur

in

some

sequentially

consistent

execution

Weak

Hardware

Already

Obeys

Condition

for

Dynamic

Data

Race

Detection

In

this

subsection

we

give

the

most

significant

result

of

our

paper

Theorem

Condition

for

dynamic

debugging

is

already

obeyed

by

all

implementations

of

WO

and

RCsc

and

all

proposed

implementations

of

and

Due

to

space

constraints

in

this

paper

we

only

give

an

informal

intuition

for

this

result

A

formal

proof

of

the

theorem

appears

in

-LSB-

-RSB-

For

brevity

we

use

the

term

all

weak

implementations

to

imply

all

possible

implementations

of

WO

and

RCsc

and

all

implementations

of

and

that

have

been

proposed

to

date

Condition

-LRB-

-RRB-

requires

that

for

a

data-race-free

execution

on

a

weak

system

the

weak

hardware

should

appear

sequentially

consistent

All

weak

implementations

guarantee

sequential

consistency

to

data-race-free

programs

They

achieve

this

by

guaranteeing

sequential

consistency

to

every

data-race-free

execution

on

the

weak

sys

tem

-LSB-

-RSB-

thereby

obeying

Condition

-LRB-

-RRB-

Condition

-LRB-

-RRB-

requires

that

every

data

race

in

a

weak

execution

that

is

not

affected

by

any

other

data

race

should

also

occur

in

a

specific

sequentially

consistent

execution

of

the

same

program

The

data

races

that

are

not

affected

by

any

others

are

intuitively

the

first

data

races

in

an

execution

Therefore

Condition

-LRB-

-RRB-

can

be

obeyed

by

ensuring

that

an

execution

provides

sequential

consistency

until

a

data

race

actually

occurs

Even

then

a

violation

of

sequential

consistency

should

be

allowed

to

occur

only

for

operations

that

are

directly

affected

by

the

data

race

Intuitively

this

is

true

for

all

weak

implementations

for

the

following

reason

Weak

implementations

are

allowed

to

violate

sequential

consistency

only

for

executions

that

exhibit

data

races

Practically

however

we

expect

that

it

is

not

possible

to

predict

whether

an

execution

will

exhibit

a

data

race

until

a

data

race

actually

occurs

in

the

execution

Thus

all

weak

implementations

provide

sequential

consistency

until

a

data

race

actually

occurs

in

the

execution

and

violate

sequential

consistency

only

in

the

parts

of

the

execution

that

are

affected

by

the

data

races

The

formal

proof

for

this

condition

shows

the

existence

of

an

SCP

for

every

execution

on

a

weak

implementation

such

that

either

all

operations

of

a

processor

are

in

the

SCP

or

the

first

operation

of

a

processor

not

in

the

SCP

is

affected

by

a

data

race

in

the

SCP

Detecting

Data

Races

on

Weak

Hardware

In

this

section

we

show

how

Condition

can

be

used

to

dynamically

detect

sequentially

consistent

data

races

on

weak

systems

We

employ

a

post-mortem

approach

to

locate

sets

of

data

races

each

of

which

contains

at

least

one

that

belongs

to

a

specific

SCP

Recall

that

a

data

race

occurs

in

an

SCP

only

if

it

also

occurs

in

the

corresponding

sequentially

consistent

execution

Our

approach

thus

directs

the

programmer

to

data

races

that

would

have

also

occurred

in

some

sequentially

consistent

execution

of

the

program

In

Section

we

argue

that

any

limitations

of

this

technique

are

analogous

to

those

that

occur

with

dynamic

detection

on

sequentially

consistent

hardware

We

also

explore

the

possibility

of

using

an

onthe-fly

approach

in

Section

Program

Instrumentation

So

far

data

races

and

hardware

constraints

have

been

expressed

in

terms

of

individual

operations

Detecting

data

races

at

such

a

low

level

would

require

tracing

the

program

order

of

all

memory

operations

performed

by

each

processor

which

in

general

would

be

impractical

Instead

higher-level

information

about

the

execution

can

be

recorded

and

used

for

data

race

detection

The

execution

of

each

processor

can

be

viewed

as

a

sequence

of

events

that

represent

groups

of

memory

operations

We

adopt

the

approach

used

by

previous

data

race

detection

methods

-LSB-

-RSB-

and

define

an

event

to

be

either

a

single

synchronization

operation

-LRB-

a

synchronization

event

-RRB-

or

a

group

of

consecutively

executed

data

operations

-LRB-

a

computation

event

-RRB-

For

example

the

events

of

the

execution

in

Figure

are

shown

in

Figure

as

sets

of

memory

operations

enclosed

within

rectangles

Each

event

A

has

two

attributes

READ

-LRB-

A

-RRB-

and

WRITE

-LRB-

A

-RRB-

the

sets

of

memory

locations

read

and

written

by

the

event

-LRB-

upper

case

letters

denote

events

-RRB-

Such

a

higher-level

view

is

useful

since

recording

the

READ

and

WRITE

sets

is

in

general

more

efficient

than

tracing

every

memory

operation

performed

by

the

event

For

example

bit-vectors

representing

those

-LRB-

shared

-RRB-

variables

that

might

be

accessed

between

two

synchronization

events

can

be

constructed

and

when

a

variable

is

accessed

the

corresponding

bit

is

set

The

bit-vector

is

then

written

to

a

trace

file

at

the

end

of

each

computation

event

Such

an

approach

avoids

writing

a

trace

record

for

every

memory

operation

write

-LRB-

Q

-RRB-

Unset

-LRB-

s

-RRB-

read

-LRB-

Q

-RRB-

Unset

-LRB-

s

-RRB-

write

-LRB-

-RRB-

write

-LRB-

-RRB-

write

-LRB-

-RRB-

write

-LRB-

-RRB-

read

-LRB-

-RRB-

write

-LRB-

-RRB-

Non-first

partition

First

partition

write

-LRB-

QEmpty

-RRB-

read

-LRB-

QEmpty

-RRB-

End

of

SCP

Figure

Augmented

graph

showing

first

and

non-first

data

race

partitions

We

assume

that

the

program

is

instrumented

to

produce

trace

files

containing

-LRB-

-RRB-

the

execution

order

of

events

issued

by

the

same

processor

-LRB-

-RRB-

the

relative

execution

order

of

synchronization

events

involving

the

same

location

and

-LRB-

-RRB-

the

READ

and

WRITE

sets

for

each

computa

tion

event

From

-LRB-

-RRB-

and

-LRB-

-RRB-

the

relation

-LRB-

defined

over

events

instead

of

operations

-RRB-

can

be

represented

by

constructing

the

graph

which

contains

one

node

for

every

event

and

edges

between

nodes

to

represent

the

po

and

relations

Data

races

are

defined

for

computation

events

in

the

same

way

as

they

were

defined

for

data

operations

a

data

race

between

two

computation

events

A

B

exists

if

A

and

B

access

a

common

location

that

at

least

one

of

them

writes

and

no

path

connects

A

and

B

in

the

graph

We

use

the

term

lower-level

data

race

to

refer

to

a

data

race

among

memory

operations

-LRB-

e.g.

a

b

-RRB-

and

the

term

higher-level

data

race

-LRB-

or

just

data

race

-RRB-

to

refer

to

a

data

race

among

events

-LRB-

e.g.

A

B

-RRB-

Since

A

and

B

represent

groups

of

memory

operations

the

data

race

A

B

may

represent

many

lower-level

data

races

among

the

individual

operations

comprising

A

and

B

If

at

least

one

of

these

lowerlevel

data

races

belongs

to

an

SCP

then

we

say

that

the

higher-level

data

race

does

also

Locating

Data

Races

in

an

SCP

We

now

show

how

to

locate

data

races

that

are

in

an

SCP

Intuitively

we

try

to

identify

the

first

data

races

that

occurred

We

add

edges

to

the

graph

G

to

construct

an

augmented

graph

G

The

purpose

of

these

edges

is

to

capture

the

possible

effect

one

data

race

may

have

on

another

For

each

race

we

add

a

doubly

directed

edge

between

the

two

nodes

involved

in

the

race

Then

given

two

races

A

B

and

C

D

a

path

exists

in

G

from

A

-LRB-

or

B

-RRB-

to

C

-LRB-

or

D

-RRB-

iff

A

B

A

C

D

The

graph

G

for

the

execution

in

Figure

is

shown

in

Figure

If

for

an

execution

the

augmented

graph

G

were

acyclic

it

would

define

a

partial

ordering

among

the

data

races

By

construction

the

first

data

races

as

given

by

the

partial

order

would

not

be

affected

by

any

other

races

and

hence

by

Condition

-LRB-

-RRB-

these

races

would

be

guaranteed

to

belong

to

an

SCP

However

since

G

may

contain

cycles

we

need

to

partition

the

data

races

and

define

a

partial

ordering

among

the

partitions

The

partitions

will

be

defined

so

that

at

least

one

data

race

in

each

partition

is

guaranteed

to

belong

to

an

SCP

We

use

the

strongly

connected

of

G

to

partition

the

data

races

two

data

races

belong

to

the

same

partition

if

and

only

if

the

events

involved

in

the

races

belong

to

the

same

strongly

connected

component

The

first

partitions

are

then

identified

by

defining

a

partial

ordering

P

on

the

partitions

as

follows

Definition

Part

P

Part

iff

a

path

exists

in

G

from

some

event

in

Part

to

some

event

in

Part

hhhhhhhhhhhhhhhhhh

A

strongly

connected

component

of

a

directed

graph

has

the

property

that

paths

exist

between

each

pair

of

nodes

in

the

component

but

no

paths

exist

from

a

node

in

one

component

to

a

node

in

another

component

and

back

A

partition

is

first

if

it

is

not

ordered

by

P

after

any

other

partitions

containing

at

least

one

data

race

For

the

execution

of

Figure

the

partitions

and

their

ordering

are

shown

in

Figure

We

now

state

the

following

theorems

regarding

the

first

partitions

The

proofs

of

these

theorems

are

relatively

straightforward

and

appear

in

-LSB-

-RSB-

Theorem

There

are

no

first

partitions

containing

data

races

iff

no

data

races

were

exhibited

in

the

execution

Theorem

In

each

first

partition

containing

data

races

at

least

one

data

race

belongs

to

an

SCP

We

now

claim

that

for

an

execution

on

a

weak

system

only

the

first

partitions

containing

data

races

should

be

reported

to

the

programmer

This

is

sufficient

because

of

the

following

reason

If

for

some

execution

no

first

partitions

are

reported

then

by

Theorem

no

data

races

must

have

occurred

in

the

execution

and

by

Condition

-LRB-

-RRB-

this

execution

must

be

sequentially

consistent

If

first

partitions

are

reported

then

by

Theorem

at

least

one

of

the

data

races

in

each

of

these

partitions

would

occur

on

a

sequentially

consistent

execution

of

the

program

A

limitation

of

our

method

is

that

for

first

partitions

with

more

than

one

data

race

we

can

not

detect

precisely

which

race

is

sequentially

consistent

Nevertheless

the

portion

of

the

execution

that

has

to

be

examined

by

the

programmer

to

detect

the

sequentially

consistent

data

race

is

significantly

narrowed

making

debugging

much

easier

In

the

next

section

we

discuss

how

dynamic

race

detection

techniques

in

sequentially

consistent

systems

suffer

from

an

analogous

limitation

Data

Race

Detection

on

Sequentially

Consistent

Systems

vs.

Weak

Systems

The

limitations

of

our

approach

for

detecting

data

races

on

weak

systems

are

analogous

to

those

for

detecting

data

races

on

sequentially

consistent

systems

We

next

discuss

these

analogies

focusing

on

accuracy

and

overhead

The

effectiveness

of

data

race

detection

depends

on

the

accuracy

with

which

the

reported

data

races

reflect

program

bugs

Our

method

reports

partitions

of

data

races

where

each

partition

contains

at

least

one

sequentially

consistent

or

SCP

data

race

i.e.

a

true

program

bug

However

the

method

suffers

from

the

following

limitations

-LRB-

-RRB-

we

can

not

always

pinpoint

exactly

which

data

races

in

each

first

partition

belong

to

the

SCP

and

hence

are

the

valid

races

and

-LRB-

-RRB-

we

do

not

locate

data

races

in

non-first

partitions

that

may

also

belong

to

the

SCP

Dynamic

techniques

for

sequentially

consistent

systems

also

suffer

from

limitations

analogous

to

the

above

There

is

an

analogy

between

data

races

in

weak

systems

that

are

not

in

an

SCP

and

data

races

in

sequentially

consistent

systems

that

are

artifacts

of

other

data

races

A

data

race

is

an

artifact

if

it

occurs

only

because

a

previous

data

race

left

the

program

's

data

in

an

inconsistent

state

unexpected

by

the

programmer

and

hence

is

not

a

direct

manifestation

of

a

program

bug

Methods

for

accurately

locating

data

races

on

sequentially

consistent

systems

-LSB-

-RSB-

also

order

partitions

of

data

races

to

enable

detection

of

the

non-artifact

races

These

methods

also

suffer

from

the

above

limitations

-LRB-

-RRB-

they

identify

partitions

of

data

races

each

of

which

is

guaranteed

to

contain

at

least

one

non-artifact

data

race

but

can

not

determine

precisely

which

races

within

each

partition

are

not

artifacts

-LSB-

-RSB-

and

-LRB-

-RRB-

they

can

not

identify

non-artifact

data

races

that

happen

to

belong

to

non-first

partitions

When

using

the

first

partitions

we

expect

that

the

difference

between

debugging

on

weak

or

sequentially

consistent

systems

will

be

insignificant

we

believe

reasoning

about

an

artifact

data

race

on

a

sequentially

consistent

system

is

as

difficult

as

reasoning

about

non-SCP

data

races

on

weak

systems

Nevertheless

for

both

the

sequentially

consistent

and

weak

systems

identifying

the

first

partitions

significantly

narrows

the

portion

of

the

execution

that

must

be

investigated

to

locate

the

non-artifact

or

the

SCP

data

races

Another

issue

impacting

accuracy

is

the

reliability

of

the

trace

data

Again

sequentially

consistent

systems

and

weak

systems

share

common

limitations

Program

instrumentation

can

be

added

by

a

trusted

facility

-LRB-

such

as

a

compiler

-RRB-

and

so

can

generally

be

assumed

correct

However

pathological

programs

can

be

constructed

that

behave

unpredictably

after

a

data

race

-LRB-

because

the

program

's

data

is

left

in

an

inconsistent

state

-RRB-

and

randomly

overwrite

the

program

's

own

address

space

In

the

worst

case

the

traces

can

be

overwritten

with

erroneous

data

suggesting

that

no

data

races

were

exhibited

when

in

fact

data

races

occurred

Such

pathological

cases

can

be

constructed

for

both

weak

and

sequentially

consistent

systems

the

reliability

problem

is

no

worse

in

weak

systems

In

practice

however

we

expect

trace

corruption

to

rarely

occur

Another

important

factor

influencing

the

effectiveness

of

data

race

detection

is

overhead

Section

presented

a

post-mortem

approach

in

which

trace

files

are

written

during

execution

and

analyzed

during

a

post-mortem

phase

Overhead

is

incurred

both

during

execution

and

during

post-mortem

analysis

However

the

overhead

of

our

method

is

no

worse

than

post-mortem

methods

on

sequentially

consistent

systems

we

require

no

more

execution-time

information

that

these

methods

and

our

analysis

requires

computation

similar

to

the

more

accurate

techniques

for

sequentially

consistent

systems

-LSB-

-RSB-

Another

approach

for

locating

data

races

is

on-the-fly

detection

On-the-fly

approaches

have

the

advantage

of

not

consuming

secondary

storage

-LSB-

-RSB-

but

existing

methods

are

typically

less

accurate

and

have

higher

run-time

overhead

than

post-mortem

techniques

The

loss

of

accuracy

is

a

result

of

attempts

to

keep

space

overhead

low

by

only

buffering

limited

trace

information

in

memory

As

a

result

some

of

the

first

data

races

can

remain

undetected

This

is

a

problem

for

both

sequentially

consistent

systems

-LRB-

where

non-first

data

races

can

be

artifacts

-RRB-

and

weak

systems

-LRB-

where

non-first

data

races

may

not

belong

to

an

SCP

-RRB-

Future

work

includes

investigating

how

our

method

might

be

employed

on-the-fly

to

locate

the

first

data

races

We

believe

that

once

data

races

are

detected

using

our

methods

other

debugging

tools

for

sequentially

consistent

systems

can

be

effectively

applied

on

weak

systems

as

well

This

is

because

the

part

of

the

execution

that

contains

the

first

bugs

is

sequentially

consistent

and

can

be

debugged

as

on

a

sequentially

consistent

execution

Since

the

overhead

and

accuracy

of

our

data

race

detection

method

on

weak

systems

are

similar

to

that

on

sequentially

consistent

systems

we

expect

that

debugging

in

general

on

weak

systems

will

also

be

no

more

difficult

Programs

for

weak

systems

can

therefore

be

debugged

while

taking

advantage

of

the

performance

gains

achieved

by

these

systems

a

slower

sequentially

consistent

mode

for

debugging

is

not

necessary

Conclusions

The

shared

memory

models

of

weak

ordering

release

consistency

with

sequentially

consistent

synchronization

operations

and

provide

high

performance

by

guaranteeing

sequential

consistency

mainly

to

programs

that

do

not

exhibit

data

races

on

sequentially

consistent

hardware

To

allow

programmers

to

use

the

intuition

and

algorithms

already

developed

for

sequentially

consistent

systems

it

is

important

to

determine

when

a

program

is

data-race-free

and

when

it

is

not

to

identify

the

parts

where

data

races

could

occur

Detecting

data

races

is

also

crucial

for

programs

written

for

sequentially

consistent

systems

Static

techniques

for

sequentially

consistent

systems

can

be

directly

applied

to

weak

systems

as

well

Dynamic

techniques

on

the

other

hand

may

report

data

races

that

could

never

occur

on

sequentially

consistent

systems

This

can

complicate

debugging

because

programmers

can

no

longer

assume

the

model

of

sequential

consistency

We

have

shown

how

a

post-mortem

dynamic

approach

can

be

used

to

detect

data

races

effectively

even

on

weak

systems

The

key

observation

we

make

is

that

in

practice

weak

systems

preserve

sequential

consistency

at

least

until

the

first

data

races

-LRB-

those

not

affected

by

any

oth

ers

-RRB-

since

they

can

not

predict

if

a

data

race

will

occur

We

formalize

this

condition

by

using

the

notion

of

a

sequentially

consistent

prefix

For

an

execution

on

a

system

that

obeys

this

condition

we

show

how

we

can

either

-LRB-

-RRB-

correctly

report

no

data

races

and

conclude

the

execution

to

be

sequentially

consistent

or

-LRB-

-RRB-

report

the

first

data

races

that

also

occur

on

a

sequentially

consistent

execution

-LRB-

within

the

limitation

discussed

in

Section

-RRB-

Since

our

condition

is

already

met

by

all

implementations

of

WO

and

RCsc

and

all

proposed

implementations

of

and

our

technique

can

exploit

the

full

performance

of

the

weak

systems

Further

we

have

shown

that

any

limitations

of

our

technique

are

also

shared

by

dynamic

techniques

on

sequentially

consistent

systems

Finally

we

believe

that

the

presence

of

the

sequentially

consistent

prefix

also

allows

other

debugging

tools

of

sequentially

consistent

systems

to

be

used

on

weak

systems

We

have

demonstrated

the

use

of

our

hardware

condition

with

a

post-mortem

dynamic

technique

Using

onthe-fly

techniques

for

weak

systems

has

problems

analogous

to

those

on

sequentially

consistent

systems

and

is

a

subject

of

future

work

Although

the

definitions

and

theorems

presented

in

this

paper

may

be

considered

complex

only

those

who

wish

to

verify

our

results

must

confront

this

complexity

Designers

of

memory

systems

can

simply

check

that

their

hardware

obeys

Condition

while

users

of

our

data

race

detection

technique

can

reason

about

their

program

as

if

the

hardware

were

sequentially

consistent

References

-LSB-

-RSB-

S.

V.

ADVE

and

M.

D.

HILL

Weak

Ordering

A

New

Definition

Proc

Annual

Intl.

Symp

on

Computer

Architecture

May

-LSB-

-RSB-

S.

V.

ADVE

M.

D.

HILL

B.

P.

MILLER

and

R.

H.

B.

NETZER

Detecting

Data

Races

on

Weak

Memory

Systems

Computer

Sciences

Technical

Report

To

be

Published

University

of

Wisconsin

Madison

-LSB-

-RSB-

S.

V.

ADVE

and

M.

D.

HILL

An

Approach

for

Specifying

Shared

Memory

Models

Computer

Sciences

Technical

Report

To

be

Published

University

of

Wisconsin

Madison

-LSB-

-RSB-

T.

R.

ALLEN

and

D.

A.

PADUA

Debugging

Fortran

on

a

Shared

Memory

Machine

Proc

Intl.

Conf

on

Parallel

Processing

August

-LSB-

-RSB-

V.

BALASUNDARAM

and

K.

KENNEDY

Compile-time

Detection

of

Race

Conditions

in

a

Parallel

Program

Intl.

Conf

on

Supercomputing

June

-LSB-

-RSB-

A.

J.

BERNSTEIN

Analysis

of

Programs

for

Parallel

Processing

IEEE

Trans

on

Electronic

Computers

-LRB-

October

-RRB-

-LSB-

-RSB-

J.

CHOI

and

S.

L.

MIN

Race

Frontier

Reproducing

Data

Races

in

Parallel

Program

Debugging

Proc

ACM

Symp

on

Principles

and

Practice

of

Parallel

Programming

April

-LSB-

-RSB-

A.

DINNING

and

E.

SCHONBERG

An

Empirical

Comparison

of

Monitoring

Algorithms

for

Access

Anomaly

Detection

Proc

ACM

SIGPLAN

Notices

Symp

on

Principles

and

Practice

of

Parallel

Programming

March

-LSB-

-RSB-

M.

DUBOIS

C.

SCHEURICH

and

F.

A.

BRIGGS

Memory

Access

Buffering

in

Multiprocessors

Proc

Annual

Intl.

Symp

on

Computer

Architecture

-LRB-

June

-RRB-

-LSB-

-RSB-

P.

A.

EMRATH

and

D.

A.

PADUA

Automatic

Detection

of

Nondeterminacy

in

Parallel

Programs

Proc

SIGOPS

Workshop

on

Parallel

and

Distributed

Debugging

May

Also

appears

in

SIGPLAN

Notices

-LRB-

-RRB-

-LRB-

January

-RRB-

-LSB-

-RSB-

K.

GHARACHORLOO

D.

LENOSKI

J.

LAUDON

P.

GIBBONS

A.

GUPTA

and

J.

HENNESSY

Memory

Consistency

and

Event

Ordering

in

Scalable

SharedMemory

Multiprocessors

Proc

Annual

Intl.

Symp

on

Computer

Architecture

May

-LSB-

-RSB-

J.

R.

GOODMAN

Cache

Consistency

and

Sequential

Consistency

Computer

Sciences

Technical

Report

Univ.

of

Wisconsin

Madison

February

-LSB-

-RSB-

R.

HOOD

K.

KENNEDY

and

J.

MELLOR-CRUMMEY

Parallel

Program

Debugging

with

On-the-fly

Anomaly

Detection

Supercomputing

November

-LSB-

-RSB-

L.

LAMPORT

Time

Clocks

and

the

Ordering

of

Events

in

a

Distributed

System

Communications

of

the

ACM

-LRB-

July

-RRB-

-LSB-

-RSB-

L.

LAMPORT

How

to

Make

a

Multiprocessor

Computer

That

Correctly

Executes

Multiprocess

Programs

IEEE

Trans

on

Computers

-LRB-

September

-RRB-

-LSB-

-RSB-

R.

H.

B.

NETZER

and

B.

P.

MILLER

Detecting

Data

Races

in

Parallel

Program

Executions

To

appear

in

Research

Monographs

in

Parallel

and

Distributed

Computing

MIT

Press

Also

available

as

Proc

Workshop

on

Programming

Languages

and

Compilers

for

Parallel

Computing

August

-LSB-

-RSB-

R.

H.

B.

NETZER

and

B.

P.

MILLER

Improving

the

Accuracy

of

Data

Race

Detection

Proc

ACM

Symp

on

Principles

and

Practice

of

Parallel

Programming

April

-LSB-

-RSB-

R.

N.

TAYLOR

A

General-Purpose

Algorithm

for

Analyzing

Concurrent

Programs

Communications

of

the

ACM

-LRB-

May

-RRB-

-LSB-

-RSB-

R.

N.

TAYLOR

Complexity

of

Analyzing

the

Synchronization

Structure

of

Concurrent

Programs

Acta

Informatica

-LRB-

-RRB-

O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

data races
release consistency
sequential consistency
weak ordering