Abstract

Networks

of

workstations

-LRB-

NOWs

-RRB-

are

gaining

popularity

as

lower-cost

alternatives

to

massively-parallel

processors

-LRB-

MPPs

-RRB-

because

of

their

ability

to

leverage

highperformance

commodity

workstations

and

data

networks

However

fast

data

networks

will

not

suffice

if

applications

require

frequent

global

synchronization

e.g.

barriers

reductions

and

broadcasts

Many

MPPs

provide

hardware

support

specifically

to

accelerate

these

operations

Separate

synchronization

networks

have

also

been

proposed

for

NOWs

but

such

add-on

hardware

only

makes

sense

if

the

performance

improvement

is

commensurate

with

its

cost

In

this

study

we

examine

the

cost/performance

tradeoff

of

add-on

synchronization

hardware

for

an

emulated

NOW

running

an

aggregate

workload

of

nine

shared-memory

message-passing

and

data-parallel

applications

For

low-latency

messaging

-LRB-

e.g.

ms

-RRB-

add-on

hardware

is

cost-effective

only

if

its

per-node

cost

is

less

than

of

the

base

workstation

cost

For

higherlatency

messages

-LRB-

e.g.

ms

-RRB-

add-on

hardware

is

costeffective

if

it

costs

less

than

of

the

base

cost

Of

course

individual

applications

behave

differently

four

of

the

nine

applications

show

no

benefit

from

the

extra

hardware

while

one

application

improves

by

over

a

factor

of

three

with

higher-latency

messages

This

work

is

supported

in

part

by

Wright

Laboratory

Avionics

Directorate

Air

Force

Material

Command

USAF

under

grant

and

ARPA

order

no

NSF

PYI

Award

NSF

Grant

and

donations

from

A.T.

T.

Bell

Laboratories

Digital

Equipment

Corporation

Sun

Microsystems

Thinking

Machines

Corporation

and

Xerox

Corporation

Our

Thinking

Machines

was

purchased

through

NSF

Institutional

Infrastructure

Grant

No

CDA

with

matching

funding

from

the

University

of

Wisconsin

Graduate

School

The

U.S.

Government

is

authorized

to

reproduce

and

distribute

reprints

for

Governmental

purposes

notwithstanding

any

copyright

notation

thereon

The

views

and

conclusions

contained

herein

are

those

of

the

authors

and

should

not

be

interpreted

as

necessarily

representing

the

official

policies

or

endorsements

either

expressed

or

implied

of

the

Wright

Laboratory

Avionics

Directorate

or

the

U.S.

Government

Introduction

Networks

of

workstations

-LRB-

NOWs

-RRB-

are

gaining

popularity

as

lower-cost

alternatives

to

the

current

generation

of

massively-parallel

processors

-LRB-

MPPs

-RRB-

NOWs

exploit

commodity

systems?entire

workstations?rather

than

individual

components

In

addition

NOWs

leverage

commodity

network

technology

to

further

reduce

engineering

cost

While

current

local-area

network

performance

is

poor

by

MPP

standards

-LRB-

latencies

in

the

to

of

microseconds

and

bandwidths

in

the

to

of

Mbits

second

-RRB-

emerging

networks

promise

better

performance

For

example

repackaged

multicomputer

interconnects

such

as

Myrinet

-LSB-

-RSB-

and

Shrimp

-LSB-

-RSB-

may

yield

up

to

two

orders-of-magnitude

performance

improvement

over

previous-generation

local-area

networks

However

high-performance

CPUs

and

data

networks

will

not

be

sufficient

for

NOWs

to

achieve

good

speedups

for

all

existing

parallel

applications

Some

applications

require

frequent

synchronization

to

coordinate

computation

-LRB-

e.g.

barriers

-RRB-

to

compute

global

results

-LRB-

e.g.

reductions

-RRB-

or

update

common

data

structures

-LRB-

e.g.

broadcasts

-RRB-

To

address

the

requirements

of

these

applications

many

MPPs?the

Cray

-LSB-

-RSB-

the

Fujitsu

-LSB-

-RSB-

and

the

Thinking

Machines

-LSB-

-RSB-

provide

explicit

hardware

support

for

global

synchronization

NOWs

can

also

employ

synchronization

hardware

in

the

form

of

a

separate

add-on

synchronization

network

For

example

Dietz

et

al.

have

developed

barrier

hardware

which

connects

to

the

standard

Centronics

parallel

port

of

an

IBM-compatible

PC

-LSB-

-RSB-

Hall

and

Driscoll

have

proposed

a

synchronization

network

for

Sun

workstation

clusters

that

suppors

barriers

reductions

and

broadcasts

-LSB-

-RSB-

Shang

and

Hwang

have

proposed

barrier

hardware

for

cluster-based

multiprocessors

including

workstation

clusters

-LSB-

-RSB-

However

whether

or

not

such

add-on

synchronization

hardware

is

cost-effective

depends

upon

its

cost

the

cost

of

the

base

NOW

and

the

performance

improvement

which

the

synchronization

hardware

provides

For

example

if

the

base

NOW

costs

per

node

-LRB-

including

the

network

-RRB-

and

the

add-on

synchronization

hardware

Synchronization

Hardware

for

Networks

of

Workstations

Performance

vs.

Cost

Rahmat

S.

Hyder

and

David

A.

Wood

Computer

Sciences

Department

University

of

Wisconsin?Madison

West

Dayton

Street

Madison

WI

wwt@cs.wisc.edu

costs

per

node

then

performance

must

improve

by

at

least

for

the

synchronization

hardware

to

be

costeffective

-LSB-

-RSB-

While

individual

applications

may

improve

this

much

or

more

the

add-on

hardware

is

only

costeffective

if

the

aggregate

performance

of

the

NOW?s

entire

workload

improves

by

In

this

paper

we

examine

the

cost/performance

tradeoff

of

add-on

synchronization

hardware

for

a

NOW

We

focus

specifically

on

global

synchronization

-LRB-

e.g.

barriers

-RRB-

not

pairwise

synchronization

-LRB-

e.g.

locks

-RRB-

We

study

a

range

of

benchmarks

to

understand

which

existing

applications

will

benefit?and

by

how

much?from

explicit

hardware

synchronization

support

We

examine

the

synchronization

requirements

for

three

important

classes

of

applications

shared-memory

message-passing

and

dataparallel

We

consider

two

alternative

synchronization

networks

a

low-cost

version

that

supports

only

simple

barriers

and

single-bit

AND

operations

called

-LRB-

Hardware-One

-RRB-

and

a

higher-cost

version

which

additionally

supports

integer

reductions

and

broadcasts

called

HW-All

-LRB-

Hardware-All

-RRB-

For

these

applications

we

calculate

the

break-even

cost?the

price

at

which

synchronization

hardware

becomes

cost-effective

We

use

a

Thinking

Machines

to

model

a

NOW

both

with

and

without

hardware

synchronization

support

and

use

the

measured

performance

improvement

to

calculate

the

cost/performance

break-even

point

We

study

the

effects

of

two

different

network

latencies

a

fast

data

network

modeled

by

native

messages

-LRB-

ms

latency

-RRB-

and

a

slow

data

network

modeled

by

messages

delayed

by

ms

using

a

relay-node

technique

described

in

Section

For

our

emulated

NOW

we

find

the

following

results

For

our

aggregate

workload?a

weighted

average

of

nine

shared-memory

message-passing

and

data-parallel

applications?add

on

synchronization

hardware

for

a

NOW

with

a

fast

data

network

is

only

cost-effective

if

it

costs

less

than

of

the

base

system

cost

or

per

node

for

a

base

per-node

cost

of

For

a

NOW

with

a

slow

data

network

add-on

hardware

is

cost-effective

if

it

costs

less

than

of

the

base

system

cost

or

per

node

Individual

applications

may

benefit

much

more

from

synchronization

hardware

Among

our

applications

the

hardware

improves

performance

by

up

to

on

a

NOW

with

the

fast

data

network

and

up

to

a

factor

of

with

the

slow

data

network

The

HW-All

hardware

yields

further

improvement

increasing

performance

by

factors

of

up

to

and

for

the

two

network

latencies

Some

applications

can

be

restructured

to

avoid

using

global

synchronization

operations

decreasing

the

performance

benefit

However

for

the

one

code

we

examined

in

detail

water

-LSB-

-RSB-

the

barrier

version

is

faster

than

the

-LRB-

original

-RRB-

locking

version

even

without

hardware

barrier

support

Other

applications

can

not

be

restructured

as

easily

to

avoid

global

synchronization

such

as

the

Wisconsin

Wind

Tunnel

-LRB-

WWT

-RRB-

-LSB-

-RSB-

While

most

applications

synchronize

to

ensure

that

messages

have

been

delivered

WWT

frequently

synchronizes

to

ensure

that

no

messages

are

in

flight

For

WWT

the

network

improves

performance

up

to

on

a

NOW

with

the

fast

data

network

and

up

to

a

factor

of

with

the

slow

data

network

the

HW-All

network

improves

performance

by

up

to

and

a

factor

of

Our

results

show

that

global

synchronization

hardware

can

be

a

cost-effective

addition

to

a

NOW

for

some

workloads

The

benefit

is

relatively

greater

for

slower

data

networks

since

for

each

synchronization

operation

the

hardware

eliminates

O

-LRB-

log

N

-RRB-

messages

latencies

from

the

critical

path

On

the

other

hand

nearly

half

our

applications

received

no

benefit

from

synchronization

support

additional

hardware

is

not

justified

to

support

workloads

dominated

by

these

applications

A

key

advantage

of

addon

synchronization

hardware?as

opposed

to

the

integrated

synchronization

hardware

in

most

MPPs?is

that

only

those

people

that

can

benefit

from

it

need

to

buy

it

The

remainder

of

the

paper

is

organized

as

follows

Section

presents

the

implementation

and

performance

of

the

synchronization

operations

which

our

applications

require

Section

presents

our

benchmark

suite

and

the

benchmarks

synchronization

requirements

Section

presents

our

experimental

performance

results

Section

presents

our

model

for

determining

the

cost/performance

break-even

point

estimations

of

system

cost

and

the

cost

performance

break-even

points

of

our

applications

Section

discusses

related

work

and

Section

summarizes

our

results

and

conclusions

Implementation

of

synchronization

operations

The

applications

in

this

study

require

a

variety

of

global

synchronization

operations

of

differing

complexities

The

simplest

operations

include

simple

barriers

and

single-bit

AND

reductions

The

more

complex

operations

involve

many-to-one

ADD

reductions

which

deliver

the

result

to

a

single

node

ADD

and

MAX

reductions

which

deliver

the

result

to

all

nodes

and

broadcasts

In

this

section

we

discuss

the

implementation

and

performance

of

these

synchronization

operations

both

with

and

without

hardware

support

Section

discusses

software

implementations

Section

discusses

hardware

implementations

and

Section

examines

their

performance

We

defer

the

discussion

of

how

our

applications

use

these

operations

until

Section

Software

synchronization

Synchronization

operations

can

be

implemented

in

software

by

using

explicit

messages

For

example

N

processors

can

perform

a

barrier

synchronization

simply

by

designating

one

node

as

a

master

node

and

then

indicate

barrier

arrival

by

sending

messages

to

that

master

node

Once

the

master

receives

arrival

messages

from

all

processors

it

sends

wakeup

messages

to

all

processors

informing

each

that

all

have

reached

the

barrier

Intuitively

this

barrier

is

very

inefficient

because

of

the

contention

at

the

master

To

quantify

its

speed

let

L

be

the

message

latency

let

S

be

the

overhead

of

sending

a

message

and

let

R

be

the

overhead

of

receiving

a

message

Then

the

barrier

latency

is

-LRB-

N

-RRB-

R

-LRB-

N

-RRB-

S.

Assuming

sufficient

network

bandwidth

the

message

latencies

can

overlap

but

the

overheads

which

the

master

incurs

must

serialize

The

overheads

can

not

be

eliminated

but

their

impact

can

be

lessened

by

distributing

them

among

the

processors

Tournament

barriers

-LSB-

-RSB-

distribute

these

overheads

by

having

processors

perform

the

arrival

phase

in

pairs

-LRB-

combining

-RRB-

forming

a

tree

With

sufficient

network

bandwidth

the

wakeup

phase

can

also

be

performed

in

a

tree

fashion

-LSB-

-RSB-

Since

for

every

message

sent

there

is

exactly

one

send

and

one

receive

overhead

latency

incurred

the

latency

of

this

barrier

is

then

O

-LRB-

N

-LRB-

L

S

R

-RRB-

-RRB-

where

the

factor

of

comes

from

the

two

separate

trees

However

in

the

presence

of

significant

messaging

overhead

combining

does

not

hide

enough

overhead

higher-radix

combining

is

needed

Consider

Figure

which

depicts

and

tournament

barriers

for

processors

with

significant

send

and

receive

overhead

The

solution

hides

much

more

of

the

messaging

overhead

resulting

in

lower

latency

O

-LRB-

-LRB-

L

S

R

-RRB-

-RRB-

Butterfly

barriers

-LSB-

-RSB-

eliminate

the

factor

of

by

effectively

performing

multiple

tournament-arrival

binary

trees

in

parallel

with

each

processor

at

the

root

of

a

different

arrival

tree

Figure

illustrates

the

messaging

pattern

for

and

butterfly

barriers

for

nodes

Once

again

the

barrier

allows

more

send/receive

overhead

to

be

hidden

than

in

the

barrier

The

butterfly

barrier

has

the

potential

for

better

latency

than

the

tournament

barrier

but

it

sends

O

-LRB-

N

logkN

-RRB-

messages

for

a

fixed

radix

k

while

the

tournament

barrier

sends

only

O

-LRB-

logkN

-RRB-

messages

The

butterfly

barrier

will

then

outperform

the

tournament

barrier

only

if

there

is

sufficient

network

bandwidth

Our

emulated

NOW

satisfies

this

assumption

so

we

will

concentrate

on

only

the

butterfly

barrier

in

the

remainder

of

this

paper

Reductions

and

broadcasts

can

also

be

implemented

via

data

messages

as

follows

Reductions

which

require

delivering

the

result

to

all

nodes

are

essentially

simple

barriers

which

also

send

data

and

hence

we

implement

them

with

a

butterfly-style

combining

pattern

Reductions

which

deliver

the

result

to

a

single

node

and

broadcasts

are

implemented

via

unbalanced

trees

-LSB-

-RSB-

in

which

the

fanout

of

nodes

is

set

according

to

the

latency

and

overhead

of

messages

Hardware

synchronization

Add-on

synchronization

hardware

for

a

NOW

consists

of

two

separate

components

the

workstation

interface

and

the

synchronization

network

itself

Modern

workstations

provide

a

range

of

possible

interfaces

each

with

different

latency

bandwidth

and

cost

considerations

At

the

lowend

most

workstations

provide

a

parallel

port

that

can

be

used

for

low-bandwidth

operations

such

as

simple

barriers

Parallel

ports

typically

require

system

calls

for

userlevel

access

but

in

some

systems

can

be

memory-mapped

directly

into

user

space

-LSB-

-RSB-

Higher

performance

at

higher

cost

can

be

obtained

by

interfacing

to

the

workstation?s

I

time

FIGURE

Tournament

barriers

of

-LRB-

left

-RRB-

and

-LRB-

right

-RRB-

for

nodes

The

I-bars

represent

send/receive

overhead

O

bus

This

is

a

better

choice

for

more

complex

operations

such

as

broadcasts

that

require

lower

latency

and/or

higher

bandwidth

At

the

high-end

the

synchronization

hardware

can

interface

directly

to

the

memory

bus

in

some

workstations

however

the

increase

in

performance

is

unlikely

to

outweigh

the

increase

in

cost

Regardless

of

which

interface

location

is

chosen

the

latencies

should

be

relatively

low

from

a

few

tens

of

cycles

to

a

few

hundreds

of

cycles

The

actual

combining

operation

in

the

switch

should

be

even

faster

For

example

since

a

barrier

operation

is

simply

a

logical

AND

it

could

be

implemented

as

a

large

AND

tree

with

delays

measured

in

nanoseconds

More

complex

operations

such

as

integer

reductions

require

more

complex

logic

but

the

basic

combining

operation

will

be

at

most

a

few

processor

cycles

Control

logic

e.g.

for

partitioning

or

virtualizing

the

network

will

introduce

additional

delays

Nonetheless

overheads

will

be

minimal

within

the

synchronization

network

We

consider

two

possible

hardware

synchronization

networks

each

with

different

complexity

and

cost

The

first

is

lower-complexity

and

lower-cost

supporting

simple

barriers

and

single-bit

AND

reductions

we

call

this

network

-LRB-

Hardware-One

-RRB-

We

include

the

singlebit

AND

because

some

applications

make

heavy

use

of

this

reduction

operation

and

it

requires

hardware

comparable

in

complexity

to

the

simple

barrier

The

second

is

higher-complexity

and

higher-cost

supporting

the

features

plus

the

reductions

described

earlier

and

broadcasts

we

call

this

network

HW-All

-LRB-

Hardware-All

-RRB-

We

assume

that

the

networks

connect

to

the

workstation?s

memory

bus

which

provides

a

low-latency

high-bandwidth

interface

The

importance

of

hardware

synchronization

is

magnified

by

longer

network

latencies

because

software

synchronization

requires

many

-LRB-

slower

-RRB-

messages

To

illustrate

this

point

we

express

parallel-program

runtime

as

the

sum

of

computation

time

-LRB-

Twork

-RRB-

communication

time

-LRB-

Tcomm

-RRB-

and

synchronization

time

-LRB-

Tglobal

-RRB-

In

the

absence

of

synchronization

hardware

Tcomm

and

Tglobal

increase

linearly

with

message

latency

while

Twork

remains

constant

As

a

result

synchronization

time

becomes

a

larger

portion

of

the

total

runtime

Conversely

with

synchronization

hardware

Tglobal

remains

constant

and

becomes

a

smaller

portion

of

the

total

runtime

We

expect

synchronization

hardware

to

be

more

important

for

NOWs

with

long

message

latencies

especially

for

finegrain

codes

which

perform

little

work

and

send

few

messages

between

synchronization

operations

Synchronization

performance

We

now

measure

the

performance

of

hardware

and

software

synchronization

operations

on

the

emulated

NOW

We

use

a

Thinking

Machines

to

emulate

node

NOWs

with

and

without

hardware

synchronization

support

To

approximate

the

higher-latency

messages

of

a

NOW

we

divide

a

partition

into

compute

nodes

and

relay

nodes

using

the

relay

nodes

to

slow

down

messages

as

follows

a

message

destined

for

node

N

is

first

sent

to

node

N

a

relay

node?where

it

is

delayed

for

the

appropriate

latency

and

then

sent

to

node

N.

Figure

illustrates

this

process

for

an

emulated

NOW

with

relay

nodes

We

do

not

use

the

compute

nodes

as

relay

nodes

because

doing

so

would

perturb

the

computation

on

those

nodes

For

hardware

synchronization

the

conveniently

provides

hardware

support

for

all

the

synchronization

operations

our

benchmarks

need

In

addition

to

the

basic

NOW

some

of

our

applications

require

the

existence

of

a

separate

host

node

which

connects

to

the

processing

nodes

The

broadcasts

T

Twork

Tcomm

Tg

l

oba

l

FIGURE

Butterfly

barriers

of

-LRB-

left

-RRB-

and

-LRB-

right

-RRB-

for

nodes

The

I-bars

represent

send

receive

overhead

time

and

many-to-one

reductions

are

used

in

this

extended

NOW

the

host

broadcasts

data

to

all

nodes

and

receives

reduction

results

from

all

nodes

The

directly

implements

this

model

with

a

separate

workstation

serving

as

the

host

Table

presents

the

latency

in

microseconds

of

each

type

of

synchronization

operation

These

measurements

were

taken

in

the

context

of

the

software

runtime

environments

for

our

benchmarks

which

are

discussed

in

Section

We

consider

two

network

latencies

base

message

latency

-LRB-

approximately

ms

-RRB-

or

zero-delayed

messages

and

messages

delayed

by

ms

or

ms-delayed

messages

All

software

operations

were

optimized

with

respect

to

messaging

latency

and

overhead

for

example

the

simple

barriers

for

ms

messages

yield

the

lowest

latency

when

the

butterfly

pattern

uses

two

phases

combining

the

nodes

in

groups

of

eight

nodes

in

the

first

phase

and

then

in

groups

of

four

nodes

in

the

second

phase

The

individual

hardware

operations

are

roughly

comparable

in

speed

This

result

is

expected

because

the

bulk

of

the

latency

is

spent

in

the

network-interface

hardware

rather

than

in

the

actual

combining

hardware

The

software

operations

range

from

one

to

two

orders

of

magnitude

slower

than

the

hardware

implementation

depending

on

the

message

latency

Synchronization

requirements

of

benchmarks

We

consider

three

classes

of

applications

sharedmemory

applications

-LRB-

SM

-RRB-

message-passing

applications

-LRB-

MP

-RRB-

and

data-parallel

applications

-LRB-

DP

-RRB-

Table

lists

the

benchmarks

we

study

in

each

of

the

three

classes

along

FIGURE

NOW

with

long-latency

messages

emulated

on

an

host

machine

Node

sends

a

message

to

node

by

sending

the

message

to

node

s

relay

which

is

node

The

message

is

delayed

at

node

and

then

sent

to

node

Compute

nodes

Relay

nodes

Software

Message

delay

Synchronization

type

Hardware

ms

Simple

barrier

ms

ms

ms

AND

ms

ms

ms

ADD

ms

ms

ms

ADD

-LRB-

to

one

node

-RRB-

ms

ms

ms

MAX

ms

ms

ms

Broadcast

ms

ms

ms

TABLE

Latency

of

hardware

and

software

synchronization

operations

Message

delay

is

the

amount

by

which

messages

-LRB-

typically

ms

-RRB-

are

delayed

ADD

reductions

to

a

single

node

and

broadcasts

involve

data

transmission

between

a

host

workstation

and

a

NOW

all

other

operations

involve

only

a

NOW

Note

that

ADD

reductions

to

a

single

node

are

significantly

slower

than

broadcasts

because

sending

a

message

from

a

node

to

the

host

on

the

involves

a

system

call

However

these

reductions

are

very

infrequent

in

our

benchmarks

so

this

is

not

a

concern

with

their

input

data

sets

and

frequencies

of

synchronization

on

the

Shared-memory

benchmarks

The

shared-memory

benchmarks

are

barnes

DSMC

moldyn

and

water

Barnes

uses

the

Barnes-Hut

algorithm

to

calculate

N-body

interactions

-LSB-

-RSB-

To

obtain

better

speedup

from

barnes

we

used

a

modified

version

which

allocates

free

cells

in

a

processor

s

local

portion

of

shared

memory

obviating

the

original

global

free

cell

Barnes

invokes

barriers

after

creation

and

traveral

of

octrees

and

after

updates

are

made

to

global

values

DSMC

is

a

rarefied

gas

simulation

computing

interactions

between

molecules

in

a

box

-LSB-

-RSB-

DSMC

invokes

barriers

after

new

molecules

enter

the

box

after

simulating

the

collision

of

molecules

and

after

moving

molecules

into

new

space

cells

Moldyn

calculates

the

motion

of

atomic

particles

based

on

forces

acting

on

each

particle

from

particles

within

a

certain

radius

-LSB-

-RSB-

Barriers

are

invoked

after

calculating

the

forces

on

and

velocities

of

molecules

and

after

updating

the

coordinates

of

molecules

Water

is

a

molecular

dynamics

simulation

computing

the

interactions

among

water

molecules

-LSB-

-RSB-

For

water

we

use

a

modified

version

which

is

restructured

to

use

barriers

instead

of

locks

this

version

provides

better

performance

This

modification

is

similar

to

the

newly-released

version

-LSB-

-RSB-

on

our

platform

Barriers

are

invoked

after

updates

to

molecules

All

of

these

benchmarks

were

parallelized

by

hand

using

a

locally-modified

version

of

the

PARMACS

macro

package

and

were

run

on

Blizzard

a

software

system

that

provides

fine-grain

distributed

shared

memory

on

the

Thinking

Machines

-LSB-

-RSB-

Blizzard

uses

the

userlevel

Stache

protocol?a

COMA-like

invalidation

protocol?to

maintain

sequentially

consistent

shared

memory

-LSB-

-RSB-

Barnes

and

water

use

Stache

to

maintain

coherence

on

blocks

and

DSMC

and

moldyn

maintain

coherence

on

blocks

Message-passing

benchmarks

The

message-passing

benchmarks

are

applu

dycore

and

the

Wisconsin

Wind

Tunnel

-LRB-

WWT

-RRB-

Applu

is

a

computational

fluid

dynamics

code

which

solves

five

coupled

parabolic/elliptic

partial

differential

equations

-LSB-

-RSB-

The

computation

consists

of

successive

over-relaxation

iterations

with

a

barrier

after

each

iteration

Dycore

computes

the

equations

of

motion

for

a

grid-point

atmospheric

global

climate

model

-LSB-

-RSB-

Barriers

are

invoked

between

phases

of

computation

and

near-neighbor

communication

We

ran

applu

and

dycore

on

Blizzard

using

Blizzard-provided

mechanisms

to

implement

message-passing

functions

for

these

applications

Unlike

the

above

applications

WWT

is

a

native

program

WWT

is

a

parallel

discrete-event

simulator

which

simulates

cache-coherent

distributed-shared-memory

multiprocessors

-LSB-

-RSB-

WWT

has

much

more

heavy

and

diverse

Synchronization

frequency

-LRB-

events/sec

-RRB-

Benchmark

Class

Input

Barriers

Integer

reductions

Broadcasts

barnes

SM

mols

dsmc

SM

mols

iter

moldyn

SM

mols

iter

water

SM

mols

reg

lattice

applu

MP

iter

dycore

MP

grid

iter

Appbt

-LRB-

iter

-RRB-

WWT

MP

Tomcatv

-LRB-

-RRB-

Sparse

-LRB-

dense

-RRB-

Water

-LRB-

mols

iter

-RRB-

metspin

DP

grid

iter

467 101 2565

nbody

DP

bodies

958 839 10009

TABLE

Benchmark

suite

Synchronization

frequencies

are

as

measured

on

the

i.e.

when

run

with

zerodelayed

messages

and

hardware

support

for

all

synchronization

operations

synchronization

requirements

up

to

three

types

of

synchronization

may

occur

after

every

phase

First

after

every

quantum

-LRB-

cycles

-RRB-

of

target-program

execution

WWT

must

ensure

that

all

messages

sent

during

the

current

quantum

have

been

received

prior

to

the

start

of

the

next

quantum

This

Network-Done

operation

involves

halting

further

network

activity

until

all

messages

currently

in-flight

have

been

delivered

We

can

perform

Network-Done

by

either

-LRB-

i

-RRB-

waiting

until

the

number

of

messages

received

by

all

nodes

equals

the

number

sent

by

all

nodes

or

-LRB-

ii

-RRB-

sending

an

explicit

acknowledgement

-LRB-

ACK

-RRB-

for

each

message

and

waiting

at

the

end

of

the

quantum

for

all

ACKs

to

be

collected

Solution

-LRB-

i

-RRB-

requires

repeated

ADD

while

solution

-LRB-

ii

-RRB-

requires

a

simple

barrier

Second

after

every

quantum

WWT

must

determine

if

all

nodes

encountered

a

barrier

in

the

target

program

in

the

last

quantum

This

can

be

accomplished

with

a

single-bit

AND

reduction

In

fact

if

option

-LRB-

ii

-RRB-

above

is

used

the

single-bit

AND

reduction

can

double

as

the

barrier

Third

if

all

nodes

indicate

that

they

have

arrived

at

a

target-program

barrier

each

target

node

must

set

its

local

clock

to

the

maximum

time

of

all

target

nodes

local

clocks

where

the

clocks

are

values

this

requirement

necessitates

MAX

reductions

Data-parallel

benchmarks

The

data-parallel

benchmarks

are

metspin

and

nbody

written

in

CM-Fortran

and

linked

with

a

modified

version

of

the

CM-Fortran

communication

library

CM-Fortran

employs

a

host-node

model

of

computation

where

a

frontend

host

machine

coordinates

computation

on

a

set

of

parallel

nodes

by

broadcasting

parallel

functions

and

data

to

the

nodes

and

by

receiving

results

from

the

nodes

In

studying

these

applications

we

assume

that

the

The

network

interface

provides

hardware

support

to

automatically

repeat

the

ADD

reduction

However

WWT

does

not

employ

this

feature

NOW

is

connected

to

an

additional

workstation

which

functions

as

the

host

Metspin

uses

the

Metropolis

Monte

Carlo

algorithm

to

simulate

an

Ising

spin

model

of

a

ferromagnet

and

calculate

the

energy

and

magnetization

at

a

particular

temperature

-LSB-

-RSB-

The

basic

computation

is

successive

overrelaxation

with

red-black

iterations

performed

on

a

grid

Nbody

calculates

the

force

between

N

bodies

interacting

via

long-range

forces

-LSB-

-RSB-

Both

metspin

and

nbody

use

barriers

after

cyclic

shifts

of

arrays

broadcasts

to

distribute

both

code

pointers

and

data

to

nodes

and

a

Network-Done

operation

to

determine

the

completion

of

cyclic

shifts

of

arrays

Metspin

also

invokes

barriers

after

phases

of

near-neighbor

computation

and

performs

manyto-one

ADD

reductions

to

the

host

to

track

the

sum

of

all

cells

in

the

grid

whose

values

have

stabilized

Performance

results

This

section

presents

the

performance

for

our

sharedmemory

benchmarks

-LRB-

Section

-RRB-

our

message-passing

benchmarks

-LRB-

Section

-RRB-

and

our

data-parallel

benchmarks

-LRB-

Section

-RRB-

as

we

vary

the

synchronization

methods

and

the

network

latency

Shared-memory

benchmarks

We

ran

each

of

our

shared-memory

benchmarks

on

the

emulated

NOW

with

zero-delayed

and

msdelayed

messages

comparing

a

system

with

the

synchronization

network

against

a

system

with

no

synchronization

hardware

-LRB-

these

benchmarks

do

not

need

the

added

features

of

HW-All

-RRB-

Table

presents

the

resultant

speedups

execution

time

on

a

single

node

divided

by

the

execution

time

on

nodes

Three

of

the

four

benchmarks?barnes

DSMC

and

moldyn?perform

little

global

thus

using

the

network

does

not

significantly

improve

Barnes

makes

extensive

use

of

locks

but

not

barriers

Message

delay

ms

Benchmark

No

HW

No

HW

barnes

DSMC

moldyn

water

TABLE

Speedups

for

shared-memory

benchmarks

No

HW

indicates

synchronization

was

done

in

software

Barnes?s

insignificant

slowdown

with

fast

messages

results

from

random

system

perturbations

performance

regardless

of

the

message

latency

On

the

other

hand

water

synchronizes

much

more

frequently

and

thus

hardware

barrier

support

improves

performance

substantially

Performance

improves

with

fast

messages

and

with

ms-delayed

messages

In

this

study

we

used

a

restructured

version

of

water

-LSB-

-RSB-

that

uses

barriers

instead

of

locks

as

the

primary

synchronization

operation

Surprisingly

this

version

runs

consistently

faster

than

the

-LRB-

original

-RRB-

locking

version

even

without

hardware

barrier

support

Specifically

the

barrier

version

is

faster

than

the

locking

version

with

software

barriers

and

fast

messages

and

faster

with

software

barriers

and

slow

messages

With

hardware

support

the

barrier

version

is

faster

than

the

locking

version

with

zero-delayed

messages

and

times

faster

than

the

locking

version

with

ms-delayed

messages

This

result

illustrates

that

restructuring

applications

to

avoid

global

synchronization

may

not

always

be

easy

Message-passing

benchmarks

Table

presents

the

speedups

for

the

first

two

message-passing

benchmarks

applu

and

dycore

comparing

a

system

with

the

synchronization

network

against

a

system

with

no

synchronization

hardware

-LRB-

These

two

benchmarks

also

do

not

require

the

added

features

of

HW

All

-RRB-

For

applu

its

low

synchronization

frequency

implies

minimal

potential

for

improvement

regardless

of

the

message

latency

For

dycore

with

its

higher

degree

of

synchronization

hardware

synchronization

support

yields

a

improvement

with

zero-delayed

messages

and

a

improvement

with

ms-delayed

messages

In

contrast

to

the

above

two

applications

WWT

can

utilize

either

the

or

HW-All

synchronization

networks

We

compare

the

performance

improvement

on

systems

using

these

networks

to

a

system

without

synchronization

hardware

In

the

system

without

synchronization

hardware

the

Network-Done

operation

is

performed

using

and

the

quantum

barrier

and

maximum-target-barrier-time

operations

are

performed

using

software

reductions

In

the

system

with

the

We

found

that

performing

Network-Done

with

ACKs

was

faster

than

performing

it

with

a

software

reduction

network

the

quantum

barrier

uses

the

hardware

AND

reduction

Network-Done

is

performed

using

ACKs

and

the

maximum-target-barrier-time

operation

is

performed

with

a

software

reduction

Finally

in

the

system

with

the

HW-All

network

the

Network-Done

operation

uses

the

ADD

reduction

the

maximum-time

operation

uses

the

MAX

reduction

and

the

quantum

barrier

uses

the

AND

reduction

Table

presents

the

speedups

for

WWT

for

the

three

systems

The

experiments

involve

WWT

simulating

a

node

shared-memory

multiprocessor

running

four

sharedmemory

programs

with

the

protocol

-LSB-

-RSB-

We

use

four

different

programs

in

order

to

observe

how

different

communication

patterns

affect

WWT?s

performance

We

first

examine

the

system

Looking

at

the

performance

improvements

we

see

that

with

zero-delayed

messages

three

of

the

four

inputs

exhibit

modest

improvement

-LRB-

no

more

than

-RRB-

while

Water

the

fourth

exhibits

a

improvement

Water

has

substantially

less

communication

than

the

other

four

inputs

so

the

bulk

of

the

time

spent

in

simulating

Water

is

in

the

quantum

barrier

in

contrast

to

the

other

three

inputs

As

a

result

performing

the

quantum

barrier

in

hardware

for

Water

has

a

greater

overall

impact

than

for

the

other

three

inputs

With

ms-delayed

messages

the

system

delivers

significantly

greater

performance

three

of

the

four

inputs

improve

by

nearly

and

Water

improves

by

a

factor

of

three

The

performance

of

the

software

reduction

used

to

implement

the

single-bit

AND

in

the

system

without

synchronization

hardware

degrades

significantly

with

the

higher

latency

network

and

hence

the

system

speeds

up

more

by

comparison

We

next

examine

the

HW-All

system

We

first

notice

that

with

zero-delayed

messages

for

all

inputs

except

Sparse

the

HW-All

system

is

actually

slower

than

the

system

This

slowdown

is

tied

to

the

method

used

for

performing

Network-Done

At

a

quantum

boundary

the

HW-All

system

repeatedly

sums

the

number

of

messages

sent

and

received

across

all

of

the

nodes

waiting

for

this

value

to

reach

zero

After

that

point

it

encounters

the

quantum

barrier

at

which

the

target-barrier

flags

on

all

nodes

are

AND?ed

However

the

system

skips

the

reduction

phase

and

instead

each

node

waits

for

all

its

ACKs

to

be

received

before

proceeding

to

the

quantum

barrier

resulting

in

only

one

synchronization

operation

Message

delay

ms

Benchmark

No

HW

No

HW

applu

dycore

TABLE

Speedups

for

message-passing

benchmarks

applu

and

dycore

instead

of

two

The

system

should

then

outperform

the

HW-All

system

if

the

target

application

has

low

enough

communication

For

all

inputs

except

Sparse

the

communication

is

in

fact

low

enough

Appbt

and

Tomcatv

are

stencil

computations

with

small

data

sets

and

Water

communicates

in

only

of

the

quanta

Sparse

on

the

other

hand

requires

multiple

broadcasts

and

a

reduction

each

iteration

With

ms-delayed

messages

the

HW-All

system

consistently

outperforms

the

system

This

is

because

the

cost

of

sending

ACKs

increases

drastically

and

therefore

the

HW-All

system?s

repeated

hardware

ADD

reductions

to

determine

network

quiescence

is

a

better

solution

Data-parallel

benchmarks

Table

presents

the

speedups

for

the

data-parallel

applications

comparing

systems

with

the

and

HW

All

synchronization

networks

against

a

system

without

synchronization

hardware

In

the

system

without

synchronization

hardware

the

barriers

reductions

and

broadcasts

are

implemented

using

messages

and

the

Network-Done

operation

is

implemented

with

ACKs

and

a

messagebased

barrier

similar

to

the

implementation

in

In

the

system

the

barriers

are

performed

in

hardware

with

the

Network-Done

operation

implemented

via

ACKs

and

a

hardware

barrier

all

other

operations

are

implemented

as

in

Finally

in

the

HW-All-based

system

the

barriers

reductions

and

broadcasts

are

all

implemented

in

dedicated

hardware

with

the

NetworkDone

operation

using

a

hardware

reduction

For

the

system

and

the

system

without

synchronization

hardware

we

additionally

throttle

the

software

broadcasts

by

forcing

the

host

to

delay

for

some

amount

of

time

-LRB-

ms

depending

on

the

benchmark

and

the

Again

the

ACK-based

method

was

found

to

be

faster

than

performing

software

reductions

network

latency

-RRB-

after

a

number

of

broadcasts

instead

of

letting

the

host

broadcast

data

to

the

nodes

ad

infinitum

Brewer

and

Kuszmaul

found

that

inserting

barriers

between

successive

communication

operations

such

as

cyclic

shifts

actually

improved

performance

by

up

to

a

factor

of

three

because

a

message

destined

for

a

processor

A

still

working

on

the

first

operation

could

not

be

delayed

by

a

message

sent

to

A

from

a

processor

B

working

on

the

second

operation

-LSB-

-RSB-

We

encounter

similar

conflicts

between

sets

of

messages

in

our

benchmarks

and

throttling

yields

substantial

benefit

We

determine

the

delaying

parameters

experimentally

We

first

examine

the

system

With

zero-delayed

messages

the

hardware

synchronization

provides

only

modest

benefit

for

metspin

and

for

nbody

This

result

is

expected

given

the

low

barrier

frequencies

of

these

applications

Nbody

benefits

less

than

metspin

because

nbody?s

increased

volume

of

broadcasts

dominates

communication

and

hence

its

runtime

decreasing

the

room

for

improvement

for

hardware

barriers

With

ms-delayed

messages

yields

a

improvement

for

metspin

and

no

noticeable

improvement

for

nbody

The

increased

benefit

of

hardware

barriers

in

metspin

with

ms-delayed

messages

occurs

simply

from

having

slower

software

barriers

We

now

examine

the

HW-All

system

This

system

yields

much

more

performance

improvement

for

metspin

and

for

nbody

For

both

applications

the

bulk

of

the

improvement

relative

to

comes

from

the

hardware

broadcasts

when

hardware

reductions

are

used

as

well

as

hardware

broadcasts

performance

improves

by

no

more

than

Since

nbody

broadcasts

more

often

than

metspin

we

would

expect

that

nbody

would

benefit

much

more

from

broadcast

hardware

than

metspin

However

metspin

benefits

a

great

deal

from

the

broadcast

hardware

because

its

computation-to-communication

ratio

is

lower

and

hence

the

separate

broadcast

network

prevents

broadcast

traffic

from

interfering

with

data

traffic

As

evidence

of

metspin?s

higher

degree

of

data

communication

we

Message

delay

Input

No

HW

HW-All

Appbt

Tomcatv

Sparse

Water

Appbt

ms

Tomcatv

Sparse

Water

TABLE

Speedups

for

message-passing

benchmark

WWT

hardware

will

be

cost-effective

if

the

performance

improvement

is

greater

than

the

increase

in

cost

speedup

-LRB-

synchronization

hardware

-RRB-

costup

-LRB-

synchronization

hardware

-RRB-

To

make

this

result

concrete

let

the

cost

of

a

base

NOW

-LRB-

without

hardware

synchronization

support

-RRB-

be

Cbase

and

let

the

cost

of

the

additional

synchronization

hardware

be

Csynch

then

the

cost

of

a

NOW

with

hardware

synchronization

support

is

Cbase

Csynch

Let

the

runtime

of

a

workload

W

on

the

base

NOW

be

Tbase

-LRB-

W

-RRB-

and

on

the

NOW

with

hardware

synchronization

support

be

Tsynch

-LRB-

W

-RRB-

Then

synchronization

hardware

is

more

costeffective

whenever

Tbase

-LRB-

W

-RRB-

and

Tsynch

-LRB-

W

-RRB-

assume

a

fixed

workload

W

that

could

either

be

a

single

application

or

the

weighted

mean

of

the

runtimes

of

many

different

jobs

To

make

our

results

independent

of

any

particular

hardware

implementation

we

calculate

the

break-even

cost

-LSB-

-RSB-

for

add-on

synchronization

hardware

by

making

Equation

an

equality

and

solving

for

Csynch

Thus

synchronization

hardware

is

cost-effective

-LRB-

for

a

particular

workload

W

-RRB-

if

its

actual

cost

is

less

than

the

break-even

cost

C

synch

System

cost

Without

loss

of

generality

we

make

C

synch

more

concrete

by

choosing

reasonable

estimates

of

the

base

system

cost

Cbase

and

synchronization

hardware

Csynch

In

this

section

and

throughout

the

rest

of

the

paper

we

treat

Cbase

Csynch

and

C

synch

as

per-node

costs

with

the

cost

of

shared

resources

-LRB-

e.g.

network

routers

-RRB-

amortized

over

all

nodes

Cbase

depends

heavily

on

the

particular

choice

of

workstation

node

List

prices

can

range

from

a

few

thou

Tbase

W

-LRB-

-RRB-

Tsynch

W

-LRB-

-RRB-

Cba

s

e

Csynch

Cbase

-LRB-

-RRB-

C

synch

Cbase

Tbase

W

-LRB-

-RRB-

Tsynch

W

-LRB-

-RRB-

-LRB-

-RRB-

found

that

when

we

throttled

broadcasts

in

the

system

without

synchronization

hardware

metspin?s

performance

improved

by

while

nbody?s

performance

improved

by

only

With

ms-delayed

messages

HW-All

yields

an

improvement

for

metspin

and

a

factor

of

improvement

for

nbody

These

improvements

are

the

opposite

of

those

achieved

with

zero-delayed

messages

In

metspin

we

see

much

less

improvement

than

we

saw

with

zero-delayed

messages

the

high

degree

of

data

communication

limits

the

potential

speedup

Conversely

for

nbody

we

see

much

more

improvement

than

we

saw

with

zero-delayed

messages

broadcasts

form

the

bulk

of

the

communication

since

data

communication

is

minimal

and

thus

the

speedup

improves

greatly

Cost/performance

In

this

section

we

examine

the

cost/performance

tradeoffs

for

global

synchronization

hardware

Section

presents

a

break-even

model

for

cost/performance

Section

examines

the

expected

cost

of

a

system

with

synchronization

hardware

Section

presents

the

resulting

cost

performance

break-even

points

for

our

applications

and

for

the

entire

workload

A

cost/performance

model

Intuitively

a

performance

enhancement

to

a

computer

system

is

cost-effective

only

if

the

increase

in

performance

exceeds

the

increase

in

cost

Wood

and

Hill

-LSB-

-RSB-

recently

formalized

this

intuition

by

showing

that

parallel

computing

is

more

cost-effective

than

uniprocessor

computing

whenever

the

following

inequality

holds

speedup

-LRB-

p

-RRB-

costup

-LRB-

p

-RRB-

where

speedup

-LRB-

p

-RRB-

is

the

runtime

on

one

processor

divided

by

the

runtime

on

p

processors

and

costup

-LRB-

p

-RRB-

is

the

cost

of

a

p-processor

system

divided

by

the

cost

of

a

system

In

particular

they

show

that

when

memory

is

a

significant

fraction

of

uniprocessor

cost

parallel

computing

can

be

cost-effective

even

with

very

low

speedups

This

same

intuition

also

applies

to

add-on

synchronization

hardware

for

a

parallel

computer

Synchronization

Msg

delay

Benchmark

No

H/W

HW-All

Metspin

Nbody

ms

Metspin

Nbody

TABLE

Data-parallel

speedups

sand

dollars

for

an

IBM-compatible

PC

to

many

tens

of

thousands

of

dollars

per

node

for

a

high-end

workstation

depending

on

the

processor

speed

memory

capacity

and

I/O

configuration

Similarly

fast

networks

range

from

approximately

per

node

for

Myrinet

-LRB-

adapter

plus

switches

-RRB-

to

over

per

node

for

Fore?s

ATM

network

For

the

purposes

of

this

paper

we

assume

Cbase

is

which

is

roughly

the

per-node

cost

of

the

Wisconsin

COW

our

local

network

of

dual-processor

SPARC

station

Csynch

depends

on

the

cost

of

the

synchronization-network

switch

and

workstation

interface

For

hardware

that

only

supports

simple

barriers

the

switch

cost

is

a

minor

component

since

it

can

be

implemented

using

a

few

standard

PALs

For

example

Dietz

et

al.

estimate

that

the

PAPERS

add-on

hardware?which

interfaces

to

the

workstation?s

parallel

port

and

implements

simple

barriers

and

binary

reductions?has

a

parts

cost

of

less

than

-LSB-

-RSB-

We

assume

a

commercial

implementation

of

PAPERS

would

cost

at

least

per

node

More

complex

functionality?for

example

integer

or

floating-point

reductions?requires

more

expensive

hardware

such

as

FPGAs

or

ASICs

Interfacing

to

the

workstation?s

I/O

or

memory

bus

can

also

be

expensive

particularly

for

low-volume

parts

like

synchronization

hardware

Nonetheless

it

seems

reasonable

that

the

total

per-node

cost

for

synchronization

hardware

should

not

exceed

the

cost

of

high-end

network

hardware

Thus

we

assume

that

synchronization

hardware

will

always

cost

less

than

per

node

Synchronization

hardware

is

cost-effective

when

the

actual

cost

Csynch

is

less

than

the

break-even

cost

C

synch

Thus

given

our

estimated

costs

add-on

synchronization

hardware

is

always

cost-effective

when

C

synch

is

greater

than

and

is

never

cost-effective

when

C

synch

is

less

than

Results

From

the

performance

results

from

Section

we

calculate

the

weighted

means

of

Tbase

and

Tsynch

for

our

entire

combined

workload

weighting

each

application

by

its

fraction

of

the

cumulative

runtime

of

all

applications

in

the

workload

For

the

WWT

application

we

used

the

average

runtime

of

all

four

inputs

Table

presents

the

values

of

C

synch

for

the

individual

applications

and

the

aggregate

workload

For

clarity

values

less

than

are

indicated

by

-LRB-

never

costeffective

-RRB-

values

greater

than

are

indicated

by

-LRB-

always

cost-effective

-RRB-

For

our

aggregate

workload

on

a

system

with

zero-delayed

messages

we

find

that

synchronization

hardware

is

cost-effective

when

the

hardware

costs

less

than

of

the

system

cost

or

given

our

cost

assumptions

On

a

system

with

ms-delayed

messages

synchronization

is

cost-effective

when

the

hardware

costs

less

than

of

the

system

cost

or

given

our

cost

assumptions

These

results

indicate

that

simple

synchronization

hardware

i.e.

is

likely

to

be

justifiable

for

a

system

with

a

fast

data

network

since

such

hardware

can

easily

be

built

for

much

less

than

More

complex

hardware

like

HW-All

is

less

clearly

justifiable

due

to

its

higher

parts

cost

and

greater

design

requirements

The

results

for

ms-delayed

messages

accentuate

the

importance

of

synchronization

hardware

for

systems

with

slow

data

networks

for

the

aggregate

workload

synchronization

hardware

is

nearly

always

cost-effective

with

ms-delayed

messages

delay

ms

delay

Application

HW-All

HW-All

barnes

DSMC

moldyn

water

applu

dycore

WWT

metspin

nbody

AGGREGATE

TABLE

Break-even

cost

of

synchronization

hardware

for

all

applications

and

for

the

aggregate

workload

for

zero-delayed

messages

and

ms-delayed

messages

Focusing

on

individual

benchmarks

we

see

that

of

the

shared-memory

benchmarks

only

water

strongly

motivates

purchasing

synchronization

hardware

the

costeffectiveness

of

synchronization

hardware

for

the

other

three

is

doubtful

However

shared-memory

overheads

in

Blizzard

are

quite

high

reducing

these

overheads

e.g.

with

hardware

support

for

shared

memory

will

increase

the

relative

benefit

of

synchronization

hardware

For

our

message-passing

benchmarks

only

dycore

and

WWT

motivate

purchasing

synchronization

hardware

For

WWT

synchronization

hardware

makes

sense

regardless

of

message

latency

WWT

is

unique

compared

to

the

rest

of

our

workload

in

that

it

must

synchronize

to

ensure

that

no

messages

are

in

flight

-LRB-

Network-Done

-RRB-

For

our

data-parallel

benchmarks

the

HW-All

network

is

cost-effective

and

economically

feasible

regardless

of

message

latency

and

is

arguably

necessary

in

order

to

garner

acceptable

speedups

The

parallel

function

invocations

from

the

host

processor

necessitate

the

HW-All

network

a

decentralized

SPMD-style

computational

model

would

lessen

this

necessity

Related

work

Numerous

proposals

for

add-on

synchronization

hardware

have

recently

appeared

PAPERS

-LSB-

-RSB-

is

a

low-cost

synchronization

network

which

supports

fine-grain

execution

on

workstation

clusters

specifically

operations

on

data

aggregates

as

in

a

data-parallel

program

and

VLIW

style

execution

Among

other

operations

PAPERS

supports

a

simple

barrier

and

a

single-bit

AND

operation

both

with

latencies

of

ms.

The

design

interfaces

to

a

network

of

PCs

running

Linux

with

connections

through

each

PC?s

parallel

port

Hall

and

Driscoll?s

COP

network

-LSB-

-RSB-

provides

synchronization

support

equivalent

to

the

HW-All

network

they

claim

that

its

cost

is

of

overall

system

cost

which

for

our

workload

is

clearly

feasible

Shang

and

Hwang

have

designed

add-on

synchronization

hardware

for

cluster-based

multiprocessors

allowing

synchronization

to

be

performed

both

within

and

between

clusters

-LSB-

-RSB-

The

ALLNODE

barrier

synchronization

hardware

uses

the

broadcast

facility

of

the

Allnode

switch

to

perform

barrier

operations

an

arbitrary

number

of

nodes

can

synchronize

in

a

few

microseconds

and

the

mechanism

consumes

less

than

of

network

bandwidth

-LSB-

-RSB-

Other

machines

besides

the

and

have

provided

synchronization

hardware

separate

from

the

data

network

PASM

a

hybrid

SIMD/MIMD

machine

uses

its

SIMD

synchronous

instruction-fetch

mechanism

as

a

barrier

when

in

MIMD

mode

-LSB-

-RSB-

A

MIMD

version

of

a

FFT

benchmark

with

the

hardware

barrier

support

was

faster

than

a

MIMD

version

without

the

hardware

barrier

Fast

barrier

synchronization

has

been

found

to

speed

up

specific

patterns

of

communication

and

computation

Brewer

and

Kuszmaul

found

that

using

the

hardware

barrier

on

the

to

limit

the

rate

of

message

injection

and

limit

congestion

improved

performance

by

more

than

a

factor

of

three

-LSB-

-RSB-

The

direct

deposit

message-passing

library

of

Stricker

et

al.

-LSB-

-RSB-

uses

hardware

barriers

rather

than

employing

buffering

or

handshaking

to

ensure

that

messages

have

been

delivered

to

their

destination

For

a

D

FFT

code

their

system

runs

approximately

times

faster

than

an

optimized

request-response

message-passing

library

Ramakrishnan

et

al.

-LSB-

-RSB-

present

two

methods

for

efficiently

supporting

deep

control

nesting

in

data-parallel

programs

by

using

synchronization

hardware

The

first

solution

employs

a

pair

of

single-bit

OR

and

AND

reductions

and

code

transformations

and

the

second

solution

requires

a

MAX

reduction

but

no

code

transformations

Additional

work

has

looked

at

synchronization

hardware

specifically

for

parallel

discrete-event

simulators

Reynolds

proposed

a

separate

synchronization

network

to

compute

minimum

next-event

times

minimum

timestamps

of

unacknowledged

messages

and

to

compute

Network-Done

-LSB-

-RSB-

Beaumont

et

al.

propose

to

dynamically

synthesize

application-specific

hardware

synchronization

for

a

desired

simulator

by

using

FPGAs

-LSB-

-RSB-

For

synchronous

parallel

discrete

event

simulators

they

suggest

synthesizing

a

barrier

and

a

MIN

reduction

to

determine

the

minimum

of

all

next-event

times

Summary

and

conclusions

This

paper

examined

the

cost/performance

tradeoffs

of

adding

a

separate

synchronization

network

to

a

network

of

workstations

We

studied

the

synchronization

requirements

of

three

important

classes

of

applications

sharedmemory

message-passing

and

data-parallel

experimentally

measured

the

performance

benefit

obtained

from

hardware

synchronization

support

We

combined

these

experimental

results

with

cost

estimates

to

calculate

the

cost/performance

break-even

point

where

hardware

synchronization

support

becomes

cost-effective

For

our

aggregate

workload

add-on

synchronization

hardware

is

cost-effective

if

it

costs

less

than

of

the

base

system

cost

for

a

fast

data

network

and

less

than

of

the

base

system

cost

for

a

slow

data

network

Individual

applications

may

benefit

much

more

from

synchronization

hardware

Among

our

applications

the

hardware

improves

performance

by

up

to

on

a

NOW

with

the

fast

data

network

and

up

to

a

factor

of

with

the

slow

data

network

The

HW-All

hardware

yields

further

improvement

increasing

performance

by

factors

of

up

to

and

for

the

two

network

latencies

Our

results

show

that

global

synchronization

hardware

can

be

a

cost-effective

addition

to

a

NOW

for

some

workloads

On

the

other

hand

nearly

half

our

applications

received

little

benefit

from

synchronization

support

A

key

advantage

of

add-on

synchronization

hardware?as

opposed

to

the

integrated

synchronization

hardware

in

most

MPPs?is

that

only

those

people

that

can

benefit

from

it

need

to

buy

it

References

-LSB-

-RSB-

David

Bailey

John

Barton

Thomas

Lasinski

and

Horst

Simon

The

NAS

Parallel

Benchmarks

Technical

Report

Revision

Ames

Research

Center

August

-LSB-

-RSB-

C.

Beaumont

P.

Boronat

J.

Champeau

J.-M

Filloque

and

B.

Pottier

Reconfigurable

technology

An

innovative

solution

for

parallel

discrete

event

simulation

support

In

Proceedings

of

the

Workshop

on

Parallel

and

Distributed

Simulation

-LRB-

PADS

-RRB-

pages

July

-LSB-

-RSB-

Matthias

A.

Blumrich

Kai

Li

Richard

Alpert

Cezary

Dubnicki

Edward

W.

Felten

and

Jonathon

Sandberg

Virtual

Memory

Mapped

Network

Interface

for

the

SHRIMP

Multicomputer

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

April

-LSB-

-RSB-

Nanette

J.

Boden

Danny

Cohen

Robert

E.

Felderman

Alan

E.

Kulawik

Charles

L.

Seitz

Jakov

N.

Seizovic

and

Wen-King

Su

Myrinet

A

Gigabit-per-Second

Local

Area

Network

IEEE

Micro

-LRB-

-RRB-

February

-LSB-

-RSB-

Eric

A.

Brewer

and

Bradley

C.

Kuszmaul

How

to

Get

Good

Performance

from

the

Data

Network

In

Proceedings

of

the

Eighth

International

Parallel

Processing

Symposium

-LRB-

IPPS

-RRB-

pages

Cancun

Mexico

June

-LSB-

-RSB-

Edward

C.

Bronson

Thomas

L.

Casavant

and

Leah

H.

Jamieson

Experimental

Application-Driven

Architecture

Analysis

of

an

SIMD/MIMD

Parallel

Processing

System

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

April

-LSB-

-RSB-

E.

D.

Brooks

III

The

butterfly

barrier

International

Journal

of

Parallel

Programming

-LRB-

-RRB-

-LSB-

-RSB-

David

Culler

Richard

Karp

David

Patterson

Abhijit

Sahay

Klaus

Erik

Schauser

Eunice

Santos

Ramesh

Subramonian

and

Thorsten

von

Eicken

LogP

Toward

a

Realistic

Model

of

Parallel

Computation

In

Fourth

ACM

SIGPLAN

Symposium

on

Principles

Practice

of

Parallel

Programming

-LRB-

PPOPP

-RRB-

pages

May

-LSB-

-RSB-

H.

G.

Dietz

W.

E.

Cohen

T.

Muhammad

and

T.

I.

Mattox

Compiler

Techniques

For

Fine-Grain

Execution

On

Workstation

Clusters

Using

PAPERS

In

Proceedings

of

the

Seventh

Annual

Workshop

on

Languages

and

Compilers

for

Parallel

Computing

August

-LSB-

-RSB-

G.

Fox

et

al.

Solving

Problems

on

Concurrent

Processors

Prentice

Hall

Englewood

Cliffs

NJ

-LSB-

-RSB-

Douglas

V.

Hall

and

Michael

A.

Driscoll

Hardware

for

Fast

Global

Operations

on

Multicomputers

In

Proceedings

of

the

Ninth

International

Parallel

Processing

Symposium

-LRB-

IPPS

-RRB-

pages

Santa

Barbara

CA

April

-LSB-

-RSB-

D.

Hensgen

R.

Finkel

and

U.

Manber

Two

algorithms

for

barrier

synchronization

International

Journal

of

Parallel

Programming

-LRB-

-RRB-

-LSB-

-RSB-

R.

E.

Kessler

and

J.

L.

Schwarzmeier

CRAY

A

New

Dimension

for

Cray

Research

In

Proceedings

of

COMPCON

pages

San

Francisco

California

Spring

-LSB-

-RSB-

Charles

E.

Leiserson

Zahi

S.

Abuhamdeh

David

C.

Douglas

Carl

R.

Feynman

Mahesh

N.

Ganmukhi

Jeffrey

V.

Hill

W.

Daniel

Hillis

Bradley

C.

Kuszmaul

Margaret

A.

St.

Pierre

David

S.

Wells

Monica

C.

Wong

Shaw-Wen

Yang

and

Robert

Zak

The

Network

Architecture

of

the

Connection

Machine

In

Proceedings

of

the

Fifth

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

-LRB-

SPAA

-RRB-

July

-LSB-

-RSB-

B.

Lubachevsky

Synchronization

barrier

and

related

tools

for

shared

memory

parallel

programming

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

II

II

August

-LSB-

-RSB-

Shubhendu

S.

Mukherjee

Shamik

D.

Sharma

Mark

D.

Hill

James

R.

Larus

Anne

Rogers

and

Joel

Saltz

Efficient

Support

for

Irregular

Applications

on

Distributed-Memory

Machines

In

Fifth

ACM

SIGPLAN

Symposium

on

Principles

Practice

of

Parallel

Programming

-LRB-

PPOPP

-RRB-

To

appear

-LSB-

-RSB-

Howard

T.

Olnowich

ALLNODE

Barrier

Synchronization

Network

In

Proceedings

of

the

Ninth

International

Parallel

Processing

Symposium

-LRB-

IPPS

-RRB-

pages

Santa

Barbara

CA

April

-LSB-

-RSB-

G.

Parisi

Statistical

Field

Theory

Addison-Wesley

Reading

MA

-LSB-

-RSB-

Steven

Przybylski

Mark

Horowitz

and

John

Hennessy

Performance

Tradeoffs

in

Cache

Design

In

Annual

International

Symposium

on

Computer

Architecture

pages

June

-LSB-

-RSB-

Vara

Ramakrishnan

Isaac

D.

Scherson

and

Raghu

Subramanian

Efficient

Techniques

for

Fast

Nested

Barrier

Synchronization

In

Proceedings

of

the

Seventh

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

-LRB-

SPAA

-RRB-

-LSB-

-RSB-

Steven

K.

Reinhardt

Mark

D.

Hill

James

R.

Larus

Alvin

R.

Lebeck

James

C.

Lewis

and

David

A.

Wood

The

Wisconsin

Wind

Tunnel

Virtual

Prototyping

of

Parallel

Computers

In

Proceedings

of

the

ACM

Sigmetrics

Conference

on

Measurement

and

Modeling

of

Computer

Systems

pages

May

-LSB-

-RSB-

Steven

K.

Reinhardt

James

R.

Larus

and

David

A.

Wood

Tempest

and

Typhoon

User-Level

Shared

Memory

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

April

-LSB-

-RSB-

Paul

F.

Reynolds

Jr.

An

Efficient

Framework

for

Parallel

Simulations

In

Proceedings

of

the

SCS

multi-conference

on

Advances

in

Parallel

and

Distributed

Simulation

pages

Anaheim

CA

January

-LSB-

-RSB-

Ioannis

Schoinas

Babak

Falsafi

Alvin

R.

Lebeck

Steven

K.

Reinhardt

James

R.

Larus

and

David

A.

Wood

Fine-grain

Access

Control

for

Distributed

Shared

Memory

In

Proceedings

of

the

Sixth

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

-LRB-

ASPLOS

VI

-RRB-

pages

October

-LSB-

-RSB-

Shisheng

Shang

and

Kai

Hwang

Distributed

Hardwired

Barrier

Synchronization

for

Scalable

Multiprocessor

Clusters

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

June

-LSB-

-RSB-

Jaswinder

Pal

Singh

Wolf-Dietrich

Weber

and

Anoop

Gupta

SPLASH

Stanford

Parallel

Applications

for

Shared

Memory

Computer

Architecture

News

-LRB-

-RRB-

March

-LSB-

-RSB-

T.

Stricker

J.

Stichnoth

D.

O?Hallaron

S.

Hinrichs

and

T.

Gross

Decoupling

Synchronization

and

Data

Transfer

in

Message

Passing

Systems

of

Parallel

Computers

In

Proceedings

of

the

Ninth

International

Conference

on

Supercomputing

-LRB-

ICS

-RRB-

Barcelona

Spain

July

-LSB-

-RSB-

Max

J.

Suarez

and

Lawrence

L.

Takacs

Documentation

of

the

ARIES/GEOS

Dynamical

Core

Version

Technical

Report

Vol

NASA

Goddard

Space

Flight

Center

March

-LSB-

-RSB-

Teruo

Utsumi

Masayuki

Ikeda

and

Moriyuki

Takamura

Architecture

of

the

Parallel

Supercomputer

In

Proceedings

of

Supercomputing

pages

Washington

D.C.

November

-LSB-

-RSB-

Steven

Cameron

Woo

Moriyoshi

Ohara

Evan

Torrie

Jaswinder

Pal

Singh

and

Anoop

Gupta

The

Programs

Characterization

and

Methodological

Considerations

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

Santa

Margherita

Ligure

Italy

June

-LSB-

-RSB-

David

A.

Wood

Satish

Chandra

Babak

Falsafi

Mark

D.

Hill

James

R.

Larus

Alvin

R.

Lebeck

James

C.

Lewis

Shubhendu

S.

Mukherjee

Subbarao

Palacharla

and

Steven

K.

Reinhardt

Mechanisms

for

Cooperative

Shared

Memory

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

May

Also

appeared

in

CMG

Transactions

Spring

-LSB-

-RSB-

David

A.

Wood

and

Mark

D.

Hill

Cost-Effective

Parallel

Computing

IEEE

Computer

-LRB-

-RRB-

February

networks of workstations
synchronization