SIMPLE

A

Methodology

for

Programming

High

Performance

Algorithms

on

Clusters

of

Symmetric

Multiprocessors

-LRB-

SMPs

-RRB-

-LRB-

Preliminary

Version

-RRB-

David

A.

Bader

Joseph

J?aJ?ay

Institute

for

Advanced

Computer

Studies

University

of

Maryland

College

Park

MD

fdbader

josephg@umiacs.umd.edu

May

Abstract

We

describe

a

methodology

for

developing

high

performance

programs

running

on

clusters

of

SMP

nodes

Our

methodology

is

based

on

a

small

kernel

-LRB-

SIMPLE

-RRB-

of

collective

communication

primitives

that

make

efficient

use

of

the

hybrid

shared

and

message

passing

environment

We

illustrate

the

power

of

our

methodology

by

presenting

experimental

results

for

sorting

integers

two-dimensional

fast

Fourier

transforms

-LRB-

FFT

-RRB-

and

constraint-satisfied

searching

Our

testbed

is

a

cluster

of

DEC

AlphaServer

2100 4/275

nodes

interconnected

by

an

ATM

switch

Problem

Overview

With

the

cost

of

commercial

off-the-shelf

-LRB-

COTS

-RRB-

high

performance

interconnects

falling

and

the

respective

performance

of

microprocessors

increasing

workstation

clusters

have

become

an

attractive

computing

platform

offering

potentially

a

superior

cost

effective

performance

-LSB-

-RSB-

Indeed

this

trend

highly

leverages

both

workstation-focused

technologies

including

systems

software

and

networking

infrastructure

for

example

COTS

networks

-LRB-

e.g.

Ethernet

Myrinet

FDDI

or

ATM

-RRB-

In

recent

years

we

have

seen

the

maturing

of

Symmetric

Multiprocessors

-LRB-

SMPs

-RRB-

technology

-LRB-

for

example

hardware

The

support

by

NSF

CISE

Postdoctoral

Research

Associate

in

Experimental

Computer

Science

No.

is

gratefully

acknowledged

ySupported

in

part

by

NSF

grant

No

and

NSF

HPCC/GCAG

grant

No

support

for

hierarchical

memory

management

multithreaded

operating

system

kernels

and

optimizing

compilers

-RRB-

and

the

heavy

reliance

upon

SMPs

as

the

work-intensive

servers

for

client/server

applications

It

can

be

argued

that

-RRB-

many

future

workstations

will

be

SMPs

with

more

than

one

processor

and

-RRB-

SMP

nodes

will

be

the

basis

of

workstation

clusters

There

are

already

several

examples

of

clusters

of

SMPs

such

as

clusters

of

DEC

AlphaServer

-LSB-

-RSB-

SGI

Challenge/PowerChallenge

-LSB-

-RSB-

or

Sun

Ultra

HPC

machines

and

the

IBM

SP

system

with

SMP

High

nodes

-LSB-

-RSB-

With

the

acceptance

of

message

passing

standards

such

as

MPI

-LSB-

-RSB-

it

has

become

easier

to

design

portable

parallel

algorithms

making

use

of

these

primitives

However

the

focus

of

MPI

is

a

standard

for

communicating

between

shared-nothing

processors

and

although

MPI

programs

run

on

clusters

of

SMPs

this

is

not

necessarily

the

optimal

methodology

for

these

platforms

Programming

Message

Passing

Simple

Programming

Shared

Memory

Programming

Shared

Everything

Shared

Nothing

Figure

The

SIMPLE

methodology

efficiently

combines

shared

memory

programming

on

a

node

with

message

passing

between

nodes

This

paper

describes

a

methodology

for

programming

clusters

of

SMP

nodes

-LRB-

herein

referred

to

as

COSMOS

-RRB-

which

aids

in

the

design

and

implementation

of

efficient

high

performance

parallel

algorithms

We

call

this

model

SIMPLE

referring

to

the

joining

of

the

SMP

and

MPI-like

message

passing

paradigms

and

the

simple

programming

approach

-LRB-

see

Figures

and

-RRB-

Programming

methodologies

for

COSMOS

fall

into

two

categories

The

first

distributed

shared

memory

-LRB-

DSM

-RRB-

systems

-LRB-

for

example

TreadMarks

-LSB-

-RSB-

from

Rice

University

Multigrain

Shared

Memory

-LRB-

MGS

-RRB-

-LSB-

-RSB-

from

MIT

and

Coherent

Virtual

Machine

-LRB-

CVM

-RRB-

-LSB-

-RSB-

from

University

of

Maryland

-RRB-

provides

a

software

layer

which

simulates

coherent

shared

memory

between

nodes

by

internally

using

messaging

to

move

around

specific

data

or

referenced

memory

pages

The

second

based

on

message

passing

primitives

-LRB-

for

example

MPI

-LSB-

-RSB-

-RRB-

enforces

a

shared-nothing

paradigm

between

tasks

and

all

communication

and

coordination

between

tasks

are

performed

through

the

exchange

of

explicit

messages

even

between

tasks

on

a

node

with

physically

shared

memory

For

example

the

model

assumed

in

-LSB-

-RSB-

is

that

each

processor

in

the

cluster

will

be

assigned

a

message

passing

-LRB-

MPI-level

-RRB-

-LRB-

k?az

m?os

-RRB-

noun

Greek

kosmos

c.

an

orderly

harmonious

systematic

universe

a

complex

orderly

self-inclusive

system

Cluster

Of

Shared

Memory

Nodes

Communication

Phase

Communication

Phase

Computation

Phase

Computation

Phase

Message

Passing

Tasks

Message

Passing

Algorithm

Communication

Phase

Communication

Phase

SMP

Algorithm

SMP

Algorithm

SMP

Algorithm

SMP

Algorithm

SMP

Algorithm

SMP

Algorithm

SMP

Algorithm

SMP

Algorithm

Simple

Methodology

Computation

Phase

Computation

Phase

SIMPLE

Algorithm

Figure

On

the

left

we

show

a

message

passing

algorithm

where

each

task

uses

sequential

code

during

computation

phases

On

the

right

the

SIMPLE

approach

replaces

each

computation

step

with

an

optimal

SMP

algorithm

process

with

lower

latency

communication

between

processes

on

the

same

SMP

node

than

with

internode

messages

However

our

work

differs

from

both

of

these

approaches

in

that

we

advocate

a

hybrid

methodology

which

maps

directly

to

underlying

architectural

aspects

As

such

we

combine

shared

memory

programming

on

shared

memory

nodes

with

message

passing

communication

between

these

nodes

The

main

results

of

this

paper

are

A

programming

methodology

for

COSMOS

which

is

both

efficient

and

portable

This

methodology

provides

a

path

for

optimizing

message

passing

algorithms

to

clusters

of

SMPs

A

small

message

passing

kernel

for

clusters

connected

by

ATM

switches

which

is

superior

in

performance

when

compared

with

the

known

MPI

implementations

High

performance

algorithms

based

on

our

methodology

for

sorting

integers

constraint-satisfied

searching

and

computing

the

two-dimensional

FFT

The

organization

of

this

paper

is

as

follows

Section

addresses

our

computation

methodology

and

target

parallel

machine

architectures

The

design

of

algorithms

for

COSMOS

is

described

in

Section

Our

SIMPLE

communication

primitives

are

described

in

Sections

and

which

include

collective

communication

and

computation

operations

as

well

as

functions

for

spreading

work

among

processors

on

a

node

or

across

an

entire

cluster

of

machines

We

present

several

examples

of

efficient

algorithms

using

the

SIMPLE

model

for

design

analysis

and

empirical

testing

The

first

algorithm

given

in

Section

sorts

integers

using

a

radix-based

approach

The

performance

of

this

algorithm

is

compared

with

that

of

an

efficient

MPI

radix

sort

highlighting

the

significant

improvement

introduced

by

our

methodology

Section

presents

the

second

algorithm

two-dimensional

FFT

which

is

the

cornerstone

computation

in

many

applications

The

third

algorithm

an

example

of

constraint-satisfied

searching

finds

all

solutions

to

the

n-queens

problem

and

can

be

found

in

Section

Experimental

results

are

provided

from

implementations

on

a

cluster

of

DEC

AlphaServer

2100 4/275

nodes

each

with

a

DEC

-LRB-

-RRB-

Mbps

PCI

card

connected

to

a

DEC

Gigaswitch/ATM

switch

and

using

the

MPI

-LRB-

e.g.

LAM

-LSB-

-RSB-

MPICH

-LSB-

-RSB-

or

CHIMP

c

-LSB-

-RSB-

-RRB-

and

POSIX

threads

-LRB-

DECthreads

-LSB-

-RSB-

or

freely

available

pthreads

implementations

-LSB-

-RSB-

-RRB-

packages

Finally

Section

presents

a

direction

for

future

work

The

SIMPLE

Parallel

Computation

Methodology

We

use

a

simple

paradigm

for

designing

efficient

and

portable

parallel

algorithms

First

we

will

describe

characteristics

of

our

target

parallel

machine

architecture

followed

in

the

next

section

by

a

set

of

SIMPLE

communication

and

computation

primitives

which

are

implemented

efficiently

and

are

intended

as

user

level

directives

PP

P

P

P

PPP

P

P

P

P

Interconnection

Network

Figure

Cluster

of

processing

elements

Our

architecture

consists

of

a

collection

of

SMP

nodes

interconnected

by

a

communication

network

-LRB-

as

shown

in

Figure

-RRB-

that

can

be

modeled

as

a

complete

graph

on

which

communication

is

subject

to

the

restrictions

imposed

by

the

latency

and

the

bandwidth

properties

of

the

network

Each

SMP

node

contains

several

identical

processors

each

typically

with

its

own

on-chip

cache

and

a

larger

off-chip

cache

which

have

uniform

access

to

a

shared

memory

and

other

resources

such

as

the

network

interface

We

view

a

parallel

algorithm

as

a

sequence

of

local

computation

interleaved

with

communication

steps

Shared

Memory

Bus

or

Switching

Network

Interface

Network

CPU

CPU

CPU

CPU

Figure

A

typical

symmetric

multiprocessing

-LRB-

SMP

-RRB-

node

used

in

a

cluster

is

on-chip

level-one

cache

and

is

off-chip

level-two

cache

We

use

the

parameter

r

to

represent

the

number

symmetric

processors

per

node

-LRB-

see

Figure

for

a

diagram

of

a

typical

node

-RRB-

Notice

that

each

CPU

typically

has

its

own

on-chip

cache

-LRB-

-RRB-

and

a

larger

off-chip

level

two

cache

-LRB-

-RRB-

which

can

be

tightly

integrated

into

the

memory

system

to

provide

fast

memory

accesses

and

cache

coherence

In

practice

SMP

configurations

range

between

and

CPU

modules

attached

to

a

shared

bus

and

main

memory

The

shared

memory

programming

of

each

SMP

node

is

based

on

threads

which

communicate

via

coordinated

accesses

to

shared

memory

Several

primitives

will

be

discussed

in

the

following

section

which

for

example

synchronize

the

threads

at

a

barrier

enable

one

thread

to

broadcast

data

to

the

other

threads

or

calculate

reductions

across

the

threads

In

our

methodology

only

the

CPUs

from

a

certain

node

have

access

to

that

node

's

configuration

In

this

manner

there

is

no

restriction

that

all

nodes

must

be

identical

and

certainly

COSMOS

can

be

constructed

from

SMP

nodes

of

different

sizes

Thus

the

number

of

threads

on

a

specific

remote

node

is

not

globally

available

Because

of

this

our

methodology

supports

only

nodeoriented

communication

meaning

we

restrict

communication

such

that

given

any

source

node

s

and

destination

node

d

with

s

d

only

one

thread

on

node

s

can

send

-LRB-

receive

-RRB-

a

message

to

-LRB-

from

-RRB-

node

d

at

any

given

time

We

will

show

later

that

no

performance

loss

will

be

incurred

by

this

restriction

Next

we

describe

the

SIMPLE

primitives

which

aid

in

the

design

of

efficient

and

portable

parallel

algorithms

For

ease

of

discussion

we

separate

the

primitives

into

two

categories

communication

-LRB-

in

Section

-RRB-

and

computation

-LRB-

in

Section

-RRB-

where

communication

directs

the

flow

of

information

between

threads

and

computation

refers

to

the

control

mechanisms

among

threads

Communication

Primitives

Recv

SendRecv

Barrier

Reduce

Broadcast

Barrier

Send

SMP

Node

Library

Internode

Communication

Library

SIMPLE

Communication

Library

Barrier

Reduce

Broadcast

Allreduce

Alltoall

Alltoallv

Gather

Scatter

Alltoall

Alltoallv

Gather

Scatter

Reduce

Broadcast

Allreduce

Figure

Hierarchy

of

SMP

message

passing

and

SIMPLE

communication

libraries

The

communication

primitives

are

grouped

into

three

modules

Internode

Communication

Library

-LRB-

ICL

-RRB-

SMP

Node

and

SIMPLE

The

ICL

communication

library

provides

a

small

kernel

for

internode

communication

similar

to

MPI

but

with

less

overhead

than

several

of

the

freely

available

implementations

of

MPI

-LRB-

for

example

MPICH

LAM

or

CHIMP

-RRB-

and

is

based

upon

a

reliable

application-layer

send

and

receive

primitive

as

well

as

a

send-and-receive

primitive

which

handles

the

exchanging

of

messages

between

sets

of

nodes

where

each

participating

node

is

the

source

and

destination

of

one

message

The

library

also

provides

a

barrier

operation

based

upon

the

send

and

receive

which

halts

the

execution

at

each

node

until

all

nodes

check

into

the

barrier

at

which

time

the

nodes

may

continue

execution

In

addition

ICL

includes

collective

communication

primitives

for

example

reduce

broadcast

allreduce

alltoall

alltoallv

gather

and

scatter

The

SMP

Node

Library

contains

three

important

primitives

for

an

SMP

node

barrier

broadcast

and

reduce

whereby

on

a

single

node

barrier

synchronizes

the

threads

broadcast

ensures

that

each

thread

has

the

most

recent

copy

of

a

shared

memory

location

and

reduce

performs

a

reduction

operation

with

a

binary

associative

operator

-LRB-

for

example

addition

multiplication

maximum

minimum

bitwise-AND

and

bitwise-OR

-RRB-

with

one

datum

per

thread

Finally

the

SIMPLE

communication

library

built

on

top

of

ICL

and

SMP

Node

includes

the

primitives

for

the

SIMPLE

model

barrier

reduce

broadcast

allreduce

alltoall

alltoallv

gather

and

scatter

These

hierarchical

layers

of

our

communication

libraries

are

pictured

in

Figure

The

SMP

Node

ICL

and

SIMPLE

libraries

are

implemented

at

a

high-level

completely

in

user

space

-LRB-

see

Figure

-RRB-

Because

no

kernel

modification

is

required

these

libraries

easily

port

to

new

platforms

Message

Passing

-LRB-

e.g.

MPI

ICL

-RRB-

SMP

Node

Library

Kernel

User

Libraries

Kernel

Space

User

Space

POSIX

threads

SIMPLE

User

Program

Figure

User

code

can

access

SIMPLE

SMP

message

passing

and

standard

user

libraries

Note

that

SIMPLE

operates

completely

in

user

space

Parameter

Description

NODES

p

Total

number

of

nodes

in

the

cluster

MYNODE

My

node

rank

from

to

NODES

THREADS

r

Total

number

of

threads

on

my

node

MYTHREAD

The

rank

of

my

thread

on

this

node

from

to

THREADS

TID

Total

number

of

threads

in

the

cluster

ID

My

thread

rank

with

respect

to

the

cluster

from

to

TID

Table

I

The

local

context

parameters

available

to

each

SIMPLE

thread

As

mentioned

previously

the

number

of

threads

per

node

can

vary

along

with

machine

size

Thus

each

thread

has

a

small

set

of

context

information

which

holds

such

parameters

as

the

number

of

threads

on

the

given

node

the

number

of

nodes

in

the

machine

the

rank

of

that

node

in

the

machine

and

the

rank

of

the

thread

both

-RRB-

on

the

node

and

-RRB-

across

the

machine

Table

I

describes

these

parameters

in

detail

Because

the

design

of

the

communication

libraries

is

modular

it

is

easy

to

experiment

with

different

implementations

For

example

the

MPI

libraries

offer

a

more

robust

communication

suite

than

our

ICL

Library

at

a

significant

cost

However

the

lower-level

ICL

and

SMP

Node

primitives

can

be

replaced

by

vendor-supplied

MPI

and

SMP

primitives

We

ran

a

simple

experiment

whereby

a

message

is

sent

between

two

nodes

and

plotted

the

results

Figure

shows

the

communication

time

and

respective

bandwidth

for

sending

a

message

between

two

DEC

AlphaServer

nodes

using

the

Digital

Gigaswitch/ATM

and

adapter

cards

which

have

a

peak

bandwidth

rating

of

Mbps

The

results

summarized

in

Table

II

reflect

the

latency

and

bandwidth

characteristics

of

point-topoint

messages

between

a

pair

of

DEC

AlphaServer

nodes

using

the

Internode

Communication

Library

-LRB-

ICL

-RRB-

and

the

best

MPI

implementation

-LRB-

MPICH

-RRB-

The

theoretical

raw

peak

bandwidth

is

Time

Bandwidth

Figure

Internode

Communication

Performance

Communication

Library

Latency

Bandwidth

ICL

s

Mbps

MPI

-LRB-

MPICH

-RRB-

s

Mbps

Table

II

The

latency

and

bandwidth

characteristics

of

point-to-point

messages

between

a

pair

of

DEC

AlphaServer

nodes

with

ATM

adapters

using

both

the

Internode

Communication

Library

-LRB-

ICL

-RRB-

and

MPI

-LRB-

MPICH

-RRB-

Mbps

and

our

application

level

measurement

finds

the

ICL

library

achieving

Mbps

while

MPI/MPICH

only

reaches

about

half

of

that

In

addition

our

latency

measurements

are

less

than

half

of

that

incurred

using

MPI/MPICH

The

SIMPLE

library

can

use

either

MPI

or

ICL

for

passing

messages

between

the

nodes

taking

into

account

the

following

two

important

considerations

First

ICL

is

not

a

replacement

for

MPI

The

ICL

library

offers

only

a

small

subset

of

the

functionality

available

in

MPI

for

example

ICL

uses

only

a

single

communication

group

specializes

the

implementation

for

an

ATM

network

instead

of

implementing

communication

on

an

abstract

channel

device

restricts

the

number

of

outstanding

communication

events

and

provides

less

status

information

and

no

additional

debugging

hooks

Second

ICL

provides

both

weak

support

for

multithreading

where

the

user

is

responsible

for

maintaining

mutually

exclusive

use

of

communication

channels

via

implicit

algorithmic

design

or

explicit

locks

and

strong

support

where

internal

locking

mechanisms

automatically

protect

the

user

from

corrupting

the

communication

layer

However

the

MPI

implementation

must

be

thread-safe

Thus

the

ICL

communication

library

achieves

the

higher

performance

for

two

main

reasons

first

latency

is

reduced

because

by

purpose

ICL

is

not

as

generalized

as

MPI

and

second

bandwidth

is

increased

in

ICL

by

optimizing

the

network

parameters

for

an

ATM

switch

Implementation

of

the

SMP

Node

Library

As

Figure

shows

the

SMP

Node

library

can

be

implemented

on

top

of

a

portable

threads

layer

such

as

POSIX

threads

-LRB-

pthreads

-RRB-

or

if

available

via

possibly

faster

native

primitives

Our

SMP

Node

library

is

based

upon

pthreads

and

thus

is

portable

to

POSIX

standard

platforms

The

three

SMP

Node

primitives

which

we

require

for

SIMPLE

are

reduce

barrier

and

broadcast

For

example

if

the

number

of

threads

is

small

each

thread

entering

a

reduce

primitive

first

acquires

a

lock

stores

the

reduction

of

its

element

with

the

shared

element

and

increases

the

counter

of

waiting

threads

If

it

is

not

the

last

to

enter

the

thread

releases

the

lock

and

blocks

waiting

for

a

condition

If

in

fact

the

thread

is

the

last

to

enter

it

resets

the

operation

and

uses

a

condition

broadcast

to

wake

up

the

other

threads

Finally

all

threads

return

the

result

For

a

larger

number

of

threads

the

reduce

primitive

can

be

implemented

with

an

efficient

parallel

k-ary

tree

for

a

suitable

value

of

k

The

pthreads

standard

requires

primitives

for

synchronization

with

condition

variables

and

mutual

exclusion

locks

but

does

not

include

a

primitive

for

barrier

synchronization

The

barrier

primitive

can

be

implemented

similarly

to

reduce

since

all

threads

must

enter

before

each

thread

can

continue

Since

a

side

effect

of

the

pthreads

locking

mechanism

is

an

SMP

memory

coherence

barrier

the

thread

with

data

to

broadcast

writes

this

information

in

a

shared

memory

location

and

then

all

threads

enter

a

barrier

Afterwards

each

thread

reads

this

shared

memory

location

which

is

guaranteed

to

be

consistent

Now

that

the

basics

of

the

communication

system

and

node

library

have

been

presented

we

are

ready

to

describe

some

of

the

SIMPLE

communication

primitives

The

Alltoall

primitive

One

of

the

most

important

collective

communication

events

is

the

Alltoall

-LRB-

or

transpose

-RRB-

primitive

which

transmits

regular

sized

blocks

of

data

between

each

pair

of

nodes

More

formally

given

a

collection

of

p

nodes

each

with

an

n

element

sending

buffer

where

p

divides

n

the

Alltoall

operation

consists

of

each

node

i

sending

its

jth

block

of

np

data

elements

to

node

j

where

node

j

stores

the

data

from

i

in

the

ith

block

of

its

receiving

buffer

for

all

-LRB-

i

j

p

-RRB-

An

efficient

message

passing

implementation

of

Alltoall

would

be

as

follows

The

notation

vari

refers

to

memory

location

var

-LRB-

np

i

-RRB-

and

src

and

dst

point

to

the

source

and

destination

arrays

respectively

ffl

Step

-LRB-

-RRB-

Copy

the

appropriate

np

elements

from

srcMYNODE

to

dstMYNODE

ffl

Step

-LRB-

-RRB-

For

i

to

NODES

do

-LCB-

A

-RRB-

Set

k

MYNODE

i

-LCB-

B

-RRB-

Send

np

elements

from

srck

to

node

k

and

Receive

np

elements

from

node

k

to

dstk

To

implement

this

algorithm

we

use

multiple

threads

per

node

The

local

memory

copy

in

Step

-LRB-

-RRB-

trivially

can

be

performed

concurrently

by

one

thread

while

the

remaining

threads

handle

the

internode

communication

as

follows

The

p

iterations

of

the

loop

in

Step

-LRB-

-RRB-

are

partitioned

in

a

straightforward

manner

to

the

remaining

threads

Each

thread

has

the

information

necessary

to

calculate

its

subset

of

loop

indices

and

thus

this

loop

partitioning

step

requires

no

synchronization

overheads

In

Figure

we

compare

the

performance

of

three

Alltoall

primitives

using

the

MPI

ICL

and

SIMPLE

communication

libraries

on

four

and

eight

DEC

AlphaServer

2100 4/275

nodes

In

all

cases

the

SIMPLE

primitive

is

the

fastest

typically

by

a

factor

or

two

or

three

over

MPI

Now

with

only

a

single

network

interface

per

node

why

would

one

expect

a

performance

improvement

by

using

multiple

threads

Our

algorithm

exploits

two

main

sources

of

parallelism

The

first

is

task

level

concurrently

exhibited

by

one

thread

performing

the

local

memory

copy

while

other

threads

utilizing

the

network

The

second

form

of

parallelism

is

less

obvious

but

nonetheless

an

important

observation

Four

Nodes

Eight

Nodes

Figure

Comparison

of

Alltoall

-LRB-

Transpose

-RRB-

Primitives

Unlike

clusters

of

workstations

where

each

network

interface

is

closed

coupled

to

a

single

processor

's

communication

stream

on

an

SMP

node

the

operating

system

is

itself

capable

of

internal

parallelism

-LRB-

via

multi-threaded

kernel

routines

-RRB-

and

can

more

efficiently

pipeline

requests

between

the

processors

and

the

network

interface

Computation

Primitives

In

the

previous

section

we

provided

an

overview

of

our

communication

library

Next

we

will

explore

the

set

of

user

level

directives

called

SIMPLE

computation

primitives

which

do

not

communicate

data

but

affect

a

thread

's

execution

through

-RRB-

loop

parallelization

-RRB-

restriction

or

-RRB-

shared

memory

management

Basic

support

for

data

parallelism

that

is

parallel

do

concurrent

execution

of

loops

across

processors

on

one

or

more

nodes

is

discussed

first

Next

we

describe

the

control

primitives

which

restrict

-LRB-

or

contextualize

-RRB-

thread

execution

for

example

to

some

subset

of

threads

or

nodes

Lastly

we

cover

a

few

shared

memory

management

directives

which

make

it

easier

for

the

user

to

develop

portable

shared

memory

code

by

standardizing

the

interface

for

allocating

and

deallocating

shared

memory

locations

Data

Parallel

The

SIMPLE

methodology

contains

several

basic

pardo

directives

for

executing

loops

concurrently

on

one

or

more

SMP

nodes

provided

that

no

dependencies

exist

in

the

loop

Typically

this

is

useful

when

an

independent

operation

is

to

be

applied

to

every

location

in

an

array

for

example

in

the

element-wise

addition

of

two

arrays

Pardo

implicitly

partitions

the

loop

to

the

threads

without

the

need

for

coordinating

overheads

such

as

synchronization

or

communication

between

processors

By

default

pardo

uses

block

partitioning

of

the

loop

assignment

values

to

the

threads

which

typically

results

in

better

cache

utilization

due

to

the

array

locations

on

left-hand

side

of

the

assignment

being

owned

by

local

caches

more

often

than

not

However

SIMPLE

explicitly

provides

both

block

and

cyclic

partitioning

interfaces

for

the

pardo

directive

Similar

mechanisms

exist

for

parallelizing

loops

across

a

COSMOS

The

all

pardo

cyclic

-LRB-

i

a

b

-RRB-

directive

will

cyclically

assign

each

iteration

of

the

loop

across

the

entire

collection

of

processors

For

example

i

a

will

be

executed

on

the

first

processor

of

the

first

node

i

a

on

the

second

processor

of

the

first

node

and

so

on

with

i

a

r

on

the

last

processor

of

the

first

node

The

iteration

with

i

a

r

is

executed

by

the

first

processor

on

the

second

node

After

r

p

iterations

are

assigned

the

next

index

will

again

be

assigned

to

the

first

processor

on

the

first

node

A

similar

directive

called

all

pardo

block

which

accepts

the

same

arguments

assigns

the

iterations

in

a

block

fashion

to

the

processors

thus

the

first

b?a

rp

iterations

are

assigned

to

the

first

processor

the

next

block

of

iterations

are

assigned

to

the

second

processor

and

so

forth

With

either

of

these

SIMPLE

directives

each

processor

will

execute

at

most

l

nrp

m

iterations

for

a

loop

of

size

n.

Control

The

second

category

of

SIMPLE

computation

primitives

control

which

threads

can

participate

in

the

context

by

using

restrictions

Control

Primitives

max

number

of

MYNODE

MYTHREAD

Primitive

Definition

participating

restriction

restriction

threads

on

one

thread

only

one

thread

per

node

p

on

one

node

all

threads

on

a

single

node

r

on

one

only

one

thread

on

a

single

node

on

thread

-LRB-

i

-RRB-

one

thread

-LRB-

i

-RRB-

per

node

p

i

on

node

-LRB-

j

-RRB-

all

threads

on

node

j

r

j

Table

III

Subset

of

SIMPLE

Control

Primitives

Table

III

defines

each

control

primitive

and

gives

the

largest

number

of

threads

able

to

execute

the

portion

of

the

algorithm

restricted

by

this

statement

For

example

if

only

one

thread

per

node

needs

to

execute

a

command

it

can

be

preceded

with

the

on

one

thread

directive

Suppose

data

has

been

gathered

to

a

single

node

Work

on

this

data

can

be

accomplished

on

that

node

by

preceding

the

statement

with

on

one

node

The

combination

of

these

two

primitives

restricts

execution

to

exactly

one

thread

and

can

be

shortcut

with

the

on

one

directive

Memory

Management

Finally

shared

memory

allocations

are

the

third

category

of

SIMPLE

computation

primitives

Two

directives

are

used

node

malloc

for

dynamically

allocating

a

shared

structure

and

node

free

for

releasing

this

memory

back

to

the

heap

The

node

malloc

primitive

is

called

by

all

threads

on

a

given

node

and

takes

as

a

parameter

the

number

of

bytes

to

be

allocated

dynamically

from

the

heap

The

primitive

returns

to

each

thread

a

valid

pointer

to

the

shared

memory

location

In

addition

a

thread

may

allow

others

to

access

local

data

by

broadcasting

the

corresponding

memory

address

When

this

shared

memory

is

no

longer

required

the

node

free

primitive

releases

it

back

to

the

heap

Thus

we

have

described

the

fundamental

elements

of

the

SIMPLE

methodology

and

can

now

present

a

high-level

approach

for

designing

algorithms

on

COSMOS

SIMPLE

Algorithmic

Design

In

this

section

we

describe

the

SIMPLE

programming

model

as

seen

by

the

user

and

the

runtime

support

for

executing

SIMPLE

code

Programming

Model

The

user

writes

an

algorithm

for

an

arbitrary

cluster

size

p

and

SMP

size

r

-LRB-

where

each

node

can

assign

possibly

different

values

to

r

at

runtime

-RRB-

using

the

parameters

from

Table

I.

SIMPLE

expects

a

standard

main

function

-LRB-

called

SIMPLE

main

-LRB-

-RRB-

-RRB-

that

to

the

user

's

view

is

immediately

up

and

running

on

each

thread

in

the

COSMOS

Thus

the

user

does

not

need

to

make

any

special

calls

to

initialize

the

libraries

or

communication

channels

SIMPLE

makes

available

the

rank

of

each

thread

on

its

node

or

across

the

cluster

and

algorithms

can

use

these

ranks

in

a

straightforward

fashion

to

break

symmetries

and

partition

work

The

only

argument

of

SIMPLE

main

-LRB-

-RRB-

is

THREADED

a

macro

pointing

to

a

private

data

structure

which

holds

local

thread

information

If

the

user

's

main

function

needs

to

call

subroutines

which

make

use

of

the

SIMPLE

library

this

information

is

easily

passed

via

another

macro

TH

in

the

calling

arguments

After

all

threads

exit

from

the

main

function

the

SIMPLE

code

performs

a

shut

down

process

Runtime

Support

When

a

SIMPLE

algorithm

first

begins

its

execution

on

a

COSMOS

the

SIMPLE

runtime

support

has

already

initialized

parallel

mechanisms

such

as

barriers

and

established

the

network-based

internode

communication

channels

which

remain

open

for

the

life

of

the

program

The

various

libraries

described

in

Section

have

runtime

initializations

which

take

place

as

follows

The

runtime

startup

routines

for

a

SIMPLE

algorithm

are

performed

in

two

steps

First

the

ICL

initialization

expands

computation

across

the

nodes

in

a

cluster

by

launching

a

master

thread

on

each

of

the

p

nodes

and

establishing

communication

channels

between

each

pair

of

nodes

Second

each

master

thread

launches

r

user

threads

where

each

node

is

at

least

an

r-way

It

is

assumed

that

the

r

CPUs

concurrently

execute

the

r

threads

The

thread

flow

of

an

example

SIMPLE

algorithm

is

shown

in

Figure

As

mentioned

previously

our

methodology

supports

only

node-oriented

communication

that

is

given

any

source

node

s

and

destination

node

d

with

s

d

only

one

thread

on

node

s

can

send

-LRB-

receive

-RRB-

a

message

to

-LRB-

from

-RRB-

node

d

during

a

communication

step

Also

note

that

the

master

thread

does

not

participate

in

any

computation

but

sits

idle

until

the

completion

of

the

user

code

at

which

time

it

coordinates

the

joining

of

threads

and

exiting

of

processes

rule

of

thumb

in

practice

is

to

use

r

threads

on

an

r

SMP

node

which

allows

operating

system

tasks

to

fully

utilize

at

least

one

CPU

Master

Thread

Master

Thread

Master

Thread

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Thread

-LRB-

-RRB-

Node

Node

Node

Communication

Phase

Node

Barrier

-LRB-

Collective

-RRB-

Time

Irregular

Communication

using

sends

and

receives

Figure

Example

of

a

SIMPLE

algorithm

flow

of

master

and

compute-based

user

threads

Note

that

the

only

responsibility

of

each

master

thread

is

only

to

launch

and

later

join

user

threads

but

never

to

participate

in

computation

or

communication

Our

model

is

simply

implemented

using

a

portable

thread

package

called

POSIX

threads

-LRB-

pthreads

-RRB-

which

is

a

standard

-LRB-

IEEE

Std

c

-RRB-

supplied

with

POSIX

-LRB-

-LSB-

-RSB-

-RRB-

Note

that

pthreads

are

also

available

in

the

standard

Distributed

Computing

Environment

-LRB-

DCE

-RRB-

used

in

operating

systems

such

as

OSF

-LSB-

-RSB-

and

AIX

-LSB-

-RSB-

A

Possible

Approach

The

latency

for

message

passing

is

an

order

of

magnitude

higher

than

accessing

local

memory

Thus

the

most

costly

operation

in

a

SIMPLE

algorithm

is

internode

communication

and

algorithmic

design

must

attempt

to

minimize

the

communication

costs

between

the

nodes

Since

this

is

a

similar

optimization

criterion

used

when

designing

efficient

message

passing

algorithms

-LSB-

-RSB-

it

is

beneficial

to

first

design

an

efficient

message

passing

algorithm

on

a

COSMOS

and

then

adapt

the

algorithm

for

the

SIMPLE

paradigm

Given

an

efficient

message

passing

algorithm

an

incremental

process

can

be

used

to

design

an

efficient

SIMPLE

algorithm

The

computational

work

assigned

to

each

node

is

mapped

into

an

efficient

SMP

algorithm

For

example

independent

operations

such

as

those

arising

in

functional

parallelism

-LRB-

for

example

independent

I/O

and

computational

tasks

or

the

local

memory

copy

in

the

SIMPLE

Alltoall

primitive

presented

in

the

previous

section

-RRB-

or

loop

parallelism

typically

can

be

threaded

For

functional

parallelism

this

means

that

each

thread

acts

as

a

functional

process

for

that

task

and

for

loop

parallelism

each

thread

computes

its

portion

of

the

computation

concurrently

Note

that

we

may

need

to

apply

loop

transformations

to

reduce

data

dependencies

between

the

threads

Thread

synchronization

is

a

costly

operation

when

implemented

in

software

and

when

possible

should

be

avoided

Example

SIMPLE

Permutation

As

mentioned

briefly

in

the

previous

section

more

complex

communication

algorithms

can

be

developed

from

the

primitives

described

in

Section

For

example

the

SIMPLE

Alltoallv

communication

primitive

handles

the

case

where

the

messages

for

each

destination

are

already

collected

into

a

contiguous

block

of

an

array

holding

all

of

the

messages

and

the

messages

to

be

received

from

the

other

nodes

likewise

will

appear

in

contiguous

blocks

in

another

array

Suppose

instead

that

each

node

contains

a

set

of

messages

each

message

holding

a

destination

tag

such

that

no

node

sends

or

receives

more

than

h

messages

-LSB-

-RSB-

The

resulting

h-relation

personalized

communication

-LSB-

-RSB-

is

a

useful

communication

routine

used

in

a

variety

of

parallel

algorithms

Each

node

determines

the

number

of

its

keys

to

be

sent

to

every

other

node

announces

these

counts

to

the

destination

nodes

rearranges

the

input

elements

into

a

single

send

buffer

such

that

all

keys

for

the

destination

node

j

are

in

contiguous

memory

and

appear

before

the

keys

for

node

j

routes

the

all-to-all

communication

event

and

finally

unpacks

each

received

element

into

the

correct

destination

position

A

description

of

the

algorithm

is

as

follows

ffl

Step

-LRB-

-RRB-

For

each

node

i

count

the

number

of

keys

labeled

with

destination

node

j

for

-LRB-

j

p

-RRB-

On

each

node

each

of

r

threads

-LCB-

A

-RRB-

histograms

of

the

input

concurrently

and

-LCB-

B

-RRB-

merges

these

r

histograms

into

a

single

array

-LRB-

sendCount

-RRB-

for

the

node

ffl

Step

-LRB-

-RRB-

Using

sendCount

and

the

arrays

generated

in

Step

-LRB-

-RRB-

rearrange

the

input

elements

into

a

single

send

buffer

such

that

all

keys

with

destination

node

j

are

in

contiguous

memory

and

appear

before

keys

with

destination

j

On

each

node

each

of

r

threads

place

of

the

elements

concurrently

ffl

Step

-LRB-

-RRB-

Apply

the

SIMPLE

Alltoall

primitive

to

the

sendCount

array

using

the

block

size

Hence

at

the

end

of

this

step

each

node

will

know

the

number

of

keys

it

will

receive

from

every

other

node

-LRB-

recvCount

-RRB-

ffl

Step

-LRB-

-RRB-

Route

the

all-to-all

communication

event

-LRB-

with

the

SIMPLE

Alltoallv

communication

primitive

-RRB-

using

the

send

sendCount

and

recvCount

arrays

ffl

Step

-LRB-

-RRB-

Each

node

unpacks

its

received

elements

and

places

each

in

the

correct

array

position

Since

this

is

a

permutation

routing

no

collisions

will

occur

in

the

final

array

and

r

threads

can

each

unpack

of

the

array

concurrently

into

shared

memory

This

algorithm

relies

on

efficient

implementations

of

the

Alltoall

and

Alltoallv

primitives

and

assumes

that

the

number

of

messages

exchanged

between

each

pair

of

nodes

is

fairly

balanced

However

if

significant

imbalance

exists

an

alternative

algorithm

might

replace

the

one-phase

data

routing

in

Step

-LRB-

-RRB-

with

a

two-phase

routing

approach

using

balanced

Alltoall

primitives

in

each

phase

-LRB-

see

-LSB-

-RSB-

-RRB-

Similarly

other

complex

communication

algorithms

can

be

developed

using

the

SIMPLE

methodology

The

above

permutation

algorithm

minimizes

the

number

of

communication

steps

which

is

optimal

on

our

COSMOS

testbed

where

communication

is

expensive

compared

with

local

computation

Next

we

show

an

example

of

an

algorithm

for

sorting

which

makes

use

of

a

special

case

of

the

h-relation

personalized

communication

where

the

number

of

messages

to

be

sent

and

received

are

the

same

Radixsort

Consider

the

problem

of

sorting

n

integers

spread

evenly

across

a

cluster

of

p

shared-memory

r-way

SMP

nodes

where

n

Fast

integer

sorting

is

crucial

for

solving

problems

in

many

domains

and

as

such

is

used

as

a

kernel

in

several

parallel

benchmarks

such

as

-LSB-

-RSB-

and

SPLASH

-LSB-

-RSB-

We

present

an

efficient

sorting

algorithm

based

on

our

SIMPLE

methodology

We

chose

the

technique

of

radix

sort

since

it

is

well

known

for

sequential

programming

but

efficient

methods

for

solving

this

problem

on

clusters

of

SMPs

are

not

The

SIMPLE

approach

for

radix

sort

is

similar

to

our

efficient

message

passing

algorithm

-LSB-

-RSB-

except

when

applicable

shared

memory

computation

replaces

sequential

node

work

and

communication

uses

the

improved

SIMPLE

communication

library

Consider

the

problem

of

sorting

n

integer

keys

in

the

range

-LSB-

M

-RSB-

that

are

distributed

equally

in

the

shared

memories

of

a

p-node

cluster

of

r-way

SMPs

Radix

sort

decomposes

each

key

into

groups

of

ae-bit

digits

for

a

suitably

chosen

ae

and

sorts

the

keys

by

applying

a

counting

sort

routine

on

each

of

the

ae-bit

digits

beginning

with

the

digit

containing

the

least

significant

bit

positions

-LSB-

-RSB-

Let

R

p.

Assume

-LRB-

w.l.o.g.

-RRB-

that

the

number

of

nodes

is

a

power

of

two

say

p

and

hence

Rp

is

an

integer

k

that

the

NAS

IS

benchmark

requires

that

the

integers

be

ranked

and

not

necessarily

placed

in

sorted

order

SIMPLE

Counting

Sort

Algorithm

Counting

Sort

algorithm

sorts

n

integers

in

the

range

-LSB-

R

-RSB-

by

using

R

counters

to

accumulate

the

number

of

keys

equal

to

the

value

i

in

bucket

Bi

for

i

R

followed

by

determining

the

rank

of

each

key

Once

the

rank

of

each

key

is

known

we

can

move

each

key

into

its

correct

position

using

a

permutation

-LRB-

np

relation

-RRB-

routing

-LSB-

-RSB-

whereby

no

node

is

the

source

or

destination

of

more

than

np

keys

Counting

Sort

is

a

stable

sorting

routine

that

is

if

two

keys

are

identical

their

relative

order

in

the

final

sort

remains

the

same

as

their

initial

order

We

present

an

overview

of

the

original

message

passing

Counting

Sort

algorithm

and

follow

this

with

the

adaptations

to

the

algorithm

using

our

SIMPLE

methodology

In

a

practical

integer

sorting

problem

we

expect

R

ss

nr

The

pseudocode

for

our

Counting

Sort

algorithm

uses

six

major

steps

and

can

be

described

as

follows

ffl

Step

-LRB-

-RRB-

For

each

node

i

-LRB-

i

p

-RRB-

count

the

frequency

of

its

np

keys

that

is

compute

Hi

-LSB-

k

-RSB-

the

number

of

keys

equal

to

k

for

-LRB-

k

R

-RRB-

ffl

Step

-LRB-

-RRB-

Apply

the

Alltoall

primitive

to

the

H

arrays

using

the

block

size

Rp

Hence

at

the

end

of

this

step

each

node

will

hold

Rp

consecutive

rows

of

H

ffl

Step

-LRB-

-RRB-

Each

node

locally

computes

the

prefix-sums

of

its

rows

of

the

array

H

ffl

Step

-LRB-

-RRB-

Apply

the

-LRB-

inverse

-RRB-

Alltoall

primitive

to

the

R

corresponding

prefix-sums

augmented

by

the

total

count

for

each

bin

The

block

size

of

the

Alltoall

primitive

is

ffl

Step

-LRB-

-RRB-

On

each

node

compute

the

ranks

of

the

np

local

elements

using

the

arrays

generated

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

ffl

Step

-LRB-

-RRB-

Perform

a

personalized

communication

of

keys

to

rank

location

using

an

np

relation

algorithm

We

can

adapt

this

message

passing

algorithm

to

our

SIMPLE

methodology

with

the

following

changes

In

Step

-LRB-

-RRB-

the

computation

can

be

divided

evenly

among

the

threads

Thus

on

each

node

each

of

r

threads

A

-RRB-

histograms

of

the

input

concurrently

and

B

-RRB-

merges

these

r

histograms

into

a

single

array

for

node

i.

For

the

prefix-sum

calculations

on

each

node

in

Step

-LRB-

-RRB-

since

the

rows

are

independent

each

of

r

threads

can

compute

the

prefix-sum

calculations

for

Rrp

rows

concurrently

Also

the

computation

of

ranks

on

each

node

in

Step

-LRB-

-RRB-

can

be

handled

by

r

threads

where

each

thread

calculates

nrp

ranks

of

the

node

's

local

elements

Communication

can

also

be

improved

by

replacing

the

message

passing

Alltoall

primitive

used

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

with

the

appropriate

SIMPLE

primitive

The

h-relation

used

in

the

final

step

of

Counting

Sort

is

a

permutation

routing

since

h

np

and

was

given

in

the

previous

section

SIMPLE

Radix

Sort

Algorithm

The

message

passing

Radix

Sort

algorithm

makes

several

passes

of

the

previous

message

passing

Counting

Sort

in

order

to

completely

sort

integer

keys

Counting

Sort

can

be

used

as

the

intermediate

sorting

routine

because

it

provides

a

stable

sort

Let

the

n

integer

keys

fall

in

the

range

-LSB-

M

-RSB-

and

M

Then

we

need

bae

passes

of

Counting

Sort

each

pass

works

on

ae-bit

digits

of

the

input

keys

starting

from

the

least

significant

digit

of

ae

bits

to

the

most

significant

digit

Radix

Sort

easily

can

be

adapted

for

clusters

of

SMPs

by

using

the

SIMPLE

Counting

Sort

routine

Performance

We

now

provide

empirical

performance

results

for

the

Radix

Sort

algorithm

on

various

platforms

We

first

graph

the

performance

of

the

SIMPLE

Radix

Sort

on

a

cluster

of

SMPs

and

show

that

indeed

our

implementation

is

efficient

Next

we

show

results

of

a

good

MPI

Radix

Sort

on

an

IBM

and

compare

this

code

with

that

of

a

shared

memory

sort

on

a

single

SMP

node

Finally

we

compare

the

SIMPLE

Radix

Sort

with

that

of

DSM

and

MPI

Radix

Sorts

on

a

cluster

of

SMPs

Figure

Execution

Time

of

SIMPLE

Radix

Sort

with

r

and

p

and

nodes

The

performance

of

the

SIMPLE

Radix

Sort

algorithm

on

a

COSMOS

of

DEC

AlphaServer

nodes

is

given

in

Figure

In

this

experiment

we

use

four

user

threads

per

node

and

vary

both

the

problem

size

and

the

number

of

nodes

used

Here

the

SIMPLE

code

shows

linear

speedups

when

using

multiple

nodes

of

a

COSMOS

platform

DEC

AlphaServer

2100 4/275

IBM

Figure

Percentage

of

execution

time

of

radix

sort

spent

in

the

Alltoallv

communication

primitive

used

in

Step

-LRB-

-RRB-

of

the

permutation

routing

on

clusters

of

DEC

and

IBM

nodes

In

Figure

we

have

plotted

the

percentage

of

the

running

time

of

radix

sort

spent

performing

the

Alltoallv

communication

primitive

used

in

Step

-LRB-

-RRB-

of

the

permutation

algorithm

for

various

IBM

and

DEC

cluster

sizes

After

each

key

is

ranked

during

the

Counting

Sort

this

step

sends

each

key

to

its

destination

For

moderately

sized

inputs

on

the

DEC

cluster

roughly

a

third

of

the

execution

time

is

spent

in

this

communication

step

and

for

larger

problems

more

than

half

the

time

is

spent

in

this

step

In

comparison

for

most

inputs

the

IBM

spends

less

than

percent

of

its

execution

time

for

in

the

corresponding

step

These

performance

graphs

indicate

that

radix

sort

is

largely

communication

bound

on

the

DEC

Cluster

while

computation

bound

on

the

IBM

These

results

are

expected

as

the

IBM

has

a

faster

network

but

less

processing

power

on

each

node

than

the

DEC

cluster

As

we

claim

in

the

introduction

software

distributed

shared

memory

and

message

passing

algorithms

are

not

optimal

for

COSMOS

platforms

For

instance

we

ported

an

efficient

SMP

radix

sort

code

into

a

software

distributed

shared

memory

package

called

Coherent

Virtual

Machine

-LRB-

CVM

version

-RRB-

-LSB-

-RSB-

which

is

an

extension

of

the

commercial

TreadMarks

-LSB-

-RSB-

DSM

implementation

The

performance

of

this

DSM

radix

sort

is

given

in

Figure

In

addition

we

took

an

efficient

message

passing

code

for

radix

sort

-LRB-

the

reader

is

referred

to

-LSB-

-RSB-

for

a

complete

analysis

of

the

algorithm

and

its

performance

-RRB-

whose

performance

on

an

IBM

is

shown

in

Figure

The

IBM

contains

uniprocessor

nodes

interconnected

by

a

fast

switch

On

this

platform

the

message

passing

algorithm

Figure

Performance

of

MPI

Radix

Sort

on

an

IBM

with

p

and

thin

nodes

performs

very

well

That

is

for

a

fixed

machine

size

when

the

problem

size

is

halved

the

performance

roughly

is

cut

in

half

as

well

In

addition

for

a

fixed

problem

size

when

twice

as

many

processors

are

used

to

solve

a

given

sorting

problem

as

expected

the

time

is

again

halved

An

analysis

of

the

difference

in

raw

performance

between

the

IBM

and

the

DEC

cluster

shows

the

following

When

computation

dominates

the

DEC

platform

is

faster

in

raw

execution

time

however

as

communication

increases

the

imbalance

of

communication

bandwidth

to

computation

speed

on

the

DEC

cluster

becomes

more

pronounced

and

the

IBM

is

the

faster

platform

For

example

consider

the

problem

of

sorting

one

million

keys

A

single

node

of

the

DEC

AlphaServer

cluster

sorts

these

keys

in

approximately

seconds

whereas

one

node

of

an

IBM

requires

more

than

four

seconds

However

when

p

nodes

both

the

DEC

cluster

and

the

require

roughly

a

half

a

second

even

though

the

DEC

cluster

is

using

four

times

as

many

processors

In

Figure

we

plot

the

execution

of

the

MPI

radix

sort

code

on

a

single

DEC

AlphaServer

2100 4/275

-LRB-

SMP

-RRB-

node

using

one

two

and

four

threads

of

execution

For

large

inputs

notice

that

the

performance

improves

slightly

when

more

threads

are

used

but

still

there

is

no

great

difference

when

using

multiple

threads

on

a

single

node

In

this

same

figure

following

the

SIMPLE

methodology

we

plot

the

performance

of

a

shared

memory

radix

sort

of

the

same

input

on

this

SMP

node

In

addition

to

being

almost

an

order

of

magnitude

faster

unlike

the

message

passing

code

the

SIMPLE

algorithm

shows

significant

speedups

when

using

multiple

threads

Figure

Comparison

of

MPI

-LRB-

MPICH

-RRB-

and

SIMPLE

Radix

Sort

with

r

and

with

p

node

Four

Nodes

Eight

Nodes

Figure

Comparison

of

DSM

MPI

and

SIMPLE

Radix

Sort

on

a

cluster

of

DEC

AlphaServer

2100 4/275

nodes

Note

that

we

tested

the

DSM/CVM

radix

sort

implementation

using

one

to

four

processes

per

node

and

the

MPI/MPICH

implementation

using

both

one

and

four

MPI

tasks

per

node

The

SIMPLE

implementation

uses

r

threads

per

node

and

p

and

p

nodes

on

the

left

and

right

respectively

Figure

provides

a

summary

of

the

performance

of

the

SIMPLE

methodology

with

DSM/CVM

or

MPI/MPICH

on

our

testbed

In

this

experiment

we

compare

the

performance

of

a

SIMPLE

radix

sort

code

using

both

four

and

eight

SMP

nodes

with

that

of

both

DSM/CVM

and

MPI/MPICH

code

for

various

cases

such

as

using

one

or

multiple

threads

of

execution

per

node

In

all

situations

on

the

cluster

of

SMPs

testbed

the

SIMPLE

algorithm

substantially

outperforms

that

of

both

the

distributed

shared

memory

and

the

message

passing

implementations

Two-Dimensional

Fast

Fourier

Transform

Fourier

transforms

are

at

the

heart

of

many

computations

in

medical

image

analysis

computational

fluid

dynamics

speech

recognition

seismic

analysis

image

and

signal

processing

and

detecting

surface

defects

in

manufacturing

The

straightforward

and

well-known

FFT

takes

a

one-dimensional

signal

and

transforms

it

into

a

one-dimensional

vector

of

frequency

components

However

when

the

input

is

a

two-dimensional

digital

image

a

corresponding

two-dimensional

FFT

-LRB-

-RRB-

can

be

used

similarly

to

transform

the

image

into

its

two-dimensional

frequency

image

A

computation

can

be

reduced

to

's

by

first

performing

's

across

the

rows

followed

by

's

down

the

columns

similar

to

the

FFT

algorithms

in

-LSB-

-RSB-

which

performs

an

all-to-all

transpose

of

the

data

between

two

phases

of

local

computation

In

fact

a

k-dimensional

transform

can

be

formed

by

performing

k

-LRB-

k

-RRB-

dimensional

FFTs

along

each

axis

In

Figure

we

illustrate

the

major

steps

of

the

two-dimensional

FFT

algorithm

Assume

that

an

n

n

image

is

originally

partitioned

in

strips

among

the

p

nodes

such

that

each

node

originally

holds

np

rows

of

the

image

ffl

Step

-LRB-

-RRB-

Each

node

performs

np

n-point

FFTs

across

the

rows

of

its

local

image

strip

ffl

Step

-LRB-

-RRB-

Locally

rearrange

the

image

such

that

each

np

np

block

of

the

image

is

transposed

Thus

for

each

block

each

column

of

data

is

gathered

into

contiguous

memory

in

preparation

for

the

following

step

ffl

Step

-LRB-

-RRB-

Apply

the

Alltoall

primitive

to

transpose

the

blocks

ffl

Step

-LRB-

-RRB-

Locally

rearrange

the

data

such

that

each

node

holds

np

columns

of

the

image

in

contiguous

memory

ffl

Step

-LRB-

-RRB-

Each

node

performs

np

n-point

FFTs

down

the

of

its

local

image

strip

Note

that

the

FFT

algorithm

above

is

valid

for

both

the

message

passing

and

SIMPLE

paradigms

The

SIMPLE

optimization

assigns

nrp

rows

and

columns

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

to

each

thread

and

substitutes

the

SIMPLE

Alltoall

primitive

in

Step

-LRB-

-RRB-

-LRB-

Note

that

the

fact

the

image

strip

is

transposed

so

the

FFTs

are

performed

physically

across

rows

of

memory

local

rearrangements

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

similarly

can

be

optimized

for

shared

memory

threads

on

each

node

-RRB-

Figure

The

Two-dimensional

FFT

Algorithm

with

blocks

of

rows

initially

distributed

across

the

nodes

-LRB-

top

left

-RRB-

performs

local

one-dimensional

FFTs

across

the

rows

-LRB-

top

right

-RRB-

locally

rearranges

data

-LRB-

bottom

left

-RRB-

transposes

the

image

such

that

each

node

holds

a

block

of

columns

and

-LRB-

bottom

right

-RRB-

performs

local

one-dimensional

FFTs

across

the

columns

We

begin

with

an

efficient

message

passing

algorithm

for

the

FFT

The

one-dimensional

FFT

used

in

the

first

and

last

steps

is

a

benchmark

kernel

from

netlib

-LSB-

-RSB-

As

shown

in

Figure

the

message

passing

implementation

performs

very

well

on

the

IBM

When

we

fix

a

problem

size

and

double

the

number

of

processors

the

execution

time

scales

appropriately

Also

when

the

image

size

is

increased

four-fold

-LRB-

say

from

to

pixels

-RRB-

on

a

given

number

of

processors

again

as

expected

the

execution

time

follows

the

predicted

complexity

of

the

FFT

algorithm

Figure

MPI

Code

for

Two-dimensional

FFT

on

an

IBM

Without

any

modifications

we

ran

the

message

passing

code

on

both

a

cluster

of

DEC

AlphaServer

2100 4/275

nodes

-LRB-

with

only

one

task

per

node

-RRB-

and

using

message

passing

solely

on

a

single

node

-LRB-

see

the

left

and

right

plates

of

Figure

respectively

-RRB-

For

a

fixed

image

size

the

performance

does

not

scale

well

with

four

more

more

nodes

In

addition

the

code

running

on

one

two

and

four

processors

of

a

single

node

shows

very

little

gain

by

using

more

than

a

single

CPU

per

node

Compare

these

results

with

the

SIMPLE

execution

times

presented

in

Figure

for

a

variety

of

configurations

-LRB-

from

one

to

eight

nodes

and

from

one

to

four

CPUs

per

node

-RRB-

and

image

sizes

-LRB-

to

pixels

-RRB-

For

instance

on

a

pixel

image

using

just

a

single

node

and

four

tasks

the

message

passing

implementation

takes

approximately

seconds

while

the

SIMPLE

approach

is

about

a

second

faster

or

equivalently

two-thirds

the

execution

time

We

see

an

improvement

for

using

multiple

CPUs

on

a

node

even

at

our

largest

available

machine

configuration

of

eight

nodes

cluster

node

Figure

MPI

Code

for

Two-Dimensional

FFT

On

the

left

we

show

the

performance

on

a

cluster

of

DEC

AlphaServer

nodes

On

the

right

multiple

processors

on

a

single

DEC

AlphaServer

2100 4/275

are

used

Figure

Two-dimensional

FFT

on

a

cluster

of

DEC

AlphaServer

nodes

using

the

SIMPLE

methodology

Constrained

Search

Algorithm

The

n-Queens

Prob

lem

A

classic

puzzle

used

in

benchmarking

and

performance

analysis

is

the

n-queens

problem

Here

the

objective

is

to

place

n

queens

on

an

n

n

chessboard

such

that

of

the

queens

can

attack

each

other

For

those

readers

unfamiliar

with

the

game

of

chess

this

restricts

the

placement

of

the

queens

such

that

no

two

queens

share

the

same

rank

-LRB-

or

row

-RRB-

column

or

diagonal

Since

there

are

n

-LRB-

-RRB-

ways

to

place

n

queens

on

an

n?n

board

a

brute

force

algorithm

which

checks

each

of

these

candidate

solutions

is

infeasible

If

we

limit

the

search

space

to

include

just

those

candidates

which

have

exactly

one

queen

per

rank

then

we

reduce

the

search

space

to

nn

possible

candidates

which

is

still

too

large

Therefore

the

most

desirable

search

method

aggressively

eliminates

sets

of

candidate

solutions

which

do

not

satisfy

the

constraints

Our

algorithm

uses

a

tree-based

backtracking

approach

where

queens

are

placed

one

by

one

on

each

rank

until

all

n

queens

are

placed

If

a

constraint

is

not

met

or

a

solution

is

found

the

last

queen

placed

on

the

board

is

removed

and

re-placed

in

the

next

column

position

This

is

equivalent

to

a

depth-first

search

with

pruning

of

branches

where

the

constraints

are

not

met

Note

that

we

are

not

taking

into

consideration

the

special

topological

properties

and

symmetries

of

the

chessboard

for

example

rotating

known

solutions

by

and

to

discover

similar

solutions

or

reflecting

solutions

about

the

horizontal

vertical

or

diagonal

axes

n

-LRB-

-RRB-

n

n

n

rank

rank

rank

rank

Column

Column

Column

Column

-LRB-

-RRB-

n

-LRB-

-RRB-

n

Figure

Encoding

of

the

chessboard

A

parallel

n-queens

constraint-satisfaction

search

algorithm

with

p

processors

uses

distributed

search

tree

approach

as

follows

First

the

algorithm

enumerates

a

set

of

independent

search-tree

seed

Search

Space

k

n

k

k

Figure

Search

Tree

for

a

constrained

search

e.g.

the

nqueens

problem

nodes

and

partitions

these

to

the

processors

Suppose

we

generate

all

possible

queen

placements

on

the

first

k

ranks

of

an

n

n

chessboard

There

will

be

nk

of

these

placements

uniquely

encoded

into

the

integers

from

to

nk

by

summing

a

term

from

each

queen

placed

on

rank

i

-LRB-

i

k

-RRB-

and

column

j

-LRB-

j

n

-RRB-

equal

to

jni

For

clarity

Figure

shows

the

value

of

each

position

on

the

chessboard

Note

that

this

is

equivalent

to

converting

to

decimal

a

base

n

number

with

digit

i

-LRB-

i

k

-RRB-

representing

the

column

position

of

queen

i

These

nk

partial

placements

can

then

be

partitioned

evenly

among

the

processors

and

-RRB-

checked

for

validity

and

-RRB-

used

as

a

root

node

for

a

sequential

depth-first

search

of

the

remaining

n

k

queen

positions

from

that

starting

point

Figure

contains

an

example

of

this

search

tree

for

k

The

algorithm

which

decodes

the

array

of

k

column

positions

from

a

partial

solution

oe

with

-LRB-

oe

nk

-RRB-

is

as

follows

ffl

For

i

to

k

do

columni

j

oemodni

ni

k

We

have

looked

at

three

approaches

and

in

each

running

time

is

directly

proportional

to

the

maximum

of

the

number

of

solutions

found

on

each

of

the

threads

The

first

uses

a

block

partitioning

of

the

nk

search

nodes

to

the

p

processors

-LRB-

using

the

all

pardo

block

-LRB-

-RRB-

SIMPLE

computation

primitive

-RRB-

such

that

processor

i

searches

nodes

nk

p

i

through

nk

p

-LRB-

i

-RRB-

inclusive

The

second

approach

cyclicly

assigns

the

nk

integers

to

p

processors

-LRB-

using

the

all

pardo

cyclic

-LRB-

-RRB-

SIMPLE

computation

primitive

-RRB-

Thread

Block

Partitioning

Cyclic

Partitioning

Random

Partitioning

k

k

k

k

k

k

k

k

k

k

k

k

10 175296

11 157034

15 118964

Time

Minimum

Maximum

Mean

s.d.

Table

IV

Number

of

solutions

found

by

each

thread

-LRB-

p

r

-RRB-

with

n

The

total

number

of

solutions

is

Execution

time

is

directly

proportional

to

the

maximum

of

the

number

of

solutions

found

on

each

of

the

threads

Thread

Block

Partitioning

Cyclic

Partitioning

Random

Partitioning

k

k

k

k

k

k

k

k

k

k

k

k

10 175296

11 157034

12 122763

15 118457

17 125687

19 133456

21 122569

23 108556

Time

Minimum

Maximum

Mean

s.d.

Table

V

Number

of

solutions

found

by

each

thread

-LRB-

p

r

-RRB-

with

n

The

total

number

of

solutions

is

Execution

time

is

directly

proportional

to

the

maximum

of

the

number

of

solutions

found

on

each

of

the

threads

Thread

Block

Partitioning

Cyclic

Partitioning

Random

Partitioning

k

k

k

k

k

k

k

k

k

k

k

k

11 1006695

12 876866

13 705788

14 569531

15 710901

Time

224 111 108 122

112 112 112 112

114 118 107 107

Minimum

Maximum

Mean

s.d.

Table

VI

Number

of

solutions

found

by

each

thread

-LRB-

p

r

-RRB-

with

n

The

total

number

of

solutions

is

Execution

time

is

directly

proportional

to

the

maximum

of

the

number

of

solutions

found

on

each

of

the

threads

Thread

Block

Partitioning

Cyclic

Partitioning

Random

Partitioning

k

k

k

k

k

k

k

k

k

k

k

k

10 1160280

11 1050762

12 892999

13 736363

14 569531

15 436228

16 768037

17 450243

18 751536

19 455587

20 668850

21 508096

22 529472

23 531973

24 348402

25 531095

26 202094

27 465160

29 256061

30 485257

31 195522

Time

Minimum

Maximum

Mean

s.d.

Table

VII

Number

of

solutions

found

by

each

thread

-LRB-

p

r

-RRB-

with

n

The

total

number

of

solutions

is

Execution

time

is

directly

proportional

to

the

maximum

of

the

number

of

solutions

found

on

each

of

the

threads

Both

the

block

and

cyclic

partitioning

schemes

can

be

performed

implicitly

without

the

need

for

any

explicit

inter-processor

communication

The

third

method

however

will

require

communication

but

because

it

more

evenly

distributes

the

computational

load

-LRB-

see

the

standard

deviation

of

the

number

of

solutions

found

by

each

thread

in

Tables

IV

-LCB-

VII

-RRB-

we

find

that

it

is

superior

in

performance

to

the

first

two

methods

Algorithm

n

CPUs

Time

-LRB-

s

-RRB-

p

r

Netlib

SIMPLE

SIMPLE

SIMPLE

SIMPLE

Netlib

SIMPLE

SIMPLE

SIMPLE

SIMPLE

Netlib

SIMPLE

SIMPLE

SIMPLE

SIMPLE

Table

VIII

n-Queens

Performance

Summary

The

third

approach

randomizes

the

integers

from

to

nk

and

assigns

th

of

these

to

each

processor

The

overhead

for

randomization

and

communication

is

minimal

compared

with

the

faster

completion

time

due

to

improved

load

balance

See

Tables

IV

and

V

for

a

comparison

of

these

three

algorithms

when

n

on

p

and

p

nodes

each

an

r

SMP

varying

k

from

to

Similar

results

for

n

are

given

in

Tables

VI

and

VII

Because

of

the

special

topology

inherent

in

this

search

problem

the

block

and

cyclic

partitioning

schemes

are

inferior

to

a

randomized

approach

Table

VIII

gives

the

performance

of

our

SIMPLE

algorithm

compared

to

the

standard

netlib

queens

benchmark

results

for

n

and

Because

our

algorithm

is

generalized

for

COSMOS

it

takes

slightly

longer

to

compute

on

a

single

processor

but

scales

linearly

with

the

total

number

of

processor

used

Experimental

Platform

Our

experimental

platform

consists

of

a

cluster

of

DEC

AlphaServer

2100 4/275

nodes

each

with

a

DEC

-LRB-

-RRB-

Mbps

PCI

card

connected

to

a

DEC

Gigaswitch/ATM

switch

and

using

the

MPI

-LRB-

e.g.

LAM

MPICH

or

CHIMP

c

-RRB-

and

pthreads

-LRB-

DECthreads

-RRB-

packages

Each

DEC

AlphaServer

2100 4/275

node

is

a

symmetric

multiprocessor

with

four

dual-issue

DEC

-LRB-

-RRB-

Alpha

RISC

processors

clocked

at

MHz

Each

Alpha

chip

has

two

separate

data

and

instruction

on-chip

caches

Both

on-chip

caches

are

KB

but

the

instruction

cache

is

direct

mapped

while

the

data

cache

is

two-way

set-associative

In

addition

each

CPU

has

a

MB

backup

-LRB-

-RRB-

cache

-LSB-

-RSB-

All

CPUs

communicate

via

a

system

bus

which

connects

the

four

CPU

modules

to

a

shared

memory

up

to

GB

in

size

Future

Work

The

future

research

directions

of

the

SIMPLE

project

can

be

categorized

into

two

areas

methodology

and

algorithmics

In

methodology

we

plan

an

extension

of

the

SIMPLE

kernel

to

handle

more

communication

events

Also

in

a

cluster

of

SMPs

it

is

not

always

the

case

that

nodes

are

homogeneous

in

size

memory

speed

load

or

even

architecture

We

are

currently

researching

load

sharing

inside

SIMPLE

algorithms

such

that

a

problem

initially

is

distributed

across

the

cluster

such

that

each

node

no

longer

has

th

of

the

input

but

a

portion

of

the

input

directly

proportional

to

each

node

's

current

ability

to

solve

the

task

In

addition

tasks

may

migrate

across

nodes

during

runtime

to

reflect

changing

conditions

in

the

cluster

or

to

redistribute

work

when

the

current

pool

of

nodes

shrinks

or

grows

For

the

second

area

algorithmics

we

are

examining

various

experimental

data

sets

for

benchmarking

algorithms

on

clusters

of

SMPs

and

are

implementing

high

performance

application

codes

using

the

SIMPLE

methodology

Release

Notes

Please

see

for

additional

performance

information

In

addition

all

the

code

used

in

this

paper

is

freely

available

for

interested

parties

from

our

anonymous

ftp

site

ftp

References

-LSB-

-RSB-

R.

Alasdair

A.

Bruce

J.G.

Mills

and

A.G.

Smith

CHIMP/MPI

User

Guide

Edinburgh

Parallel

Computing

Centre

The

University

of

Edinburgh

edition

June

-LSB-

-RSB-

C.

Amza

A.L.

Cox

S.

Dwarkadas

P.

Keleher

H.

Lu

R.

Rajamony

W.

Yu

and

W.

Zwaenepoel

TreadMarks

Shared

Memory

Computing

on

Networks

of

Workstations

IEEE

Computer

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

D.A.

Bader

On

the

Design

and

Analysis

of

Practical

Parallel

Algorithms

for

Combinatorial

Problems

with

Applications

to

Image

Processing

PhD

thesis

University

of

Maryland

College

Park

Department

of

Electrical

Engineering

April

-LSB-

-RSB-

D.A.

Bader

D.R.

Helman

and

J.

J?aJ?a

Practical

Parallel

Algorithms

for

Personalized

Communication

and

Integer

Sorting

and

Technical

Report

UMIACS

and

Electrical

Engineering

University

of

Maryland

College

Park

MD

November

-LSB-

-RSB-

D.A.

Bader

D.R.

Helman

and

J.

J?aJ?a

Practical

Parallel

Algorithms

for

Personalized

Communication

and

Integer

Sorting

ACM

Journal

of

Experimental

Algorithmics

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

D.

Bailey

E.

Barszcz

J.

Barton

D.

Browning

R.

Carter

L.

Dagum

R.

Fatoohi

S.

Fineberg

P.

Frederickson

T.

Lasinski

R.

Schreiber

H.

Simon

V.

Venkatakrishnan

and

S.

Weeratunga

The

NAS

Parallel

Benchmarks

Technical

Report

Numerical

Aerodynamic

Simulation

Facility

NASA

Ames

Research

Center

Moffett

Field

CA

March

-LSB-

-RSB-

W.P.

Brown

Parallel

Computation

of

Atmospheric

Propagation

Technical

report

Maui

High

Performance

Computing

Center

and

Phillips

Laboratory

Kihei

Maui

HI

-LSB-

-RSB-

D.E.

Culler

R.M.

Karp

D.A.

Patterson

A.

Sahay

K.E.

Schauser

E.

Santos

R.

Subramonian

and

T.

von

Eicken

LogP

Towards

a

Realistic

Model

of

Parallel

Computation

In

Fourth

ACM

SIGPLAN

Symposium

on

Principles

and

Practice

of

Parallel

Programming

May

-LSB-

-RSB-

Digital

Equipment

Corp.

Guide

to

DECthreads

Maynard

MA

July

-LSB-

-RSB-

Digital

Equipment

Corporation

Maynard

MA

Digital

UNIX

-LRB-

formerly

-RRB-

c

edition

July

-LSB-

-RSB-

M.

Galles

and

E.

Williams

Performance

optimizations

implementation

and

verification

of

the

SGI

Challenge

multiprocessor

Technical

report

Silicon

Graphics

Computer

Systems

Mountain

View

CA

May

pp

Available

from

ftp

paper.ps.Z

-LSB-

-RSB-

W.

Gropp

E.

Lusk

N.

Doss

and

A.

Skjellum

A

High-Performance

Portable

Implementation

of

the

MPI

Message

Passing

Interface

Standard

Technical

report

Argonne

National

Laboratory

Argonne

IL

-LSB-

-RSB-

C.

Harris

Node

Selection

for

the

IBM

SP

System

Version

IBM

Division

November

-LSB-

-RSB-

F.M.

Hayes

Design

of

the

AlphaServer

Multiprocessor

Server

Systems

Digital

Technical

Journal

-LRB-

-RRB-

-LCB-

Summer

-LSB-

-RSB-

IBM

Corporation

AIX

DCE

Base

Version

edition

October

-LSB-

-RSB-

IBM

Corporation

SP

System

Division

-LSB-

-RSB-

P.

Keleher

CVM

The

Coherent

Virtual

Machine

University

of

Maryland

edition

November

-LSB-

-RSB-

D.E.

Knuth

The

Art

of

Computer

Programming

Sorting

and

Searching

volume

AddisonWesley

Publishing

Company

Reading

MA

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Technical

report

University

of

Tennessee

Knoxville

TN

June

Version

-LSB-

-RSB-

F.

M?uller

A

Library

Implementation

of

POSIX

Threads

under

UNIX

In

Proceedings

of

the

Winter

USENIX

Conference

pages

-LCB-

San

Diego

CA

January

-LSB-

-RSB-

Netlib

Repository

for

mathematical

software

papers

and

databases

University

of

Tennessee

and

Oak

Ridge

National

Laboratory

-LSB-

-RSB-

Ohio

Supercomputer

Center

LAM

MPI

Parallel

Computing

The

Ohio

State

University

Columbus

OH

-LSB-

-RSB-

G.F.

Pfister

In

Search

of

Clusters

Prentice

Hall

Englewood

Cliffs

NJ

-LSB-

-RSB-

Portable

Applications

Standards

Committee

of

the

IEEE

Information

technology

-LCB-

Portable

Operating

System

Interface

-LRB-

POSIX

-RRB-

-LCB-

Part

System

Application

Program

Interface

-LRB-

API

-RRB-

edition

ISO/IEC

ANSI/IEEE

Std

-LSB-

-RSB-

C.

Provenzano

Proven

Pthreads

WWW

page.

-LSB-

-RSB-

W.

Saphir

A.

Woo

and

M.

Yarrow

The

NAS

Parallel

Benchmarks

Results

Report

Numerical

Aerodynamic

Simulation

Facility

NASA

Ames

Research

Center

Moffett

Field

CA

August

-LSB-

-RSB-

Sun

Microsystems

Inc.

POSIX

Threads

WWW

page.

-LSB-

-RSB-

L.G.

Valiant

A

Bridging

Model

for

Parallel

Computation

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

S.C.

Woo

M.

Ohara

E.

Torrie

J.P.

Singh

and

A.

Gupta

The

Programs

Characterization

and

Methodological

Considerations

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

-LCB-

June

-LSB-

-RSB-

D.

Yeung

J.

Kubiatowicz

and

A.

Agarwal

MGS

A

Multigrain

Shared

Memory

System

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

Philadelphia

PA

May

message passing
mpi
parallel algorithms
shared memory
smp
symmetric multiprocessors

atm networks
cluster computing
experimental parallel algorithms
parallel performance