Express

versus

PVM

A

Performance

Comparison

Ishfaq

Department

of

Computer

Science

The

Hong

Kong

University

of

Science

and

Technology

Clear

Water

Bay

Kowloon

Hong

Kong

Email

iahmad@cs.ust.hk,

Phone

-LRB-852-RRB- 2358-6980

FAX

-LRB-852-RRB- 2358-1477

Abstract

Due

to

the

increasingly

popularity

of

networked

clusters

of

workstations

and

the

need

for

portability

across

various

parallel

and

distributed

platforms

a

number

of

programming

environments

have

been

proposed

to

develop

parallel

programs

These

environments

provide

message-passing

libraries

which

can

be

used

with

C

or

Fortran

to

write

parallel

program

based

on

a

message-passing

paradigm

capable

of

running

on

various

platforms

The

performance

of

these

environments

is

a

major

concern

since

communication

has

a

direct

impact

on

the

performance

of

a

parallel

program

In

this

paper

we

evaluate

and

compare

the

performance

of

Express

and

PVM

which

are

commonly

used

on

most

commercial

and

experimental

parallel

computers

as

well

as

a

variety

of

workstation

clusters

The

performance

evaluation

is

done

through

benchmarking

the

basic

primitives

of

Express

and

PVM

on

three

platforms

an

Intel

hypercube

parallel

computer

a

cluster

of

SUN

workstations

connected

by

an

Ethernet

and

a

cluster

of

HP

workstations

connected

by

an

FDDI

ring

The

performance

measures

include

the

timings

of

various

communication

primitives

coded

with

Express

and

PVM

The

results

of

Express

and

PVM

on

the

are

also

compared

with

the

equivalent

implementations

using

the

NX

message-passing

library

of

the

To

make

a

comparison

from

applications

point

of

view

we

have

also

benchmarked

a

suite

of

various

programsincluding

three

different

versions

of

Gaussian

elimination

and

the

N-body

problem

The

performance

results

also

enable

us

to

compare

three

different

hardware

platforms

While

it

is

not

the

purpose

of

this

study

to

make

a

qualitative

judgement

on

Express

and

PVM

we

do

highlight

their

distinctive

features

and

provide

an

overview

of

their

programming

styles

This

research

was

supported

by

a

grant

from

the

Hong

Kong

Research

Grants

Council

under

contract

number

HKUST

Introduction

In

the

recent

years

we

have

witnessed

an

unprecedented

growth

of

parallel

computing

hardware

platforms

At

the

same

time

advancements

in

the

design

of

processor

architecture

and

communication

mediums

have

resulted

in

the

emergence

of

fast

workstations

connected

by

high-speed

communication

networks

-LSB-

-RSB-

-LSB-

-RSB-

The

cumulative

performance

of

these

workstations

clusters

is

approaching

the

speeds

of

some

of

the

contemporary

massively

parallel

processors

and

traditional

supercomputers

Recent

studies

have

shown

that

clusters

of

workstations

have

the

potential

of

solving

very

large-scale

problems

-LSB-

-RSB-

Portability

of

software

across

these

platforms

is

highly

desirable

so

that

the

code

developed

for

one

platform

does

not

have

to

be

rewritten

or

modified

for

the

other

A

number

of

software

environments

providing

some

kind

of

portability

and

facilities

to

develop

parallel

programs

have

been

proposed

These

environments

can

simulate

a

cluster

of

workstations

as

a

virtual

parallel

computer

and

can

perform

communication

across

multiple

homogeneous

and

heterogeneous

parallel

machines

This

paper

compares

two

parallel

programming

environments

The

first

is

Express

which

is

a

commercial

product

from

Parasoft

Corporation

-LSB-

-RSB-

The

second

is

PVM

which

is

available

in

the

public

domain

-LSB-

-RSB-

We

call

them

environments

because

they

are

neither

operating

systems

nor

languages

rather

they

allow

the

programs

to

be

written

using

standard

C

or

Fortran

As

elaborated

in

-LSB-

-RSB-

and

-LSB-

-RSB-

both

Express

and

PVM

are

more

than

just

message-passing

libraries

since

they

provide

a

broad

set

of

tools

and

utilities

that

are

vital

for

full

exploitation

of

computing

and

networking

resources

A

number

of

other

environments

have

also

been

proposed

including

Linda

-LSB-

-RSB-

-LSB-

-RSB-

PICL

-LSB-

-RSB-

and

Zipcode

-LSB-

-RSB-

A

detailed

survey

of

software

environments

for

networked

systems

can

be

found

in

-LSB-

-RSB-

while

an

overview

of

recent

developments

for

message-passing

techniques

for

both

parallel

and

distributed

systems

can

be

found

in

-LSB-

-RSB-

We

evaluate

the

performance

of

Express

and

PVM

since

they

are

very

common

on

various

platforms

and

also

because

they

programming

models

are

quite

A

key

measure

of

the

usefulness

of

programming

environments

like

Express

and

PVM

is

the

speed

of

the

basic

primitives

used

frequently

in

parallel

programs

We

have

evaluated

the

performance

of

some

of

the

basic

primitives

of

Express

and

PVM

running

on

an

parallel

computer

a

cluster

of

HP

workstations

connected

by

an

FDDI

network

and

a

cluster

of

IPX

workstations

connected

by

an

Ethernet

The

results

from

this

evaluation

can

be

useful

in

a

number

of

ways

The

amount

of

extra

overhead

incurred

due

to

portability

can

be

determined

when

compared

to

the

host

operating

system

such

as

NX

By

using

the

timings

of

various

communication

primitives

algorithm

developers

and

MPI

-LSB-

-RSB-

was

not

fully

available

at

the

time

of

this

study

parallel

compiler

writers

can

estimate

the

performance

of

various

libraries

-LSB-

-RSB-

The

timings

of

various

communication

functions

on

different

platforms

can

help

in

making

better

problem

partitioning

and

scheduling

decisions

The

results

enable

us

to

compare

the

performance

of

the

parallel

computer

and

workstation

clusters

using

different

networks

and

shed

some

light

on

the

trade-off

between

their

performance

and

cost

These

results

also

indicate

if

there

are

any

communication

bottlenecks

that

require

improvements

in

performance

The

rest

of

this

paper

is

organized

as

follows

In

the

next

section

we

provide

some

background

of

Express

and

its

utilities

Section

provides

an

overview

of

PVM

and

its

utilities

Section

describes

the

hardware

platforms

used

in

this

study

Section

contains

the

performance

results

of

various

communication

primitives

Section

presents

the

timings

of

the

application

benchmark

suite

and

the

last

section

concludes

this

paper

Overview

of

Express

Express

allows

to

write

parallel

programs

on

a

variety

of

parallel

machines

including

nCUBE

Intel

and

hypercubes

Intel

Paragon

and

transputer

arrays

-LSB-

-RSB-

-LSB-

-RSB-

The

network

version

of

Express

allows

a

network

of

workstations

to

be

used

as

a

virtual

parallel

machine

Workstations

that

Express

can

run

on

include

DEC

HP

SGI

and

Sun

The

history

of

Express

can

be

traced

back

to

the

Caltech/JPL

machines

developed

in

the

early

eighties

-LSB-

-RSB-

At

that

time

the

simplest

model

of

a

parallel

or

distributed

computation

was

the

one

in

which

a

master

process

takes

the

responsibility

of

creating

a

number

of

worker

processes

which

perform

the

computations

required

to

produce

the

overall

desired

output

In

Express

this

style

of

programming

is

called

the

host-node

programming

model

where

the

master

is

a

host

while

the

worker

is

a

node

In

order

to

avoid

writing

a

host

program

a

host-free

programming

model

called

Cubix

was

also

developed

which

provided

an

I/O

system

for

opening

reading

and

writing

files

and

interfacing

with

the

user

Later

an

operating

system

known

as

Multitasking

object-oriented

operating

system

-LRB-

MOOSE

-RRB-

was

built

to

support

multi-tasking

remote

task

creation

scheduling

and

a

number

of

other

housekeeping

functions

Moreover

a

system

called

crystal

router

was

developed

to

support

more

efficient

concurrent

communication

-LSB-

-RSB-

Eventually

the

combined

research

efforts

at

Caltech

resulted

in

an

integrated

software

package

now

called

Express

As

shown

in

Figure

there

are

three

implementation

layers

of

Express

The

lowest

level

consists

of

utilities

for

controlling

the

hardware

such

as

allocation

of

processors

loading

of

programs

etc.

The

medium

level

provides

support

for

problem

partitioning

It

also

allows

communication

among

nodes

and

between

a

control

processor

and

nodes

The

highest

level

includes

facilities

for

node

programs

to

perform

I/O

and

utilities

for

access

to

the

host

operating

system

Since

each

level

is

logically

distinct

and

built

only

on

those

below

it

Express

is

portable

to

a

variety

of

systems

using

a

top-down

approach

A

summary

of

the

utilities

provided

by

Express

is

illustrated

in

Figure

The

communication

utilities

include

blocking

and

non-blocking

communication

among

nodes

exchange

broadcast

and

collective

communication

such

as

reading

and

writing

a

vector

The

global

communication

includes

concatenation

reduction

operations

synchronization

etc.

In

addition

Express

provides

processor

control

domain

decomposition

tools

parallel

I/O

graphics

and

debugging

and

performance

analysis

tools

Overview

of

PVM

PVM

has

been

developed

by

researchers

at

Emory

University

Oak

Ridge

National

Laboratory

-LRB-

ORNL

-RRB-

University

of

Tennessee

Carnegie

Mellon

University

and

Pittsburgh

Supercomputer

Center

-LSB-

-RSB-

It

permits

a

network

of

heterogeneous

computers

to

be

used

as

a

single

parallel

computer

PVM

has

evolved

through

many

versions

The

initial

versions

of

PVM

used

TCP/IP

sockets

to

implement

all

communication

Consequently

PVM

was

limited

to

workstation

environments

such

as

and

SPARCstation

PA-RISC

etc.

However

recent

versions

of

PVM

such

as

the

one

reported

in

this

paper

have

included

implementation

on

various

parallel

machines

Heterogeneous

network-based

computing

-LSB-

-RSB-

refers

to

general

purpose

concurrent

computing

where

ahardware

platform

consists

of

a

collection

of

computer

systems

of

varying

architectures

interconnected

by

one

or

more

network

types

such

as

FDDI

applications

are

viewed

as

comprising

several

sub-algorithms

each

of

which

is

potentially

Figure

Three

layers

of

Express

A

complete

I/O

system

Problem

decomposition

Control

of

hardware

Highest

Level

Medium

Level

Lowest

Level

allowing

parallel

node

programs

access

to

the

operating

system

facilities

of

the

control

processor

done

by

providing

utilities

for

communication

among

nodes

and

between

nodes

and

control

processor

such

as

allocating

processors

loading

programs

and

message

passing

between

arbitrary

nodes

different

in

terms

of

its

most

appropriate

programming

model

implementation

language

and

resource

requirements

PVM?s

portability

is

similar

to

Express

The

PVM

computing

model

is

illustrated

in

Figure

Under

PVM

a

user

defined

collection

of

serial

parallel

and

vector

computers

appear

as

one

large

distributed-memory

virtual

computer

PVM

supplies

functions

to

automatically

start

up

tasks

on

the

virtual

machine

and

allows

tasks

to

communicate

and

synchronize

with

each

other

A

task

is

defined

as

a

unit

of

computation

in

PVM

analogous

to

a

UNIX

process

Applications

written

in

Fortran

or

C

can

be

parallelized

by

using

message-passing

constructs

common

to

most

distributed-memory

computers

By

sending

and

receiving

messages

multiple

tasks

of

an

application

can

cooperate

to

solve

a

problem

in

parallel

PVM

supports

heterogeneity

at

an

application

machine

and

network

level

and

can

handle

data

conversion

that

may

be

required

-LSB-

-RSB-

In

other

words

PVM

allows

application

tasks

to

exploit

the

architecture

best

suited

to

their

solutions

Moreover

PVM

allows

a

virtual

machine

to

be

interconnected

by

a

variety

of

different

networks

PVM

provides

routines

of

packing

and

sending

messages

between

tasks

The

model

assumes

that

any

task

can

send

a

message

to

any

other

task

The

PVM

communication

model

provides

synchronous

and

asynchronous

blocking

send

and

receive

multicast

to

a

set

of

tasks

and

broadcast

to

a

user

defined

group

of

tasks

Since

message

buffers

are

allocated

dynamically

the

maximum

size

of

a

message

that

can

be

sent

or

received

is

limited

by

the

amount

of

available

memory

on

a

given

host

PVM

supplies

routines

that

enable

a

user

process

to

become

a

PVM

task

and

to

become

a

normal

process

again

These

routines

perform

functions

such

as

adding

and

deleting

hosts

from

the

virtual

machine

and

starting

and

terminating

PVM

tasks

PVM

also

provides

fault

Message-Passing

Synchronous

and

Asychronous

Comm

Problem

Decompostion

Partitioning

and

Load

Balancing

Cubix

Model

Concurrent

and

host-free

I/O

Collective

Communication

etc.

Broadcast

Concatenation

Graphic

Plotix

Library

Control

Allocating

nodes

Multitasking

Topology

Setup

Performance

Monitoring

and

Profiling

Tools

Debugging

NDB

Source

Level

Debugger

Express

Unix

C

and

Fortran

Figure

The

Express

utilities

tolerance

If

a

host

fails

PVM

can

detect

this

and

delete

the

host

from

the

virtual

machine

The

status

of

host

can

be

requested

by

an

application

and

if

required

a

replacement

host

can

be

added

by

the

application

Test

Environment

Our

test

environment

included

three

platforms

an

Intel

hypercube

an

Ethernet

based

network

of

SUN

workstations

and

an

FDDI

based

ring

of

HP

workstations

All

tests

programs

were

written

in

Fortran

The

hardware

platforms

are

briefly

described

below

Our

first

platform

was

the

Intel

parallel

computer

at

Caltech?s

Concurrent

Supercomputing

Center

This

system

is

based

on

hypercube

interconnection

network

topology

and

it

can

scale

up

to

nodes

It

is

controlled

from

a

host

computer

called

System

Resource

Manager

or

SRM

which

runs

System

V

UNIX

-LSB-

-RSB-

An

node

is

based

on

the

RISC

microprocessor

-LSB-

-RSB-

A

node

also

includes

a

hardware

communication

module

called

DCM

-LRB-

direct

connect

module

-RRB-

that

connects

the

node

to

the

interconnection

network

-LSB-

-RSB-

Each

DCM

router

can

support

up

to

eight

channels

which

are

bit-serial

and

full

duplex

Table

I

gives

a

summary

of

the

configuration

of

the

In

our

experiments

for

small

data

sizes

we

used

repetitions

for

each

primitive

to

improve

the

accuracy

while

for

large

messages

the

number

of

repetitions

was

varied

from

to

Some

of

our

results

using

the

NX

library

concur

with

those

of

earlier

studies

-LSB-

-RSB-

-LSB-

-RSB-

For

performance

evaluation

of

workstation

clusters

we

used

two

platforms

The

first

was

a

cluster

of

homogeneous

SUN

SPARC

IPX

workstations

connected

by

Ethernet

segments

Component

A

Component

B

Component

C

Component

D

Input

Partitioning

Output

Display

Synchronization

Data

Communication

Instance

Instance

Instance

Instance

Instance

Instance

Instance

Instance

n

Instance

Instance

n

Instance

Figure

The

PVM

computing

model

The

second

was

a

cluster

of

homogeneous

HP

workstations

connected

by

an

FDDI

Table

II

gives

a

summary

of

the

configuration

of

IPX

and

HP

clusters

The

experiments

on

these

clusters

were

conducted

at

night

when

the

systems

were

idle

In

each

experiment

for

small

data

sizes

we

used

about

repetitions

for

each

primitive

For

larger

messages

the

number

of

repetitions

used

was

In

addition

each

experiment

was

repeated

times

and

the

average

was

taken

across

repetitions

within

an

experiment

and

across

experiments

Communication

Performance

The

communication

tests

include

one-to-one

communication

exchange

broadcast

global

reduction

operations

ring

communication

and

complete

exchange

These

primitives

were

tested

by

writing

programs

using

Express

PVM

and

NX

The

timing

of

a

primitive

was

taken

by

calling

that

primitive

a

number

of

times

and

using

a

clock

before

and

after

that

loop

The

elapsed

time

was

then

divided

by

the

number

of

repetitions

During

our

experiments

we

found

that

the

differences

in

the

performance

of

Express

and

PVM

had

considerable

variations

on

the

three

hardware

platforms

In

addition

we

also

discovered

that

the

size

of

the

communication

data

had

a

great

impact

on

the

relative

performance

of

all

three

software

environments

Therefore

for

each

primitive

we

have

divided

the

results

into

two

parts

The

first

part

presents

the

timing

results

on

the

and

the

second

part

presents

the

results

on

the

HP

and

IPX

clusters

Each

part

is

presented

with

results

for

small

-LRB-

to

bytes

-RRB-

large

-LRB-

to

bytes

-RRB-

and

very

large

-LRB-

to

bytes

-RRB-

messages

One-to-One

Communication

For

measuring

the

communication

speed

between

two

nodes

we

performed

the

standard

echo

test

In

the

echo

test

the

communication

time

between

sending

and

receiving

nodes

is

measured

by

starting

a

clock

at

the

sending

node

and

then

invoking

the

send

and

receive

rou

Number

of

nodes

Node

CPU

Clock

Frequency

MHz

Main

Memory/Node

Mbytes

Express

Version

PVM

Version

iPSC

Node

O/S

-LRB-

rel

-RRB-

Table

I

Configurations

of

the

used

in

the

experiments

Table

II

Configurations

of

the

SUN

IPX

and

HP

clusters

Network

SUN-IPX/Ethernet

HP/FDDI-Ring

Number

of

Workstations

System

Model

SPARC

IPX

Main

Memory

Mbytes

OS

Name

SUN

OS

HP-UX

Express

Version

PVM

Version

tines

to

send

out

a

message

and

wait

for

a

reply

On

the

destination

node

receive

and

send

routines

are

used

to

echo

this

message

back

to

the

sending

node

This

process

is

repeated

N

times

and

the

elapsed

time

is

divided

by

to

get

the

timing

for

one-way

communication

The

results

of

one-to-one

communication

using

NX

Express

and

PVM

on

the

are

shown

in

Figure

This

figure

indicates

that

the

timings

of

Express

and

NX

are

relatively

close

for

small

medium

and

large

data

sizes

PVM

on

the

other

hand

is

about

times

slower

than

Express

and

NX

for

small

messages

But

as

the

message

size

increases

the

difference

between

the

performance

of

PVM

and

Express

decreases

For

example

as

shown

in

Figure

-LRB-

b

-RRB-

the

difference

between

PVM

and

Express

decreases

steadily

as

the

message

size

is

increased

from

to

bytes

For

very

large

messages

as

indicated

in

Figure

-LRB-

c

-RRB-

the

timings

of

PVM

and

NX

are

about

the

same

The

results

for

the

IPX

and

HP

cluster

are

given

in

Figure

From

this

figure

we

can

ob

Figure

Times

for

one-to-one

communication

using

NX

Express

and

PVM

on

the

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

-LRB-

c

-RRB-

Very

large

messages

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

One-to-One

Communication

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

One-to-One

Communication

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

One-to-One

Communication

serve

that

on

the

HP

cluster

PVM

is

times

slower

than

Express

But

this

difference

reduces

to

times

when

the

messages

get

larger

A

similar

trend

can

be

observed

using

IPX

workstations

with

the

Ethernet

Notice

that

Express

timings

on

the

cluster

of

HP

workstations

and

are

comparable

but

PVM

on

the

HP

cluster

is

slower

than

that

on

the

Compared

to

the

HP

cluster

Express

and

PVM

are

about

to

times

slower

when

run

on

the

IPX

cluster

Some

timing

values

for

various

message

sizes

are

provided

in

Table

III

The

Exchange

Operation

The

exchange

operation

simultaneously

sends

a

message

to

a

node

and

receives

a

reply

The

advantage

of

this

function

is

that

data

can

be

read

from

and

sent

to

the

same

or

different

nodes

in

one

step

Another

advantage

is

that

data

transmission

in

reading

and

writing

can

be

Figure

Times

for

one-to-one

communication

using

Express

and

PVM

on

the

HP

and

IPX

clusters

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

-LRB-

c

-RRB-

Very

large

messages

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

One-to-One

Communication

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

One-to-One

Communication

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

One-to-One

Communication

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

overlapped

Moreover

the

user

is

free

from

worrying

about

which

node

should

read

first

and

which

should

write

Since

PVM

does

not

provide

the

exchange

operation

we

implemented

it

by

using

a

pvmfsend

and

a

pvmfrecv

routine

in

one

node

and

correspondingly

a

pvmfrecv

and

a

pvmfsend

routine

in

the

other

node

As

shown

in

Figure

on

the

PVM

is

about

two

times

slower

than

Express

However

with

a

larger

message

size

the

difference

between

the

timings

of

PVM

and

Express

becomes

smaller

From

Figure

we

can

observe

the

overall

trend

in

the

performance

of

PVM

and

Express

on

the

HP

and

IPX

clusters

is

similar

to

that

of

Figure

For

small

message

sizes

Express

running

on

the

IPX

cluster

is

to

times

faster

than

PVM

On

the

HP

cluster

Express

is

about

times

faster

than

PVM

But

with

larger

message

sizes

Express

is

only

about

times

faster

than

PVM

on

both

clusters

Express

on

the

HP

cluster

is

about

times

slower

than

Express

on

the

Similarly

PVM

on

the

HP

cluster

is

times

slower

than

PVM

on

the

The

Broadcast

Operation

Broadcast

is

performed

to

send

the

same

data

to

more

than

one

node

It

can

be

performed

between

host

and

nodes

or

from

one

node

to

multiple

nodes

It

is

one

of

the

frequently

used

primitive

and

is

important

for

a

performance

comparison

Both

Express

and

PVM

provide

routines

for

broadcast

In

NX

on

the

other

hand

broadcast

is

done

using

csend

and

crecv

routines

Message

can

also

be

sent

to

a

subcube

composed

of

a

set

of

nodes

surrounding

the

source

node

Express

and

PVM

have

an

extra

advantage

of

specifying

arbitrarily

any

node

-LRB-

s

-RRB-

that

should

receive

the

broadcast

message

The

timings

for

the

broadcast

operation

on

the

using

processors

are

shown

in

Figure

From

this

figure

we

observe

that

the

timings

of

broadcast

operation

of

Express

and

NX

are

very

close

while

the

broadcast

operation

of

PVM

is

much

slower

as

compared

to

both

NX

and

Express

As

shown

in

Figure

the

broadcast

operation

of

PVM

on

the

cluster

of

IPX

workstations

is

about

times

slower

than

Express

when

the

messages

size

is

small

But

for

large

messages

PVM

is

about

times

slower

than

Express

On

the

HP

cluster

for

small

message

size

and

nodes

PVM

is

about

times

slower

than

Express

For

very

large

messages

the

timings

for

PVM

are

comparable

to

those

of

Express

For

very

large

messages

PVM

even

outperforms

Express

Comparing

different

hardware

platforms

PVM

on

the

is

times

slower

than

on

Table

III

Times

required

for

one-to-one

communication

with

messages

of

different

sizes

-LRB-

s

-RRB-

m

Bytes

68 83 736 189

677 643 2225

74 92 741 189

858 647 2380

100 115 134 841

271 1252 754 2563

200 255 270 852

303 1342 820 2774

1000 566 584 929

598 1350 1270 3211

5941 5958 5815 4570

FDDI

Ethernet

NX

Express

PVM

Express

PVM

Express

PVM

the

HP

cluster

for

small

messages

But

this

difference

reduces

to

about

times

for

large

messages

Compared

to

the

Express

is

about

times

slower

on

the

IPX

cluster

and

times

slower

on

HP

cluster

Global

Communication

Global

communication

operations

are

frequently

required

in

parallel

programs

In

such

global

operations

all

the

nodes

participate

in

the

same

operation

The

reduction

operations

are

examples

that

require

global

communication

A

reduction

operation

takes

as

input

a

value

in

each

processor

and

outputs

a

single

value

in

every

processor

There

are

various

reduction

operations

such

as

add

prod

max

min

and

or

xor

etc.

While

NX

provides

different

routines

for

each

of

these

operations

Express

has

a

unified

function

for

performing

reduction

operations

But

since

routines

for

these

operations

are

not

provided

by

PVM

we

implemented

them

Ex

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Exchange

Operation

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Exchange

Operation

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Exchange

Operation

Figure

Times

for

the

exchange

operation

between

nearest

neighbors

using

NX

Express

and

PVM

on

the

-LRB-

processors

-RRB-

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

press

has

an

extra

facility

of

specifying

which

nodes

should

participate

in

the

global

operation

In

this

paper

only

the

results

of

the

sum

operation

are

given

Table

IV

and

Table

V

include

times

to

perform

the

global

sum

on

and

words

The

results

for

the

are

repeated

for

and

processors

It

can

be

seen

that

both

Express

and

NX

are

much

faster

than

PVM

The

global

operations

of

PVM

do

not

seem

to

scale

very

well

with

an

increase

in

the

number

of

processors

Moreover

compared

to

these

operations

are

even

slower

on

the

HP

and

IPX

clusters

We

also

observed

similar

results

-LRB-

not

reported

here

-RRB-

for

some

other

global

reduction

operations

such

as

multiply

and

logical

AND

The

Ring

Communication

Test

In

the

ring

communication

test

each

node

takes

a

message

and

circulates

it

through

each

Figure

Times

for

the

exchange

operation

using

Express

and

PVM

on

the

HP

and

IPX

clusters

-LRB-

workstations

-RRB-

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Exchange

Operation

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Exchange

Operation

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Exchange

Operation

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

of

the

other

nodes

in

the

network

This

communication

pattern

is

generated

in

a

straight

forward

manner

Each

node

finds

its

forward

and

backward

neighbor

with

whom

it

communicates

Each

node

then

writes

a

message

to

its

forward

neighbor

and

reads

from

the

back

Figure

Times

for

the

broadcast

operation

using

NX

Express

and

PVM

on

the

-LRB-

processors

-RRB-

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Broadcast

-LRB-

processors

-RRB-

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Broadcast

-LRB-

processors

-RRB-

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Broadcast

-LRB-

processors

-RRB-

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

Table

V

Times

for

global

sum

-LRB-

milliseconds

-RRB-

on

the

workstation

clusters

Exp-HP

PVM-HP

ExpIPX

PVM-IP

Words

Processors

ward

neighbor

This

is

repeated

times

where

N

is

the

total

number

of

nodes

In

general

the

efficiency

of

this

communication

pattern

depends

on

the

underlying

network

topology

Determination

of

the

forward

and

backward

neighbors

can

be

done

using

Express?s

grid

utilities

which

can

convert

a

physical

topology

to

a

virtual

topology

On

the

hypercube

forward

and

backward

neighbors

can

be

easily

found

using

gray

codes

for

implementing

the

ring

pattern

using

NX

primitives

For

a

bus

based

system

using

PVM

this

determination

can

not

be

done

meaningfully

Hence

we

can

arbitrarily

determine

the

forward

and

backward

nodes

for

each

node

such

that

the

message

goes

through

each

workstation

only

once

As

can

be

seen

from

Figure

on

the

PVM

is

to

times

slower

than

Express

Figure

Times

for

the

broadcast

operation

using

Express

and

PVM

on

the

HP

and

IPX

clusters

-LRB-

workstations

-RRB-

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Broadcast

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Broadcast

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Broadcast

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

and

NX

primitives

for

and

nodes

When

the

message

size

is

large

the

timings

for

PVM

Express

and

NX

are

almost

identical

On

the

HP

and

IPX

clusters

PVM

is

slower

than

Express

-LRB-

see

Figure

-RRB-

Compared

to

the

the

ring

test

on

the

HP

cluster

is

about

times

slower

for

small

messages

For

large

messages

timings

for

HP

cluster

are

comparable

to

those

on

the

and

sometimes

even

smaller

The

ring

test

on

the

HP

cluster

is

times

faster

than

on

the

IPX

cluster

for

small

message

size

The

Complete

Exchange

In

the

complete

exchange

communication

pattern

also

known

as

all-to-all

personalized

communication

-LSB-

-RSB-

each

of

the

N

nodes

sends

a

different

block

of

data

to

each

of

the

remaining

nodes

This

communication

pattern

is

equivalent

to

a

complete

graph

It

is

used

in

a

number

of

algorithms

including

matrix

transpose

matrix-vector

multiply

FFTs

distributed

table

look-ups

etc.

The

time

required

to

carry

out

the

complete

exchange

operation

is

an

important

measure

of

the

power

of

a

distributed-memory

parallel

computer

system

since

it

is

the

densest

communication

requirement

that

can

be

implemented

on

a

network

-LSB-

-RSB-

The

timings

for

the

complete

exchange

operation

using

nodes

with

PVM

Express

and

NX

on

the

are

given

in

Figure

Here

PVM

is

about

times

slower

than

Express

for

small

messages

For

medium

and

large

messages

this

difference

reduces

considerably

For

larger

message

sizes

timings

for

NX

primitives

and

Express

are

about

the

same

while

PVM

is

about

times

slower

than

NX

and

Express

On

the

cluster

of

HP

workstations

-LRB-

Figure

-RRB-

PVM

is

times

slower

than

Express

However

for

large

messages

PVM

is

only

times

slower

Comparing

different

platforms

for

small

messages

Express

and

PVM

on

the

are

times

faster

than

on

the

HP

cluster

For

large

messages

the

timings

difference

reduces

to

a

small

value

Comparing

the

clusters

of

HP

and

Table

IV

Times

for

global

sum

-LRB-

milliseconds

-RRB-

on

the

ords

Processors

IPX

workstations

the

complete

exchange

operation

with

small

messages

is

about

times

faster

on

the

HP

cluster

as

compared

to

the

IPX

cluster

For

larger

message

size

this

difference

increases

to

times

Evaluation

with

an

Application

Benchmark

Suite

To

compare

the

performance

of

Express

PVM

and

NX

with

real

applications

we

used

an

application

benchmark

suite

that

included

three

different

versions

of

Gaussian

elimination

and

the

N-body

problem

These

applications

were

coded

using

Fortran

with

Express

PVM

and

NX

primitives

The

execution

times

were

obtained

with

and

workstations

on

the

IPX

cluster

while

and

workstations

were

used

on

the

HP

cluster

On

the

we

used

and

nodes

The

execution

time

of

an

application

was

measured

by

taking

the

Figure

Times

for

the

ring

test

on

the

using

NX

Express

and

PVM

-LRB-

processors

-RRB-

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Ring

Communication

-LRB-

processors

-RRB-

-LRB-

n

-RRB-

-LRB-

n

-RRB-

-LRB-

n

-RRB-

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Ring

Communication

-LRB-

processors

-RRB-

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Ring

Communication

-LRB-

processors

-RRB-

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

average

of

all

the

nodes

The

results

are

given

in

the

following

sections

Gaussian

Elimination

-LRB-

Row-block

Partitioned

-RRB-

The

three

versions

of

Gaussian

elimination

for

solving

linear

equations

are

based

on

the

partial

pivoting

technique

but

with

different

data

partitioning

strategies

As

a

result

the

algorithms

used

in

these

versions

are

quite

different

The

first

version

is

based

on

row-block

partitioning

that

is

an

equal

number

of

contiguous

rows

of

the

coefficient

matrix

are

assigned

to

each

processor

The

program

consists

of

three

routines

one

for

generating

the

random

data

and

other

two

are

for

performing

Gaussian

elimination

and

backward

substitution

Only

the

Gaussian

elimination

routine

was

timed

The

execution

times

of

Gaussian

elimination

with

row-block

partitioning

using

Express

PVM

and

NX

primitives

on

different

platforms

are

included

in

Table

VI

to

Table

VIII

These

ta

Figure

Times

for

the

ring

test

using

Express

and

PVM

on

the

HP

and

IPX

clusters

-LRB-

workstations

-RRB-

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Ring

Communication

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Ring

Communication

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Ring

Communication

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

bles

also

include

the

times

for

a

serial

execution

of

Gaussian

elimination

using

one

processor

On

the

the

serial

version

was

implemented

using

NX

only

From

these

figures

we

can

make

the

following

observations

On

the

the

Express

version

yields

speedup

with

the

number

of

processors

varying

from

to

The

speedup

is

better

with

a

larger

matrix

size

However

the

execution

times

start

to

increase

with

processors

On

the

other

hand

the

NX

version

still

yields

some

speedup

with

processors

Using

the

PVM

version

speedup

is

observed

with

the

number

of

processors

varying

from

to

However

the

execution

times

start

increasing

with

or

more

processors

From

Table

VI

one

can

observe

that

for

small

number

of

processors

on

the

the

difference

between

the

performance

of

Express

and

NX

is

insignificant

Express

performs

poorly

with

a

large

number

of

processors

The

performance

of

PVM

is

even

worse

for

the

large

number

of

processors

The

main

reason

is

that

the

algorithm

Figure

Times

for

the

complete

exchange

test

using

NX

Express

and

PVM

on

the

-LRB-

processors

-RRB-

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Complete

Exchange

-LRB-

processors

-RRB-

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Complete

Exchange

-LRB-

processors

-RRB-

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Complete

Exchange

-LRB-

processors

-RRB-

-LRB-

c

-RRB-

Very

large

messages

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

used

makes

an

extensive

use

of

broadcast

operations

for

sending

the

pivot

row

to

other

processors

Moreover

no

optimizations

are

made

in

communication

calls

to

exploit

the

hypercube

topology

As

a

result

the

algorithm

uses

a

number

of

global

operations

which

are

quite

slow

with

Express

and

particularly

with

PVM

The

relative

performance

of

this

version

of

Gaussian

elimination

using

Express

and

NX

is

also

indicated

in

Figure

-LRB-

a

-RRB-

This

figure

includes

the

plots

of

the

ratios

of

execution

times

of

Express

to

NX

The

ratios

of

execution

times

of

PVM

to

Express

are

plotted

Figure

-LRB-

a

-RRB-

On

both

the

IPX

and

HP

clusters

we

make

the

following

observations

The

execution

times

for

Express

and

PVM

indicate

speedup

with

an

increasing

number

of

processors

using

a

reasonably

large

matrix

size

The

timings

of

Express

version

are

again

better

than

those

of

the

PVM

Figure

Times

for

the

complete

exchange

test

using

Express

and

PVM

on

the

HP

and

IPX

clusters

-LRB-

workstations

-RRB-

-LRB-

a

-RRB-

Small

messages

-LRB-

b

-RRB-

Large

messages

-LRB-

c

-RRB-

Very

large

messages

60 80 100 120

140 160 180 200

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Complete

Exchange

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

200 300 400 500

600 700 800 900

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Complete

Exchange

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

1000 3000 5000 7000

Time

-LRB-

ms

-RRB-

Message

Size

-LRB-

bytes

-RRB-

Complete

Exchange

-LRB-

processors

-RRB-

PVM-IPX

EXP-IPX

PVM-HPPA

EXP-HPPA

version

Gaussian

Elimination

-LRB-

Column-block

Partitioned

-RRB-

In

this

version

of

Gaussian

elimination

the

data

is

partitioned

across

processors

in

terms

of

blocks

of

columns

The

execution

times

for

this

version

are

shown

in

Table

IX

to

Table

XI

These

tables

indicate

that

on

the

Express

and

PVM

perform

well

for

large

matrices

when

the

number

of

processors

is

between

and

but

perform

poorly

for

processors

Furthermore

the

execution

times

of

the

column-block

partitioned

versions

are

in

general

larger

than

those

of

the

row-block

partitioned

versions

This

is

because

the

row-block

partitioned

algorithm

the

determination

of

the

pivoting

row

is

done

in

parallel

On

the

other

hand

the

column-block

partitioned

algorithm

performs

this

step

serially

When

the

number

of

processors

is

large

the

PVM-based

column-block

partitioned

version

takes

less

time

than

the

PVM-based

row-partitioned

version

because

the

broadcast

operation

is

used

less

frequently

in

the

former

version

The

inefficient

broadcast

operation

of

PVM

therefore

is

the

main

cause

of

performance

degradation

when

using

a

large

number

of

processors

NX

Express

PVM

NX

ExpressPVM

NX

Express

PVM

NX

ExpressPVM

NX

ExpressPVM

PE

PE

PE

Table

VI

Timings

-LRB-

seconds

-RRB-

for

row-block

partitioned

Gaussian

elimination

using

NX

Express

and

PVM

on

the

PE

PE

Matrix

Size

PE

NX

Express

PVM

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

VII

Timings

-LRB-

seconds

-RRB-

for

row-block

partitioned

Gaussian

elimination

using

Express

and

PVM

on

the

IPX

cluster

connected

by

Ethernet

PE

PE

Matrix

Size

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

VIII

Timings

-LRB-

seconds

-RRB-

for

row-block

partitioned

Gaussian

elimination

using

Express

and

PVM

on

the

HP

cluster

connected

by

FDDI

PE

Matrix

Size

On

the

IPX

cluster

of

workstations

it

is

interesting

to

see

the

results

of

Express

showing

speedup

if

the

number

of

workstations

is

increased

even

up

to

PVM

however

is

not

shown

to

perform

well

beyond

workstations

Both

Express

and

PVM

seem

to

perform

better

on

the

HP

cluster

than

on

the

with

or

workstations

or

processors

The

relative

performance

of

Express

to

NX

and

PVM

to

Express

is

indicated

in

Figure

-LRB-

b

-RRB-

and

Figure

-LRB-

b

-RRB-

respectively

Gaussian

Elimination

-LRB-

Column-Scattered

Partitioned

-RRB-

In

this

version

of

Gaussian

elimination

the

data

is

partitioned

using

a

cyclic

distribution

of

the

columns

of

the

coefficient

matrix

The

execution

times

of

this

version

using

Express

PVM

and

NX

primitives

on

different

platforms

are

shown

in

Table

XII

to

Table

XIV

The

results

indicate

that

in

general

this

algorithm

is

better

than

the

column-block

partitioned

version

but

is

comparable

to

the

row-block

partitioned

version

This

is

because

the

column-scatter

parti

NX

ExpressPVM

NX

Express

PVM

NX

ExpressPVM

NX

Express

PVM

NX

Express

PVM

PE

PE

PE

Table

IX

Timings

-LRB-

seconds

-RRB-

for

column-block

partitioned

Gaussian

elimination

using

NX

Express

and

PVM

on

the

PE

PE

Matrix

Size

Express

PVM

Express

PVM

Express

PVM

Express

PVM

PE

Table

X

Timings

-LRB-

seconds

-RRB-

for

column-block

partitioned

Gaussian

elimination

using

Express

and

PVM

on

the

IPX

cluster

connected

by

Ethernet

PE

PE

Matrix

Size

PE

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

XI

Timings

-LRB-

seconds

-RRB-

for

column-block

partitioned

Gaussian

elimination

using

Express

and

PVM

on

the

HP

cluster

connected

by

FDDI

PE

Matrix

Size

tioned

can

balance

load

well

which

results

in

reduction

of

processor

waiting

times

However

it

requires

excessive

exchanges

of

messages

which

can

delay

the

execution

When

the

matrix

is

small

and

the

number

of

processors

is

large

the

benefit

of

load

balancing

is

small

compared

to

the

extra

cost

of

communication

We

observe

that

on

the

for

small

matrix

size

the

Express

version

yields

speedup

with

the

number

of

processors

varying

from

to

Speedup

is

also

observed

for

processors

but

only

for

large

matrix

sizes

but

the

execution

times

start

to

increase

with

processors

On

the

other

hand

the

NX

version

yields

speedup

even

with

processors

The

execution

times

for

the

PVM

version

are

comparable

to

the

Express

version

For

large

matrix

size

the

PVM

version

can

sometimes

outperform

the

Express

version

Inspecting

the

timings

on

the

cluster

of

HP

and

IPX

clusters

we

can

observe

reasonable

speedup

with

increasing

number

of

workstations

with

both

Express

and

PVM

The

times

on

the

and

the

HP

cluster

are

comparable

for

smaller

matrix

size

the

times

on

the

HP

cluster

are

even

better

than

those

on

the

The

relative

performance

of

Express

to

NX

and

PVM

to

Express

is

indicated

in

Figure

-LRB-

c

-RRB-

and

Figure

-LRB-

c

-RRB-

respectively

The

N-body

Problem

The

program

for

the

N-body

problem

was

written

using

the

algorithm

reported

in

-LSB-

-RSB-

The

execution

times

of

the

three

platforms

are

shown

in

Table

XV

to

Table

XVII

These

results

indicate

that

on

the

all

three

version

yield

speedup

as

the

number

of

processors

increases

For

large

number

of

bodies

the

times

of

Express

and

are

very

close

However

the

NX

Express

PVM

NX

Express

PVM

NX

ExpressPVM

NX

Express

PVM

NX

ExpressPVM

PE

PE

PE

Table

XII

Timings

-LRB-

seconds

-RRB-

for

column-scattered

partitioned

Gaussian

elimination

using

NX

Express

and

PVM

on

the

PE

PE

Matrix

Size

Express

PVM

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

XIII

Timings

-LRB-

seconds

-RRB-

for

column-scattered

partitioned

Gaussian

elimination

using

Express

and

PVM

on

the

IPX

cluster

connected

by

Ethernet

PE

PE

Matrix

Size

PVM

version

is

slower

than

the

other

two

versions

by

a

factor

of

to

-LRB-

see

Figure

-RRB-

On

the

HP

cluster

the

PVM

version

is

found

to

be

in

general

times

slower

than

the

Express

version

On

the

IPX

cluster

the

times

of

both

Express

and

PVM

versions

are

found

to

be

close

Com

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

XIV

Timings

-LRB-

seconds

-RRB-

for

column-scattered

partitionedGaussian

elimination

using

Express

and

PVM

on

the

HP

cluster

connected

by

FDDI

PE

Matrix

Size

Figure

Ratios

of

execution

times

of

Express

to

NX

for

different

Gaussian

elimination

algorithms

on

the

-LRB-

c

-RRB-

Column-scattered

partitioned

-LRB-

a

-RRB-

Row-block

partitioned

-LRB-

b

-RRB-

Column-block

partitioned

Express

to

NX

Number

of

nodes

Row-block

partitioned

Gaussian

Elimination

by

by

by

by

by

Express

to

NX

Number

of

nodes

Column-block

partitioned

Gaussian

Elimination

by

by

by

by

by

Express

to

NX

Number

of

nodes

Column-scatter

partitioned

Gaussian

Elimination

by

by

by

by

by

pared

to

the

the

HP

cluster

is

faster

but

IPX

is

slower

Because

of

less

communication

involved

both

Express

and

PVM

seem

to

scale

well

on

the

workstation

clusters

Conclusions

In

this

paper

we

made

an

experimental

performance

comparison

of

Express

and

PVM

The

performance

study

was

carried

out

on

an

parallel

computer

a

cluster

of

IPX

workstations

connected

by

an

Ethernet

and

a

cluster

of

HP

workstations

connected

by

an

FD

DI

The

summary

of

our

results

is

that

Express

primitives

are

in

general

faster

than

those

of

PVM

but

marginally

slower

than

those

of

NX

When

the

message

size

is

small

PVM

primitives

are

significantly

slower

than

NX

and

Express

on

the

In

general

PVM

is

also

slower

Figure

Ratios

of

execution

times

of

PVM

to

Express

for

different

Gaussian

elimination

algorithms

on

the

-LRB-

c

-RRB-

Column-scattered

partitioned

-LRB-

a

-RRB-

Row-block

partitioned

-LRB-

b

-RRB-

Column-block

partitioned

PVM

to

Express

Number

of

nodes

Column-scatter

partitioned

Gaussian

Elimination

by

by

by

by

by

PVM

to

Express

Number

of

nodes

Row-block

partitioned

Gaussian

Elimination

by

by

by

by

by

PVM

to

Express

Number

of

nodes

Column-block

partitioned

Gaussian

Elimination

by

by

by

by

by

than

Express

on

the

IPX

and

HP

clusters

As

the

message

size

increases

the

difference

between

the

performance

of

PVM

and

Express

becomes

less

significant

For

very

large

messages

PVM

even

outperforms

Express

This

makes

Express

more

suitable

for

fine-grained

applications

on

parallel

systems

and

PVM

more

suitable

for

coarse-grained

problems

on

distributed

systems

The

global

operations

of

both

Express

and

PVM

do

not

scale

very

well

with

an

increasing

number

of

processors

Comparing

different

platforms

the

FDDI

is

comparable

to

the

However

since

only

workstations

were

available

the

scalability

of

FDDI

could

not

be

determined

In

general

the

FDDI

is

faster

than

the

Ethernet

by

factors

of

to

Comparing

the

timings

of

applications

we

noticed

that

PVM

outperformed

Express

on

the

when

the

granularity

of

the

problem

was

large

the

number

of

processors

was

small

and

the

message

sizes

were

large

Aside

from

the

performance

issue

the

choice

of

using

Express

or

PVM

would

depend

on

a

number

of

factors

such

as

the

type

of

application

programming

paradigm

communication

NX

Express

PVM

NX

ExpressPVM

NX

Express

PVM

NX

ExpressPVM

NX

Express

PVM

PE

PE

PE

Table

XV

Timings

-LRB-

seconds

-RRB-

for

the

N-body

problem

using

NX

Express

and

PVM

on

the

PE

PE

Points

PE

Express

PVM

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

XVI

Timings

-LRB-

seconds

-RRB-

for

the

N-body

problem

using

Express

and

PVM

on

the

IPX

cluster

connected

by

Ethernet

PE

PE

Points

Express

PVM

Express

PVM

Express

PVM

PE

PE

Table

XVII

Timings

-LRB-

seconds

-RRB-

for

the

N-body

problem

using

Express

and

PVM

on

the

IPX

cluster

connected

by

FDDI

PE

Points

patterns

and

typical

sizes

of

the

messages

in

the

application

The

type

of

application

may

determine

whether

the

application

can

be

better

implemented

as

coarse-grained

or

fine-grained

A

networked

hardware

platform

would

be

more

useful

for

a

coarse-grained

implementation

and

PVM

would

be

a

more

appropriate

choice

However

Express

can

be

more

efficient

for

a

fine-grained

implementation

on

a

parallel

machine

or

a

workstations

cluster

The

advantage

of

Express

is

that

by

using

its

portable

message-passing

library

the

development

phase

of

an

application

can

be

carried

out

on

a

workstations

cluster

but

final

experimentation

can

be

done

on

a

parallel

machine

Contrary

to

PVM

Express

provides

a

very

large

number

of

primitives

and

a

number

of

additional

tools

and

utilities

The

programming

paradigm

could

be

SPMD

or

MIMD

using

a

combination

of

different

processes

These

paradigms

can

be

efficiently

implemented

using

Express?s

grid

facilities

The

advantages

of

PVM

include

its

small

size

the

simplicity

in

programming

and

support

for

a

complete

heterogeneous

supercomputing

enviornment

The

initial

versions

of

PVM

used

TCP/IP

sockets

to

implement

all

communications

Consequently

PVM

was

limited

to

distributed

systems

However

recent

versions

of

PVM

such

as

the

one

reported

in

this

paper

have

included

implementation

within

parallel

machines

On

the

other

hand

Express

which

was

originally

developed

for

homogeneous

multiprocessors

has

evolved

towards

distributed

systems

It

has

tackled

the

problem

of

heterogeneity

only

recently

PVM

can

be

useful

for

running

an

application

across

multiple

parallel

machines

as

a

combination

of

parallel

and

distributed

computing

paradigms

Communications

patterns

and

sizes

of

the

messages

within

a

parallel

program

can

also

affect

the

choice

of

Express

and

PVM

-LRB-

for

example

PVM

does

not

provide

collective

communication

and

global

operations

-RRB-

Perhaps

the

major

advantage

of

PVM

over

Express

is

that

the

former

is

available

in

a

public

domain

while

the

latter

is

a

commercial

product

Figure

Ratios

of

execution

times

of

Express

to

NX

and

PVM

to

Express

for

the

N-Body

problem

on

the

-LRB-

b

-RRB-

Ratios

of

execution

times

of

PVM

to

Express

-LRB-

a

-RRB-

Ratios

of

execution

times

of

Express

to

NX

Express

to

NX

Number

of

nodes

N-body

problem

PVM

to

Express

Number

of

nodes

N-body

problem

References

-LSB-

-RSB-

I.

Ahmad

et

al.

Implementation

and

Scalability

of

Fortran

Intrinsic

Functions

on

Distributed

Memory

Machines

Technical

Report

Northeast

Parallel

Architectures

Center

Syracuse

University

Syracuse

NY

May

-LSB-

-RSB-

S.

Ahuja

N.

Carriero

and

D.

Gelernter

Linda

and

Friends

IEEE

Computer

no.

August

-LSB-

-RSB-

I.

Angus

G.

Fox

J.

Kim

D.

Walker

Solving

Problems

on

Concurrent

Processors

Volume

II

Prentice

Hall

Englewood

Cliffs

New

Jersey

-LSB-

-RSB-

G.

Bell

Ultracomputers

A

Teraflop

Before

its

Time

Communications

of

the

ACM

Vol

No.

August

pp.

-LSB-

-RSB-

A.

Beguelin

J.

Dongarra

A.

Geist

R.

Manchek

V.

Sunderam

Paradigms

and

Tools

for

Heterogeneous

Network

Computing

Supercomputing

Nov.

-LSB-

-RSB-

A.

Beguelin

J.

Dongarra

A.

Geist

R.

Manchek

V.

Sunderam

Solving

computational

grand

challenges

using

a

network

of

heterogeneous

supercomputers

Proc

of

Fifth

SIAM

conference

on

Parallel

Processing

Philadelphia

PA

SIAM

-LSB-

-RSB-

R.

Berrendorf

and

J.

Helin

Evaluating

the

Basic

Performance

of

the

Intel

Parallel

Computer

Concurrency

Practice

and

Experience

vol

-LRB-

-RRB-

May

pp.

-LSB-

-RSB-

S.

Bokhari

Communication

Overhead

on

the

Intel

Hypercube

ICASE

Interim

Report

10 182055

NASA

Langley

Research

Center

Hampton

VA

May

-LSB-

-RSB-

S.

Bokhari

Multiphase

Complete

Exchange

on

a

Circuit

Switched

Hypercube

Tech

Report

ICASE

NASA

Langly

Research

Center

Hampton

Virginia

January

-LSB-

-RSB-

R.M.Butler

and

E.

Lusk

Monitors

Messages

and

Clusters

The

Parallel

Programming

System

Technical

Report

TM-ANL

Argonne

National

Laboratory

-LSB-

-RSB-

N.

Carriero

and

D.

Gelernter

Linda

in

context

Communications

of

the

ACM

vol

no.

April

pp.

-LSB-

-RSB-

J.

Flower

and

A.

Kolawa

A

Packet

History

of

Message

Passing

Systems

Parasoft

Corporation

-LSB-

-RSB-

J.

Flower

and

A.

Kolawa

Express

is

not

just

a

Message

Passing

System

Current

and

Future

Directions

in

Express

Parallel

Computing

-LRB-

-RRB-

pp.

-LSB-

-RSB-

G.C.

Fox

M.A.

Johnson

G.A.

Lyzenga

S.W.

Otto

J.A.

Salmon

and

D.A.

Walker

Solving

Problems

on

Concurrent

Processors

vol

I

Prentice

Hall

-LSB-

-RSB-

R.

F.

Freund

and

H.

J.

Siegel

Heterogeneous

Processing

IEEE

Computer

Vol

No.

June

pp.

-LSB-

-RSB-

G.A.

Geist

M.T.

Heath

B.W.

Peyton

and

P.H.

Worley

A

User?s

Guide

to

Picl

a

Portable

Instrumented

Communication

Library

Technical

Report

ORNL/TM

Oak

Ridge

National

Laboratory

Oak

Ridge

TN

Oct

-LSB-

-RSB-

A.

Geist

A.

Beguelin

J.

Dongarra

W.

Jiang

R.

Manchek

V.

Sunderam

PVM

User?s

Guide

and

Reference

Manual

Oakridge

National

Labs

-LSB-

-RSB-

Intel

Corporation

and

Programmers

Reference

Manual

June

-LSB-

-RSB-

O.A.McBryan

An

Overview

of

Message-Passing

Environments

Parallel

Computing

-LRB-

-RRB-

pp.

-LSB-

-RSB-

S.A.

Moyer

Performance

of

the

Node

Architecture

Tech

Report

Institute

of

Parallel

Computations

University

of

Virginia

May

-LSB-

-RSB-

Parasoft

Corporation

Express

Fortran

User?s

Guide

-LSB-

-RSB-

Parasoft

Corporation

Express

Introductory

Guide

version

-LSB-

-RSB-

P.

Pierce

The

NX

Message

Passing

Interface

Parallel

Computing

-LRB-

-RRB-

pp.

-LSB-

-RSB-

A.

Skjellum

and

A.P.

Leung

Zipcode

A

Portable

Multicomputer

Communication

Library

atop

the

Reactive

Kernal

Proc

of

Distributed

Memory

Computing

Conference

April

-LSB-

-RSB-

V.

Sunderam

PVM

A

Framework

for

Parallel

Distributed

Computing

Concurrency

Practice

Experience

Vol

No.

Dec

pp

-LSB-

-RSB-

V.S.

Sunderam

G.A.

Geist

J.

Dongarra

R.

Manchek

The

PVM

Concurrent

Computing

System

Evolution

Experiences

and

Trends

Parallel

Computing

-LRB-

-RRB-

pp.

-LSB-

-RSB-

L.

Turcotte

A

Survey

of

Software

Environments

for

Exploiting

Networked

Computing

Resources

Technical

Report

Engineering

research

center

for

Computational

Field

Simulations

Mississippi

State

University

January

benchmarking
express
parallel programming
performance evaluation
pvm

hypercube computers
interprocessor communication
parallel algorithms