Efficient

Scatter

Communication

in

Wormhole

k-ary

n-cubes

with

Multidestination

Message

Passing

Mohammad

Banikazemi

and

Dhabaleswar

K.

Panda

Technical

Report

Efficient

Scatter

Communication

in

Wormhole

k-ary

n-cubes

with

Multidestination

Message

Passing

Mohammad

Banikazemi

and

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

E-mail

fbanikaze,pandag@cis.ohio-state.edu

Contact

Author

Dhabaleswar

K.

Panda

Abstract

This

paper

presents

efficient

ways

to

implement

one-to-all

scatter

communication

on

k-ary

n-cubes

with

multidestination

message-passing

Earlier

researchers

have

applied

multidestination

message-passing

to

support

non-personalized

communications

In

this

paper

for

the

first

time

in

the

literature

we

demonstrate

that

multidestination

message-passing

can

also

be

used

to

support

efficient

personalized

communication

Current

parallel

systems

use

either

a

sequential

tree-based

-LRB-

ST

-RRB-

scheme

or

a

binomial

tree-based

-LRB-

BT

-RRB-

scheme

with

unicast

message-passing

to

implement

scatter

First

we

show

that

the

ST

scheme

achieves

the

lower

bound

just

for

a

narrow

range

of

message

lengths

Next

we

show

that

the

BT

scheme

has

a

better

overall

performance

compared

to

ST

but

it

can

never

achieve

the

lower

bound

Finally

we

propose

a

sequential

multidestination

tree-based

-LRB-

SMT

-RRB-

scheme

using

multidestination

messages

which

minimizes

the

scatter

latency

as

well

as

achieves

the

lower

bound

This

scheme

allows

data

for

multiple

destinations

to

be

grouped

together

and

sent

as

a

single

message

using

multidestination

multicast/broadcast

worms

All

three

schemes

are

evaluated

for

a

range

of

system

sizes

-LRB-

k

n

-RRB-

communication

start-up

times

-LRB-

ts

-RRB-

propagation

times

-LRB-

tp

-RRB-

and

message

lengths

-LRB-

l

-RRB-

For

small

message

sizes

on

an

torus

it

is

shown

that

the

SMT

scheme

can

reduce

the

scatter

latency

up

to

a

factor

of

compared

to

the

ST

scheme

and

up

to

a

factor

of

compared

to

the

BT

scheme

For

a

range

of

system

sizes

and

-LRB-

ts

tp

-RRB-

ratios

it

is

shown

that

the

SMT

scheme

performs

better

than

the

ST

and

BT

schemes

for

a

wide

range

of

message

lengths

Thus

these

results

demonstrate

significant

potential

to

be

applied

to

design

scalable

collective

communication

libraries

for

current

and

future

generation

wormhole

systems

research

is

supported

in

part

by

NSF

Grant

and

NSF

Career

Award

Contents

Introduction

Current

Approaches

for

Scatter

Communication

Scatter

Communication

and

Its

Lower

Bound

Sequential

Tree

-LRB-

ST

-RRB-

Algorithm

Binomial

Tree

-LRB-

BT

-RRB-

Algorithm

Overview

of

Multidestination

Message

Passing

Sequential

Multidestination

Tree

-LRB-

SMT

-RRB-

Algorithm

Motivation

The

General

Algorithm

The

SMT

Algorithm

for

Tori

and

Meshes

The

SMT

Algorithm

for

k-ary

n-cubes

Performance

Benefits

Performance

Evaluation

Effect

of

Communication

Start-up

Effect

of

System

Size

Effect

of

Propagation

Cost

Overall

Comparison

Conclusions

and

Future

Research

Introduction

The

wormhole-routing

switching

technique

has

become

the

trend

in

building

parallel

systems

due

to

its

inherent

advantages

like

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

Cray

-LSB-

-RSB-

-LSB-

-RSB-

MIT

J-Machine

-LSB-

-RSB-

Stanford

FLASH

-LSB-

-RSB-

Myrinet

-LSB-

-RSB-

use

the

wormhole-routing

switching

technique

Such

systems

are

being

used

for

supporting

either

distributedmemory

shared-memory

or

distributed-shared

memory

programming

paradigms

The

need

for

collective

communication

operations

such

as

broadcast

multicast

global

reduction

scatter

gather

complete

exchange

and

barrier

synchronization

arises

frequently

in

parallel

applications

-LSB-

-RSB-

The

usage

of

collective

communication

operations

simplify

the

programming

of

parallel

applications

and

facilitate

the

implementation

of

efficient

and

portable

communication

schemes

on

different

machines

The

Message

Passing

Interface

Standard

-LSB-

-RSB-

has

recently

defined

the

importance

of

collective

communication

operations

One

of

the

important

collective

communication

operations

is

one-to-all

personalized

transfer

or

simply

known

as

scatter

-LSB-

-RSB-

In

this

operation

one

process

sends

different

messages

-LRB-

each

of

l

bytes

long

-RRB-

to

every

other

process

of

an

user-defined

group

Most

of

the

current

wormhole

systems

support

only

unicast

message

passing

-LSB-

-RSB-

It

has

been

shown

recently

that

collective

communication

libraries

use

either

a

sequential

tree-based

-LRB-

ST

-RRB-

scheme

or

binomial

tree-based

-LRB-

BT

-RRB-

scheme

to

support

scatter

communication

The

ST

scheme

is

used

on

IBM

under

the

MPICH

implementation

-LSB-

-RSB-

and

the

BT

scheme

under

the

MPL

implementation

-LSB-

-RSB-

The

ST

scheme

achieves

the

lower

bound

for

scatter

communication

for

a

narrow

range

of

system

parameters

For

a

wide

range

of

system

parameters

its

latency

is

much

higher

than

the

lower

bound

On

the

other

hand

the

BT

scheme

shows

a

better

overall

performance

for

large

range

of

system

parameters

However

it

can

never

achieve

the

lower

bound

Thus

it

is

challenging

to

determine

whether

scatter

communication

can

be

implemented

with

reduced

latency

such

that

it

achieves

the

lower

bound

for

a

wide

range

of

system

parameters

In

this

paper

we

take

on

such

challenges

and

demonstrate

that

efficient

scatter

communication

using

multidestination

message

passing

can

satisfy

the

above

objectives

The

multidestination

message

passing

mechanism

has

recently

been

proposed

and

used

to

implement

a

set

of

collective

communication

operations

using

non-personalized

message

exchanges

-LSB-

-RSB-

However

to

the

best

of

our

knowledge

it

has

not

been

applied

to

implement

any

collective

communication

operations

with

personalized

message

exchange

In

this

paper

for

the

first

time

in

the

literature

we

demonstrate

the

capability

of

multidestination

message

passing

to

implement

one-to-all

scatter

operation

We

propose

a

sequential

multidestination

tree-based

-LRB-

SMT

-RRB-

scheme

using

only

multidestination

multicast/broadcast

worms

-LSB-

-RSB-

Since

the

scatter

communication

uses

personalized

messages

its

lower

bound

is

mainly

dependent

on

the

amount

of

time

needed

for

message

propa

gation

Thus

any

algorithm

with

a

capability

to

overlap

communication

start-up

times

with

message

propagation

times

can

meet

the

lower

bound

Such

overlap

is

dependent

on

many

factors

like

system

size

message

size

communication

start-up

time

-LRB-

ts

-RRB-

and

message

propagation

time

-LRB-

tp

-RRB-

Under

the

ST

scheme

it

is

possible

to

achieve

the

lower

bound

if

ts

ltp

where

l

is

the

message

size

-LRB-

in

bytes

-RRB-

being

transfered

to

every

node

from

the

source

node

Since

the

current

generation

systems

have

higher

ts

it

is

not

possible

to

achieve

the

lower

bound

under

the

ST

scheme

unless

message

sizes

are

quite

long

The

BT

scheme

alleviates

this

bottleneck

up

to

certain

extent

but

does

not

reach

the

lower

bound

In

our

SMT

scheme

we

take

advantage

of

the

multicast/broadcast

feature

of

the

multidestination

mechanism

For

a

set

of

destinations

falling

along

a

base-routing-conformedpath

-LRB-

BRCP

-RRB-

-LSB-

-RSB-

we

group

their

data

together

and

send

it

as

a

single

message

For

different

values

of

ts

tp

and

l

we

choose

appropriate

size

for

grouping

the

destinations

so

that

ts

ltp

This

allows

us

to

achieve

the

lower

bound

for

a

wide

range

of

system

parameters

and

message

length

l.

Using

such

an

approach

we

develop

and

present

scatter

algorithms

for

k-ary

n-cubes

We

evaluate

all

three

algorithms

for

a

range

of

system

and

technological

parameters

It

is

shown

that

the

SMT

scheme

can

implement

scatter

operation

with

reduced

latency

compared

to

the

ST

and

BT

schemes

Thus

the

SMT

scheme

promises

great

potential

to

design

scalable

library

for

scatter

communication

in

current

and

future

generation

wormhole

systems

The

Paper

is

organized

as

follows

The

ST

and

BT

schemes

for

scatter

communication

are

discussed

in

section

An

overview

of

the

multidestination

mechanism

is

presented

in

section

In

section

we

develop

the

SMT

scheme

and

propose

algorithms

for

k-ary

n-cube

systems

Performance

evaluation

results

are

presented

in

section

Finally

we

conclude

the

paper

with

conclusions

and

future

research

directions

Current

Approaches

for

Scatter

Communication

In

this

section

we

define

the

scatter

collective

communication

operation

and

derive

its

lower

bound

for

wormhole-routed

systems

Then

we

present

two

unicast-based

scatter

algorithms

which

are

used

on

current

systems

We

also

investigate

the

conditions

under

which

these

algorithms

can

meet

the

lower

bound

Without

loss

of

generality

we

assume

that

the

original

data

at

the

source

node

is

ordered

in

a

column-major

fashion

Scatter

Communication

and

Its

Lower

Bound

Scatter

is

one

of

the

personalized

communication

operations

in

which

one

processor

-LRB-

source

node

-RRB-

sends

a

different

message

to

every

other

processor

in

an

user-defined

group

-LRB-

Fig.

-RRB-

For

a

system

with

P

processors

and

message

size

of

l

bytes

the

volume

of

the

data

which

has

to

be

sent

out

from

the

source

node

to

the

network

is

-LRB-

P

-RRB-

l

and

at

P

P

Figure

The

semantic

of

scatter

operation

least

one

message

should

be

sent

out

Therefore

the

lower

bound

-LRB-

LB

-RRB-

of

the

scatter

operation

latency

for

wormhole

routed

systems

can

be

derived

as

TLB

ts

-LRB-

P

-RRB-

ltp

-LRB-

-RRB-

where

ts

is

the

start-up

overhead

and

tp

is

the

transmission

time

per

byte

Sequential

Tree

-LRB-

ST

-RRB-

Algorithm

The

ST

algorithm

-LRB-

also

known

as

separate

addressing

-RRB-

is

based

on

a

sequential

tree

-LSB-

-RSB-

In

this

algorithm

the

source

node

sends

a

separate

message

to

each

of

the

other

processors

participating

in

scatter

Therefore

the

number

of

required

communication

steps

is

P

Figure

shows

the

sequential

tree

for

a

system

with

processors

The

implementation

of

ST

on

a

mesh/torus

is

illustrated

in

Fig

The

algorithm

for

implementing

the

ST

algorithm

is

presented

in

Fig.

Figure

Sequential

tree

ST

-RRB-

scheme

for

an

system

Let

us

analyze

the

latency

of

the

scatter

communication

under

the

ST

algorithm

Depending

on

the

system

parameters

two

different

cases

can

be

recognized

ts

ltp

and

ts

ltp

Thus

the

scatter

latency

can

be

written

as

TST

-LRB-

ts

-LRB-

P

-RRB-

ltp

if

ts

ltp

-LRB-

P

-RRB-

ts

ltp

if

ts

ltp

-LRB-

-RRB-

ignore

the

impact

of

node

delay

for

simplicity

Source

Unicast

Worm

Figure

Sequential

tree

-LRB-

ST

-RRB-

algorithm

on

a

mesh/torus

The

ST

Algorithm

Inputs

root

the

source

node

of

scatter

msg

the

address

of

the

message

l

message

length

P

number

of

participating

nodes

Output

Each

node

receives

its

portion

of

data

Strt

returns

the

starting

address

of

the

message

which

should

be

transmitted

to

a

specific

node

procedure

scatter

-LRB-

root

msg

l

P

-RRB-

begin

if

-LRB-

MyId

root

-RRB-

then

for

i

to

do

begin

if

-LRB-

i

MyId

-RRB-

then

begin

tmsg

Strt

-LRB-

msg

i

l

-RRB-

send

tmsg

to

i

length

l

end

endfor

else

receive

msg

from

root

end

Figure

Pseudo-code

for

the

ST

algorithm

It

is

obvious

that

only

if

ts

ltp

then

the

lower

bound

can

be

met

If

this

condition

is

not

satisfied

the

latency

of

the

scatter

operation

will

exceeds

the

lower

bound

Because

of

the

-LRB-

P

-RRB-

ts

term

the

latency

of

scatter

can

increase

rapidly

when

the

system

size

increases

or

the

start-up

overhead

is

significant

This

leads

to

the

following

observations

Observation

The

ST

algorithm

can

achieve

the

lower

bound

only

when

l

tstp

Observation

With

l

tstp

the

latency

of

scatter

increases

rapidly

with

increase

in

system

size

Binomial

Tree

-LRB-

BT

-RRB-

Algorithm

The

BT

algorithm

is

based

on

the

well

known

binomial

trees

-LSB-

-RSB-

Such

an

algorithm

is

used

in

the

MPL

library

-LSB-

-RSB-

for

scatter

implementation

In

this

approach

the

number

of

processors

which

receive

the

message

is

doubled

in

each

communication

step

Figure

illustrates

the

BT

algorithm

on

a

system

In

the

first

step

the

source

node

-LRB-

P

-RRB-

sends

the

data

for

half

of

the

processors

-LRB-

and

P

-RRB-

to

node

P

In

the

following

steps

becomes

responsible

to

deliver

the

data

to

processors

P

P

and

P

In

the

second

step

sends

the

data

of

and

to

processor

P

At

the

same

time

sends

the

data

of

and

to

P

Finally

processors

P

P

P

and

send

the

appropriate

data

to

P

P

P

and

P

respectively

Figure

illustrates

the

implementation

of

the

BT

scatter

algorithm

on

a

mesh/torus

The

algorithm

for

implementing

the

BT

algorithm

is

presented

in

Fig.

Figure

Binomial

Tree

-LRB-

BT

-RRB-

algorithm

on

an

system

Let

us

analyze

the

scatter

latency

using

the

BT

algorithm

Since

the

number

of

processors

which

receive

their

messages

is

doubled

in

each

step

the

number

of

required

communication

steps

for

the

entire

operation

is

P

On

the

other

hand

the

data

size

decreases

by

a

factor

of

two

in

every

successive

step

Therefore

the

latency

of

scatter

can

be

written

as

TBT

P

X

i

ts

l

tp

P

ts

-LRB-

P

-RRB-

ltp

-LRB-

-RRB-

It

can

be

seen

that

the

BT

algorithm

can

never

meet

the

lower

bound

due

to

additional

number

of

start-ups

This

leads

to

Observation

The

BT

algorithm

is

not

capable

of

achieving

the

lower

bound

Unicast

Worm

Source

Figure

BT

algorithm

on

a

mesh/torus

Overview

of

Multidestination

Message

Passing

In

unicast

or

point-to-point

message

passing

each

message

has

exactly

one

destination

In

contrast

systems

with

multidestination

message

passing

allows

a

multicast/broadcast

message

to

be

absorbed

at

one

of

the

destination

nodes

while

being

forwarded

-LRB-

i.e.

absorband-forward

-RRB-

to

the

next

node

concurrently

-LSB-

-RSB-

The

header

of

multidestination

messages

consist

of

multiple

destinations

as

an

ordered

list

depending

on

their

intended

order

of

traversal

A

Hamiltonian

path-based

routing

scheme

has

been

introduced

in

-LSB-

-RSB-

to

use

multidestination

messages

Recently

a

Base-Routing-Confirmed-Path

-LRB-

BRCP

-RRB-

model

has

been

proposed

in

-LSB-

-RSB-

to

apply

multidestination

message

passing

to

dimensionordered

and

other

adaptive

routing

schemes

Formally

the

BRCP

model

can

be

described

as

follows

Let

us

assume

a

network

supporting

a

deadlock-free

routing

scheme

R

for

unicast

message

passing

We

identify

R

as

the

base-routing

A

multidestination

worm

with

an

ordered

destination

list

dn

dn

on

a

network

supporting

routing

scheme

R

conforms

to

the

this

base

routing

if

the

destination

set

dn

can

be

covered

as

intermediate

nodes

on

one

of

the

possible

paths

from

s

to

dn

under

the

routing

constraint

R.

Thus

a

multidestination

worm

gets

routed

in

a

piece-wise

manner

from

the

source

to

the

final

destination

The

BT

Algorithm

Inputs

root

the

source

node

of

scatter

msg

the

address

of

the

message

l

message

length

P

number

of

participating

nodes

Output

Each

node

receives

its

portion

of

data

AbsID

converts

the

relative

ID

of

a

node

with

respect

to

the

root

to

the

real

ID

of

that

node

in

the

system

procedure

scatter

-LRB-

root

msg

l

P

-RRB-

begin

NumSteps

P

for

i

to

NumSteps

do

begin

diff

-LRB-

NumSteps-i

-RRB-

couples

P

-LRB-

diff

-RRB-

for

j

to

do

begin

AbsID

converts

the

relative

ID

of

a

node

with

respect

to

the

root

to

the

real

ID

of

that

node

in

the

system

sndr

AbsID

-LRB-

root

j

diff

-RRB-

rcvr

AbsID

-LRB-

root

sndr

diff

-RRB-

if

-LRB-

MyId

sndr

-RRB-

then

send

msg

to

rcvr

length

-LRB-

P

l

-RRB-

-LRB-

i

-RRB-

else

receive

msg

from

sndr

endfor

endfor

end

Figure

Pseudo-code

for

the

BT

algorithm

through

intermediate

destinations

Such

a

worm

is

capable

of

delivering

data

to

a

set

of

destinations

using

a

single

message

passing

step

We

now

show

how

multidestination

message

passing

can

be

used

to

improve

the

performance

of

scatter

operation

on

wormhole-routed

systems

for

a

wide

range

of

technological

and

system

parameters

Sequential

Multidestination

Tree

-LRB-

SMT

-RRB-

Algorithm

In

this

section

we

use

multidestination

message

passing

to

develop

better

algorithms

for

scatter

First

we

provide

the

motivation

behind

such

algorithms

Next

we

propose

a

generalized

algorithm

After

that

we

propose

algorithms

for

tori/meshes

and

k-ary

n-cubes

Finally

we

discuss

the

performance

benefits

of

the

SMT

algorithm

over

the

ST

and

BT

algorithms

Motivation

As

presented

in

observations

and

only

ST

algorithm

can

achieve

the

lower

bound

Nevertheless

observation

shows

that

the

ST

algorithm

can

achieve

the

lower

bound

just

for

a

limited

variations

of

system

parameters

and

message

sizes

This

lower

bound

is

achieved

only

when

the

start-up

overhead

of

sending

messages

are

overlapped

with

the

transmission

time

of

other

messages

To

achieve

this

requirement

the

ts

ltp

condition

should

hold

In

other

words

the

size

of

the

messages

being

transmitted

should

be

large

enough

so

that

the

next

message

can

become

ready

for

transmission

not

later

than

when

the

previous

message

has

been

completely

injected

to

the

interconnection

network

-LRB-

l

tstp

-RRB-

In

this

section

we

propose

a

new

algorithm

that

extends

the

cases

in

which

the

lower

bound

latency

can

be

met

by

increasing

the

size

of

the

messages

which

are

being

transmitted

The

multidestination

message

passing

mechanism

helps

us

to

achieve

such

increase

of

message

length

The

General

Algorithm

In

the

SMT

algorithm

all

processors

which

are

participating

in

the

scatter

communication

operation

are

grouped

into

smaller

groups

The

messages

of

all

processors

in

each

group

are

combined

to

form

a

multidestination

message

Then

these

different

multidestination

messages

are

sent

out

to

the

members

of

the

corresponding

groups

Depending

on

the

routing

scheme

being

used

in

the

system

and

the

arrangement

of

the

data

at

the

source

node

different

groupings

can

be

used

For

example

as

illustrated

in

Fig.

nodes

in

a

system

are

grouped

into

four

groups

-LRB-

-RRB-

With

such

grouping

the

source

node

uses

four

messages

to

cover

these

groups

If

the

smallest

group

of

processors

has

g

-LRB-

g

-RRB-

members

then

the

size

of

the

shortest

message

that

is

going

to

be

transmitted

is

gl

Therefore

the

condition

under

which

the

lower

bound

is

met

can

be

written

as

gl

tstp

The

minimum

value

of

g

to

achieve

the

lower

bound

can

be

found

from

the

following

equation

gmin

ts

ltp

-LRB-

-RRB-

Obviously

for

bigger

g

's

there

are

less

constraints

on

system

parameters

and

message

size

under

which

the

lower

bound

can

be

achieved

However

the

need

for

rearrangement

of

the

original

data

and

the

routing

constraints

do

n't

allow

selecting

any

arbitrary

g

's

Assuming

that

all

groups

have

g

members

the

execution

time

of

the

SMT

scatter

algorithm

can

be

derived

as

TSMT

-LRB-

ts

-LRB-

P

-RRB-

ltp

if

ts

lgtp

Pg

ts

-LRB-

g

-RRB-

ltp

if

ts

lgtp

-LRB-

-RRB-

Observation

The

SMT

algorithm

can

achieve

the

lower

bound

when

l

tsg

tp

Figure

Illustration

of

the

Sequential

Multidestination

Tree

-LRB-

SMT

-RRB-

algorithm

Using

this

generalized

approach

we

show

how

this

algorithm

can

be

used

to

implement

scatter

on

meshes/tori

and

higher

dimensional

systems

with

dimension-ordered

routing

The

SMT

Algorithm

for

Tori

and

Meshes

In

a

k

k

torus

it

can

be

easily

observed

that

the

processors

of

each

column

can

form

a

group

A

multidestination

message

can

deliver

the

data

for

all

members

of

each

group

under

dimension-ordered

routing

The

minimum

value

of

g

for

which

the

lower

bound

latency

is

achievable

can

be

found

from

Equn

The

latency

of

scatter

can

be

found

from

Equn

If

gmin

k

then

the

lower

bound

is

achieved

Figure

illustrates

the

implementation

of

the

SMT

algorithm

on

a

torus

Detailed

steps

for

implementing

the

SMT

algorithm

on

a

k

k

torus

is

presented

in

Fig.

Multidestination

Worm

Source

Figure

Sequential

Multidestination

Tree

-LRB-

SMT

-RRB-

algorithm

on

a

torus

Let

us

consider

applying

the

SMT

algorithm

to

meshes

Exactly

the

same

approach

as

used

for

tori

can

be

used

for

meshes

when

the

source

node

is

in

the

up-most

row

of

the

mesh

-LRB-

Fig.

-RRB-

In

this

case

the

latency

of

scatter

on

a

mesh

will

be

the

same

as

that

on

a

torus

and

can

be

calculated

from

Equn

However

if

the

source

node

is

not

in

the

up-most

row

an

extra

step

is

required

to

transfer

the

data

to

one

of

the

nodes

in

the

first

row

This

will

increase

the

overall

execution

time

of

scatter

The

SMT

Algorithm

for

k-ary

n-cubes

The

SMT

algorithm

for

tori

can

be

easily

generalized

for

k-ary

n-cubes

To

implement

the

SMT

algorithm

on

k-ary

n-cubes

with

dimension-ordered

routing

it

is

enough

to

partition

the

processors

along

the

lowest

dimension

and

select

g

equal

to

k

One

multidestination

message

is

enough

for

delivering

the

messages

of

all

members

of

each

group

The

scatter

latency

can

be

found

from

Equn

and

the

lower

bound

is

met

when

gmin

k

The

detailed

steps

for

implementing

the

SMT

algorithm

on

a

k-ary

n-cube

is

presented

in

Fig

Performance

Benefits

As

mentioned

in

Observations

and

both

the

ST

and

SMT

algorithms

can

achieve

the

lower

bound

under

some

conditions

The

condition

under

which

the

latency

of

the

ST

algorithm

will

not

exceed

the

lower

bound

is

l

tstp

On

the

other

hand

l

tsg

tp

condition

should

hold

so

that

the

SMT

algorithm

can

achieve

the

lower

bound

latency

From

these

conditions

it

is

obvious

that

the

SMT

algorithm

provides

a

more

relaxed

constraint

From

Observation

we

can

conclude

that

smaller

communication

start-up

The

SMT

Algorithm

for

Tori

Inputs

root

the

source

node

of

scatter

msg

the

address

of

the

message

l

message

length

k

number

of

nodes

in

each

dimension

Output

Each

node

receives

its

portion

of

data

Strt

returns

the

starting

address

of

the

message

which

should

be

transmitted

to

a

specific

node

NodeRowCol

converts

the

ID

of

a

node

to

the

row

and

column

in

which

it

is

located

NodeID

finds

the

ID

of

a

node

located

at

a

specific

row

and

column

procedure

scatter

-LRB-

root

msg

l

k

-RRB-

begin

if

-LRB-

MyId

root

-RRB-

then

for

i

to

do

begin

NodeRowCol

-LRB-

MyId

row

col

-RRB-

dests

fg

for

j

to

do

dests

NodeID

-LRB-

j

mod

-LRB-

col

-RRB-

-RRB-

-LSB-

dests

tmsg

Strt

-LRB-

buffer

NodeID

-LRB-

mod

-LRB-

col

-RRB-

-RRB-

l

-RRB-

sendmulti

tmsg

to

dests

length

k

l

endfor

else

receive

msg

from

root

end

Figure

Pseudo-code

for

the

SMT

algorithm

for

tori

Multidestination

Source

Worm

Figure

SMT

algorithm

on

a

mesh

The

SMT

Algorithm

for

k-ary

n-cubes

Inputs

root

the

source

node

of

scatter

buffer

the

address

of

the

message

l

message

length

k

number

of

nodes

in

each

dimension

n

number

of

dimensions

Output

Each

node

receives

its

portion

of

data

Strt

returns

the

starting

address

of

the

message

which

should

be

transmitted

to

a

specific

node

procedure

scatter

-LRB-

root

buffer

l

k

n

-RRB-

begin

if

-LRB-

MyId

root

-RRB-

then

for

dn

to

for

dn

to

ffl

ffl

ffl

for

to

begin

dests

fg

base

dn

dn

for

to

dests

dests

-LSB-

base

Strt

returns

the

starting

address

of

the

message

of

node

i

msg

Strt

-LRB-

buffer

base

-RRB-

multisend

msg

to

dests

length

k

l

endfor

else

receive

msg

from

root

end

Figure

Outline

of

the

SMT

algorithm

for

k-ary

n-cubes

overhead

-LRB-

ts

-RRB-

will

increase

the

number

of

cases

for

which

the

lower

bound

is

achievable

It

can

also

be

seen

that

having

a

bigger

k

will

have

the

same

effect

However

decreasing

the

per-byte

transmission

time

-LRB-

tp

-RRB-

will

have

the

opposite

effect

The

effect

of

changes

in

system

parameters

and

size

on

the

minimum

message

length

for

which

the

lower

bound

is

achievable

-LRB-

lmin?lb

-RRB-

is

summarized

in

Table

In

the

next

section

we

analyze

and

compare

the

effects

of

these

parameters

on

the

latency

of

the

ST

BT

and

SMT

algorithms

for

a

range

of

system

and

technological

parameters

Table

The

effect

of

changes

in

system

parameters

on

the

minimum

message

length

for

which

the

lower

bound

is

achievable

P

ts

tp

lmin?lb

fixed

fixed

fixed

fixed

fixed

fixed

Performance

Evaluation

In

this

section

we

compare

the

latency

of

ST

BT

and

SMT

algorithms

for

different

system

parameters

and

message

sizes

We

consider

a

torus

with

ts

microsec

and

tp

nsec

as

our

base

system

and

investigate

the

effect

of

the

variations

of

system

parameters

Effect

of

Communication

Start-up

Figure

shows

the

comparison

results

for

three

different

start-ups

-LRB-

ts

and

microsec

-RRB-

It

can

be

seen

that

the

SMT

algorithm

performs

better

than

the

ST

algorithm

for

all

different

message

sizes

It

can

also

be

verified

that

the

SMT

performs

better

than

BT

for

a

wide

range

of

message

sizes

As

ts

decreases

the

minimum

message

length

for

which

the

lower

bound

can

be

achieved

decreases

-LRB-

l

tsg

tp

-RRB-

With

ts

microsec

the

SMT

algorithm

achieves

the

lower

bound

for

messages

of

size

bytes

and

more

With

ts

and

microsec

lower

bound

is

achievable

for

messages

of

size

and

bytes

respectively

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

Figure

Latency

of

three

different

algorithms

for

different

start-up

times

Figure

shows

the

factor

of

improvement

of

the

SMT

algorithm

over

the

ST

algorithm

for

three

different

startups

It

can

be

observed

that

the

ST

algorithm

can

never

perform

better

than

the

SMT

algorithm

For

a

message

size

the

SMT

algorithm

can

reduce

the

scatter

latency

up

to

a

factor

of

compared

to

the

ST

algorithm

As

ts

increases

the

SMT

scheme

performs

better

than

the

ST

scheme

for

a

given

message

length

Similarly

Fig.

shows

the

factor

of

improvement

of

the

SMT

scheme

over

the

BT

scheme

It

can

be

observed

that

the

BT

scheme

can

perform

better

than

the

SMT

only

for

very

short

messages

For

other

message

sizes

the

SMT

algorithm

can

show

up

to

improvement

compared

to

the

BT

algorithm

It

can

be

seen

that

as

ts

reduces

the

maximum

message

size

for

which

the

BT

scheme

can

perform

better

than

the

SMT

algorithm

decreases

As

ts

keeps

on

reducing

for

new

generation

systems

these

results

indicate

that

the

SMT

algorithm

can

outperform

the

unicast-based

algorithms

-LRB-

ST

and

BT

-RRB-

for

a

wider

range

of

system

parameters

and

message

sizes

100 200 300 400

Factor

of

improvement

-LRB-

ST/SMT

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

100 200 300 400

Factor

of

improvement

-LRB-

ST/SMT

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

100 200 300 400

Factor

of

improvement

-LRB-

ST/SMT

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

Figure

Factor

of

improvement

-LRB-

ST/SMT

-RRB-

for

different

start-up

times

100 200 300 400

Factor

of

improvement

-LRB-

BT/SMT

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

100 200 300 400

Factor

of

improvement

-LRB-

BT/SMT

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

100 200 300 400

Factor

of

improvement

-LRB-

BT/SMT

Percentage

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

Figure

Factor

of

improvement

-LRB-

BT/SMT

-RRB-

for

different

start-up

times

Effect

of

System

Size

The

effect

of

system

size

on

the

scatter

algorithms

is

shown

in

Fig

It

is

interesting

to

observe

that

by

increasing

the

system

size

the

minimum

length

requirement

for

achieving

the

lower

bound

in

the

SMT

algorithm

decreases

It

can

be

seen

that

while

the

SMT

algorithm

can

not

achieve

the

lower

bound

for

messages

shorter

than

bytes

on

a

system

the

lower

bound

is

achieved

even

for

and

messages

on

and

systems

respectively

Such

performance

improvement

demonstrates

the

scalability

of

the

SMT

algorithm

with

respect

to

increase

in

system

size

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

Figure

Latency

of

three

different

algorithms

for

different

system

sizes

Effect

of

Propagation

Cost

Figure

shows

the

latency

of

the

three

scatter

algorithms

for

different

propagation

times

-LRB-

tp

and

nsec

-RRB-

It

can

be

observed

that

as

propagation

time

increases

the

performance

of

the

ST

algorithm

becomes

worse

On

the

other

hand

it

can

be

seen

that

as

tp

reduces

the

factor

of

improvement

obtained

by

using

the

SMT

algorithm

over

the

BT

algorithm

increases

The

factor

of

improvement

for

message

size

of

bytes

and

tp

is

For

a

similar

system

with

tp

and

nsec

the

factors

of

improvement

are

and

respectively

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

100 200 300 400

Scatter

Latency

-LRB-

microsec

-RRB-

Message

length

-LRB-

byte

-RRB-

ts

tp

ST

BT

SMT

Figure

Latency

of

three

different

algorithms

for

different

propagation

times

Overall

Comparison

From

Equns

and

it

is

obvious

that

in

the

worst

case

the

SMT

algorithm

performs

as

good

as

the

ST

algorithm

Figures

and

illustrate

the

minimum

message

size

above

which

the

SMT

algorithm

outperforms

the

BT

algorithm

A

range

of

system

sizes

-LRB-

k

k

-RRB-

and

ratios

of

-LRB-

ts

tp

-RRB-

are

considered

It

can

be

observed

that

for

a

wide

range

of

system

parameters

system

sizes

and

and

message

sizes

the

SMT

algorithm

performs

better

than

the

BT

algorithm

For

systems

the

minimum

l

is

bytes

for

a

wide

range

of

ts

tp

ratios

For

systems

with

smaller

sized

the

minimum

l

increases

with

ts

tp

ratio

However

it

is

around

bytes

even

for

ts

tp

This

overall

comparison

demonstrates

the

potential

of

the

SMT

algorithm

to

develop

scalable

library

for

scatter

communication

over

a

range

of

system

and

technological

parameters

Minimum

l

-LRB-

byte

-RRB-

k

ts/tp

Figure

Minimum

message

length

-LRB-

l

-RRB-

required

for

the

SMT

algorithm

to

perform

better

than

the

BT

algorithm

in

systems

A

range

of

system

sizes

-LRB-

k

k

-RRB-

and

ratios

of

-LRB-

ts

tp

-RRB-

are

considered

Conclusions

and

Future

Research

In

this

paper

we

have

presented

a

new

approach

to

implement

scatter

communication

in

k-ary

n-cube

wormhole

systems

using

multidestination

message

passing

mechanism

Compared

to

the

current

unicast-based

schemes

the

proposed

framework

can

meet

the

lower

bound

latency

for

a

much

wider

range

of

system

technological

and

application

parameters

As

ts

continues

to

decrease

in

current

and

future

generation

systems

it

is

shown

that

the

proposed

algorithm

can

outperform

the

unicast-based

algorithms

for

a

wider

range

of

system

parameters

and

message

sizes

It

is

also

shown

that

the

new

algorithm

is

scalable

with

respect

to

increase

in

system

size

Thus

the

new

framework

demonstrates

significant

potential

to

design

scalable

library

for

scatter

communication

in

current

and

future

generation

wormhole

systems

In

this

paper

we

have

studied

the

usage

of

multidestination

mechanism

for

the

scatter

personalized

communication

operation

in

k-ary

n-cubes

under

dimension-ordered

Minimum

l

-LRB-

byte

-RRB-

k

ts/tp

Figure

Minimum

message

length

-LRB-

l

-RRB-

required

for

the

SMT

algorithm

to

perform

better

than

the

BT

algorithm

in

systems

A

range

of

system

sizes

-LRB-

k

k

k

-RRB-

and

ratios

of

-LRB-

ts

tp

-RRB-

are

considered

in

k-ary

n-cubes

routing

We

will

be

evaluating

the

scheme

for

other

topologies

and

routing

schemes

We

are

also

extending

our

work

to

other

personalized

collective

communication

primitives

like

combined

reduction

and

scatter

and

complete

exchange

Additional

Information

A

number

of

related

papers

and

technical

reports

are

available

electronically

through

the

home

page

of

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

The

URL

is

Alternatively

the

papers

can

be

obtained

by

anonymous

ftp

from

ftp.cis.ohio-state

edu

cd

pub/communication

References

-LSB-

-RSB-

V.

Bala

J.

Bruck

R.

Cypher

P.

Elustondo

A.

Ho

C.-T

Ho

S.

Kipnis

and

M.

Snir

CCL

A

Portable

and

Tunable

Collective

Communication

Library

for

Scalable

Parallel

Computers

In

Proceedings

of

the

International

Parallel

Processing

Symposium

-LSB-

-RSB-

M.

Barnett

S.

Gupta

D.

G.

Payne

L.

Shuler

R.

van

de

Geijn

and

J.

Watts

Interprocessor

Collective

Communication

Library

-LRB-

Intercom

-RRB-

In

Scalable

High

Performance

Computing

Conference

pages

-LCB-

-LSB-

-RSB-

N.

J.

Boden

D.

Cohen

and

et

al.

Myrinet

A

Gigabit-per-Second

Local

Area

Network

IEEE

Micro

pages

-LCB-

Feb

-LSB-

-RSB-

R.

V.

Boppana

and

S.

Chalasani

A

Comparison

of

Adaptive

Wormhole

Routing

Algorithms

In

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

Cray

Research

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

Deadlock-Free

Message

Routing

in

Multiprocessor

Interconnection

Networks

IEEE

Transactions

on

Computers

pages

-LCB-

May

-LSB-

-RSB-

J.

Duato

A

New

Theory

of

Deadlock-Free

Adaptive

Routing

in

Wormhole

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

B.

Duzett

and

R.

Buck

An

Overview

of

the

Supercomputer

In

Proceedings

of

the

Frontiers

of

Massively

Parallel

Computation

pages

-LCB-

-LSB-

-RSB-

M.

Heinrich

J.

Kuskin

and

Others

The

performance

impact

of

flexibility

in

the

stanford

flash

multiprocessor

In

The

sixth

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

-LRB-

ASPLOS-VI

-RRB-

San

Jose

CA

October

-LSB-

-RSB-

C.

Huang

Y.

Huang

and

P.

K.

McKinley

A

Thread-based

Interface

for

Collective

Communication

on

ATM

Networks

In

International

Conference

on

Distributed

Computing

Systems

-LSB-

-RSB-

Y.

Huang

C.

C.

Huang

and

P.

K.

McKinley

Multicast

Virtual

Topologies

for

Collective

Communication

in

MPCs

and

ATM

Clusters

In

Proceedings

of

the

Supercomputing

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

V.

Kumar

A.

Grama

A.

Gupta

and

G.

Karypis

Introduction

to

Parallel

Computing

Design

and

Analysis

of

Algorithms

Benjamin/Cummings

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

P.

K.

McKinley

and

D.

F.

Robinson

Collective

Communication

in

Wormhole-Routed

Massively

Parallel

Computers

IEEE

Computer

pages

-LCB-

Dec

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Dec

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

P.

Mitra

D.

G.

Payne

and

et

al.

Fast

Collective

Communication

Libraries

Please

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

M.

D.

Noakes

D.

A.

Wallach

and

W.

J.

Dally

The

J-Machine

Multicomputer

An

Architectural

Evaluation

In

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

N.

Nupairoj

and

L.M.

Ni

Benchmarking

of

Multicast

Communication

Services

Technical

Report

Michigan

State

University

Sep

-LSB-

-RSB-

S.

Pakin

M.

Lauria

and

A.

Chien

High

Performance

Messaging

on

Workstations

Illinois

Fast

Messages

-LRB-

FM

-RRB-

In

Proceedings

of

the

Supercomputing

-LSB-

-RSB-

D.

K.

Panda

Fast

Barrier

Synchronization

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Worms

Future

Generation

Computer

Systems

-LCB-

Nov

-LSB-

-RSB-

D.

K.

Panda

Global

Reduction

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Exchange

Worms

In

International

Parallel

Processing

Symposium

pages

-LCB-

Apr

-LSB-

-RSB-

D.

K.

Panda

Issues

in

Designing

Efficient

and

Practical

Algorithms

for

Collective

Communication

in

Wormhole-Routed

Systems

In

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

-LSB-

-RSB-

Z.

Xu

and

K.

Hwang

Modeling

Communication

Overhead

MPI

and

MPL

Performance

on

the

IBM

IEEE

Parallel

and

Distributed

Technology

pages

-LCB-

Spring

O B O O O O O O B I I O O O O O O O O O O O O B O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I I O O O O O O O B I I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

collective communication
multidestination message passing
personalized communication
scatter
wormhole routing


Efficient Multicast in IP/ATM Networks Walid Mostafa Mukesh Singhal Department of Computer and Information Science The Ohio State University Columbus , OH 43210 email : -LRB- mostafa , singhal -RRB- @cis . ohio-state . edu Phone : 614-292-5839 Fax : 614-292-2991 Abstract Real-time transport protocols rely on IP to disseminate audio/video packets from a sender to one or more receivers . In IP over ATM broadband networks , studies have shown that the effective bandwidth is reduced by at least 10 % and that larger protocol payload sizes yield higher throughput . However , a dropped cell corrupts the IP packet it belongs to and reduces the rate successfully-received data , or goodput . This results in quality of service degradation to multimedia traffic , particularly for intra-frame encoded video raffic that is sensitive to a packet loss . A key observation for multimedia applications is that achieving maximum throughput versus reaching optimum goodput is determined by the end-to-end quality of service requirements . In this report , we derive analytic expressions for the maximum throughput and packet-loss probability which provides us with a better understanding of the goodput of real-time transport protocols . We also derive an analytic expression for optimum PDU that maximizes the goodput with respect to cell-loss probability and number of ATM switches . We show that optimum PDUs yield higher goodput than the default MTU for IP over ATM , particularly as the cell-loss probability and/or the number of ATM switches increase . For reliable multicast , We also derive analytic expressions to the expected total number of transmissions per packet and to the goodput of reliable multicast . 1 Introduction The continuous growth in multimedia applications over the Internet , e.g , video conferencing -LSB- Jac94 -RSB- and distributed whiteboards -LSB- McC92 -RSB- , and the availability of high-speed broadband ATM networks have led to considerable interests in the interoperability of IP and ATM -LSB- Lau94 , CSV96 -RSB- . Real-time transport protocols , e.g. , RTP -LSB- SCFJ96 , Sch96 -RSB- , cater for the needs of audio/video traffic of multimedia applications -LSB- HFG96 , BFFM96 , SH96 , Vau96 -RSB- . These protocols provide high throughput because they neither guarantee delivery or ordering of transport packets nor provide flow control . Multimedia applications over the Internet disseminate audio/video traffic in real-time and are sensitive to network delays . However , they have various degrees of tolerance to packet loss . For example , the ITU recommendation H. 261 -LSB- IT93 -RSB- that uses the temporal redundancy of video to perform compression , is sensitive to packet loss . Thus , in specifying the RTP payload format for H. 261 video , Turletti and Muitema -LSB- TH96 -RSB- defined negative acknowledgement control packets to request packet retransmission . This approach mimics reliable transport protocols . Multimedia transport requires either high throughput of redundant data that tolerate packet loss or lower throughput of compressed data that is sensitive to packet loss . In other words , techniques that reduce multimedia bandwidth requirements increase its quality of service demands . This fundamental observation is the motivation to analyze the tradeoff between the maximum rate of transmitted traffic , throughput , at a source and the maximum rate of successfully-received traffic , goodput , at one or more destinations . In this report , we derive analytic expressions for the maximum throughput and packetloss probability which provide us with a better understanding of the goodput of real-time transport protocols . We also derive an analytic expression for the optimum protocol payload data unit size that maximizes the goodput of data transport with respect to cell-loss probability and number of ATM switches . We show that optimum PDUs yield higher goodput than the default MTU for IP over ATM -LRB- 9180 bytes -RRB- , specially as the cell-loss probability and/or the number of ATM switches increase . We also derive similar expressions to the goodput of reliable multicast where a packet is retransmitted until it is correctly received by all receivers . The rest of the report is organized as follows : Section 2 describes the protocol stack overhead for multimedia transport using RTP and IP over ATM . In Section 3 , we derive analytic expressions for maximum throughput , packet loss probability , and optimum PDU using real-time multicast with no retransmissions of lost packets . In Section 4 , we derive analytic expressions for the expected total number of retransmissions per lost packet and the goodput for reliable multicast . Section 5 presents numerical results for the analytic expressions derived earlier and compares them with simulation results . It also presents a goodput comparison between optimum PDU and default IP/ATM MTU . A summary is presented in Section 6 . 2 Protocol Stack Overhead In this section , we present the model and assumptions of multimedia transport using IP over ATM networks . The ATM layer reduces the available bandwidth to the higher protocol layer by at least 10 % due to ATM cell header overhead . Cavanaugh -LSB- Cav94 -RSB- evaluated the available bandwidth to an application using Internet protocols and ATM over a variety of physical layers . Other studies evaluated the performance of Internet protocols over ATM -LSB- RF94 , LO95 , LFEN96 -RSB- . The real-time transport protocol , RTP , provides end-to-end network transport services . It supports both point-to-point and point-to-multipoint transport . It does not address resource reservation and does not guarantee quality of service for real-time services . It provides a control protocol to allow monitoring of data delivery and to provide minimal control and identification functionality . Figure 1 shows the various protocol layers that are involved in multimedia data transmission using RTP . The application data is referred to as RTP payload -LRB- PDU -RRB- . RTP adds an

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O