Multicast

on

Irregular

Switch-based

Networks

with

Wormhole

Routing

Ram

Kesavan

Kiran

Bondalapati

and

Dhabaleswar

K.

Panda

Technical

Report

Manuscript

has

been

submitted

to

International

Symposium

on

High

Performance

Computer

Architecture

-LRB-

HPCA

-RRB-

Multicast

on

Irregular

Switch-based

Networks

with

Wormhole

Routing

Ram

Kesavan

Kiran

Bondalapati

and

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

E-mail

fkesavan,bondalap,pandag@cis.ohio-state.edu

Contact

Author

Dhabaleswar

K.

Panda

Abstract

This

paper

presents

efficient

multicasting

with

reduced

contention

on

irregular

networks

with

switch-based

wormhole

interconnection

and

unicast

message

passing

First

it

is

proved

that

for

an

arbitrary

irregular

network

with

a

typical

deadlock-free

adaptive

routing

it

may

not

be

possible

to

create

an

ordered

list

of

nodes

to

implement

an

arbitrary

multicast

in

a

contention-free

manner

with

minimal

number

of

communication

steps

Next

three

different

multicast

algorithms

are

proposed

with

their

respective

node

orderings

to

reduce

contention

switch-based

ordering

-LRB-

SO

-RRB-

switch-based

hierarchical

ordering

-LRB-

SHO

-RRB-

and

chain

concatenation

ordering

-LRB-

CCO

-RRB-

A

variation

of

a

binomial

tree-based

communication

pattern

with

unicast

message

passing

is

used

on

the

above

ordered

lists

to

implement

multicast

The

proposed

multicast

algorithms

are

compared

with

each

other

as

well

as

with

the

naive

random

ordering

-LRB-

RO

-RRB-

algorithm

for

a

range

of

system

sizes

switch

sizes

message

lengths

degrees

of

connectivity

destination

set

sizes

and

communication

start-up

times

With

larger

system

size

and

larger

destination

set

size

the

CCO

algorithm

is

shown

to

be

the

best

to

implement

multicast

with

reduced

contention

and

minimum

latency

The

SHO

algorithm

emerges

as

the

second

best

Such

results

related

to

multicast

on

irregular

networks

are

the

first

of

their

kind

in

the

wormhole

literature

Thus

these

results

demonstrate

significant

potential

to

be

applied

to

current

and

future

generation

NOW

systems

with

irregular

interconnection

research

is

supported

in

part

by

NSF

Grant

and

NSF

Career

Award

Contents

Introduction

Irregular

Networks

Network

Model

Routing

Issues

Contention-Free

Multicast

in

Irregular

Networks

Contention-Free

Multicast

with

Ordered

Chain

Non-existence

of

Ordered

Chain

in

Irregular

Networks

Multicast

Algorithms

Random

Ordering

-LRB-

RO

-RRB-

Algorithm

Switch-based

Ordering

-LRB-

SO

-RRB-

Algorithm

Switch-based

Hierarchical

Ordering

-LRB-

SHO

-RRB-

Algorithm

Chain

Concatenation

Ordering

-LRB-

CCO

-RRB-

Algorithm

Simulation

Experiments

and

Results

Effect

of

system

size

Effect

of

message

length

Effect

of

communication

startup

time

Effect

of

switch

size

Effect

of

degree

of

network

connectivity

Related

Work

Conclusions

Introduction

Multicast/broadcast

is

a

common

collective

communication

operation

as

defined

by

the

MPI

standard

-LSB-

-RSB-

Parallel

systems

supporting

distributed

memory

or

distributed-shared

memory

programming

paradigms

require

fast

implementation

of

multicast

and

broadcast

operations

in

order

to

support

various

application

and

system

level

data

distribution

functions

Multicast

and

broadcast

also

get

used

for

other

collective

communication

operations

like

barrier

synchronization

and

global

combining

-LSB-

-RSB-

Current

generation

parallel

systems

like

IBM

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

Cray

-LSB-

-RSB-

-LSB-

-RSB-

J-Machine

-LSB-

-RSB-

and

Stanford

FLASH

-LSB-

-RSB-

use

the

wormhole-routing

switching

technique

due

to

its

inherent

advantages

like

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

These

systems

use

regular

network

topologies

with

various

deadlock-free

routing

schemes

For

such

regular

wormhole

networks

many

multicast/broadcast

algorithms

have

been

proposed

in

the

literature

in

recent

years

-LSB-

-RSB-

More

recently

wormhole

routing

is

being

applied

to

switch-based

interconnects

like

Myrinet

-LSB-

-RSB-

and

ServerNet

-LSB-

-RSB-

to

build

networks

of

workstations

for

cost-effective

parallel

computing

Switch-based

networks

are

typically

built

for

supporting

irregular

topology

Such

irregularity

provides

flexibility

to

design

scalable

systems

with

incremental

expansion

capability

The

irregularity

also

makes

the

routing

on

such

systems

quite

complicated

Typically

deadlock-free

adaptive

routing

schemes

-LSB-

-RSB-

are

supported

on

such

systems

to

achieve

low-latency

and

high-bandwidth

communication

Such

routing

flexibility

also

leads

to

difficulty

in

implementing

a

multicast/broadcast

operation

in

a

contention-free

manner

Multicast

algorithms

are

typically

hierarchical

in

nature

to

achieve

reduced

latency

In

these

algorithms

some

nodes

work

as

intermediate

nodes

which

receive

a

copy

of

the

message

from

the

source

and

forward

it

to

other

nodes

Typically

tree-structured

algorithms

are

used

to

minimize

the

number

of

communication

start-ups

-LRB-

steps

-RRB-

required

for

multicast

-LSB-

-RSB-

The

efficiency

of

an

algorithm

is

determined

by

the

required

number

of

start-ups

for

a

multicast

to

complete

and

the

degree

of

contention

experienced

among

the

messages

of

the

multicast

For

regular

networks

with

e-cube

routing

the

concept

of

a

dimension-ordered

chain

has

been

developed

-LSB-

-RSB-

to

implement

contention-free

multicast

with

minimum

latency

However

for

irregular

wormhole

networks

with

adaptive

routing

developing

such

contention-free

multicast

algorithms

is

a

non-trivial

task

To

the

best

of

our

knowledge

no

solutions

exist

in

the

literature

in

this

direction

In

this

paper

we

take

on

such

a

challenge

and

develop

efficient

multicast

algorithms

for

irregular

wormhole

networks

Without

loss

of

generality

we

consider

a

deadlock-free

routing

scheme

similar

to

that

used

in

DEC

networks

-LSB-

-RSB-

In

addition

to

providing

deadlockfreedom

this

routing

provides

adaptive

communication

between

nodes

in

an

irregular

network

With

respect

to

such

routing

we

first

prove

that

no

ordered

chain

similar

to

that

proposed

in

-LSB-

-RSB-

exists

to

implement

contention-free

multicast

in

-LRB-

d

-RRB-

e

steps

for

d

destinations

Next

we

develop

multicast

algorithms

which

-RRB-

minimize

the

number

of

communication

start-ups

-LRB-

steps

-RRB-

for

a

given

number

of

destinations

and

-RRB-

minimize

contention

among

the

communication

steps

We

assume

a

system

consisting

of

S

switches

with

k

ports

per

switch

We

propose

three

different

multicast

algorithms

with

their

respective

orderings

of

destinations

The

first

algorithm

switch-based

ordering

-LRB-

SO

-RRB-

groups

the

destinations

based

on

the

switches

to

which

they

are

connected

to

generate

an

ordered

list

of

destinations

This

algorithm

implements

multicast

with

-LRB-

d

-RRB-

e

steps

with

contention

among

the

steps

The

second

algorithm

switch-based

hierarchical

ordering

-LRB-

SHO

-RRB-

provides

enhancement

by

using

a

two-step

hierarchical

multicast

-LRB-

inter-switch

and

intra-switch

-RRB-

This

algorithm

implements

a

multicast

with

-LRB-

-LRB-

L

-RRB-

e

-LRB-

k

-RRB-

e

-RRB-

steps

where

a

leader

node

set

of

size

L

is

generated

after

grouping

the

destinations

based

on

the

switches

This

algorithm

guarantees

that

the

final

-LRB-

k

-RRB-

e

intraswitch

steps

are

contention-free

Finally

we

propose

a

chain

concatenation

ordering

-LRB-

CCO

-RRB-

algorithm

For

a

given

network

and

a

set

of

destinations

this

algorithm

first

determines

chains

of

switches

-LRB-

defined

as

partial-ordered-chains

or

POCs

-RRB-

which

can

implement

multicast

within

themselves

in

a

contention-free

manner

A

depth-first-search

on

the

routing

graph

is

used

to

generate

such

POCs

Finally

these

POCs

are

concatenated

to

generate

the

overall

ordered

list

in

order

to

minimize

contention

The

above

three

algorithms

are

evaluated

on

a

simulator

testbed

In

addition

to

comparing

these

algorithms

with

each

other

they

are

compared

against

a

naive

random

ordering

-LRB-

RO

-RRB-

algorithm

The

evaluation

is

done

for

a

wide

range

of

system

sizes

switch

sizes

message

lengths

degrees

of

connectivity

destination

set

sizes

and

communication

start-up

times

In

most

of

the

cases

it

is

demonstrated

that

the

CCO

algorithm

is

capable

of

implementing

multicast

with

reduced

latency

Detailed

simulation

analysis

also

exhibits

that

the

CCO

algorithm

leads

to

minimum

contention

among

the

multicast

steps

The

SHO

algorithm

does

not

perform

well

for

small

number

of

destinations

However

for

large

number

of

destinations

in

many

places

it

comes

closer

to

the

performance

of

the

CCO

algorithm

For

small

number

of

destinations

and

system

sizes

the

RO

SO

and

CCO

algorithms

are

shown

to

perform

equally

well

Thus

the

results

indicate

that

the

CCO

algorithm

can

be

adopted

on

any

irregular

wormhole

network

to

implement

efficient

multicasting

The

rest

of

the

paper

is

organized

as

follows

Section

provides

an

overview

of

irregular

networks

and

some

associated

issues

related

to

routing

Section

shows

why

implementing

contention-free

multicast

in

irregular

networks

is

a

non-trivial

problem

Section

presents

the

four

multicast

algorithms

in

detail

Simulation

experiments

and

results

comparing

the

relative

merits

of

the

multicasting

schemes

are

presented

in

Section

Related

works

in

the

literature

are

described

in

Section

Finally

concluding

remarks

are

made

in

Section

Irregular

Networks

In

this

section

we

provide

an

irregular

switch-based

wormhole

network

model

The

associated

issues

related

to

deadlock-free

adaptive

routing

for

such

a

network

are

discussed

Network

Model

Figure

-LRB-

a

-RRB-

shows

a

typical

parallel

system

using

switch-based

interconnect

with

irregular

topology

Such

a

network

consists

of

a

set

of

switches

where

each

switch

can

have

a

set

of

ports

The

system

in

the

figure

consists

of

eight

switches

with

eight

ports

per

switch

Some

of

the

ports

in

each

switch

are

connected

to

processors

where

as

the

remaining

ports

are

connected

to

ports

of

other

switches

to

provide

connectivity

between

the

processors

Such

connectivity

is

typically

irregular

and

the

only

thing

that

is

guaranteed

is

that

the

network

is

connected

Thus

the

interconnection

topology

of

the

network

can

be

denoted

by

a

graph

G

-LRB-

V

E

-RRB-

where

V

is

the

set

of

switches

and

E

is

the

set

of

bidirectional

links

between

the

switches

-LSB-

-RSB-

Figure

-LRB-

b

-RRB-

shows

the

interconnection

graph

for

the

irregular

network

in

Fig.

-LRB-

a

-RRB-

It

is

to

be

noted

that

all

links

are

bidirectional

and

multiple

links

between

two

switches

are

allowed

A

typical

switch-based

irregular

network

can

be

described

by

using

the

following

parameters

P

number

of

processors

S

number

of

switches

k

number

of

ports

per

switch

f

fraction

of

the

number

of

ports

per

switch

which

are

connected

to

processors

P

f

S

k

c

percentage

connectivity

out

of

remaining

-LRB-

f

-RRB-

SK

ports

for

interconnection

Without

loss

of

generality

we

assume

f

in

this

paper

so

ports

of

each

switch

are

connected

to

processors

Such

a

configuration

allows

a

system

with

a

given

number

of

processors

to

be

built

using

less

number

of

switches

while

allowing

reasonable

number

of

external

Processing

Element

-LRB-

Workstation

-RRB-

Switches

Bidirectional

Link

Figure

a

-RRB-

An

example

system

with

switch-based

interconnect

and

irregular

topology

b

-RRB-

corresponding

interconnection

graph

G.

communication

ports

per

processor

-LSB-

-RSB-

We

vary

c

in

our

model

to

provide

different

types

of

irregular

connectivity

Figure

shows

a

generic

switch

with

k

ports

Each

port

consists

of

one

input

and

one

output

link

As

shown

in

Fig.

-LRB-

a

-RRB-

a

port

can

be

connected

to

the

port

of

another

switch

a

processor

or

kept

open

The

switch

can

implement

different

types

of

switching

techniques

wormhole

virtual

cut-through

or

ATM

In

this

paper

we

restrict

ourselves

to

switches

implementing

wormhole

routing

-LSB-

-RSB-

Each

port

consists

of

an

input

and

an

output

flit

buffer

These

buffers

are

big

enough

to

capture

the

header

flit

of

an

incoming

worm

so

that

the

routing

decision

can

be

made

as

soon

as

the

header

flit

arrives

A

k-port

switch

typically

provides

a

k

k

crossbar

connectivity

in

order

to

enable

a

message

from

any

input

flit

buffer

to

be

transfered

to

any

output

flit

buffer

However

in

many

instances

some

routing

restrictions

are

used

to

achieve

deadlock-free

routing

We

consider

some

of

these

issues

in

the

following

subsection

SwitchCrossbar

Input

Links

Output

Links

Input

Flit

Buffer

Output

Flit

Buffer

k

k

Figure

Organization

of

a

typical

k-port

switch

supporting

wormhole

routing

Routing

Issues

Several

deadlock-free

routing

schemes

have

been

proposed

in

the

literature

for

irregular

networks

-LSB-

-RSB-

Without

loss

of

generality

in

this

paper

we

assume

the

routing

scheme

for

our

irregular

network

to

be

similar

to

that

used

in

Autonet

-LSB-

-RSB-

due

to

its

simplicity

and

its

commercial

implementation

Such

routing

allows

deadlock-freedom

as

well

as

adaptivity

In

this

routing

scheme

a

breadth-first

spanning

tree

-LRB-

BFS

-RRB-

on

the

graph

G

is

first

computed

using

a

distributed

algorithm

The

algorithm

has

the

property

that

all

nodes

will

eventually

agree

on

a

unique

spanning

tree

Now

the

edges

of

G

can

be

partitioned

into

tree

edges

and

cross

edges

According

to

the

property

of

BFS

trees

a

cross

edge

does

not

connect

two

switches

which

are

at

a

difference

of

more

than

one

level

in

the

tree

Deadlock-free

routing

is

based

on

a

loop-free

assignment

of

direction

to

the

operational

links

In

particular

the

up

end

of

each

link

is

defined

as

-RRB-

the

end

whose

switch

is

closer

to

the

root

in

the

spanning

tree

-RRB-

the

end

whose

switch

has

the

lower

UID

if

both

ends

are

at

switches

at

the

same

tree

level

Links

looped

back

to

the

same

switch

are

omitted

from

the

configuration

The

result

of

this

assignment

is

that

the

directed

links

do

not

form

loops

Figure

-LRB-

a

-RRB-

shows

the

BFS

spanning

tree

corresponding

to

the

example

irregular

network

shown

in

Fig.

-LRB-

a

-RRB-

The

assignment

of

up

direction

to

the

links

on

this

network

is

illustrated

The

down

direction

is

along

the

reverse

direction

of

the

link

up

direction

root

Figure

BFS

spanning

tree

rooted

at

node

corresponding

to

the

example

irregular

network

shown

in

Fig.

To

eliminate

deadlocks

while

still

allowing

all

links

to

be

used

this

routing

uses

the

following

up/down

rule

a

legal

route

must

traverse

zero

or

more

links

in

the

up

direction

followed

by

zero

or

more

links

in

the

down

direction

Putting

it

in

the

negative

a

packet

may

never

traverse

a

link

along

the

up

direction

after

having

traversed

one

in

the

down

direction

Details

of

this

routing

scheme

can

be

found

in

-LSB-

-RSB-

In

order

to

implement

the

above

routing

each

switch

has

an

indexed

forwarding

table

When

a

worm

reaches

a

switch

the

destination

address

is

captured

from

the

header

flit

of

the

incoming

worm

This

address

is

concatenated

with

the

incoming

port

number

and

the

result

is

used

to

index

the

switch

's

forwarding

table

The

table

lookup

returns

the

outgoing

port

number

that

the

worm

should

be

routed

through

The

forwarding

tables

can

be

constructed

to

support

both

minimal

and

non-minimal

adaptive

routing

In

this

paper

we

only

consider

minimal

adaptive

routing

Thus

the

forwarding

tables

allow

only

legal

routes

with

the

minimum

hop

count

When

multiple

minimal

routes

exist

from

the

source

to

the

destination

the

forwarding

table

entries

show

alternative

forwarding

ports

The

choice

of

the

outgoing

port

is

decided

dynamically

based

on

the

ports

which

are

free

when

the

header

flits

arrive

at

the

switch

In

case

multiple

outgoing

ports

are

free

the

routing

scheme

randomly

selects

one

of

them

Contention-Free

Multicast

in

Irregular

Networks

In

this

section

we

discuss

the

significance

of

ordered

chains

to

achieve

contention-free

multicast

with

optimal

number

of

communication

steps

We

prove

that

there

does

not

exist

an

ordered

chain

of

nodes

to

implement

contention

free

multicast

with

binomial

tree-based

message

pattern

on

an

arbitrary

irregular

network

with

the

routing

scheme

discussed

in

Section

Contention-Free

Multicast

with

Ordered

Chain

Typically

binomial

tree-based

algorithms

have

been

used

in

the

literature

-LSB-

-RSB-

to

implement

multicast

on

meshes

tori

and

hypercubes

with

optimal

number

of

communication

start-ups

-LRB-

steps

-RRB-

Such

an

approach

requires

-LRB-

d

-RRB-

e

communication

steps

for

a

multicast

with

d

destinations

Besides

the

number

of

startups

an

important

factor

which

affects

the

overall

multicast

latency

is

the

contention

that

messages

undergo

between

different

steps

of

the

binomial

tree-based

algorithm

In

-LSB-

-RSB-

it

has

been

shown

that

if

an

ordered

chain

can

be

generated

among

the

nodes

participating

in

the

multicast

a

contention

free

binomial

multicast

tree

can

be

constructed

Let

the

symbol

d

denote

such

an

ordering

Such

ordered

chains

typically

exhibit

the

following

property

Property

-LSB-

-RSB-

If

there

exist

four

nodes

w

x

y

and

z

in

an

ordered

chain

such

that

w

d

x

d

y

d

z

then

messages

between

processors

w

and

x

will

not

contend

for

any

links

with

messages

between

processors

y

and

z.

Non-existence

of

Ordered

Chain

in

Irregular

Networks

Using

the

above

property

the

contention-free

multicast

problem

in

irregular

networks

reduces

to

generating

an

ordered

chain

among

the

participating

nodes

However

in

switch-based

networks

concurrent

communication

between

the

processors

connected

to

a

switch

are

contention-free

Thus

the

above

problem

further

reduces

to

generating

an

ordered

chain

among

participating

switches

where

a

participating

switch

is

defined

as

a

switch

having

at

least

one

node

connected

to

it

which

is

participating

in

the

multicast

In

the

worst

case

of

a

broadcast

an

ordered

chain

consisting

of

all

the

switches

in

the

network

must

be

generated

This

chain

can

be

easily

reduced

to

generate

the

ordered

chain

for

an

arbitrary

multicast

The

following

theorem

indicates

that

it

is

not

always

possible

to

generate

such

an

ordered

chain

for

an

arbitrary

irregular

network

Theorem

Given

an

arbitrary

irregular

network

using

the

deadlock-free

adaptive

routing

discussed

in

Section

there

does

not

always

exist

an

ordered

chain

satisfying

Property

consisting

of

all

the

switches

in

the

network

Proof

Consider

an

irregular

network

with

the

routing

scheme

as

discussed

in

Section

Let

the

graph

G

reflect

the

connectivity

between

the

participating

switches

for

a

broadcast

Let

us

take

five

switches

in

the

BFS

spanning

tree

of

G

such

that

the

subgraph

in

Fig

-LRB-

a

-RRB-

shows

their

relative

positions

in

the

BFS

tree

Let

there

be

no

cross

links

incident

on

the

switches

It

can

be

easily

seen

that

the

shortest

valid

route

from

switch

si

to

switch

sj

is

along

the

links

of

where

i

j

We

claim

that

any

ordered

chain

in

G

containing

the

switches

to

must

have

either

-LSB-

-RSB-

p

p

-LSB-

-RSB-

or

-LSB-

-RSB-

p

p

-LSB-

-RSB-

We

prove

this

by

contradiction

In

this

discussion

the

square

brackets

indicate

that

the

relative

ordering

of

the

switches

enclosed

within

square

brackets

is

not

important

If

-LSB-

-RSB-

p

p

then

a

message

from

a

processor

connected

to

the

switch

to

a

processor

connected

to

the

switch

will

contend

for

the

link

e

with

a

message

from

a

processor

connected

to

the

switch

to

a

processor

connected

to

the

switch

This

scenario

is

shown

in

Fig

-LRB-

b

-RRB-

This

violates

the

Property

of

ordered

chains

Similarily

it

can

be

proved

that

-LSB-

-RSB-

p

p

can

not

exist

Thus

any

ordered

chain

in

G

containing

the

switches

to

must

have

either

-LSB-

-RSB-

p

p

-LSB-

-RSB-

or

-LSB-

-RSB-

p

p

-LSB-

-RSB-

Now

let

us

take

seven

switches

to

in

the

BFS

spanning

tree

of

G

such

that

the

subgraph

in

Fig

-LRB-

c

-RRB-

shows

their

relative

positions

in

the

BFS

tree

Let

there

be

no

cross

links

incident

on

the

switches

to

excepting

Using

the

above

reasoning

any

ordered

chain

of

G

containing

the

switches

to

must

have

between

the

switches

and

between

the

switches

and

and

between

the

switches

and

with

respect

to

the

symbol

p

It

can

be

easily

observed

that

such

an

ordered

chain

is

impossible

to

generate

Therefore

there

exists

no

ordered

chain

for

an

arbitrary

irregular

graph

with

the

routing

discussed

in

Section

-LRB-

a

-RRB-

n

worm

e

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Figure

A

part

of

an

example

BFS

tree

with

five

switches

-LRB-

a

-RRB-

their

relative

positions

in

the

BFS

tree

-LRB-

b

-RRB-

and

a

possible

scenario

of

contention

-LRB-

c

-RRB-

A

part

of

an

example

BFS

tree

with

seven

switches

Since

it

is

not

possible

to

construct

an

ordered

chain

it

is

a

non-trivial

problem

to

implement

contention-free

multicast

with

optimal

number

of

communication

steps

in

irregular

networks

Thus

in

the

next

section

we

propose

alternative

ordering

schemes

and

the

associated

multicast

algorithms

to

implement

multicast

with

reduced

contention

as

well

as

with

minimum

number

of

steps

Multicast

Algorithms

In

this

section

we

present

several

multicast

algorithms

A

naive

random

ordering

algorithm

is

introduced

first

Then

we

propose

three

new

algorithms

with

capability

for

reduced

contention

during

multicast

These

multicast

algorithms

are

illustrated

with

examples

to

demonstrate

their

performance

and

contention-freedom

capability

Random

Ordering

-LRB-

RO

-RRB-

Algorithm

Let

the

source

of

a

multicast

be

s

and

the

destination

processors

be

in

a

set

D

The

naive

RO

algorithm

randomly

orders

the

elements

of

D

-LSB-

fsg

into

a

list

and

performs

the

Umesh

algorithm

-LSB-

-RSB-

on

it

This

algorithm

is

very

simple

to

implement

and

it

takes

-LRB-

jDj

-RRB-

e

communication

start-ups

-LRB-

steps

-RRB-

to

complete

Since

the

destinations

and

the

source

are

ordered

randomly

nothing

can

be

said

about

contention

among

messages

of

the

multicast

Therefore

with

increasing

jDj

this

algorithm

is

prone

to

severe

contention

Let

us

consider

a

sample

multicast

shown

in

Fig.

on

the

example

irregular

network

of

Fig.

-LRB-

a

-RRB-

Processor

is

its

source

and

g

is

its

destination

set

Figure

-LRB-

a

-RRB-

shows

the

multicast

tree

generated

using

the

RO

algorithm

for

the

sample

multicast

It

also

shows

the

list

which

is

a

random

ordering

of

the

elements

of

D

-LSB-

fsg

for

the

multicast

For

this

sample

destination

set

the

multicast

takes

three

communication

steps

Source

Destination

Figure

A

sample

multicast

destination

set

on

the

example

irregular

network

Switch-based

Ordering

-LRB-

SO

-RRB-

Algorithm

Let

the

source

of

a

multicast

be

s

and

the

destination

processors

be

in

a

set

D

The

SO

algorithm

sorts

the

elements

of

D

-LSB-

fsg

into

a

list

such

that

participating

processors

on

the

same

switch

appear

adjacent

to

each

other

in

This

is

done

by

doing

a

switch-based

grouping

of

the

processors

and

then

randomly

ordering

these

groups

into

the

list

Similar

to

the

RO

algorithm

the

Umesh

algorithm

-LSB-

-RSB-

is

now

performed

on

Figure

-LRB-

b

-RRB-

shows

the

multicast

tree

generated

using

the

SO

algorithm

for

the

sample

multicast

shown

in

Fig.

on

an

irregular

network

It

also

details

the

list

for

the

multicast

For

this

sample

destination

set

the

multicast

takes

three

communication

steps

Like

the

RO

algorithm

the

SO

algorithm

takes

-LRB-

jDj

-RRB-

e

start-ups

to

complete

However

it

reduces

contention

compared

to

the

RO

algorithm

In

the

latter

phases

of

the

Umesh

algorithm

nodes

send

messages

to

their

neighboring

nodes

in

Due

to

the

grouping

there

is

a

higher

probability

of

these

communications

taking

place

between

processors

on

the

same

switch

This

reduces

inter-switch

traffic

considerably

during

the

latter

phases

of

the

multicast

when

the

number

of

messages

is

quite

large

Intra-switch

messages

do

not

contribute

to

contention

since

these

messages

do

not

use

inter-switch

links

Therefore

the

SO

algorithm

promises

better

performance

compared

to

the

RO

algorithm

L

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

a

-RRB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

b

-RRB-

L

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

c

-RRB-

Figure

Multicast

trees

for

the

sample

multicast

destination

set

using

algorithms

a

-RRB-

RO

b

-RRB-

SO

and

c

-RRB-

SHO

Switch-based

Hierarchical

Ordering

-LRB-

SHO

-RRB-

Algorithm

The

SHO

algorithm

uses

the

concepts

of

leader

and

hierarchy

to

guarantee

contention

freedom

in

the

latter

phases

of

the

multicast

Let

the

source

of

a

multicast

be

s

and

the

destination

processors

be

in

a

set

D.

Let

be

the

set

D

-LSB-

fsg

This

set

can

be

partitioned

into

disjoint

subsets

so

that

each

subset

is

represented

by

a

leader

node

This

partitioning

is

done

in

a

way

such

that

all

participating

processors

connected

to

a

switch

form

a

disjoint

subset

For

subsets

not

containing

the

source

node

s

the

processor

with

the

least

UID

within

the

subset

is

chosen

as

the

leader

node

The

source

node

s

is

chosen

as

the

leader

node

of

its

subset

A

list

is

formed

by

randomly

ordering

all

the

leader

nodes

The

multicast

takes

place

in

two

stages

The

first

stage

involves

performing

the

Umesh

algorithm

-LSB-

-RSB-

on

the

elements

of

the

list

This

stage

takes

start-ups

to

complete

It

is

to

be

noted

that

there

is

no

contention

freedom

guaranteed

during

this

stage

During

the

second

stage

each

leader

node

does

a

multicast

over

its

associated

subset

members

using

the

Umesh

algorithm

This

stage

of

the

algorithm

takes

-LRB-

f

k

-RRB-

e

start-ups

to

complete

Since

this

stage

of

the

multicast

consists

solely

of

intra-switch

messages

they

do

not

experience

any

contention

with

other

messages

Therefore

the

SHO

algorithm

has

reduced

contention

compared

to

the

SO

algorithm

However

the

SHO

algorithm

takes

-LRB-

-RRB-

e

-LRB-

f

k

-RRB-

e

start-ups

which

could

be

more

than

the

number

of

start-ups

for

the

SO

algorithm

for

small

values

of

jDj

This

advantage

is

offset

as

the

the

size

of

jDj

increases

and

the

message

length

increases

Figure

-LRB-

c

-RRB-

shows

the

multicast

tree

generated

using

the

SHO

algorithm

for

the

sample

multicast

shown

in

Fig.

It

also

shows

the

list

for

the

multicast

The

communication

steps

are

identified

by

-LSB-

i

j

-RSB-

where

i

corresponds

to

the

step

number

-LRB-

as

in

the

examples

for

the

RO

and

SO

algorithms

-RRB-

and

j

corresponds

to

the

stage

number

For

this

sample

destination

set

the

multicast

takes

four

steps

to

finish

Chain

Concatenation

Ordering

-LRB-

CCO

-RRB-

Algorithm

The

above

three

algorithms

do

not

attempt

to

reduce

contention

during

the

inter-switch

multicast

steps

In

order

to

reduce

such

contention

we

use

a

new

concept

of

partial

ordered

chain

-LRB-

POC

-RRB-

to

order

the

participating

switches

A

POC

is

formally

defined

as

Definition

A

partial

ordered

chain

-LRB-

POC

-RRB-

is

defined

as

an

ordered

list

of

a

subset

of

the

switches

in

an

arbitrary

irregular

network

satisfying

Property

As

proved

in

Theorem

there

does

not

exist

a

global

ordered

chain

among

the

switches

of

an

arbitrary

irregular

network

with

deadlock-free

adaptive

routing

discussed

in

Section

Thus

we

attempt

to

construct

as

many

longest

POCs

as

possible

and

concatenate

them

to

form

an

overall

ordering

Such

a

concatenated

chain

promises

reduced

contention

during

multicast

steps

The

following

theorem

suggests

a

method

of

constructing

POCs

on

an

irregular

network

with

the

routing

scheme

discussed

in

Section

Theorem

Let

P

be

any

ordered

list

of

switches

sn

where

si

is

connected

to

si

by

a

down

tree

link

-LRB-

from

the

BFS

spanning

tree

-RRB-

or

a

down

cross

link

connecting

switches

at

different

levels

of

the

BFS

spanning

tree

Then

P

forms

a

partial

ordered

chain

-LRB-

POC

-RRB-

Proof

Let

us

use

the

symbol

p

to

denote

the

order

in

the

above

list

P

Therefore

si

p

si

Let

E

en

denote

the

list

of

down

links

such

that

switch

si

is

connected

to

the

switch

si

by

the

down

link

ei

where

si

and

si

P

A

message

from

a

processor

connected

to

the

switch

si

to

a

processor

connected

to

the

switch

sj

where

si

p

sj

will

take

only

the

links

from

P

if

all

the

links

ei

ei

ej

are

down

tree

links

of

the

BFS

spanning

tree

Figure

-LRB-

a

-RRB-

shows

the

only

other

possible

minimal

route

for

worm

wi

j

from

si

to

sj

The

switches

in

P

are

highlighted

This

scenario

can

not

occur

in

a

BFS

tree

because

during

the

construction

of

the

tree

either

the

cross

link

em

would

be

a

tree

edge

of

sub-tree

Tm

or

the

cross

link

en

would

be

a

tree

edge

of

sub-tree

Tn

However

if

there

is

a

cross

link

ec

in

the

list

fei

ei

ej

then

a

message

from

a

processor

connected

to

si

to

a

processor

connected

to

sj

can

take

links

that

are

not

in

P

as

shown

in

Fig.

-LRB-

b

-RRB-

In

the

figure

worm

wi

j

takes

links

not

in

the

list

E

These

links

can

not

be

taken

by

a

message

from

sk

to

sl

where

sj

p

sk

p

sl

This

is

because

these

links

are

at

higher

level

than

sk

and

any

minimal

path

from

sk

to

sl

can

not

contain

these

links

Therefore

P

is

a

partial

ordered

chain

sj

Tn

Tm

em

en

si

-LRB-

a

-RRB-

sj

ec

si

-LRB-

b

-RRB-

s

j

s

i

-LRB-

c

-RRB-

wi

j

sa

sb

wa

b

scs

d

Figure

Possible

minimal

paths

from

si

to

sj

which

take

links

not

in

P

a

-RRB-

all

links

between

si

and

sj

in

P

are

down

tree

links

of

the

BFS

spanning

tree

b

-RRB-

one

cross

link

between

si

and

sj

in

P

c

-RRB-

an

example

of

contention

between

messages

of

two

disjoint

POCs

The

CCO

algorithm

constructs

as

many

longest

POCs

as

possible

from

the

participating

processors

concatenates

the

POCs

and

executes

the

Umesh

algorithm

on

this

concatenated

list

Such

an

approach

promises

to

minimize

the

contention

because

a

-RRB-

messages

within

a

POC

do

not

contend

with

each

other

and

b

-RRB-

messages

within

one

POC

contend

minimally

with

messages

within

another

disjoint

POC

An

example

of

the

latter

situation

is

given

in

Fig.

-LRB-

c

-RRB-

In

the

figure

switches

in

two

POCs

P

and

P

are

highlighted

with

different

shading

The

worm

wi

j

going

from

si

to

sj

takes

links

that

are

not

in

E.

Therefore

there

is

contention

between

the

worms

wi

j

and

wa

b

for

the

links

between

switches

sc

and

sd

As

the

example

illustrates

this

kind

of

contention

only

occurs

when

messages

within

a

POC

take

links

which

are

not

contained

in

the

POC

A

formal

specification

of

the

CCO

algorithm

is

given

in

Fig.

as

a

six-step

approach

In

the

first

step

a

depth-first-search

-LRB-

DFS

-RRB-

is

applied

on

the

BFS

spanning

tree

discussed

in

Section

starting

with

the

root

node

r

and

considering

only

the

down

links

specified

in

Theorem

This

is

to

facilitate

the

construction

of

the

longest

POCs

The

step

results

in

a

DAG

T

Figure

-LRB-

a

-RRB-

shows

the

DAG

T

which

is

created

when

the

above

DFS

is

applied

on

the

BFS

tree

in

Fig.

Like

in

the

SHO

algorithm

a

participating

switch

is

defined

as

one

with

at

least

one

participating

processor

connected

to

it

In

the

third

step

the

resultant

DAG

T

from

the

DFS

is

reduced

to

a

DAG

T

which

contains

only

the

participating

switches

Figure

-LRB-

b

-RRB-

shows

the

T

created

when

the

T

from

Fig.

-LRB-

a

-RRB-

is

reduced

In

order

to

determine

the

longest

POCs

and

concatenate

them

to

form

an

overall

ordered

list

we

carry

out

a

weighted

descendents

approach

As

indicated

in

step

each

switch

is

The

CCO

Algorithm

Inputs

G

irregular

graph

B

a

breadth

first

spanning

tree

on

the

switches

of

G

D

set

of

destinations

s

the

source

Output

Send

the

message

to

all

destinations

with

minimum

contention

and

minimum

number

of

start-ups

Procedure

Apply

the

DFS

on

B

from

the

root

node

r

of

B.

Only

down

tree

links

belonging

to

B

and

down

cross

links

which

connect

switches

in

different

levels

of

B

are

used

Let

the

resulting

DAG

be

T

S

fs

j

s

is

a

switch

in

G

and

p

D

-LSB-

fsg

such

that

p

is

connected

to

sg

S

is

the

set

of

participating

switches

Reduce

T

to

the

DAG

T

which

contains

all

switches

in

S

and

no

other

switches

Reduction

is

done

by

removing

all

non-participating

switches

from

T

Every

time

a

switch

s

and

all

links

incident

on

s

are

removed

directed

down

links

are

drawn

from

each

parent

of

s

to

each

child

of

s.

Calculate

the

weight

wt

of

each

switch

in

T

as

follows

wt

-LRB-

s

-RRB-

P

-LRB-

s

-RRB-

ae

-LRB-

si

-RRB-

where

Desc

-LRB-

s

-RRB-

is

the

set

containing

s

and

all

the

descendents

of

s

in

T

and

ae

-LRB-

s

-RRB-

is

the

number

of

participating

processors

connected

to

the

switch

s.

L

i

while

T

is

not

empty

-LRB-

a

-RRB-

Let

be

a

switch

with

maximum

weight

in

T

li

such

that

pj

is

the

switch

with

the

maximum

weight

among

all

children

switches

of

pj

-LRB-

b

-RRB-

T

T

fp

j

p

lig

-LRB-

c

-RRB-

where

fnj

nj

g

are

the

participating

processors

connected

to

switch

pj

in

li

-LRB-

d

-RRB-

L

concat

-LRB-

L

-RRB-

i

i

end

while

Apply

the

Umesh

algorithm

on

list

L.

Figure

Outline

of

the

CCO

algorithm

given

an

appropriate

weight

according

to

the

number

of

participating

processors

connected

to

it

and

to

all

its

descendent

switches

Therefore

the

child

with

the

largest

weight

indicates

how

to

proceed

while

building

the

longest

POC

from

the

parent

After

the

weights

have

been

calculated

chains

of

switches

are

stripped

off

from

T

according

to

their

weights

in

step

In

other

words

the

heaviest

chain

gets

stripped

first

from

T

and

the

lightest

the

last

These

chains

are

concatenated

together

in

chronological

order

and

each

switch

is

replaced

by

the

participating

processors

connected

to

it

to

form

L

The

chains

of

switches

stripped

off

from

the

T

in

Fig.

-LRB-

b

-RRB-

are

and

in

chronological

order

The

switches

in

and

are

replaced

by

the

participating

processors

connected

to

them

to

generate

the

POCs

and

The

POCs

and

are

concatenated

to

form

the

list

L.

Finally

the

Umesh

algorithm

is

performed

on

this

list

L

as

indicated

in

step

Figure

-LRB-

c

-RRB-

shows

the

resultant

multicast

tree

generated

in

step

and

the

list

L.

down

links

-LRB-

a

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

w

-RRB-

down

links

weight

-LRB-

b

-RRB-

L

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

c

-RRB-

Figure

Illustrating

the

steps

of

the

CCO

algorithms

on

the

multicast

set

of

Fig.

a

-RRB-

DAG

T

created

by

step

b

-RRB-

DAG

T

created

by

step

and

c

-RRB-

the

list

L

created

by

step

and

the

corresponding

multicast

tree

It

can

be

observed

that

the

CCO

algorithm

has

significant

potential

to

reduce

contention

compared

to

the

SHO

and

the

SO

algorithms

It

incorporates

the

grouping

effect

of

the

SO

algorithm

by

reducing

participating

processors

to

participating

switches

It

counteracts

the

extra

start-ups

due

to

the

hierarchical

effect

of

the

SHO

algorithm

by

expanding

the

switches

to

the

participating

processors

before

the

last

step

By

constructing

as

many

longest

POCs

as

possible

and

concatenating

them

together

the

contention

among

messages

within

POCs

is

eliminated

and

contention

among

messages

of

disjoint

POCs

is

minimized

The

CCO

algorithm

also

takes

only

jDje

steps

to

complete

Thus

this

algorithm

promises

potential

to

implement

a

multicast

with

minimum

number

communication

start-ups

as

well

as

reduced

contention

In

the

next

section

we

present

simulation

results

corresponding

to

these

four

algorithms

Simulation

Experiments

and

Results

We

performed

simulation

experiments

to

evaluate

and

compare

performance

of

the

above

algorithms

for

single

multicast

A

flit-level

wormhole-routed

simulator

WORMULSim

was

used

to

model

the

irregular

network

as

discussed

in

Section

This

simulator

was

designed

using

CSIM

-LSB-

-RSB-

For

all

simulation

experiments

we

assumed

system

and

technological

parameters

representing

the

current

trend

in

technology

The

following

parameters

were

used

ts

-LRB-

communication

start-up

time

-RRB-

microseconds

tphy

-LRB-

link

propagation

time

-RRB-

nanoseconds

tnode

-LRB-

router

delay

at

node

-RRB-

nanoseconds

tsw

-LRB-

switching

time

across

the

router

crossbar

for

a

flit

-RRB-

nanoseconds

tinj

-LRB-

time

to

inject

a

flit

into

network

-RRB-

nanoseconds

and

tcons

-LRB-

time

to

consume

a

flit

from

network

-RRB-

nanoseconds

For

all

experiments

-LRB-

unless

otherwise

stated

-RRB-

we

assumed

the

following

default

system

configuration

a

processor

system

interconnected

by

eight-port

switches

and

network

having

connectivity

as

defined

in

Section

The

default

message

length

was

assumed

to

be

flits

We

performed

several

experiments

to

study

the

impact

of

system

size

message

length

communication

start-up

time

switch

size

and

degree

of

connectivity

on

multicast

latency

for

varying

sizes

of

destination

sets

The

destinations

and

network

topologies

were

generated

randomly

For

each

data

point

the

multicast

latency

was

averaged

over

different

sets

of

destinations

for

each

of

different

network

configurations

Effect

of

system

size

We

simulated

the

four

multicast

algorithms

on

four

different

system

configurations

with

and

processors

respectively

Figure

shows

these

results

It

can

be

observed

that

the

CCO

algorithm

performs

the

best

for

all

system

sizes

and

destinations

As

the

system

size

increases

the

benefit

of

the

CCO

algorithm

becomes

more

prominent

For

example

on

a

processor

system

with

destinations

the

reduction

in

multicast

latency

achieved

by

the

CCO

algorithm

is

around

and

compared

to

the

RO

SO

and

SHO

algorithms

respectively

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

ro

so

sho

cco

60 80 100 120

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

ro

so

sho

cco

100 200 300 400

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

ro

so

sho

cco

Figure

Multicast

latency

vs.

number

of

destinations

for

four

different

system

configurations

and

processors

It

can

be

observed

that

the

RO

algorithm

performs

the

worst

The

multicast

latency

using

the

RO

algorithm

increases

considerably

as

we

move

to

larger

systems

and

larger

number

of

destinations

The

SO

algorithm

performs

well

for

small

sizes

of

destination

sets

However

its

latency

also

increases

as

we

move

to

larger

systems

and

larger

number

of

destinations

The

SHO

algorithm

does

not

perform

well

for

smaller

sizes

of

destination

sets

because

of

its

additional

start-up

requirement

However

as

the

number

of

destinations

increases

it

performs

reasonably

well

and

its

performance

falls

between

the

SO

and

CCO

algorithms

The

SO

and

SHO

algorithms

exhibit

the

same

latency

for

broadcast

because

they

lead

to

exactly

the

same

sequence

of

message

passing

steps

To

get

more

insight

into

the

behavior

of

these

algorithms

we

also

studied

the

contention

effect

associated

with

these

algorithms

Figure

shows

the

results

Since

the

RO

algorithm

undergoes

severe

contention

for

a

fair

comparison

we

present

average

message

contention

only

for

the

other

three

algorithms

These

contention

factors

are

obtained

from

simulation

experiments

corresponding

to

those

in

Fig.

The

average

message

contention

is

defined

as

the

following

parameter

-LRB-

tactual

lat?tideal

lat

-RRB-

tideal

lat

where

tactual

lat

is

the

average

of

actual

latencies

experienced

by

messages

involved

in

the

multicast

and

tideal

lat

is

the

average

of

ideal

latencies

for

the

corresponding

messages

The

ideal

latency

for

a

message

is

computed

based

on

the

basic

wormhole

routing

principle

depending

on

its

length

and

the

distance

it

travels

It

can

be

observed

that

the

average

message

contention

of

the

CCO

algorithm

is

very

low

compared

to

the

SO

and

SHO

algorithms

for

all

different

system

sizes

and

number

of

destinations

The

average

message

contention

for

the

SHO

algorithm

is

lower

than

that

of

the

SO

algorithm

because

the

second

stage

of

the

SHO

algorithm

is

guaranteed

to

take

place

in

a

contention-free

manner

as

indicated

in

Section

Effect

of

message

length

We

studied

the

impact

of

message

length

on

the

four

algorithms

We

considered

six

different

message

lengths

and

flits

A

processor

system

with

default

parameters

was

considered

Figure

shows

the

respective

results

It

can

be

easily

observed

that

the

CCO

algorithm

wins

over

all

other

algorithms

for

varying

message

lengths

and

number

of

destinations

For

message

length

of

flits

the

algorithms

perform

in

the

following

manner

for

all

destinations

CCO

SHO

SO

RO

where

reflects

capability

to

implement

the

multicast

with

reduced

latency

Average

message

contention

Number

of

destinations

System

Size

so

sho

cco

60 80 100 120

Average

message

contention

Number

of

destinations

System

Size

so

sho

cco

50 100 150 200

Average

message

contention

Number

of

destinations

System

Size

so

sho

cco

100 200 300 400

Average

message

contention

Number

of

destinations

System

Size

so

sho

cco

Figure

Average

message

contention

vs.

number

of

destinations

for

four

different

system

configurations

and

processors

Effect

of

communication

startup

time

The

effect

of

communication

start-up

time

on

the

performance

of

the

four

algorithms

was

studied

The

default

processor

system

configuration

was

used

with

three

different

communication

start-up

times

and

microseconds

Figure

shows

the

respective

multicast

latencies

It

can

be

observed

that

with

higher

communication

start-up

time

the

CCO

algorithm

shows

smaller

benefit

compared

to

the

SO

and

SHO

algorithms

However

it

performs

much

better

compared

to

the

RO

algorithm

This

is

expected

because

for

a

given

message

length

higher

start-up

times

reduce

the

contention

between

different

phases

of

the

multicast

algorithm

However

as

the

start-up

time

diminishes

the

CCO

algorithm

clearly

performs

the

best

for

all

destination

sizes

Currently

researchers

are

exploring

multiple

directions

to

design

efficient

network

interface

architectures

and

messaging

layers

to

reduce

communication

start-up

time

In

this

context

the

current

results

indicate

that

message

contention

in

multicast

will

gradually

become

severe

with

reduction

in

communication

start-up

time

Thus

algorithms

like

the

CCO

must

be

used

in

future

systems

to

implement

multicast

with

reduced

latency

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

ro

so

sho

cco

Figure

Multicast

latency

vs.

number

of

destinations

for

six

different

message

lengths

and

flits

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Startup

Time

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Startup

Time

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Startup

Time

ro

so

sho

cco

Figure

Multicast

latency

vs.

number

of

destinations

for

three

different

communication

start-up

times

and

microseconds

Effect

of

switch

size

We

also

studied

the

effect

of

switch

size

on

the

overall

performance

of

the

four

algorithms

The

default

processor

system

was

considered

with

three

different

switch

sizes

and

ports

Figure

shows

the

performance

results

It

can

be

observed

that

with

smaller

switch

size

the

CCO

algorithm

performs

the

best

As

switch

size

increases

more

number

of

communication

steps

become

intra-switch

steps

Since

intra-switch

steps

are

contention-free

it

leads

to

reduced

contention

for

the

overall

multicast

and

the

algorithms

start

delivering

equal

performance

However

for

larger

number

of

destinations

contention

still

exists

for

the

RO

algorithm

Thus

for

bigger

switch

size

and

larger

number

of

destinations

any

of

SO

SHO

and

CCO

algorithms

can

be

used

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Switch

Size

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Switch

Size

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Switch

Size

ro

so

sho

cco

Figure

Multicast

latency

vs.

number

of

destinations

for

three

different

switch

sizes

and

ports

Effect

of

degree

of

network

connectivity

Finally

we

studied

the

impact

of

degree

of

network

connectivity

on

multicast

latency

The

default

system

size

of

processors

was

considered

with

three

different

degrees

of

network

connectivity

and

Figure

shows

the

associated

performance

results

With

lesser

connectivity

the

number

of

communication

links

reduces

in

an

irregular

network

leading

to

less

number

of

adaptive

paths

and

more

contention

for

multicast

Under

such

circumstances

the

CCO

algorithm

delivers

the

best

performance

As

the

degree

of

connectivity

increases

the

contention

effect

reduces

but

does

not

get

completely

eliminated

Thus

with

higher

connectivity

the

CCO

algorithm

still

performs

better

compared

to

other

algorithms

but

the

benifits

are

reduced

Related

Work

Several

multicast

schemes

have

been

recently

proposed

and

evaluated

for

networks

of

workstations

with

ATM

switching

In

-LSB-

-RSB-

McKinley

et

al.

studied

the

trade-offs

between

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Connectivity

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Connectivity

ro

so

sho

cco

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Connectivity

ro

so

sho

cco

Figure

Multicast

latency

vs.

number

of

destinations

for

different

degrees

of

network

connectivity

and

implementing

multicast

through

software

message

passing

vs.

hardware-assisted

replication

at

the

ATM

switch

They

also

proposed

a

thread-based

interface

for

collective

communication

on

ATM

networks

in

-LSB-

-RSB-

In

-LSB-

-RSB-

Du

et

al.

evaluated

the

impact

of

different

messaging

layers

and

APIs

to

implement

multicast

on

networks

of

workstations

with

ATM

and

Ethernet

connections

In

-LSB-

-RSB-

Thekkath

et

al.

proposed

efficient

messaging

layers

to

achieve

low

latency

communication

on

ATM

interconnected

networks

of

workstations

Our

results

presented

in

this

paper

focus

on

irregular

networks

with

wormhole

routing

We

use

the

software

approach

with

unicast

message

passing

and

develop

different

multicast

algorithms

while

taking

network

topology

and

routing

into

account

In

-LSB-

-RSB-

Ni

et

al.

proposed

a

deadlock-free

adaptive

routing

scheme

for

irregular

networks

with

cut-through

switches

The

routing

is

based

on

Eulerian

trails

In

this

paper

we

have

considered

deadlock-free

adaptive

routing

scheme

proposed

in

-LSB-

-RSB-

due

to

its

simplicity

and

commercial

implementation

In

the

near

future

we

will

be

extending

our

multicast

schemes

for

the

routing

proposed

in

-LSB-

-RSB-

In

-LSB-

-RSB-

Bal

et

al.

proposed

reliable

multicast

scheme

on

Myrinet

using

Fast

Messages

-LSB-

-RSB-

In

this

work

a

credit-based

scheme

has

been

introduced

for

managing

buffers

at

senders

and

receivers

to

support

reliable

multicast

Our

emphasis

on

this

paper

has

been

on

developing

alternative

multicast

algorithms

and

evaluating

their

relative

performance

It

will

be

interesting

to

see

how

our

algorithms

can

be

integrated

with

the

framework

in

-LSB-

-RSB-

to

achieve

high

performance

and

reliable

multicast

on

Myrinet

Conclusions

In

this

paper

we

have

shown

efficient

ways

of

implementing

multicast

on

the

emerging

irregular

switch-based

networks

with

wormhole

routing

and

unicast

message

passing

Using

a

typical

deadlock-free

adaptive

routing

for

irregular

networks

first

we

have

proved

that

it

is

not

feasible

to

construct

a

complete

ordered

chain

of

destinations

to

implement

multicast

in

a

contentionfree

manner

with

optimal

number

of

communication

steps

Then

we

have

proposed

three

new

multicast

algorithms

-LRB-

SO

SHO

and

CCO

-RRB-

with

their

respective

orderings

of

destinations

These

algorithms

together

with

a

naive

random

ordering

-LRB-

RO

-RRB-

have

been

evaluated

through

simulation

for

a

wide

range

of

system

sizes

message

lengths

switch

sizes

degrees

of

connectivity

and

start-up

times

The

simulation

results

demonstrate

the

CCO

algorithm

to

be

the

best

for

a

wide

range

of

system

and

technological

parameters

This

algorithm

implements

a

multicast

with

least

amount

of

contention

and

minimum

latency

The

SO

algorithm

does

better

than

the

SHO

algorithm

for

small

sizes

of

destination

sets

However

the

SHO

outperforms

the

SO

as

the

system

size

and

the

number

of

destinations

increase

Overall

for

relatively

large

systems

and

large

number

of

destinations

the

four

algorithms

have

been

demonstrated

to

perform

in

the

following

order

CCO

-LRB-

best

-RRB-

SHO

SO

RO

-LRB-

worst

-RRB-

These

results

demonstrate

significant

potential

to

be

applied

to

current

and

future

generation

networks

of

workstations

with

irregular

interconnection

In

this

paper

we

have

proposed

algorithms

for

single

multicast

on

irregular

networks

We

will

be

extending

this

framework

to

see

how

other

collective

communication

operations

like

multiple

multicast

barrier

synchronization

and

complete

exchange

can

be

implemented

on

irregular

networks

with

minimum

latency

Besides

wormhole

routing

many

irregular

networks

currently

support

buffered

wormhole

routing

virtual

cut-through

switching

and

ATM

switching

It

will

be

interesting

to

see

how

the

algorithms

developed

in

this

paper

for

wormhole

routing

can

be

applied

to

systems

with

the

above

mentioned

switching

techniques

Additional

Information

A

number

of

related

papers

and

technical

reports

are

available

electronically

through

the

home

page

of

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

The

URL

is

Alternatively

the

papers

can

be

obtained

by

anonymous

ftp

from

ftp.cis.ohio-state

edu

cd

pub/communication

References

-LSB-

-RSB-

A.

Bar-Noy

and

S.

Kipnis

Multiple

Message

Broadcasting

in

the

Postal

Model

In

International

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

M.

Barnett

S.

Gupta

D.

G.

Payne

L.

Shuler

R.

van

de

Geijn

and

J.

Watts

Interprocessor

Collective

Communication

Library

-LRB-

Intercom

-RRB-

In

Scalable

High

Performance

Computing

Conference

pages

-LCB-

-LSB-

-RSB-

N.

J.

Boden

D.

Cohen

and

et

al.

Myrinet

A

Gigabit-per-Second

Local

Area

Network

IEEE

Micro

pages

-LCB-

Feb

-LSB-

-RSB-

R.

V.

Boppana

S.

Chalasani

and

C.

S.

Raghavendra

On

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

J.

Bruck

R.

Cypher

and

C.-T

Ho

Multiple

Message

Broadcasting

with

Generalized

Fibonacci

Trees

In

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

L.

De

Coster

N.

Dewulf

and

C.-T

Ho

Efficient

Multi-packet

Multicast

Algorithms

on

Meshes

with

Wormhole

and

Dimension-Ordered

Routing

In

International

Conference

on

Parallel

Processing

pages

III

-LCB-

Aug

-LSB-

-RSB-

Cray

Research

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

J.

Duato

A

New

Theory

of

Deadlock-Free

Adaptive

Routing

in

Wormhole

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

B.

Duzett

and

R.

Buck

An

Overview

of

the

Supercomputer

In

Proceedings

of

the

Frontiers

of

Massively

Parallel

Computation

pages

-LCB-

-LSB-

-RSB-

M.

D.

Schroeder

et

al.

Autonet

A

High-speed

Self-configuring

Local

Area

Network

Using

Point-to-point

Links

Technical

Report

SRC

research

report

DEC

Apr

-LSB-

-RSB-

M.

Heinrich

J.

Kuskin

et

al.

The

performance

impact

of

flexibility

in

the

stanford

flash

multiprocessor

In

The

sixth

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

-LRB-

ASPLOS-VI

-RRB-

San

Jose

CA

October

-LSB-

-RSB-

R.

Horst

ServerNet

Deadlock

Avoidance

and

Fractahedral

Topologies

In

Proceedings

of

the

International

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

C.

Huang

Y.

Huang

and

P.

K.

McKinley

A

Thread-based

Interface

for

Collective

Communication

on

ATM

Networks

In

International

Conference

on

Distributed

Computing

Systems

-LSB-

-RSB-

C.

Huang

and

P.

K.

McKinley

Communication

Issues

in

Parallel

Computing

Across

ATM

Networks

IEEE

Parallel

and

Distributed

Technology

Winter

-LCB-

-LSB-

-RSB-

Y.

Huang

C.

C.

Huang

and

P.

K.

McKinley

Multicast

Virtual

Topologies

for

Collective

Communication

in

MPCs

and

ATM

Clusters

In

Proceedings

of

the

Supercomputing

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

S.

L.

Johnsson

and

C.-T

Ho

Optimum

Broadcasting

and

Personalized

Communication

in

Hypercubes

IEEE

Transactions

on

Computers

pages

-LCB-

September

-LSB-

-RSB-

Ram

Kesavan

and

D.

K.

Panda

Minimizing

Node

Contention

in

Multiple

Multicast

on

Wormhole

k-ary

n-cube

Networks

In

International

Conference

on

Parallel

Processing

Chicago

IL

Aug

To

appear

-LSB-

-RSB-

Y.

Lan

A.-H

Esfahanian

and

L.

Ni

Multicast

in

Hypercube

Multiprocessors

Journal

of

Parallel

and

Distributed

Computing

-LCB-

-LSB-

-RSB-

M.

Lin

J.

Hsieh

D.

H.

C.

Du

J.

P.

Thomas

and

J.

A.

MacDonald

Distributed

Network

Computing

over

Local

ATM

Networks

IEEE

Journal

on

Selected

Areas

in

Communications

-LRB-

-RRB-

May

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

P.

K.

McKinley

and

D.

F.

Robinson

Collective

Communication

in

Wormhole-Routed

Massively

Parallel

Computers

IEEE

Computer

pages

-LCB-

Dec

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Dec

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

M.

D.

Noakes

D.

A.

Wallach

and

W.

J.

Dally

The

J-Machine

Multicomputer

An

Architectural

Evaluation

In

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

S.

Pakin

M.

Lauria

and

A.

Chien

High

Performance

Messaging

on

Workstations

Illinois

Fast

Messages

-LRB-

FM

-RRB-

In

Proceedings

of

the

Supercomputing

-LSB-

-RSB-

D.

K.

Panda

Issues

in

Designing

Efficient

and

Practical

Algorithms

for

Collective

Communication

in

Wormhole-Routed

Systems

In

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

Technical

Report

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

-LSB-

-RSB-

H.

Park

and

D.

P.

Agrawal

Generic

Methodologies

for

Deadlock-free

Routing

In

Proceedings

of

the

International

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

W.

Qiao

and

L.

M.

Ni

Adaptive

Routing

in

Irregular

Networks

Using

Cut-Through

Switches

In

International

Conference

on

Parallel

Processing

Chicago

IL

Aug

To

appear

-LSB-

-RSB-

H.D.

Schwetman

CSIM

Reference

Manual

-LRB-

Revision

-RRB-

Technical

report

Microelectronics

and

Computer

Technology

Corporation

Austin

Texas

January

-LSB-

-RSB-

C.

B.

Stunkel

D.

G.

Shea

B.

Abali

et

al.

The

High-Performance

Switch

IBM

System

Journal

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

C.

A.

Thekkath

H.

Levy

and

E.

D.

Lazowska

Efficient

Support

for

Multicomputing

on

ATM

Networks

Technical

Report

University

of

Minnesota

-LSB-

-RSB-

K.

Verstoep

K.

Langendoen

and

H.

Bal

Efficient

Reliable

Multicast

on

Myrinet

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

broadcast
collective communication
irregular networks
multicast
networks of workstations
wormhole routing