Fast

Broadcast

and

Multicast

in

Wormhole

Multistage

Networks

with

Multidestination

Worms

Dhabaleswar

K.

Panda

and

Rajeev

Sivaram

Technical

Report

Fast

Broadcast

and

Multicast

in

Wormhole

Multistage

Networks

with

Multidestination

Dhabaleswar

K.

Panda

and

Rajeev

Sivaram

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

E-mail

fpanda,sivaramg@cis.ohio-state.edu

Abstract

This

paper

presents

a

new

approach

to

implement

fast

broadcast

and

multicast

operations

in

bidirectional

wormhole

Multistage

Interconnection

Networks

-LRB-

MIN

-RRB-

with

loopback

as

used

in

IBM

network

The

novelty

lies

in

using

multidestination

message

passing

mechanism

instead

of

single

destination

-LRB-

unicast

-RRB-

messages

For

broadcast/multicast

operation

it

is

shown

that

a

single

worm

with

multiple

destinations

is

sufficient

to

allow

pipelined

replication

of

flits

at

appropriate

intermediate

switches

and

deliver

copies

to

the

required

destinations

For

higher

communication

start-up

-LRB-

ts

-RRB-

for

an

n-processor

system

this

new

approach

leads

to

an

asymptotic

improvement

by

a

factor

of

compared

to

the

unicast-based

messagepassing

Two

schemes

for

broadcast

and

multicast

are

presented

together

with

the

necessary

architectural

supports

at

a

switch-level

Storage

requirements

at

a

switch

to

ensure

deadlock

freedom

are

also

derived

These

schemes

are

evaluated

and

compared

with

the

unicast-based

message-passing

for

different

values

of

communication

start-up

time

link

propagation

time

switch

delay

system

size

and

switch

size

It

is

shown

that

the

multidestination

approach

demonstrates

superiority

and

scalability

for

higher

ts

and

smaller

switch

size

-LRB-

-RRB-

For

example

with

ts

microsec

a

broadcast

operation

on

a

processor

MIN

with

switches

and

flits

of

message

can

be

implemented

in

just

microsec

using

multidestination

approach

compared

to

microsec

with

unicast

message-passing

Similarly

for

a

processor

MIN

a

multicast

operation

with

arbitrary

number

of

destinations

takes

a

constant

time

of

microsec

for

a

flit

message

These

results

indicate

that

the

proposed

scheme

can

be

easily

applied

to

current

and

future

generation

multistage

systems

like

to

provide

fast

and

scalable

collective

communication

operations

as

defined

by

the

Message

Passing

Interface

-LRB-

MPI

-RRB-

standard

research

is

supported

in

part

by

the

National

Science

Foundation

Grant

Contents

Introduction

Multidestination

Mechanism

for

MINs

Omega

Network

Model

with

Loopback

Supporting

Multidestination

Mechanism

Broadcast

No

Loopback

on

Forward

Propagation

Algorithm

-LRB-

NLF

-RRB-

Loopback

on

Forward

Propagation

Algorithm

-LRB-

LF

-RRB-

Estimating

Broadcast

Latency

Multicast

Loop

Back

on

Forward

Propagation

Algorithm

-LRB-

LF

-RRB-

Architectural

supports

at

a

switch

Estimating

Multicast

Latency

Storage

Requirement

for

Deadlock

Freedom

An

Example

of

Deadlock

Requirement

Under

Generalized

Framework

Requirement

With

Specific

Mechanisms

No

Buffering

When

All

Output

Links

are

Busy

Allowing

Buffering

of

Worms

Performance

Evaluation

Broadcast

Overall

Comparison

Impact

of

Message

Length

Multicast

Overall

Comparison

Impact

of

System

Size

Conclusions

and

Future

Research

Introduction

The

wormhole-routing

switching

technique

is

becoming

the

trend

in

building

future

parallel

systems

due

to

its

inherent

advantages

like

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

IBM

Intel

Paragon

Cray

Ncube

J-Machine

and

Stanford

DASH

are

representative

systems

falling

into

this

category

All

systems

except

IBM

use

k-ary

n-cube

interconnections

The

IBM

system

-LSB-

-RSB-

uses

bidirectional

multistage

interconnection

with

loopback

Such

systems

are

being

used

for

supporting

either

distributed-memory

or

distributed-shared

memory

programming

paradigms

For

efficient

support

of

the

above

paradigms

these

systems

require

fast

broadcast

and

multicast

support

from

the

underlying

network

as

defined

by

the

Message

Passing

Interface

-LRB-

MPI

-RRB-

Standard

-LSB-

-RSB-

Many

software

schemes

have

been

recently

proposed

in

the

literature

to

efficiently

implement

broadcast

and

multicast

in

wormhole-routed

k-ary

n-cube

-LSB-

-RSB-

and

multistage

networks

-LSB-

-RSB-

All

these

schemes

use

multiple

contention-free

phases

using

point-to-point

-LRB-

unicast

-RRB-

message

passing

For

an

n-processor

system

these

schemes

require

n

communication

phases

for

broadcast

Similarly

for

multicasting

to

d

destinations

-LRB-

d

n

-RRB-

de

communication

phases

are

required

Since

the

communication

start-up

time

-LRB-

ts

-RRB-

in

current

generation

systems

is

around

microseconds

the

overall

latency

of

broadcast/multicast

operation

using

these

algorithms

is

quite

high

and

is

proportional

to

the

number

of

phases

in

the

algorithm

because

the

communication

start-up

time

dominates

in

every

phase

For

example

with

ts

microsec

it

takes

at

least

microseconds

-LRB-

ignoring

data

propagation

and

computation

time

-RRB-

to

perform

broadcast

operation

on

a

system

This

raises

a

fundamental

question

whether

the

latency

of

these

operations

can

be

reduced

by

using

a

new

and

efficient

mechanism

In

our

recent

works

-LSB-

-RSB-

we

have

introduced

a

new

concept

of

multidestination

wormhole

routing

In

contrast

to

unicast

messages

which

have

a

single

destination

this

mechanism

allows

a

message

to

have

multiple

destinations

Such

worms

can

be

designed

to

have

different

functionality

We

have

used

the

multidestination

broadcast

worm

in

-LSB-

-RSB-

to

design

algorithms

for

broadcast

and

multicast

operations

in

wormhole-routed

k-ary

n-cube

systems

with

reduced

latency

In

-LSB-

-RSB-

we

have

introduced

the

concept

of

multidestination

gather

worms

and

in

conjunction

with

the

broadcast

worm

we

have

shown

how

to

implement

complete

and

arbitrary

barrier

synchronization

in

wormhole-routed

systems

with

reduced

latency

In

-LSB-

-RSB-

we

have

introduced

a

multidestination

exchange

worm

and

have

demonstrated

that

complete

global

reduction

operations

-LRB-

including

barrier

synchronization

-RRB-

can

be

efficiently

implemented

with

very

small

latency

Contrary

to

the

Hamiltonian

path-based

scheme

-LSB-

-RSB-

our

scheme

uses

a

Base

Routing

Conformed

Path

-LRB-

BRCP

-RRB-

model

and

can

be

applied

to

systems

supporting

deterministic

-LRB-

e-cube

-LSB-

-RSB-

-RRB-

or

adaptive

-LRB-

planar-adaptive

-LSB-

-RSB-

and

turn-model

-LSB-

-RSB-

-RRB-

routing

We

have

demonstrated

that

the

latency

of

collective

communication

in

k-ary

n-cube

networks

can

be

significantly

reduced

by

using

multidestination

worms

instead

of

unicast

worms

The

most

interesting

thing

that

has

been

shown

in

these

works

is

that

the

cost

of

multicast

subset

barrier

synchronization

and

subset

global

reduction

operations

can

be

reduced

or

kept

constant

as

the

number

of

processors

participating

in

such

an

operation

increases

beyond

a

certain

number

These

results

have

established

the

fact

that

the

multidestination

mechanism

can

be

used

in

future

generation

wormhole

systems

to

provide

scalable

collective

communication

While

wormhole

systems

using

k-ary

n-cube

topology

are

being

built

the

traditional

multistage

interconnection

-LRB-

MIN

-RRB-

is

coming

back

and

gaining

gradual

popularity

Systems

like

IBM

-LSB-

-RSB-

use

a

wormhole-routed

bidirectional

MIN

with

loopback

to

exploit

locality

of

references

As

parallel

systems

start

using

such

wormhole

interconnection

the

challenge

is

to

incorporate

efficient

architectural

supports

into

the

interconnection

so

as

to

provide

fast

collective

communication

In

this

paper

we

take

on

such

a

challenge

and

apply

our

multidestination

message-passing

concept

to

bidirectional

MINs

with

loopback

We

present

two

schemes

No

Loopback

on

Forward

Propagation

-LRB-

NLF

-RRB-

and

Loopback

on

Forward

Propagation

-LRB-

LF

-RRB-

where

a

single

worm

with

multiple

destinations

as

its

header

can

deliver

copies

of

a

message

to

multiple

destinations

with

a

single

communication

start-up

We

propose

a

pipelined

replication

mechanism

at

a

switch-level

to

process

the

header

list

and

replicate

the

incoming

flits

as

required

The

associated

architectural

supports

at

a

switch-level

are

presented

Storage

requirements

at

a

switch

to

ensure

deadlock

freedom

are

also

derived

These

schemes

are

evaluated

and

compared

with

unicast-based

approach

-LSB-

-RSB-

to

implement

one-to-all

broadcast

and

one-to-many

multicast

operations

The

interplay

between

different

technological

parameters

-LRB-

communication

start-up

time

link

propagation

time

and

switch

delay

-RRB-

and

architectural

parameters

-LRB-

number

of

nodes

and

size

of

switches

-RRB-

is

studied

The

results

indicate

the

proposed

multidestination

approach

to

be

superior

and

scalable

compared

to

the

unicast-based

scheme

for

various

system

size

switch

size

and

message

length

The

paper

is

organized

as

follows

The

multidestination

mechanism

for

MINs

is

introduced

in

section

In

section

we

present

two

multidestination

schemes

for

broadcast

operation

and

derive

analytical

expressions

for

the

associated

broadcast

latency

In

section

we

present

a

scheme

for

multicast

and

analyze

the

associated

latency

In

section

we

examine

the

issue

of

deadlock

and

derive

storage

requirements

in

the

central

queue

for

deadlock

avoidance

Com

parison

of

the

proposed

schemes

with

respect

to

the

unicast-based

scheme

for

different

system

and

technological

parameters

is

done

in

section

Finally

we

conclude

the

paper

with

future

works

Multidestination

Mechanism

for

MINs

In

this

section

we

present

a

method

for

implementing

the

multidestination

mechanism

on

Multistage

Interconnection

Networks

-LRB-

MINs

-RRB-

For

simplicity

we

introduce

the

mechanism

with

respect

to

the

popular

Omega

network

model

-LSB-

-RSB-

However

the

concept

can

be

applied

to

any

other

MIN

with

appropriate

modification

We

also

indicate

the

suitable

architectural

supports

at

a

switch

to

implement

this

mechanism

Omega

Network

Model

with

Loopback

Consider

an

n-node

Omega

network

with

bidirectional

interconnection

A

typical

such

organization

with

k

k

switches

consists

of

logk

n

stages

Each

stage

uses

a

shuffle

interconnection

Bidirectional

communication

links

connect

intermediate

stages

Similar

to

the

interconnection

structure

being

used

in

IBM

-LSB-

-RSB-

we

assume

the

processors

to

be

connected

to

one

side

of

the

network

The

switches

support

the

loopback

feature

wherein

a

typical

single-destination

unicast

message

can

go

up

-LRB-

forward

-RRB-

a

few

stages

and

come

down

-LRB-

backward

-RRB-

to

reach

its

destination

Such

movement

allows

a

message

going

to

a

closer

destination

to

traverse

less

number

of

stages

This

feature

helps

in

exploiting

communication

locality

in

the

network

The

switch

organization

is

assumed

to

be

closer

to

that

of

the

high-performance

switch

used

in

IBM

-LSB-

-RSB-

A

k?k

switch

has

input

buffers

output

buffers

and

controllers

associated

with

the

input

and

output

buffers

If

the

communication

links

are

w-bits

wide

we

assume

the

flit

size

to

be

w

bits

too

We

also

assume

virtual

cut-through

routing

Each

switch

is

equipped

with

a

central

queue

where

blocked

messages

can

be

stored

As

described

later

we

will

demonstrate

that

such

an

organization

allows

deadlock

freedom

during

the

replication

process

Supporting

Multidestination

Mechanism

Our

earlier

work

-LSB-

-RSB-

on

multidestination

mechanism

for

k-ary

n-cube

networks

assumes

that

a

worm

moves

along

a

path

passing

through

the

routers

of

the

intermediate

destinations

The

routers

were

assumed

to

support

forward

as

well

as

absorb

capability

However

such

paths

are

not

feasible

in

a

MIN

Instead

we

propose

the

use

of

a

replication

mechanism

at

every

switch

When

an

incoming

message

arrives

at

a

switch

depending

on

the

destination

list

the

switch

replicates

the

message

to

appropriate

output

buffers

to

create

multiple

streams

of

outgoing

messages

At

a

given

switch

the

selection

of

appropriate

output

buffers

to

which

a

message

needs

to

be

replicated

depends

on

the

destination

list

and

the

topology

of

the

MIN

The

basic

idea

is

that

if

a

new

stream

gets

generated

at

a

switch

from

an

incoming

message

stream

it

should

be

able

to

deliver

the

message

to

its

destination

list

-LRB-

a

subset

of

the

destination

list

of

the

incoming

message

-RRB-

with

successive/no

replication

on

its

way

of

traversal

Figure

shows

a

logical

picture

of

such

a

replication

process

in

two

adjacent

stages

of

a

MIN

consisting

of

switches

The

incoming

message

to

the

upper-left

switch

consists

of

destinations

Since

one

destination

node

k

is

connected

to

this

switch

itself

a

new

stream

is

replicated

for

it

with

the

header

having

only

one

destination

The

remaining

list

of

destinations

are

partitioned

into

two

subsets

-LRB-

a

b

c

d

e

-RRB-

going

to

the

lower

output

and

-LRB-

f

g

h

i

j

-RRB-

going

to

the

upper

output

At

the

second

stage

of

the

network

the

above

header

lists

are

further

partitioned

and

directed

to

appropriate

outputs

-LRB-

including

loopback

-RRB-

When

the

data

flits

follow

the

header

flits

the

switch

appropriately

replicates

them

and

forwards

them

to

the

respective

output

buffers

In

this

example

it

can

be

seen

that

the

original

multidestination

worm

gets

transformed

to

multidestination

worms

There

are

different

ways

to

encode

the

addresses

in

a

multidestination

worm

-LSB-

-RSB-

Depending

on

the

header

format

being

used

and

the

ease

of

implementation

the

destination

list

may

or

may

not

be

actually

partitioned

The

all-destination

encoding

scheme

uses

one

or

more

flits

for

each

destination

However

as

the

number

of

destinations

increases

such

encoding

makes

the

header

quite

long

and

increases

the

message

size

In

contrast

a

bit-string

encoding

scheme

can

be

used

to

represent

each

destination

by

a

bit

With

a

channel

width

of

w

bits

the

header

of

a

multidestination

worm

to

d

destinations

can

be

encapsulated

in

just

dd

we

flits

This

makes

the

header

size

quite

compact

and

cost-effective

In

this

paper

we

assume

such

a

bit-string

encoding

scheme

dst

-LSB-

a

b

c

-RSB-

dst

-LSB-

d

e

-RSB-

dst

-LSB-

f

g

-RSB-

dst

-LSB-

h

i

j

-RSB-

dst

-LSB-

f

g

h

i

j

-RSB-

dst

-LSB-

a

b

c

d

e

-RSB-

dst

-LSB-

k

-RSB-

dst

-LSB-

a

b

c

d

e

f

g

h

i

j

k

-RSB-

Figure

Logical

representation

of

destination

partitioning

and

data

replication

in

a

multistage

network

to

route

multidestination

worms

On

the

arrival

of

a

new

flit

at

an

input

buffer

it

is

replicated

to

the

respective

output

buffers

As

soon

as

a

flit

is

ready

at

an

output

buffer

it

can

start

propagating

During

this

replication

it

may

happen

that

the

previous

flit

at

an

output

buffer

might

not

have

propagated

to

the

next

stage

The

previous

flit

may

correspond

either

to

the

worm

being

replicated

or

to

a

different

worm

In

either

case

the

replicated

flit

corresponding

to

the

blocked

output

link

gets

stored

in

the

central

queue

associated

with

the

switch

When

the

output

buffer

corresponding

to

the

blocked

link

becomes

free

the

stored

flits

from

the

central

queue

start

moving

If

an

output

buffer

becomes

free

when

the

ith

flit

of

a

message

is

being

replicated

with

the

previous

i

flits

intended

for

the

corresponding

output

link

having

been

stored

in

the

central

queue

the

replicated

ith

flit

gets

stored

in

the

central

queue

The

storing

of

the

replicated

flits

to

the

central

queue

may

get

overlapped

with

the

forwarding

of

previously

stored

flits

to

their

respective

output

links

when

they

become

free

When

all

required

copies

of

an

incoming

flit

have

been

made

at

a

switch

a

signal

is

sent

to

its

previous

stage

to

send

the

following

flit

A

sequential

replication

mechanism

instead

of

concurrent

replication

is

assumed

to

reduce

complexity

of

a

switch

to

reduce

the

blocking

time

and

to

provide

asynchronous

coupling

between

the

incoming

message

stream

and

the

outgoing

message

streams

at

a

switch

Independent

replication

logics

are

assumed

for

every

input

in

a

switch

To

provide

more

concurrency

pipelined

flit

buffers

at

each

input

and

output

are

assumed

to

provide

overlap

between

replication

of

flit

i

the

arrival

of

flit

i

and

the

propagation

of

flit

i

Depending

on

the

header

format

being

used

and

the

ease

of

implementation

the

destination

list

may

or

may

not

be

actually

partitioned

In

the

following

sections

we

use

a

fixed

header

format

-LRB-

bit-string

encoding

-RRB-

and

the

switches

operate

on

appropriate

bit

patterns

to

route

multidestination

worms

It

is

to

be

noted

that

such

replication

is

different

from

the

hardware

tree-based

broadcast/multicast

which

leads

to

deadlock

as

discussed

in

-LSB-

-RSB-

The

hardware

tree-based

replication

scheme

requires

all

output

buffers

to

be

free

before

replication

is

done

to

any

of

them

In

our

scheme

replication

proceeds

as

long

as

at

least

one

of

the

required

output

buffers

is

free

If

any

of

the

required

output

buffers

are

full

the

corresponding

worms

are

stored

in

the

central

queue

as

described

above

In

our

scheme

if

the

central

queue

has

adequate

storage

deadlock

can

be

avoided

In

section

we

describe

the

nature

of

this

deadlock

propose

mechanisms

to

deal

with

blocked

worms

and

derive

the

storage

requirement

of

the

central

queue

so

as

to

ensure

deadlock

freedom

for

each

of

the

mechanisms

Broadcast

In

this

section

we

propose

two

algorithms

for

one-to-all-broadcast

using

multidestination

worms

We

determine

the

respective

latencies

of

these

algorithms

and

compare

them

with

that

of

the

unicast-based

approach

No

Loopback

on

Forward

Propagation

Algorithm

-LRB-

NLF

-RRB-

This

algorithm

consists

of

two

phases

During

the

first

phase

a

multidestination

broadcast

worm

is

allowed

to

route

adaptively

in

the

forward

direction

from

the

first

stage

to

the

last

stage

of

the

network

The

second

phase

starts

when

the

worm

reaches

the

last

stage

Assuming

k

k

switches

at

the

last

stage

k

copies

of

the

incoming

flits

are

made

-LRB-

one

for

each

of

the

k

outgoing

channels

-RRB-

and

sent

back

to

the

switches

of

the

penultimate

stage

As

discussed

in

the

previous

section

the

k

copies

are

made

in

a

pipelined

fashion

with

k

copies

of

an

incoming

flit

being

made

-LRB-

and

forwarded

-RRB-

before

copies

of

the

next

flit

are

made

Note

that

a

flit

is

forwarded

as

soon

at

is

made

-LRB-

while

the

remaining

copies

are

being

made

-RRB-

These

worms

start

traveling

in

the

backward

direction

At

every

stage

k

copies

of

the

flits

are

again

made

and

forwarded

to

unique

switches

of

the

next

lower

stage

When

the

worms

reach

first

stage

switches

a

copy

of

the

packet

is

forwarded

to

every

processor

connected

to

the

switch

thus

completing

the

broadcast

Figure

-LRB-

a

-RRB-

illustrates

this

algorithm

for

a

node

Omega

network

consisting

of

switches

The

algorithm

is

more

formally

described

in

Fig.

NLF

-LCB-

Broadcast

Algorithm

Phase

-RRB-

if

-LRB-

NOT

-LRB-

last-stage

-RRB-

then

Forward

worm

to

any

switch

at

the

next

higher

stage

Phase

-RRB-

for

last-stage

downto

second-stage

do

Make

k

copies

of

the

incoming

flit

and

forward

it

to

the

switches

of

the

next

lower

stage

-RRB-

if

-LRB-

first-stage

-RRB-

then

Make

k

copies

of

the

incoming

flit

and

forward

it

to

the

k

processors

connected

to

the

switch

Figure

NLF

-LCB-

Broadcast

Algorithm

-LRB-

a

-RRB-

Broadcast

using

NLF

scheme

Phase

Phase

-LRB-

b

-RRB-

Broadcast

using

LF

schemeBackwardWormsWormsForward

Figure

An

example

of

broadcast

operation

using

-LRB-

a

-RRB-

No

Loopback

on

Forward

Propagation

-LRB-

NLF

-RRB-

algorithm

and

-LRB-

b

-RRB-

Loopback

on

Forward

Propagation

-LRB-

LF

-RRB-

algorithm

on

an

Omega

network

The

source

is

assumed

to

be

node

Loopback

on

Forward

Propagation

Algorithm

-LRB-

LF

-RRB-

This

algorithm

makes

use

of

the

fact

that

switches

at

all

stages

are

equipped

with

loopback

capabilities

Hence

copies

of

a

broadcast

message

are

also

made

to

backward

outputs

of

a

switch

in

the

first

phase

while

forwarding

flits

to

higher

stages

At

the

first

stage

copies

are

made

for

processors

in

the

same

switch

while

forwarding

a

copy

to

the

next

stage

Similarly

at

the

next

stage

copies

are

made

for

switches

at

the

previous

stage

connecting

to

this

stage

switch

In

addition

a

copy

is

forwarded

to

a

switch

at

the

next

higher

stage

To

reduce

the

pattern

completion

time

at

every

switch

the

first

copy

is

forwarded

to

the

next

higher

stage

This

process

continues

till

a

final

stage

switch

is

reached

Now

the

second

phase

begins

At

the

final

stage

switch

copies

are

made

and

forwarded

in

the

reverse

direction

Note

that

compared

to

the

NLF

algorithm

one

less

copy

is

made

in

the

final

stage

as

it

is

not

necessary

to

make

copy

for

the

link

on

which

the

message

arrives

Similar

to

the

NLF

algorithm

on

the

backward

path

k

copies

of

each

flit

are

made

and

forwarded

Fig

-LRB-

b

-RRB-

illustrates

this

algorithm

for

a

node

Omega

network

with

switches

The

algorithm

is

more

formally

described

in

Fig.

It

is

to

be

noted

that

the

unicast-based

scheme

-LSB-

-RSB-

takes

n

steps

to

perform

broadcast

in

an

n-node

multistage

network

This

leads

to

Theorem

Both

NLF

and

LF

algorithms

can

implement

one-to-all

broadcast

operation

in

an

n-processor

multistage

network

using

only

a

single

communication

start-up

Compared

to

the

unicast-based

approach

it

leads

to

an

asymptotic

improvement

by

a

factor

of

n.

Estimating

Broadcast

Latency

Let

us

estimate

the

latency

of

the

above

two

algorithms

for

an

omega

network

with

n

processors

and

k

k

switches

Since

the

broadcast

message

needs

to

go

to

all

processors

we

do

not

need

a

list

of

destinations

in

the

message

header

A

special

ID

-LRB-

of

length

flit

-RRB-

indicating

that

it

is

a

broadcast

pattern

will

be

sufficient

Let

us

consider

the

following

parameters

tsw

as

the

delay

for

making

the

routing

decision

at

a

switch

tcopy

as

the

time

to

replicate

a

flit

from

an

input

buffer

to

an

output

buffer

tlink

as

the

link

delay

encountered

by

a

flit

between

adjacent

stages

ts

as

the

communication

start-up

time

and

a

message

length

of

l

flits

Under

the

NLF

algorithm

during

phase

no

loopback

copies

are

made

till

the

last

stage

is

reached

Therefore

the

header

flit

encounters

a

delay

of

-LRB-

logk

n

-RRB-

-LRB-

tsw

tcopy

tlink

-RRB-

in

phase

Since

k

copies

are

made

in

each

switch

in

the

backward

path

the

slowest

moving

header

encounters

a

delay

of

-LRB-

logk

n

-RRB-

-LRB-

tsw

k

tcopy

tlink

-RRB-

during

phase

Once

the

header

arrives

the

remaining

-LRB-

l

-RRB-

flits

arrive

in

a

pipelined

manner

with

a

delay

of

-LRB-

k

tcopy

-RRB-

per

flit

Thus

broadcast

latency

using

the

NLF

algorithm

is

LF

-LCB-

Broadcast

Algorithm

Phase

-RRB-

forall

-LRB-

stage

last-stage

-RRB-

do

a

-RRB-

Forward

a

copy

of

the

incoming

flit

to

the

next

higher

stage

switch

Make

k

more

copies

of

the

flit

b

-RRB-

if

-LRB-

stage

first-stage

-RRB-

then

Forward

the

k

copies

to

the

switches

at

the

previous

stage

to

which

the

switch

is

connected

else

Forward

the

k

copies

to

the

k

other

processors

connected

to

the

switch

Phase

-RRB-

if

-LRB-

last-stage

-RRB-

then

Make

k

copies

of

the

incoming

flit

and

forward

them

to

switches

at

the

penultimate

stage

-RRB-

for

penultimate-stage

downto

second-stage

do

Make

k

copies

of

the

incoming

flit

with

forwarding

to

the

next

lower

stage

switch

-RRB-

if

-LRB-

first-stage

-RRB-

then

Make

k

copies

of

the

incoming

flit

and

forward

to

the

k

processors

connected

to

the

switch

Figure

LF

-LCB-

Broadcast

Algorithm

tbroad

NLF

ts

-LRB-

logk

n

-RRB-

-LRB-

tsw

tcopy

tlink

-RRB-

logk

n

-LRB-

tsw

k

tcopy

tlink

-RRB-

-LRB-

l

-RRB-

k

tcopy

In

the

LF

algorithm

copies

are

made

even

when

the

flits

move

in

the

forward

direction

However

taking

into

account

of

the

fact

that

the

copy

which

is

made

first

at

a

switch

is

forwarded

to

the

next

stage

the

delay

encountered

by

the

header

flit

in

phase

remains

the

same

as

that

for

the

NLF

algorithm

In

the

second

phase

-LRB-

-RRB-

copies

are

made

at

the

last

stage

switch

and

k

copies

are

made

in

all

subsequent

stages

in

the

backward

direction

Once

the

slowest

header

arrives

at

a

destination

the

remaining

flits

arrive

in

a

pipelined

manner

similar

to

the

NLF

algorithm

Thus

the

broadcast

latency

using

the

LF

algorithm

is

tbroad

LF

ts

-LRB-

logk

n

-RRB-

-LRB-

tsw

tcopy

tlink

-RRB-

-LRB-

tsw

-LRB-

k

-RRB-

tcopy

tlink

-RRB-

-LRB-

logk

n

-RRB-

-LRB-

tsw

k

tcopy

tlink

-RRB-

-LRB-

l

-RRB-

k

tcopy

This

leads

to

the

following

observation

Observation

The

broadcast

latency

of

both

NLF

and

LF

algorithms

are

almost

identical

with

a

minor

difference

that

the

LF

algorithm

is

faster

by

tcopy

units

The

above

latencies

are

computed

from

the

start

of

the

operation

to

the

time

the

last

destination

completely

receives

the

message

However

the

LF

scheme

allows

many

destinations

to

receive

the

message

earlier

due

to

less

number

of

stages

being

traversed

Hence

we

suggest

using

the

LF

algorithm

for

practical

use

In

the

following

section

we

focus

only

on

the

LF

algorithm

and

use

it

for

multicast

operation

Multicast

We

use

the

LF

scheme

outlined

in

the

previous

section

to

propose

implementation

for

oneto-many

multicast

operations

Unlike

the

broadcast

operation

a

multicast

worm

should

have

appropriate

destination

IDs

in

its

header

list

In

this

section

we

propose

an

algorithm

for

multicast

operation

together

with

the

required

architectural

supports

at

a

switch

Loop

Back

on

Forward

Propagation

Algorithm

-LRB-

LF

-RRB-

This

algorithm

is

an

enhancement

to

the

LF

algorithm

presented

for

broadcast

operation

in

the

previous

section

Consider

a

multidestination

multicast

worm

having

a

set

of

destinations

in

its

header

A

bit-string

encoded

address

as

discussed

in

section

is

assumed

We

use

a

fixed

header

format

of

nw

flits

to

encode

the

multicast

destination

set

for

an

n-processor

system

with

w-bits

wide

links

These

flits

are

organized

as

an

n-bit

destination

vector

-LRB-

D

-RRB-

where

a

bit

position

i

being

on

-LRB-

-RRB-

indicates

that

the

processor

i

should

receive

a

copy

of

the

message

As

the

message

propagates

through

a

switch

from

vector

D

the

routing

logic

determines

the

set

of

destinations

that

can

be

reached

by

traversing

each

of

its

outgoing

links

Below

we

describe

a

simple

logic

to

determine

such

reachability

During

phase

a

switch

at

a

given

stage

checks

its

outgoing

loopback

links

first

If

a

link

leads

to

any

of

the

destinations

a

copy

of

an

incoming

flit

is

made

and

forwarded

to

that

link

If

the

destinations

can

not

be

completely

covered

by

these

loopback

links

a

copy

is

forwarded

to

the

next

higher

stage

in

an

adaptive

manner

similar

to

the

LF

broadcast

algorithm

During

phase

a

switch

in

a

given

stage

only

checks

its

outgoing

links

in

the

backward

direction

If

maximum

copies

need

to

be

made

at

a

switch

it

can

be

easily

observed

that

k

The

detailed

multicast

algorithm

is

formally

presented

in

Fig.

Architectural

supports

at

a

switch

In

addition

to

the

replication

mechanism

needed

at

a

switch

as

described

in

section

the

multicast

operation

requires

special

routing

logic

at

every

switch

We

propose

the

following

routing

logic

and

the

associated

architectural

supports

Every

switch

maintains

for

each

of

its

output

channels

a

bit-encoded

list

of

the

reachable

destinations

say

RDi

for

output

link

i

When

the

header

of

a

multidestination

multicast

arrives

at

a

switch

-LRB-

it

needs

to

wait

for

all

the

nw

header

flits

to

arrive

-RRB-

it

performs

-LRB-

D

RDi

-RRB-

bit-wise

AND

operation

for

each

of

its

output

links

For

a

given

output

link

if

the

result

of

the

operation

is

non-zero

it

needs

to

replicate

a

message

stream

on

that

link

With

suitable

hardware

support

this

bit-wise

AND

operation

for

all

outgoing

links

can

be

done

in

parallel

To

facilitate

routing

during

phase

a

switch

also

stores

a

bit-encoded

list

of

non-reachable

destinations

-LRB-

NRD

-RRB-

viz

the

destinations

which

can

not

be

reached

by

its

loopback

links

A

non-zero

value

during

operation

-LRB-

D

NRD

-RRB-

indicates

that

the

switch

needs

to

forward

a

message

stream

in

the

upward

direction

These

bit

vectors

can

be

computed

statically

and

down

loaded

to

a

switch

like

the

routing

table

of

IBM

This

helps

in

making

fast

routing

decisions

The

length

of

each

of

these

address

lists

can

also

be

reduced

from

n-bits

to

n

k

bits

by

redefining

the

reachability

property

so

that

we

consider

the

switches

of

the

first

stage

that

are

reachable

from

the

output

links

of

a

switch

-LRB-

rather

than

the

processors

that

are

reachable

from

these

output

links

-RRB-

With

the

above

architectural

supports

it

can

be

easily

seen

that

the

LF

-LCB-

multicast

algorithm

can

implement

arbitrary

multicast

with

a

single

communication

start-up

This

leads

to

LF

-LCB-

Multicast

Algorithm

Phase

-RRB-

forall

-LRB-

stage

last-stage

-RRB-

do

a

-RRB-

Compare

header

bits

with

encoding

of

destinations

reachable

from

the

switch

b

-RRB-

if

-LRB-

first-stage

-RRB-

then

Make

p

copies

of

the

message

where

p

k

is

the

number

of

destinations

connected

to

the

same

switch

Send

these

copies

to

the

appropriate

processors

else

Make

p

copies

of

the

message

where

p

k

is

the

number

of

switches

which

lead

to

destinations

reachable

from

the

switch

Send

these

copies

to

the

appropriate

output

buffers

c

-RRB-

if

-LRB-

-LRB-

-LRB-

header-bits

-RRB-

-LRB-

destinations

reachable

from

current

switch

-RRB-

-RRB-

-RRB-

then

Forward

a

copy

of

the

message

to

a

switch

at

the

next

higher

level

Phase

-RRB-

for

last-stage

downto

first-stage

do

a

-RRB-

Execute

Step

-RRB-

a

-RRB-

b

-RRB-

Execute

Step

-RRB-

b

-RRB-

Figure

LF

-LCB-

Multicast

Algorithm

Theorem

The

proposed

Loopback

on

Forward

Propagation

-LRB-

LF

-RRB-

algorithm

can

implement

arbitrary

multicast

operation

in

an

n-processor

multistage

network

using

only

a

single

communication

start-up

Compared

to

the

unicast-based

approach

it

leads

to

an

asymptotic

improvement

by

a

factor

of

d

where

d

is

the

number

of

destinations

in

a

multicast

pattern

Estimating

Multicast

Latency

While

deriving

an

expression

for

the

multicast

latency

we

need

to

take

into

account

the

additional

routing

delay

to

process

the

multidestination

header

and

to

wait

for

all

header

flits

to

arrive

before

making

a

routing

decision

Assume

the

routing

delay

for

a

multicast

message

at

a

switch

to

be

-LRB-

as

opposed

to

tsw

for

the

broadcast

operation

-RRB-

Let

us

consider

the

worst

case

scenario

where

the

multicast

is

targeted

to

-LRB-

n

-RRB-

nodes

Let

us

assume

that

nw

is

the

header

length

This

indicates

that

the

delay

encountered

by

the

header

while

moving

from

the

first

stage

to

the

second

stage

is

-LRB-

tlink

k

tcopy

tlink

-RRB-

The

k

tcopy

term

is

for

the

replication

of

the

header

flits

A

delay

of

-LRB-

tcopy

tlink

-RRB-

is

encountered

for

each

of

the

remaining

-LRB-

logkn

-RRB-

switches

along

the

upward

path

in

phase

keeping

in

mind

that

the

copy

which

is

made

first

at

a

switch

is

forwarded

to

the

next

stage

A

delay

of

-LRB-

-LRB-

k

-RRB-

tcopy

tlink

-RRB-

is

encountered

at

the

highest

stage

switch

at

the

start

of

phase

Finally

a

delay

of

-LRB-

k

tcopy

tlink

-RRB-

is

encountered

at

each

of

the

remaining

-LRB-

-RRB-

switches

along

the

downward

path

of

phase

After

the

slowest

header

arrives

at

the

processor

the

remaining

-LRB-

l

-RRB-

flits

arrive

in

a

pipelined

fashion

at

a

delay

of

k

tcopy

per

flit

This

leads

to

the

worst

case

multicast

latency

as

tmulticast

LF

ts

-LRB-

tlink

k

tcopy

tlink

-RRB-

-LRB-

logkn

-RRB-

-LRB-

tcopy

tlink

-RRB-

-LRB-

-LRB-

k

-RRB-

tcopy

tlink

-RRB-

-LRB-

logkn

-RRB-

-LRB-

k

tcopy

tlink

-RRB-

-LRB-

l

-RRB-

k

tcopy

Storage

Requirement

for

Deadlock

Freedom

In

this

section

we

first

analyze

how

buffer

dependencies

lead

to

deadlock

Next

we

examine

the

storage

requirements

for

deadlock

avoidance

under

a

generalized

framework

We

then

propose

two

mechanisms

for

handling

blocked

unicast

worms

at

a

switch

and

derive

the

respective

storage

requirements

An

Example

of

Deadlock

As

mentioned

in

the

context

of

the

replication

mechanism

in

section

we

assume

that

replicated

flits

corresponding

to

blocked

output

buffers

-LRB-

links

-RRB-

are

stored

in

the

switch

's

central

queue

Because

of

this

approach

adopted

in

the

presence

of

blocked

links

channel

dependency

does

not

lead

to

deadlock

However

deadlock

can

occur

because

of

buffer

dependencies

across

multiple

worms

involving

the

central

queue

Consider

a

switch

with

a

central

queue

of

size

flits

Assume

two

downward

bound

worms

A

and

B

using

the

NLF

scheme

to

perform

broadcast

Each

worm

requires

to

make

copies

and

forward

them

to

the

downward

bound

output

links

of

the

switch

Consider

an

instance

where

worm

A

grabs

two

of

the

downward

bound

output

links

and

worm

B

grabs

the

other

two

Four

replicas

are

made

of

A

's

first

flit

Out

of

these

four

replicas

two

are

forwarded

to

the

links

that

A

holds

and

the

other

two

are

forwarded

to

the

central

queue

as

their

corresponding

links

are

unavailable

to

A

The

same

happens

for

B

's

first

flit

Now

when

the

second

flits

of

A

and

B

arrive

they

can

be

forwarded

to

the

links

that

the

worms

have

held

However

no

replication

can

be

made

for

the

unavailable

links

because

the

central

queue

is

full

According

to

the

mechanism

described

in

section

the

third

flits

of

the

worms

can

not

be

received

Hence

neither

A

nor

B

can

make

progress

This

indefinite

waiting

leads

to

deadlock

It

can

be

noted

that

such

deadlock

could

have

been

easily

prevented

by

providing

a

sufficiently

large

central

queue

Let

us

now

examine

the

amount

of

storage

required

in

the

central

queue

to

avoid

deadlocks

for

broadcast

Consider

a

k

k

switch

with

input

buffers

and

output

buffers

Out

of

these

k

input

and

k

output

buffers

are

used

for

propagating

upward

worms

to

later

stages

of

the

MIN

Similarly

the

remaining

k

input

and

k

output

buffers

are

used

for

downward

worms

Let

l

bytes

be

the

maximum

worm

length

allowed

in

the

system

We

analyze

below

the

storage

requirements

under

a

generalized

framework

and

under

two

proposed

mechanisms

Requirement

Under

Generalized

Framework

The

degree

of

replication

of

a

worm

at

a

switch

affects

the

storage

requirement

for

deadlock

avoidance

Therefore

let

us

first

classify

worms

based

on

their

degree

of

replication

at

a

switch

Definition

If

the

flits

of

a

worm

are

replicated

p

times

at

a

switch

the

worm

is

defined

to

be

a

p-replicated

worm

at

that

switch

Definition

If

a

worm

is

simply

forwarded

to

the

corresponding

output

port

at

a

switch

i.e.

it

undergoes

no

replication

it

is

defined

to

be

a

worm

at

that

switch

Let

us

now

examine

the

storage

requirements

at

a

switch

when

m

d-replicated

worms

which

contend

for

the

same

set

of

d

buffers

-LRB-

output

links

-RRB-

arrive

at

the

switch

In

our

discussion

below

we

use

the

terms

output

buffer

and

output

links

interchangeably

Lemma

If

m

d-replicated

worms

which

arrive

at

a

switch

simultaneously

contend

for

the

same

d

output

buffers

and

each

of

them

manages

to

grab

pi

buffers

-LRB-

pi

m

i

m

Pmi

pi

d

-RRB-

then

a

central

queue

with

l

-LRB-

d

Pmi

-LRB-

d

pi

-RRB-

-RRB-

bytes

storage

is

required

to

prevent

deadlock

Proof

Deadlock

can

be

prevented

only

if

progress

can

occur

for

all

the

d

links

and

they

become

free

in

a

finite

amount

of

time

A

link

is

freed

only

if

a

complete

worm

has

been

forwarded

on

it

Consider

the

ith

d-replicated

worm

This

worm

can

be

forwarded

on

each

of

the

pi

channels

it

is

holding

only

if

there

is

sufficient

space

in

the

central

queue

to

accommodate

the

d

pi

replicated

worms

intended

for

the

output

links

that

are

not

available

to

it

Therefore

for

the

links

held

by

the

ith

d-replicated

worm

to

be

freed

there

must

be

-LRB-

d

pi

-RRB-

l

bytes

of

space

in

the

central

queue

Considering

all

m

d-replicated

worms

and

the

fact

that

all

d

links

have

to

be

freed

it

follows

that

l

Pmi

-LRB-

d

pi

-RRB-

bytes

are

required

in

the

central

queue

to

avoid

deadlock

The

above

analysis

only

considers

the

snapshot

at

a

given

switch

However

a

replicated

worm

on

one

of

the

output

links

may

be

blocked

at

the

next

stage

switch

due

to

the

nonavailability

of

output

links

at

that

stage

This

indicates

that

even

for

the

pi

links

being

held

by

the

ith

d-replicated

worm

lpi

bytes

of

storage

are

required

in

the

central

queue

Considering

all

m

worms

l

Pmi

pi

ld

bytes

of

additional

storage

are

required

in

the

central

queue

Therefore

for

deadlock

avoidance

we

require

a

central

queue

of

overall

size

l

-LRB-

d

Pmi

-LRB-

d

pi

-RRB-

-RRB-

bytes

Requirement

With

Specific

Mechanisms

We

now

propose

two

mechanisms

to

handle

worms

at

a

given

switch

For

each

of

the

mechanisms

we

derive

the

storage

requirements

to

avoid

deadlock

No

Buffering

When

All

Output

Links

are

Busy

Under

this

mechanism

when

a

p-replicated

worm

arrives

at

a

switch

a

check

is

made

to

detect

if

at

least

one

of

the

required

p

output

buffers

is

free

If

of

the

p

buffers

are

free

the

worm

is

blocked

in

place

i

e

no

replication/buffering

occurs

This

prevents

any

successive

flits

of

the

same

worm

from

arriving

on

the

same

input

link

It

also

prevents

flits

belonging

to

any

other

worm

from

arriving

on

the

input

link

If

at

least

one

of

the

p

buffers

is

free

the

corresponding

output

link

is

reserved

p

copies

are

made

of

each

of

the

arriving

flits

of

the

worm

with

the

copies

corresponding

to

the

unavailable

links

being

forwarded

to

the

central

queue

as

discussed

earlier

By

extension

of

this

mechanism

to

worms

if

the

link

required

by

the

worm

is

free

the

worm

is

forwarded

on

that

link

otherwise

the

worm

is

buffered

in

place

as

with

traditional

wormhole

routing

Let

us

now

derive

the

storage

requirements

in

the

central

queue

for

deadlock

freedom

in

a

system

using

this

mechanism

Lemma

For

any

n

if

n

worms

arrive

at

a

switch

with

no

central

queue

space

deadlock

can

not

occur

Proof

Let

us

consider

the

worst

case

where

all

n

worms

contend

for

the

same

output

link

If

one

of

these

worms

gets

the

link

it

holds

on

to

it

till

the

entire

worm

has

been

transmitted

Since

a

worm

is

of

finite

length

the

link

will

be

freed

in

finite

time

-LRB-

the

other

n

worms

may

be

blocked

in

place

-RRB-

One

of

the

remaining

n

worms

can

now

grab

the

output

link

and

will

hold

it

for

finite

time

Similarly

each

of

the

other

worms

holds

the

link

for

finite

time

before

releasing

it

Therefore

of

the

worms

waits

for

an

infinite

amount

of

time

for

the

link

Therefore

deadlock

does

not

occur

Let

us

examine

the

buffer

usage

at

a

given

switch

at

any

given

instant

It

can

be

observed

that

Observation

A

worm

which

holds

an

output

buffer

at

a

switch

can

either

be

-LRB-

a

-RRB-

a

replica

of

a

worm

which

holds

an

input

buffer

-LRB-

b

-RRB-

a

replica

of

a

worm

that

has

been

enqueued

in

the

central

queue

or

-LRB-

c

-RRB-

a

worm

whose

last

flits

are

about

to

be

forwarded

Theorem

To

avoid

deadlocks

in

a

MIN

system

with

k?k

switches

using

the

NLF

-LCB-

Broadcast

algorithm

a

central

queue

of

size

bytes

is

required

where

l

bytes

is

the

maximum

supported

worm

length

To

avoid

deadlocks

in

a

similar

MIN

system

that

uses

the

LF

-LCB-

Broadcast

algorithm

a

central

queue

of

size

bytes

is

sufficient

Proof

If

the

system

adopts

the

NLF-Broadcast

scheme

in

the

worst

case

it

can

receive

k

phase

-LRB-

upward

bound

-RRB-

NLF

-LCB-

Broadcast

worms

and

k

phase

-LRB-

downward

bound

-RRB-

worms

Since

the

upward

bound

worms

are

worms

by

Lemma

no

central

queue

space

is

required

for

them

If

the

k

upward

bound

worms

arrive

and

find

all

the

upward

bound

links

blocked

they

can

be

blocked

in

place

as

with

traditional

wormhole

routing

since

the

upward

bound

links

will

be

released

in

finite

time

The

k

phase

worms

are

k-replicated

worms

which

contend

for

the

k

output

buffers

meant

for

the

downward

bound

worms

In

the

worst

case

each

of

them

could

have

got

hold

of

only

one

of

the

k

output

links

required

By

Lemma

a

central

queue

of

size

l

-LRB-

k

Pki

-LRB-

k

-RRB-

-RRB-

bytes

is

required

to

avoid

deadlock

Therefore

to

avoid

deadlock

using

the

NLF

scheme

a

central

queue

of

size

bytes

is

required

Note

that

in

the

case

where

of

the

k-replicated

downward

worms

-LRB-

there

are

k

such

worms

-RRB-

arriving

at

a

switch

is

able

to

get

hold

of

an

output

buffer

the

output

buffers

are

occupied

by

worms

that

satisfy

cases

-LRB-

b

-RRB-

or

-LRB-

c

-RRB-

of

Observation

The

case

therefore

represents

either

a

later

snapshot

of

the

worst

case

mentioned

above

or

a

later

snapshot

of

a

better

case

Either

way

the

links

will

be

freed

in

finite

time

because

the

worms

are

of

finite

length

and

they

have

either

been

fully

queued

or

have

their

last

flits

occupying

the

output

buffer

Once

the

output

links

are

free

a

scenario

such

as

the

worst

case

above

can

now

occur

Since

worms

which

have

not

yet

grabbed

an

output

buffer

are

blocked

in

place

the

scenario

where

of

the

k-replicated

worms

gets

hold

of

an

output

buffer

does

not

represent

a

worse

case

in

terms

of

central

queue

storage

requirement

for

deadlock

freedom

Therefore

the

case

that

we

have

considered

above

is

the

worst

case

-LRB-

in

terms

of

buffer

requirements

-RRB-

On

the

other

hand

if

the

system

uses

the

LF

scheme

both

upward

and

downward

bound

messages

arriving

at

a

switch

are

k-replicated

As

with

the

NLF

scheme

in

the

worst

case

k

upward

and

k

downward

bound

worms

arrive

at

a

switch

The

downward

bound

k-replicated

worms

-LRB-

there

are

k

of

these

worms

-RRB-

contend

for

the

same

k

output

buffers

and

therefore

require

bytes

of

central

queue

storage

for

deadlock

avoidance

Of

the

upward

bound

k-replicated

worms

-LRB-

there

are

k

of

these

worms

-RRB-

in

phase

any

two

can

contend

for

at

most

k

output

buffers

However

every

pair

of

worms

can

contend

for

a

different

set

of

k

buffers

The

union

of

the

sets

of

these

k

buffers

that

the

worms

contend

for

includes

the

set

of

k

output

buffers

for

downward

bound

worms

If

we

assume

that

the

k

upward

bound

worms

contend

for

the

same

k

output

buffers

-LRB-

a

case

that

is

worse

than

the

actual

-RRB-

then

bytes

of

space

are

sufficient

for

the

k

phase

worms

to

pass

through

a

switch

without

deadlock

Therefore

a

central

queue

of

size

bytes

is

sufficient

to

avoid

deadlocks

with

the

LF

scheme

The

above

result

implies

that

for

a

system

such

as

the

IBM

with

switches

and

a

maximum

supported

worm

length

of

bytes

a

central

queue

of

size

KBytes

can

be

used

to

prevent

deadlock

while

using

the

NLF

scheme

For

the

LF

scheme

KBytes

are

required

Even

with

the

current

configuration

of

having

a

central

queue

of

size

KByte

the

proposed

broadcast

can

be

achieved

for

messages

of

length

up

to

bytes

-LRB-

for

LF

-RRB-

and

bytes

-LRB-

for

NLF

-RRB-

without

deadlock

Allowing

Buffering

of

Worms

Under

this

mechanism

p-replicated

worms

-LRB-

p

-RRB-

behave

as

discussed

earlier

under

Mechanism

If

the

worm

is

a

worm

and

the

required

output

buffer

is

free

the

worm

is

forwarded

on

the

corresponding

link

If

the

required

output

buffer

is

not

free

the

worm

is

stored

in

the

central

queue

The

above

approach

increases

the

storage

requirement

in

the

central

queue

for

deadlock

avoidance

if

we

assume

that

and

p-replicated

-LRB-

p

-RRB-

worms

share

the

same

central

queue

space

since

blocked

worms

need

to

be

enqueued

-LRB-

and

not

buffered

in

place

as

with

the

previous

mechanism

-RRB-

However

the

mechanism

has

the

advantage

that

contention

will

be

reduced

for

traffic

with

larger

proportions

of

worms

The

assumption

of

a

common

central

queue

storage

for

and

p-replicated

-LRB-

p

-RRB-

worms

makes

estimation

of

buffer

requirements

for

deadlock

very

complicated

when

we

consider

the

case

where

waves

of

multiple

broadcasts/multicasts

occur

A

worm

enqueued

in

a

given

wave

may

prevent

a

p-replicated

-LRB-

p

-RRB-

worm

of

a

next

wave

from

progressing

Consider

the

following

scenario

where

a

p-replicated

worm

is

able

to

grab

a

few

of

the

output

links

it

requires

but

there

is

no

space

in

the

central

queue

for

the

replicas

corresponding

to

the

blocked

links

because

the

central

queue

space

has

been

used

up

by

worms

of

previous

waves

The

p-replicated

worm

will

now

hold

onto

the

links

that

it

has

grabbed

but

it

can

not

make

progress

till

there

is

enough

space

in

the

central

queue

to

hold

the

replicas

corresponding

to

the

links

that

are

unavailable

to

it

Space

may

be

created

in

the

central

queue

by

the

forwarding

of

some

of

the

enqueued

worms

on

links

that

become

available

However

if

the

switch

controller

is

not

well

designed

this

space

may

be

taken

up

by

arriving

worms

destined

for

one

of

the

links

held

by

the

blocked

p-replicated

worm

The

p-replicated

worm

may

thus

get

starved

for

central

queue

space

and

this

eventually

leads

to

deadlock

We

therefore

propose

that

a

separate

part

of

the

central

queue

space

be

reserved

for

replicated

worms

The

amount

of

space

that

should

be

reserved

should

be

decided

by

the

designer

based

on

performance

-LRB-

reduced

latency

and

higher

throughput

-RRB-

considerations

Note

that

such

space

reserved

for

the

unicast

worms

is

solely

for

increasing

performance

by

reducing

link

contention

The

amount

of

space

reserved

for

worms

can

be

varied

from

one

application

to

another

by

taking

into

account

the

maximum

degree

of

replication

required

for

broadcasts/multicasts

of

the

application

The

buffer

space

for

deadlock

avoidance

derived

above

represents

the

worst

case

space

requirement

for

multiple

broadcasts

it

is

clear

that

different

applications

have

different

multicast/broadcast

patterns

Hence

the

amount

of

space

reserved

for

worms

can

be

varied

from

one

application

to

another

by

taking

into

account

the

maximum

degree

of

replication

required

for

broadcasts/multicasts

of

the

application

Depending

on

the

application

multicasts

may

require

lower

degrees

of

replication

at

the

switches

and

therefore

may

have

a

lower

storage

requirement

for

deadlock

avoidance

In

such

cases

the

unused

part

of

the

queue

space

reserved

for

p-replicated

worms

-LRB-

p

-RRB-

may

be

used

for

blocked

worms

to

increase

network

performance

Since

the

space

reserved

for

worms

is

a

design

parameter

we

will

not

consider

it

as

part

of

the

storage

requirements

for

deadlock

freedom

Hence

we

do

not

derive

the

related

storage

requirement

in

this

paper

Performance

Evaluation

In

this

section

we

present

results

of

our

performance

evaluation

for

broadcast

and

multicast

operations

We

compare

multidestination

and

unicast-based

schemes

for

different

system

architectural

and

technological

parameters

Broadcast

We

first

considered

single-source

broadcast

operation

Based

on

the

analytical

models

derived

in

the

earlier

section

we

compared

both

schemes

The

following

parameters

representing

the

current

trend

in

technology

were

assumed

ts

-LRB-

communication

start-up

time

-RRB-

as

and

microseconds

tp

-LRB-

link

propagation

time

-RRB-

as

nsec

tsw

-LRB-

switch

delay

-RRB-

as

nsec

and

tcopy

-LRB-

time

for

copying

flit

from

the

input

buffer

to

one

output

buffer

of

a

switch

-RRB-

as

nsec

Overall

Comparison

Figure

shows

the

comparison

results

for

different

MIN

sizes

-LRB-

and

processors

-RRB-

Four

different

switch

configurations

-LRB-

and

-RRB-

were

assumed

The

message

length

was

kept

fixed

at

flits

It

can

be

observed

that

for

all

values

of

ts

and

switch

sizes

the

multidestination

scheme

implements

faster

broadcast

than

the

unicast-based

scheme

The

reduction

in

broadcast

latency

can

be

observed

to

be

in

a

range

of

-LCB-

depending

on

the

system

and

switch

size

With

increase

in

switch

size

the

unicast-based

scheme

performs

better

compared

to

itself

for

for

a

given

system

size

This

is

because

of

the

reduced

number

of

stages

in

a

system

with

larger

switch

For

the

multidestination

scheme

the

impact

of

switch

size

is

minimal

for

higher

values

of

ts

However

as

ts

reduces

the

broadcast

latency

using

multidestination

worm

increases

with

switch

size

This

increase

is

attributed

to

the

replication

cost

at

a

given

switch

In

our

current

implementation

we

assume

sequential

copying

of

a

flit

at

an

input

buffer

to

all

output

buffers

Hence

the

overall

latency

increases

with

increase

in

switch

size

Such

increase

in

latency

can

be

avoided

by

using

parallel

copying

at

the

switch

For

systems

like

IBM

which

uses

switches

it

can

be

observed

that

even

with

a

very

low

ts

-LRB-

around

microsec

-RRB-

the

broadcast

latency

can

be

reduced

by

a

factor

of

for

a

system

by

using

the

proposed

multidestination

scheme

Impact

of

Message

Length

Figure

shows

the

impact

of

message

length

for

broadcast

operation

in

a

MIN

It

can

be

seen

that

the

multidestination

scheme

always

delivers

lower

latency

-LRB-

up

to

a

factor

of

500 1000 1500 2000

2500 3000 3500 4000

Broadcast

Latency

-LRB-

microsec

-RRB-

System

size

-LRB-

total

number

of

nodes

-RRB-

Broadcast

-LRB-

ts

microsec

L

flits

-RRB-

unicast

k

unicast

k

unicast

k

unicast

k

multidest

k

multidest

k

multidest

k

multidest

k

500 1000 1500 2000

2500 3000 3500 4000

Broadcast

Latency

-LRB-

microsec

-RRB-

System

size

-LRB-

total

number

of

nodes

-RRB-

Broadcast

-LRB-

ts

microsec

L

flits

-RRB-

unicast

k

unicast

k

unicast

k

unicast

k

mutlidest

k

multidest

k

multidest

k

multidest

k

500 1000 1500 2000

2500 3000 3500 4000

Broadcast

Latency

-LRB-

microsec

-RRB-

System

size

-LRB-

total

number

of

nodes

-RRB-

Broadcast

-LRB-

ts

microsec

L

flits

-RRB-

unicast

k

unicast

k

unicast

k

unicast

k

multidest

k

multidest

k

multidest

k

multidest

k

500 1000 1500 2000

2500 3000 3500 4000

Broadcast

Latency

-LRB-

microsec

-RRB-

System

size

-LRB-

total

number

of

nodes

-RRB-

Broadcast

-LRB-

ts

microsec

L

flits

-RRB-

unicast

k

unicast

k

unicast

k

unicast

k

multidest

k

multidest

k

multidest

k

multidest

k

Figure

Comparison

of

multidestination

and

unicast-based

schemes

to

implement

single

source

broadcast

on

MINs

of

different

size

-LRB-

and

-RRB-

with

different

switch

sizes

-LRB-

k

k

with

k

and

-RRB-

and

communication

start-up

time

-LRB-

and

microseconds

-RRB-

The

message

length

was

kept

fixed

at

flits

-RRB-

for

smaller

message

size

and

smaller

switch

size

Even

for

long

messages

-LRB-

message

length

of

flits

-RRB-

on

a

system

having

switches

and

ts

microsec

the

multidestination

scheme

implements

broadcast

with

a

latency

of

only

microsec

compared

to

microsec

needed

by

the

unicast-based

scheme

For

ts

microsec

the

respective

values

are

and

microsec

respectively

Hence

it

can

be

observed

that

for

systems

with

smaller

switches

the

multidestination

scheme

provides

scalable

broadcasting

200 400 600 800

Broadcast

Latency

-LRB-

microsec

-RRB-

Packet

Size

-LRB-

flits

-RRB-

Broadcast

-LRB-

ts

microsec

-RRB-

unicast

k

unicast

k

unicast

k

unicast

k

muitidest

k

multidest

k

multidest

k

multidest

k

200 400 600 800

Broadcast

Latency

-LRB-

microsec

-RRB-

Packet

Size

-LRB-

flits

-RRB-

Broadcast

-LRB-

ts

microsec

-RRB-

unicast

k

unicast

k

unicast

k

unicast

k

muitidest

k

multidest

k

multidest

k

multidest

k

Figure

Impact

of

message

length

in

implementing

single

source

broadcast

on

MIN

with

multidestination

and

unicast-based

schemes

for

different

switch

sizes

-LRB-

k

k

with

k

and

-RRB-

and

communication

start-up

time

-LRB-

and

microseconds

-RRB-

Multicast

We

considered

the

performance

of

the

multidestination

based

LF

algorithm

and

the

unicast

based

multicast

algorithm

under

the

following

two

frameworks

-LCB-

-LRB-

a

-RRB-

multicast

latency

for

fixed

system

sizes

with

varying

number

of

destinations

and

switch

sizes

and

-LRB-

b

-RRB-

best

and

worst

case

multicast

latency

for

varying

system

size

Overall

Comparison

We

studied

the

total

completion

time

for

varying

degree

of

multicast

for

systems

with

size

of

and

processors

for

different

switch

sizes

-LRB-

k

and

additionally

for

the

system

size

of

-RRB-

We

assumed

a

startup

time

-LRB-

ts

-RRB-

of

microseconds

and

a

fixed

message

length

of

bytes

It

was

observed

that

the

multidestination

based

scheme

takes

almost

constant

time

for

multicast

with

increase

in

the

number

of

destinations

For

a

system

the

scheme

shows

an

improvement

by

a

factor

of

over

the

unicast

based

scheme

for

multicasts

beyond

destinations

Considering

differing

switch

sizes

we

find

that

the

switch

with

k

performs

marginally

better

than

the

switch

with

k

The

reason

for

this

is

that

for

larger

switch

size

more

time

is

spent

on

replication

in

Phase

of

the

multicast

scheme

-LRB-

more

replicas

have

to

be

made

at

every

stage

-RRB-

For

a

system

size

of

the

multidestination

scheme

shows

an

improvement

by

a

factor

of

to

for

multicasts

involving

or

more

destinations

With

increase

in

system

size

the

header

length

starts

dominating

and

contribute

significantly

to

the

delay

-LRB-

using

the

bit

string

encoding

for

a

processor

system

the

header

has

a

fixed

length

of

bytes

with

link

-RRB-

The

use

of

better

encoding

schemes

can

result

in

even

better

performance

using

the

multidestination

scheme

For

a

system

size

of

the

header

length

begins

to

make

a

prominent

impact

on

the

delay

encountered

by

the

multidestination

based

multicast

-LRB-

the

header

length

is

bytes

under

the

bit

encoding

scheme

-RRB-

Despite

this

the

multidestination

scheme

shows

a

factor

of

two

improvement

for

switch

sizes

of

and

and

a

factor

of

for

a

switch

size

of

50 100 150 200

Multicast

Latency

-LRB-

microsec

-RRB-

Number

of

destinations

Multicast

-LRB-

ts

microsec

L

flits

System

Size

-RRB-

unicast

-LRB-

k

-RRB-

unicast

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

200 400 600 800

Multicast

Latency

-LRB-

microsec

-RRB-

Number

of

destinations

Multicast

-LRB-

ts

microsec

L

flits

System

Size

-RRB-

unicast

-LRB-

k

-RRB-

unicast

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

500 1000 1500 2000

2500 3000 3500 4000

Multicast

Latency

-LRB-

microsec

-RRB-

Number

of

destinations

Multicast

-LRB-

ts

microsec

L

flits

System

Size

-RRB-

unicast

-LRB-

k

-RRB-

unicast

-LRB-

k

-RRB-

unicast

-LRB-

k

-RRB-

unicast

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

multidest

-LRB-

k

-RRB-

Figure

Comparison

of

multidestination

and

unicast-based

schemes

to

implement

single

source

multicast

on

MINs

of

different

system

sizes

-LRB-

-RRB-

for

varying

number

of

destinations

and

switch

sizes

-LRB-

k

k

with

k

and

-RRB-

The

communication

start-up

time

was

assumed

to

be

microsec

and

the

message

length

was

kept

fixed

at

flits

Impact

of

System

Size

Figure

compares

the

best

case

and

worst

case

latency

of

both

schemes

The

best

case

occurs

when

the

number

of

destinations

is

only

Similarly

the

worst

case

is

indicated

by

multicasting

to

-LRB-

n

-RRB-

destinations

For

best

case

it

can

be

seen

that

the

multidestination

scheme

performs

better

for

both

switch

sizes

For

the

worst

case

the

multidestination

scheme

also

perform

better

For

system

with

switches

the

multidestination

scheme

is

able

to

reduce

the

cost

by

a

factor

of

Even

for

system

the

improvement

is

by

a

factor

of

It

is

to

be

noted

that

the

unicast-based

scheme

being

considered

is

a

static

algorithm

where

the

destinations

are

known

apriori

and

the

send-recv

communication

pairs

are

compiled

statically

Hence

the

unicast-based

scheme

does

not

carry

the

destinations

in

its

header

The

multidestination

scheme

carries

the

destinations

in

its

header

and

can

be

used

for

both

static

and

dynamic

implementation

Hence

the

multidestination

scheme

provides

fast

and

scalable

multicast

for

large

system

sizes

The

increase

in

the

worst

case

multicast

time

with

increasing

system

size

is

due

to

the

header

becoming

very

long

as

explained

in

the

previous

section

500 1000 1500 2000

2500 3000 3500 4000

Multicast

Latency

-LRB-

microsec

-RRB-

System

size

-LRB-

total

number

of

nodes

-RRB-

Multicast

-LRB-

ts

microsec

L

flits

k

-RRB-

unicast

-LRB-

worst

case

-RRB-

unicast

-LRB-

best

case

-RRB-

multidest

-LRB-

worst

case

-RRB-

multidest

-LRB-

best

case

-RRB-

500 1000 1500 2000

2500 3000 3500 4000

Multicast

Latency

-LRB-

microsec

-RRB-

System

size

-LRB-

total

number

of

nodes

-RRB-

Multicast

-LRB-

ts

microsec

L

flits

k

-RRB-

unicast

-LRB-

worst

case

-RRB-

unicast

-LRB-

best

case

-RRB-

multidest

-LRB-

worst

case

-RRB-

multidest

-LRB-

best

case

-RRB-

Figure

Comparison

of

multidestination

and

unicast-based

schemes

to

implement

single

source

multicast

on

MINs

of

different

size

-LRB-

and

-RRB-

with

different

switch

sizes

-LRB-

k?k

with

k

and

-RRB-

The

communication

start-up

time

was

assumed

to

be

microsec

and

the

message

length

was

kept

fixed

at

flits

Conclusions

and

Future

Research

In

this

paper

we

have

presented

a

new

approach

to

implement

fast

and

scalable

broadcast

and

multicast

in

wormhole

multistage

networks

using

multidestination

worms

The

basic

mechanism

is

presented

together

with

the

architectural

supports

at

a

switch

Two

algorithms

No

Loopback

on

Forward

Propagation

-LRB-

NLF

-RRB-

and

Loopback

on

Forward

Propagation

-LRB-

LF

-RRB-

are

presented

and

evaluated

for

various

architectural

system

and

technological

parameters

The

storage

requirement

at

a

switch

to

achieve

deadlock

free

broadcast

and

multicast

using

the

proposed

schemes

are

derived

It

is

shown

that

both

these

algorithms

can

implement

broadcast

and

arbitrary

multicast

with

single

communication

start-up

For

an

n-processor

system

compared

to

the

unicast-based

scheme

the

proposed

framework

demonstrates

an

asymptotic

reduction

of

n

communication

steps

for

broadcast

operation

Similarly

the

reduction

is

by

a

factor

of

d

for

arbitrary

multicast

to

d

destinations

Analytical

results

indicate

that

the

proposed

scheme

is

far

superior

and

scalable

compared

to

the

unicast-based

scheme

for

smaller

switch

sizes

and

higher

communication

start-up

cost

For

the

contemporary

MIN

technology

with

and

switches

the

scheme

reduces

broadcast

and

multicast

latency

significantly

Hence

this

framework

is

quite

suitable

for

current

and

future

generation

wormhole

MINs

In

this

paper

we

have

studied

the

usage

of

multidestination

mechanism

for

broadcast

and

multicast

operation

We

are

extending

our

work

to

other

collective

communication

patterns

like

barrier

synchronization

global

reduction

and

parallel-prefix

operations

As

the

switch

size

grows

the

proposed

scheme

encounters

more

delay

due

to

sequential

replication

cost

at

a

switch

We

are

exploring

alternative

schemes

to

make

the

scheme

profitable

even

for

bigger

switches

Acknowledgements

The

authors

would

like

to

thank

Dr.

Craig

Stunkel

of

IBM

T.

J.

Watson

Research

Center

for

having

initial

discussion

on

this

topic

The

authors

would

also

like

to

thank

other

members

of

the

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

in

the

department

for

providing

comments

criticisms

and

suggestions

to

this

work

References

-LSB-

-RSB-

M.

Barnett

D.

G.

Payne

and

R.

Van

de

Geijn

Optimal

Broadcasting

in

Mesh-Connected

Architectures

Technical

Report

Dept.

of

Computer

Science

Universityof

Texas

at

Austin

Dec

-LSB-

-RSB-

C.-M

Chiang

and

L.

M.

Ni

Multi-Address

Encoding

for

Multicast

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

May

-LSB-

-RSB-

A.

A.

Chien

and

J.

H.

Kim

Planar-Adaptive

Routing

Low-Cost

Adaptive

Networks

for

Multiprocessors

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

Deadlock-Free

Message

Routing

in

Multiprocessor

Interconnection

Networks

IEEE

Transactions

on

Computers

pages

-LCB-

May

-LSB-

-RSB-

C.

J.

Glass

and

L.

Ni

The

Turn

Model

for

Adaptive

Routing

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

V.

Kumar

A.

Grama

A.

Gupta

and

G.

Karypis

Introduction

to

Parallel

Computing

Design

and

Analysis

of

Algorithms

Benjamin/Cummings

-LSB-

-RSB-

X.

Lin

P.

K.

McKinley

and

L.

M.

Ni

Performance

Evaluation

of

Multicast

Wormhole

Routing

in

Multicomputers

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Direct

Networks

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

II

-LCB-

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

D.

K.

Panda

Fast

Barrier

Synchronization

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Worms

In

International

Symposium

on

High

Performance

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

Global

Reduction

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Exchange

Worms

In

International

Parallel

Processing

Symposium

Apr

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

-LSB-

-RSB-

C.

B.

Stunkel

D.

G.

Shea

B.

Abali

M.

M.

Denneeau

P.

H.

Hochschild

D.

J.

Joseph

B.

J.

Nathanson

M.

Tsao

and

P.

R.

Varker

Architecture

and

Implementation

of

Vulcan

In

Proceedings

of

the

Int

l

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

Craig

Stunkel

D.

Shea

D.

G.

Grice

P.

H.

Hochschild

and

M.

Tsao

The

High

Performance

Switch

In

Scalable

High

Performance

Computing

Conference

pages

-LCB-

-LSB-

-RSB-

H.

Xu

Y.-D

Gui

and

L.

M.

Ni

Optimal

Software

Multicast

in

Wormhole-Routed

Multistage

Networks

In

Proceedings

of

the

Supercomputing

Conference

pages

pp.

-LCB-

broadcast
collective communication
multicast
multistage networks
wormhole routing