Multicasting

on

Switch-based

Irregular

Networks

using

Multi-drop

Path-based

Multidestination

Worms

Ram

Kesavan

and

Dhabaleswar

K.

Panda

Technical

Report

Multicasting

on

Switch-based

Irregular

Networks

using

Multi-drop

Path-based

Multidestination

Worms

Ram

Kesavan

and

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

E-mail

fkesavan,pandag@cis.ohio-state.edu

Contact

Author

Dhabaleswar

K.

Panda

Abstract

This

paper

presents

a

novel

concept

of

multi-drop

path-based

multidestination

message

passing

on

switch-based

irregular

networks

First

the

multi-drop

mechanism

is

defined

with

an

associated

header

encoding

scheme

and

this

mechanism

is

used

to

develop

path-based

multidestination

worms

Next

a

method

is

proposed

to

identify

valid

multidestination

paths

on

arbitrary

irregular

networks

with

a

typical

deadlock-free

routing

Then

the

deadlock-free

property

of

multi-drop

path-based

worms

is

emphasized

Using

the

above

concepts

three

multicast

algorithms

are

proposed

multi-drop

path-based

greedy

-LRB-

MDP-G

-RRB-

multi-drop

path-based

lessgreedy

-LRB-

MDP-LG

-RRB-

and

multi-drop

path-based

binomial

-LRB-

MDP-B

-RRB-

The

proposed

algorithms

are

compared

with

each

other

and

with

the

best

unicast

based

algorithm

the

CCO

-LSB-

-RSB-

for

a

range

of

system

and

technological

parameters

The

MDP-LG

scheme

is

shown

to

be

the

best

to

implement

multicast

with

reduced

latency

research

is

supported

in

part

by

NSF

Grant

and

NSF

Career

Award

Contents

Introduction

Switch-based

Irregular

Networks

Irregular

Interconnection

The

Autonet

Routing

Multidestination

Message

Passing

Multidestination

Message

Passing

and

Path-based

Worms

Tree-based

Multidestination

Worms

Multi-drop

Path-based

Multidestination

Worms

The

Multi-drop

Mechanism

The

Multi-drop

Header

Encoding

Scheme

Creation

of

Multi-drop

Worms

on

Irregular

Networks

Valid

Multidestination

Paths

with

the

Autonet

Routing

Deadlock-freedom

Multicast

Algorithms

Construction

of

POCs

The

Multi-drop

Path-based

Greedy

-LRB-

MDP-G

-RRB-

Algorithm

The

Multi-drop

Path-based

Less

Greedy

-LRB-

MDP-LG

-RRB-

Algorithm

The

Multi-drop

Path-based

Binomial

-LRB-

MDP-B

-RRB-

Algorithm

Simulation

Experiments

and

Results

Effect

of

system

size

Effect

of

message

length

Effect

of

communication

start-up

time

Effect

of

switch

size

Effect

of

degree

of

network

connectivity

Conclusions

Introduction

The

cut-through/wormhole-routing

switching

technique

is

becoming

the

trend

in

building

future

parallel

systems

due

to

its

inherent

advantages

like

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

Intel

Paragon

Cray

Ncube

J-Machine

and

Stanford

DASH

are

systems

using

this

switching

technique

These

systems

have

regular

network

topologies

with

various

deadlock-free

routing

schemes

More

recently

this

switching

technique

is

being

applied

to

switch-based

interconnects

like

Myrinet

-LSB-

-RSB-

and

ServerNet

-LSB-

-RSB-

to

build

networks

of

workstations

for

cost-effective

parallel

computing

Such

networks

typically

have

irregular

topology

which

provides

flexibility

to

design

scalable

systems

with

incremental

expansion

capability

The

irregularity

also

makes

the

routing

on

such

systems

quite

complicated

Typically

deadlock-free

adaptive

routing

schemes

-LSB-

-RSB-

are

supported

on

such

systems

to

achieve

low-latency

and

high-bandwidth

communication

In

order

to

to

support

the

paradigms

of

distributed

memory

or

distributed-shared

memory

these

systems

need

fast

implementation

of

collective

communication

operations

-LRB-

broadcast

multicast

global

combine

and

barrier

synchronization

-RRB-

-LSB-

-RSB-

Traditionally

wormhole-routed

systems

support

only

point-to-point

-LRB-

unicast

-RRB-

message

passing

This

mechanism

allows

a

message

to

have

exactly

one

destination

This

leads

to

collective

communication

operations

being

implemented

as

multiple

phases

of

unicast

message

exchange

-LSB-

-RSB-

In

order

to

reduce

the

number

of

phases

to

implement

collective

communication

operations

a

Hamiltonian

Path-based

scheme

was

proposed

on

MPP

networks

-LSB-

-RSB-

This

scheme

uses

a

modified

router

organization

to

support

propagation

of

multidestination

messages

-LRB-

a

message

with

multiple

destinations

-RRB-

and

allows

a

message

to

be

consumed

at

a

destination

node

while

being

forwarded

to

the

next

node

concurrently

Such

an

approach

demonstrated

that

collective

communication

operations

can

be

implemented

on

MPP

networks

with

reduced

latency

This

concept

was

futher

extended

to

allow

multidestination

worms

to

take

row-column

paths

-LSB-

-RSB-

and

all

paths

conforming

to

the

base

routing

using

the

Base

Routing

Conformed

Path

-LRB-

BRCP

-RRB-

model

-LSB-

-RSB-

Modern

cut-through

switches

possess

the

capability

of

simultaneous

replication

of

an

incoming

worm

to

multiple

output

ports

-LSB-

-RSB-

Although

this

facility

has

been

exploited

for

implementing

broadcast

by

flooding

the

network

there

has

been

little

prior

work

on

using

this

capability

for

arbitrary

multicast

Recently

a

multiport

encoding

scheme

has

been

proposed

to

construct

multidestination

worms

in

multistage

interconnection

networks

-LSB-

-RSB-

and

the

associated

architectural

issues

in

supporting

such

worms

have

been

discussed

in

-LSB-

-RSB-

This

leads

to

a

challenge

of

designing

an

efficient

path-based

multidestination

message

passing

mechanism

on

irregular

networks

This

challenge

includes

design

of

an

efficient

header

encoding

scheme

a

scheme

for

constructing

such

worms

determining

valid

paths

in

the

system

and

designing

efficient

multicast

algorithms

In

this

paper

we

take

on

the

above

challenges

First

the

multi-drop

mechanism

with

an

associated

header

encoding

is

proposed

Next

the

path-based

multidestination

message

passing

mechanism

is

developed

on

switch-based

irregular

networks

A

method

to

identify

valid

multidrop

paths

with

respect

to

the

Autonet

routing

is

proposed

and

proved

to

be

deadlock-free

Then

we

propose

three

new

multicast

algorithms

using

the

proposed

path-based

multidestination

message

passing

mechanism

a

-RRB-

the

Multi-drop

Path-based

Greedy

-LRB-

MDP-G

-RRB-

algorithm

b

-RRB-

the

Multi-drop

Path-based

Less-Greedy

-LRB-

MDP-LG

-RRB-

algorithm

and

c

-RRB-

the

Multi-drop

Pathbased

Binomial

-LRB-

MDP-B

-RRB-

algorithm

These

algorithms

are

evaluated

over

a

range

of

system

parameters

and

compared

with

the

best

unicast-based

multicast

algorithm

the

CCO

algorithm

-LSB-

-RSB-

The

MDP-LG

algorithm

shows

benefits

of

and

for

multicast

latency

when

compared

to

the

CCO

MDP-G

MDP-B

algorithms

respectively

These

results

clearly

demonstrate

that

the

multi-drop

path-based

multidestination

mechanism

promises

significant

potential

to

implement

fast

and

efficient

collective

communication

operations

in

future

switchbased

irregular

systems

The

rest

of

the

paper

is

organized

as

follows

Section

provides

an

overview

of

switchbased

irregular

networks

and

deadlock-free

routing

on

such

networks

Section

describes

the

basic

concepts

of

multidestination

message

passing

Section

presents

the

multi-drop

path-based

multidestination

mechanism

A

method

of

creating

valid

multi-drop

worms

on

irregular

networks

is

described

in

Section

Section

discusses

three

new

multicast

algorithms

Performance

evaluation

results

of

these

algorithms

are

presented

and

compared

with

those

of

the

CCO

algorithm

in

Section

Finally

concluding

remarks

are

made

in

Section

Switch-based

Irregular

Networks

In

this

section

we

provide

a

brief

overview

of

switch-based

irregular

networks

and

focus

on

a

typical

deadlock-free

routing

on

such

networks

Irregular

Interconnection

Figure

-LRB-

a

-RRB-

shows

a

typical

parallel

system

using

switch-based

interconnect

with

irregular

topology

Such

a

network

consists

of

a

set

of

switches

where

each

switch

can

have

a

set

of

bidirectional

ports

The

system

in

the

figure

consists

of

eight

switches

with

eight

ports

per

switch

Some

of

the

ports

in

each

switch

are

connected

to

processors

some

ports

are

connected

to

ports

of

other

switches

to

provide

connectivity

between

the

processors

and

some

ports

are

left

open

for

further

connections

Such

connectivity

is

typically

irregular

and

the

only

thing

that

is

guaranteed

is

that

the

network

is

connected

Thus

the

interconnection

topology

of

the

network

can

be

denoted

by

a

graph

G

-LRB-

V

E

-RRB-

where

V

is

the

set

of

switches

and

E

is

the

set

of

bidirectional

links

between

the

switches

-LSB-

-RSB-

Figure

-LRB-

b

-RRB-

shows

the

interconnection

graph

for

the

irregular

network

in

Fig.

-LRB-

a

-RRB-

The

switches

can

implement

different

types

of

switching

techniques

wormhole

virtual

cut-through

or

ATM

These

switches

can

also

implement

different

routing

schemes

-LSB-

-RSB-

In

this

paper

we

focus

on

wormhole

switches

and

assume

the

Autonet

-LSB-

-RSB-

routing

scheme

due

to

its

simplicity

and

its

commercial

implementation

The

scheme

is

briefly

described

below

The

Autonet

Routing

In

the

Autonet

routing

scheme

-LSB-

-RSB-

a

breadth-first

spanning

tree

-LRB-

BFS

-RRB-

on

G

is

first

computed

using

a

distributed

algorithm

The

edges

of

G

can

be

partitioned

into

tree

edges

and

cross

edges

According

to

the

property

of

BFS

trees

a

cross

edge

does

not

connect

two

switches

which

are

at

a

difference

of

more

than

one

level

in

the

tree

Deadlock-free

routing

is

based

on

a

loop-free

assignment

of

direction

to

the

operational

links

In

particular

the

up

end

of

each

link

is

defined

as

-RRB-

the

end

whose

switch

is

closer

to

the

root

in

the

spanning

tree

or

-RRB-

the

end

whose

switch

has

the

lower

UID

if

both

ends

are

at

switches

at

the

same

tree

level

Links

looped

back

to

the

same

switch

are

omitted

from

the

configuration

The

result

of

this

assignment

is

that

the

directed

links

do

not

form

loops

Figure

-LRB-

c

-RRB-

shows

the

BFS

spanning

tree

embedded

on

graph

G

shown

in

Fig.

-LRB-

b

-RRB-

The

assignment

of

the

up

direction

to

the

links

on

this

network

is

illustrated

The

down

direction

is

along

the

reverse

direction

of

the

link

ProcessingElement

Switch

Bidirectional

Link

-LRB-

a

-RRB-

-LRB-

b

-RRB-

up

direction

root

-LRB-

c

-RRB-

tree

link

cross

link

Figure

-LRB-

a

-RRB-

An

example

system

with

switch-based

interconnect

and

irregular

topology

-LRB-

b

-RRB-

the

corresponding

interconnection

graph

G

and

-LRB-

c

-RRB-

the

BFS

spanning

tree

rooted

at

node

of

G.

To

eliminate

deadlocks

while

still

allowing

all

links

to

be

used

this

routing

uses

the

following

up/down

rule

a

legal

route

must

traverse

zero

or

more

links

in

the

up

direction

followed

by

zero

or

more

links

in

the

down

direction

Putting

it

in

the

negative

a

packet

may

never

traverse

a

link

along

the

up

direction

after

having

traversed

one

in

the

down

direction

Therefore

this

routing

is

also

called

up

down

routing

In

order

to

implement

the

above

routing

each

switch

has

an

indexed

forwarding

table

When

a

worm

reaches

a

switch

the

destination

address

is

captured

from

the

header

flit

of

the

incoming

worm

This

address

is

concatenated

with

the

incoming

port

number

and

the

result

is

used

to

index

the

switch

's

forwarding

table

The

table

lookup

returns

the

outgoing

port

number

that

the

worm

should

be

routed

through

The

forwarding

tables

can

be

constructed

to

support

both

minimal

and

non-minimal

adaptive

routing

During

configuration

the

current

implementation

of

Autonet

switches

constructs

tables

to

support

minimal

adaptive

routing

-LSB-

-RSB-

In

this

paper

we

assume

that

the

forwarding

tables

allow

only

legal

routes

with

the

minimum

hop

count

Multidestination

Message

Passing

In

this

section

we

describe

the

basic

concepts

of

multidestination

message

passing

and

pathbased

multidestination

worms

Next

we

discuss

tree-based

multidestination

worms

on

switchbased

irregular

networks

and

show

the

motivation

for

path-based

multidestination

worms

on

such

networks

Multidestination

Message

Passing

and

Path-based

Worms

The

concept

of

wormhole

message

passing

with

multiple

destinations

was

first

introduced

by

Lin

and

Ni

for

-LSB-

-RSB-

Hamiltonian

meshes

Then

the

multidestination

message

passing

was

introduced

-LSB-

-RSB-

for

k-ary

n-cube

networks

with

different

base

routing

schemes

These

schemes

were

proposed

for

regular

direct

networks

typical

of

MPPs

In

such

networks

each

node

consists

of

a

router

and

a

processor

The

processor

is

connected

to

the

router

through

a

set

of

injection

and

consumption

channels

The

routers

are

connected

in

some

given

topology

A

path-based

multidestination

worm

in

such

networks

carries

the

ids

of

multiple

destinations

in

its

header

After

being

injected

into

the

network

by

the

source

the

worm

is

routed

based

on

the

first

destination

in

the

header

When

the

worm

reaches

the

router

of

an

intermediate

destination

the

address

of

the

destination

is

removed

from

the

header

and

the

next

destination

is

used

to

route

the

worm

onwards

As

the

data

flits

pass

through

routers

corresponding

to

intermediate

destinations

they

are

forwarded

to

the

next

router

and

absorbed

to

the

processor

The

final

destination

absorbs

the

complete

worm

Tree-based

Multidestination

Worms

The

above

path-based

multidestination

mechanism

-LRB-

with

a

header

of

destination

ids

-RRB-

can

not

be

directly

applied

to

switch-based

networks

because

there

is

no

one-to-one

correspondence

between

switches/routers

and

processors

on

such

networks

Thus

a

tree-based

mechanism

-LSB-

-RSB-

has

been

proposed

which

allows

a

multidestination

worm

to

replicate

at

switches

with

multiple

replicas

of

the

worm

following

different

paths

in

the

network

In

this

approach

the

switches

possess

the

capability

to

replicate

flits

of

an

arriving

multidestination

worm

to

multiple

output

ports

The

issues

of

synchronous

and

asynchronous

replication

of

flits

in

a

deadlock-free

manner

within

the

switch

have

been

discussed

in

detail

in

-LSB-

-RSB-

Figure

shows

an

example

of

a

tree-based

multidestination

worm

for

MINs

using

the

multiport

encoding

scheme

-LSB-

-RSB-

It

must

be

noted

that

such

a

worm

branches

out

in

a

symmetric

fashion

For

example

switches

and

replicate

to

output

ports

and

and

switches

and

replicate

to

output

ports

and

The

symmetry

is

required

because

all

replicas

of

a

multiport

encoded

multidestination

worm

created

at

a

switch

must

possess

the

same

header

information

It

can

be

easily

observed

that

it

is

difficult

to

create

such

multiport

encoded

worms

for

irregular

networks

In

-LSB-

-RSB-

the

authors

show

how

precomputed

reachability

information

at

switches

can

be

used

to

generate

asymmetric

multidestination

worms

with

bit-string

encoded

headers

The

authors

have

also

extended

this

bit-string

encoded

mechanism

to

irregular

switchbased

networks

-LSB-

-RSB-

However

computation

of

reachability

information

at

each

switch

for

an

arbitrary

switch-based

irregular

network

may

be

quite

a

non-trivial

task

Therefore

the

question

arises

whether

path-based

multidestination

mechanism

can

be

adapted

to

switch-based

irregular

networks

using

the

replication

capability

at

the

switches

The

challenges

to

doing

this

are

in

deriving

solutions

to

the

following

problems

a

-RRB-

the

concept

of

path-based

worms

on

a

switch-based

network

b

-RRB-

a

suitable

header

encoding

scheme

c

-RRB-

defining

valid

paths

on

a

given

base

routing

scheme

d

-RRB-

prevention

of

deadlock

due

to

coexistence

of

such

worms

with

unicast

worms

and

e

-RRB-

developing

multicast

algorithms

In

the

next

sections

we

look

at

these

problems

and

propose

solutions

Multi-drop

Path-based

Multidestination

Worms

In

this

section

we

describe

the

concept

of

multi-drop

path-based

multidestination

worms

and

a

new

header

encoding

scheme

for

such

worms

source

destination

Figure

Example

of

tree-based

multidestination

worm

for

MINs

using

the

multiport

encoding

mechanism

-LSB-

-RSB-

The

Multi-drop

Mechanism

In

this

mechanism

the

capability

at

switches

to

replicate

incoming

worms

and

to

forward

the

copies

to

several

output

ports

simultaneously

is

used

but

the

replication

is

restricted

The

multi-drop

mechanism

is

defined

as

follows

Definition

A

multidestination

worm

is

defined

as

a

multi-drop

worm

if

a

-RRB-

it

can

cover

multiple

destinations

on

a

traversed

switch

and

b

-RRB-

for

every

switch

in

which

it

replicates

to

multiple

outgoing

ports

no

more

than

one

of

these

ports

is

connected

to

another

switch

and

the

remaining

are

connected

to

processors

Such

a

mechanism

allows

easy

construction

of

paths

-LRB-

of

switches

-RRB-

in

an

irregular

network

It

simplifies

handling

of

deadlock

problems

because

a

worm

does

not

get

replicated

into

several

worms

branched

over

multiple

paths

It

also

allows

simplified

header

encoding

by

using

a

variation

of

the

all-destination

encoding

-LSB-

-RSB-

where

destination

ids

are

replaced

by

switch

ids

and

ids

of

destinations

connected

to

a

switch

Figure

-LRB-

a

-RRB-

shows

an

example

of

a

multidrop

path-based

multidestination

worm

If

we

abstract

a

switch-based

irregular

network

to

the

interconnection

graph

G

then

multi-drop

worms

can

be

abstracted

to

strict

path-based

multidestination

worms

on

G

The

path

of

a

multi-drop

worm

is

defined

by

the

set

of

interswitch

links

taken

by

the

worm

From

this

point

on

we

refer

to

a

path

as

a

sequence

of

inter-switch

links

Although

the

number

of

destinations

covered

in

a

single

step

by

such

a

multi-drop

multidestination

worm

can

be

less

than

that

covered

by

a

worm

using

the

treebased

mechanism

this

scheme

is

much

simpler

It

also

requires

minimal

modification

to

the

switch

and

results

in

low

complexity

for

creation

of

multidestination

worms

source

destination

port

numbering

-LRB-

a

-RRB-

addr

-LRB-

s

-RRB-

data

flits

-LRB-

b

-RRB-

addr

-LRB-

s

-RRB-

addr

-LRB-

s

-RRB-

tag

Figure

The

Multi-drop

mechanism

-LRB-

a

-RRB-

example

of

a

multidestination

worm

and

-LRB-

b

-RRB-

the

header

encoding

The

Multi-drop

Header

Encoding

Scheme

This

new

scheme

is

a

hybrid

of

the

all-destination

-LSB-

-RSB-

and

multiport

-LSB-

-RSB-

encoding

schemes

It

makes

use

of

the

fact

that

switches

too

have

unique

addresses

in

a

typical

switch-based

irregular

network

like

Autonet

-LSB-

-RSB-

The

current

implementation

of

the

Autonet

switch

can

route

messages

addressed

to

a

particular

switch

The

multi-drop

worm

header

encoding

for

each

set

of

destinations

on

a

switch

contains

the

switch

address

followed

by

the

multiport

encoding

of

destinations

for

that

switch

A

tag

bit

can

distinguish

whether

it

is

a

unicast

or

multidestination

worm

Figure

-LRB-

b

-RRB-

shows

the

multi-drop

scheme

encoded

header

of

the

worm

shown

in

Fig.

-LRB-

a

-RRB-

Since

switch

is

the

first

switch

with

destinations

on

it

the

header

encoding

begins

with

the

address

of

The

following

bits

encode

the

information

about

which

output

ports

the

worm

replicates

to

The

bits

correspond

to

output

ports

to

from

right

to

left

In

the

first

switch

the

worm

is

intended

to

be

replicated

to

two

destinations

and

one

outgoing

link

to

switch

Similarily

information

about

switches

and

is

encoded

in

the

header

In

this

scheme

the

source

does

not

completely

specify

the

route

taken

by

the

worm

intermediate

switches

with

no

destinations

are

not

included

Since

the

routing

tables

at

the

switches

are

used

to

route

a

worm

any

adaptivity

provided

by

the

underlying

routing

scheme

is

fully

utilized

between

any

two

consecutive

participating

switches

The

length

of

the

header

of

a

worm

only

increases

linearly

with

the

number

of

participating

switches

in

the

worm

and

not

the

number

of

hops

traversed

by

the

worm

or

the

number

of

destinations

Also

this

new

scheme

requires

only

a

minor

change

in

the

routing

logic

of

a

switch

which

already

provides

the

replication

facility

like

the

Autonet

switch

Now

the

challenge

is

in

defining

valid

paths

on

a

given

base

routing

scheme

and

prevention

of

deadlock

due

to

coexistence

of

such

worms

with

unicast

worms

In

the

next

section

we

solve

this

problem

in

the

context

of

Autonet

routing

Creation

of

Multi-drop

Worms

on

Irregular

Networks

A

multi-drop

path-based

multidestination

worm

needs

to

get

routed

in

a

piece-wise

manner

from

the

source

switch

to

the

final

switch

This

raises

the

following

fundamental

problems

at

any

intermediate

switch

a

-RRB-

it

should

be

feasible

for

the

multidestination

worm

to

find

a

path

leading

to

the

next

switch

in

its

list

and

b

-RRB-

the

piece-wise

routing

of

a

multidestination

worm

should

not

violate

the

routing

scheme

to

prevent

deadlock

In

this

section

we

first

propose

a

method

to

identify

valid

multi-drop

multidestination

paths

which

conform

to

the

base

Autonet

routing

on

an

arbitrary

irregular

graph

Then

we

discuss

the

requirements

for

deadlock-freedom

Valid

Multidestination

Paths

with

the

Autonet

Routing

Let

us

define

the

list

of

participating

switches

Lsw

-LRB-

w

-RRB-

of

a

multi-drop

multidestination

worm

w

as

follows

Lsw

-LRB-

w

-RRB-

consists

of

switches

directly

connected

to

the

destinations

of

w

in

the

order

in

which

the

corresponding

destinations

will

be

covered

Let

us

denote

the

jth

processor

on

the

ith

switch

as

pi

j

If

w

starts

from

source

processor

ps

and

covers

the

processors

pn

m

in

that

order

then

Lsw

-LRB-

w

-RRB-

is

given

by

the

switches

ss

sn

Therefore

generating

a

valid

multi-drop

multidestination

path

for

a

worm

w

on

an

arbitrary

switch-based

irregular

network

N

is

equivalent

to

generating

a

valid

path

which

covers

the

switches

in

Lsw

-LRB-

w

-RRB-

in

that

order

on

the

interconnection

graph

G

of

that

network

Definition

Let

Psd

and

Pmsd

be

defined

as

the

sets

of

all

paths

allowed

between

a

source

switch

s

to

a

destination

switch

d

under

the

up

down

routing

and

up

down

minimal

routing

respectively

Definition

A

partial

ordered

chain

-LRB-

POC

-RRB-

is

any

ordered

list

of

switches

sn

where

si

is

connected

to

si

by

a

down

tree

link

-LRB-

from

the

BFS

spanning

tree

-RRB-

or

a

down

cross

link

connecting

switches

at

different

levels

of

the

BFS

spanning

tree

In

our

recent

work

-LSB-

-RSB-

we

defined

the

concept

of

a

POC

with

the

specific

purpose

of

constructing

an

ordering

among

switches

in

an

irregular

network

to

reduce

link

contention

between

two

messages

of

the

same

multicast

using

unicast

message

passing

In

this

paper

we

use

this

concept

to

construct

valid

multi-drop

multidestination

paths

Definition

The

path

set

of

a

multi-drop

multidestination

worm

w

denoted

as

PSm

-LRB-

w

-RRB-

is

defined

as

the

set

of

all

paths

that

can

be

taken

by

w

when

routed

in

a

piece-wise

manner

using

the

routing

tables

which

support

up

down

minimal

routing

Lemma

The

path

set

of

a

multi-drop

multidestination

worm

w

PSm

-LRB-

w

-RRB-

is

given

by

Pmsn

where

Lsw

-LRB-

w

-RRB-

ss

sn

Proof

The

worm

w

starts

from

the

switch

ss

and

covers

sn

in

that

order

Since

w

is

routed

in

a

piece-wise

manner

using

up

down

minimal

routing

it

first

takes

one

of

the

paths

from

Then

it

takes

one

of

the

paths

from

and

so

on

Thus

PSm

-LRB-

w

-RRB-

Pmsn

Theorem

PSm

-LRB-

w

-RRB-

Psssn

if

Lsw

-LRB-

w

-RRB-

ss

sn

and

if

the

switches

sn

lie

on

some

POC

P

in

that

particular

order

Proof

It

can

be

easily

observed

that

all

paths

in

are

up

down

paths

If

it

can

be

shown

that

-LRB-

i

n

-RRB-

all

paths

in

Pmsisi

are

down

paths

then

using

the

result

from

Lemma

it

can

be

seen

that

all

paths

in

PSm

-LRB-

w

-RRB-

are

up

down

paths

Let

P

be

given

by

the

ordered

list

of

switches

swm

Then

-LRB-

i

n

-RRB-

si

P

Let

us

use

the

symbol

poc

to

denote

the

order

in

the

above

list

P

Therefore

-LRB-

i

m

-RRB-

swi

poc

swi

and

-LRB-

i

n

-RRB-

si

poc

si

Let

E

en

denote

the

list

of

down

links

such

that

switch

swi

is

connected

to

the

switch

swi

by

the

down

link

ei

where

swi

and

swi

P

Let

us

consider

two

arbitrary

switches

sa

and

sa

such

that

a

n

sa

swi

and

sa

swj

Therefore

si

poc

sj

It

can

be

easily

observed

that

the

path

described

by

list

of

links

pi

j

ei

ei

ej

is

one

of

the

valid

minimal

paths

between

si

and

sj

Also

each

link

in

this

path

is

a

down

link

and

the

path

is

j

i

links

long

Any

path

from

si

to

sj

that

contains

an

up

link

is

definitely

longer

than

pi

j

because

an

up

link

takes

the

worm

higher

up

the

BFS

routing

tree

whereas

sj

is

exactly

j

i

levels

lower

than

si

in

the

BFS

tree

Since

a

routed

worm

from

sa

to

sa

takes

only

down

links

all

paths

in

Pmsasa

are

down

paths

Therefore

piece-wise

routing

from

to

sn

results

in

a

down

path

only

Therefore

all

paths

in

PSm

-LRB-

w

-RRB-

are

up

down

paths

Theorem

A

multi-drop

path-based

multidestination

worm

w

where

Lsw

-LRB-

w

-RRB-

ss

sn

and

the

switches

sn

lie

on

some

POC

P

in

that

particular

order

always

follows

a

valid

up

down

path

and

introduces

no

deadlocks

Proof

This

can

be

easily

derived

from

Theorem

and

the

fact

that

non-minimal

up

down

routing

is

deadlock-free

and

non-minimal

and

minimal

up

down

paths

can

coexist

without

deadlock

It

must

be

noted

that

the

concept

of

valid

multi-drop

multidestination

worms

using

POCs

on

Autonet

routing

can

also

be

extended

to

the

case

when

the

routing

tables

support

nonminimal

paths

In

this

case

multi-drop

multidestination

worms

are

not

routed

using

the

routing

tables

at

intermediate

switches

instead

the

source

specifies

the

exact

route

-LRB-

all

switches

in

a

path

-RRB-

in

the

header

encoding

of

the

worm

Such

a

header

encoding

for

the

sample

multidrop

worm

in

Fig

-LRB-

a

-RRB-

is

shown

in

Fig

-LRB-

a

-RRB-

-LRB-

it

includes

the

address

of

with

a

single

port

replication

-RRB-

It

can

be

easily

observed

that

the

switch

addresses

in

this

encoding

is

redundant

information

and

the

header

encoding

can

be

further

simplified

by

dropping

the

switch

addresses

completely

and

retaining

only

the

multiport

encoding

as

shown

in

Fig.

-LRB-

b

-RRB-

addr

-LRB-

s

-RRB-

data

flits

-LRB-

a

-RRB-

addr

-LRB-

s

-RRB-

addr

-LRB-

s

-RRB-

-LRB-

s

-RRB-

data

flits

-LRB-

b

-RRB-

Figure

Header

encoding

of

the

multi-drop

of

Fig.

-LRB-

a

-RRB-

for

routing

tables

supporting

nonminimal

routing

-LRB-

a

-RRB-

each

switch

in

the

route

is

specified

with

its

multiport

encoding

and

-LRB-

b

-RRB-

an

enhancement

by

eliminating

the

switch

addresses

Deadlock-freedom

It

is

shown

in

-LSB-

-RSB-

that

asynchronous

replication

with

buffers

at

least

as

large

as

the

largest

multidestination

worm

ensures

deadlock

freedom

on

multistage

interconnection

networks

for

tree-based

multidestination

worms

The

Autonet

switch

provides

similar

guarantee

with

input

buffer

size

of

bytes

-LSB-

-RSB-

-LRB-

This

feature

is

provided

to

prevent

deadlock

that

arises

in

the

case

of

broadcast

worms

which

use

the

replication

mechanism

at

the

switches

to

flood

the

network

-RRB-

Since

multi-drop

worms

form

a

subset

of

all

tree-based

worms

addition

of

multi-drop

worms

can

not

cause

deadlock

in

an

Autonet

switch-based

network

provided

the

message

length

is

less

than

the

given

buffer

size

Interestingly

the

buffer

requirement

for

deadlock-freedom

with

multi-drop

worms

can

be

actually

lowered

depending

on

the

scheduling

policy

used

at

the

switch

There

are

two

basic

kinds

of

scheduling

policies

at

a

switch

centralized

or

distributed

When

a

worm

arrives

at

the

switch

the

header

flits

are

used

to

index

into

the

routing

table

and

the

appropriate

output

port

-LRB-

s

-RRB-

are

read

from

the

table

to

form

the

request

These

requests

can

be

handled

at

each

input

port

independently

-LRB-

distributed

-RRB-

or

all

the

requests

can

be

handled

centrally

-LRB-

centralized

-RRB-

The

buffer

requirement

for

deadlock-freedom

should

be

larger

than

the

maximum

worm

length

in

the

case

of

the

distributed

scheduling

policy

but

much

lesser

in

the

case

of

the

centralized

scheduling

policy

The

Autonet

switch

uses

a

central

scheduling

policy

to

schedule

output

ports

among

the

various

incoming

worms

both

unicast

and

broadcast

The

central

scheduling

engine

maintains

a

queue

of

requests

for

ports

If

a

broadcast

request

is

for

multiple

output

ports

then

the

engine

only

grants

the

request

after

it

accumulates

all

the

requested

output

ports

in

the

free

state

Therefore

the

replication

of

flits

to

output

ports

only

begins

after

all

the

required

output

ports

have

been

captured

by

the

worm

If

the

same

centralized

scheduling

policy

is

used

with

the

multi-drop

mechanism

the

buffer

size

requirement

for

deadlock-freedom

can

be

relaxed

as

shown

below

Theorem

Multi-drop

path

based

multidestination

worms

using

the

valid

paths

as

defined

in

Section

with

the

Autonet

central

scheduling

is

deadlock-free

with

even

single

flit

buffers

at

each

input

port

of

a

switch

Proof

A

multidestination

worm

can

not

capture

some

ports

of

a

switch

while

waiting

on

some

other

ports

This

is

because

the

centralized

scheduler

only

grants

all

requested

ports

to

a

worm

after

they

have

been

captured

Once

a

multi-drop

worm

reaches

its

first

destination

switch

it

can

only

proceed

on

down

links

This

is

because

after

reaching

any

switch

which

is

adjacent

to

a

destination

the

worm

is

forced

to

take

down

links

down

a

POC

as

described

in

Theorem

Therefore

no

multi-drop

worm

can

wait

on

an

up

link

after

it

has

reached

even

one

destination

This

means

there

can

exist

no

cycle

of

multi-drop

worms

which

have

reached

at

least

one

destination

each

since

there

exists

no

cycle

of

up

links

This

also

holds

for

a

combination

of

unicast

and

multi-drop

worms

since

there

can

exist

no

worm

holding

on

to

a

down

link

and

waiting

on

an

up

link

Therefore

the

multi-drop

mechanism

can

be

implemented

without

deadlock

with

single

flit

size

buffers

at

input

ports

on

any

switch

which

uses

the

central

scheduling

policy

Multicast

Algorithms

In

this

section

we

propose

three

multicast

algorithms

using

the

multi-drop

path-based

mechanism

All

these

algorithms

use

the

POC

concept

to

construct

as

many

longest

POCs

as

possible

from

the

participating

switches

and

group

these

POCs

in

some

hierarchy

to

implement

multicast

Let

us

first

look

at

how

the

POCs

are

constructed

Construction

of

POCs

In

-LSB-

-RSB-

we

described

a

method

to

construct

as

many

longest

POCs

as

possible

from

the

participating

switches

of

a

multicast

Figure

in

the

Appendix

formally

describes

the

method

We

explain

the

method

using

an

example

multicast

The

interested

reader

can

refer

to

-LSB-

-RSB-

for

details

Figure

-LRB-

a

-RRB-

shows

the

BFS

graph

of

a

sample

irregular

network

Let

us

assume

that

there

are

two

processors

connected

to

each

switch

of

the

irregular

graph

and

let

the

processors

on

the

switch

numbered

i

be

labelled

as

pi

and

pi

Let

us

consider

a

multicast

with

the

source

as

and

two

destinations

on

each

switch

except

switches

and

In

the

first

step

a

depth-first

search

-LRB-

DFS

-RRB-

is

applied

on

the

BFS

spanning

tree

starting

with

the

root

node

r

and

considering

only

the

down

links

specified

in

Definition

This

is

to

facilitate

the

construction

of

the

longest

POCs

Figure

-LRB-

b

-RRB-

shows

the

DFS

tree

T

on

the

sample

irregular

network

A

participating

switch

is

one

with

at

least

one

participating

processor

connected

to

it

The

resultant

DAG

T

from

the

DFS

can

be

reduced

to

a

DAG

T

which

contains

only

the

participating

switches

Fig.

-LRB-

c

-RRB-

shows

the

reduced

DAG

T

Each

switch

is

given

a

weight

equal

to

the

number

of

participating

processors

connected

to

it

and

to

all

its

descendent

switches

Subsequently

chains

of

switches

are

stripped

off

from

T

according

to

their

weights

In

other

words

the

heaviest

chain

gets

stripped

first

from

T

and

the

lightest

the

last

Figure

-LRB-

d

-RRB-

shows

the

obtained

set

of

chains

C

In

all

these

figures

the

participating

switches

are

highlighted

root

-LRB-

a

-RRB-

root

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

Illustrating

steps

for

construction

of

POCs

a

-RRB-

sample

BFS

graph

of

an

irregular

network

b

-RRB-

DFS

tree

T

c

-RRB-

DAG

T

and

the

associated

switch

weights

and

-LRB-

d

-RRB-

the

set

C

of

chains

The

Multi-drop

Path-based

Greedy

-LRB-

MDP-G

-RRB-

Algorithm

This

algorithm

uses

a

greedy

approach

on

the

set

of

POCs

generated

by

the

above

steps

to

implement

multicast

First

the

source

node

covers

the

destination

processors

on

the

switches

of

the

heaviest

chain

with

a

single

multi-drop

worm

Then

in

each

successive

step

the

destinations

that

have

been

covered

in

previous

steps

cover

destinations

on

chains

that

have

not

been

covered

yet

-LRB-

in

C

-RRB-

using

multi-drop

worms

Figure

-LRB-

a

-RRB-

shows

these

steps

for

the

example

multicast

covers

all

the

destinations

on

switches

and

with

a

single

multi-drop

worm

in

the

first

step

In

the

second

step

covers

all

destinations

on

switches

and

using

a

multidestination

worm

covers

all

destinations

on

switches

and

and

covers

all

destinations

on

switch

Therefore

the

multicast

completes

in

two

steps

-LRB-

a

-RRB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

b

-RRB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

c

-RRB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Figure

Illustrating

the

multicast

algorithms

for

the

sample

multicast

of

Fig.

a

-RRB-

the

MDP-G

algorithm

b

-RRB-

the

MDP-LG

algorithm

and

c

-RRB-

the

MDP-B

algorithm

The

numbers

in

brackets

denote

step

numbers

Since

at

each

step

maximum

possible

destinations

are

being

covered

using

multi-drop

worms

constructed

with

longest

possible

POCs

this

algorithm

takes

minimum

number

of

steps

to

execute

However

it

has

the

drawback

that

the

number

of

multi-drop

worms

starting

from

a

single

switch

might

be

large

leading

to

link

contention

The

Multi-drop

Path-based

Less

Greedy

-LRB-

MDP-LG

-RRB-

Algorithm

We

propose

the

MDP-LG

algorithm

to

address

the

above

drawback

This

algorithm

also

uses

the

output

from

the

POC

construction

method

like

the

MDP-G

algorithm

Following

the

first

step

only

one

destination

from

each

switch

that

has

been

covered

chooses

chains

that

have

not

been

covered

yet

-LRB-

in

C

-RRB-

and

covers

them

using

multidestination

worms

Since

all

destinations

covered

in

the

particular

step

do

not

send

out

worms

this

scheme

is

given

the

name

Less

Greedy

As

shown

in

Fig

-LRB-

b

-RRB-

covers

all

the

destination

on

switches

and

with

a

multi-drop

worm

in

the

first

step

In

the

second

step

covers

all

destinations

on

switches

and

covers

all

destinations

on

switches

and

and

covers

all

destinations

on

switch

This

algorithm

may

take

more

steps

to

complete

but

it

reduces

link

contention

by

making

sure

that

only

one

multi-drop

worm

starts

from

a

switch

during

one

step

The

Multi-drop

Path-based

Binomial

-LRB-

MDP-B

-RRB-

Algorithm

We

propose

the

MDP-B

algorithm

to

further

reduce

link

contention

in

the

MDP-LG

algorithm

Again

the

MDP-B

algorithm

starts

the

same

way

as

the

MDP-G

algorithm

with

the

construction

of

the

POCs

Following

the

first

step

only

the

source

processor

and

the

leader

processor

of

each

chain

are

involved

in

sending

worms

The

leader

processor

is

chosen

as

a

destination

processor

on

the

first

switch

of

the

chain

The

source

covers

all

destinations

on

one

chain

in

the

first

step

using

a

multi-drop

worm

In

the

second

step

the

source

and

the

leader

of

the

covered

chain

cover

all

destinations

on

two

other

chains

using

multi-drop

worms

and

so

on

As

shown

in

Fig.

-LRB-

c

-RRB-

covers

all

the

destination

on

switches

and

with

a

multi-drop

worm

in

the

first

step

In

the

second

step

covers

all

destinations

on

switches

and

with

a

single

worm

and

covers

all

destinations

on

switches

and

using

a

single

worm

In

the

third

step

covers

all

destinations

on

switch

Therefore

this

algorithm

requires

steps

But

the

link

contention

as

compared

to

the

MDP-G

and

MDP-LG

algorithms

is

reduced

since

there

are

no

two

destinations

on

the

same

chain

trying

to

send

out

worms

at

the

same

time

Simulation

Experiments

and

Results

We

performed

simulation

experiments

to

evaluate

and

compare

performance

of

the

MDP-G

MDP-LG

MDP-B

and

CCO

-LSB-

-RSB-

algorithms

for

single

multicast

A

flit-level

wormhole-routed

simulator

WORMULSim

was

used

to

model

the

irregular

network

For

all

simulation

experiments

we

assumed

system

and

technological

parameters

representing

the

current

trend

in

technology

The

following

default

parameters

were

used

ts

-LRB-

communication

start-up

time

-RRB-

microseconds

tphy

-LRB-

link

propagation

time

-RRB-

nanoseconds

troute

-LRB-

routing

delay

at

switch

-RRB-

ns

tsw

-LRB-

switching

time

across

the

router

crossbar

for

a

flit

-RRB-

ns

tsched

-LRB-

time

for

scheduler

to

process

a

request

-RRB-

ns

tinj

-LRB-

time

to

inject

a

flit

into

network

-RRB-

ns

and

tcons

-LRB-

time

to

consume

a

flit

from

network

-RRB-

ns

For

all

experiments

-LRB-

unless

otherwise

stated

-RRB-

we

assumed

the

following

default

system

configuration

a

processor

system

interconnected

by

eight-port

switches

and

network

having

connectivity

as

defined

in

-LSB-

-RSB-

The

default

message

length

was

assumed

to

be

flits

We

performed

several

experiments

to

study

the

impact

of

system

size

message

length

communication

start-up

time

switch

size

and

degree

of

connectivity

on

single

multicast

latency

for

varying

sizes

of

destination

sets

The

destinations

and

network

topologies

were

generated

randomly

For

each

data

point

the

multicast

latency

was

averaged

over

different

sets

of

destinations

for

each

of

different

network

configurations

The

confidence

intervals

generated

for

the

data

points

were

observed

to

be

extremely

narrow

Effect

of

system

size

We

simulated

the

four

multicast

algorithms

on

four

different

system

configurations

with

and

processors

respectively

Figure

shows

these

results

It

can

be

observed

that

the

MDP-LG

algorithm

performs

the

best

for

almost

all

system

sizes

and

destinations

As

system

size

increases

the

MDP-LG

algorithm

shows

steady

benefits

compared

to

the

CCO

algorithm

For

example

on

a

processor

system

with

destinations

the

reduction

in

multicast

latency

achieved

by

the

MPD-LG

algorithm

is

around

and

compared

to

the

CCO

MPD-G

and

MDP-B

algorithms

respectively

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

60 80 100 120

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

100 200 300 400

Multicast

Latency

in

microseconds

Number

of

destinations

System

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

Figure

Multicast

latency

vs.

number

of

destinations

for

four

different

system

configurations

and

processors

As

expected

the

multi-drop

algorithms

outperform

the

CCO

algorithm

Since

the

number

of

destinations

covered

in

one

step

is

increased

by

using

multidestination

worms

the

total

number

of

steps

for

the

completion

of

the

multicast

algorithm

is

reduced

As

discussed

in

Section

the

MDP-G

suffers

from

link

contention

at

higher

destination

set

sizes

since

multiple

destinations

on

the

same

switch

send

worms

simultaneously

and

this

leads

to

severe

link

contention

The

MDP-B

algorithm

incurs

less

link

contention

but

undergoes

more

number

of

steps

and

hence

it

is

outperformed

by

the

MDP-LG

which

strikes

a

balance

between

number

of

steps

and

link

contention

Effect

of

message

length

We

studied

the

impact

of

message

length

on

the

four

algorithms

Four

different

message

lengths

and

flits

on

a

processor

system

with

default

parameters

were

considered

Figure

shows

the

respective

results

It

can

be

easily

observed

that

as

message

length

increases

the

link

contention

effect

dominates

the

results

and

the

MDP-B

algorithm

outperforms

the

remaining

algorithms

although

it

requires

more

steps

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Message

Length

uni-CCO

MDP-G

MDP-LG

MDP-B

Figure

Multicast

latency

vs.

number

of

destinations

for

four

different

message

lengths

and

flits

Effect

of

communication

start-up

time

We

studied

the

effect

of

communication

start-up

time

on

the

performance

of

the

four

algorithms

The

default

processor

system

configuration

was

used

with

three

different

communication

start-up

times

and

microseconds

Figure

shows

the

respective

multicast

latencies

It

can

be

observed

that

with

higher

communication

start-up

time

the

MDP-G

and

MDP-LG

algorithms

outperform

the

other

two

algorithms

This

is

expected

because

for

a

given

message

length

higher

start-up

times

reduce

the

effect

of

link

contention

and

the

MDP-G

and

MDP-LG

algorithms

take

the

minimum

steps

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Startup

Time

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Startup

Time

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Startup

Time

uni-CCO

MDP-G

MDP-LG

MDP-B

Figure

Multicast

latency

vs.

number

of

destinations

for

three

different

communication

start-up

times

and

microseconds

Effect

of

switch

size

We

studied

the

effect

of

switch

size

on

the

overall

performance

of

the

four

algorithms

The

default

processor

system

was

considered

with

three

different

switch

sizes

and

ports

Figure

shows

the

performance

results

It

can

be

observed

that

with

increasing

switch

size

the

MDP-G

algorithm

performs

the

best

As

switch

size

increases

more

number

of

communication

steps

become

intra-switch

steps

Since

intra-switch

steps

are

contention-free

it

reduces

the

effect

of

link

contention

on

the

overall

multicast

latency

Therefore

the

algorithm

which

completes

in

the

minimum

number

of

steps

the

MDP-G

algorithm

outperforms

the

other

algorithms

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Switch

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Switch

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Switch

Size

uni-CCO

MDP-G

MDP-LG

MDP-B

Figure

Multicast

latency

vs.

number

of

destinations

for

three

different

switch

sizes

and

ports

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Connectivity

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Connectivity

uni-CCO

MDP-G

MDP-LG

MDP-B

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

destinations

Connectivity

uni-CCO

MDP-G

MDP-LG

MDP-B

Figure

Multicast

latency

vs.

number

of

destinations

for

different

degrees

of

network

connectivity

and

Effect

of

degree

of

network

connectivity

Finally

we

studied

the

impact

of

degree

of

network

connectivity

on

multicast

latency

The

default

system

size

of

processors

was

considered

with

three

different

degrees

of

network

connectivity

and

Figure

shows

the

associated

performance

results

With

lesser

connectivity

the

number

of

communication

links

reduces

in

an

irregular

network

leading

to

less

number

of

adaptive

paths

and

more

contention

between

messages

Under

such

circumstances

the

MDP-B

and

MDP-LG

algorithms

deliver

the

best

performance

As

the

degree

of

connectivity

increases

the

contention

effect

reduces

but

does

not

get

completely

eliminated

Thus

with

higher

connectivity

the

MPD-G

and

MDP-LG

algorithms

outperform

the

rest

Conclusions

In

this

paper

we

have

proposed

a

new

multi-drop

path-based

multidestination

message

passing

mechanism

on

switch-based

irregular

networks

A

method

to

identify

valid

multi-drop

paths

with

respect

to

a

typical

deadlock-free

routing

on

arbitrary

irregular

networks

has

been

proposed

Using

this

new

mechanism

three

multicast

algorithms

MDP-G

MDP-LG

and

MDP-B

have

been

developed

These

algorithms

have

been

simulated

and

compared

with

each

other

and

the

best

unicast-based

multicast

algorithm

the

CCO

algorithm

Compared

to

the

CCO

algorithm

the

MDP-LG

algorithm

shows

an

improvement

in

multicast

latency

up

to

a

factor

of

These

results

demonstrate

significant

potential

for

the

multi-drop

multidestination

mechanism

to

be

applied

to

current

and

future

switch-based

irregular

networks

Since

our

simulator

currently

models

only

pure

wormhole

routing

-LRB-

single

flit

buffer

size

at

input

ports

of

each

switch

-RRB-

we

believe

that

the

observed

benefits

of

the

multi-drop

multidestination

mechanism

were

diminished

due

to

the

effect

of

link

contention

We

expect

this

effect

to

reduce

with

any

realistic

buffer

size

at

input

ports

of

switches

and

increased

benefits

of

the

multi-drop

mechanism

We

are

working

on

extending

the

simulator

to

model

larger

buffer

sizes

at

the

switches

and

results

will

be

reported

during

the

workshop

We

are

also

look

ing

at

applying

this

new

mechanism

at

other

collective

communication

operations

like

barrier

synchronization

gather

multiple

multicast

etc.

Additional

Information

A

number

of

related

papers

can

be

obtained

from

Alternatively

the

papers

can

be

obtained

by

anonymous

ftp

from

ftp.cis.ohio-state

edu

cd

pub/communication

Acknowledgements

The

authors

would

like

to

thank

other

members

of

the

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

in

the

department

for

providing

comments

criticisms

and

suggestions

to

this

work

References

-LSB-

-RSB-

N.

J.

Boden

D.

Cohen

and

et

al.

Myrinet

A

Gigabit-per-Second

Local

Area

Network

IEEE

Micro

pages

-LCB-

Feb

-LSB-

-RSB-

R.

V.

Boppana

S.

Chalasani

and

C.

S.

Raghavendra

On

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

C.

M.

Chiang

and

L.

M.

Ni

Deadlock-Free

Multi-Head

Wormhole

Routing

In

Proceedings

of

the

First

High

Performance

Computing-Asia

-LSB-

-RSB-

R.

Horst

ServerNet

Deadlock

Avoidance

and

Fractahedral

Topologies

In

Proceedings

of

the

International

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

R.

Kesavan

K.

Bondalapati

and

D.

K.

Panda

Multicast

on

Irregular

Switch-based

Networks

with

Wormhole

Routing

In

Proceedings

of

the

International

Symposium

on

High

Performance

Computer

Architecture

-LRB-

-RRB-

pages

-LCB-

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Dec

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

Lionel

Ni

Should

Scalable

Parallel

Computers

Support

Efficient

Hardware

Multicasting

In

ICPP

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

Issues

in

Designing

Efficient

and

Practical

Algorithms

for

Collective

Communication

in

Wormhole-Routed

Systems

In

ICPP

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

-LSB-

-RSB-

W.

Qiao

and

L.

M.

Ni

Adaptive

Routing

in

Irregular

Networks

Using

Cut-Through

Switches

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

M.

D.

Schroeder

et

al.

Autonet

A

High-speed

Self-configuring

Local

Area

Network

Using

Point-to-point

Links

Technical

Report

SRC

research

report

DEC

Apr

-LSB-

-RSB-

R.

Sivaram

D.

K.

Panda

and

C.

B.

Stunkel

Efficient

Broadcast

and

Multicast

on

Multistage

Interconnection

Networks

using

Multiport

Encoding

In

Proceedings

of

the

IEEE

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

Oct

Available

from

-LSB-

-RSB-

R.

Sivaram

D.

K.

Panda

and

C.

B.

Stunkel

Multicasting

in

Irregular

Networks

with

Cut-Through

Switches

using

Tree-based

Multidestination

Worms

Technical

report

The

Ohio

State

Univeristy

March

-LSB-

-RSB-

C.

B.

Stunkel

R.

Sivaram

and

D.

K.

Panda

Implementing

Multidestination

Worms

in

Switch-Based

Parallel

Systems

Architectural

Alternatives

and

their

Impact

In

Proceedings

of

the

ACM

Annual

International

Symposium

on

Computer

Architecture

-LRB-

-RRB-

June

Available

from

-LSB-

-RSB-

H.

Xu

P.

K.

McKinley

and

L.

Ni

Efficient

Implementation

of

Barrier

Synchronization

in

Wormhole-routed

Hypercube

Multicomputers

Journal

of

Parallel

and

Distributed

Computing

-LCB-

Appendix

Algorithm

for

POC

Construction

Inputs

G

irregular

graph

B

a

breadth

first

spanning

tree

on

the

switches

of

G

D

set

of

destinations

ns

the

source

c

-LRB-

n

s

-RRB-

node

n

is

connected

to

switch

s.

Output

Construct

longest

possible

POCs

from

participating

switches

in

G.

Procedure

Apply

the

DFS

on

B

from

the

root

node

r

of

B.

Only

down

tree

links

belonging

to

B

and

down

cross

links

which

connect

switches

in

different

levels

of

B

are

used

Let

the

resulting

DAG

be

T

S

fs

j

s

is

a

switch

in

G

and

d

D

-LSB-

fnsg

and

c

-LRB-

d

s

-RRB-

g.

S

is

the

set

of

participating

switches

Reduce

T

to

a

DAG

T

which

contains

all

switches

in

S

and

no

other

switches

Reduction

is

done

by

removing

all

non-participating

switches

from

T

Every

time

a

switch

s

and

all

links

incident

on

s

are

removed

directed

down

links

are

drawn

from

each

parent

of

s

to

each

child

of

s.

Calculate

the

weight

wt

of

each

switch

in

T

as

follows

wt

-LRB-

s

-RRB-

P

-LRB-

s

-RRB-

ae

-LRB-

si

-RRB-

where

Desc

-LRB-

s

-RRB-

is

the

set

containing

s

and

all

the

descendents

of

s

in

T

and

ae

-LRB-

s

-RRB-

is

the

number

of

participating

processors

connected

to

the

switch

s.

C

i

while

T

is

not

empty

-LRB-

a

-RRB-

Let

be

a

switch

with

maximum

weight

in

T

li

such

that

pj

is

the

switch

with

the

maximum

weight

among

all

children

switches

of

pj

-LRB-

b

-RRB-

T

T

fp

j

p

lig

-LRB-

c

-RRB-

where

fnj

nj

g

are

the

participating

processors

connected

to

switch

pj

in

li

-LRB-

d

-RRB-

C

concat

-LRB-

C

li

-RRB-

i

i

end

while

Figure

Algorithm

for

construction

of

POCs

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

broadcast
collective communication
multicast
networks of workstations
wormhole routing

cut-through routing
interconnection network
path-based routing