Multidestination

Message

Passing

in

Wormhole

k-ary

n-cube

Networks

with

Base

Routing

Conformed

Paths

Dhabaleswar

K.

Panda

Sanjay

Singal

and

Ram

Kesavan

Technical

Report

Multidestination

Message

Passing

in

Wormhole

k-ary

n-cube

Networks

with

Base

Routing

Conformed

Dhabaleswar

K.

Panda

Sanjay

Singal

and

Ram

Kesavan

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

E-mail

fpanda,singal,kesavang@cis.ohio-state.edu

Abstract

This

paper

proposes

multidestination

message

passing

on

wormhole

k-ary

n-cube

networks

using

a

new

base-routing-conformed-path

-LRB-

BRCP

-RRB-

model

This

model

allows

both

unicast

-LRB-

single-destination

-RRB-

and

multidestination

messages

to

co-exist

in

a

given

network

without

leading

to

deadlock

The

model

is

illustrated

with

several

common

routing

schemes

-LRB-

deterministic

as

well

as

adaptive

-RRB-

and

the

associated

deadlock-freedom

properties

are

analyzed

Using

this

model

a

set

of

new

algorithms

for

popular

collective

communication

operations

broadcast

and

multicast

are

proposed

and

evaluated

It

is

shown

that

the

proposed

algorithms

can

considerably

reduce

the

latency

of

these

operations

compared

to

the

Umesh

-LRB-

unicast-based

multicast

-RRB-

-LSB-

-RSB-

and

the

HamiltonianPath-based

-LSB-

-RSB-

schemes

A

very

interesting

result

that

is

presented

shows

that

a

multicast

can

be

implemented

with

reduced

or

near-constant

latency

as

the

number

of

processors

participating

in

the

multicast

increases

beyond

a

certain

number

It

is

also

shown

that

the

BRCP

model

can

take

advantage

of

adaptivity

in

routing

schemes

to

further

reduce

the

latency

of

these

operations

The

multidestination

mechanism

and

the

BRCP

model

establish

a

new

foundation

to

provide

fast

and

scalable

collective

communication

support

on

wormhole-routed

systems

Introduction

The

wormhole-routing

switching

technique

is

becoming

the

trend

in

building

future

parallel

systems

due

to

its

inherent

advantages

like

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

Intel

Paragon

Cray

Ncube

J-Machine

and

Stanford

DASH

are

representative

systems

falling

into

this

category

Such

systems

are

being

used

for

supporting

either

distributed-memory

shared-memory

or

distributed-shared

memory

programming

paradigms

In

order

to

support

these

paradigms

these

systems

need

fast

implementation

of

collective

communication

operations

-LRB-

broadcast

multicast

global

combine

and

barrier

synchronization

-RRB-

-LSB-

-RSB-

Traditionally

wormhole-routed

systems

have

supported

only

point-to-point

-LRB-

unicast

-RRB-

message

passing

mechanism

This

mechanism

allows

a

message

to

have

exactly

one

destination

which

leads

to

collective

communication

operations

being

implemented

as

multiple

phases

of

unicast

message

exchange

-LSB-

-RSB-

Each

of

these

phases

incurs

a

communication

start-up

cost

which

is

quite

high

on

research

is

supported

in

part

by

NSF

Grant

and

NSF

Career

Award

A

preliminary

version

of

this

paper

-LSB-

-RSB-

has

been

presented

in

Parallel

Computer

Routing

and

Communication

Workshop

Seattle

May

current

systems

-LRB-

microsec

-RRB-

Thus

these

operations

are

implemented

on

current

systems

with

considerable

latency

Since

collective

communication

operations

typically

fall

into

the

critical

paths

of

program

execution

high

latency

implementation

of

these

operations

limits

the

parallel

speedup

of

programs

Therefore

it

is

essential

to

implement

these

operations

with

reduced

latency

In

order

to

reduce

the

number

of

phases

to

implement

collective

communication

operations

Lin

and

Ni

proposed

a

Hamiltonian

Path-based

scheme

in

-LSB-

-RSB-

This

scheme

uses

a

modified

router

organization

to

support

propagation

of

multidestination

messages

-LRB-

a

message

with

multiple

destinations

-RRB-

It

allows

a

message

to

be

concurrently

copied

into

one

destination

node

while

it

is

being

forwarded

to

the

next

destination

The

message

propagation

is

restricted

to

a

Hamiltonian

Path

in

the

network

to

prevent

deadlock

Such

an

approach

demonstrates

that

collective

communication

operations

can

be

implemented

on

wormhole-routed

systems

with

reduced

latency

All

messages

get

routed

with

limited

adaptivity

because

they

conform

to

the

Hamiltonian

Path

Although

this

approach

tries

to

reduce

the

number

of

phases

the

path

length

dominates

the

communication

cost

on

larger

systems

which

leads

to

high

latency

collective

operations

Also

the

Hamiltonian

path-based

scheme

assumes

the

existence

of

a

hamiltonian

path

in

the

routing

scheme

supported

by

the

system

Current

commercial

and

experimental

systems

-LRB-

Intel

Paragon

Cray

Stanford

DASH

-RRB-

use

e-cube

routing

-LSB-

-RSB-

Recently

various

adaptive

routing

schemes

have

been

proposed

in

the

literature

-LSB-

-RSB-

It

is

conceivable

that

future

systems

will

adopt

these

adaptive

schemes

In

fact

the

recent

Cray

uses

such

a

scheme

Since

these

schemes

do

not

allow

the

existence

of

a

Hamiltonian

path

in

k-ary

n-cube

networks

it

is

infeasible

to

implement

the

Hamiltonian-path-based

scheme

on

such

systems

without

major

modification

to

the

router

architecture

Therefore

it

is

unlikely

that

the

Hamiltonian-path-based

scheme

will

be

used

in

future

systems

This

leads

to

the

following

challenge

Can

a

multidestination

communication

mechanism

be

proposed

for

k-ary

n-cube

wormhole

systems

to

allow

a

-RRB-

easy

implementation

on

any

supported

routing

scheme

b

-RRB-

implementation

of

collective

communication

operations

with

reduced

latency

compared

to

unicast-based

implementations

c

-RRB-

utilization

of

any

underlying

routing

adaptivity

for

further

reduction

in

latency

and

d

-RRB-

co-existence

of

unicast

messages

with

multidestination

messages

without

deadlocks

In

this

paper

we

take

on

such

a

challenge

and

propose

a

new

framework

of

Base-RoutingConformed-Path

-LRB-

BRCP

-RRB-

model

with

multidestination

messages

Depending

on

the

routing

scheme

used

in

a

network

this

model

allows

a

multidestination

message

to

propagate

through

valid

paths

conforming

to

the

base-routing

scheme

The

model

is

described

and

illustrated

for

a

set

of

representative

routing

schemes

e-cube

-LSB-

-RSB-

planar-adaptive

-LSB-

-RSB-

fully-adaptive

-LSB-

-RSB-

and

turn-model

-LSB-

-RSB-

Architectural

support

to

implement

multidestination

message-passing

on

current

generation

networks

and

the

associated

deadlock-freedom

properties

are

analyzed

It

is

also

shown

how

collective

communication

operations

can

be

implemented

with

reduced

latency

by

using

the

BRCP

model

In

particular

we

present

algorithms

for

two

commonly

used

collective

communication

operations

broadcast

and

multicast

These

algorithms

are

developed

and

evaluated

for

and

wormhole

networks

with

different

routing

schemes

It

is

shown

that

the

BRCP

model

reduces

the

latency

of

these

operations

considerably

compared

to

Umesh

-LSB-

-RSB-

and

Hamiltonian-path-based

-LSB-

-RSB-

schemes

As

the

number

of

participating

destinations

in

a

multicast

increases

beyond

certain

number

the

BRCP

model

allows

a

steady

reduction

in

the

multicast

latency

With

increase

in

adaptivity

in

the

underlying

routing

scheme

further

reduction

in

multicast

latency

is

demonstrated

In

this

paper

we

only

emphasize

broadcast

and

multicast

operations

Recently

we

have

also

shown

that

the

BRCP

model

is

equally

powerful

for

implementing

other

collective

communication

operations

like

barrier

synchronization

and

global

reduction

-LSB-

-RSB-

with

reduced

latency

Thus

the

BRCP

model

promises

significant

potential

to

implement

fast

and

efficient

collective

communication

operations

in

future

wormhole

systems

with

minimal

increase

in

cost

The

paper

is

organized

as

follows

Section

describes

the

basics

of

k-ary

n-cube

wormhole

networks

and

the

multidestination

mechanism

The

BRCP

model

together

with

its

intrinsic

benefits

and

deadlock-freedom

properties

are

presented

in

Section

Algorithms

for

broadcast

and

multicast

operations

are

developed

in

Sections

and

respectively

Simulation

experiments

and

results

are

presented

in

Section

Finally

concluding

remarks

are

made

in

Section

Multidestination

Message

Passing

in

Wormhole

k-ary

n-cubes

In

this

section

we

provide

an

overview

of

wormhole

k-ary

n-cube

networks

and

then

discuss

the

basic

concepts

behind

multidestination

message

passing

Wormhole

k-ary

n-cube

Networks

Multicomputers

with

direct

networks

supporting

wormhole

switching

have

become

a

popular

trend

in

building

large

parallel

systems

The

most

commonly

used

direct

network

topologies

are

meshes/tori

which

belong

to

the

class

of

generalized

k-ary

n-cube

interconnections

-LSB-

-RSB-

These

interconnections

are

n-dimensional

grids

with

kn

nodes

along

the

dimensions

n

respectively

Figure

shows

a

two

dimensional

torus

-LRB-

with

wrap-around

connection

-RRB-

system

Each

node

in

the

system

comprises

of

a

router

and

a

processor

The

routers

are

connected

in

a

configuration

Each

processor

is

connected

to

its

router

through

a

set

of

injection

and

consumption

channels

The

injection

channels

are

used

to

inject

messages

to

the

network

and

the

consumption

channels

are

used

to

consume

messages

from

the

network

The

injection

and

consumption

channels

are

also

called

internal

channels

-LSB-

-RSB-

processorrouter

consumption

channels

injection

channels

Figure

A

system

with

wrap-around

connection

and

a

typical

processor-router

interface

consisting

of

injection

and

consumption

channels

Multidestination

Message

Passing

Mechanism

The

concept

of

wormhole

message

passing

mechanism

with

multiple

destinations

was

first

introduced

by

Lin

and

Ni

in

-LSB-

-RSB-

A

single

destination

-LRB-

unicast

-RRB-

message

on

wormhole

networks

typically

consists

of

three

parts

a

header

flit

one

or

more

data

flit

-LRB-

s

-RRB-

and

a

a

tail

flit

A

multidestination

message

has

data

and

tail

flits

similar

to

those

of

a

unicast

message

However

its

header

carries

ids

of

a

set

of

destinations

Clearly

a

multidestination

message

with

a

single

destination

is

equivalent

to

a

unicast

message

There

are

different

ways

to

encode

the

ids

of

multiple

destinations

into

the

header

flit

-LRB-

s

-RRB-

of

a

multidestination

message

-LSB-

-RSB-

In

this

paper

we

assume

the

all-destination

encoding

where

a

set

of

d

destinations

are

encoded

as

a

set

of

d

flits

For

example

consider

the

multidestination

worm

shown

in

Fig.

Three

flits

are

required

to

encode

three

destinations

-LRB-

and

-RRB-

in

its

header

using

the

all-destination

encoding

scheme

-LSB-

-RSB-

The

worm

also

carries

five

data

flits

-LRB-

A-E

-RRB-

and

a

tail

flit

data

header

ABCD

CD

D

BAB

RouterRouterRouter

Node

Node

Node

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

a

-RRB-

tailx

absorb

forward

BNode

Router

BCBAB

RouterRouterRouter

Node

Node

Node

ANode

D

Router

E

Figure

Principles

of

multidestination

wormhole

mechanism

-LRB-

a

-RRB-

an

example

multidestination

multicast

worm

with

three

destinations

using

the

all-destination

encoding

-LRB-

b

-RRB-

snapshot

of

the

system

when

data

flit

A

of

the

worm

has

reached

the

router

of

node

and

-LRB-

c

-RRB-

snapshot

one

clock

cycle

later

Such

a

multidestination

worm

can

be

used

to

deliver

a

message

to

multiple

destinations

-LRB-

a

multicast

operation

-RRB-

with

a

single

communication

start-up

The

ordering

of

the

destinations

in

the

header

flits

of

a

multidestination

message

determines

the

path

traversed

by

the

worm

In

the

following

section

we

will

introduce

the

BRCP

model

and

show

how

destination

orderings

in

the

header

flits

can

be

constructed

to

avoid

deadlock

on

a

given

routing

scheme

For

the

time

being

for

the

purpose

of

illustration

let

us

assume

that

nodes

and

are

consecutive

nodes

of

a

linear

array

and

that

the

multidestination

worm

shown

in

Fig.

is

injected

by

a

source

node

located

to

the

left

of

node

On

injection

into

the

network

the

worm

is

routed

based

on

the

address

carried

by

the

leading

header

flit

corresponding

to

the

first

destination

-LRB-

node

-RRB-

As

soon

as

the

worm

reaches

the

router

of

node

the

first

header

flit

-LRB-

containing

the

address

of

node

-RRB-

is

removed

from

the

worm

by

the

router

Subsequently

the

worm

is

routed

to

the

second

destination

using

the

next

header

flit

of

the

message

-LRB-

node

-RRB-

Such

piece-wise

routing

continues

till

the

worm

reaches

the

last

destination

-LRB-

node

in

this

example

-RRB-

and

it

does

not

get

routed

any

further

At

each

intermediate

destination

router

every

data

flit

of

the

worm

is

forwarded

to

an

adjacent

router

as

well

as

absorbed

-LRB-

or

copied

-RRB-

to

the

system

buffer

by

the

router-processor

interface

As

in

conventional

unicast

message-passing

the

router

of

the

final

destination

of

a

multidestination

worm

only

consumes

the

flits

and

each

router

of

a

non-destination

node

along

the

path

traversed

by

the

worm

simply

forwards

all

flits

Figure

and

illustrate

the

mechanism

by

showing

snapshots

of

the

system

in

two

consecutive

cycles

Every

router

is

assumed

to

have

only

a

single

flit

buffer

Figure

indicates

the

system

state

when

data

flit

A

of

the

worm

has

reached

the

router

of

the

final

destination

node

At

this

time

flits

B

C

and

D

are

at

the

routers

of

nodes

and

respectively

In

the

next

cycle

routers

of

nodes

and

forward-and-absorb

flits

D

and

C

respectively

Also

the

router

of

node

forwards

flit

B

and

the

router

of

node

absorbs

flit

A

It

can

be

easily

observed

that

after

one

more

cycle

node

receives

the

complete

message

Nodes

and

receive

the

complete

message

in

one

and

three

additional

cycles

respectively

It

is

to

be

noted

that

such

a

multidestination

mechanism

is

quite

powerful

and

can

deliver

a

message

to

multiple

destinations

much

faster

than

multiple

unicast

messages

This

scheme

is

also

quite

general

because

unicast

messages

can

always

be

implemented

under

this

scheme

as

a

subset

operation

with

only

one

destination

In

order

to

support

multidestination

message-passing

the

router

interface

needs

to

have

logic

to

concurrently

absorb

as

well

as

forward

flits

at

intermediate

destination

routers

Current

wormhole

routers

contain

logic

to

absorb

-LRB-

at

final

destinations

-RRB-

and

forward

-LRB-

at

intermediate

destinations

-RRB-

flits

Only

a

small

amount

of

glue

logic

is

required

for

routers

to

provide

both

these

features

concurrently

To

support

the

all-destination

encoding

each

router

also

needs

a

small

amount

of

additional

logic

to

strip

away

a

header

flit

once

a

worm

reaches

a

destination

In

the

following

section

we

introduce

additional

consumption

channel

requirements

per

router

to

support

deadlock-free

multidestination

message-passing

Depending

on

the

base

routing

scheme

being

used

we

show

that

this

requirement

can

also

be

satisfied

with

minimum

additional

logic

Therefore

very

little

additional

logic

is

needed

to

enhance

current

routers

to

support

multidestination

message

passing

The

multidestination

message

passing

mechanism

is

quite

general

and

can

be

used

with

different

functionalities

to

implement

different

collective

communication

operations

-LSB-

-RSB-

The

example

in

Fig.

illustrated

a

multidestination

multicast

worm

to

implement

multicast/broadcast

Similarly

a

multidestination

gather

worm

can

be

used

for

efficient

barrier

synchronization

-LSB-

-RSB-

and

a

multidestination

exchange

worm

can

be

used

for

efficient

global

reduction

-LSB-

-RSB-

In

this

paper

we

focus

on

multidestination

multicast

worms

to

implement

multicast/broadcast

To

enable

implementation

of

such

operations

we

first

have

to

solve

the

following

problem

how

can

the

multidestination

mechanism

be

applied

to

k-ary

n-cube

networks

supporting

different

types

of

base

routing

schemes

In

the

following

section

we

develop

a

BRCP

model

to

solve

this

problem

Base

Routing

Conformed

Path

-LRB-

BRCP

-RRB-

Model

In

this

section

we

introduce

the

base-routing-conformed-path

-LRB-

BRCP

-RRB-

model

and

apply

it

to

common

wormhole

routing

schemes

First

we

classify

the

current

wormhole

routing

schemes

into

two

classes

with

and

without

cyclic

dependencies

Next

the

BRCP

model

is

developed

and

illustrated

for

both

these

classes

Finally

deadlock-freedom

properties

of

this

model

are

discussed

Two

classes

of

Routing

Schemes

Current

wormhole-routing

schemes

with

unicast

message

passing

can

be

broadly

classified

into

two

classes

-LSB-

-RSB-

The

routing

schemes

belonging

to

the

first

class

are

based

on

routing

functions

which

avoid

cyclic

dependencies

between

messages

Routing

schemes

belonging

to

this

class

are

the

popular

e-cube

routing

-LSB-

-RSB-

planar-adaptive

routing

-LSB-

-RSB-

turn-model

-LSB-

-RSB-

etc.

The

routing

schemes

belonging

to

the

second

class

allow

cyclic

dependencies

between

messages

but

achieve

deadlock-free

message

passing

by

providing

escape

channels

-LSB-

-RSB-

The

basic

idea

is

to

provide

a

routing

function

R

as

well

as

a

connected

routing

sub-function

of

R

The

routing

sub-function

is

chosen

in

such

a

way

that

there

are

no

cycles

in

its

extended

channel

dependency

graph

The

channels

under

work

as

escape

channels

to

provide

deadlock-freedom

in

the

presence

of

cycles

under

routing

constraint

R

This

routing

class

was

developed

by

Duato

to

provide

fullyadaptive

routing

with

minimal

number

of

virtual

channels

-LSB-

-RSB-

Recently

many

other

routing

schemes

-LSB-

-RSB-

have

been

added

to

this

class

The

BRCP

Model

Let

us

consider

the

routing

sequence

of

a

multidestination

worm

As

discussed

in

Section

it

carries

ids

of

multiple

destinations

in

its

header

as

an

ordered

list

It

gets

routed

in

a

piece-wise

manner

The

first

routing

step

happens

between

the

source

and

the

first

destination

of

this

worm

-LRB-

This

step

may

involve

multiple

sub-steps

depending

on

the

number

of

routers

between

the

source

and

the

first

destination

-RRB-

The

second

routing

step

happens

between

the

first

and

second

destination

and

so

on

If

all

these

routing

steps

conform

to

the

base

routing

of

a

system

no

modification

of

the

routing

scheme

is

required

for

implementing

multidestination

worms

Also

a

multidestination

worm

holds

on

to

the

set

of

traversed

communication

channels

-LRB-

until

the

tail

flit

passes

through

-RRB-

even

after

reaching

an

intermediate

destination

and

attempts

to

obtain

additional

communication

channels

to

reach

the

next

destination

This

raises

two

fundamental

problems

a

-RRB-

at

any

intermediate

destination

it

should

be

feasible

for

a

multidestination

worm

to

find

a

path

leading

to

the

remaining

destinations

and

b

-RRB-

the

piece-wise

routing

of

a

multidestination

worm

should

not

introduce

additional

communication

channel

dependency

leading

to

deadlock

We

introduce

a

BRCP

model

for

multidestination

worms

to

solve

both

of

the

above

problems

This

model

is

also

applicable

to

both

classes

of

routing

schemes

which

were

discussed

earlier

Let

us

assume

the

network

supports

a

deadlock-free

routing

scheme

R

for

unicast

message

passing

This

routing

R

could

be

e-cube

-LSB-

-RSB-

planar-adaptive

-LSB-

-RSB-

turn-model

-LSB-

-RSB-

fully-adaptive

-LSB-

-RSB-

or

any

other

routing

scheme

We

identify

R

as

the

base-routing

Then

the

BRCP

model

with

respect

to

multidestination

worms

can

be

formally

defined

as

follows

Definition

A

multidestination

worm

from

source

s

with

an

ordered

destination

list

-LRB-

dm

dm

-RRB-

conforms

to

the

base

routing

R

if

and

only

if

the

destination

set

-LRB-

dm

dm

-RRB-

can

be

covered

as

intermediate

nodes

on

one

of

the

possible

paths

from

s

to

dm

under

R

where

R

R

if

R

is

a

routing

function

with

no

cyclic

dependencies

and

R

if

R

is

a

routing

function

with

cyclic

dependencies

and

is

the

associated

routing

sub-function

of

R

with

no

cycles

in

its

extended

channel

dependency

graph

Next

let

us

determine

a

simple

procedure

to

find

out

whether

an

ordered

list

of

destinations

can

be

used

as

the

header

of

a

BRCP

multidestination

worm

for

a

given

routing

scheme

Let

P

rdi

dj

be

defined

as

the

set

of

all

paths

allowed

between

two

nodes

di

and

dj

under

unicast

routing

constraint

r.

Let

the

symbol

denote

the

cartesian

product

of

elements

of

two

sets

Consider

routing

R

as

defined

in

Definition

Definition

An

ordered

list

of

destinations

-LRB-

dm

dm

-RRB-

can

be

used

as

the

header

of

a

BRCP

multidestination

worm

for

a

given

routing

R

if

PR

PR

PR

dm

PR

sdm

Theorem

Multidestination

worms

under

the

BRCP

model

as

defined

above

can

co-exist

with

unicast

worms

without

leading

to

additional

communication

channel

dependency

in

the

network

Proof

Let

us

take

a

simple

BRCP

multidestination

worm

w

with

source

s

and

only

two

destinations

and

in

that

order

Let

w

take

the

path

from

s

to

and

the

path

from

to

It

can

be

easily

observed

from

Defn

and

Defn

that

PR

PR

and

the

composite

path

-LRB-

ffi

-RRB-

PR

Since

R

has

no

cyclic

dependencies

there

are

no

additional

communication

channel

dependency

while

routing

w.

Therefore

multidestination

worms

under

the

BRCP

model

can

co-exist

with

unicast

worms

without

leading

to

additional

communication

channel

dependency

The

argument

can

be

easily

extended

to

BRCP

multidestination

worms

with

more

than

two

destinations

It

can

be

observed

from

the

above

results

that

a

BRCP

multidestination

worm

will

always

be

able

to

find

a

path

to

cover

the

destinations

It

is

to

be

noted

that

for

the

class

of

routing

schemes

with

cyclic

dependencies

it

is

not

necessary

to

restrict

multidestination

messages

to

use

only

escape

channels

-LRB-

under

routing

constraints

of

-RRB-

These

messages

are

allowed

to

use

adaptive

channels

However

by

arranging

the

destinations

according

to

it

is

guaranteed

that

a

multidestination

message

can

select

an

escape

channel

at

any

node

Besides

the

communication

channel

dependency

multidestination

worms

have

potential

to

create

additional

dependency

due

to

router-processor

consumption

channels

Later

in

this

section

we

analyze

this

problem

and

determine

the

required

number

of

consumption

channels

to

be

provided

in

a

system

to

alleviate

such

dependency

and

the

associated

deadlocks

Illustration

of

the

BRCP

Model

We

illustrate

the

BRCP

model

by

applying

it

to

four

common

routing

schemes

e-cube

-LSB-

-RSB-

planaradaptive

-LSB-

-RSB-

turn-model

-LSB-

-RSB-

and

fully-adaptive

-LSB-

-RSB-

We

use

Definition

to

determine

valid

paths

for

multidestination

worms

Let

us

apply

the

BRCP

model

to

a

mesh

supporting

e-cube

routing

Figure

shows

valid

paths

for

a

mesh

supporting

e-cube

routing

under

the

BRCP

model

It

can

be

observed

that

three

different

kinds

of

paths

are

valid

row

path

-LRB-

R

-RRB-

covering

a

set

of

nodes

along

a

row

column

path

-LRB-

C

-RRB-

along

a

column

and

row-column

path

-LRB-

RC

-RRB-

along

a

row-column

It

is

to

be

noted

that

a

similar

but

restricted

method

of

using

only

column-path

has

also

been

independently

proposed

recently

by

Boppana

Chalasani

and

Raghavendra

-LSB-

-RSB-

Ecube

Planar

adaptiveTurn

model

source

destination

-LRB-

west-first

non-minimal

-RRB-

Fully-adaptive

-LRB-

Duato

-RRB-

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

Figure

Feasible

paths

for

multidestination

worms

under

the

BRCP

model

conforming

to

different

base

routing

schemes

in

a

mesh

In

addition

to

ecube

paths

added

flexibility

for

new

paths

under

planar

and

west

first

routing

schemes

are

shown

in

bold

lines

The

BRCP

model

can

be

extended

to

higher

dimensional

e-cube

systems

For

example

let

us

consider

an

e-cube

routed

system

with

x

y

and

z

dimensions

where

unicast

routing

gets

done

from

lower

dimension

x

to

highest

dimension

z

In

this

system

seven

kinds

of

paths

are

feasible

under

the

BRCP

model

X

Y

Z

XY

YZ

XZ

and

XYZ

Similar

paths

can

also

be

enumerated

for

higher

dimensional

e-cube

routed

systems

Let

us

consider

the

turn-model

routing

scheme

-LSB-

-RSB-

which

is

an

adaptive

routing

scheme

that

does

not

use

virtual

channels

This

scheme

can

support

either

minimal

or

non-minimal

routing

Figure

shows

valid

BRCPs

for

a

system

supporting

west-first

turn

model

with

non-minimal

routing

It

can

be

observed

that

in

addition

to

the

BRCPs

for

e-cube

routing

-LRB-

R

C

and

RC

-RRB-

multiple

destinations

falling

along

a

non-minimal

west-first

-LRB-

NMWF

-RRB-

path

can

be

covered

by

a

valid

multidestination

worm

It

is

to

be

noted

that

only

minimal

west-first

-LRB-

MWF

-RRB-

paths

are

valid

if

the

base

routing

supports

minimal

routing

in

this

system

These

results

for

west-first

turn

model

can

easily

be

extended

to

other

turn-model

systems

-LRB-

north-last

and

all-but-one-negative

first

-RRB-

and

systems

of

higher

dimension

Next

let

us

consider

planar-adaptive

routing

-LSB-

-RSB-

Figure

shows

the

valid

BRCPs

for

a

system

supporting

planar

adaptive

routing

In

addition

to

the

BRCPs

for

e-cube

routing

this

routing

provides

flexibility

for

a

multidestination

worm

to

take

any

combination

of

x

and

y

links

along

a

specific

direction

For

a

system

these

can

be

in

the

directions

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

-LRB-

x

y

-RRB-

and

-LRB-

x

y

-RRB-

We

identify

such

BRCPs

as

diagonal

-LRB-

D

-RRB-

paths

For

a

system

such

diagonal

paths

can

be

along

the

the

respective

planes

For

example

a

sequence

of

destinations

in

the

-LRB-

x

y

-RRB-

direction

along

the

XY

plane

followed

by

a

sequence

of

destinations

in

the

-LRB-

y

z

-RRB-

direction

along

the

YZ

plane

is

a

valid

BRCP

Similar

paths

can

be

derived

for

other

combinations

of

planes

in

and

higher

dimensional

systems

In

the

current

version

of

planar-adaptive

routing

-LSB-

-RSB-

some

ambiguity

exists

for

unicast

messagepassing

with

respect

to

selection

of

virtual

channels

along

dimension

di

in

adaptive

plane

Ai

This

happens

when

the

source

-LRB-

or

the

first

node

where

the

worm

enters

adaptive

plane

Ai

-RRB-

and

the

destination

have

same

offset

along

dimension

di

In

such

scenarios

a

unicast

worm

may

get

routed

along

virtual

channel

di

-LRB-

increasing

network

-RRB-

or

channel

di

-LRB-

decreasing

network

-RRB-

without

leading

to

deadlock

However

multidestination

messages

with

piece-wise

routing

between

destinations

may

lead

to

deadlock

with

such

ambiguity

Figure

in

Appendix

A

shows

an

example

of

such

deadlock

With

a

consistent

channel

selection

policy

-LRB-

sticking

with

either

virtual

channel

di

or

di

-RRB-

such

deadlock

scenarios

can

be

easily

avoided

for

multidestination

messages

Finally

let

us

apply

the

BRCP

model

to

Duato

's

fully-adaptive

routing

scheme

-LSB-

-RSB-

Let

us

assume

that

the

routing

sub-function

used

for

this

fully-adaptive

scheme

is

e-cube

Under

this

situation

as

shown

in

Fig.

R

C

and

RC

are

valid

BRCPs

These

paths

are

similar

to

those

for

e-cube

routing

as

shown

in

Fig.

Depending

on

the

routing

sub-function

of

the

fully-adaptive

routing

scheme

valid

BRCPs

will

conform

to

Deadlock

due

to

Consumption

Channel

Dependency

As

discussed

in

Section

a

router-to-processor

consumption

channel

is

required

by

a

multidestination

worm

at

each

of

its

destinations

in

order

to

implement

the

forward-and-absorb

mechanism

A

consumption

channel

is

captured

by

the

first

arriving

flit

of

a

message

and

is

released

when

the

tail

flit

passes

through

If

a

consumption

channel

is

not

available

at

a

destination

the

worm

gets

blocked

at

that

node

Thus

additional

dependencies

among

multidestination

worms

may

arise

if

each

node

in

the

system

has

limited

number

of

consumption

channels

leading

to

deadlock

Such

deadlock

situations

may

vary

depending

on

the

topology

the

base

routing

scheme

and

the

BRCPs

used

Let

us

first

consider

the

consumption

channel

dependency

on

a

simple

meshes

with

e-cube

routing

Theorem

On

an

e-cube

routed

mesh

minimum

number

of

consumption

channels

required

to

achieve

deadlock-free

multidestination

message

passing

are

two

if

either

R

paths

or

C

paths

are

used

three

if

both

R

and

C

paths

are

used

and

four

if

R

C

and

RC

paths

are

used

Proof

On

an

e-cube

routed

mesh

it

is

possible

that

two

multidestination

worms

going

in

opposite

directions

along

a

R

path

have

common

destinations

Figure

shows

such

an

example

where

multidestination

worms

and

have

common

destinations

B

and

D

During

the

propagation

of

these

worms

it

may

happen

that

a

consumption

channel

at

B

is

allocated

to

worm

and

a

consumption

channel

at

D

is

allocated

to

If

we

assume

one

consumption

channel

per

node

no

consumption

channels

are

available

for

and

when

they

arrive

at

nodes

B

and

D

respectively

This

leads

to

a

cyclic

dependency

between

these

two

worms

and

deadlock

occurs

Such

deadlocks

can

be

avoided

if

each

node

has

at

least

two

consumption

channels

This

argument

also

holds

good

for

worms

going

in

opposite

directions

along

C

paths

If

multidestination

worms

along

both

R

and

C

paths

are

allowed

in

the

system

then

there

might

be

a

cyclic

dependency

between

worms

along

R

and

C

paths

due

to

limited

number

of

consumption

channels

per

node

Figure

shows

such

an

example

with

two

consumption

channels

per

node

Such

deadlocks

can

be

avoided

if

each

node

has

at

least

three

consumption

channels

If

worms

along

RC

paths

are

allowed

then

a

node

may

receive

multidestination

messages

from

these

paths

This

leads

to

added

dependency

and

three

consumption

channels

may

not

be

sufficient

as

shown

in

Figure

In

such

cases

at

least

four

consumption

channels

per

node

are

needed

Similar

solutions

have

also

been

proposed

in

-LSB-

-RSB-

to

provide

two

consumption

channels

per

node

for

deadlock

freedom

in

column-path

routing

Since

the

BRCP

model

provides

more

flexibility

more

consumption

channels

are

required

The

above

results

derived

for

mesh

supporting

e-cube

routing

can

be

easily

extended

to

k-ary

n-cube

topology

and

other

routing

schemes

The

basic

idea

is

to

determine

the

maximum

number

of

multidestination

worms

which

can

enter

a

node

under

the

BRCP

model

for

a

given

-LRB-

B

A

-RRB-

-LRB-

C

B

-RRB-

-LRB-

D

C

-RRB-

-LRB-

A

D

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

A

BCD

-LRB-

D

A

-RRB-

-LRB-

C

D

-RRB-

-LRB-

B

C

-RRB-

-LRB-

A

B

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

C

B

A

-RRB-

-LRB-

C

B

-RRB-

-LRB-

A

D

C

-RRB-

-LRB-

A

D

-RRB-

-LRB-

-RRB-

A

BCD

-LRB-

D

A

-RRB-

-LRB-

B

C

D

-RRB-

-LRB-

B

C

-RRB-

-LRB-

D

A

B

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

multidestination

worm

with

destinations

-LRB-

consumption

channel

being

held

by

a

worm

-RRB-

B

C

D

E

-LRB-

D

B

-RRB-

-LRB-

B

D

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

A

-LRB-

a

-RRB-

Figure

Examples

of

deadlock

due

to

multidestination

worms

through

consumption

channel

dependency

in

a

mesh

with

e-cube

routing

a

-RRB-

a

set

of

R

or

C

paths

with

one

consumption

channel

per

node

b

-RRB-

a

set

of

R

and

C

paths

with

two

consumption

channels

per

node

and

-LRB-

b

-RRB-

a

set

of

C

and

RC

paths

with

three

consumption

channels

per

node

topology

and

routing

scheme

This

leads

to

Theorem

If

a

base

routing

R

in

a

k-ary

n-cube

system

requires

v

virtual

channels

per

physical

channel

to

support

deadlock-free

unicast

communication

then

consumption

channels

are

sufficient

for

deadlock-free

multidestination

communication

Proof

If

v

virtual

channels

are

used

then

each

router

has

input

channels

and

output

channels

Under

the

BRCP

model

it

may

be

feasible

for

a

multidestination

worm

to

enter

along

any

input

channel

of

the

router

and

to

go

out

along

any

output

channel

Thus

consumption

channels

are

sufficient

for

not

introducing

any

additional

dependency

among

the

multidestination

worms

The

above

result

holds

good

for

any

routing

and

topology

without

virtual

lane

-LSB-

-RSB-

support

Virtual

lanes

are

typically

used

to

reduce

congestion

and

do

not

introduce

any

additional

communication

channel

dependency

Each

set

of

virtual

lanes

entering

a

node

from

a

given

dimension

and

direction

can

share

a

consumption

channel

Consider

a

multidestination

worm

arriving

on

a

virtual

lane

of

a

given

set

and

reserving

a

consumption

channel

If

a

second

multidestination

worm

arrives

using

another

virtual

channel

of

the

same

set

then

it

must

wait

This

waiting

can

not

produce

deadlock

because

both

messages

follow

the

same

direction

Thus

the

minimum

number

of

consumption

channels

required

for

a

system

with

virtual

lane

support

remains

the

same

as

that

of

its

underlying

deadlock-free

routing

R

as

indicated

in

Theorem

However

more

consumption

channels

may

be

used

to

reduce

blocking

and

obtain

better

performance

-LSB-

-RSB-

For

adaptive

routing

schemes

with

cyclic

dependencies

such

as

the

one

proposed

by

Duato

-LSB-

-RSB-

the

BRCP

model

allows

paths

conforming

to

the

associated

acyclic

routing

sub-function

as

discussed

earlier

Therefore

the

consumption

channel

requirement

for

fully-adaptive

routing

systems

is

the

same

as

that

of

systems

with

the

associated

acyclic

routing

sub-function

Table

shows

the

minimum

number

of

consumption

channels

required

per

node

for

a

mesh

with

different

routing

schemes

and

BRCP

paths

It

can

be

observed

that

only

four

consumption

channels

are

sufficient

when

the

base

routing

is

e-cube

west-first

non-minimal

or

fully-adaptive

with

e-cube

routing

as

escape

channels

Table

Minimum

number

of

consumption

channels

per

node

required

to

achieve

deadlock-free

multidestination

message-passing

on

a

mesh

under

different

routing

and

BRCPs

Routing

BRCP

Paths

No

of

Consumption

Channels

e-cube

R

e-cube

C

e-cube

R

C

e-cube

R

C

RC

turn

-LRB-

west-first

non-minimal

-RRB-

R

C

RC

NMWF

planar-adaptive

R

C

D

fully-adaptive

-LRB-

Duato

-RRB-

R

C

RC

-LRB-

e-cube

routing

as

-RRB-

The

above

consumption

channels

are

required

for

deadlock-freedom

Typically

the

probability

of

all

the

consumption

channels

at

a

node

being

accessed

simultaneously

is

quite

low

Therefore

these

consumption

channels

can

be

implemented

efficiently

as

virtual

channels

by

multiplexing

the

available

bandwidth

at

the

network

interface

Such

virtual

channel

implementation

-LRB-

using

multiple

DMA

channels

-RRB-

requires

very

little

additional

logic

considering

the

existing

complexity

of

today

's

routers

Thus

the

BRCP

model

is

a

feasible

solution

for

current

and

future

generation

wormhole-routed

systems

Potential

Benefits

for

Collective

Communication

The

significant

benefit

of

this

BRCP

model

is

that

a

message

can

be

delivered

to

multiple

destinations

with

the

same

overhead

as

that

of

sending

it

to

a

single

destination

if

the

destinations

can

be

grouped

into

a

single

worm

Even

the

simplest

e-cube

routing

scheme

can

take

advantage

of

this

model

by

grouping

destinations

along

R

C

and

RC

paths

for

broadcast

and

multicast

operations

As

the

adaptivity

of

the

base

routing

increases

more

destinations

can

be

covered

by

a

single

multidestination

worm

If

the

set

of

destinations

demonstrate

some

spatial

organization

-LRB-

row

column

diagonal

or

block

-RRB-

then

a

suitable

scheme

based

on

the

underlying

routing

constraint

can

be

developed

to

cover

them

using

as

few

multidestination

worms

as

possible

Thus

this

model

opens

up

an

entirely

new

framework

together

with

its

challenges

for

developing

optimal

algorithms

to

implement

collective

communication

operations

-LSB-

-RSB-

in

minimal

time

In

this

paper

we

demonstrate

the

advantages

of

the

BRCP

model

by

developing

and

evaluating

algorithms

for

broadcast

and

multicast

which

are

important

collective

communication

operations

Application

of

this

model

to

other

collective

communication

operations

like

synchronization

and

global

reduction

can

be

found

in

-LSB-

-RSB-

Broadcasting

To

demonstrate

the

potential

of

the

BRCP

model

we

consider

two

commonly

used

collective

communication

operations

one-to-all

broadcasting

-LSB-

-RSB-

and

one-to-many

multicasting

-LSB-

-RSB-

In

this

section

we

show

how

the

BRCP

model

can

be

used

to

implement

broadcasting

on

kary

n-cube

systems

with

reduced

latency

First

we

propose

efficient

broadcasting

algorithms

for

e-cube

systems

Next

we

propose

algorithms

for

adaptive-routed

systems

and

higher

dimensional

systems

E-cube

Routed

Mesh

Let

us

consider

a

mesh

with

e-cube

routing

As

discussed

in

Sec.

the

BRCP

model

provides

three

different

kinds

of

paths

-LRB-

R

C

and

RC

-RRB-

on

an

e-cube

routed

mesh

We

consider

two

scenarios

the

source

node

being

a

corner

node

of

the

mesh

and

being

an

arbitrary

node

Corner

Source

Let

the

source

of

a

broadcast

be

a

corner

node

of

the

mesh

Figure

shows

two

ways

to

implement

the

broadcast

As

shown

in

Fig.

the

source

can

initiate

a

multidestination

worm

along

the

leftmost

column

of

the

mesh

in

the

first

step

In

the

second

step

each

node

on

this

column

acts

like

a

secondary

source

and

initiates

a

multidestination

worm

along

its

row

We

can

denote

this

twostep

broadcasting

scheme

as

fC

Rg

Using

row-column

paths

a

two-step

fRC

Cg

scheme

can

also

be

used

as

shown

in

Fig.

An

alternative

fRC

Rg

scheme

is

also

possible

where

the

destinations

along

the

rightmost

column

initiate

row

worms

in

the

second

step

All

these

schemes

encounter

two

communication

start-ups

to

implement

broadcast

on

a

mesh

Step

Step

multidestination

wormsStep

Step

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

One-to-all

broadcasting

in

a

mesh

from

a

corner

source

node

using

multidestination

worms

-LRB-

a

-RRB-

scheme

fC

Rg

using

row

and

column

paths

and

-LRB-

b

-RRB-

scheme

fRC

Cg

using

row-column

and

column

paths

Arbitrary

Source

Let

us

consider

broadcasting

from

an

arbitrary

source

The

schemes

in

Fig.

can

be

trivially

extended

to

schemes

where

the

source

first

sends

an

unicast

-LRB-

U

-RRB-

message

to

a

corner

node

and

then

the

remaining

steps

are

carried

out

These

schemes

can

be

defined

as

fU

C

Rg

and

fU

RC

Cg

schemes

respectively

Besides

these

schemes

RC

paths

enable

broadcast

implementation

in

fewer

steps

Consider

broadcasting

on

a

mesh

as

shown

Fig.

This

scheme

can

be

identified

as

fRC

-LRB-

RC/R/C

-RRB-

g

scheme

Fig.

shows

an

alternative

scheme

where

the

same

source

sends

a

multidestination

worm

to

the

-LRB-

x,-y

-RRB-

corner

node

during

the

first

step

The

following

theorem

shows

how

such

a

scheme

is

possible

for

arbitrary

source

positions

for

most

meshes

Theorem

One-to-all

broadcasting

from

an

arbitrary

source

node

in

an

e-cube

routed

mesh

can

be

implemented

in

two

steps

under

the

BRCP

model

using

a

combination

of

row-column

-LRB-

RC

-RRB-

row

-LRB-

R

-RRB-

and

column

-LRB-

C

-RRB-

paths

provided

Proof

Let

us

consider

a

e-cube

routed

mesh

with

the

source

position

at

-LRB-

-RRB-

as

shown

in

Fig.

In

the

first

step

the

source

can

send

a

multidestination

worm

to

any

one

of

the

four

corner

nodes

of

the

mesh

Let

us

consider

the

choice

of

sending

a

multidestination

worm

to

-LRB-

x

y

-RRB-

corner

node

If

-LRB-

-RRB-

-LRB-

-RRB-

then

the

broadcasting

can

be

finished

in

one

more

step

similar

to

the

case

shown

in

Fig.

This

condition

leads

to

an

inequality

-LRB-

-RRB-

-LRB-

-RRB-

Similarly

the

inequalities

for

sending

the

multidestination

worm

in

the

first

step

to

the

other

three

corners

can

be

derived

These

inequalities

are

shown

below

-LRB-

-RRB-

-LRB-

-RRB-

worm

to

-LRB-

x

y

-RRB-

corner

node

worm

to

-LRB-

x

y

-RRB-

corner

node

-LRB-

-RRB-

-LRB-

-RRB-

worm

to

-LRB-

x

y

-RRB-

corner

node

-LRB-

-RRB-

-LRB-

-RRB-

worm

to

-LRB-

x

y

-RRB-

corner

node

-LRB-

-RRB-

Multiple

choices

are

available

for

some

positions

of

the

source

We

can

show

by

contradiction

that

the

inequalities

cover

all

possible

positions

-LRB-

-RRB-

of

the

source

Let

us

assume

that

-LRB-

-RRB-

such

that

it

is

not

covered

in

any

of

the

four

cases

This

implies

that

the

negation

of

each

of

the

above

inequalities

is

true

This

is

given

as

-LRB-

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

-LRB-

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

Using

the

first

and

last

inequality

from

this

expression

the

inequality

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

can

be

derived

This

inequality

can

never

true

for

any

-LRB-

-RRB-

if

Therefore

all

possible

positions

of

the

source

are

covered

by

the

above

cases

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

c

-RRB-

source

step

x

y

Figure

Two-step

broadcasting

schemes

in

ecube-routed

mesh

from

an

arbitrary

source

node

using

a

combination

of

RC

R

and

C

paths

-LRB-

a

-RRB-

sending

first

step

multidestination

worm

to

-LRB-

x

y

-RRB-

corner

node

-LRB-

b

-RRB-

an

alternative

scheme

by

sending

first

step

worm

to

-LRB-

x,-y

-RRB-

corner

node

and

-LRB-

c

-RRB-

four

different

choices

in

sending

first

step

worm

to

one

of

the

four

corner

nodes

depending

on

the

values

of

and

as

discussed

in

Theorem

Taking

Advantage

of

Adaptivity

The

BRCP

model

offers

flexibility

in

choosing

paths

with

adaptive

routing

schemes

Now

the

question

is

whether

this

added

flexibility

helps

in

implementing

faster

broadcast

Figure

shows

broadcasting

in

a

mesh

supporting

planar-adaptive

routing

This

fD

D/C

Cg

scheme

takes

three

steps

Since

R

C

and

RC

paths

are

always

realizable

under

planar-adaptive

routing

the

two-step

broadcasting

scheme

as

discussed

earlier

can

be

used

to

support

fast

broadcasting

if

Otherwise

the

trivial

broadcasting

mentioned

at

the

beginning

of

Sec.

can

be

used

Therefore

the

adaptivity

does

not

help

in

reducing

broadcast

latency

In

the

next

section

we

will

show

that

D

paths

can

in

fact

help

in

reducing

latency

for

multicast

operation

For

systems

supporting

fully-adaptive

routing

-LSB-

-RSB-

with

e-cube-routed

escape

channels

the

supportable

BRCP

paths

are

R

C

and

RC

Thus

broadcasting

can

be

done

in

two

steps

by

using

the

fRC

-LRB-

RC/R/C

-RRB-

g

scheme

-LRB-

if

-RRB-

Otherwise

a

three

step

fU

RC

Cg

scheme

can

be

used

step

-LRB-

a

-RRB-

-LRB-

b

-RRB-

step

step

-LRB-

c

-RRB-

Planar-adaptive

Turn

model

-LRB-

west-first

-RRB-

Turn

model

-LRB-

north-last

-RRB-

Figure

One-to-all

broadcasting

on

systems

with

adaptive

routing

a

-RRB-

three-step

broadcasting

with

planar-adaptive

routing

b

-RRB-

one-step

scheme

for

turn

model

with

west-first

non-minimal

routing

and

c

-RRB-

two-step

scheme

for

turn

model

with

north-last

non-minimal

routing

Now

let

us

consider

the

turn-model

-LSB-

-RSB-

with

west-first

non-minimal

routing

As

shown

in

Fig.

one-to-all

broadcasting

can

be

implemented

in

just

one

step

using

a

fNMWFg

scheme

Here

the

adaptivity

as

well

as

the

non-minimal

property

in

routing

help

in

reducing

the

broadcast

latency

Though

the

multidestination

worm

along

a

NMWF

path

visits

some

nodes

twice

the

header

of

this

worm

can

be

appropriately

ordered

so

that

no

node

-LRB-

including

the

source

node

-RRB-

receives

more

than

one

copy

of

the

message

It

is

to

be

noted

that

such

one-step

broadcasting

is

not

possible

if

the

routing

supports

only

minimal

paths

Also

one-step

broadcasting

is

not

feasible

for

other

turn

model

systems

such

as

north-last

nonminimal

routing

Two

steps

are

required

in

this

case

as

shown

in

Fig.

The

first

step

uses

an

RC

path

and

the

second

step

uses

a

non-minimal

north-last

-LRB-

NMNL

-RRB-

path

leading

to

a

fRC

NMNLg

scheme

It

can

be

observed

that

one-step

scheme

is

possible

if

the

routing

supports

north-first

instead

of

north-last

Thus

on

systems

supporting

turn

model

the

complexity

of

broadcasting

depends

on

the

exact

nature

of

paths

being

supported

by

the

model

Here

onwards

without

loss

of

generality

we

consider

only

west-first

non-minimal

routing

for

systems

supporting

turn

model

For

higher

dimensional

systems

with

turn

model

we

consider

the

counterpart

of

the

west-first

routing

all-but-one-negative-first

routing

-LSB-

-RSB-

In

the

absence

of

the

BRCP

model

it

takes

-LRB-

-RRB-

e

communication

steps

to

support

broadcasting

in

a

mesh

using

unicast

message

passing

-LSB-

-RSB-

This

leads

to

Theorem

One-to-all

broadcasting

from

an

arbitrary

source

in

a

wormhole-routed

mesh

can

be

implemented

in

two

-LRB-

if

-RRB-

or

three

-LRB-

if

-RRB-

communication

steps

under

the

BRCP

model

if

the

underlying

routing

is

e-cube

planar-adaptive

or

fully-adaptive

For

turn-model

with

west-first

non-minimal

routing

only

one

communication

step

is

sufficient

Compared

to

the

unicast-based

message-passing

the

BRCP

model

reduces

the

number

of

communication

steps

by

a

factor

of

up

to

-LRB-

-RRB-

e

depending

on

the

base

routing

Higher

Dimensional

Systems

The

above

broadcasting

schemes

for

mesh

can

be

easily

extended

to

higher

dimensional

systems

k-ary

n-meshes

and

generalized

n-D

meshes

-LRB-

kn

-RRB-

We

first

consider

systems

with

e-cube

planar-adaptive

and

fully-adaptive

schemes

Later

systems

supporting

the

turn

model

are

considered

In

an

n-dimensional

system

let

the

source

node

send

its

message

to

a

corner

node

of

the

system

during

the

first

step

Now

two

additional

steps

can

be

used

to

cover

nodes

on

a

plane

containing

this

corner

node

Next

the

nodes

on

this

plane

can

send

multidestination

worms

along

the

third

dimension

to

cover

all

nodes

in

a

cube

containing

the

above

plane

Now

the

nodes

in

this

cube

can

cover

nodes

along

the

fourth

dimension

with

an

additional

communication

step

This

leads

to

a

broadcasting

scheme

with

-LRB-

n

-RRB-

steps

By

using

unicast-based

approach

-LSB-

-RSB-

one-to-all

broadcasting

in

a

k-ary

n-mesh

takes

-LRB-

kn

-RRB-

e

steps

This

leads

to

Theorem

One-to-all

broadcasting

from

an

arbitrary

source

node

in

a

k-ary

n-mesh

or

in

a

generalized

n-D

mesh

supporting

e-cube

planar-adaptive

or

fully-adaptive

routing

can

be

implemented

in

-LRB-

n

-RRB-

communication

steps

under

the

BRCP

model

Compared

to

the

unicast-based

approach

the

BRCP

model

reduces

the

number

of

communication

steps

by

a

factor

of

-LRB-

N

-RRB-

e

-LRB-

n

-RRB-

where

N

kn

for

k-ary

n-mesh

and

N

kn

for

generalized

n-D

mesh

Next

let

us

consider

a

system

supporting

turn

model

with

all-but-one-negative-first

routing

which

supports

unicast

messages

getting

routed

adaptively

in

x

and

y

directions

first

before

getting

routed

adaptively

in

the

remaining

directions

As

shown

in

Fig.

broadcasting

can

be

implemented

on

this

system

in

two-steps

Such

broadcasting

schemes

can

be

easily

extended

to

higher

dimensional

systems

This

leads

to

the

following

result

Theorem

One-to-all

broadcasting

on

an

n-dimensional

mesh

supporting

turn

model

and

all-butone-negative-first

routing

can

be

implemented

in

-LRB-

-RRB-

steps

Detailed

proof

is

given

in

Appendix

B.

step

-LRB-

a

-RRB-

-LRB-

b

-RRB-

step

x

y

z

Figure

Two-step

broadcasting

in

a

mesh

supporting

all-but-one-negative-first

turn

model

routing

Two

alternative

schemes

a

-RRB-

left-most

yz

plane

is

getting

covered

first

and

b

-RRB-

bottom-most

xz

plane

is

getting

covered

first

Since

the

main

goal

of

this

paper

is

to

demonstrate

the

significance

of

the

BRCP

model

we

do

not

emphasize

proposing

an

optimal

broadcast

algorithm

for

the

turn-model

In

the

above

algorithm

we

have

used

non-minimal

paths

only

during

the

first

step

It

may

be

possible

to

use

such

paths

in

the

following

steps

to

reduce

number

of

steps

for

broadcasting

We

are

investigating

such

possibilities

In

this

section

we

have

only

dealt

with

meshes

For

systems

with

wrap-around

BRCP

paths

with

wrap-around

can

be

used

For

example

in

a

torus

with

e-cube

routing

the

R

and

C

paths

can

be

extended

to

row

path

with

wrap-around

-LRB-

RW

-RRB-

and

column

path

with

wrap-around

-LRB-

CW

-RRB-

However

with

minimal

routing

such

RW

and

CW

paths

do

not

satisfy

the

constraints

presented

in

Definition

to

become

valid

BRCP

paths

This

can

be

easily

observed

in

the

example

shown

in

Fig.

The

CW

path

followed

by

a

multidestination

worm

going

from

s

to

d

is

not

the

same

as

the

path

of

a

unicast

worm

going

from

s

to

d

Such

paths

become

valid

only

under

non-minimal

routing

Since

most

of

the

current

torus

configurations

support

minimal

routing

we

do

not

consider

such

wrap-around

paths

in

this

paper

any

further

source

multidestination

worm

-LRB-

a

-RRB-

-LRB-

b

-RRB-

sd

sd

unicast

worm

destination

Figure

An

example

of

an

RW

path

which

does

not

satisfy

the

BRCP

definition

under

minimal

routing

-LRB-

a

-RRB-

the

RW

multidestination

worm

from

source

s

to

the

last

destination

d

-LRB-

b

-RRB-

a

unicast

worm

from

source

s

to

destination

d.

Multicasting

In

this

section

our

emphasis

is

on

one-to-many

multicasting

It

may

be

argued

that

multicasting

can

always

be

implemented

by

treating

it

as

a

subset

operation

of

broadcasting

i.e.

forcing

a

message

to

go

to

every

node

in

the

system

However

this

may

lead

to

performance

degradation

because

a

node

not

participating

in

the

multicast

has

to

process

the

message

and

discard

it

-LRB-

with

cache

coherency

for

example

-RRB-

With

multiple

programs

running

at

disjoint

partitions

of

a

system

such

forced

implementation

of

multicasting

as

broadcast

operation

may

also

lead

to

ambiguity

Thus

our

emphasis

is

on

developing

multicasting

schemes

which

deliver

messages

to

only

those

nodes

which

participate

in

a

given

multicast

operation

We

propose

a

Hierarchical

Leader-based

-LRB-

HL

-RRB-

scheme

and

demonstrate

its

operational

principle

under

the

BRCP

model

for

different

routing

schemes

adaptivity

and

topologies

Hierarchical

Leader-based

-LRB-

HL

-RRB-

scheme

The

broadcasting

algorithms

presented

in

the

last

section

can

be

viewed

as

multi-step

-LRB-

hierarchical

-RRB-

algorithms

where

a

set

of

nodes

in

every

step

try

to

cover

additional

nodes

using

BRCP

paths

Since

the

broadcasting

is

a

regular

pattern

it

is

easier

to

derive

these

steps

-LRB-

which

node

needs

to

cover

which

other

nodes

-RRB-

For

multicast

with

arbitrary

number

of

nodes

we

present

a

bottom-up

approach

using

a

concept

of

leader

and

multiple

hierarchies

to

derive

these

steps

automatically

Definition

A

node

is

defined

to

be

a

leader

node

among

a

set

of

nodes

if

it

can

forward

a

message

to

other

members

-LRB-

nodes

-RRB-

in

this

set

in

one

communication

step

under

the

BRCP

model

using

a

single

multidestination

worm

Consider

a

multicast

pattern

from

a

source

s

with

a

destination

set

D.

Let

D

-LSB-

fsg

denote

the

base

-LRB-

level

-RRB-

set

This

set

can

be

partitioned

into

disjoint

subsets

so

that

each

subset

is

represented

by

a

leader

node

As

we

will

see

later

such

partitioning

can

be

done

based

on

the

allowable

BRCP

paths

Let

's

define

these

leaders

as

level

leaders

and

identify

the

set

of

these

level

leaders

by

a

new

set

This

set

can

further

be

partitioned

into

disjoint

subsets

by

applying

Defn

to

level

leaders

Let

us

identify

this

new

set

of

leaders

-LRB-

level

leaders

-RRB-

as

The

above

partitioning

process

can

be

applied

repeatedly

until

a

-RRB-

the

set

of

leaders

can

not

be

grouped

any

further

based

on

the

allowable

BRCP

paths

or

b

-RRB-

further

grouping

is

not

profitable

to

reduce

multicast

latency

We

emphasize

on

the

profitability

aspect

little

later

Let

us

assume

that

the

above

grouping

process

can

be

repeated

m

times

for

a

set

of

multicast

destinations

and

a

given

system

This

HL

grouping

process

leads

to

m

sets

of

level

i

leaders

i

m

where

Lm

ae

Lm

ae

ae

With

the

formation

of

the

above

leader

sets

the

multicast

can

take

place

in

-LRB-

m

-RRB-

phases

First

we

form

a

source-augmented

level

m

leader

set

Lsm

Lm

-LSB-

fsg

If

s

is

not

a

member

in

Lm

we

have

jLsmj

jLm

Otherwise

both

sets

are

identical

The

first

phase

involves

performing

unicast-based

multicast

-LRB-

using

the

Umesh

algorithm

-LSB-

-RSB-

-RRB-

on

the

set

Lsm

using

point-to-point

-LRB-

single

destination

-RRB-

messages

This

phase

takes

-LRB-

jLsmj

-RRB-

e

communication

steps

If

the

source

happens

to

be

the

sole

level

m

leader

then

this

phase

is

eliminated

After

the

Umesh

phase

every

level

m

leader

has

a

copy

of

the

message

During

the

second

phase

each

level

m

leader

forwards

the

message

to

a

set

of

level

-LRB-

m

-RRB-

leaders

for

which

it

was

chosen

as

a

leader

during

mth

step

of

the

grouping

using

a

multidestination

worm

Similar

to

this

second

phase

additional

-LRB-

m

-RRB-

phases

get

repeated

where

level

i

leaders

cover

level

-LRB-

i

-RRB-

leaders

i

-LRB-

m

-RRB-

using

multidestination

worms

Thus

the

multicast

gets

completed

in

-LRB-

m

-RRB-

phases

This

leads

to

Theorem

Using

hierarchical

leader-based

grouping

and

the

BRCP

model

a

multicast

operation

can

be

achieved

in

-LRB-

m

-RRB-

phases

using

-LRB-

jLsmj

-RRB-

e

m

communication

steps

where

the

sourceaugmented

set

Lsm

consists

of

source

node

s

and

the

set

of

level

m

leaders

Lm

If

the

source

node

happens

to

be

the

sole

level

m

leader

then

the

multicast

can

take

place

in

m

phases

using

only

m

communication

steps

Let

us

discuss

about

the

profitability

in

grouping

mentioned

earlier

If

the

grouping

is

terminated

at

jth

step

it

can

be

observed

that

Lsj

leaders

will

be

involved

in

the

first

phase

and

the

overall

multicast

can

be

done

in

-LRB-

jLsj

j

-RRB-

e

j

steps

Assume

it

is

possible

to

go

for

one

more

level

of

grouping

based

on

the

allowable

BRCP

paths

This

will

lead

to

Lj

level

-LRB-

j

-RRB-

leaders

and

the

overall

multicast

will

require

-LRB-

jLsj

j

-RRB-

e

-LRB-

j

-RRB-

steps

This

leads

to

Lemma

Under

the

HL

scheme

it

is

profitable

to

go

for

-LRB-

j

-RRB-

th

step

of

the

grouping

as

long

as

-LRB-

jLsj

j

-RRB-

e

-LRB-

jLsj

j

-RRB-

e

where

Lsj

and

Lsj

are

source-augmented

sets

of

level

-LRB-

j

-RRB-

and

level

j

leaders

respectively

Figure

outlines

the

grouping

and

multicasting

procedures

using

the

HL

scheme

Now

the

question

is

how

to

take

care

of

the

allowable

BRCP

paths

in

different

steps

of

the

HL

scheme

In

the

source

node

s

is

added

to

the

set

it

is

not

guaranteed

that

it

will

be

present

in

the

set

Lm

In

its

absence

the

algorithm

prohibits

the

source

node

to

initiate

any

multidestination

worm

We

include

source

node

to

implement

multicast

with

reduced

latency

the

following

subsections

we

illustrate

the

HL

scheme

for

different

routing

schemes

and

topologies

Procedure

Grouping

Input

source

s

a

set

of

destinations

D

and

allowable

BRCP

paths

Output

max

level

m

a

set

of

level

m

leaders

Lm

max

level

value

-LRB-

-RRB-

for

all

destinations

and

source

and

member

set

-LSB-

-LRB-

-RRB-

-LRB-

max

level

-RRB-

-RSB-

for

all

destinations

and

source

Group

-LRB-

-RRB-

f

i

fsg

-LSB-

D

k

k

max

level

-LRB-

k

-RRB-

repeat

f

form

disjoint

grouping

of

Li

based

on

allowable

BRCP

paths

i

i

Li

leaders

from

set

Li

form

source-augmented

sets

Lsi

Li

-LSB-

fsg

and

Lsi

Li

-LSB-

fsg

if

-LRB-

-LRB-

jLsi

j

-RRB-

e

-LRB-

jLsi

-RRB-

e

-RRB-

f

k

k

Li

f

max

level

-LRB-

k

-RRB-

i

member

set

-LSB-

-LRB-

k

-RRB-

i

-RSB-

sorted

list

of

nodes

reachable

from

node

k

on

a

BRCP

path

g

g

until

-LRB-

-LRB-

jLsi

j

-RRB-

e

-LRB-

jLsi

-RRB-

e

-RRB-

g

Return

-LRB-

m

Lm

Li

max

level

-LRB-

-RRB-

member

set

-LSB-

-LRB-

-RRB-

-LRB-

-RRB-

-RSB-

-RRB-

g

Procedure

Multicast

Input

source

s

a

set

of

destinations

D

max

level

m

a

set

of

level

m

leaders

Lm

max

level

value

-LRB-

-RRB-

for

all

destinations

and

source

and

member

set

-LSB-

-LRB-

-RRB-

-LRB-

max

level

-RRB-

-RSB-

for

all

destinations

and

source

Multicast

-LRB-

-RRB-

f

if

-LRB-

my

id

s

-RRB-

or

-LRB-

my

id

Lm

-RRB-

f

participate

in

Umesh

multicast

and

receive

a

copy

of

message

g

if

-LRB-

max

level

-LRB-

my

id

-RRB-

m

-RRB-

f

for

-LRB-

i

m

i

i

-RRB-

f

if

member

set

-LSB-

-LRB-

my

id

-RRB-

i

-RSB-

OE

send

out

multidestination

worm

to

member

set

-LSB-

-LRB-

my

id

-RRB-

i

-RSB-

g

g

else

f

receive

a

copy

of

the

message

for

-LRB-

i

max

level

-LRB-

my

id

-RRB-

i

i

-RRB-

f

if

member

set

-LSB-

-LRB-

my

id

-RRB-

i

-RSB-

OE

send

out

multidestination

worm

to

member

set

-LSB-

-LRB-

my

id

-RRB-

i

-RSB-

g

g

g

Figure

Procedures

for

hierarchical

grouping

and

multicast

using

the

hierarchical

leader

-LRB-

HL

-RRB-

scheme

under

the

BRCP

model

E-cube

Routed

Meshes

We

consider

two

cases

The

first

multicast

algorithm

uses

the

HL

scheme

with

only

R

and

C

paths

The

second

one

uses

RC

paths

with

multi-level

grouping

Using

Row

-LRB-

R

-RRB-

and

Column

-LRB-

C

-RRB-

Paths

The

grouping

in

every

step

of

the

HL

scheme

can

be

viewed

as

a

function

which

takes

a

set

of

nodes/leaders

as

input

and

returns

a

set

of

leaders

based

on

one

or

more

allowable

paths

Let

us

formally

define

such

grouping

as

a

function

Ffpathg

Li

Li

A

function

FR

indicates

that

it

allows

leaders

to

be

selected

along

rows

Similarly

a

function

FC

allows

selection

along

columns

Figure

shows

an

example

of

the

HL

scheme

with

two

levels

of

grouping

where

FR

-LRB-

-RRB-

and

FC

-LRB-

-RRB-

FC

-LRB-

FR

-LRB-

-RRB-

-RRB-

The

destination

nodes

are

initially

grouped

along

rows

to

determine

their

level

leaders

-LRB-

row-leaders

-RRB-

The

next

step

involves

determining

level

leaders

-LRB-

column-leaders

-RRB-

In

this

example

we

have

seven

level

leaders

and

three

level

leaders

It

is

to

be

noted

that

each

level

leader

is

also

a

level

leader

By

reversing

the

ordering

of

grouping

-LRB-

FC

-LRB-

-RRB-

and

FR

-LRB-

-RRB-

-RRB-

we

could

have

obtained

seven

level

leaders

and

the

same

three

level

leaders

Since

only

R

and

C

paths

are

available

no

further

grouping

of

leaders

in

is

possible

under

either

ordering

source

destination

level

leader

level

leader

unicast

worm

-LRB-

step

-RRB-

unicast

worm

-LRB-

step

-RRB-

multidest

C

worm

-LRB-

step

-RRB-

multidest

R

worm

-LRB-

step

-RRB-

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

multidest

C

worm

-LRB-

step

-RRB-

multidest

R

worm

-LRB-

step

-RRB-

Figure

An

example

multicast

in

a

e-cube

routed

mesh

using

hierarchical

leader-based

-LRB-

HL

-RRB-

scheme

with

R

and

C

paths

After

the

grouping

is

over

the

multicast

takes

place

as

follows

In

the

first

phase

the

Umesh

algorithm

-LSB-

-RSB-

is

used

to

cover

three

level

leaders

in

two

communication

steps

The

second

phase

involves

level

leaders

sending

multidestination

worms

along

C

paths

In

the

last

phase

all

level

leaders

send

multidestination

worms

along

R

paths

Thus

the

multicast

is

finished

in

four

steps

We

can

identify

this

multicast

scheme

as

fU

C

Rg

scheme

If

only

unicast

communication

was

used

-LRB-

using

the

Umesh

algorithm

-RRB-

it

would

have

taken

five

steps

to

cover

destinations

in

this

example

The

BRCP

model

implements

it

faster

than

the

unicast-based

approach

More

interestingly

let

us

consider

all

nodes

along

rightmost

three

columns

of

the

mesh

in

Fig.

being

included

in

a

new

multicast

set

This

will

add

destinations

to

the

example

making

the

size

of

the

multicast

set

as

It

requires

six

steps

to

cover

all

nodes

in

this

new

set

using

unicast

messages

However

the

BRCP

model

can

still

achieve

this

multicast

with

four

steps

For

a

k

k

e-cube

routed

mesh

it

can

be

easily

observed

that

k

under

the

BRCP

model

with

R

and

C

paths

This

leads

to

Lemma

The

upper

bound

in

number

of

communication

steps

to

implement

an

arbitrary

multicast

on

an

e-cube

routed

mesh

with

the

BRCP

model

using

R

and

C

paths

is

-LRB-

k

-RRB-

e

As

discussed

in

Lemma

the

HL

scheme

uses

a

profitable

grouping

approach

and

thus

always

performs

equal

to

or

better

than

the

Umesh

scheme

With

certain

spatial

distribution

of

the

multicast

destination

set

the

HL

scheme

with

allowable

BRCP

paths

may

not

perform

better

than

the

Umesh

scheme

For

example

if

a

multicast

set

consists

of

nodes

along

the

diagonal

of

a

e-cube

mesh

the

BRCP

model

can

not

group

these

destinations

with

R

or

C

paths

In

this

case

both

Umesh

and

BRCP

models

perform

equally

However

a

multidestination

worm

along

a

RC

path

will

be

able

cover

at

least

two

nodes

along

the

diagonal

Similarly

if

the

system

supports

planar-adaptive

routing

then

all

the

destinations

along

the

diagonal

can

be

covered

with

a

single

multidestination

worm

Even

with

simple

R

and

C

paths

on

an

e-cube

routed

system

as

the

degree

-LRB-

number

of

destination

nodes

-RRB-

of

a

multicast

increases

more

and

more

destinations

can

be

grouped

along

R

paths

Similarly

the

row-leaders

can

also

be

grouped

along

columns

leading

to

less

number

of

column-leaders

Table

shows

the

average

values

of

and

obtained

on

a

ecube-routed

mesh

using

R

and

C

paths

For

each

multicast

size

the

destinations

were

chosen

randomly

and

the

average

was

taken

over

runs

It

can

be

clearly

observed

from

the

table

that

as

the

number

of

destinations

in

a

multicast

increases

the

HL

grouping

becomes

more

and

more

effective

and

the

value

of

decreases

This

reduces

the

number

of

communication

steps

required

by

the

BRCP

model

to

perform

multicast

compared

to

the

Umesh

This

leads

to

Theorem

The

BRCP

model

always

implements

a

multicast

using

lesser

or

equal

number

of

communication

steps

than

the

Umesh

scheme

As

the

number

of

destinations

in

a

multicast

set

increases

beyond

certain

number

the

BRCP

model

allows

the

multicast

latency

to

get

reduced

with

fewer

number

of

steps

-LRB-

communication

startups

-RRB-

The

exact

number

beyond

which

the

BRCP

model

performs

better

than

the

Umesh

depends

on

the

topology

routing

the

allowable

BRCP

paths

and

the

spatial

distribution

of

the

destinations

The

above

interesting

result

demonstrated

by

the

BRCP

model

is

the

strong

contribution

of

this

paper

In

the

next

section

we

present

results

of

simulation

experiments

for

different

topology

system

size

routing

allowable

BRCP

paths

and

technological

parameters

These

results

consider

both

communication

startup

and

message

propagation

latency

and

validate

the

results

stated

in

Theorem

for

the

overall

multicast

latency

Table

Average

values

of

and

on

a

e-cube

routed

mesh

using

HL

grouping

with

R

and

C

paths

To

illustrate

how

the

the

number

of

communication

steps

decreases

with

with

increase

in

multicast

degree

the

number

of

communication

steps

for

the

HL

scheme

is

compared

with

the

Umesh

scheme

no

of

dests

64 128 192 256

320 384 448 512

576 640 704 768

HL

steps

Umesh

steps

Using

Row-Column

-LRB-

RC

-RRB-

Paths

with

Multi-level

Grouping

In

the

above

solution

the

destinations

are

grouped

along

valid

R

and

C

paths

in

a

bottom-up

manner

to

determine

the

respective

leaders

Here

we

present

an

alternative

top-down

approach

with

RC

paths

and

multi-level

grouping

to

determine

the

leaders

Consider

a

RC

path

covering

some

destinations

in

a

multicast

Now

a

destination

along

the

row

-LRB-

column

-RRB-

portion

of

this

path

after

receiving

the

message

can

cover

other

destinations

along

its

column

-LRB-

row

-RRB-

Thus

the

first

destination

along

this

RC

path

can

be

designated

as

a

level

leader

The

destinations

along

the

RC

path

which

can

cover

other

destinations

using

R/C

paths

can

be

designated

as

level

leaders

In

this

approach

a

RC

path

directly

leads

to

a

two-level

grouping

By

doing

a

systematic

top-down

search

for

suitable

RC

paths

all

level

and

level

leaders

can

be

easily

decided

for

a

given

set

of

destinations

The

basic

idea

is

to

search

for

RC

paths

with

non-zero

destinations

along

its

row

and

column

Figure

in

Appendix

C

outlines

the

associated

grouping

procedure

for

a

mesh

Once

the

grouping

is

done

the

level

leaders

can

be

covered

with

unicast

messages

using

the

Umesh

algorithm

Then

level

leaders

can

send

multidestination

messages

along

RC

paths

Finally

level

leaders

can

send

multidestination

messages

along

R/C

paths

to

cover

remaining

destinations

Figure

shows

how

the

multicast

destination

set

of

Fig.

can

be

covered

using

the

current

scheme

The

grouping

leads

to

two

level

leaders

and

it

takes

four

steps

to

finish

the

multicast

This

scheme

can

be

identified

as

fU

RC

-LRB-

R/C

-RRB-

g

scheme

For

this

example

the

RC-path-based

scheme

does

not

provide

any

benefit

compared

to

the

earlier

scheme

with

R

and

C

paths

However

as

shown

in

Table

it

shows

benefits

for

a

wide

range

of

destinations

For

a

given

number

of

destinations

though

the

number

of

level

leaders

is

higher

than

that

in

the

R/C

scheme

there

are

less

number

of

level

leaders

due

to

longer

RC

paths

This

reduces

the

overall

multicast

latency

Taking

Advantage

of

Adaptivity

As

discussed

in

Sec.

planar-adaptive

and

turn-model

allow

flexibility

of

additional

D

and

NMWF

paths

respectively

Since

these

added

paths

have

capability

to

cover

more

number

of

nodes

with

unicast

worm

-LRB-

step

-RRB-

unicast

worm

-LRB-

step

-RRB-

multidest

RC

worm

-LRB-

step

-RRB-

multidest

R/C

worms

-LRB-

step

-RRB-

multidesti

RC

worm

-LRB-

step

-RRB-

multidest

R/C

worms

-LRB-

step

-RRB-

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

source

destination

level

leader

level

leader

Figure

Multicasting

with

RC

paths

and

multi-level

grouping

in

a

mesh

for

the

example

destination

set

of

Fig.

In

-LRB-

c

-RRB-

where

two

worms

belonging

to

steps

and

seem

to

cover

a

single

destination

it

is

actually

covered

by

the

step

worm

In

cases

where

the

two

worms

belong

to

the

same

step

as

can

be

inferred

from

the

grouping

procedure

in

Fig.

the

column

worm

covers

the

destination

Table

Average

values

of

and

on

a

e-cube

routed

mesh

using

multi-level

grouping

with

RC

paths

The

number

of

communication

steps

required

for

multicasting

under

this

scheme

is

compared

against

the

HL

grouping

with

R

and

C

paths

no

of

dests

64 128 192 256

320 384 448 512

576 640 704 768

RC

steps

R/C

steps

a

single

multidestination

worm

there

is

potential

to

do

better

grouping

using

these

paths

This

indicates

that

for

a

given

set

of

destinations

in

a

multicast

the

HL

grouping

can

lead

to

either

a

smaller

m

-LRB-

maximum

number

of

levels

in

the

grouping

-RRB-

or

less

number

of

level

m

leaders

Lm

or

both

Figure

shows

the

effect

of

adaptivity

on

multicast

implementation

The

same

multicast

example

from

Fig.

is

considered

As

shown

in

Fig.

the

D

paths

-LRB-

along

x

and

y

directions

-RRB-

of

planar-adaptive

routing

reduce

the

set

of

destinations

into

five

level

leaders

Now

all

these

level

leaders

can

in

fact

be

grouped

together

along

another

D

path

-LRB-

x

and

y

directions

-RRB-

to

form

a

single

level

leader

Such

grouping

can

lead

to

a

two-step

multicast

scheme

fD

Dg

as

shown

in

the

figure

There

is

a

savings

of

two

steps

compared

to

the

implementation

on

e-cube

routed

system

using

R

and

C

paths

source

destination

level

leader

diagonal

path

worm

-LRB-

D

-LRB-

x

y

-RRB-

-RRB-

diagonal

path

worm

-LRB-

D

-LRB-

x

y

-RRB-

-RRB-

west-first

non-minimal

column

path

worm

-LRB-

C

-RRB-

path

worm

-LRB-

WFNM

-RRB-

west-first

non-minimal

path

worm

-LRB-

WFNM

-RRB-

Figure

Multicast

with

hierarchical

leader

-LRB-

HL

-RRB-

scheme

in

a

mesh

supporting

a

-RRB-

planar-adaptive

routing

b

-RRB-

turn

model

west-first

non-minimal

routing

with

a

two-step

approach

and

c

-RRB-

turn

model

with

one-step

approach

In

this

example

the

source

node

position

helps

in

covering

the

level

leader

as

well

as

all

level

leaders

in

a

single

step

Otherwise

the

source

node

needs

to

send

a

unicast

worm

to

level

leader

first

leading

to

a

three-step

fU

D

Dg

scheme

The

addition

of

D

paths

provides

a

lot

of

flexibility

in

grouping

Unlike

the

maximum

two

levels

of

grouping

possible

on

e-cube

system

with

R

and

C

paths

we

can

go

here

for

multiple

levels

of

grouping

as

long

as

they

remain

profitable

Using

our

function

F

introduced

earlier

to

define

grouping

sequences

we

can

have

many

grouping

choices

for

a

planar-adaptive

system

FD

-LRB-

FC

-LRB-

FR

-RRB-

-RRB-

FD

-LRB-

FD

-LRB-

FR

-RRB-

-RRB-

FD

-LRB-

FR

-RRB-

FD

-LRB-

FD

-RRB-

etc.

The

effectiveness

of

these

grouping

sequences

totally

depends

on

the

spatial

distribution

of

the

destinations

In

the

next

section

we

present

simulation

results

to

compare

the

effectiveness

of

several

such

grouping

sequences

Now

we

consider

the

same

multicast

example

on

a

system

with

turn-model

west-first

nonminimal

routing

As

shown

in

Fig.

the

destinations

can

be

first

grouped

along

C

paths

Now

all

level

leaders

-LRB-

column-leaders

-RRB-

can

be

covered

by

a

single

multidestination

worm

from

the

source

using

a

NMWF

path

This

two-step

multicast

scheme

can

be

identified

as

fNMWF

Cg

scheme

Figure

illustrates

an

one-step

fNMWFg

scheme

with

longer

path

length

The

relative

merits

between

these

two

schemes

will

be

illustrated

through

simulation

experiments

later

Again

it

can

be

observed

that

the

extra

adaptivity

provided

by

the

turn

model

helps

in

reducing

the

number

of

communication

steps

required

for

multicasting

This

leads

to

Theorem

An

arbitrary

multicast

on

a

system

supporting

turn-model

west-first

non-minimal

routing

can

be

implemented

using

one

or

two

communication

steps

under

the

BRCP

model

Multidestination

message

passing

under

the

BRCP

model

provides

flexibility

to

reduce

multicast

latency

on

an

adaptive

routed

system

compared

to

an

e-cube

routed

system

Such

advantages

are

not

possible

with

unicast

message

passing

using

the

Umesh

algorithm

Higher

Dimensional

Systems

The

HL

scheme

developed

for

meshes

in

the

previous

subsections

can

be

easily

extended

to

multidimensional

systems

For

an

n-dimensional

system

there

will

be

several

possibilities

in

grouping

depending

on

the

routing

scheme

supported

by

the

system

For

example

for

a

mesh

with

e-cube

routing

the

allowable

BRCP

paths

are

X

Y

Z

XY

YZ

XZ

and

XYZ

-LRB-

the

R

C

and

RC

paths

mentioned

earlier

for

a

system

are

equal

to

X

Y

and

XY

paths

respectively

-RRB-

For

a

system

with

planar-adaptive

routing

paths

like

Dxy

and

Dyz

are

possible

in

addition

to

e-cube

paths

For

a

given

system

and

routing

the

HL

scheme

can

be

applied

to

derive

the

best

grouping

using

the

allowable

BRCP

paths

The

concepts

of

profitability

and

multi-level

grouping

discussed

earlier

can

also

be

used

to

reduce

multicast

latency

In

order

to

keep

the

paper

focused

and

under

limited

length

we

do

not

elaborate

more

in

this

direction

In

the

following

section

we

present

some

simulation

results

for

e-cube

routed

system

for

completeness

Simulation

Experiments

and

Results

To

verify

the

effectiveness

of

our

new

BRCP

model

we

simulated

single-source

broadcasting

and

multicasting

on

a

CSIM-based

-LSB-

-RSB-

simulator

testbed

This

testbed

implements

flit-level

simulation

and

can

be

configured

for

different

topological

and

technological

parameters

We

assumed

system

parameters

representing

the

current

trend

in

technology

A

range

of

values

for

communication

start-up

time

-LRB-

ts

-RRB-

was

used

and

microsec

Link

propagation

time

-LRB-

tp

-RRB-

was

assumed

to

be

microsec

for

Mbytes/sec

link

For

unicast

message-passing

router

delay

-LRB-

tnode

-RRB-

was

assumed

as

nanoseconds

For

multidestination

message-passing

a

higher

value

of

router

delay

nanosec

was

considered

For

a

fair

comparison

consumption

channels

were

assumed

for

both

multidestination

and

unicast

message-passing

Each

experiment

was

carried

out

times

to

obtain

average

results

In

this

section

we

present

experimental

results

for

single-source

multicast

and

broadcast

in

the

absence

of

background

traffic

The

main

objective

behind

these

experiments

is

to

understand

the

complete

dynamics

of

the

BRCP

model

for

different

topological

and

technological

parameters

Interested

readers

are

requested

to

refer

to

-LSB-

-RSB-

for

a

more

detailed

study

of

multiple

multicasts

Broadcasting

Effect

of

System

Size

and

Adaptivity

Figure

compares

latency

of

broadcast

from

an

arbitrary

source

on

systems

with

various

sizes

It

also

illustrates

the

effect

of

adaptivity

It

can

be

observed

that

as

the

system

size

increases

both

Umesh

-LSB-

-RSB-

and

Hamiltonian-path-based

schemes

-LSB-

-RSB-

incur

more

latency

to

implement

broadcast

However

the

multidestination

based

schemes

keep

the

latency

almost

constant

across

different

system

sizes

The

HL

scheme

with

RC

paths

and

the

two-step

broadcasting

with

turn

model

demonstrate

identical

performance

For

a

system

these

two

schemes

demonstrate

a

performance

improvement

by

a

factor

of

compared

to

the

Umesh

algorithm

The

HL

scheme

with

R

and

C

paths

implement

broadcast

with

one

additional

start-up

compared

to

the

RC

path

scheme

and

the

two-step

turn

scheme

Since

the

one-step

turn

scheme

requires

only

one

communication

start-up

it

performs

best

for

systems

smaller

than

However

as

the

system

size

increases

the

propagation

latency

dominates

and

it

performs

worse

than

the

Hamiltonian-path-based

scheme

Effect

of

Communication

Start-up

Figure

shows

the

effect

of

varying

communication

start-up

time

-LRB-

ts

-RRB-

on

the

performance

of

broadcasting

schemes

on

a

mesh

The

broadcast

latency

increases

linearly

with

ts

for

Umesh

algorithm

However

multidestination

schemes

-LRB-

including

the

Hamiltonian

path-based

scheme

-RRB-

are

not

very

sensitive

to

ts

because

they

use

less

number

of

communication

start-ups

The

multides

200 400 600 800

Broadcast

Latency

in

microseconds

Mesh

size

-LRB-

k

x

k

-RRB-

Effect

of

System

Size

microsec

umesh

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

turn

-LRB-

step

-RRB-

turn

-LRB-

step

-RRB-

hampath

Multicast

Latency

in

microseconds

in

microseconds

Effect

of

Communication

Startup

Mesh

umesh

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

turn

-LRB-

step

-RRB-

turn

-LRB-

step

-RRB-

hampath

Figure

Analyzing

one-to-all

broadcasting

latency

a

-RRB-

Effect

of

system

size

and

adaptivity

and

b

-RRB-

Effect

of

communication

start-up

tination

schemes

provide

significant

advantage

in

implementing

faster

broadcast

for

systems

with

higher

ts

Even

for

systems

with

very

low

ts

value

-LRB-

microsec

-RRB-

the

HL

scheme

with

RC

paths

is

capable

of

implementing

broadcast

times

faster

than

the

Umesh

Multicasting

Overall

Comparison

for

Meshes

with

E-cube

Routing

Figure

compares

the

latency

of

single-source

multicast

for

varying

system

sizes

Four

different

schemes

were

considered

HL

scheme

using

-LRB-

R

C

-RRB-

and

-LRB-

R

C

RC

-RRB-

paths

Umesh

and

Hamiltonian

path-based

scheme

We

used

two

different

ts

values

and

microseconds

The

simulation

results

conform

to

the

fact

that

the

BRCP

model

is

capable

of

reducing

multicast

latency

as

the

number

of

destinations

participating

in

the

multicast

increases

beyond

a

certain

number

This

cutoff

number

was

observed

to

be

and

nodes

for

the

three

system

configurations

considered

For

large

number

of

destinations

the

HL

schemes

are

able

to

reduce

multicast

latency

by

a

factor

of

compared

to

the

Umesh

scheme

The

Hamiltonian-path-based

scheme

performs

best

for

smaller

system

size

higher

ts

and

less

number

of

destinations

per

multicast

However

the

HL

schemes

perform

better

than

the

Hamiltonian-path-based

scheme

as

the

system

size

increases

ts

reduces

and

the

number

of

destinations

per

multicast

increases

In

Section

it

was

demonstrated

that

RC

paths

are

effective

than

R

and

C

paths

For

systems

with

higher

ts

-LRB-

microsec

-RRB-

the

HL

scheme

with

RC

paths

was

observed

to

implement

multicast

faster

than

using

R

and

C

paths

However

for

systems

with

ts

microsec

the

scheme

using

R

and

C

paths

performed

better

This

is

because

a

multidestination

worm

along

a

RC

path

encounters

longer

propagation

delay

-LRB-

up

to

hops

in

a

kxk

system

-RRB-

compared

to

that

along

a

R/C

path

-LRB-

up

to

k

hops

-RRB-

The

propagation

delay

dominates

for

systems

with

lower

ts

Thus

the

HL

scheme

with

R

and

C

paths

is

better

for

systems

with

lower

ts

value

where

as

the

RC

path-based

scheme

is

better

for

other

systems

Multicast

Latency

in

microseconds

Number

of

Destinations

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

umesh

hampath

Multicast

Latency

in

microseconds

Number

of

Destionations

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

umesh

hampath

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

Destinations

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

umesh

hampath

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

Destinations

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

umesh

hampath

200 400 600 800

Multicast

Latency

in

microseconds

Number

of

Destinations

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

umesh

hampath

200 400 600 800

Multicast

Latency

in

microseconds

Number

of

nodes

participating

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

c

rc

-RRB-

umesh

hampath

Figure

Comparison

of

single-source

multicast

latency

on

three

different

meshes

with

two

different

communication

start-up

-LRB-

ts

-RRB-

values

Four

different

schemes

are

compared

hierarchical

leader-based

-LRB-

HL

-RRB-

scheme

with

R

and

C

paths

HL

scheme

with

R

C

and

RC

paths

Umesh

and

Hamiltonian

Path-based

scheme

Effect

of

Adaptivity

on

Meshes

We

studied

the

effect

of

adaptivity

on

reducing

multicast

latency

under

the

BRCP

model

We

considered

a

mesh

with

three

different

routing

schemes

e-cube

planar-adaptive

and

turn

model

Under

e-cube

routing

we

considered

the

HL

scheme

with

R

and

C

paths

For

planaradaptive

systems

we

considered

two

different

grouping

sequences

as

discussed

in

Sec.

The

-LRB-

R

D

-RRB-

grouping

sequence

first

determines

row

leaders

by

grouping

the

destinations

along

R

paths

and

then

determines

the

column

leaders

by

grouping

along

D

paths

Similarly

the

-LRB-

R

C

D

-RRB-

grouping

sequence

determines

the

final

set

of

column

leaders

by

applying

hierarchical

grouping

along

R

C

and

D

paths

successively

For

turn

model

both

one-step

and

two-step

schemes

were

considered

Figure

shows

the

performance

comparison

between

the

above

schemes

to

implement

arbitrary

multicast

on

and

systems

with

ts

microsec

It

can

be

observed

that

the

turn

model

with

two-step

multicasting

scheme

performs

the

best

For

a

wide

range

of

destinations

this

scheme

implements

multicast

with

a

near-constant

latency

Though

the

one-step

scheme

uses

only

one

communication

start-up

the

latency

increases

with

increased

number

of

destinations

due

to

longer

path

traversal

50 100 150 200

Multicast

Latency

in

microseconds

Number

of

nodes

participating

Effect

of

Adaptivity

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

d

-RRB-

-LRB-

r

c

d

-RRB-

turn

-LRB-

step

-RRB-

turn

-LRB-

step

-RRB-

200 400 600 800

Multicast

Latency

in

microseconds

Number

of

nodes

participating

Effect

of

Adaptivity

x

Mesh

microsec

-LRB-

r

c

-RRB-

-LRB-

r

d

-RRB-

-LRB-

r

c

d

-RRB-

turn

-LRB-

step

-RRB-

turn

-LRB-

step

-RRB-

Figure

Effect

of

the

BRCP

model

to

reduce

multicast

latency

in

meshes

with

increase

in

adaptivity

Both

-LRB-

R

D

-RRB-

and

-LRB-

R

C

D

-RRB-

groupings

were

observed

to

perform

better

than

the

-LRB-

R

C

-RRB-

grouping

for

multicast

with

small

number

of

destinations

The

-LRB-

R

D

-RRB-

grouping

was

slightly

better

than

the

-LRB-

R

C

D

-RRB-

grouping

For

large

number

of

destinations

the

-LRB-

R

C

-RRB-

grouping

performed

the

best

This

observation

can

be

explained

as

follows

The

row

leaders

determined

after

the

first

step

grouping

along

R

paths

are

distributed

among

few

left

columns

of

the

system

For

multicast

with

small

number

of

destinations

the

row

leaders

are

sparsely

distributed

among

these

columns

In

this

case

the

second

grouping

along

D

paths

becomes

beneficial

compared

to

that

along

C

paths

to

minimize

the

number

of

column

leaders

Thus

-LRB-

R

D

-RRB-

grouping

performs

better

than

-LRB-

R

C

-RRB-

grouping

However

if

the

grouping

along

D

paths

is

done

as

the

third

step

-LRB-

i.e.

after

-LRB-

R

C

-RRB-

grouping

-RRB-

then

the

flexibility

of

D

paths

to

reduce

the

number

of

column

leaders

reduces

Thus

-LRB-

R

C

D

-RRB-

grouping

does

not

perform

as

good

as

-LRB-

R

D

-RRB-

grouping

For

multicast

with

larger

number

of

destinations

the

row

leaders

get

distributed

among

very

few

left

columns

In

this

case

grouping

along

C

paths

is

much

more

effective

Thus

for

a

planar-adaptive

system

grouping

along

D

paths

can

be

only

used

for

multicast

with

small

number

of

destinations

Comparison

for

Meshes

Figure

shows

how

the

HL

scheme

with

X

Y

and

Z

paths

as

discussed

in

Sec.

performs

in

meshes

compared

to

other

schemes

Two

different

system

configurations

and

a

very

small

ts

value

-LRB-

microsec

-RRB-

were

assumed

Here

too

a

similar

trend

can

be

observed

that

the

HL

scheme

performs

better

as

the

number

of

destinations

increases

The

HL

scheme

reduces

multicast

latency

by

factors

of

and

compared

to

the

Umesh

and

the

Hamiltonian-path

schemes

respectively

100 200 300 400

Multicast

Latency

in

microseconds

Number

of

nodes

participating

x

x

Mesh

microsec

-LRB-

r

c

-RRB-

umesh

hampath

200 400 600 800

Multicast

Latency

in

microseconds

Number

of

nodes

participating

x

x

Mesh

microsec

-LRB-

r

c

-RRB-

umesh

hampath

Figure

Comparing

multicast

latency

in

meshes

with

HL

Umesh

and

Hamiltonian-path-based

schemes

Scalability

with

Increasing

System

Size

Finally

we

studied

scalability

issues

of

our

multicast

schemes

as

the

system

size

increases

and

compared

them

with

other

schemes

like

Umesh

and

Hamiltonian

path-based

Figure

shows

the

comparison

results

for

three

different

system

sizes

and

Results

for

both

high

-LRB-

-RRB-

and

medium

-LRB-

-RRB-

degree

of

multicasting

are

illustrated

A

small

value

of

ts

-LRB-

microsec

-RRB-

was

assumed

It

can

easily

be

observed

that

the

HL

schemes

are

much

more

scalable

than

Umesh

and

Hamiltonian

path-based

schemes

When

the

system

size

increases

by

a

factor

of

-LRB-

from

to

-RRB-

the

multicast

latency

to

cover

nodes

of

the

system

increases

only

by

a

factor

of

using

HL

-LRB-

R

C

-RRB-

scheme

In

comparison

Umesh

increases

the

latency

by

a

factor

of

and

Hamiltonian

path-based

scheme

by

a

factor

of

As

the

system

size

increases

the

HL

scheme

with

RC

paths

suffers

a

little

due

to

increased

path

length

However

it

is

quite

scalable

Similar

trends

can

also

be

observed

for

multicast

covering

nodes

These

results

clearly

show

the

potential

of

the

BRCP

model

to

implement

scalable

multicast

in

large

systems

200 400 600 800

Multicast

Latency

in

microseconds

Mesh

size

-LRB-

k

x

k

-RRB-

Scalability

of

Multicast

Schemes

Nodes

Participating

hl

-LRB-

r

c

-RRB-

hl

-LRB-

r

c

rc

-RRB-

umesh

hampath

200 400 600 800

Multicast

Latency

in

microseconds

Mesh

size

-LRB-

k

x

k

-RRB-

Scalability

of

Multicast

Schemes

Nodes

Participating

hl

-LRB-

r

c

-RRB-

hl

-LRB-

r

c

rc

-RRB-

umesh

hampath

Figure

Comparing

multicast

latency

of

different

schemes

with

increase

in

system

size

Related

Research

Many

recent

researchers

have

focused

on

multidestination

message

passing

to

support

efficient

collective

communication

Broadcasting

on

hypercube

networks

with

e-cube

routing

has

been

discussed

in

-LSB-

-RSB-

Multicasting

using

column-oriented

paths

on

mesh

with

Hamiltonian-routing

has

been

proposed

in

-LSB-

-RSB-

The

focus

on

these

papers

has

been

purely

towards

the

development

of

optimal

algorithms

In

this

paper

we

introduce

the

BRCP

model

in

a

general

manner

for

any

k-ary

n-cube

network

with

arbitrary

routing

scheme

It

focuses

on

both

architectural

and

algorithmic

aspects

This

paper

introduces

the

BRCP

model

and

studies

its

impact

on

implementing

single-source

broadcast

and

multicast

operations

Application

of

this

scheme

for

multiple

multicast

has

been

discussed

in

-LSB-

-RSB-

The

multidestination

mechanism

has

been

extended

for

other

types

of

collective

communication

in

the

following

papers

barrier

synchronization

-LSB-

-RSB-

and

global

reduction

-LSB-

-RSB-

Multidestination-based

multicast

and

gather

operations

have

also

been

applied

and

evaluated

for

distributed

shared

memory

systems

in

-LSB-

-RSB-

Recently

multidestination

mechanism

has

also

been

studied

for

multistage

networks

Such

studies

include

multicasting

for

unidirectional

and

bidirectional

MINs

-LSB-

-RSB-

Conclusions

In

this

paper

we

have

introduced

a

new

BRCP

model

for

supporting

multidestination

message

passing

on

wormhole

k-ary

n-cube

networks

It

allows

multidestination

worms

to

get

propagated

along

paths

conforming

to

a

given

base

routing

scheme

The

principles

behind

creating

multidestination

worms

for

popular

routing

schemes

are

emphasized

It

is

also

shown

that

the

current

routers

can

support

the

BRCP

model

with

very

little

additional

logic

The

potential

of

this

model

to

implement

fast

and

scalable

collective

communication

operations

has

been

illustrated

by

developing

and

evaluating

algorithms

for

single-source

broadcasting

and

multicasting

It

is

shown

that

these

operations

can

be

implemented

significantly

faster

under

the

BRCP

model

compared

to

the

Umesh

and

the

Hamiltonian-path-based

schemes

The

paper

demonstrates

the

following

two

unique

and

interesting

results

a

-RRB-

the

multicast

latency

can

in

fact

be

reduced

or

kept

near-constant

as

the

number

of

processors

participating

in

the

multicast

increases

beyond

certain

number

and

b

-RRB-

the

multicast

latency

can

be

reduced

by

taking

advantage

of

routing

adaptivity

Thus

multidestination

message

passing

with

the

BRCP

model

demonstrates

significant

potential

to

be

incorporated

into

future

wormhole

systems

to

implement

fast

and

scalable

collective

communication

operations

Acknowledgments

The

authors

would

like

to

thank

members

of

the

PAC

research

group

for

providing

comments

criticisms

and

suggestions

to

this

work

Special

thanks

go

to

Professors

Jos?e

Duato

Rajendra

Boppana

and

Suresh

Chalasani

for

their

valuable

feedback

on

several

aspects

of

the

BRCP

model

Additional

Information

A

number

of

related

papers

and

technical

reports

are

available

electronically

through

the

home

page

of

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

The

URL

is

References

-LSB-

-RSB-

S.

Balakrishnan

and

D.

K.

Panda

Impact

of

Multiple

Consumption

Channels

on

Wormhole

Routed

k-ary

n-cube

Networks

In

Proceedings

of

the

International

Parallel

Processing

Symposium

pages

-LCB-

April

-LSB-

-RSB-

A.

Bar-Noy

and

S.

Kipnis

Multiple

Message

Broadcasting

in

the

Postal

Model

In

International

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

M.

Barnett

et

al.

Interprocessor

Collective

Communication

Library

-LRB-

InterCom

-RRB-

In

Proceedings

of

the

Scalable

High

Performance

Computing

Conference

pages

-LCB-

-LSB-

-RSB-

M.

Barnett

D.

G.

Payne

and

R.

Van

de

Geijn

Optimal

Broadcasting

in

Mesh-Connected

Architectures

Technical

Report

Dept.

of

Computer

Science

Universityof

Texas

at

Austin

Dec

-LSB-

-RSB-

R.

V.

Boppana

S.

Chalasani

and

C.

S.

Raghavendra

On

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

Y.

M.

Boura

and

C.

R.

Das

Modeling

Virtual

Channel

Flow

Control

in

Hypercubes

In

International

Symposium

on

High

Performance

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

J.

Bruck

R.

Cypher

and

C.-T

Ho

Multiple

Message

Broadcasting

with

Generalized

Fibonacci

Trees

In

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

C.

M.

Chiang

and

L.

M.

Ni

Multi-Address

Encoding

for

Multicast

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

May

-LSB-

-RSB-

C.

M.

Chiang

and

L.

M.

Ni

Deadlock-Free

Multi-Head

Wormhole

Routing

In

Proceedings

of

the

First

High

Performance

Computing-Asia

-LSB-

-RSB-

A.

A.

Chien

and

J.

H.

Kim

Planar-Adaptive

Routing

Low-Cost

Adaptive

Networks

for

Multiprocessors

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

Dai

and

D.

K.

Panda

Reducing

Cache

Invalidation

Overheads

in

Wormhole

DSMs

Using

Multidestination

Message

Passing

In

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

Deadlock-Free

Message

Routing

in

Multiprocessor

Interconnection

Networks

IEEE

Transactions

on

Computers

pages

-LCB-

May

-LSB-

-RSB-

J.

Duato

A

New

Theory

of

Deadlock-Free

Adaptive

Routing

in

Wormhole

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

J.

Duato

S.

Yalamanchili

and

L.

Ni

Interconnection

Networks

An

Engineering

Approach

The

IEEE

Press

-LSB-

-RSB-

E.

Fleury

and

P.

Fraigniaud

Multicasting

in

Meshes

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

C.

J.

Glass

and

L.

Ni

The

Turn

Model

for

Adaptive

Routing

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

C.-T

Ho

and

M.-Y

Kao

Optimal

Broadcast

on

Hypercubes

with

Wormhole

and

E-cube

Routing

In

Proceedings

of

the

Int

l

Conference

on

Parallel

and

Distributed

Systems

pages

-LCB-

-LSB-

-RSB-

S.

L.

Johnsson

and

C.-T

Ho

Optimum

Broadcasting

and

Personalized

Communication

in

Hypercubes

IEEE

Transactions

on

Computers

pages

-LCB-

September

-LSB-

-RSB-

R.

Kesavan

and

D.

K.

Panda

Minimizing

Node

Contention

in

Multiple

Multicast

on

Wormhole

k-ary

n-cube

Networks

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

X.

Lin

P.

K.

McKinley

and

L.

M.

Ni

Performance

Evaluation

of

Multicast

Wormhole

Routing

in

Multicomputers

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Dec

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

D.

K.

Panda

Fast

Barrier

Synchronization

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Worms

In

International

Symposium

on

High

Performance

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

Global

Reduction

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Exchange

Worms

In

International

Parallel

Processing

Symposium

pages

-LCB-

Apr

-LSB-

-RSB-

D.

K.

Panda

Issues

in

Designing

Efficient

and

Practical

Algorithms

for

Collective

Communication

in

Wormhole-Routed

Systems

In

ICPP

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

-LSB-

-RSB-

H.D.

Schwetman

Using

CSIM

to

Model

Complex

Systems

In

Proceedings

of

the

Winter

Simulation

Conference

pages

-LCB-

-LSB-

-RSB-

R.

Sivaram

D.

K.

Panda

and

C.

B.

Stunkel

Efficient

Broadcast

and

Multicast

on

Multistage

Interconnection

Networks

using

Multiport

Encoding

In

Proceedings

of

the

IEEE

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

Oct

Available

from

-LSB-

-RSB-

C.

B.

Stunkel

R.

Sivaram

and

D.

K.

Panda

Implementing

Multidestination

Worms

in

SwitchBased

Parallel

Systems

Architectural

Alternatives

and

their

Impact

In

Proceedings

of

the

ACM

Annual

International

Symposium

on

Computer

Architecture

-LRB-

-RRB-

June

Available

from

-LSB-

-RSB-

J.

H.

Upadhyay

V.

Varavithya

and

P.

Mohapatra

Efficient

and

Balanced

Adaptive

Routing

in

Two-Dimensional

Meshes

In

International

Symposium

on

High

Performance

Computer

Architecture

pages

-LCB-

Appendix

A.

Deadlock-free

Multidestination

Message-Passing

in

Planar-Adaptive

Systems

For

a

pair

of

source-destination

having

the

same

di

address

within

an

adaptive

plane

Ai

the

planar-adaptive

routing

-LSB-

-RSB-

does

not

clearly

specify

on

which

network

this

unicast

worm

needs

to

get

routed

For

example

such

a

worm

may

take

either

virtual

channel

di

corresponding

to

increasing

network

or

channel

di

corresponding

to

decreasing

network

In

spite

of

such

ambiguity

with

a

closer

look

it

can

be

observed

that

the

routing

algorithm

does

not

lead

to

any

deadlock

for

unicast

message

passing

A

multidestination

worm

gets

routed

in

a

piece-wise

manner

from

source

to

the

first

destination

as

well

as

between

consecutive

destinations

For

a

multidestination

worm

it

is

possible

that

its

source

-LRB-

or

the

first

node

where

the

worm

enters

adaptive

plane

Ai

-RRB-

and

the

next

destination

have

same

offset

along

dimension

di

Under

such

circumstances

if

the

multidestination

worm

selects

virtual

channel

di

or

di

in

an

arbitrary

manner

it

may

lead

to

deadlock

with

other

multidestination

worms

Figure

shows

an

example

of

such

a

deadlock

on

a

plane

Based

on

their

first

destinations

the

worms

and

can

be

categorized

as

increasing

and

decreasing

worms

respectively

However

the

first

destinations

of

worms

and

do

not

identify

them

as

increasing

or

decreasing

worms

Based

on

the

flexibility

of

channel

selection

discussed

above

let

us

assume

that

these

two

worms

select

virtual

channels

and

respectively

Now

worms

and

wait

for

virtual

channels

and

respectively

This

is

a

circular

wait

on

virtual

channels

and

it

leads

to

deadlock

Such

deadlock

could

have

been

avoided

if

the

worm

would

have

chosen

channel

Alternatively

the

worm

could

have

chosen

channel

also

This

leads

to

Theorem

To

support

deadlock-free

multidestination

message

passing

with

planar-adaptive

routing

worms

while

getting

routed

in

adaptive

plane

Ai

should

consistently

select

virtual

channel

di

-LRB-

or

di

-RRB-

when

the

source

-LRB-

or

the

first

node

where

the

worm

enters

adaptive

plane

Ai

-RRB-

have

same

offset

along

dimension

di

worm

propagation

A

C

D

F

G

E

B

H

I

K

L

M

N

O

P

Jd

d

d

Worm

W

From

F

to

J

K

Worm

W

From

I

to

K

C.

Worm

W

From

G

to

F

N

Worm

W

From

O

to

G

E

Figure

Example

of

a

deadlock

with

multidestination

worms

in

planar

adaptive

systems

using

the

existing

routing

algorithm

Such

deadlocks

can

be

avoided

by

removing

the

existing

ambiguity

in

channel

selection

policy

when

source

and

destination

in

adaptive

plane

Ai

have

same

offset

along

dimension

di

B.

Detailed

Proof

for

Theorem

-LRB-

Sec.

-RRB-

Proof

In

the

above

example

the

first

multidestination

message

after

reaching

the

farthest

node

along

x

and

y

directions

has

flexibility

to

go

along

x

y

z

and

z

directions

In

order

to

cover

all

nodes

along

a

plane

using

a

single

multidestination

worm

as

shown

in

the

figure

a

message

needs

flexibility

in

routing

along

positive

and

negative

directions

of

one

only

dimension

In

the

above

example

such

flexibility

exists

for

the

z

dimension

Thus

by

combining

this

dimension

with

routing

along

x

or

y

direction

it

is

possible

for

the

first

message

to

cover

all

nodes

along

the

xz

or

yz

plane

For

an

n-dimensional

system

all-but-one-negative-first

routing

allows

a

message

to

get

routed

first

along

negative

directions

of

all

dimensions

except

one

Let

this

dimension

be

w

After

the

first

multidestination

worm

reaches

the

farthest

node

by

traveling

maximum

hops

along

all

allowed

negative

directions

potentially

all

the

nodes

along

any

plane

consisting

of

the

dimension

w

can

be

covered

There

is

flexibility

in

selecting

the

other

dimension

of

this

plane

However

no

plane

with

three

or

more

dimensions

can

be

covered

without

violating

the

routing

restrictions

Thus

the

first

multidestination

worm

can

cover

at

best

all

the

nodes

on

a

plane

Now

single

dimensional

multidestination

worms

from

the

nodes

on

this

plane

can

be

used

to

cover

nodes

along

any

of

the

remaining

dimensions

With

-LRB-

-RRB-

dimensions

remaining

it

will

require

-LRB-

-RRB-

additional

steps

to

cover

all

nodes

Therefore

broadcast

can

be

implemented

in

-LRB-

-RRB-

steps

C.

Grouping

Procedure

for

Mesh

using

RC

Paths

-LRB-

Sec.

-RRB-

Procedure

Grouping

Input

source

s

k

k

mesh

and

a

set

of

destinations

D

rows

are

numbered

from

bottom

to

top

columns

are

from

left

to

right

Output

a

set

of

level

leaders

member

set

-LSB-

-LRB-

-RRB-

-LRB-

-RRB-

-RSB-

for

all

destinations

Group

-LRB-

-RRB-

f

row

column

OE

while

-LRB-

D

OE

-RRB-

do

f

while

-LRB-

no

of

destinations

along

R

path

-LRB-

row

-RRB-

OE

-RRB-

do

row

row

while

-LRB-

no

of

destinations

along

C

path

-LRB-

column

-RRB-

OE

-RRB-

do

column

j

destination

along

RC

path

-LRB-

row

column

-RRB-

with

minimum

column

value

-LSB-

fjg

member

set

-LSB-

-LRB-

j

-RRB-

-RSB-

sorted

list

of

destinations

reachable

from

j

along

RC

path

-LRB-

row

column

-RRB-

D

D

member

set

-LSB-

-LRB-

j

-RRB-

-RSB-

for

each

destination

i

along

RC

path

-LRB-

row

column

-RRB-

f

if

ix

row

and

iy

column

f

S

sorted

list

of

destinations

in

D

reachable

from

i

along

a

column

path

C

member

set

-LSB-

-LRB-

i

-RRB-

-RSB-

S

D

D

S

g

if

ix

row

and

iy

column

f

S

sorted

list

of

destinations

in

D

reachable

from

i

along

a

row

path

R

member

set

-LSB-

-LRB-

i

-RRB-

-RSB-

S

D

D

S

g

g

row

row

column

g

Return

-LRB-

member

set

-LSB-

-LRB-

-RRB-

-LRB-

-RRB-

-RSB-

-RRB-

g

Figure

A

top-down

grouping

procedure

with

RC

paths

and

multi-level

grouping

in

a

mesh

broadcast
collective communication
interconnection networks
meshes
multicast
wormhole routing

deadlock-freedom
interprocessor communication
k-ary n-cubes