Designing

Scalable

Systems

with

two-level

k-ary

n-cube

Wormhole-routed

Interconnections

Debashis

Basak

and

Dhabaleswar

K.

Panda

Department

of

Computer

and

Information

Science

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

basak,panda@cis.ohio-state.edu

August

Abstract

Recent

advancements

in

VLSI

and

packaging

technologies

are

making

it

cost

effective

to

integrate

multiple

processing

units

into

a

chip

or

a

board

This

demonstrates

attractiveness

in

building

scalable

parallel

systems

using

clustered

configurations

while

exploiting

communication

locality

Variety

of

clustered

architectures

proposed

in

the

past

using

buses

or

MINs

as

the

inter-cluster

interconnection

do

not

satisfy

both

the

above

objectives

This

paper

focuses

on

design

issues

in

building

two-level

clustered

systems

with

k-ary

n-cube

interconnections

at

both

the

intra-cluster

and

the

intercluster

levels

A

new

framework

of

composite-routing

is

proposed

to

route

messages

in

such

clustered

systems

in

a

deadlock-free

manner

The

interplay

between

various

parameters

such

as

cluster

size

inter

and

intra-cluster

topologies

channel

widths

routing

schemes

message

length

and

locality

of

communication

in

determining

system

performance

and

cost

have

been

analyzed

under

the

constant

bisection

bandwidth

constraint

Network

latency

in

such

a

system

is

analytically

modeled

by

taking

into

account

demand

multiplexing

of

multiple

virtual

channels

on

each

physical

channel

Predictions

from

this

model

together

with

channel

capacity

constraints

and

simulation

experiments

are

used

in

determining

optimal

configurations

for

a

given

system

size

Our

analysis

indicates

that

small

sized

clusters

with

a

ring

intra-cluster

topology

and

a

inter-cluster

network

connecting

these

clusters

offer

best

system

performance

This

provides

guidelines

to

system

designers

for

building

large-scale

parallel

systems

research

is

supported

in

part

by

the

National

Science

Foundation

Grant

and

by

the

Ohio-State

University

Seed

Grant

Contents

Introduction

Two-level

k-ary

n-cube

configurations

Definition

Cluster

Interface

Topological

Alternatives

Maintaining

Constant

Bisection

Bandwidth

Cost

Analysis

Routing

Alternatives

Absorb

Retransmit

-LRB-

AB

-RRB-

Strategy

No

Absorb

at

Interface

-LRB-

NAB

-RRB-

Strategy

Deadlock-free

Routing

Suitable

routing

algorithm

on

the

added

virtual

channel

network

Performance

Analysis

Bandwidth

and

capacity

constraints

A

Contention

Model

For

-LRB-

k

n

-RRB-

systems

Estimating

inter-cluster

message

latency

Estimating

intra-cluster

message

latency

Simulation

Experiments

and

Results

Effect

of

Cluster

Size

Effect

of

Topology

Intranet

or

Cluster

Topology

Internet

topology

Effect

of

Locality

Conclusions

Introduction

Traditionally

research

in

interconnection

networks

have

assumed

one

processing

element

per

node

However

advancements

in

VLSI

and

packaging

technologies

are

making

it

cost-effective

to

integrate

multiple

processing

elements

into

a

chip

multiple

chips

into

a

board

and

multiple

boards

into

a

system

-LSB-

-RSB-

Any

of

these

building

blocks

can

represent

a

processor

cluster

Such

clusters

can

be

interconnected

together

to

build

large-scale

systems

A

variety

of

hierarchical

configurations

have

been

proposed

by

researchers

in

the

last

decade

to

build

scalable

systems

using

either

single

processor

or

processor

cluster

per

node

Some

examples

include

cluster

of

processors

with

buses

-LSB-

-RSB-

cluster

of

processors

with

crossbar

switches

-LSB-

-RSB-

combination

of

crossbar

and

omega

networks

-LSB-

-RSB-

local

and

global

meshes

-LSB-

-RSB-

two-level

systems

based

on

hypercube

and

other

network

topologies

-LSB-

-RSB-

and

combination

of

intra-cluster

bus

and

inter-cluster

mesh/hypercube

networks

-LSB-

-RSB-

Two

desired

features

in

parallel

architectures

are

scalability

of

the

system

and

its

ability

to

exploit

the

locality

of

communication

inherent

in

most

applications

Most

of

the

above

architectures

using

buses

or

MINs

as

the

inter-cluster

interconnection

do

not

satisfy

both

the

above

objectives

In

the

architectures

supporting

scalable

inter-cluster

interconnections

the

focus

has

been

restricted

to

meshes

or

hypercubes

as

the

intercluster

topology

with

packet

switched

or

circuit

switched

routing

The

analysis

in

-LSB-

-RSB-

dealt

with

fixed

cluster

sizes

and

did

not

consider

the

optimality

of

the

network

configuration

in

terms

of

inter-cluster

and

intra-cluster

network

sizes

and

topologies

Research

by

Dally

-LSB-

-RSB-

has

demonstrated

the

attractiveness

of

k-ary

n-cube

topology

with

wormhole

routing

switching

technique

It

has

been

shown

that

low

dimensional

-LRB-

n

-RRB-

k-ary

n-cube

networks

have

potential

to

build

scalable

systems

with

low

average

message

latency

under

physical

wiring

constraint

In

-LSB-

-RSB-

we

have

proposed

a

general

class

of

k-ary

n-cube

cluster-c

scalable

architectures

by

combining

the

scalability

of

k-ary

n-cube

wormhole-routed

networks

with

the

cost-effectiveness

of

processor

cluster

designs

Each

cluster

consists

of

c

processors

and

the

possible

intra-cluster

interconnections

can

be

either

k-ary

n-cube

direct

network

or

indirect

networks

like

MIN

or

bus-based

Such

clustered

organizations

using

wormhole-routing

as

the

switching

technique

are

becoming

the

trend

for

building

scalable

parallel

systems

Example

commercial

systems

are

Intel

Paragon

-LSB-

-RSB-

Stanford

DASH

-LSB-

-RSB-

and

the

KSR

system

-LSB-

-RSB-

In

the

Paragon

system

a

cluster

consists

of

processors

connected

by

bus

and

the

clusters

are

connected

through

a

mesh

The

KSR

system

uses

a

ring-ring

topology

The

Stanford

DASH

system

interconnects

processor

clusters

through

a

mesh

network

Though

machines

of

this

type

are

being

built

there

is

still

lack

of

understanding

and

analysis

as

to

how

such

clustered

systems

can

be

designed

with

optimal

configurations

under

different

technological

and

packaging

constraints

The

interplay

between

routing

strategies

inter-cluster

topology

intra-cluster

topologies

and

communication

locality

in

determining

the

system

performance

is

also

not

known

In

this

paper

we

focus

on

two-level

clustered

systems

with

k-ary

n-cube

interconnections

at

both

intra-cluster

and

inter-cluster

levels

For

a

given

system

size

different

topolog

ical

alternatives

and

their

associated

cost

under

constant

bisection

bandwidth

constraint

-LSB-

-RSB-

are

analyzed

We

propose

two

minimal

deadlock-free

wormhole

routing

schemes

to

route

messages

in

such

two-level

k-ary

n-cube

systems

We

analyze

the

interplay

between

various

parameters

such

as

cluster

size

inter

and

intra-cluster

topologies

channel

widths

routing

schemes

message

length

and

locality

of

communication

in

determining

system

performance

and

cost

A

throughput-latency

analytical

model

for

such

systems

is

developed

for

one

of

the

routing

schemes

by

taking

into

account

the

demand

multiplexing

of

virtual

channels

on

each

physical

channel

Predictions

from

this

model

together

with

channel

capacity

constraints

and

simulation

experiments

are

used

in

determining

optimal

system

configurations

The

results

provide

guidelines

to

system

designers

about

how

to

build

large-scale

parallel

systems

using

two-level

k-ary

n-cube

wormhole

routed

interconnections

Section

formally

introduces

the

two-level

k-ary

n-cube

architecture

Section

presents

various

topological

alternatives

to

build

such

systems

under

the

constant

bisection

bandwidth

constraint

An

estimate

of

the

cost

of

each

choice

in

terms

of

router

and

link

complexity

is

also

provided

Section

proposes

two

routing

alternatives

The

advantages

disadvantages

and

deadlock

freedom

of

both

schemes

are

discussed

Section

models

the

system

performance

Section

presents

simulation

results

showing

the

interplay

between

the

various

system

parameters

Two-level

k-ary

n-cube

configurations

Consider

a

two-level

hierarchical

system

as

shown

in

Fig.

The

inter-cluster

network

is

the

higher

level

network

which

connects

multiple

clusters

Each

cluster

is

a

group

of

processors

interconnected

by

an

intra-cluster

network

and

connected

to

the

inter-cluster

network

through

a

cluster

interface

-LRB-

CI

-RRB-

Intercluster

Network

Intra-cluster

Network

Clusters

ClusterInterface

-LRB-

CI

-RRB-

Figure

A

two-level

clustered

system

Definition

In

our

hierarchical

system

both

the

inter-cluster

and

intra-cluster

networks

are

k-ary

n-cubes

Since

the

topologies

in

both

networks

can

be

different

we

identify

the

inter-cluster

network

as

a

k-ary

n-cube

and

the

intra-cluster

network

as

a

Thus

there

are

kn

clusters

and

processors

in

each

cluster

This

leads

to

a

total

of

N

processors

in

the

whole

system

From

now

on

we

will

refer

to

our

two

level

architecture

using

a

shorthand

-LRB-

k

n

-RRB-

We

will

also

refer

to

the

inter-cluster

network

as

internet

and

the

intra-cluster

network

as

intranet

Fig.

shows

two

examples

of

-LRB-

k

n

-RRB-

systems

The

first

example

is

a

-LRB-

-RRB-

configuration

with

processors

The

internet

and

intranet

are

ring

and

torus

topologies

respectively

We

identify

this

as

a

ring-torus

architecture

The

second

example

a

-LRB-

-RRB-

system

is

a

torus-ring

architecture

with

processors

It

is

possible

to

build

a

two-level

system

with

different

topologies

for

different

clusters

However

in

this

paper

we

consider

intranet

topologies

to

be

homogeneous

i.e.

same

and

for

all

clusters

-LRB-

-RRB-

A

network

-LRB-

k-ary

n-cube

-RRB-

Clusters

A

cluster

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Two

examples

of

-LRB-

k

n

-RRB-

architectures

with

processors

a

-RRB-

ring-torus

and

b

-RRB-

torus-ring

topologies

Cluster

Interface

In

our

system

messages

targeted

to

processors

within

the

cluster

are

routed

using

the

intra-cluster

network

Messages

destined

for

processors

in

other

clusters

are

routed

through

the

cluster

interface

Such

an

organization

is

shown

in

Fig.

If

a

message

needs

to

go

across

to

a

processor

in

another

cluster

it

gets

routed

out

of

the

source-cluster

through

the

source-CI

traverses

the

internet

to

reach

the

destination-CI

enters

the

destination

cluster

and

gets

routed

to

the

destination

processor

LM

LM

LM

P

P

P

LM

LM

LMP

P

P

LM

LM

LM

P

P

P

Inter-Cluster

Network

Cluster

Interface

Clusters

Intra-Cluster

Network

P

Processor

LM-Local

Memory

Router

-LRB-

CI

-RRB-

Figure

Cluster

interface

connecting

the

intra-cluster

networks

to

the

inter-cluster

network

from

associate

processor

to

associate

processor

input

channels

from

intranet

to

intranet

output

channels

input

channels

from

internet

output

channels

to

internet

-LRB-

-RRB-

x

-LRB-

-RRB-

cluster

interface

-LRB-

CI

-RRB-

router

switch

Figure

Organization

of

the

cluster

interface

router

switch

Let

us

look

at

the

organization

of

the

CI

in

more

detail

Figure

shows

the

router

switch

which

is

a

part

of

the

interface

As

shown

in

Fig.

one

of

the

intra-cluster

routers

is

augmented

to

form

a

larger

switch

in

order

to

support

the

inter-cluster

routing

The

cluster

processor

directly

connected

to

the

augmented

router

is

identified

as

associate

processor

of

the

CI

This

associate

processor

may

be

designated

to

carry

out

specialized

communication

tasks

like

message

combining

and

decombining

in

order

to

implement

dense

communication

patterns

like

all-to-all

efficiently

In

this

paper

we

do

not

focus

on

developing

communication

algorithms

on

such

systems

We

only

emphasize

on

the

architectural

aspects

The

associate

processor

is

allowed

one

input

-LRB-

injection

-RRB-

and

one

output

-LRB-

consumption

-RRB-

channel

from

the

switch

There

are

input

and

output

channels

-LRB-

called

intranet

channels

-RRB-

from

and

to

the

neighboring

processors

of

the

associate

processor

inside

the

cluster

The

neighbors

of

the

CI

in

the

internet

are

each

connected

to

one

input

and

one

output

channel

-LRB-

called

internet

channels

-RRB-

of

the

switch

Thus

the

switch

complexity

of

the

CI

router

is

-LRB-

-RRB-

x

-LRB-

-RRB-

The

figure

also

shows

the

possible

flow

of

messages

A

message

entering

on

an

intranet

channel

may

get

consumed

by

the

associate

processor

may

continue

on

an

outgoing

intranet

channel

or

may

leave

through

one

of

the

outgoing

internet

channels

for

going

to

a

remote

cluster

A

message

entering

on

an

internet

channel

also

has

similar

choices

Topological

Alternatives

In

this

section

we

consider

the

architectural

options

in

the

design

of

a

-LRB-

k

n

-RRB-

system

Different

possible

topologies

for

intranet

and

internet

are

analyzed

under

constant

bisection

bandwidth

constraint

We

also

develop

a

cost

measure

which

takes

into

account

the

number

of

routers

and

their

complexity

and

the

number

of

physical

links

required

This

measure

is

used

to

determine

the

cost

of

an

option

Maintaining

Constant

Bisection

Bandwidth

Given

N

processors

there

are

many

different

alternatives

to

form

a

hierarchical

configuration

For

example

with

N

we

can

have

clusters

of

processors

each

or

clusters

of

processors

each

Each

alternative

leads

to

a

different

clustered

architecture

with

different

properties

and

performance

capabilities

These

properties

depend

on

various

parameters

like

cluster

size

the

intranet

and

the

internet

topologies

and

the

routing

schemes

used

Our

objective

is

to

select

the

configuration

that

offers

best

system

performance

For

a

given

hierarchical

configuration

we

use

the

constant

bisection

bandwidth

constraint

-LSB-

-RSB-

to

determine

channel

widths

in

each

network

The

number

of

wires

that

need

to

cross

the

bisection

of

a

network

is

called

bisection

width

In

general

this

width

can

not

be

increased

arbitrarily

and

is

limited

by

factors

like

available

layout

area

-LSB-

-RSB-

allowable

system

size

cost

and

power

considerations

With

a

given

set

of

factors

the

bisection

width

can

be

held

constant

at

some

limit

This

limit

directly

affects

the

channel

width

and

indirectly

determines

the

number

of

flits

required

for

a

given

message

Consider

a

k-ary

n-cube

with

bidirectional

channels

each

of

width

W

bits

It

has

a

bisection

width

of

-LSB-

-RSB-

With

N

being

total

number

of

nodes

this

width

is

equal

to

k.

For

a

linear

array

of

processors

this

width

is

For

comparison

across

different

topologies

we

normalize

bisection

widths

to

that

of

a

n-cube

-LRB-

hypercube

-RRB-

with

unitwidth

bidirectional

channels

Thus

the

bisection

width

of

such

a

n-cube

becomes

equal

to

Consequently

the

channel

width

or

flit

size

W

of

a

k-ary

n-cube

is

derived

as

k

The

constant

bisection

width

constraint

imposed

within

a

cluster

comes

from

lower

packaging

levels

like

VLSI

chip

or

board-levels

restrictions

-LRB-

depending

on

how

a

cluster

is

defined

-RRB-

Similar

constraints

for

the

internet

are

imposed

by

more

higher

levels

of

packaging

like

board-level

or

cabinet-level

Since

the

constraints

imposed

on

the

internet

and

intranet

arise

from

different

levels

of

packaging

we

will

consider

them

independent

Thus

the

intranet

channel

width

is

determined

solely

by

the

intranet

size

and

topology

Similarly

the

internet

channel

width

W

is

determined

by

internet

size

and

topology

W

W

-LRB-

-RRB-

or

-LRB-

-RRB-

-LRB-

-RRB-

or

-LRB-

-RRB-

Figure

Two

possible

-LRB-

k

n

-RRB-

configurations

to

implement

N

processor

system

and

the

associated

channel

width

under

constant

bisection

bandwidth

constraint

intra-cluster

channel

width

and

W

inter-cluster

channel

width

Consider

a

system

with

N

processors

We

have

the

following

possible

alternatives

to

build

it

as

a

two-level

system

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

Figure

shows

two

of

the

choices

and

the

associated

channel

widths

This

great

amount

of

available

flexibility

makes

the

task

of

choosing

the

most

suitable

architecture

a

very

challenging

one

Physical

constraints

like

packaging

fixed

pin-out

board-size

etc.

do

help

to

narrow

down

the

choices

but

still

leaves

a

large

number

of

possible

configurations

to

choose

from

In

this

paper

we

consider

throughput

and

latency

metrics

to

guide

us

in

making

a

good

choice

Cost

Analysis

An

important

consideration

while

choosing

amongst

various

topological

designs

is

the

cost

An

ideal

choice

would

be

one

which

offers

good

performance

at

a

reasonable

price

The

cost

of

an

architecture

depends

on

several

factors

including

number

of

processors

cost

of

a

processor

number

of

links

number

of

virtual

channels

supported

on

a

physical

channel

number

of

routers

and

router

complexity

etc.

To

compare

the

cost

of

various

-LRB-

k

n

-RRB-

alternatives

possible

with

N

nodes

we

use

total

router

cost

and

number

of

physical

links

as

a

measure

of

system

cost

We

can

assume

the

other

factors

influencing

cost

remaining

fixed

once

system

size

N

is

given

Let

us

consider

the

cost

of

a

single

n

x

n

router

switch

as

units

and

the

cost

of

a

Table

Summary

of

Symbols

Used

for

k-ary

n-cube

or

-LRB-

k

n

-RRB-

Clustered

System

N

Total

number

of

processors

C

Number

of

processors

in

a

cluster

d

Average

distance

on

an

internet

dim

ss

-LRB-

-RRB-

Average

distance

on

an

intranet

dim

ss

-LRB-

-RRB-

D

Average

distance

between

two

nodes

in

the

internet

nd

Average

distance

between

two

nodes

in

the

intranet

W

Width

of

internet

channel

in

bytes

Width

of

intranet

channel

in

bytes

L

Message

length

in

bytes

p

Prob

of

a

generated

message

being

intranet

C/N

under

uniform

traffic

pattern

m

message

generation

rate

per

unit

time

per

node

M

effective

message

generation

rate

by

a

CI

mC

-LRB-

-RRB-

under

uniform

traffic

pattern

physical

link

as

unit

Under

this

model

the

cost

of

an

internet

router

switch

is

-LRB-

-RRB-

and

the

cost

of

an

intranet

router

switch

is

-LRB-

-RRB-

Since

there

are

kn

internet

routers

and

N?kn

intranet

routers

the

total

internet

router

cost

Costinter

router

kn

-LRB-

-RRB-

and

the

total

intranet

router

cost

Costintra

router

-LRB-

N

kn

-RRB-

-LRB-

-RRB-

The

total

router

cost

is

given

by

-LRB-

Costinter

router

Costintra

router

-RRB-

This

can

be

simplified

to

-LRB-

N

-LRB-

-RRB-

kn

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

This

shows

that

for

a

given

N

total

router

cost

goes

up

if

the

internet

size

is

increased

without

lowering

the

dimensionality

of

any

of

the

levels

It

also

increases

if

the

dimensionality

of

any

of

the

levels

is

increased

without

reducing

the

internet

size

We

also

determine

the

cost

of

internet

physical

links

Costinter

links

-LRB-

number

of

internet

channels

-RRB-

x

-LRB-

channel

width

-RRB-

Under

the

constant

bisection

width

assumption

W

k

and

Costinter

links

simplifies

to

knkn

This

implies

that

the

internet

link

cost

increases

with

internet

size

-LRB-

kn

-RRB-

It

also

increases

keeping

internet

size

fixed

if

the

dimensionality

of

the

internet

is

lowered

thus

increasing

the

radix

k

and

hence

the

product

kn

The

cost

of

all

intranet

physical

links

Costintra

links

is

given

by

kn

N

Under

the

constant

bisection

width

constraint

Hence

Costintra

links

simplifies

to

-LRB-

N

-RRB-

Thus

the

intranet

link

cost

is

minimized

by

having

smaller

cluster

size

-LRB-

which

makes

small

-RRB-

or

large

cluster

size

with

high

dimensionality

-LRB-

which

makes

the

radix

small

hence

the

product

is

small

-RRB-

Let

us

define

the

total

cost

of

the

system

to

be

the

sum

of

all

router

and

link

costs

Table

shows

the

cost

components

for

various

-LRB-

k

n

-RRB-

configurations

with

N

It

can

be

noted

that

the

following

configurations

are

the

less

expensive

alternatives

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

with

total

cost

of

each

being

less

than

units

There

exist

other

choices

like

-LRB-

-RRB-

which

has

a

total

cost

as

high

as

If

one

of

the

less

expensive

configurations

also

provides

reasonable

system

performance

then

it

would

become

an

attractive

choice

In

our

simulation

studies

we

found

that

-LRB-

-RRB-

topology

also

offers

good

system

performance

and

hence

becomes

an

excellent

candidate

topology

to

build

a

two-level

clustered

system

with

processors

Table

Router

and

Link

Costs

for

Various

Topologies

to

Implement

a

Two-level

system

with

Processors

Topology

Total

Total

Total

Total

Total

Total

Total

-LRB-

k

n

-RRB-

internet

intranet

internet

intranet

router

link

cost

router

router

link

link

cost

cost

cost

cost

cost

cost

-LRB-

-RRB-

3136 1728 1024 1024

4864 2048 6912

-LRB-

-RRB-

1024 2048 2816 3072

-LRB-

-RRB-

400 2160 256 4096

-LRB-

-RRB-

784 2160 128 4096

2944 4224 7168

-LRB-

-RRB-

16 3072 43744

-LRB-

-RRB-

1125 1500 500 11250

-LRB-

-RRB-

Routing

Alternatives

The

problem

of

routing

a

message

using

wormhole-routing

switching

technique

in

a

hierarchical

architecture

is

discussed

in

this

section

We

present

two

strategies

to

transfer

a

message

in

a

-LRB-

k

n

-RRB-

system

at

the

cluster

interfaces

The

associated

problems

and

advantages

of

each

strategy

are

analyzed

We

propose

a

general

scheme

where

routing

restrictions

in

intranet

and

internet

can

be

different

yet

the

composite

routing

for

the

two-level

system

is

deadlock

free

Absorb

Retransmit

-LRB-

AB

-RRB-

Strategy

This

strategy

uses

a

message

forwarding

mechanism

at

the

cluster

interfaces

An

intercluster

message

starting

from

a

source

processor

travels

to

the

source-CI

using

intra-cluster

routing

restrictions

At

the

source-CI

the

message

is

fully

absorbed

and

then

retransmitted

into

the

internet

The

message

now

travels

from

the

source-CI

to

the

destination-CI

using

inter-cluster

routing

strategy

At

the

destination-CI

the

message

is

again

fully

absorbed

before

being

retransmitted

into

the

destination

cluster

The

message

travels

towards

the

destination

node

using

intra-cluster

routing

restrictions

The

advantage

of

this

strategy

is

that

it

facilitates

advanced

operations

like

message

combining

at

the

interfaces

This

results

in

more

efficient

communication

in

case

of

localized

traffic

patterns

like

all-to-one

Another

advantage

of

this

technique

is

that

it

avoids

loss

in

system

bandwidth

due

to

mismatch

between

the

internet

and

intranet

flit

sizes

-LRB-

channel

widths

-RRB-

This

aspect

is

discussed

in

the

following

section

A

disadvantage

of

this

strategy

is

that

since

all

inter-cluster

messages

have

to

be

absorbed

and

retransmitted

at

the

CI

associate

processor

the

injection

and

consumption

channels

in

this

processor

becomes

a

serious

bottleneck

This

leads

to

serialization

of

all

traffic

going

out/coming

into

the

cluster

This

problem

can

be

solved

by

providing

more

than

one

injection

and

consumption

channels

at

this

processor

-LSB-

-RSB-

Another

disadvantage

with

this

technique

is

that

messages

incur

longer

latencies

because

they

need

to

be

absorbed

fully

at

two

intermediate

places

For

longer

messages

the

latency

becomes

more

critical

This

leads

us

to

the

following

alternative

strategy

No

Absorb

at

Interface

-LRB-

NAB

-RRB-

Strategy

In

this

strategy

messages

are

not

absorbed

at

interfaces

Once

a

message

-LRB-

worm

-RRB-

reaches

the

source-CI

it

continues

moving

into

the

internet

Again

when

the

worm

reaches

the

destination-CI

it

continues

moving

into

the

cluster

towards

the

destination

The

main

advantage

of

this

technique

is

that

it

leads

to

lower

message

latencies

An

important

observation

to

be

made

is

that

the

rate

of

message

travel

in

this

scheme

is

determined

by

the

slowest

of

the

two

networks

-LRB-

the

speed

is

determined

by

the

slowest

stage

of

the

pipelined

worm

movement

-RRB-

When

channel

cycle

times

are

similar

in

both

levels

the

network

with

smaller

flit

size

determines

the

speed

of

the

message

We

consider

W

W

for

At

the

cluster

interface

there

are

two

ways

to

handle

the

mismatch

in

flit

sizes

A

smaller

flit

arriving

at

the

interface

from

a

cluster

is

immediately

pumped

into

a

wider

internet

channel

thus

wasting

W

bits

in

every

cycle

Consequently

in

all

channel

transfers

in

the

internet

we

do

not

utilize

W

bits

in

every

cycle

This

leads

to

a

wastage

of

W

of

the

internet

bandwidth

The

arriving

small

flits

from

intranet

are

not

dispatched

immediately

into

the

internet

A

bigger

W

sized

flit

is

formed

by

concatenating

W

of

the

small

flits

at

the

cluster

interface

and

then

the

concatenated

wider

flit

is

released

into

the

internet

However

now

the

injection

of

these

wider

flits

can

only

take

place

at

W

cycles

Thus

the

worm

gets

stretched

out

in

the

internet

On

any

physical

link

on

the

worm

's

path

in

the

internet

flits

can

only

appear

at

periodic

intervals

of

W

cycles

This

leads

to

an

identical

wastage

of

bandwidth

as

before

For

and

it

implies

we

do

not

utilize

as

much

as

and

of

the

internet

bandwidth

respectively

We

define

this

problem

as

flit

mismatch

problem

since

it

arises

because

of

mismatch

in

flit

sizes

in

the

two

networks

We

suggest

the

use

of

virtual

channels

with

demand

multiplexing

to

alleviate

this

flit

mismatch

problem

Virtual

channels

as

proposed

by

Dally

-LSB-

-RSB-

are

logical

channels

which

are

time-multiplexed

over

a

physical

link

Routing

is

done

over

the

virtual

channel

network

The

NAB

scheme

now

works

like

this

We

use

the

strategy

of

concatenating

larger

flits

from

smaller

flits

and

then

releasing

them

into

the

internet

We

saw

that

on

any

physical

link

on

the

worm

's

path

in

the

internet

flits

can

only

appear

at

periodic

intervals

of

W

cycles

By

allowing

at

least

W

virtual

channels/lanes

to

be

demand

multiplexed

on

a

physical

channel

we

avoid

the

wastage

of

the

physical

channel

bandwidth

The

reason

being

that

during

the

period

a

message

on

a

virtual

channel

is

unable

to

transfer

a

flit

some

other

message

on

a

different

virtual

channel

is

able

to

utilize

the

multiplexed

physical

link

In

sec.

we

will

show

through

simulation

experiments

that

increasing

the

number

of

virtual

channels

does

actually

increase

the

system

throughput

by

preventing

the

wastage

of

bandwidth

Thus

the

NAB

strategy

with

requisite

virtual

channels

is

an

efficient

routing

scheme

for

clustered

systems

as

it

offers

more

throughput

at

low

latency

For

W

i.e.

intranet

channel

width

is

greater

than

internet

channel

width

we

can

use

an

analogous

strategy

of

adding

virtual

channels

in

the

intranet

to

alleviate

the

mismatch

problem

Both

AB

and

NAB

are

general

strategies

in

the

sense

that

they

do

not

specify

the

actual

routing

algorithm

to

be

used

in

each

network

Numerous

deadlock

free

algorithms

exist

in

literature

for

minimal

wormhole-routing

with

virtual

channels

for

k-ary

n-cubes

e.g.

dimension-order

-LSB-

-RSB-

fully-adaptive

-LSB-

-RSB-

or

planar

-LSB-

-RSB-

etc.

However

no

such

algorithm

exists

for

-LRB-

k

n

-RRB-

systems

One

simple

solution

might

be

to

use

a

composite-routing

scheme

e.g.

use

dimension

order

in

the

k-ary

n-cube

internet

and

fully

adaptive

in

the

cube

cluster

intranets

For

the

AB

strategy

it

is

easy

to

see

that

any

such

composition

of

deadlock-free

routing

algorithms

is

guaranteed

to

be

also

deadlock-free

This

is

by

virtue

of

the

fact

that

messages

are

fully

absorbed

at

network

interfaces

thus

making

routing

decisions

in

one

network

independent

of

routing

decisions

in

other

networks

This

implies

that

each

network

be

it

the

internet

or

cluster

intranets

has

the

flexibility

to

choose

its

own

routing

algorithm

However

one

can

see

that

for

the

NAB

strategy

such

simple

composite-routing

schemes

need

not

be

deadlock-free

This

is

because

under

this

scheme

worms

can

simultaneously

occupy

channels

in

more

than

one

network

In

the

following

section

we

show

an

example

of

such

a

deadlock

problem

and

offer

a

solution

to

design

deadlock-free

routing

schemes

for

-LRB-

k

n

-RRB-

systems

Deadlock-free

Routing

Consider

a

-LRB-

-RRB-

system

using

dimension-order

minimal

routing

in

both

networks

as

shown

in

Fig.

The

two

clusters

are

basically

bidirectional

rings

Consider

a

situation

where

all

processors

except

processors

and

are

trying

to

send

messages

to

their

respective

hop

away

right

cyclic

neighbor

in

the

cluster

Processors

and

on

the

other

hand

are

trying

to

send

inter-cluster

messages

to

processors

and

respectively

There

are

two

virtual

channels

per

physical

channel

and

the

algorithm

dictates

that

a

message

can

travel

on

virtual

channel

in

a

dimension

iff

it

is/has

taking/taken

the

wrap-around

connection

in

that

dimension

Otherwise

virtual

channel

is

used

Though

this

dimension-order

routing

is

proven

to

be

deadlock-free

in

a

flat

-LRB-

non-clustered

-RRB-

system

it

is

not

so

in

a

-LRB-

k

n

-RRB-

system

Fig.

-LRB-

b

-RRB-

shows

a

deadlock

situation

with

the

specified

communication

pattern

The

reason

why

such

compositions

of

deadlock-free

routing

schemes

do

not

work

is

because

they

assume

that

when

a

worm

reaches

the

local

destination

-LRB-

some

CI

-RRB-

in

a

network

it

gets

absorbed

fully

However

in

a

clustered

system

using

NAB

scheme

the

worm

is

not

absorbed

but

released

into

the

next

network

where

it

may

get

blocked

This

violates

the

basic

assumption

Since

efficient

algorithms

already

exist

for

handling

routing

in

k-ary

ncube

networks

it

is

desirable

to

come

up

with

a

scheme

for

composite-routing

in

-LRB-

k

n

-RRB-

systems

that

use

these

algorithms

We

provide

a

solution

to

design

deadlock-free

routing

schemes

for

-LRB-

k

n

-RRB-

systems

requiring

just

one

extra

virtual

channel

for

each

intranet

physical

link

while

retaining

the

simplicity

and

flexibility

of

composite-routing

scheme

Consider

a

composite-routing

scheme

which

uses

a

routing

algorithm

Rintra

inside

each

cluster

and

Rinter

in

the

internet

Let

Rintra

require

v

virtual

channels

per

physical

link

We

use

Rintra

together

with

its

v

virtual

channels

to

route

only

intra-cluster

messages

It

is

to

be

noted

that

such

scheme

is

different

from

the

example

in

Fig.

where

Rintra

was

also

used

for

routing

inter-cluster

messages

For

routing

inter-cluster

messages

when

they

are

traveling

inside

the

cluster

we

use

a

separate

virtual

channel

network

as

follows

We

add

an

extra

virtual

channel

on

all

physical

links

inside

the

cluster

thus

requiring

v

virtual

channels

per

physical

link

in

total

These

set

of

extra

virtual

channels

form

a

virtual

network

say

S.

Any

minimal

routing

strategy

R

can

be

used

on

S

to

route

inter-cluster

messages

inside

the

cluster

We

identify

this

composite

scheme

as

modified-composite-routing

The

inter-cluster

messages

using

network

S

of

a

cluster

can

be

divided

into

exactly

two

classes

A

-RRB-

messages

coming

from

other

clusters

B

-RRB-

messages

going

out

of

the

cluster

to

other

clusters

Given

this

we

can

state

the

following

Lemma

The

set

of

channels

used

by

messages

of

class

A

and

class

B

in

the

virtual

channel

network

S

are

disjoint

Proof

by

contradiction

Assume

two

messages

ma

from

class

A

and

mb

from

class

B

share

a

channel

under

the

routing

strategy

R

It

can

be

shown

easily

that

the

path

used

by

Cluster

Cluster

A

-LRB-

-RRB-

system

with

clusters

of

processors

each

Each

of

the

clusters

are

bidirectional

rings

with

virtual

channels

on

each

physical

channel

A

mesage

uses

Assume

that

the

following

messages

each

of

greater

than

flits

get

initiated

at

same

time

intra-cluster

msgs

Cluster

Cluster

A

label

on

a

channel

depicts

the

source

id

of

the

message

occupying

the

channel

inter-cluster

msgs

-LRB-

a

-RRB-

-LRB-

b

-RRB-

virtual

channel

virtual

channel

virtual

channel

if

it

has

not

takena

wrap

around

otherwise

it

uses

virtual

channel

Figure

An

example

of

a

deadlock

in

a

-LRB-

-RRB-

system

using

dimension

order

routing

with

virtual

channels

a

-RRB-

system

connectivity

with

its

virtual

channels

and

b

-RRB-

channels

occupied

by

messages

leading

to

a

deadlock

one

of

the

messages

is

non-minimal

and

a

minimal

path

can

be

established

without

using

the

shared

channel

This

leads

us

to

Corollary

Messages

belonging

to

the

two

different

classes

can

not

get

into

a

deadlock

inside

a

cluster

virtual

channel

that

may

be

used

router

Cluster

interface

router

Messages

from

other

clusters

enters

here

by

inter-cluster

messages

entering

the

cluster

to

get

to

a

destination

in

the

cluster

using

minimal

routing

dimension

set

-LRB-

radix

-RRB-

Figure

Partition

of

the

added

virtual

channel

network

under

the

modified-compositerouting

scheme

to

route

messages

entering

the

cluster

interface

to

reach

a

destination

inside

the

cluster

The

routing

is

minimal

The

partitioned

virtual

channel

network

is

acyclic

From

Lemma

we

know

that

the

set

of

channels

of

S

used

by

class

A

messages

say

SA

is

disjoint

from

the

set

of

messages

used

by

class

B

messages

say

SB

Thus

we

can

partition

the

channels

in

S

into

SA

and

SB

It

is

easy

to

see

that

SB

contains

exactly

the

reverse

of

the

channels

in

SA

Figure

shows

SA

in

a

cluster

Let

vcd

represent

a

set

of

virtual

channels

of

S

in

dimension

d

which

point

from

position

to

in

d

Here

d

and

It

is

easy

to

see

that

either

all

links

in

set

vcd

belong

to

SA

or

no

link

in

vcd

belong

to

SA

Also

if

vcd

belongs

to

SA

then

the

reverse

channel

set

vcd

can

not

belong

to

SA

This

leads

us

to

the

fact

that

SA

is

acyclic

-LRB-

this

can

be

also

observed

in

Fig.

-RRB-

Similarly

SB

can

also

be

shown

to

acyclic

This

leads

us

to

Lemma

An

inter-cluster

message

after

entering

the

cluster

and

while

being

routed

to

its

destination

inside

the

cluster

can

not

get

permanently

blocked

on

a

virtual

channel

of

S.

Theorem

The

modified-composite-routing

scheme

using

an

added

virtual

channel

per

physical

channel

of

the

intranet

is

deadlock-free

for

a

two-level

clustered

system

Proof

Intra-cluster

messages

travel

inside

each

cluster

using

deadlock-free

Rintra

routing

Since

they

use

an

independent

virtual

network

it

is

ensured

that

they

can

not

get

into

a

deadlock

The

deadlock-free

inter-cluster

routing

algorithm

Rinter

ensures

that

a

message

injected

into

the

internet

at

a

source

cluster

interface

is

bound

to

reach

the

destination

cluster

interface

By

Lemma

inter-cluster

messages

entering

a

cluster

can

not

get

permanently

blocked

inside

the

cluster

Using

the

fact

that

inter-cluster

messages

going

out

of

the

cluster

can

not

get

into

a

deadlock

they

are

guaranteed

to

reach

the

cluster

interface

All

these

facts

together

imply

that

inter-cluster

messages

also

can

not

deadlock

Suitable

routing

algorithm

on

the

added

virtual

channel

net

work

Intuitively

the

traffic

pattern

on

the

virtual

channel

network

S

is

quite

specific

Messages

are

either

traveling

towards

the

CI

or

away

from

it

Figure

show

the

traffic

distribution

of

messages

entering

the

cluster

on

various

virtual

channels

for

a

cluster

We

assume

that

messages

entering

the

cluster

are

uniformly

distributed

inside

the

cluster

As

expected

virtual

channels

near

the

CI

have

heavier

demand

than

those

away

from

the

CI

However

in

case

of

dimension-order

routing

which

routes

on

higher

dimensions

first

there

is

a

big

imbalance

in

the

usage

of

the

virtual

channels

next

to

the

CI

In

dimension-order

routing

the

traffic

on

the

highest

dimension

virtual

channel

next

to

the

CI

is

and

on

the

lowest

dimension

virtual

channel

it

is

However

the

distribution

of

the

traffic

on

virtual

channels

next

to

the

CI

in

case

of

fully

adaptive

routing

algorithm

is

more

balanced

and

is

Thus

using

fully

adaptive

algorithm

on

S

to

route

inter-cluster

messages

inside

the

cluster

demonstrates

potential

to

give

much

better

performance

Performance

Analysis

This

section

analyzes

some

important

parameters

that

influence

performance

in

-LRB-

k

n

-RRB-

clustered

systems

and

derives

useful

guidelines

for

architects

to

design

such

systems

Two

important

metrics

of

network

performance

latency

and

throughput

are

considered

Throughput

for

clustered

systems

is

estimated

under

various

bandwidth

constraints

An

analytical

contention

model

is

also

developed

to

predict

system

behavior

Bandwidth

and

capacity

constraints

A

method

to

estimate

the

system

throughput

is

to

calculate

the

capacity

of

the

network

or

the

number

of

messages

that

can

be

handled

by

the

network

in

unit

time

Typically

the

actual

maximum

throughput

is

some

fraction

of

this

estimated

theoretical

throughput

For

virtual

channel

that

may

be

used

router

Cluster

interface

router

Messages

from

other

clusters

enters

here

by

inter-cluster

messages

entering

the

cluster

to

get

to

a

destination

in

the

cluster

using

minimal

routing

-LRB-

a

-RRB-

Traffic

distribution

with

dimension

order

routing

Cluster

interface

router

Messages

from

other

clusters

enters

here

-LRB-

b

-RRB-

A

possible

traffic

distribution

with

fully

adaptive

routing

derived

experimentally

virtual

channel

that

may

be

used

router

by

inter-cluster

messages

entering

the

cluster

to

get

to

a

destination

in

the

cluster

using

minimal

routing

Figure

Traffic

density

on

the

virtual

channels

for

routing

inter-cluster

messages

from

the

cluster

interface

to

the

respective

cluster

destinations

using

-LRB-

a

-RRB-

dimension-order

routing

and

-LRB-

b

-RRB-

fully

adaptive

routing

a

-LRB-

k

n

-RRB-

system

consider

each

node

generating

messages

of

length

L

at

the

rate

m.

Under

uniform

traffic

assumption

it

can

be

observed

that

M

mC

-LRB-

p

-RRB-

where

p

CN

is

the

effective

rate

at

which

a

cluster

-LRB-

or

equivalently

its

CI

-RRB-

injects

messages

into

the

internet

In

a

clustered

system

the

maximum

value

of

m

can

be

potentially

restricted

by

a

number

of

constraints

like

channel

capacities

of

the

intranet

and

internet

and

the

total

bandwidth

available

to

access

the

CI

from

the

inter

and

intra

sides

These

constraints

are

graphically

shown

in

Fig.

The

objective

is

to

derive

an

upper

bound

of

m

by

each

constraint

and

to

generate

a

constraint

vector

The

least

value

in

this

constraint

vector

is

the

ideal

maximum

m

that

the

system

can

handle

Cluster

Inter-cluster

network

Internet

channel

capacity

constraint

Intranet

channel

capacity

constraint

Incoming

bandwidth

constraint

at

CI

Outgoing

bandwidth

constraint

at

CI

Cluster

Interface

Figure

Various

channel

and

bandwidth

constraints

which

can

restrict

the

maximum

ideal

throughput

in

a

two-level

clustered

system

Internet

channel

capacity

constraint

Consider

the

total

traffic

outgoing

from

each

cluster

This

rate

is

M

and

it

has

to

travel

an

average

distance

d

-LRB-

k

-RRB-

ss

k

on

each

of

the

n

internet

dimensions

Thus

the

required

bandwidth

on

the

internet

is

-LRB-

NC

-RRB-

MdnL

while

the

offered

bandwidth

by

all

internet

channels

is

W

where

NC

is

the

total

number

of

nodes

-LRB-

CIs

-RRB-

in

the

internet

and

W

is

the

channel

width

in

the

internet

Under

constant

bisection

width

assumption

we

have

W

k

Thus

we

have

-LRB-

NC

-RRB-

MdnL

W

-RRB-

M

-RRB-

m

dLC

-LRB-

p

-RRB-

LC

-LRB-

p

-RRB-

-LRB-

-RRB-

Let

us

define

mInter

sat

as

LC

-LRB-

p

-RRB-

Thus

mInter

sat

is

the

maximum

ideal

throughput

per

node

that

can

be

supported

by

the

internet

channels

Intranet

channel

capacity

constraint

Consider

each

cluster

generating

messages

at

the

rate

of

Cm

per

unit

time

Out

of

this

a

fraction

Cmp

is

for

intra-cluster

and

the

rest

Cm

-LRB-

p

-RRB-

is

for

inter-cluster

The

types

of

traffic

utilizing

the

cluster

channels

and

their

respective

generation

rates

can

be

categorized

as

follows

Intracluster

messages

require

a

bandwidth

of

Cmp

Intercluster

messages

leaving

the

cluster

require

a

bandwidth

of

Cm

-LRB-

p

-RRB-

The

third

type

is

intercluster

messages

entering

the

cluster

Assuming

steady

state

this

traffic

is

exactly

the

same

as

the

traffic

leaving

the

cluster

Therefore

the

entry

rate

is

Cm

-LRB-

p

-RRB-

Any

of

these

messages

use

on

the

average

of

the

cluster

channel

cycles

Thus

to

sustain

a

throughput

of

m

we

require

a

total

bandwidth

of

-LRB-

p

-RRB-

The

cluster

offers

a

bandwidth

of

Thus

we

have

-LRB-

p

-RRB-

-RRB-

m

-LRB-

p

-RRB-

-RRB-

m

L

-LRB-

p

-RRB-

L

-LRB-

p

-RRB-

-LRB-

-RRB-

Let

us

define

mintra

sat

as

L

-LRB-

p

-RRB-

Since

C

N

we

get

p

or

p

Consider

this

fact

in

deriving

relationships

between

some

of

the

constraints

For

example

mInter

sat

minter

sat

p

-LRB-

p

-RRB-

C

minter

sat

-LRB-

p

-RRB-

C

minter

sat

Thus

for

all

cluster

sizes

with

C

mInter

sat

minter

sat

Hence

the

internet

channel

capacity

constraint

imposes

a

stricter

bound

on

the

value

of

m

than

the

intranet

channel

capacity

constraint

Outgoing

bandwidth

constraint

at

CI

Let

us

look

at

the

total

bandwidth

of

the

channels

which

connect

the

CI

to

the

rest

of

the

cluster

This

value

is

and

should

be

greater

than

the

desired

bandwidth

of

ML

This

leads

to

ML

-RRB-

m

C

-LRB-

p

-RRB-

L

C

-LRB-

p

-RRB-

L

-LRB-

-RRB-

Let

us

define

mout

sat

as

C

-LRB-

p

-RRB-

L

We

can

observe

that

mout

sat

minter

sat

-LRB-

p

-RRB-

-LRB-

p

-RRB-

Now

from

our

earlier

discussion

p

p

Also

C

and

it

can

be

easily

observed

that

for

all

values

of

the

fraction

Thus

mout

sat

minter

sat

which

implies

that

the

outgoing

bandwidth

constraint

at

CI

is

stricter

than

the

intranet

channel

capacity

constraint

On

the

other

hand

since

mInter

sat

mout

sat

we

conclude

that

mInter

sat

mout

sat

for

cluster

sizes

larger

than

In

other

words

if

we

assume

similar

channel

cycle

times

in

both

hierarchies

the

internet

channel

capacity

constraint

is

stricter

than

the

outgoing

bandwidth

constraint

at

CI

Incoming

bandwidth

constraint

at

CI

Let

us

look

at

the

total

bandwidth

of

the

channels

which

connect

the

CI

to

the

other

clusters

This

value

is

and

should

be

greater

than

the

desired

bandwidth

of

ML

This

leads

to

m

C

-LRB-

p

-RRB-

L

nk

C

-LRB-

p

-RRB-

L

-LRB-

-RRB-

Let

us

define

minsat

as

nk

C

-LRB-

p

-RRB-

L

Note

that

mInter

sat

minsat

Now

always

except

in

the

pathological

case

that

the

system

has

just

two

or

three

clusters

Therefore

minsat

is

more

restrictive

than

mInter

sat

in

almost

all

cases

Similarly

we

can

show

that

minsat

is

more

restrictive

than

mintra

sat

The

relationship

among

the

above

constraints

can

be

summarized

as

follows

mInter

sat

mout

sat

minsat

mintra

sat

-LRB-

-RRB-

Thus

out

of

the

four

constraints

discussed

mInter

sat

is

most

strict

It

is

to

be

noted

that

the

value

of

mInter

sat

is

dependent

only

on

the

size

of

the

cluster

and

quite

independent

of

the

cluster

topology

This

fact

leads

to

Observation

Keeping

the

internet

topology

internet

size

and

cluster

size

fixed

varying

the

cluster

topology

has

no

effect

on

the

maximum

system

throughput

The

next

question

is

what

happens

if

we

do

change

the

cluster

topology

while

keeping

the

cluster

size

fixed

Given

a

cluster

size

of

C

the

bandwidth

to

reach

CI

is

proportional

to

It

is

easy

to

see

that

the

bandwidth

falls

very

sharply

as

we

move

from

one

dimensional

cluster

topology

-LRB-

ring

-RRB-

to

other

higher

dimensional

cluster

topologies

From

Obs

we

know

that

increasing

cluster

dimensionality

does

not

affect

system

throughput

but

the

average

message

latency

increases

This

leads

to

Observation

Keeping

the

internet

topology

internet

size

and

cluster

size

fixed

a

ring

cluster

offers

the

least

average

latency

Finally

the

expression

mInter

sat

CL

-LRB-

p

-RRB-

leads

us

to

Observation

The

maximum

system

throughput

falls

as

cluster

size

and

message

length

grow

From

the

above

observations

one

would

tend

to

conclude

that

clustering

is

not

that

a

good

idea

However

all

the

above

analysis

is

determining

saturated

throughput

under

uniform

traffic

which

is

not

truly

representative

of

the

traffic

patterns

on

clustered

systems

In

fact

using

compiler

tools

and

good

mapping

schemes

for

a

given

program

one

would

expect

to

generate

as

high

localized

communication

as

possible

to

exploit

clustered

architectures

Through

simulation

experiments

we

will

be

showing

in

sec.

that

even

for

small

localization

factor

in

traffic

patterns

relatively

large

cluster

sizes

offer

good

system

performance

A

Contention

Model

For

-LRB-

k

n

-RRB-

systems

Let

us

develop

an

analytical

contention

model

for

-LRB-

k

n

-RRB-

systems

The

routing

strategy

chosen

to

route

messages

directly

determines

the

methodology

and

scope

of

the

analytical

model

used

For

example

an

analytical

model

developed

for

dimension-order

routing

on

kary

n-cubes

would

not

predict

system

behavio

-LSB-

accurately

with

fully

adaptive

routing

The

model

developed

in

this

work

assumes

AB

routing

strategy

i.e.

messages

are

fully

absorbed

at

network

interfaces

before

being

released

into

another

network

In

sec.

we

saw

that

AB

strategy

made

better

use

of

the

internet

bandwidth

Since

dimension-order

routing

is

commonly

used

in

contemporary

direct

networks

we

use

it

in

routing

intra-cluster

messages

inside

an

intranet

and

inter-cluster

messages

in

the

internet

For

routing

intercluster

messages

when

they

are

traveling

inside

the

cluster

we

use

fully

adaptive

algorithm

on

an

added

virtual

network

as

suggested

in

sec.

We

derive

steady

state

expressions

for

Average

latency

of

inter-cluster

messages

as

a

function

of

m

the

network

request

rate

by

each

processor

and

p

the

probability

of

a

message

being

intra-cluster

Average

latency

of

intra-cluster

messages

as

a

function

of

m

and

p.

Average

latency

of

a

message

in

the

system

as

a

function

of

m

and

p.

Estimating

inter-cluster

message

latency

An

inter-cluster

message

goes

through

three

phases

one

in

the

inter-cluster

and

two

symmetric

ones

in

the

source

and

the

destination

clusters

We

derive

the

latency

for

each

phase

below

A.

Estimating

inter-cluster

message

latency

in

the

internet

We

analyze

the

latency

of

an

average

inter-cluster

message

from

the

point

it

is

released

into

the

internet

at

the

source-CI

to

the

point

it

is

fully

absorbed

at

its

destination-CI

in

the

presence

of

contention

The

effective

network

request

rate

by

a

CI

is

M

mC

-LRB-

p

-RRB-

An

analysis

similar

to

Agrawal

-LSB-

-RSB-

predicts

the

contention

delay

-LRB-

w

-RRB-

through

a

CIP

switch

for

an

intercluster

message

as

w

aeB

-LRB-

ae

-RRB-

-LRB-

d

-RRB-

-LRB-

-RRB-

where

B

is

the

message

size

in

flits

L

W

and

ae

MBd

is

the

channel

utilization

of

an

intercluster

channel

However

this

analysis

is

based

on

the

traffic

on

physical

channels

and

does

not

consider

multiple

virtual

channels

being

multiplexed

on

a

single

physical

channel

We

extend

this

model

below

to

include

the

effect

of

virtual

channels

The

propagation

time

of

a

flit

through

one

node

-LRB-

Tnode

-RRB-

without

considering

contention

is

simply

the

sum

of

the

switch

delay

-LRB-

Tsw

-RRB-

and

the

wire

delay

-LRB-

Tpch

-RRB-

across

a

physical

link

Assume

that

the

routing

algorithm

dictates

the

multiplexing

of

v

virtual

channels

on

a

physical

link

and

there

are

no

additional

virtual

channels

The

effect

of

the

virtual

channels

can

be

approximated

by

stretching

the

wire

delay

Tpch

to

Tvch

At

high

system

load

one

would

expect

Tvch

Tpch

v

However

the

policy

of

demand

multiplexing

i.e.

allotting

a

time-slice

to

a

virtual

channel

only

if

it

has

a

flit

ready

to

transfer

implies

that

at

low

system

load

this

stretching

would

be

less

than

v.

To

reflect

such

behavior

we

model

Tvch

as

Tpch

-LRB-

ae

-LRB-

v

-RRB-

-RRB-

by

taking

utilization

into

account

This

is

a

reasonable

model

as

it

conforms

to

both

the

cases

of

zero

stretching

of

wire

delay

at

low

loads

and

a

stretc

-LSB-

by

a

factor

v

at

high

loads

Under

this

model

the

propagation

time

through

a

node

without

considering

contention

is

given

by

Tnode

Tsw

Tpch

-LRB-

ae

-LRB-

v

-RRB-

-RRB-

-LRB-

-RRB-

The

average

transit

time

-LRB-

Tinter

-RRB-

of

an

inter-cluster

message

through

the

internet

taking

into

account

contention

can

be

computed

in

terms

of

the

channel

utilization

For

wormhole

routed

switches

with

single

cycle

transit

time

through

a

switch

the

average

delay

through

a

switch

is

thus

-LRB-

w

-RRB-

cycles

Given

that

the

average

number

of

inter

cluster

links

traversed

by

a

message

is

D

nd

we

have

Tinter

aeB

-LRB-

ae

-RRB-

-LRB-

d

-RRB-

nd

B

Tnode

-LRB-

-RRB-

B.

Estimating

inter-cluster

message

latency

in

the

intranet

Here

we

analyze

the

latency

of

an

average

inter-cluster

message

when

it

is

traversing

from

the

source

to

the

source-CI

or

from

the

destination-CI

to

the

actual

destination

We

assume

a

separate

virtual

channel

network

S

inside

the

cluster

and

use

a

fully

adaptive

algorithm

for

routing

inter-cluster

messages

as

suggested

in

sec.

Recall

that

intracluster

messages

do

not

use

this

virtual

channel

network

We

assume

that

the

algorithm

for

routing

intra-cluster

messages

requires

virtual

channels

per

physical

channel

Thus

the

total

number

of

virtual

channels

on

a

physical

channel

in

the

intranet

is

-LRB-

-RRB-

The

time

of

propagation

through

an

intranet

node

without

considering

contention

is

given

by

-LRB-

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

where

and

are

the

switch

and

wire

delays

respectively

in

the

intranet

These

switch

and

wire

delays

can

be

different

from

the

internet

delays

however

here

we

assume

them

to

be

the

same

The

physical

channel

utilization

in

the

intranet

is

given

by

Messages

on

S

either

traverse

towards

the

CI

or

away

from

it

Therefore

the

channel

utilization

on

this

network

is

bound

to

be

more

near

the

CI

and

fall

quite

sharply

as

we

move

away

from

it

Therefore

it

is

a

good

approximation

to

assume

that

the

only

switching

delay

encountered

by

the

message

in

this

phase

is

when

it

is

traveling

the

link

to/from

the

CI

For

these

links

we

have

an

average

of

M

messages

arriving

per

Tpch

time

We

assume

this

arrival

distribution

to

be

Poisson

Again

L

cycles

are

required

to

service

each

message

by

a

link

where

is

the

channel

width

in

the

intranet

In

terms

of

time

this

is

equivalent

to

an

average

service

time

of

We

model

this

service

time

as

having

an

exponential

distribution

with

mean

Thus

the

delay

seen

at

the

CI

is

M

Tpch

A

-LRB-

-RRB-

Given

that

the

average

number

of

intranet

links

traversed

is

we

have

the

latency

of

the

inter-cluster

message

to

travel

from

the

destination-CI

to

the

destination

as

Tinter

given

by

Tinter

M

Tpch

A

-LRB-

-RRB-

-LRB-

-RRB-

We

come

up

with

a

similar

expression

for

the

latency

of

an

inter-cluster

message

when

it

travels

from

the

source

to

the

source-CI

Thus

the

total

latency

of

an

intercluster

message

Tinter

is

given

by

Tinter

Tinter

-LRB-

-RRB-

Estimating

intra-cluster

message

latency

The

expression

for

the

average

transit

time

Tintra

for

an

intra-cluster

message

through

the

intranet

by

taking

into

account

the

contention

can

be

derived

similar

to

Eq

-LRB-

-RRB-

Given

that

the

average

number

of

intranet

links

traversed

is

we

have

the

latency

as

Tintra

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

We

know

that

a

message

generated

by

a

processor

can

be

either

intra-cluster

with

a

probability

p

or

inter-cluster

with

a

probability

-LRB-

p

-RRB-

Therefore

the

average

latency

of

a

message

in

the

system

is

estimated

as

Tave

Tinter

-LRB-

p

-RRB-

Tintra

p

-LRB-

-RRB-

Simulation

Experiments

and

Results

To

compare

the

performance

of

various

clustered

configurations

we

performed

simulation

experiments

on

an

event

driven

simulated

cluster

testbed

developed

using

CSIM

-LSB-

-RSB-

The

testbed

uses

composite-routing

schemes

with

flexibility

to

choose

the

two

components

-LRB-

intra

and

inter

-RRB-

from

amongst

dimension

order/fully

adaptive/planar

adaptive

options

It

implements

flit-level

simulation

and

allows

for

demand

multiplexing

of

virtual

channels

on

physical

channels

leading

to

better

utilization

of

channel

bandwidth

The

simulator

has

flexibility

for

varying

architectural

parameters

and

timing

parameters

In

our

simulations

we

fixed

the

channel

cycle

times

in

both

levels

to

be

same

This

is

reasonable

assuming

similar

technology

being

used

in

both

levels

Channel

widths

were

determined

by

the

constant

bisection

bandwidth

constraint

We

experimented

with

both

AB

and

NAB

routing

strategies

The

timing

parameters

in

the

simulator

were

as

follows

Tsw

s

and

Tpch

s.

Simulations

were

done

for

uniform

distribution

of

traffic

i.e.

a

generated

message

by

a

processor

is

directed

with

equal

probability

to

any

other

processor

in

the

system

However

on

clustered

architectures

one

would

expect

communication

patterns

to

be

more

localized

There

are

various

ways

of

defining

locality

of

communication

of

applications

We

used

the

locality

model

developed

by

Agarwal

in

-LSB-

-RSB-

in

which

a

processor

generates

messages

with

equal

probability

to

some

nearest

subset

of

processors

We

compared

the

performance

for

various

cluster

sizes

and

topologies

for

uniform

and

localized

traffic

patterns

Effect

of

Cluster

Size

Average

message

latency

versus

throughput

per

node

were

experimented

for

system

sizes

of

N

and

processors

with

various

topologies

Simulation

results

verified

our

earlier

observation

that

throughput

falls

with

growth

in

cluster

size

Figure

shows

the

trade-off

for

N

It

was

observed

that

configuration

-LRB-

-RRB-

with

cluster

size

of

saturates

around

a

throughput

whereas

configuration

-LRB-

-RRB-

with

cluster

size

of

saturates

at

and

configuration

-LRB-

-RRB-

with

cluster

size

of

saturates

even

earlier

at

around

Similar

results

were

also

observed

in

other

system

sizes

as

well

with

both

AB

and

NAB

routing

strategies

Average

message

latency

remained

around

s

for

all

system

configurations

when

operating

at

or

below

of

their

respective

saturation

points

Average

latency

of

a

message

in

us

Throughput

per

node

in

messages/us

N

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Latency

vs.

throughput

for

system

size

of

N

processors

under

different

-LRB-

k

n

-RRB-

topologies

Uniform

traffic

pattern

dimension-order

routing

and

absorb-andretransmit

-LRB-

AB

-RRB-

scheme

were

used

Effect

of

Topology

Intranet

or

Cluster

Topology

To

observe

the

effect

of

varying

cluster

topology

on

system

performance

we

kept

the

cluster

size

the

internet

size

and

the

internet

topology

fixed

and

varied

only

the

cluster

dimensionality

Observations

and

in

sec.

predict

that

with

increasing

dimensionality

of

cluster

only

system

latency

increases

but

system

throughput

does

not

get

affected

This

observation

was

verified

by

running

simulations

on

the

configurations

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

with

dimension-order

routing

and

AB

scheme

The

results

shown

in

Fig.

indicate

that

maximum

system

throughput

remained

around

for

all

three

configurations

but

the

message

latencies

increased

from

to

to

as

dimensionality

of

the

cluster

went

up

from

to

to

However

our

experimental

results

were

different

with

NAB

strategy

in

that

not

only

latency

went

up

with

increased

dimensionality

but

throughput

also

went

down

For

example

we

ran

simulations

on

the

same

configurations

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

with

NAB

strategy

The

results

shown

in

Fig.

indicate

that

performance

degrades

both

in

terms

of

latency

and

throughput

The

reason

for

this

is

the

flit

mismatch

problem

discussed

in

sec.

which

leads

to

wastage

of

a

considerable

fraction

of

the

internet

bandwidth

As

cluster

dimensionality

goes

up

the

flit

size

falls

and

increases

the

mismatch

This

leads

to

a

greater

loss

in

bandwidth

which

causes

throughput

to

fall

As

stated

earlier

in

sec.

we

proposed

a

way

to

solve

this

mismatch

problem

by

using

more

virtual

channels

on

a

physical

channel

with

demand

multiplexing

so

that

Average

latency

of

a

message

in

us

Throughput

per

node

in

messages/us

Inter-cluster

flit

size

Cluster

Cluster

Cluster

Figure

Latency

vs.

throughput

for

system

size

of

N

processors

under

different

cluster

dimensionality

-LRB-

and

-RRB-

Uniform

traffic

pattern

dimension-order

routing

and

absorb-and-retransmit

-LRB-

AB

-RRB-

scheme

were

used

the

bandwidth

gets

utilized

by

some

other

message

To

verify

this

fact

we

switched

to

fully

adaptive

routing

which

uses

more

virtual

channels

than

dimension

order

routing

We

ran

the

simulations

on

the

configurations

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

with

fully

adaptive

routing

using

virtual

channels

The

results

are

also

shown

in

Fig.

We

observed

an

improvement

of

in

throughput

with

an

increase

of

just

virtual

channel

It

implies

that

even

with

a

small

number

of

virtual

channels

per

physical

channel

the

flit

mismatch

problem

can

be

solved

effectively

and

the

system

throughput

becomes

oblivious

to

cluster

dimensionality

It

was

also

observed

that

latency

increased

with

dimensionality

and

the

lowest

latencies

were

obtained

when

the

cluster

topology

was

a

ring

Internet

topology

Higher

dimensional

internet

implies

smaller

distances

to

be

traveled

in

the

internet

On

the

other

hand

it

also

implies

narrower

internet

flit

size

-LRB-

W

-RRB-

and

therefore

messages

should

take

longer

to

travel

However

it

is

to

be

noted

that

with

NAB

routing

strategy

if

the

intranet

flit

size

-LRB-

W

-RRB-

is

smaller

than

W

then

message

speeds

are

governed

by

W

Under

these

circumstances

increasing

internet

dimensionality

helps

to

reduce

the

flit

size

mismatch

till

the

point

W

W

This

was

observed

through

simulations

as

shown

in

Fig.

The

cluster

was

with

and

the

internet

flit

size

W

was

varied

from

to

to

as

dimensionality

was

increased

With

increased

dimensionality

performances

became

better

These

results

correspond

to

dimension

order

routing

With

fully

adaptive

routing

we

hypothesize

that

flit

mismatch

will

not

be

critical

to

the

system

performance

Hence

lowdimensional

internets

with

and

topology

have

potential

to

deliver

best

system

performance

Average

latency

of

a

message

in

us

Throughput

per

node

in

messages/us

Inter-cluster

flit

size

Cluster

flit

size

vchs

Cluster

flit

size

vchs

Cluster

flit

size

vchs

Cluster

flit

size

vchs

Cluster

flit

size

vchs

Cluster

flit

size

vchs

Figure

Latency

vs.

throughput

for

system

size

of

N

processors

under

different

cluster

dimensionality

-LRB-

and

-RRB-

and

intercluster

routing

schemes

-LRB-

dimension

order

routing

with

virtual

channels

and

fully

adaptive

with

virtual

channels

-RRB-

Uniform

traffic

pattern

and

no-absorb-at-interface

-LRB-

NAB

-RRB-

scheme

were

used

Average

latency

of

a

message

in

us

Throughput

per

node

in

messages/us

N

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Effect

of

increasing

internet

dimension

for

system

size

of

N

processors

with

cluster

size

The

intranet

flit

size

and

internet

flit

size

W

changed

from

to

to

as

dimension

was

increased

W

remained

greater

than

Uniform

traffic

pattern

dimension-order

routing

and

NAB

scheme

were

used

Effect

of

Locality

To

observe

the

impact

of

increasing

locality

on

performance

we

repeated

our

experiments

on

larger

cluster

sizes

like

with

more

localized

traffic

patterns

Figure

shows

the

latency

vs.

throughput

for

configuration

-LRB-

-RRB-

under

both

uniform

traffic

and

localized

traffic

with

p

Significant

improvement

in

throughput

and

reasonable

improvement

in

latency

were

observed

under

this

localized

traffic

Figure

shows

the

latency

vs.

throughput

for

configuration

-LRB-

-RRB-

under

localized

traffic

with

p

and

p

and

for

configurations

-LRB-

-RRB-

and

-LRB-

-RRB-

under

uniform

traffic

It

was

observed

that

with

a

locality

of

p

cluster

size

of

performed

as

well

as

cluster

size

of

with

uniform

traffic

Similarly

with

a

locality

of

p

cluster

size

of

outperformed

cluster

size

of

with

uniform

traffic

This

shows

that

with

higher

locality

in

communication

it

is

attractive

to

build

systems

with

larger

cluster

sizes

Average

latency

of

a

message

in

us

Throughput

per

node

in

messages/us

uniform

traffic

localized

p

Figure

Effect

of

locality

on

-LRB-

-RRB-

system

with

uniform

traffic

and

localized

traffic

-LRB-

p

-RRB-

under

dimension

order

routing

It

was

also

observed

that

for

a

given

cluster

size

increasing

the

system

size

i.e.

the

internet

size

did

not

lead

to

any

substantial

degradation

This

confirms

the

fact

that

clustered

architectures

are

indeed

scalable

Conclusions

In

this

paper

we

have

proposed

how

to

design

scalable

parallel

systems

using

two-level

k-ary

n-cube

wormhole-routed

interconnections

The

proposed

architecture

exploits

com

Average

latency

of

a

message

in

us

Throughput

per

node

in

messages/us

N

-LRB-

-RRB-

localized

p

-LRB-

-RRB-

localized

p

-LRB-

-RRB-

uniform

traffic

-LRB-

-RRB-

uniform

traffic

Figure

The

performance

of

-LRB-

-RRB-

system

with

localized

traffic

-LRB-

p

and

-RRB-

being

compared

to

systems

with

smaller

cluster

sizes

-LRB-

-RRB-

and

-LRB-

-RRB-

with

uniform

traffic

munication

locality

and

takes

advantage

of

VLSI

integration

by

using

processor

clusters

Two

minimal

deadlock-free

wormhole

routing

schemes

with

associated

virtual

channels

and

message

protocols

at

the

cluster

interface

have

been

proposed

to

route

messages

in

such

twolevel

k-ary

n-cube

systems

An

analytical

model

and

simulation

experiments

were

used

in

determining

optimal

configurations

for

such

systems

under

the

constant

bisection

bandwidth

constraint

The

research

provides

the

following

guidelines

to

system

designers

for

building

large-scale

parallel

systems

Small

sized

clusters

with

a

ring

intra-cluster

topology

and

a

inter-cluster

network

connecting

these

clusters

offer

maximum

system

performance

with

close-tominimum

cost

The

intranet

and

internet

topologies

together

with

the

composite-routing

scheme

should

minimize

flit

mismatch

problem

in

order

to

maximally

utilize

the

system

bandwidth

With

communication

locality

systems

with

large

cluster

sizes

are

attractive

For

clustered

systems

NAB

routing

strategy

provides

minimum

latency

compared

to

AB

routing

strategy

in

order

to

support

longer

messages

This

phase

of

the

research

has

emphasized

on

direct-network

based

clusters

In

our

next

phase

we

are

planning

on

determining

optimal

intra-cluster

size

and

topology

with

indirect

networks

at

the

cluster

level

Once

the

optimal

topologies

are

derived

it

will

be

interesting

to

see

how

to

implement

efficient

communication

and

synchronization

on

these

systems

to

support

various

programming

models

such

as

distributed-memory

cache-coherent

shared-memory

and

distributed-shared-memory

paradigms

References

-LSB-

-RSB-

Basak

D.

and

Panda

D.K.

Scalable

Architectures

with

k-ary

n-cube

cluster-c

organization

Submitted

to

the

Symposium

of

Parallel

and

Distributed

Processing

-LSB-

-RSB-

Dally

William

J.

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Transactions

on

Computers

Vol

No.

June

-LSB-

-RSB-

Agarwal

Anant

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

Vol

No.

Oct.

-LSB-

-RSB-

Hsu

W.

Yew

P.

The

Performance

of

Hierarchical

Systems

with

wiring

constraints

Int

Conf

on

Parallel

Processing

Aug.

-LSB-

-RSB-

Hsu

W.

Yew

P.

The

Impact

of

Wiring

Constraints

on

Hierarchical

Network

Performance

Int

Parallel

Processing

symp

Mar

-LSB-

-RSB-

Hsu

W.

Yew

P.

The

Performance

Evaluation

of

Wire-Limited

Hierarchical

Network

CSRD

report

no.

Aug

-LSB-

-RSB-

Carlson

D.

The

Mesh

with

a

Global

Mesh

a

Flexible

High-speed

Organization

for

Parallel

Computation

Int

Conf

on

Supercomputer

Systems

IEEE

Comp

Soc

Press

-LSB-

-RSB-

Wu

S.

Liu

M.

A

Cluster

Structure

as

an

Interconnection

Network

for

Large

Multiprocessor

Systems

IEEE

Trans

on

Computers

Vol

No.

Apr.

-LSB-

-RSB-

Agrawal

A.

Mahgoub

I.

Performance

Analysis

of

cluster-based

supersystems

Int

Conf

on

Supercomputer

Systems

IEEE

Comp

Soc

Press

-LSB-

-RSB-

Kuck

D.J.

Kuhn

R.H.

Leasure

B.R.

and

Sameh

A.

Parallel

Computing

Today-the

Cedar

Approach

Science

Feb.

-LSB-

-RSB-

Dandamudi

S.

Eager

D.

Hierarchical

Interconnection

Networks

for

Multicomputer

Systems

IEEE

Trans

on

Computers

Vol

No.

June

-LSB-

-RSB-

Padmanabhan

K.

Effective

Architectures

for

Data

Access

in

a

Shared

Memory

Hierarchy

Jour

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

Mabbs

S.A.

Forward

K.E.

Performance

Analysis

of

a

Hierarchical

Communication

Architecture

for

a

Shared-Memory

Multiprocessor

Report

No.

Feb.

Dept.

of

Electrical

and

Electronic

Engineering

The

University

of

Melbourne

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

Lenoski

D.

et

al

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LSB-

-RSB-

Kendall

Square

Research

KSR

Technical

Summary

-LSB-

-RSB-

Lenoski

D.

et

al

The

Directory-Based

Cache

Coherence

Protocol

for

the

DASH

Multiprocessor

In

Proceedings

of

the

Annual

Symposium

on

Computer

Architecture

pp

May

-LSB-

-RSB-

Asthana

A.

Jagdish

H.

and

Mathews

B.

Impact

of

Advanced

VLSI

packaging

on

the

design

of

a

large

parallel

computer

Int

Conf

on

Parallel

Processing

August

-LSB-

-RSB-

Chien

A.

A.

and

Kim

J.H.

Planar

Adaptive

Routing

Low

cost

adaptive

networks

for

multiprocessors

Proc

Ann.Int.Symp.on

Comp.Arch

Pgs

-LSB-

-RSB-

Duato

J.

On

the

design

of

deadlock-free

adaptive

routing

algorithms

for

multicomputers

Theoretical

aspects

PARLE

Parallel

Architectures

and

Languages

pp

-LSB-

-RSB-

Gaughan

P.T.

and

Yalamanchili

S.

Analytical

Models

of

Bandwidth

Allocation

in

Pipelined

k-ary

n-cubes

Int

Parallel

Processing

Symposium

pp

-LSB-

-RSB-

Balakrishnan

S.

and

Panda

D.K.

Impact

of

Multiple

Consumption

Channels

on

Wormhole

Routed

k-ary

n-cubes

Int

Parallel

Processing

Symposium

pp

-LSB-

-RSB-

Schwetman

H.

CSIM

A

C-based

process-oriented

simulation

language

In

Proc

of

Winter

Simulation

Conf.

pp.

clustered architectures
hierarchical systems
interconnection networks
multiprocessors
parallel architectures