Designing

Processor-cluster

Based

Systems

Interplay

Between

Cluster

Organizations

and

Collective

Communication

Algorithms

Debashis

Basak

and

Dhabaleswar

K.

Panda

Technical

Report

Designing

Processor-cluster

Based

Systems

Interplay

Between

Cluster

Organizations

and

Collective

Communication

Algorithms

D.

Basak

and

D.

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

fbasak,panda,g@cis.ohio-state.edu

Contact

Author

D.

K.

Panda

Abstract

Advancements

in

VLSI

has

made

it

attractive

to

package

multiple

processors

into

a

single

multichip

or

a

board

module

There

is

an

increasing

trend

towards

using

such

processor-clusters

in

large

multiprocessor

design

Past

research

on

designing

processor-cluster

based

systems

has

focused

mainly

in

studying

the

packaging

technologies

affecting

the

inter-cluster

network

To

make

processor-cluster

based

multiprocessor

design

more

attractive

there

is

a

strong

need

to

understand

the

details

about

the

topology

inside

the

cluster

its

memory

organization

and

the

impact

of

this

organization

on

system

performance

In

this

paper

we

focus

on

such

aspects

of

processor-cluster

design

with

an

overall

objective

to

support

a

logically

shared

address

programming

model

We

analyze

the

communication

costs

for

accesing

inter-cluster

and

intra-cluster

memories

under

different

cluster

organizations

The

merits

of

these

organizations

are

evaluated

based

on

the

performance

of

collective

communication

algorithms

which

occur

frequently

in

appplications

In

this

paper

we

focus

on

implementing

the

broadcast

collective

communication

algorithm

Umesh

on

clustered

systems

Our

results

indicate

that

cluster

organizations

like

bus

and

crossbar

which

allow

memory

inside

a

cluster

to

be

accessed

without

messaging

overheads

outperform

other

organizations

because

of

faster

intra-cluster

access

We

also

demonstrate

that

such

faster

access

can

be

exploited

to

design

better

algorithms

on

clustered

systems

We

propose

a

new

algorithm

clus

mesh

for

broadcasting

on

clustered

meshes

For

reasonably

faster

communication

within

clusters

this

algorithm

can

outperform

the

existing

umesh

algorithm

by

upto

research

is

supported

in

part

by

NSF

Grant

and

an

Ohio

State

University

Presidential

Fellowship

Contents

Introduction

Processor-cluster

based

systems

Processor-cluster

organizations

Star

Direct

Bus

Crossbar

Evaluation

of

different

processor-cluster

organizations

Deriving

the

time

for

broadcast

Comparision

of

broadcast

times

Broadcast

on

clustered

meshes

Single

port

model

with

multiple

injection

channels

Illustrating

U

mesh

in

terms

of

inter

and

intra-cluster

steps

Usage

of

injection

channels

by

U

mesh

Illustrating

clus

mesh

The

clus

mesh

algorithm

Comparison

of

Clus

mesh

and

U

mesh

performance

Impact

of

increasing

number

of

clusters

Impact

of

increasing

the

cluster

size

Conclusion

Introduction

With

advancements

in

VLSI

and

packaging

technologies

it

has

become

cost-effective

to

integrate

multiple

processing

elements

into

a

chip

or

a

board

-LSB-

-RSB-

This

is

leading

to

the

development

of

parallel

systems

using

such

processor-clusters

as

building

blocks

instead

of

single

processors

thus

allowing

a

modular

and

hierarchical

approach

to

building

large

systems

Prominent

examples

of

processorcluster

based

systems

are

the

Stanford

DASH

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

and

-LSB-

-RSB-

Typically

the

interconnections

connecting

the

processor-clusters

of

these

systems

are

scalable

meshes

tori

or

multistage

networks

This

is

referred

to

as

the

inter-cluster

network

The

interconnection

inside

a

processor

cluster

is

referred

to

as

the

intra-cluster

topology

Current

processor-clustered

organizations

connect

each

processor-cluster

to

the

inter-cluster

network

via

a

single

interface

referred

to

as

the

cluster

interface

or

network

interface

This

allows

a

system

with

a

given

total

number

of

processors

to

be

interconnected

by

using

fewer

network

interfaces

and

network

routers

leading

to

a

more

cost-effective

design

In

recent

years

there

has

been

a

lot

of

research

related

to

designing

processor-clustered

based

systems

However

most

of

these

research

have

emphasized

on

packaging

aspects

-LSB-

-RSB-

These

results

are

geared

towards

deriving

optimal

inter-cluster

topologies

under

packaging

and

pinout

constraints

While

deriving

such

optimal

topologies

researchers

have

primarily

emphasized

on

the

load

on

network

bisection

and

evaluated

different

topologies

with

respect

to

average

message

latency

and

throughput

However

the

performance

of

a

real

system

depends

on

several

other

factors

like

locality

of

reference

communication

overheads

and

performance

of

frequently-used

communication

patterns

In

our

recent

work

-LSB-

-RSB-

we

have

considered

communication

overheads

and

have

demonstrated

how

and

when

processor-clustered

systems

provide

benefits

leading

to

costeffective

system

designs

However

in

all

the

above

works

the

impact

of

intra-cluster

organization

on

the

performance

of

processor-clustered

systems

have

been

ignored

The

intra-cluster

organization

heavily

depends

on

the

programming

model

intended

for

a

system

In

order

to

support

shared

memory

programming

it

will

be

beneficial

to

allow

shared

memory

between

processors

inside

a

cluster

Recently

it

has

been

shown

that

DASH

like

architecture

providing

common

directory

access

for

processors

within

a

cluster

may

lead

to

performance

degradation

-LSB-

-RSB-

However

it

is

not

clear

whether

distributed

memory

systems

need

to

provide

shared-memory

access

between

sibling

processors

-LRB-

processors

within

a

cluster

-RRB-

From

a

naive

point

of

view

such

sharing

does

not

seem

to

be

beneficial

due

to

the

programming

model

However

with

a

closer

look

it

can

be

observed

that

in

the

absence

of

such

sharing

the

benefits

of

clustering

to

exploit

locality

of

reference

are

minimal

If

four

processors

in

a

processor-clustered

system

are

sitting

on

a

single

board

it

is

natural

that

faster

communication

between

these

processors

can

be

achieved

by

providing

some

portions

of

shared

memory

between

them

-LRB-

even

though

higher-level

communications

will

be

message-passing

-RRB-

This

will

lead

to

faster

message

passing

between

these

processors

and

the

applications

can

take

advantage

of

locality

Otherwise

processor-clustering

does

not

lead

to

any

performance

gain

If

we

believe

that

such

shared

memory

access

are

required

the

next

questions

are

-RRB-

which

intra-cluster

organization

will

provide

the

best

benefit

and

-RRB-

Can

the

existing

communication

algorithms

take

advantage

of

such

memory

sharing

to

deliver

better

performance

In

this

paper

we

take

on

such

a

challenge

to

provide

answers

to

the

above

questions

We

study

the

interplay

between

intra-cluster

organizations

and

the

performance

of

collective

communication

operations

which

are

frequently

used

operations

on

distributed

memory

systems

-LSB-

-RSB-

We

carry

out

our

study

in

four

phases

During

the

first

phase

we

perform

experiments

on

Cray

to

study

the

impact

of

processor

clustering

on

communication

between

remote

processors

and

sibling

processors

We

observe

that

there

are

no

performance

benefits

while

doing

communication

between

sibling

processors

on

Cray

During

next

phase

we

analyze

alternative

intra-cluster

organizations

-LRB-

star

bus

direct

and

crossbar

-RRB-

to

study

their

respective

capabilities

to

provide

faster

communication

between

sibling

processors

We

demonstrate

that

such

faster

communication

can

also

lead

to

faster

global

operation

like

broadcasting

Our

results

indicate

that

bus

and

crossbar

organizations

can

provide

better

performance

improvement

for

broadcast

During

the

third

phase

we

evaluate

the

existing

broadcast

algorithm

-LRB-

U

mesh

designed

for

non-clustered

systems

-RRB-

to

find

out

whether

these

algorithm

are

sufficient

to

take

advantage

of

faster

communication

between

sibling

processors

We

demonstrate

that

they

are

not

sufficient

Thus

in

the

final

phase

we

propose

a

new

broadcast

algorithm

-LRB-

clus

mesh

-RRB-

to

deliver

better

performance

for

processor-clustered

systems

A

comparative

evaluation

indicates

that

the

new

algorithm

is

capable

of

delivering

up-to

better

performance

depending

on

the

relative

speed

of

intra

to

inter-cluster

communication

costs

This

study

provides

significant

insight

on

how

future

processorclusters

should

be

designed

for

supporting

distributed-memory

paradigms

Processor-cluster

based

systems

Computing

nodes

having

more

than

one

processor

on

a

single

multi-chip

module

or

processor-board

are

becoming

increasingly

available

In

the

recent

past

many

parallel

multiprocessor

systems

have

been

developed

using

such

processor-clusters

e.g.

the

CRAY

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

and

the

Stanford

DASH

-LSB-

-RSB-

Most

of

these

systems

are

two-level

architectures

as

shown

in

Figure

The

processor-clusters

are

interconnected

through

a

scalable

inter-cluster

network

e.g.

torus

mesh

The

cluster

configuration

can

vary

from

a

simple

star

connection

as

in

the

to

a

bus

interconnection

as

in

the

DASH

or

Paragon

Cluster

organizations

are

discussed

in

detail

in

the

next

section

The

number

of

processors

in

a

cluster

is

usually

small

ranging

from

However

with

advancements

in

VLSI

larger

clusters

are

expected

to

become

common

processorcluster

P

M

M

PR

RP

P

P

PM

processor

cluster

Cray

torus

DASH

Mesh

R

P

PM

processor

cluster

Intel

Paragon

Mesh

P

processor

M

memory

R

router

to

otherrouters

fromroutersother

injection/consumptionchannels

Figure

Recent

examples

of

processor-cluster

based

systems

A

cluster

is

connected

to

its

inter-cluster

network

router

through

a

cluster

interface

The

interface

offers

a

number

of

channels

to

and

from

the

network

router

These

channels

-LRB-

referred

to

as

injection/consumption

channels

-LSB-

-RSB-

-RRB-

are

used

to

inject/consume

messages

to/from

other

clusters

We

use

i

to

denote

the

number

of

injection

or

consumption

channels

per

cluster

Typically

most

systems

have

i

this

value

being

limited

by

the

maximum

available

router

pinout

-LSB-

-RSB-

Processor-cluster

organizations

Processor-cluster

based

architectures

have

been

motivated

by

the

convenience

of

packaging

multiple

processors

into

a

single

cluster

-LSB-

-RSB-

Details

about

the

topology

inside

the

cluster

the

memory

organization

impact

of

this

organization

on

system

performance

etc.

have

not

been

addressed

Thus

there

is

a

need

to

determine

whether

there

is

a

requirement

for

tightly

coupled

clusters

that

can

take

advantage

of

the

higher

integration

available

inside

processor-clusters

In

the

following

subsections

we

analyze

the

communication

delay

to

transfer

m

words

to

a

processor

's

own

memory

from

the

memory

of

another

processor

in

a

-RRB-

a

different

cluster

and

b

-RRB-

the

same

cluster

This

kind

of

operation

is

similar

to

the

basic

shmem

get

primitive

-LSB-

-RSB-

available

on

the

Cray

system

The

shmem

get

transfer

across

clusters

can

be

achieved

only

through

inter-processor

message

communication

This

requires

a

cost

of

-LRB-

ts

mtp

-RRB-

cycles

where

ts

denotes

the

messaging

startup

overhead

in

network

cycles

and

tp

denotes

the

per

word

transfer

cost

We

assume

the

currently

popular

wormhole

mechanism

for

routing

in

the

inter-cluster

Wormhole

routing

being

distance

insensitive

-LSB-

-RSB-

our

expression

for

cost

is

independent

of

a

distance

factor

Typical

value

of

ts

in

modern

systems

ranges

from

network

cycles

while

value

of

tp

ranges

from

cycles/word

-LSB-

-RSB-

The

inter-cluster

communication

cost

is

independent

of

the

cluster

organization

and

is

assumed

to

be

the

same

in

all

cluster

organizations

discussed

below

The

time

for

a

shmem

get

operation

from

a

memory

inside

the

same

cluster

depends

on

the

cluster

organization

Figure

shows

a

set

of

four

common

alternatives

for

cluster

organization

In

the

following

subsections

we

analyze

the

communication

delay

for

shmem

get

to

transferring

m

words

PMPM

i

injection

channels

network-clusterinterface

-LRB-

mux/demux

-RRB-

cprocessors

No

intra-cluster

topology

links

fromother

routerslinks

toother

routers

Router

Star

organization

PMPM

i

injection

channels

links

fromother

routerslinks

toother

routers

Router

PMPM

Direct

organization

The

processors

in

the

cluster

are

connected

through

a

direct

network

i

injection

channels

links

fromother

routerslinks

toother

routers

Router

PM

PM

PM

No

shared

access

to

processor

memory

by

other

cluster

processors

No

shared

access

to

processor

memory

by

other

cluster

processors

Intra-cluster

communication

possible

only

through

messaging

Intra-cluster

communication

possible

only

through

messaging

Bus

organization

The

processors

in

the

cluster

are

connected

through

a

shared

busShared

access

toprocessor

memory

by

other

cluster

processors

Intra-cluster

communication

possible

through

direct

access

to

memory

i

injection

channels

links

fromother

routerslinks

toother

routers

Router

Crossbar

organization

The

processors

are

connected

to

memories

through

a

crossbarShared

access

to

processor

memory

by

other

cluster

processors.Intra-cluster

communication

possible

through

direct

access

to

memory

PPPP

MMMM

Shared

Bus

Crossbar

cprocessors

cprocessors

cprocessors

Figure

Four

alternate

processor-cluster

organizations

Star

This

is

the

simplest

cluster

organization

All

processors

in

the

cluster

are

only

connected

to

the

cluster

interface

There

is

no

other

interconnection

between

the

processors

Thus

an

intra-cluster

communication

has

to

go

from

the

source

processor

through

an

injection

channel

into

the

network

router

and

then

come

back

to

the

destination

processor

through

a

consumption

channel

Such

communication

needs

to

be

implemented

as

a

message

undergoing

the

same

messaging

penalties

as

an

inter-cluster

message

Thus

the

time

required

for

an

intra-cluster

transfer

of

m

words

is

-LRB-

ts

mtp

-RRB-

cycles

The

above

observation

of

intra-cluster

cost

being

similar

to

inter-cluster

cost

was

verified

by

us

through

experimentation

on

a

-LRB-

processor

-RRB-

Cray

whose

nodes

are

star

clusters

of

processors

each

We

compared

the

time

for

a

shmem

get

operation

on

processor

from

two

different

processors

The

first

processor

was

within

the

same

cluster

as

while

the

second

processor

belonged

to

a

remote

cluster

The

result

of

this

comparison

is

depicted

in

Fig.

The

plots

show

the

average

access

time

per

word

versus

the

transfer

size

From

the

plots

we

observe

very

little

difference

in

the

cost

of

an

intra

and

inter-cluster

communication

of

the

same

size

thus

confirming

our

observation

To

estimate

the

overheads

of

messaging

we

also

conducted

an

experiment

to

determine

the

time

for

a

shmem

get

from

a

processor

's

own

local

memory

The

plot

for

the

average

access

time

per

word

from

this

experiment

is

also

shown

in

Fig.

as

a

to

transfer

Comparison

with

the

earlier

plots

revealed

a

drop

in

the

startup

overhead

ts

from

cycles

to

only

cycles

Similarly

the

per

word

cost

dropped

from

its

earlier

asymptotic

value

of

tp

cycles/word

to

cycles/word

This

indicates

that

being

able

to

access

memory

without

messaging

costs

has

potential

to

offer

much

faster

communication

This

is

exploited

in

the

tightly-coupled

bus

and

crossbar

organizations

To

distinguish

the

lower

startup

and

per

word

cost

while

accessing

memory

without

messaging

we

denote

this

startup

as

and

per

word

transfer

cost

as

where

ts

and

tp

Direct

In

this

organization

the

processors

inside

a

cluster

are

interconnected

by

a

point-point

direct

network

e.g.

a

mesh

as

shown

in

Fig.

As

in

the

star

cluster

a

processor

is

restricted

to

directly

access

only

its

own

memory

Accessing

the

memory

of

another

processor

in

the

same

cluster

has

to

be

implemented

through

message-passing

The

messaging

startup

overheads

are

expected

to

be

close

to

the

inter-cluster

overhead

of

ts

However

the

per

word

transfer

cost

inside

the

cluster

denoted

as

is

expected

to

be

lower

than

tp

due

to

the

higher

integration

The

cost

for

a

m

word

Latency

per

word

-LRB-

cycles

-RRB-

of

Transfer

Size

-LRB-

words

-RRB-

across

clusters

-LRB-

-RRB-

within

a

cluster

-LRB-

-RRB-

local

memory

-LRB-

-RRB-

Figure

Comparison

of

the

shmem

get

timings

for

three

different

accesses

a

-RRB-

processor

in

a

remote

cluster

b

-RRB-

processor

in

the

same

cluster

and

c

-RRB-

local

memory

intra-cluster

transfer

therefore

takes

-LRB-

ts

-RRB-

In

this

paper

we

assume

typical

values

of

to

network

cycles

per

word

Bus

In

this

organization

all

processors

inside

a

cluster

are

connected

through

a

shared

bus

Clearly

in

such

an

organization

the

maximum

bus

bandwidth

limits

the

number

of

processors

that

can

be

accommodated

in

a

cluster

Let

us

assume

that

within

a

cluster

a

processor

can

inject

a

word

every

cycles

The

maximum

rate

at

which

a

processor

can

inject

into

the

cluster

bus

is

therefore

words/cycle

Thus

the

cost

for

a

m

word

intra-cluster

transfer

over

the

cluster

bus

is

-LRB-

-RRB-

cycles

where

is

the

startup

overhead

for

a

memory

transfer

as

defined

earlier

Let

the

cluster

bus

bandwidth

be

denoted

as

Bbus

words/cycle

Such

a

bus

can

support

processor

injections

every

cycle

We

denote

the

expression

as

parameter

b

to

be

used

in

later

analysis

This

parameter

b

can

be

visualized

as

the

number

of

parallel

transfers

allowed

on

the

bus

Crossbar

Similar

to

the

bus

organization

the

crossbar

organization

also

allows

all

memory

inside

a

cluster

to

be

accessed

by

all

processors

in

the

same

cluster

Similar

to

the

bus

bandwidth

limitation

in

the

bus

organization

the

maximum

allowable

size

of

the

crossbar

can

limit

the

size

of

the

cluster

The

cost

for

a

m

word

intra-cluster

transfer

is

-LRB-

-RRB-

cycle

assuming

that

within

a

cluster

a

processor

can

inject

a

word

every

cycles

In

the

next

section

we

evaluate

these

cluster

organizations

for

efficiently

supporting

collective

communication

algorithms

which

occur

frequently

in

applications

In

this

paper

we

restrict

our

attention

to

the

broadcast

collective

communication

algorithm

However

the

ideas

presented

in

this

paper

can

be

applied

with

minor

modifications

to

other

collective

communication

operations

as

such

as

gather

reduction

and

barrier

synchronization

Evaluation

of

different

processor-cluster

organizations

The

U

mesh

algorithm

has

been

proposed

in

-LSB-

-RSB-

as

an

optimal

algorithm

to

achieve

multicasts

and

broadcasts

in

non-clustered

mesh

interconnected

systems

To

perform

a

multicast/broadcast

to

-LRB-

N

-RRB-

destinations

the

algorithm

orders

the

N

destinations

and

source

into

a

dimensionordered

chain

and

achieves

the

operation

in

Ne

contention-free

steps

We

extend

the

same

U

mesh

algorithm

for

broadcasting

to

clustered

meshes

To

have

contention-free

steps

we

label

the

clusters

in

the

system

on

a

dimension

ordered

chain

along

the

dimensions

of

the

inter-cluster

network

The

c

processors

in

a

cluster

labeled

i

are

numbered

from

ci

to

c

-LRB-

i

-RRB-

Deriving

the

time

for

broadcast

We

derive

expressions

for

the

time

taken

by

the

U

mesh

algorithm

to

achieve

a

broadcast

in

clustered

systems

with

different

cluster

organizations

The

U

mesh

algorithm

naturally

divides

into

two

phases

an

inter-cluster

phase

followed

by

intra-cluster

phase

in

each

cluster

In

the

intercluster

phase

the

broadcast

data

is

sent

to

one

processor

in

each

cluster

For

a

given

system

with

N

processors

and

cluster

size

c

there

are

N

c

clusters

This

phase

requires

-LRB-

N

ce

-RRB-

steps

of

inter-cluster

message

communication

Assuming

the

broadcast

data

consists

of

m

words

the

time

for

the

inter-cluster

phase

is

-LRB-

N

ce

-RRB-

-LRB-

ts

mtp

-RRB-

The

time

for

the

intra-cluster

phase

is

a

function

of

the

cluster

organization

For

the

four

different

cluster

organizations

discussed

earlier

the

third

column

in

Table

summarizes

the

time

required

for

this

phase

The

basic

approach

for

broadcasting

inside

a

cluster

is

to

employ

whenever

possible

a

U

mesh-like

divide

and

conquer

strategy

Let

us

derive

the

broadcasting

time

required

in

a

star

cluster

Recall

that

in

a

star

cluster

even

an

intra-cluster

communication

needs

to

go

through

the

network

router

using

an

injection

channel

Without

loss

of

generality

in

this

discussion

we

assume

i

and

c

to

be

powers

of

and

c

i

At

the

beginning

of

the

intra-cluster

phase

one

processor

in

the

cluster

has

the

data

In

step

using

one

injection

channel

the

data

is

transferred

to

a

second

processor

Similarly

in

step

the

two

processors

now

having

the

data

after

step

concurrently

transfer

it

to

two

more

processors

using

two

injection

channels

In

step

the

four

processors

now

having

the

data

after

step

concurrently

transfer

it

to

four

more

processors

using

two

injection

channels

Similarly

we

can

show

that

in

step

i

i

processors

having

the

data

concurrently

transfer

it

to

another

i

processors

using

i

injection

channels

Thus

after

i

steps

i

processors

have

the

data

while

c

i

processors

still

have

to

get

it

We

assign

one

injection

channel

to

each

of

the

i

processors

having

the

data

These

i

processors

now

broadcast

the

data

to

-LRB-

c

i

-RRB-

i

processors

each

using

the

injection

channel

assigned

to

them

Thus

this

part

requires

c

i

steps

The

total

number

of

intra-cluster

steps

used

are

-LRB-

i

c

i

-RRB-

and

the

time

required

is

-LRB-

i

c

i

-RRB-

-LRB-

ts

mtp

-RRB-

cycles

Table

The

time

required

in

network

cycles

for

the

inter

and

intra-cluster

phases

of

broadcasting

a

data

of

size

m

words

in

a

clustered

mesh

using

the

U

mesh

algorithm

The

number

of

clusters

in

the

system

is

N

c

The

time

for

the

intra-cluster

phase

of

the

broadcast

varies

with

the

cluster

organization

The

relationships

i

c

and

b

c

are

assumed

Cluster

Inter-cluster

Intra-cluster

organization

broadcast

time

broadcast

time

star

-LRB-

N

ce

-RRB-

-LRB-

ts

mtp

-RRB-

-LRB-

ie

dc

ie

-RRB-

-LRB-

ts

mtp

-RRB-

direct

-LRB-

N

ce

-RRB-

-LRB-

ts

mtp

-RRB-

ce

-LRB-

ts

-RRB-

bus

-LRB-

N

ce

-RRB-

-LRB-

ts

mtp

-RRB-

-LRB-

be

dc

be

-RRB-

-LRB-

-RRB-

crossbar

-LRB-

N

ce

-RRB-

-LRB-

ts

mtp

-RRB-

ce

-LRB-

-RRB-

The

number

of

steps

for

broadcasting

in

a

bus

organization

with

a

cluster

bus

supporting

at

most

b

parallel

transfers

can

be

similarly

derived

to

be

-LRB-

be

c

b

-RRB-

The

time

for

an

intra-cluster

communication

in

a

bus

organization

being

-LRB-

-RRB-

as

shown

earlier

the

time

for

a

broadcast

in

this

organization

requires

-LRB-

be

c

b

-RRB-

-LRB-

-RRB-

cycles

The

number

of

steps

for

broadcasts

in

direct

and

crossbar

organizations

following

a

U

mesh-like

algorithm

can

be

easily

derived

to

be

ce

Comparision

of

broadcast

times

Figure

depicts

plots

comparing

the

time

for

broadcasting

in

a

given

system

with

a

total

of

N

processors

for

different

cluster

organizations

The

impact

of

increasing

cluster

size

from

c

to

processors

number

of

injection

channels

per

router

from

i

to

and

system

size

to

N

processors

was

also

studied

From

Fig.

we

observe

that

as

the

cluster

size

is

increased

the

broadcasting

time

in

a

star

cluster

remains

constant

while

c

beyond

which

it

rises

sharply

This

happens

because

until

c

the

intra-cluster

broadcast

can

be

achieved

in

c

steps

leading

to

a

total

number

of

-LRB-

N

c

c

-RRB-

N

steps

a

constant

for

a

given

system

size

However

with

c

an

additional

-LRB-

dc

ie

-RRB-

steps

are

necessary

to

complete

the

broadcast

For

a

given

system

size

of

N

processors

an

increase

in

cluster

size

leads

to

a

fall

in

the

number

of

clusters

in

the

system

This

leads

to

fewer

steps

being

necessary

to

achieve

the

inter-cluster

phase

of

the

broadcast

This

gets

compensated

by

the

greater

number

of

steps

in

Latency

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

Umesh

i

N

star

direct

bus

xbar

Latency

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

Umesh

i

N

star

direct

bus

xbar

Latency

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

Umesh

i

N

star

direct

bus

xbar

Latency

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

Umesh

i

N

star

direct

bus

xbar

Latency

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

Umesh

i

N

star

direct

bus

xbar

Latency

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

Umesh

i

N

star

direct

bus

xbar

Figure

Comparing

the

performance

of

U

Mesh

broadcast

algorithm

on

four

different

clustered

organizations

as

the

cluster

size

is

increased

from

c

The

total

number

of

processors

in

the

system

was

maintained

at

N

and

N

in

the

first

and

second

rows

of

plots

respectively

Typical

values

of

ts

t

s

ts

tp

tp

t

;p

b

and

m

flits

were

assumed

the

intra-cluster

network

Among

the

cluster

organizations

the

direct

bus

and

crossbar

offer

a

reduced

intra-cluster

communication

time

over

the

inter-cluster

communication

This

gain

is

higher

in

the

tightly-coupled

bus

and

crossbar

organizations

because

ts

These

organizations

therefore

demonstrate

a

substantial

fall

in

the

broadcast

time

as

the

cluster

size

is

increased

For

a

bus

cluster

the

cluster

bus

bandwidth

can

limit

this

gain

in

performance

The

depicted

plots

in

Fig.

were

derived

for

b

For

cluster

sizes

higher

than

this

limitation

becomes

evident

from

the

plots

as

the

bus

organization

can

not

match

the

crossbar

performance

Observations

similar

to

those

made

above

were

also

made

with

a

larger

system

size

of

N

processors

From

the

plots

we

also

observe

that

the

tightly

coupled

bus

and

crossbar

configurations

consistently

outperform

the

direct

and

star

configurations

by

a

margin

of

to

This

demonstrates

that

tightly-coupled

cluster

configurations

are

indeed

necessary

to

offer

the

faster

intra-cluster

communication

required

to

fully

exploit

application

locality

and

fully

harness

the

power

of

processor-cluster

based

systems

Broadcast

on

clustered

meshes

In

the

last

section

we

demonstrated

that

having

faster

intra-cluster

communication

leads

to

better

performance

for

broadcast

operations

It

was

recommended

that

clustered

systems

should

be

designed

with

cluster

organizations

like

the

bus

or

crossbar

allowing

processors

to

directly

access

the

memory

of

other

processors

within

the

same

cluster

In

this

section

we

demonstrate

that

with

such

differential

communication

costs

existing

algorithms

designed

to

be

optimal

under

the

assumption

of

flat

communication

cost

may

not

remain

optimal

in

clustered

systems

We

analyze

the

U

Mesh

-LSB-

-RSB-

algorithm

for

multicasting/broadcasting

in

terms

of

the

required

number

of

intercluster

and

intra-cluster

steps

We

propose

an

alternate

clus

mesh

algorithm

that

requires

more

overall

steps

but

outperforms

the

U

Mesh

algorithm

under

the

assumption

of

faster

intra-cluster

communication

Single

port

model

with

multiple

injection

channels

In

section

it

was

shown

that

current

and

future

router

pinout

technologies

can

support

more

than

one

injection

and

consumption

channel

from

a

processor-cluster

to

a

router

With

such

pinout

technologies

i

injection/consumption

channels

will

be

commonly

available

in

clustered

systems

The

Cray

currently

supports

i

from

each

processor-cluster

In

literature

-LSB-

-RSB-

injection

channels

have

often

been

referred

to

as

communication

ports

In

this

discussion

although

we

have

multiple

injection

channel

per

cluster

we

allow

a

processor

to

be

injecting

at

most

one

message

at

a

time

Thus

from

a

single

processor

's

perspective

ours

is

not

a

multi-port

model

However

in

a

cluster

with

i

injection

channels

up

to

i

processors

within

a

cluster

can

concurrently

inject

a

message

each

into

the

router

through

separate

injection

channels

The

one-port

model

reflects

the

limitation

that

can

arise

from

the

limited

memory

bandwidth

available

for

message

injection

When

a

message

is

injected

the

memory

needs

to

be

read

at

a

rate

matching

the

network

channel

speed

Otherwise

the

network

channels

can

remain

under-utilized

leading

to

a

fall

in

network

performance

-LSB-

-RSB-

In

this

paper

we

assume

memory

bandwidth

per

processor

to

support

only

one

message

injection

that

is

a

single-port

model

However

different

processors

in

a

cluster

can

inject

messages

concurrently

since

these

are

supported

from

different

memories

Illustrating

U

mesh

in

terms

of

inter

and

intra-cluster

steps

The

U

mesh

algorithm

achieves

a

contention-free

broadcast

in

n-dimensional

meshes

A

broadcast

in

a

system

with

N

processors

is

achieved

in

Ne

steps

The

input

to

the

U

mesh

algorithm

is

a

dimension-ordered

chain

-LSB-

-RSB-

of

all

the

processors

Let

us

analyze

the

application

of

the

U

mesh

algorithm

for

achieving

a

broadcast

in

a

system

with

clusters

each

having

c

processors

as

shown

in

Fig.

In

the

remaining

discussion

we

denote

an

inter-cluster

communication

latency

as

S

-LRB-

for

slow

-RRB-

cycles

and

an

intra-cluster

communication

latency

as

F

-LRB-

for

fast

-RRB-

cycles

We

denote

the

latency

to

perform

a

broadcast

in

a

cluster

when

one

processor

within

the

cluster

already

has

the

data

as

parameter

B.

Representative

expressions

for

B

for

different

cluster

organizations

were

presented

in

Table

For

example

B

inside

a

crossbar

based

cluster

requires

-LRB-

c

-RRB-

F

Thus

for

a

cluster

with

c

processors

B

cycles

In

the

example

broadcast

in

Fig.

-LRB-

a

-RRB-

the

source

processor

is

in

the

center

cluster

while

in

the

broadcast

in

Fig.

-LRB-

b

-RRB-

the

source

processor

is

not

in

the

center

cluster

U

Mesh

requires

steps

to

complete

the

broadcast

in

both

examples

Each

arc

representing

a

communication

is

numbered

by

the

step

in

which

it

occurs

and

by

S

or

F

depicting

inter

or

intra-cluster

communication

respectively

The

completion

times

are

derived

by

tracing

the

longest

arc-path

to

the

last

destination

As

derived

in

Fig.

the

completion

time

for

the

broadcast

with

source

in

center

cluster

is

cycles

while

for

the

broadcast

with

source

not

in

the

center

cluster

the

completion

time

is

cycles

All

communications

in

the

same

step

are

contention-free

The

U

mesh

algorithm

can

also

be

applied

to

higher

dimensional

mesh

of

clusters

Figure

depicts

the

same

broadcasts

as

in

Fig.

on

a

mesh

of

clusters

It

can

be

shown

that

the

communications

within

the

same

step

are

contention-free

leading

to

similar

broadcast

times

as

in

the

case

of

the

linear

array

2301 6745 101189

F

S

Umesh

broadcast

algorithm

in

clusters

with

source

in

center

cluster

requires

F

S

B

cycles

S

S

S

S

S

S

The

last

steps

for

broadcasting

within

a

cluster

have

not

been

shown

For

c

this

time

is

B

cycles

2301 6745 101189

Umesh

broadcast

algorithm

in

clusters

with

source

in

center

cluster

requires

B

cycles

S

S

S

S

S

S

S

source

source

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Example

demonstrating

the

communication

steps

to

perform

a

broadcast

using

the

U

mesh

algorithm

in

a

cluster

linear

array

-LRB-

mesh

-RRB-

The

source

in

figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

is

chosen

as

processor

and

respectively

Each

cluster

is

assumed

to

have

two

injection

channels

All

communication

steps

are

contention-free

S

S

S

S

S

S

S

-LRB-

a

-RRB-

-LRB-

b

-RRB-

source

source

S

S

S

xy

S

Figure

Example

demonstrating

the

communication

steps

to

perform

a

broadcast

using

the

U

mesh

algorithm

in

a

mesh

of

clusters

The

source

in

figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

is

chosen

as

processor

and

respectively

Each

cluster

is

assumed

to

have

two

injection

channels

All

communication

steps

are

contention-free

and

the

broadcast

times

are

similar

to

those

obtained

on

a

cluster

linear

array

Usage

of

injection

channels

by

U

mesh

From

Figures

-LRB-

b

-RRB-

and

-LRB-

b

-RRB-

it

can

be

observed

that

U

mesh

mostly

uses

only

one

injection

channel

from

each

cluster

The

U

mesh

broadcast

can

be

divided

into

two

distinct

phases

In

the

first

phase

referred

to

by

us

as

the

inter-cluster

phase

one

processor

in

each

cluster

is

reached

This

is

followed

by

the

second

phase

the

intra-cluster

phase

where

the

remaining

processors

are

reached

Two

injection

channels

are

used

concurrently

by

U

mesh

only

in

the

special

case

when

the

source

lies

in

the

center

cluster

as

shown

in

Figures

-LRB-

a

-RRB-

-LRB-

a

-RRB-

However

this

usage

is

limited

only

to

the

center

cluster

This

leads

us

to

the

following

observation

Observation

The

U

mesh

algorithm

for

broadcasting

when

applied

to

clustered

systems

does

not

effectively

use

more

than

one

injection

channel

from

each

cluster

Illustrating

clus

mesh

We

propose

a

new

broadcast

algorithm

clus

mesh

for

clustered

systems

which

effectively

exploits

a

-RRB-

faster

intra-cluster

communication

-LRB-

F

S

-RRB-

and

b

-RRB-

more

injection

channels

from

each

cluster

The

algorithm

presented

in

this

paper

is

designed

for

systems

with

two

injection

channels

per

cluster

However

a

similar

approach

can

be

used

to

design

an

algorithm

for

systems

with

four

injection

per

processor

We

first

convey

the

main

idea

behind

the

clus

mesh

algorithm

using

the

following

example

Figure

depicts

the

communications

performed

to

achieve

broadcast

in

a

cluster

linear

array

from

processor

in

the

center

cluster

In

the

first

step

processor

performs

an

intracluster

communication

to

involve

another

processor

in

the

same

cluster

In

step

processors

and

perform

inter-cluster

communications

to

processors

-LRB-

in

-RRB-

and

-LRB-

in

-RRB-

respectively

In

step

both

processors

and

perform

an

intra-cluster

communication

to

involve

another

processor

in

their

own

clusters

respectively

In

the

same

step

processors

and

perform

intercluster

communications

to

processors

-LRB-

in

-RRB-

and

-LRB-

in

-RRB-

respectively

In

step

cluster

communicates

with

and

concurrently

while

communicates

with

and

The

last

clusters

to

be

reached

and

perform

an

intra-cluster

broadcast

using

two

more

intra-cluster

steps

The

total

broadcast

time

as

derived

in

Fig.

is

B

cycles

-LRB-

assuming

B

-RRB-

Let

us

compare

this

to

the

cycles

for

a

similar

broadcast

using

U

mesh

as

shown

earlier

in

Fig.

-LRB-

a

-RRB-

We

derive

that

the

clus

mesh

outperforms

U

mesh

if

S

F

The

main

idea

behind

clus

mesh

for

a

system

with

injection

channels

is

as

follows

The

first

processor

in

any

cluster

to

receive

the

broadcast

data

performs

an

intra-cluster

communication

to

involve

one

more

processor

in

the

same

cluster

The

two

processors

then

exploit

the

two

injection

channels

to

send

out

two

concurrent

inter-cluster

messages

in

two

different

directions

to

cover

two

more

clusters

The

potential

gain

arises

from

the

faster

step

within

the

cluster

which

allows

more

clusters

to

be

covered

in

lesser

time

The

algorithm

offers

contention-free

performance

if

started

from

the

center

cluster

Thus

when

the

source

processor

is

not

in

the

center

cluster

an

extra

inter-cluster

communication

is

required

to

send

the

broadcast

data

to

the

center

cluster

The

total

time

required

in

the

example

shown

in

Fig.

when

the

source

cluster

is

not

in

center

is

therefore

F

cycles

Comparing

this

to

the

F

cycles

for

a

similar

broadcast

using

U

mesh

derived

earlier

in

Fig.

-LRB-

b

-RRB-

it

can

be

observed

that

the

clus

mesh

outperforms

U

mesh

if

S

We

also

observe

that

the

total

number

of

steps

obtained

by

adding

the

S

and

F

factors

may

be

higher

for

the

clus

mesh

algorithm

Typically

clus

mesh

has

fewer

S

steps

and

more

F

steps

compared

to

the

U

mesh

expression

However

with

S

F

clus

mesh

has

potential

to

offer

a

smaller

broadcast

time

2301 6745 101189

S

S

S

F

F

F

S

S

S

S

S

source

broadcast

algorithm

on

clusters

starting

from

center

cluster

requires

F

S

F

S

B

cycles

For

a

source

not

in

the

center

cluster

an

extra

S

step

is

necessary

to

initially

bring

the

data

to

a

processor

in

the

center

cluster

The

required

time

in

that

case

is

B.

cycles

Figure

Example

demonstrating

the

communication

steps

to

perform

a

broadcast

using

the

novel

clus

mesh

algorithm

in

a

cluster

linear

array

-LRB-

mesh

-RRB-

Each

cluster

is

assumed

to

have

two

injection

channels

All

communication

steps

are

contention-free

The

clus

mesh

algorithm

Let

us

consider

the

algorithm

for

clus

mesh

on

a

linear

array

with

NC

clusters

The

broadcast

data

is

first

sent

to

the

center

cluster

The

linear

array

is

then

recursively

partitioned

into

parts

At

each

recursive

step

we

apply

the

algorithm

presented

in

the

example

in

Fig

for

broadcasting

to

clusters

The

algorithm

in

Fig.

being

contention-free

and

the

communications

in

the

recursive

partitions

not

interfering

with

each

other

the

overall

algorithm

is

also

contentionfree

For

partitions

smaller

than

clusters

Table

summarizes

the

minimal

cost

for

achieving

a

broadcast

in

these

special

cases

The

clus

mesh

algorithm

is

easily

extendible

to

higher

dimensional

meshes

of

clusters

For

example

Fig.

demonstrates

the

basic

partitioning

in

a

mesh

The

time

complexity

of

the

clus

mesh

algorithm

can

be

easily

shown

to

be

-LRB-

S

-LRB-

F

-RRB-

NC

B

-RRB-

The

time

complexity

for

a

similar

broadcast

using

U

mesh

is

-LRB-

S

NC

B

-RRB-

Thus

clus

mesh

outperforms

U

mesh

only

if

the

condition

-LRB-

S

-LRB-

-RRB-

NC

-RRB-

-LRB-

S

NC

-RRB-

is

satisfied

by

the

respective

values

of

S

and

F

We

propose

a

hybrid

clustered

algorithm

which

for

a

given

system

size

and

values

for

S

and

F

evaluates

the

time

for

both

clus

mesh

and

U

mesh

and

uses

the

faster

Table

Cost

of

a

broadcast

in

a

linear

array

of

fewer

than

clusters

The

parameters

S

and

F

denote

inter

and

intra-cluster

communication

costs

respectively

in

network

cycles

while

the

parameter

B

denotes

cost

of

broadcast

inside

a

cluster

Number

Source

Source

does

not

of

belongs

to

belong

to

Clusters

center

cluster

center

cluster

S

B

S

B

S

F

B

B

B

B

F

B

F

B

F

B

B

F

B

B

B

B

algorithm

While

comparing

the

performance

of

U

mesh

versus

the

new

algorithm

we

present

results

derived

using

the

hybrid

approach

S

S

F

F

S

S

S

S

S

Perform

the

recursive

broadcast

in

each

sub-partition

Figure

Example

demonstrating

the

communication

steps

to

perform

a

broadcast

using

the

recursive

clus

mesh

algorithm

in

a

mesh

of

clusters

Each

cluster

is

assumed

to

have

two

injection

channels

All

communication

steps

are

contention-free

Comparison

of

Clus

mesh

and

U

mesh

performance

In

this

section

we

compare

the

performance

of

Clus

Mesh

and

U

mesh

algorithms

for

various

system

sizes

cluster

sizes

and

different

values

of

S

and

F

Impact

of

increasing

number

of

clusters

Figure

compares

the

time

taken

for

clus

mesh

and

U

mesh

to

achieve

a

broadcast

on

a

system

having

NC

clusters

The

parameters

NC

was

varied

from

clusters

to

study

the

impact

of

increasing

number

of

clusters

The

experiment

was

performed

for

three

different

relative

values

of

S

and

F

S

F

and

As

discussed

earlier

the

clus

mesh

achieves

a

trade-off

by

reducing

number

of

S

steps

with

increased

F

steps

This

trade-off

pays

off

when

the

S

F

is

above

a

certain

value

From

Fig.

it

can

be

observed

that

for

S

F

the

clus

mesh

is

not

able

to

deliver

significantly

better

performance

than

U

mesh

As

S

F

is

increased

to

the

gains

from

clus

mesh

become

apparent

The

relative

gain

over

U

mesh

was

observed

to

be

up

to

for

a

wide

range

of

system

sizes

These

gains

are

further

increased

with

a

higher

S

F

The

relative

gain

over

U

mesh

in

this

case

was

observed

to

be

up

to

for

a

wide

range

of

system

sizes

We

also

observed

that

the

gains

are

higher

as

the

number

of

clusters

is

increased

200 400 600 800

Latency

for

broadcast

-LRB-

cycles

-RRB-

Number

of

clusters

-LRB-

-RRB-

i

c

S

F

umesh

S/F

S/F

200 400 600 800

Latency

for

broadcast

-LRB-

cycles

-RRB-

Number

of

clusters

-LRB-

-RRB-

i

c

S

F

umesh

S/F

S/F

200 400 600 800

Latency

for

broadcast

-LRB-

cycles

-RRB-

Number

of

clusters

-LRB-

-RRB-

i

c

S

F

umesh

S/F

S/F

Figure

Comparing

the

performance

of

clus

mesh

and

U

mesh

broadcast

algorithms

with

the

number

of

clusters

in

the

system

being

increased

from

NC

The

cluster

size

is

kept

fixed

at

c

The

three

plots

depict

results

for

three

different

values

of

inter-cluster

communication

step

-LRB-

S

-RRB-

to

intra-cluster

communication

step

cost

-LRB-

F

-RRB-

of

S

F

and

respectively

Impact

of

increasing

the

cluster

size

We

first

analyzed

the

impact

of

increasing

cluster

size

while

keeping

the

number

of

clusters

in

the

system

fixed

These

results

are

presented

in

Figure

The

number

of

clusters

was

kept

fixed

at

NC

It

can

be

observed

that

the

relative

performance

remains

almost

constant

as

the

cluster

size

is

increased

from

This

is

because

in

both

clus

mesh

and

U

mesh

the

cluster

size

only

affects

the

time

taken

towards

the

end

of

the

respective

algorithms

to

achieve

the

intra-cluster

broadcasts

This

time

denoted

by

B

earlier

is

same

for

both

algorithms

Latency

for

broadcast

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

i

S

F

umesh

Figure

Comparing

the

performance

of

Clus

mesh

and

U

Mesh

broadcast

algorithms

with

increasing

cluster

size

The

number

of

clusters

is

kept

fixed

at

NC

An

inter-cluster

communication

step

of

S

cycles

and

intra-cluster

communication

step

of

F

cycles

are

assumed

We

also

analyzed

the

impact

of

increasing

cluster

size

while

keeping

the

total

number

of

processors

in

the

system

fixed

at

N

These

results

are

presented

in

Figure

As

the

cluster

size

is

increased

the

number

of

clusters

in

the

system

falls

This

can

leads

to

a

fall

in

the

relative

gains

of

clus

mesh

over

U

mesh

as

depicted

in

Fig.

However

we

observed

that

for

realistic

cluster

sizes

and

large

system

sizes

as

expected

in

the

near

future

clus

mesh

outperforms

U

mesh

Latency

for

broadcast

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

i

N

S

F

umesh

Latency

for

broadcast

-LRB-

cycles

-RRB-

cluster

size

-LRB-

c

-RRB-

i

N

S

F

umesh

Figure

Comparing

the

performance

of

Clus

Mesh

and

U

Mesh

broadcast

algorithms

The

impact

of

increasing

cluster

size

is

shown

in

two

different

systems

with

N

and

processors

An

inter-cluster

communication

step

of

S

cycles

and

intra-cluster

communication

step

of

F

cycles

are

assumed

Additional

Information

A

number

of

related

papers

and

technical

reports

are

available

electronically

through

the

home

page

of

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

The

URL

is

Alternatively

the

papers

can

be

obtained

by

anonymous

ftp

from

ftp.cis.ohio-state

edu

cd

pub/communication

Conclusion

In

this

paper

we

have

analyzed

various

cluster

organizations

for

designing

processor-cluster

based

multiprocessor

systems

Using

the

broadcast

collective

communication

operation

we

have

demonstrated

that

there

is

a

requirement

for

tightly

coupled

clusters

that

can

take

advantage

of

the

higher

integration

available

inside

processor-clusters

With

differential

communication

costs

in

the

intraand

inter-cluster

we

also

demonstrated

that

the

U

mesh

algorithm

does

not

remain

optimal

on

clustered

systems

We

have

proposed

a

new

algorithm

Clus

mesh

for

broadcasting

on

clustered

meshes

which

outperforms

U

mesh

under

the

assumption

of

faster

intra-cluster

communication

In

this

paper

we

have

only

discussed

about

the

broadcast

operation

on

clustered

meshes

However

similar

ideas

can

be

extended

to

other

collective

communication

operations

like

gather

reduction

and

barrier

synchronization

These

operations

can

be

treated

as

reverse

broadcast

or

combination

of

broadcast

and

reverse

broadcast

operations

In

the

next

phase

of

this

research

we

are

designing

algorithms

for

implementing

such

collective

communication

operations

efficiently

on

clustered

systems

We

are

also

investigating

the

best

way

to

extend

the

Clus

mesh

algorithm

for

systems

with

more

than

two

injection

channels

per

cluster

References

-LSB-

-RSB-

A.

Asthana

H.

Jagdish

and

B.

Mathews

Impact

of

Advanced

VLSI

packaging

on

the

design

of

a

large

parallel

computer

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Alleviating

Consumption

Channel

Bottleneck

in

Wormhole-Routed

k-ary

n-cube

Systems

Technical

Report

Department

of

Computer

and

Information

Science

The

Ohio

State

University

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Clustered

Multiprocessor

Systems

under

Packaging

and

Technological

Advancements

Technical

Report

Department

of

Computer

and

Information

Science

The

Ohio

State

University

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Benefits

of

Processor

Clustering

in

Designing

Large

Parallel

Systems

When

and

How

In

Proc

of

the

Int

Parallel

Processing

Symposium

Apr

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

W.

J.

Dally

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Trans

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

A.

Erlichson

et

al.

The

benefits

of

clustering

in

shared

address

space

mutliprocessors

An

application-driven

investigation

Technical

Report

Stanford

University

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

C.

T.

Ho

and

S.

L.

Johnsson

Distributed

Routing

Algorithms

for

Broadcasting

and

Personalized

Communication

in

Hypercubes

In

Proc

International

Conference

on

Parallel

processing

pages

-LCB-

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Impact

of

Wiring

Constraints

on

Hierarchical

Network

Performance

In

Proc

of

the

Int

Parallel

Processing

Symposium

Mar

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Net

works

IEEETPDS

-LRB-

-RRB-

-LCB-

Dec

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

M.

T.

Raghunath

and

A.

Ranade

Designing

interconnection

networks

for

multi-level

packaging

In

Proc

of

the

Supercomputing

pages

-LCB-

O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O

broadcasting
collective communication
interconnection networks

clustered architectures
hierarchical architectures
parallel architectures