Designing

Clustered

Multiprocessor

Systems

under

Packaging

and

Technological

Advancements

Debashis

Basak

and

Dhabaleswar

K.

Panda

Department

of

Computer

and

Information

Science

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

fbasak,pandag@cis.ohio-state.edu

Abstract

Packaging

technologies

impose

various

physical

constraints

on

bisection

bandwidth

and

channel

width

of

a

system

whereas

processor

and

interconnect

technologies

lead

to

certain

throughput

and

latency

demands

on

the

system

performance

Earlier

studies

in

literature

have

either

focused

on

only

flat

interconnections

or

proposed

hierarchical/clustered

interconnections

with

limited

packaging

constraints

Pinout

technologies

and

capacity

of

packaging

modules

have

been

ignored

often

leading

to

configurations

which

are

not

design-feasible

In

this

paper

we

propose

a

new

supply-demand

framework

for

multiprocessor

system

design

by

considering

packaging

processor

and

interconnect

technologies

in

an

integrated

manner

The

elegance

of

this

framework

lies

in

its

parameterized

representation

of

different

technologies

For

a

given

set

of

technological

parameters

the

framework

derives

the

best

configuration

while

considering

practical

design

aspects

like

maximum

board

area

maximum

available

pinout

fixed

channel

width

and

scalability

In

order

to

build

a

scalable

parallel

system

with

a

given

number

of

processors

the

framework

explores

the

design

space

of

flat

k-ary

n-cube

topologies

and

their

clustered

variations

-LRB-

k-ary

n-cube

cluster-c

-RRB-

to

derive

design-feasible

configurations

with

best

system

performance

The

study

identifies

processor

board

area

supported

channel

width

board

pinout

density

and

router

pinout

as

critical

parameters

and

analyzes

their

impact

on

deriving

design-feasible

and

best

configurations

For

a

wide

range

of

parameters

it

is

shown

that

best

configurations

are

achieved

with

cluster-based

systems

with

processors

per

cluster

and

inter-cluster

interconnection

research

is

supported

by

the

National

Science

Foundation

Grant

Contents

Introduction

Designing

Systems

with

k-ary

n-cube

cluster-c

Organization

k-ary

n-cube

cluster-c

Organization

Architectural

Alternatives

and

Choices

Demand

on

Network

Performance

Effect

of

Processor

and

Interconnect

Speed

on

Demanded

System

Throughput

Demanded

Bound

on

Average

Network

Latency

Packaging

Model

for

Clustered

Systems

Parameterizing

Packaging

Technologies

Processor

Board

Technology

Board

Pinout

Technologies

Channel

Width

Technology

Router

Pinout

Technology

Impact

of

Packaging

on

Architectural

Parameters

Offered

Inter-Cluster

Channel

Width

Supporting

a

Fixed

Inter-Cluster

Channel

Width

Designing

with

Under

Utilization

of

Resources

Offered

Inter-Cluster

Bisection

Latency-Throughput

Performance

Model

for

k-ary

n-cube

cluster-c

Systems

The

Model

Comparing

Performance

of

the

Model

with

Simulations

Design

Framework

Methodology

Applying

the

Framework

Deriving

Design-Feasible

Configurations

Deriving

Good

Configurations

Deriving

the

Best

Configuration

Impact

of

Varying

Packaging

and

Demand

Parameters

on

the

Design

Process

Processor

and

Interconnect

Technology

Increasing

Supported

Channel

Width

Increasing

Board

Pinout

Density

Increasing

the

Maximum

Router

Pinout

Maximum

Board

Capacity

Surface

Pinout

Technology

Conclusions

and

Future

Work

Introduction

Rapid

developments

in

the

field

of

processor

interconnect

and

packaging

technologies

make

the

task

of

efficient

design

of

large

multiprocessor

systems

a

difficult

one

-LSB-

-RSB-

Design

guidelines

need

to

take

into

account

technological

changes

to

yield

system

configurations

delivering

best

performance

Several

previous

studies

have

considered

packaging

constraints

-LSB-

-RSB-

while

selecting

the

best

system

configuration

These

studies

include

Dally

's

-LSB-

-RSB-

analysis

of

k-ary

n-cube

interconnection

under

a

VLSI

model

with

constant

bisection

bandwidth

and

Abraham

and

Padmanabhan

's

study

-LSB-

-RSB-

under

a

constant

pinout

from

a

processing

node

Agrawal

's

-LSB-

-RSB-

analysis

of

k-ary

n-cube

networks

considers

three

different

constraints

constant

bisection

width

constant

channel

width

and

constant

pinout

while

considering

node

and

wire

delays

However

Ranade

-LSB-

-RSB-

and

Yew

-LSB-

-RSB-

have

argued

that

neither

Dally

's

VLSI

model

with

limited

bisection

bandwidth

nor

the

limited

pinout

model

as

proposed

in

-LSB-

-RSB-

is

adequate

while

designing

very

large

systems

Both

models

confine

to

only

one

level

of

packaging

hierarchy

whereas

large

systems

typically

employ

several

levels

of

packaging

It

was

demonstrated

in

-LSB-

-RSB-

that

while

designing

a

large

multiprocessor

system

with

multiple

levels

of

packaging

multi-level/hierarchical/clustered

architecture

can

be

an

advantage

The

architectural

levels

can

be

chosen

to

closely

match

the

packaging

hierarchy

leading

to

better

designs

A

multi-level

architecture

also

widens

the

design

space

in

terms

of

alternative

configurations

possible

to

build

a

system

with

a

given

number

of

processors

A

variety

of

two-level

hierarchical

configurations

have

been

proposed

by

researchers

in

the

past

to

build

scalable

systems

Though

a

system

design

with

multiple

levels

is

a

more

generalized

approach

it

is

commonly

believed

that

two

levels

are

sufficient

to

build

parallel

systems

in

the

near

future

-LSB-

-RSB-

Moreover

the

design

techniques

which

work

for

two-level

hierarchies

can

be

easily

extended

to

accommodate

more

levels

Examples

of

previous

work

in

this

area

include

two-level

systems

based

on

hypercube

and

other

network

topologies

-LSB-

-RSB-

MINs

and

n-hop

networks

-LSB-

-RSB-

and

combination

of

bus

and

mesh/hypercube

networks

-LSB-

-RSB-

Though

these

designs

provide

alternative

ways

to

build

parallel

systems

most

of

them

do

not

take

any

packaging

constraints

into

account

Thus

it

is

difficult

to

use

these

configurations

to

build

realistic

systems

under

varying

technological

and

packaging

constraints

A

typical

hierarchy

used

in

packaging

a

large

system

consists

of

multiple

chips

on

a

board

and

multiple

such

boards

in

a

card-cage

A

larger

system

may

require

multiple

card-cages

multiple

cabinets

and

so

on

The

modules

at

each

level

of

this

packaging

hierarchy

chips

boards

card-cages

etc.

have

their

own

characteristics

in

terms

of

maximum

capacity

bisection

size

available

pinout

and

channel

width

For

example

the

maximum

board

size

available

may

be

limited

to

a

size

of

The

pinout

from

a

board

may

be

limited

to

pins

The

pinout

from

a

router

chip

may

be

limited

to

a

maximum

of

pins

Factors

like

path-width

inside

routers

and

connector

technology

restrict

channel

widths

from

being

arbitrarily

large

For

designing

a

simpler

communication

interface

the

width

of

data

lines

in

a

channel

is

also

expected

to

maintain

an

integral

relationship

with

that

of

processor

and

memory

which

are

typically

in

multiples

of

a

byte

Such

packaging

limitations

impose

constraints

on

the

design

space

For

example

a

configuration

requiring

a

pinout

of

from

each

router

chip

can

not

be

supported

under

the

router

pinout

limitation

mentioned

above

Similarly

a

configuration

requiring

sixty-four

bit

channels

from

a

board

can

not

be

supported

under

the

board

pinout

constraint

Based

on

such

packaging

constraints

only

a

small

set

of

theoretically

possible

configurations

are

package-able

or

design-feasible

Only

these

configurations

can

translate

into

real

machines

Thus

a

system

architect

needs

to

first

identify

a

set

of

design-feasible

configurations

offered

-LRB-

supplied

-RRB-

by

a

given

set

of

packaging

technologies

before

choosing

on

a

configuration

to

build

a

system

Besides

design-feasibility

processor

and

application

characteristics

also

lead

to

an

expected

minimal

demand

on

performance

from

the

system

in

terms

of

sustained

average

throughput

and

maximum

allowable

message

latency

Sustaining

a

desired

performance

places

a

demand

on

the

latency-throughput

characteristics

of

a

system

configuration

-LSB-

-RSB-

Thus

an

architect

should

identify

such

demand

on

performance

and

use

it

as

a

criterion

to

select

good

configurations

satisfying

the

demand

However

the

demanded

performance

is

not

a

fixed

parameter

It

is

a

function

of

processor

and

interconnection

speeds

For

example

faster

processors

can

compute

quicker

leading

to

faster

injection

of

messages

into

the

network

To

sustain

the

faster

computation

rate

the

demanded

latency

and

sustained

average

throughput

need

to

be

supportable

by

the

underlying

communication

network

For

example

a

given

good

design-feasible

configuration

for

MHz

processor

system

need

not

remain

good

when

processor

technology

changes

to

MHz

However

by

doubling

the

channel

speed

the

configuration

can

be

made

good

for

MHz

processors

Design

results

presented

in

previous

studies

have

not

taken

into

account

such

impact

of

processor

and

interconnect

technology

on

the

demanded

performance

Thus

a

design

framework

must

consider

packaging

processor

and

interconnect

technologies

in

an

integrated

manner

to

propose

realistic

design

solutions

In

this

paper

we

propose

such

an

integrated

supply-demand

framework

for

multiprocessor

system

design

The

overall

framework

is

summarized

in

Fig.

The

basic

objective

of

the

framework

is

to

design

the

best

system

configuration

for

a

given

number

-LRB-

N

-RRB-

of

processors

using

a

two-level

architecture

The

family

of

flat

k-ary

n-cube

topologies

and

their

clustered

variations

-LRB-

k-ary

n-cube

cluster-c

-RRB-

are

considered

to

derive

scalable

configurations

Theoretically

a

large

number

of

alternate

configurations

are

possible

to

build

such

a

system

We

consider

several

packaging

constraints

such

as

varying

board

sizes

reasonable

channel

widths

in

multiples

of

a

byte

limited

pins

from

a

router

chip

and

limited

pinout

from

board

depending

on

its

size

The

set

of

design-feasible

configurations

conforming

to

the

packaging

constraints

are

first

derived

The

demanded

performance

-LRB-

latency

and

throughput

-RRB-

are

then

derived

for

a

given

processor

and

interconnect

technology

The

latencythroughput

performance

characteristics

offered

by

each

design-feasible

configuration

is

estimated

through

analytical

modeling

and

compared

to

the

demanded

performance

Among

the

good

designfeasible

configurations

satisfying

the

demanded

performance

the

best

configuration

is

decided

on

the

basis

of

cost-effectiveness

and

scope

for

scalability

The

elegance

of

the

framework

lies

in

its

parameterized

representation

of

different

technologies

The

best

configuration

can

be

derived

for

any

set

of

technologies

and

constraints

by

choosing

appropriate

values

for

the

parameters

We

illustrate

the

framework

by

deriving

best

configurations

to

design

a

processor

system

Using

the

framework

we

also

study

the

impact

of

various

packaging

parameters

and

demanded

performance

on

the

overall

design

process

Among

others

the

effect

of

varying

maximum

board

size

higher

router

pinout

and

alternate

board

pinout

technology

on

the

set

of

design-feasible/good/best

configuration

is

analyzed

and

suitable

design

guidelines

are

derived

to

aid

an

architect

in

the

design

process

For

a

wide

range

of

parameters

it

is

shown

that

best

configurations

are

achieved

with

processors

per

cluster

and

inter-cluster

interconnection

The

proposed

guidelines

and

results

are

verified

through

accurate

simulation

modeling

Analytical/simulation

modeling

Processor

board

Board

pinout

Applications

Communication

locality

volume

Packaging

technology

technology

technology

-LRB-

Satisfying

packaging

constraints

-RRB-

Maximum

board

size

Surface

technology

Periphery

technology

maximum

sustained

average

throughput

match

determines

relationship

Processor

speed

Interconnection

Givenchannel

width

Commmunication

link

speeds

SUPPLY

DEMAND

Good

configurations

Best

Configuration

Design

Guideline

Framework

Design-Feasible

configurations

-LRB-

Meeting

Demanded

Performance

-RRB-

Traffic

bound

on

average

network

latency

router

pinoutMaximum

Best

Performance

Cost

considerations

Scalability

technology

Offered

performance

characteristics

Demanded

performance

characteristics

Figure

A

supply-demand

optimization

framework

for

designing

and

developing

scalable

architectures

under

varying

processor

interconnection

and

packaging

technologies

The

paper

is

organized

as

follows

In

Sec.

we

present

the

two-level

k-ary

n-cube

clusterc

architecture

The

impact

of

processor

speed

and

communication

link

speed

on

the

demanded

performance

is

discussed

in

Sec.

In

Sec.

we

present

a

representative

multi-level

packaging

model

for

clustered

systems

Section

discusses

the

trends

in

growth

of

processor

board

sizes

pinout

channel

width

and

router

pinout

technologies

In

Sec.

we

derive

expressions

for

offered

channel

width

and

bisection

bandwidth

under

different

packaging

processor

and

interconnect

technologies

Section

presents

a

latency-throughput

performance

model

for

k-ary

n-cube

cluster-c

clustered

systems

In

Sec.

we

present

the

integrated

framework

and

discuss

important

considerations

in

choosing

the

best

configuration

Section

illustrates

the

framework

to

design

a

processor

system

Section

demonstrates

the

impact

of

varying

packaging

and

demand

parameters

Finally

concluding

remarks

and

future

work

are

presented

Designing

Systems

with

k-ary

n-cube

cluster-c

Organization

k-ary

n-cube

cluster-c

Organization

Many

current

parallel

systems

like

the

CRAY

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

and

the

Stanford

DASH

-LSB-

-RSB-

are

taking

a

two-level

clustering

approach

Recently

we

have

introduced

a

new

k-ary

n-cube

cluster-c

organization

-LSB-

-RSB-

to

capture

this

upcoming

trend

in

building

scalable

parallel

systems

In

this

organization

the

lower

level

consists

of

kn

processor

clusters

These

clusters

are

interconnected

by

a

higher

level

direct

k-ary

n-cube

network

-LRB-

also

referred

to

as

inter-cluster

network

or

internet

-RRB-

Each

cluster

consists

of

c

processors

leading

to

a

total

of

N

-LRB-

kn

c

-RRB-

processors

in

the

system

This

interconnection

achieves

two

main

design

objectives

a

-RRB-

direct

network-based

internet

providing

easy

scalability

and

b

-RRB-

processor

clusters

providing

the

convenience

of

packaging

modularity

and

potential

for

better

exploitation

of

communication

locality

Figure

shows

the

overall

configuration

of

such

a

system

Intercluster

Network

Intra-cluster

Network

ClusterInterface

-LRB-

CI

-RRB-

k-ary

n-cube

c-processor

Clusters

Bus/MIN/direct

star

Figure

Two-level

clustering

with

k-ary

n-cube

cluster-c

-LRB-

kn

c

-RRB-

organization

The

interconnection

within

a

cluster

-LRB-

also

referred

to

as

intra-cluster

network

or

intranet

-RRB-

can

be

chosen

as

bus/MIN/star

network/direct

network

as

shown

in

Fig.

Each

cluster

is

connected

to

the

rest

of

the

system

through

a

cluster

interface

The

main

task

of

the

cluster

interface

is

to

handle

the

volume

of

communication

to/from

the

cluster

Other

functionalities

may

be

added

to

the

cluster

interface

to

efficiently

implement

various

communication

synchronization

and

cachecoherence

operations

to

enhance

overall

system

performance

-LSB-

-RSB-

However

such

discussion

is

beyond

the

scope

of

this

paper

and

we

emphasize

only

on

the

design

framework

The

memory

in

such

systems

is

distributed

physically

across

the

clusters

Organization

of

memory

within

a

cluster

is

left

as

an

open

choice

depending

on

the

size

of

cluster

and

its

configuration

The

exact

nature

of

this

distribution

is

not

critical

to

the

design

and

analysis

a

-RRB-

direct-network

based

c

-RRB-

bus-based

d

-RRB-

star-based

b

-RRB-

MIN-basedMIN

Cluster

interconnections

connections

toother

clusters

CI

CI

CI

CI

processor-memory

unit

router

CI

cluster

interface

Figure

Four

possible

cluster

interconnections

under

k-ary

n-cube

cluster-c

-LRB-

kn

c

-RRB-

organization

presented

in

this

paper

Therefore

without

loss

of

generality

we

assume

the

cluster

memory

to

be

distributed

uniformly

amongst

the

processors

in

the

cluster

Architectural

Alternatives

and

Choices

To

build

an

N-processor

system

there

are

vast

number

of

possible

alternatives

with

clustered

configurations

The

degrees

of

freedom

are

number

of

processors

-LRB-

size

-RRB-

in

each

cluster

and

topologies

of

the

two

levels

-LRB-

inter-cluster

and

intra-cluster

-RRB-

Let

us

consider

designing

a

system

with

N

processors

This

system

can

be

designed

with

clusters

of

processors

each

clusters

of

processors

each

and

so

on

Note

that

for

a

given

system

size

fixing

the

size

of

one

level

automatically

determines

the

size

of

the

other

level

Having

fixed

the

size

of

each

level

there

is

still

freedom

to

vary

the

topology

in

each

level

For

example

in

a

system

with

clusters

with

processors

each

the

topologies

can

be

internet

with

bus-based

clusters

internet

with

MIN-based

clusters

and

so

on

Our

objective

is

to

select

the

configuration

which

a

-RRB-

satisfies

packaging

constraints

b

-RRB-

meets

desired

performance

of

latency

and

throughput

and

c

-RRB-

is

easily

scalable

to

larger

sizes

It

is

easy

to

observe

that

the

flat

k-ary

n-cube

systems

can

be

derived

as

a

special

case

of

the

k-ary

n-cube

cluster-c

family

by

choosing

cluster

size

c

Since

inter-cluster

interconnections

are

more

expensive

than

intra-cluster

interconnections

-LSB-

-RSB-

in

this

paper

we

primarily

emphasize

on

internet

and

cluster

size

In

the

following

sections

we

develop

a

supply-demand

optimization

framework

to

derive

optimal

k-ary

n-cube

cluster-c

organizations

-LRB-

c

-RRB-

Table

provides

a

summary

of

the

main

symbols

and

notations

used

in

this

paper

In

the

remaining

discussion

we

represent

a

k-ary

n-cube

cluster-c

organization

using

a

compact

notation

of

-LRB-

kn

c

-RRB-

For

example

an

inter-cluster

network

with

clusters

of

four

processors

each

is

represented

as

-LRB-

-RRB-

or

-LRB-

-RRB-

Similarly

a

configuration

with

mixed

radix

inter-cluster

network

with

topology

and

cluster

size

is

represented

as

-LRB-

-RRB-

Table

Summary

of

Important

Symbols

and

Notations

Used

a

Unit

area

to

hold

a

processor

chip

associated

memory

and

router

logic

b

Capacity

of

a

board

in

units

of

a

-LRB-

number

of

processors

in

a

board

-RRB-

b

c

Number

of

clusters

in

a

board

c

Size

of

each

cluster

-LRB-

kn

c

-RRB-

Shorthand

representation

for

a

k-ary

n-cube

cluster-c

organization

n

Dimensionality

of

inter-cluster

network

pp

Pinout

density

under

periphery

pinout

technology

ps

Pinout

density

under

surface

pinout

technology

r

Computation

to

communication

ratio

-LRB-

indicates

application

characteristics

-RRB-

t

Cycle

time

to

transfer

a

bit

across

a

wire

-LRB-

indicates

communication

link

technology

-RRB-

ub

Percentage

utilization

of

board

area

b

up

Percentage

utilization

of

available

board

pinout

Pb

B

Bisection

size

in

the

inter-cluster

network

D

Computational

speed

of

a

processor

-LRB-

indicates

processor

technology

-RRB-

L

Message

length

in

bits

N

Total

number

of

processors

in

a

system

N

N

c

Total

number

of

clusters

in

system

Nboards

Total

number

of

processor

boards

used

in

a

system

-LRB-

N

b

-RRB-

Pb

Total

pincount

from

a

board

-LRB-

depends

on

pinout

technology

-RRB-

R

Maximum

pinout

supported

from

a

router

Tc

Average

message

latency

in

presence

of

contention

Tmax

Upper

bound

on

Tc

W

Inter-cluster

channel

width

-LRB-

number

of

wires

-RRB-

W

Channel

width

supported

by

technology

Wp

Maximum

value

of

W

constrained

by

board

pinout

Pb

Demanded

average

throughput

per

processor

reflecting

both

processor

and

communication

link

technologies

Dt

r

Demand

on

Network

Performance

In

this

section

we

characterize

the

demands

made

on

the

average

system

throughput

and

average

network

latency

of

an

interconnection

network

These

demands

are

characterized

by

parameterizing

processor

and

interconnection

speed

and

communication

characteristics

of

applications

We

first

analyze

the

impact

of

processor

speed

and

communication

link

technology

on

the

demanded

average

throughput

in

the

system

The

importance

of

a

bound

on

the

average

network

latency

of

a

message

is

discussed

next

Effect

of

Processor

and

Interconnect

Speed

on

Demanded

System

Throughput

With

advances

in

processor

technology

the

rate

at

which

processors

can

execute

instructions

is

going

up

Let

us

denote

the

computational

speed

of

a

processor

as

D

MFlops/sec

An

increase

in

processing

power

leads

to

a

higher

value

of

D

As

a

general

observation

in

applications

an

amount

of

computation

is

associated

with

a

quota

of

necessary

communication

expressed

as

the

computation

to

communication

ratio

-LRB-

r

Flops/bit

-RRB-

-LSB-

-RSB-

Therefore

while

computing

at

the

rate

D

MFlops/sec

a

processor

demands

a

sustained

communication

rate

or

average

throughput

of

-LRB-

D

r

-RRB-

Mbits/sec

To

allow

such

a

demanded

traffic

rate

the

underlying

interconnection

network

should

be

able

to

sustain

a

minimum

of

-LRB-

D

r

-RRB-

Mbits/sec

per

processor

A

typical

estimate

of

the

range

of

computation

to

communication

ratio

as

suggested

in

-LSB-

-RSB-

is

r

to

Flops/bit

For

an

example

system

with

D

and

r

Flops/bit

the

expected

throughput

demand

on

the

network

is

-LRB-

-RRB-

Mbits/sec

or

per

processor

To

capture

the

competitive

growth

of

the

processor

and

interconnect

technologies

we

represent

the

above

processor

demand

on

average

throughput

in

terms

of

-LRB-

bits/network

cycle

-RRB-

instead

of

-LRB-

bits/sec

-RRB-

Let

t

denote

the

network

cycle

time

-LRB-

seconds

-RRB-

the

time

to

transfer

a

bit

across

a

network

wire

A

reduction

in

t

allows

more

data

to

be

sent

across

any

wire

in

a

given

time

leading

to

higher

channel

bandwidth

The

average

throughput

of

-LRB-

D

r

-RRB-

bits/sec

can

be

rewritten

as

-LRB-

D

r

-RRB-

t

bits/cycle

Observe

that

an

increase

in

the

value

of

D

captures

the

advancement

in

processor

speed

while

a

decrease

in

the

value

of

t

captures

that

of

link

speeds

To

capture

both

these

technological

advancements

together

let

us

introduce

a

parameter

-LRB-

D

r

-RRB-

t

for

demanded

average

throughput

in

bits/cycle

It

is

to

be

noted

that

if

processor

and

link

speeds

increase

in

the

same

proportion

then

does

not

change

For

deriving

representative

values

of

in

current

and

future

systems

we

consider

predictions

made

by

Patterson

-LSB-

-RSB-

As

shown

in

Table

for

a

computation

to

communication

ratio

r

Flops/bit

the

values

of

broadly

lie

in

the

range

of

bits/cycle

For

a

lower

value

of

r

the

corresponding

values

in

the

range

of

would

be

higher

Table

Sample

values

of

for

various

representative

combinations

of

processor

and

network

speeds

A

computation

to

communication

ratio

of

r

Flops/bit

-LSB-

-RSB-

is

assumed

Processor

Speed

Processor

Network

Channel

Network

Channel

-LRB-

D

r

-RRB-

t

D

-LRB-

MFlops/sec

-RRB-

Demand

on

Network

Speed

-LRB-

MHz

-RRB-

Cycle

Time

-LRB-

bits/cycle

-RRB-

D

r

-LRB-

bits/sec

-RRB-

t

-LRB-

sec/cycle

-RRB-

The

bisection

size

of

a

network

is

defined

as

the

minimum

number

of

wires

that

need

to

be

cut

in

order

to

divide

the

network

into

two

equal

parts

-LSB-

-RSB-

It

limits

the

number

of

bits

that

can

cross

from

one

half

to

another

half

of

the

network

Let

us

analyze

the

impact

of

sustaining

an

average

throughput

of

bits/cycle

per

processor

on

the

network

bisection

size

in

the

inter-cluster

network

The

total

traffic

injected

by

all

processors

per

cycle

is

N

bits

For

uniform

traffic

the

probability

of

a

generated

message

being

intra-cluster

is

c

N

Similarly

the

probability

of

a

message

being

inter-cluster

is

-LRB-

c

N

-RRB-

Thus

on

the

average

N

-LRB-

c

N

-RRB-

bits

get

injected

into

the

inter-cluster

network

every

cycle

On

the

average

half

of

this

inter-cluster

traffic

is

destined

to

processors

on

the

other

half

of

the

system

These

messages

require

to

cross

the

inter-cluster

bisection

indicating

a

demand

of

N

-LRB-

c

N

-RRB-

bits

across

the

bisection

every

cycle

Denoting

the

inter-cluster

bisection

size

as

B

wires

it

is

clear

that

B

N

-LRB-

c

N

-RRB-

wires

are

needed

to

sustain

the

demanded

traffic

However

due

to

contention

in

the

network

the

utilization

of

the

network

channels

are

only

a

fraction

of

the

maximum

capacity

-LSB-

-RSB-

To

compensate

for

such

loss

in

bandwidth

due

to

contention

the

actual

value

of

B

required

to

support

a

throughput

of

bits/cycle

has

to

be

greater

than

N

-LRB-

c

N

-RRB-

Similar

to

the

demand

on

the

inter-cluster

bisection

size

to

sustain

a

given

throughput

the

traffic

going

in/coming

out/traversing

from

one

processor

to

another

inside

a

cluster

also

imposes

a

demand

on

the

intra-cluster

bisection

size

This

demand

is

clearly

proportional

to

the

size

of

a

cluster

Thus

to

support

larger

sized

clusters

the

intra-cluster

bisection

size

should

scale

linearly

with

size

c

Since

packaging

constraints

are

less

rigid

in

lower

hierarchies

it

is

possible

to

provide

thicker

channels/buses

to

achieve

higher

bandwidth

inside

a

cluster

-LSB-

-RSB-

Hence

in

this

paper

we

focus

only

on

the

bisection

size

of

the

inter-cluster

network

Demanded

Bound

on

Average

Network

Latency

The

achievable

average

message

latency

can

have

a

direct

impact

on

the

processor

efficiency

in

a

multiprocessor

system

For

example

consider

a

parallel

application

in

which

each

processor

in

the

system

executes

a

series

of

computation

blocks

of

cycles

each

At

the

start

of

a

block

each

processor

sends

out

request

messages

to

other

processors

the

reply

to

which

is

checked

at

the

end

of

the

block

execution

-LRB-

can

be

considered

as

prefetching

data

-RRB-

Assume

a

processor

continues

execution

of

its

next

block

only

after

it

receives

the

reply

to

the

previous

request

Let

the

amount

of

time

spent

by

a

remote

processor

in

sending

out

the

reply

message

be

small

Thus

it

is

critical

that

the

average

one-way

message

latency

in

the

network

be

less

than

cycles

for

a

round-trip

latency

of

less

than

cycles

Otherwise

a

processor

is

forced

to

idle

while

waiting

for

a

reply

message

leading

to

a

fall

in

performance

Let

Tmax

denote

such

an

upper

bound

on

the

average

network

latency

expected/demanded

in

a

system

Clearly

the

value

of

Tmax

depends

on

many

factors

like

nature

of

the

application

and

state

of

compiler

technology

For

example

the

extent

to

which

prefetching

of

data

can

be

applied

differs

from

application

to

application

Similarly

the

state

of

compiler

technology

dictates

the

degree

to

which

prefetching

can

be

exploited

For

illustrative

purposes

in

this

paper

we

assume

typical

values

of

Tmax

to

be

cycles

-LSB-

-RSB-

In

this

section

we

have

defined

demanded

network

performance

in

terms

of

a

-RRB-

sustained

average

throughput

-LRB-

bits/cycle

-RRB-

and

b

-RRB-

bound

on

average

message

latency

-LRB-

Tmax

cycles

-RRB-

We

refer

to

this

as

the

demand

side

to

our

design

framework

In

the

following

sections

we

develop

the

supply

side

to

our

framework

The

next

section

presents

a

typical

multi-level

packaging

model

for

designing

large

multiprocessor

systems

The

packaging

technologies

and

constraints

of

these

levels

are

discussed

in

Section

and

their

impact

on

the

set

of

feasible

-LRB-

or

supplied

-RRB-

configurations

derived

in

Section

Packaging

Model

for

Clustered

Systems

Some

earlier

models

proposed

to

capture

packaging

constraints

like

the

VLSI

model

with

limited

bisection

bandwidth

as

proposed

by

Dally

-LSB-

-RSB-

and

the

limited

pincount

model

as

proposed

by

Agrawal

-LSB-

-RSB-

deal

with

only

one

level

of

packaging

Large

parallel

machines

on

the

other

hand

typically

employ

several

levels

of

packaging

There

are

inherent

technological

constraints

that

limit

factors

like

VLSI

die

sizes

chip

pinout

board

areas

board

pinout

and

number

of

boards

per

card-cage

Figure

shows

a

typical

multi-level

hierarchy

encountered

while

packaging

a

large

clustered

multiprocessor

system

At

the

lowest

level

are

processor

memory

and

router

VLSI

chip

modules

A

single

processor

chip

its

associated

memory

and

router

chips

and

other

interface

logic

are

usually

referred

together

as

a

processing

node

Multiple

processing

nodes

are

organized

into

cluster

modules

and

placed

on

processor

boards

A

cluster

module

is

usually

compact

and

does

not

span

across

multiple

boards

Figure

shows

each

board

in

the

system

having

four

cluster

modules

placed

on

it

Multiple

boards

are

organized

into

a

card-cage

and

a

system

may

comprise

of

multiple

such

card-cages

For

example

the

system

in

Fig.

requires

two

card-cages

each

containing

four

boards

A

larger

system

would

require

more

boards

and

card-cages

The

modules

at

each

level

of

this

packaging

hierarchy

chips

boards

card-cages

have

their

own

characteristics

in

terms

of

maximum

capacity

bisection

size

available

pinout

and

channel

width

For

example

the

pinout

from

a

VLSI

chip

may

be

limited

to

a

maximum

of

pins

Similarly

depending

on

the

size

of

a

cluster

module

and

maximum

allowable

area

a

processor

board

may

be

able

to

hold

up

to

cluster

modules

only

Such

packaging

characteristics

are

analyzed

in

more

detail

in

the

next

section

Cluster

Interface

Router

Chip

Processor

and

Memory

Chips

Cluster

Interface

Router

Chip

Processor

and

Memory

Chips

Interface

Router

Chip

Processor

and

Memory

Chips

Board

Pinout

Cluster

module

containing

Pinout

from

a

Backplanes

Board

cluster

module

multiple

processing

nodes

Cluster

Interface

Router

Chip

Processor

and

Memory

Chips

Cluster

Interface

Router

Chip

Processor

and

Memory

Chips

Cluster

Interface

Router

Chip

Processor

and

Memory

Chips

An

inter-board

channel

being

connected

using

a

An

inter-board

channel

being

connected

using

Board

connector

cable

the

board

backplane

RouterchipsProcessorMemory

Figure

Typical

multi-level

hierarchy

used

in

packaging

a

large

clustered

multiprocessor

system

The

above

packaging

hierarchy

has

a

direct

impact

on

the

connections

of

the

inter-cluster

channels

For

example

let

us

consider

two

clusters

connected

by

an

inter-cluster

channel

The

two

clusters

may

lie

on

the

same

or

on

different

boards

A

channel

between

two

clusters

on

the

same

board

is

connected

through

wires

on

the

board

Such

wires

directly

connect

pins

of

one

cluster

module

to

another

However

connecting

a

channel

between

clusters

across

two

different

boards

is

more

difficult

Pins

from

the

cluster

modules

need

to

be

first

connected

to

pins

of

the

respective

boards

Figure

shows

such

board

pins

along

the

periphery

of

each

board

The

channel

is

completed

by

connecting

the

pins

between

the

two

board

modules

This

inter-board

part

of

the

channel

is

implemented

either

through

a

-RRB-

the

shared

backplane

for

boards

within

the

same

card-cage

or

b

-RRB-

a

cable

connector

for

boards

across

different

card-cages

Figure

shows

examples

of

these

two

types

of

inter-board

channels

Large

system

size

and

multiple

levels

of

packaging

have

significant

impact

on

the

length

of

connecting

wires

in

the

system

The

length

of

wires

to

connect

channels

depends

on

the

distance

between

the

connecting

boards

Longer

wires

have

higher

capacitance

-LSB-

-RSB-

leading

to

elongation

of

the

channel

cycle

time

However

it

has

been

shown

-LSB-

-RSB-

that

this

problem

can

be

alleviated

by

applying

pipelining

techniques

over

long

wires

Such

techniques

are

being

commonly

employed

in

recently

developed

systems

like

the

Intel

Paragon

-LSB-

-RSB-

In

this

study

we

assume

such

pipelining

techniques

being

used

to

limit

the

channel

cycle

time

in

the

inter-cluster

system

Parameterizing

Packaging

Technologies

The

characteristics

and

limitations

of

each

level

of

packaging

has

an

extreme

impact

on

the

set

of

achievable

or

feasible

configurations

and

their

cost

and

performance

In

this

section

we

discuss

important

trends

in

board

capacity

board

pinout

channel

width

and

router

pinout

technologies

These

technologies

are

parameterized

and

their

impact

on

design

constraints

are

analyzed

Processor

Board

Technology

Processor

boards

can

not

be

arbitrarily

large

in

size

The

physical

size

of

a

board

is

restricted

by

electrical

mechanical

and

board

fabrication

constraints

In

terms

of

physical

dimensions

board

sizes

being

used

in

recent

multiprocessor

systems

vary

from

to

an

aggressive

used

in

the

J-machine

-LSB-

-RSB-

The

largest

board

size

available

to

a

system

designer

usually

varies

with

technology

and

over

a

period

of

time

In

this

study

we

therefore

do

not

present

guidelines

restricted

to

a

particular

largest

board

size

The

available

maximum

board

size

is

treated

as

a

parameter

in

the

framework

Depending

on

the

size

of

a

given

board

we

can

fit

only

a

limited

number

of

cluster

modules

or

processing

nodes

on

it

For

example

consider

a

design

problem

with

board

sizes

of

and

two

processing

nodes

per

cluster

module

Let

the

size

of

a

processor

chip

be

the

area

occupied

by

associated

local

memory

of

-LRB-

say

-RRB-

be

and

required

intra-board

estate

for

router

memory

and

address

buses

and

other

support

logic

be

square

inches

In

such

a

design

we

require

around

square

inches

per

processing

node

leading

to

square

inches

per

cluster

module

This

implies

that

we

can

fit

exactly

one

cluster

module

or

two

processing

nodes

on

each

board

It

is

natural

to

express

board

area

or

capacity

in

terms

of

absolute

units

such

as

square

inches

However

for

ease

of

discussion

it

is

more

convenient

to

express

it

in

relation

to

the

area

of

a

processing

node

For

example

consider

a

given

a

board

with

its

capacity

specified

to

be

processing

nodes

It

can

be

easily

observed

that

such

a

board

is

capable

of

holding

clusters

of

processing

nodes

each

or

clusters

of

processing

nodes

each

As

discussed

earlier

we

define

the

size

of

a

processing

node

to

include

the

area

required

by

a

processor

chip

its

local

memory

router

and

other

associated

interface

logic

Such

a

combined

area

is

denoted

as

parameter

a

The

magnitude

of

a

depends

on

two

main

factors

a

-RRB-

level

of

integration

in

the

VLSI

chips

and

b

-RRB-

the

size

of

memory

associated

with

each

processor

A

higher

VLSI

integration

leads

to

smaller

a.

Similarly

supporting

larger

memory

per

processor

leads

to

larger

a

A

designer

can

choose

an

appropriate

value

for

a

depending

on

chip

sizes

and

amount

of

memory

per

processor

and

use

this

as

an

unit

of

board

capacity

A

board

with

physical

area

-LRB-

ba

-RRB-

is

defined

to

have

a

capacity

of

-LRB-

b

-RRB-

Thus

in

the

example

in

the

last

paragraph

a

and

a

board

has

a

capacity

of

b

Over

a

period

of

time

let

the

advancement

in

VLSI

technology

lead

to

higher

processor

and

memory

integration

To

reflect

this

we

can

have

a

suitably

smaller

value

of

a

in

our

design

problem

Let

the

new

value

for

a

be

This

implies

the

same

board

in

terms

of

the

new

value

of

a

has

capacity

to

hold

b

processing

nodes

In

further

discussion

unless

otherwise

mentioned

we

use

the

terms

board

size

board

area

and

board

capacity

in

an

interchangeable

manner

Assume

a

cluster

module

containing

c

processing

nodes

requires

an

area

ca

Thus

a

board

with

capacity

b

can

hold

up

to

b

c

clusters

on

it

To

build

a

system

with

N

processing

nodes

a

total

board

area

of

is

necessary

A

total

board

area

larger

than

may

be

used

but

this

clearly

leads

to

wastage

of

precious

board

estate

Thus

we

suggest

using

a

total

board

area

close

to

We

denote

the

maximum

board

size

discussed

earlier

as

parameter

bmax

For

illustrative

purposes

we

later

consider

bmax

and

in

this

paper

However

the

framework

works

with

any

value

of

bmax

as

allowed

by

technology

All

boards

used

in

a

given

system

are

assumed

to

be

of

the

same

size

and

board

capacity

satisfies

b

bmax

Total

number

of

boards

used

in

designing

an

N-processor

system

is

defined

as

Nboards

N

b.

Board

Pinout

Technologies

The

pin-count

Pb

out

of

a

board

has

a

direct

influence

on

the

data

volume

that

can

flow

in/out

of

a

given

processor

board

Currently

two

different

types

of

technologies

are

being

employed

by

the

computer

industry

a

-RRB-

Peripheral

pinout

This

is

the

traditional

technology

-LSB-

-RSB-

where

the

periphery

of

a

board

is

used

for

external

connections

The

exact

relationship

between

the

size

of

board

periphery

and

pincount

depends

on

the

pin

connector

technology

However

it

is

reasonable

to

expect

this

relationship

to

be

linear

Thus

a

larger

board

having

a

longer

periphery

can

support

more

pins

Typically

only

one

or

two

sides

of

a

board

are

used

for

periphery

pinout

Without

loss

of

generality

let

us

assume

only

one

side

being

used

for

pinout

as

shown

earlier

in

Fig.

Let

pp

denote

the

peripheral

pinout

density

the

pin-count

that

can

be

supported

from

an

unit

board

length

Thus

a

larger

square

board

with

capacity

b

having

each

side

of

length

pb

units

can

support

a

total

of

Pb

pp

pb

pins

b

-RRB-

Surface

pinout

This

is

representative

of

a

more

aggressive

pinout

technology

The

surface

of

the

board

is

utilized

for

external

connections

Representative

examples

are

electronic

interconnections

using

elastomeric

connectors

-LSB-

-RSB-

and

optical

interconnections

-LSB-

-RSB-

Let

ps

denote

the

surface

pinout

density

the

pin-count

that

can

be

supported

from

a

board

of

unit

capacity

Assuming

a

linear

relation

between

board

area

and

surface

pinout

the

pin-count

supportable

from

a

board

of

capacity

b

is

Pb

bps

Figure

shows

the

growths

of

pincount

with

board

area

under

the

respective

surface

and

peripheral

technologies

for

different

relationships

between

ps

and

pp

The

vertical

clipping

line

in

each

graph

reflects

that

the

board

size

can

not

grow

continuously

and

is

limited

by

some

maximum

size

As

expected

for

large

board

sizes

the

surface

technology

always

supersedes

the

peripheral

Board

size

-LRB-

b

-RRB-

Pincount

psb

surface

pp?b

peripheral

Board

size

-LRB-

b

-RRB-

technology

technology

bmax

technology

bmax

ps

pp

ps

pp

increase

inboard

size

Pincount

psb

surfacepp?b

peripheral

technology

Figure

Growth

of

pincount

with

board

area

under

peripheral

and

surface

technologies

shown

for

two

different

relationships

between

pp

and

ps

The

parameters

pp

and

ps

reflect

the

pincount

that

can

be

supported

from

a

board

of

unit

capacity

under

periphery

and

surface

pinout

technologies

respectively

technology

However

if

ps

pp

then

for

smaller

board

sizes

the

peripheral

technology

can

beat

the

surface

technology

Representative

values

of

pp

and

ps

derived

from

the

current

design

trends

-LSB-

-RSB-

are

pp

and

ps

Future

values

of

these

parameters

will

depend

on

the

advances

made

in

the

respective

technologies

In

the

next

section

we

compare

the

impact

on

designing

clustered

systems

under

both

these

pinout

technologies

Channel

Width

Technology

Most

current

parallel

machines

have

and

data

channels

This

corresponds

to

a

channel

width

of

W

ss

and

respectively

including

control

acknowledgment

and

parity

wires

For

a

simpler

interface

design

the

width

of

data

lines

in

a

channel

is

expected

to

maintain

an

integral

relationship

with

that

of

processor

and

memory

which

are

typically

in

multiples

of

a

byte

Further

factors

like

path-width

inside

routers

and

connector

technology

restrict

channel

widths

from

being

arbitrarily

large

Most

previous

studies

while

proposing

design

guidelines

did

not

consider

such

constraints

on

channel

width

In

our

framework

we

account

for

these

and

denote

a

channel

width

supportable

by

technology

as

W

For

example

current

technology

supports

W

and

In

the

near

future

it

is

expected

that

a

wider

channel

width

technology

with

ability

to

carry

data

-LSB-

-RSB-

corresponding

to

W

ss

would

be

feasible

In

the

next

section

we

investigate

the

impact

of

such

supportable

fixed

channel

width

-LRB-

W

-RRB-

on

the

set

of

design-feasible

configurations

Router

Pinout

Technology

Similar

to

the

restriction

on

maximum

pinout

from

a

processor

board

the

pinout

from

processor

memory

and

router

chips

are

also

limited

The

available

number

of

pins

from

a

router

restricts

the

number

of

channels

and

channel

widths

that

can

be

supported

-LSB-

-RSB-

In

a

multi-dimensional

system

a

router

chip

has

to

support

at

least

two

channels

along

any

dimension

one

for

an

incoming

and

another

for

an

outgoing

channel

For

a

system

with

bidirectional

channels

this

number

is

four

per

dimension

Let

us

consider

an

inter-cluster

router

in

a

-LRB-

kn

c

-RRB-

system

The

number

of

channels

to

be

supported

from

a

router

assuming

unidirectional

channels

is

Supporting

W

bit

wide

channels

in

such

a

system

requires

a

pinout

of

from

each

router

chip

Let

us

denote

the

maximum

pinout

from

a

chip

offered

by

the

chip

fabrication

technology

as

R.

Clearly

this

imposes

a

constraint

of

R

-LRB-

-RRB-

As

an

illustration

consider

a

representative

value

of

R

ss

Using

the

above

equation

let

us

derive

the

maximum

dimension

-LRB-

n

-RRB-

that

can

be

supported

in

an

interconnection

For

various

current

and

future

channel

width

technologies

W

and

the

maximum

dimension

supported

are

n

and

respectively

Thus

it

can

be

observed

that

even

with

a

conservative

pinout

technology

and

a

channel

width

technology

of

W

up

to

systems

are

feasible

For

thinner

channels

the

maximum

supportable

dimension

can

be

even

higher

Impact

of

Packaging

on

Architectural

Parameters

In

this

section

we

analyze

the

impact

of

packaging

constraints

on

the

channel

width

and

bisection

size

of

k-ary

n-cube

cluster-c

configurations

The

maximum

offered

channel

width

is

derived

based

on

board

size

and

pinout

constraints

We

then

analyze

the

impact

of

maintaining

a

fixed

supportable

channel

width

on

system

dimensionality

and

cluster

size

It

is

then

shown

that

realistic

design

can

lead

to

under-utilization

of

board

resources

Minimizing

such

under-utilization

is

necessary

to

minimize

system

cost

Based

on

this

a

process

of

deriving

system

configurations

minimizing

under-utilization

is

proposed

We

then

analyze

the

impact

of

increasing

system

and

cluster

size

on

the

bisection

size

of

a

system

while

maintaining

a

fixed

supportable

channel

width

Most

results

presented

in

this

section

hold

for

both

periphery

and

surface

pinout

technologies

In

case

of

differences

these

are

specifically

mentioned

All

results

are

derived

for

arbitrary

cluster

sizes

-LRB-

c

-RRB-

and

the

corresponding

results

for

flat

architectures

can

be

obtained

by

choosing

c

Offered

Inter-Cluster

Channel

Width

Let

the

clusters

in

a

system

be

placed

on

multiple

boards

and

interconnected

by

channels

to

yield

a

N

N

c

x

x

kn

inter-cluster

topology

as

discussed

earlier

in

Section

Each

board

in

this

system

holds

clusters

The

clusters

on

any

board

together

with

the

inter-cluster

channels

between

them

can

be

visualized

as

a

sub-topology

of

the

overall

inter-cluster

topology

Let

this

sub-topology

be

represented

as

-LRB-

x

x

x

bn

-RRB-

where

bi

ki

i

n

and

some

bi

's

may

be

to

depict

that

only

one

cluster

exists

on

the

board

along

that

dimension

This

implies

that

no

inter-cluster

dimension

can

be

fully

contained

within

a

board

This

is

reasonable

to

expect

for

large

system

sizes

Figure

shows

an

example

system

with

clusters

being

placed

on

boards

each

with

a

capacity

for

holding

four

processor

clusters

The

overall

inter-cluster

topology

is

assumed

to

be

-LRB-

N

x

x

-RRB-

The

sub-topology

on

each

board

is

assumed

to

be

x

x

Let

the

processor

clusterinterconnection

N

inter-cluster

b

sub-topology

on

a

board

dim

Figure

A

system

with

clusters

being

placed

on

boards

each

with

a

capacity

of

holding

processor

clusters

The

overall

inter-cluster

topology

is

The

sub-topology

placed

on

each

board

is

number

of

rows

of

processor

clusters

in

dimension

i

of

the

sub-topology

on

a

board

be

denoted

as

ri

This

can

be

derived

as

ri

bi

For

example

the

number

of

such

rows

on

each

board

in

Fig.

along

dimensions

marked

and

are

and

respectively

Note

that

inter-board

channels

go

out/come

in

from

the

two

ends

of

each

row

leading

to

inter-board

along

dimension

i.

Hence

the

total

number

of

inter-board

channels

required

to

go

out/come

into

a

board

from

all

n

dimensions

is

derived

as

-LRB-

bi

-RRB-

As

discussed

earlier

in

Section

inter-board

channels

between

boards

in

the

same

card-cage

can

be

connected

through

a

shared

backplane

The

connections

from

a

board

to

the

backplane

are

made

using

the

pinout

from

the

board

In

large

systems

with

many

boards

and

multiple

card-cages

a

single

backplane

can

not

be

used

to

connect

all

boards

Thus

cable

connectors

also

need

to

be

employed

to

connect

the

inter-board

channels

across

card-cages

Here

too

the

connections

from

the

board

to

cable

connectors

are

established

using

the

is

true

for

unidirectional

channels

For

full

duplex

bidirectional

channels

this

number

would

be

pinout

from

the

board

Thus

the

inter-board

channel

width

is

constrained

by

the

available

board

pinout

To

support

a

channel

width

of

W

a

total

pincount

of

-LRB-

bi

-RRB-

is

required

from

each

board

Thus

the

maximum

supportable

inter-board

channel

width

from

pinout

restrictions

is

derived

as

W

Pb

-LRB-

bi

-RRB-

-LRB-

-RRB-

Assuming

that

the

sub-topology

on

the

board

is

i.e.

-LRB-

bi

n

-RRB-

and

using

b

c

this

can

be

simplified

to

W

Pb

-LRB-

n

-RRB-

n

Pb

-LRB-

b

c

-RRB-

-LRB-

n

-RRB-

n

-LRB-

-RRB-

Similar

to

the

inter-board

inter-cluster

channels

connecting

clusters

across

different

boards

intra-board

inter-cluster

channels

are

required

to

connect

the

clusters

on

a

single

board

to

maintain

the

-LRB-

x

x.

x

bn

-RRB-

sub-topology

on

a

board

These

channels

can

be

implemented

using

on-board

wires

For

an

on-board

implementation

it

is

intuitive

to

expect

the

available

board

bisection

to

limit

the

size

of

such

intra-board

channels

However

with

multi-layered

boards

it

is

reasonable

to

assume

on-board

connection

density

to

be

much

higher

than

off-board

density

Hence

the

intercluster

channel

width

is

mainly

limited

by

the

inter-board

channel

width

determined

by

the

pinout

constraint

as

shown

in

Eqn

From

the

expression

for

W

in

Eqn

it

can

be

observed

that

the

channel

width

is

determined

by

board

size

cluster

size

pinout

technology

from

board

and

dimensionality

of

the

inter-cluster

network

It

does

not

depend

on

the

total

system

size

For

a

given

pinout

technology

and

board

size

let

us

consider

the

impact

of

varying

the

inter-cluster

dimensionality

while

keeping

cluster

size

fixed

In

order

to

support

larger

dimensional

networks

we

need

more

channels

This

leads

to

thinner

channels

with

a

fixed

pinout

from

a

board

Similarly

for

a

given

pinout

technology

and

board

size

decreasing

the

cluster

size

while

keeping

inter-cluster

dimensionality

fixed

also

leads

to

thinner

channels

A

smaller

cluster

size

results

in

larger

number

of

clusters

on

board

This

implies

that

pinout

from

the

board

gets

shared

among

more

clusters

leading

to

thinner

channels

These

observations

can

be

summarized

as

Result

For

a

fixed

pinout

from

a

board

of

given

size

the

offered

channel

width

-LRB-

W

-RRB-

falls

with

increase

in

the

inter-cluster

network

dimensionality

-LRB-

n

-RRB-

while

keeping

cluster

size

-LRB-

c

-RRB-

fixed

Similarly

channel

width

-LRB-

W

-RRB-

falls

as

the

the

inter-cluster

dimensionality

-LRB-

n

-RRB-

remains

fixed

and

the

cluster

size

-LRB-

c

-RRB-

reduces

of

regular

topology

wherever

made

in

this

section

is

only

for

formula

simplification

and

providing

a

more

intuitive

understanding

of

the

interplay

between

various

parameters

However

the

proposed

framework

is

also

valid

for

the

mixed

radix

topologies

Design

solutions

are

presented

later

with

mixed

radix

topologies

Figure

depicts

the

above

trends

as

cluster

size

is

increased

for

different

inter-cluster

dimensionalities

Each

board

is

assumed

to

be

large

enough

to

hold

exactly

one

cluster

-LRB-

-RRB-

Channel

widths

up

to

lines

have

only

been

shown

For

a

given

cluster

size

offered

channel

width

drops

with

increasing

inter-cluster

dimensionality

It

can

be

observed

that

the

drop

in

channel

width

is

more

significant

at

lower

dimensions

-LRB-

from

to

-RRB-

than

at

higher

dimensions

-LRB-

from

to

-RRB-

Therefore

for

reasonably

higher

dimensions

the

channel

widths

are

expected

to

be

close

Similar

results

were

also

observed

with

Channel

width

-LRB-

W

-RRB-

cluster

size

-LRB-

c

-RRB-

Periphery

Pinout

Technology

Figure

Offered

channel

width

as

a

function

of

cluster

size

and

inter-cluster

dimensionalities

Board

size

is

chosen

to

hold

one

cluster

with

c

processors

-LRB-

-RRB-

A

periphery

pinout

technology

with

pp

is

considered

Let

us

consider

the

impact

of

board

size

on

channel

width

W

Under

periphery

pinout

technology

with

Pb

pp

pb

we

can

simplify

Eqn

as

W

ppc

-LRB-

n

-RRB-

n

-LRB-

n

-RRB-

-LRB-

-RRB-

n

-LRB-

-RRB-

Thus

for

a

given

cluster

size

and

inter-cluster

dimensionality

n

an

increase

in

board

size

b

leads

to

a

fall

in

channel

width

For

a

inter-cluster

network

-LRB-

n

-RRB-

the

channel

width

remains

constant

while

that

in

an

network

rises

sharply

Under

surface

pinout

technology

with

Pb

psb

a

similar

simplification

leads

to

W

psc

-LRB-

n

-RRB-

n

-LRB-

-RRB-

Here

for

any

given

cluster

size

and

inter-cluster

dimensionality

an

increase

in

board

size

b

leads

to

a

rise

in

channel

width

However

this

rise

is

not

very

significant

for

higher

dimensions

This

leads

to

Result

Under

periphery

pinout

technology

keeping

cluster

size

c

fixed

and

dimensionality

of

inter-cluster

network

fixed

at

n

an

increase

in

board

size

results

in

channel

width

to

fall

However

under

surface

pinout

technology

for

any

fixed

inter-cluster

dimensionality

it

leads

to

a

rise

in

channel

width

Supporting

a

Fixed

Inter-Cluster

Channel

Width

Most

of

the

prior

studies

on

system

design

while

proposing

guidelines

under

different

constraints

like

constant

bisection

bandwidth

-LSB-

-RSB-

did

not

impose

any

restrictions

on

the

values

that

the

channel

width

can

take

while

satisfying

other

constraints

As

discussed

in

Section

supporting

an

arbitrarily

large

channel

width

is

not

easy

Further

the

data

lines

are

expected

to

be

integral

multiples

of

bytes

for

easier

interfacing

with

processors

and

memories

Representative

values

of

supportable

channel

width

-LRB-

W

-RRB-

as

discussed

earlier

in

Section

are

and

In

Equations

and

we

showed

that

offered

channel

width

W

is

a

function

of

system

parameters

n

b

c

and

pp

or

ps

Given

a

channel

width

technology

W

to

be

supported

an

obvious

design

objective

while

selecting

values

for

n

b

and

c

is

to

ensure

that

the

offered

channel

width

-LRB-

W

-RRB-

is

equal

to

the

supportable

channel

width

-LRB-

W

-RRB-

Based

on

Equations

and

we

analyzed

the

relationships

among

these

parameters

and

the

impact

of

varying

one

parameter

on

another

was

studied

while

maintaining

a

fixed

W

-LRB-

W

-RRB-

Table

summarizes

some

of

the

important

interplays

An

entry

in

the

table

fixed

or

corresponds

to

the

respective

parameter

being

kept

fixed

increased

or

decreased

respectively

For

example

the

first

row

of

the

table

indicates

that

for

a

given

fixed

board

size

and

pinout

density

increasing

-LRB-

-RRB-

cluster

size

requires

the

intercluster

dimensionality

to

be

also

increased

-LRB-

-RRB-

Similarly

rows

two

and

three

denote

the

impact

of

increasing

board

pinout

density

on

inter-cluster

dimensionality

and

cluster

size

respectively

With

increase

-LRB-

-RRB-

in

board

pinout

density

the

supported

pinout

from

a

given

board

is

higher

The

increased

pinout

can

be

used

to

support

more

inter-cluster

channels

from

a

board

while

maintaining

a

given

channel

width

The

extra

channels

can

support

a

-RRB-

higher

-LRB-

-RRB-

dimensional

inter-cluster

networks

requiring

more

channels

or

b

-RRB-

smaller

-LRB-

-RRB-

clusters

With

smaller

clusters

building

a

system

of

given

size

requires

more

clusters

Interconnecting

a

larger

number

of

clusters

while

maintaining

the

inter-cluster

dimensionality

and

channel

width

requires

more

channels

in

the

system

Increased

pinout

from

boards

make

it

possible

to

support

the

needed

extra

channels

It

may

be

noted

that

relationships

obtained

by

reversing

the

sense

of

all

arrows

in

Table

also

hold

true

Designing

with

Under

Utilization

of

Resources

The

expression

in

Eqn

yields

a

channel

width

W

assuming

full

utilization

of

board

area

and

pinout

resources

However

while

building

a

real

machine

it

is

unlikely

that

both

board

area

and

pinout

resources

get

fully

utilized

For

example

a

fraction

of

the

board

pinout

capacity

may

remain

unutilized

if

a

larger

board

size

is

required

to

fit

a

desired

topology

but

the

resulting

larger

pinout

is

not

required

To

make

our

design

framework

more

realistic

we

allow

such

under-utilization

of

board

and

pinout

capacities

However

in

order

to

minimize

system

cost

we

also

add

a

design

Table

Summary

of

Interplay

Between

Board

Size

-LRB-

b

-RRB-

Pinout

Density

-LRB-

pp

or

ps

-RRB-

Cluster

Size

-LRB-

c

-RRB-

and

Inter-Cluster

Dimensionality

-LRB-

n

-RRB-

to

Maintain

a

Fixed

Channel

Width

-LRB-

W

-RRB-

Board

Size

Pinout

Density

Cluster

Inter-Cluster

Size

Size

Dimensionality

-LRB-

b

-RRB-

-LRB-

pp

or

ps

-RRB-

-LRB-

c

-RRB-

-LRB-

n

-RRB-

fixed

fixed

fixed

fixed

fixed

fixed

fixed

-LRB-

periphery

-RRB-

fixed

fixed

-LRB-

surface

-RRB-

fixed

objective

to

minimize

such

under-utilization

of

resources

Let

us

denote

parameters

up

and

ub

as

the

percentage

utilization

of

board

pinout

and

board

area

respectively

Now

consider

supporting

a

channel

width

of

W

W

with

marginal

under-utilization

of

board

area

and

pinout

resources

being

allowed

Based

on

Eqn

we

can

derive

W

Pbup

-LRB-

b

c

-RRB-

-LRB-

n

-RRB-

n

W

up

ub

-LRB-

-RRB-

where

Pbup

is

the

utilized

pincount

from

board

is

the

actual

number

of

clusters

placed

on

a

board

-LRB-

out

of

the

maximum

b

c

-RRB-

and

W

is

the

channel

width

value

obtained

assuming

full-utilization

of

resources

We

assume

a

reasonable

bound

on

both

under-utilizations

such

that

up

ub

umin

for

some

umin

closer

to

Thus

observations

made

earlier

assuming

full-utilization

-LRB-

umin

-RRB-

continue

to

hold

For

illustrative

purposes

in

this

paper

we

choose

umin

Given

a

pinout

density

and

a

channel

width

technology

we

formulate

the

following

search

problem

to

determine

-LRB-

n

b

c

-RRB-

tuples

which

satisfy

the

following

inequalities

Allowing

only

under-utilization

of

pinout

we

have

up

and

ub

This

leads

to

W

W

-LRB-

ub

up

-RRB-

-RRB-

W

W

-LRB-

-RRB-

Allowing

only

under-utilization

of

board

area

we

have

ub

and

up

This

leads

to

W

W

-LRB-

ub

up

-RRB-

-RRB-

W

W

-LRB-

-RRB-

Figure

shows

the

plots

in

Fig.

with

such

under-utilization

tolerance

lines

which

provide

flexibility

of

around

the

channel

widths

of

W

and

Plots

are

depicted

for

a

periphery

pinout

technology

of

pp

and

are

shown

for

only

one

cluster

per

board

-LRB-

-RRB-

For

example

the

configurations

falling

within

the

tolerance

boundaries

of

W

-LRB-

like

c

c

and

c

-RRB-

offer

a

W

ss

Such

configurations

comply

with

supported

channel

width

-LRB-

W

-RRB-

technology

while

utilizing

board

resources

maximally

In

Section

we

demonstrate

that

a

subset

of

these

configurations

while

complying

with

other

packaging

constraints

like

maximum

router

pinout

and

maximum

board

size

form

the

set

of

design-feasible

or

package-able

configurations

Channel

width

-LRB-

W

-RRB-

cluster

size

-LRB-

c

-RRB-

Periphery

Pinout

Technology

Figure

Configurations

offering

channel

width

-LRB-

W

-RRB-

closer

to

and

with

a

reasonable

-LRB-

-RRB-

under-utilization

of

board

area

or

pinout

A

periphery

pinout

technology

with

pp

is

considered

A

board

is

assumed

to

hold

one

cluster

with

c

processors

-LRB-

-RRB-

Offered

Inter-Cluster

Bisection

Now

let

us

consider

the

impact

of

packaging

on

offered

inter-cluster

bisection

size

As

discussed

earlier

in

Section

the

bisection

size

of

the

inter-cluster

network

B

or

more

specifically

bisection

size

per

processor

B

N

is

indicative

of

the

supportable

average

throughput

per

processor

in

the

system

-LSB-

-RSB-

Observations

derived

in

this

subsection

for

the

bisection

size

per

processor

reflect

similar

trends

on

system

throughput

These

observations

are

expected

to

be

broad

guidelines

to

aid

the

design

decision

process

A

more

accurate

modeling

of

the

average

throughput

in

presence

of

contention

is

presented

in

the

next

section

Given

a

-LRB-

x

x.

x

kn

-RRB-

mesh/torus

inter-cluster

network

with

a

fixed

channel

width

the

size

of

the

inter-cluster

bisection

can

be

computed

in

the

following

manner

A

-LRB-

x

x.

x

kn

-RRB-

mesh/torus

can

have

various

bisections

which

divide

this

network

into

two

halves

For

example

in

a

torus

with

x

y

and

z

dimensions

we

can

have

three

possible

bisections

one

orthogonal

to

x

dimension

along

yz

plane

one

orthogonal

to

y

dimension

along

xz

plane

and

so

on

We

are

interested

in

the

size

of

the

smallest

bisection

in

the

system

because

it

maximally

constrains

the

performance

of

the

system

under

random

traffic

Clearly

the

smallest

bisection

has

to

be

orthogonal

to

the

dimension

having

the

largest

radix

given

by

kmax

maxi

n

-LRB-

ki

-RRB-

The

number

of

nodes

in

such

a

bisection

is

given

by

N

kmax

where

N

denotes

the

number

of

nodes

-LRB-

clusters

-RRB-

in

the

system

Assume

each

of

these

nodes

is

connected

to

another

across

this

bisection

using

unidirectional

channels

of

width

W

This

leads

to

the

offered

bisection

size

B

as

N

W

kmax

wires

Thus

the

expression

for

B

N

the

bisection

size

per

processor

can

be

derived

as

N

W

Nkmax

wires

The

presence

of

wrap-around

channels

in

torus

doubles

this

to

W

Nkmax

This

can

be

simplified

by

assuming

the

inter-cluster

network

to

be

regular

i.e.

-LRB-

kmax

-LRB-

N

c

-RRB-

n

-RRB-

and

N

N

c

to

B

N

W

Nkmax

c

-LRB-

N

c

-RRB-

n

nc

-LRB-

n

-RRB-

n

-LRB-

-RRB-

Based

on

the

above

equation

the

following

observations

can

be

made

Result

For

a

given

channel

width

-LRB-

W

W

-RRB-

and

inter-cluster

dimensionality

-LRB-

n

-RRB-

the

offered

bisection

size

per

processor

-LRB-

B

N

-RRB-

falls

with

increase

in

the

system

size

-LRB-

N

-RRB-

while

keeping

cluster

size

-LRB-

c

-RRB-

fixed

Similarly

the

bisection

size

per

processor

-LRB-

B

N

-RRB-

also

falls

as

the

system

size

-LRB-

N

-RRB-

remains

fixed

and

cluster

size

-LRB-

c

-RRB-

is

increased

Figure

shows

the

bisection

size

per

processor

for

a

system

size

of

N

processors

a

given

channel

width

of

W

and

different

inter-cluster

dimensionality

-LRB-

n

-RRB-

as

cluster

size

is

varied

from

c

It

can

be

observed

that

the

fall

in

bisection

size

per

processor

is

more

appreciable

at

smaller

cluster

sizes

and

higher

inter-cluster

dimensionalities

Bisection

Size

per

Processor

Cluster

Size

-LRB-

c

-RRB-

W

Figure

Offered

inter-cluster

bisection

per

processor

-LRB-

B

N

-RRB-

as

a

function

of

cluster

size

and

inter-cluster

dimensionality

while

maintaining

a

channel

width

W

for

a

processor

system

Note

that

for

a

given

pinout

technology

the

board

sizes

are

not

fixed

and

are

changed

with

cluster

size

to

maintain

W

It

is

also

interesting

to

observe

the

impact

of

board

size

on

offered

bisection

size

per

processor

under

two

different

pinout

technologies

Figure

shows

the

plots

of

B

N

for

a

N

system

The

trends

are

shown

with

respect

to

periphery

and

surface

pinout

technologies

as

board

size

is

increased

To

analyze

these

trends

we

first

simplify

Eqn

by

using

Eqn

to

replace

W

leading

to

B

N

nc

-LRB-

n

-RRB-

n

Pb

-LRB-

n

-RRB-

n

Pb

nb

-LRB-

n

-RRB-

nn

-LRB-

-RRB-

For

a

periphery

pinout

technology

with

Pb

pp

pb

this

can

be

further

simplified

to

B

N

pp

nb

-LRB-

n

-RRB-

-LRB-

-RRB-

n

Thus

for

a

given

system

size

-LRB-

N

-RRB-

and

inter-cluster

network

dimensionality

-LRB-

n

-RRB-

bidirectional

duplex

channels

this

value

needs

to

be

multiplied

by

another

factor

of

Bisection

Size

Per

Processor

Board

Size

-LRB-

b

-RRB-

a

-RRB-

Periphery

Pinout

Technology

Bisection

Size

Per

Processor

Board

size

-LRB-

b

-RRB-

b

-RRB-

Surface

Pinout

Technology

Figure

Offered

inter-cluster

bisection

per

processor

-LRB-

B

N

-RRB-

in

a

system

of

processors

as

a

function

of

board

size

and

different

inter-cluster

dimensionalities

under

two

different

pinout

technologies

a

-RRB-

periphery

pinout

with

pp

and

b

-RRB-

surface

pinout

with

ps

the

bisection

size

per

processor

-LRB-

B

N

-RRB-

falls

with

increase

in

board

size

-LRB-

b

-RRB-

for

n

This

can

be

explained

by

the

fact

that

under

periphery

pinout

the

total

board

pin-count

in

the

system

reduces

as

board

sizes

are

increased

For

n

the

value

of

B

N

remains

fixed

and

for

n

it

rises

slowly

The

key

point

is

that

under

periphery

pinout

technology

it

is

ideal

to

work

with

smaller

boards

as

they

offer

higher

bisection

sizes

and

hence

performance

Similarly

under

surface

pinout

technology

with

Pb

psb

we

can

simplify

Eqn

as

B

n

nn

Here

it

can

be

observed

that

for

a

given

system

size

-LRB-

N

-RRB-

and

inter-cluster

network

dimensionality

-LRB-

n

-RRB-

the

size

of

the

bisection

increases

with

n

These

lead

to

Result

Under

periphery

pinout

technology

for

a

given

system

size

-LRB-

N

-RRB-

keeping

the

dimensionality

of

inter-cluster

network

-LRB-

n

-RRB-

fixed

at

a

value

greater

than

an

increase

in

board

size

-LRB-

b

-RRB-

leads

to

a

fall

in

the

inter-cluster

bisection

size

per

processor

However

under

surface

pinout

technology

as

board

size

-LRB-

b

-RRB-

is

increased

bisection

size

per

processor

increases

for

all

inter-cluster

dimensionalities

Bisection

size

per

processor

has

direct

impact

on

the

maximum

traffic

that

can

be

supported

in

a

system

Thus

trends

similar

to

those

presented

in

this

section

with

respect

to

bisection

size

per

processor

are

expected

in

the

maximum

value

of

the

offered

average

system

throughput

In

the

following

section

we

present

a

simple

analytical

model

for

-LRB-

kn

c

-RRB-

systems

to

estimate

their

performance

in

presence

of

contention

We

use

such

a

model

in

Sec.

to

determine

a

more

accurate

trend

about

the

average

system

throughput

and

offered

average

message

latency

while

deriving

configurations

which

support

a

demanded

performance

Latency-Throughput

Performance

Model

for

k-ary

n-cube

clusterc

Systems

The

Model

We

develop

a

simple

analytical

model

to

predict

performance

in

-LRB-

kn

c

-RRB-

systems

with

small

cluster

sizes

This

model

considers

network

contention

and

determines

latency

and

throughput

parameters

for

a

given

traffic

load

Our

model

is

a

simple

extension

of

the

model

proposed

by

Agrawal

-LSB-

-RSB-

to

predict

performance

in

flat

k-ary

n-cube

networks

with

dimension-order

-LSB-

-RSB-

virtual

cut-through

routing

As

shown

in

-LSB-

-RSB-

the

average

message

latency

in

the

presence

of

contention

through

a

k-ary

n-cube

network

can

be

derived

as

Tc

mF

-LRB-

mFd

-RRB-

-LRB-

d

-RRB-

d

n

nd

F

-LRB-

-RRB-

where

Tc

denotes

the

average

message

latency

expressed

in

network

cycles

Network

cycle

as

defined

earlier

in

Sec.

is

the

time

to

send

a

flit

across

one

hop

in

the

system

The

parameter

d

represents

the

average

number

of

hops

taken

by

a

message

in

a

dimension

For

a

network

with

unidirectional

channels

and

wrap-around

connections

d

-LRB-

k

-RRB-

For

bidirectional

channels

this

value

is

-LRB-

k

-RRB-

The

message

size

expressed

in

flits

is

F

L

W

where

L

is

the

message

length

in

bits

and

W

the

channel

width

The

parameter

m

is

the

message

injection

rate

by

a

processor

in

terms

of

messages/cycle

From

the

above

equation

it

can

be

observed

that

nd

F

cycles

is

the

minimum

latency

suffered

by

a

message

This

happens

at

a

very

small

value

of

message

injection

rate

In

a

clustered

-LRB-

kn

c

-RRB-

system

depending

on

whether

a

message

is

intra-cluster

or

inter-cluster

it

encounters

different

latencies

For

relatively

small

clusters

-LRB-

c

-RRB-

it

is

reasonable

to

assume

advanced

packaging

faster

interconnect

and

wider

buses

inside

a

cluster

leading

to

very

high

intracluster

message

bandwidth

With

such

high

bandwidth

it

is

reasonable

to

expect

small

delays

inside

clusters

Inter-cluster

messages

on

the

other

hand

need

to

travel

across

clusters

The

average

delay

for

inter-cluster

traversal

is

relatively

much

larger

than

average

intra-cluster

delay

Our

objective

is

to

derive

the

average

message

latency

as

a

function

of

average

message

throughput

The

average

message

latency

in

a

clustered

system

is

determined

as

a

weighted

sum

of

the

intra-cluster

and

inter-cluster

message

latencies

The

weights

are

the

relative

frequency

of

these

messages

being

generated

by

a

processor

Under

uniform

traffic

the

probability

of

an

intracluster

message

occuring

is

given

by

the

expression

-LRB-

c

N

-RRB-

Clearly

for

reasonably

large

systems

and

relatively

small

cluster

sizes

the

value

of

c

N

is

negligible

For

example

in

a

system

with

N

processors

and

a

cluster

size

of

c

the

value

of

c

N

The

probability

of

a

message

being

inter-cluster

-LRB-

c

N

-RRB-

is

therefore

very

close

to

This

indicates

that

most

messages

generated

in

the

system

are

inter-cluster

Under

such

an

assumption

the

average

message

latency

in

the

system

is

closer

to

the

average

latency

of

an

inter-cluster

message

In

this

analysis

we

therefore

focus

on

deriving

the

average

latency

of

an

inter-cluster

message

An

uniform

traffic

model

was

chosen

because

it

is

considered

more

representative

while

designing

a

general

purpose

machine

No

prior

knowledge

is

assumed

about

the

nature

of

the

applications

to

be

run

on

it

Let

us

consider

the

latency

of

an

inter-cluster

message

It

has

three

components

Tinter

and

These

terms

denote

the

delays

from

the

source

processor

to

source-CI

through

the

source

intranet

source-CI

to

destination-CI

through

the

k-ary

n-cube

internet

and

dest-CI

to

the

destination

processor

through

the

destination

intranet

respectively

The

exact

expressions

for

and

depend

on

the

cluster

topology

cluster

size

and

rate

of

traffic

However

from

earlier

discussion

we

know

that

these

latency

components

inside

clusters

and

are

much

smaller

as

compared

to

the

inter-cluster

component

Tinter

For

star-based

clusters

which

is

a

popular

trend

in

current

multi-processor

systems

as

discussed

earlier

in

Section

these

components

are

negligible

Assuming

a

clustered

system

offering

small

intra-cluster

latencies

the

average

latency

seen

by

a

message

Tc

is

dominated

by

the

average

inter-cluster

latency

To

estimate

this

factor

we

modify

the

model

in

-LSB-

-RSB-

in

the

following

manner

Given

the

message

injection

rate

from

each

processor

to

be

m

messages/cycle

the

combined

traffic

generated

by

all

the

processors

in

a

cluster

is

mc

messages/cycle

Under

uniform

traffic

assumption

most

of

this

traffic

is

inter-cluster

We

assume

this

resultant

traffic

to

be

injected

by

the

CI

into

the

internet

Assuming

the

resultant

traffic

stream

to

be

Poisson

the

inter-cluster

latency

can

be

predicted

from

Equation

by

replacing

the

parameter

m

by

the

expression

mc

leading

to

Tc

mcF

-LRB-

mcFd

-RRB-

-LRB-

d

-RRB-

d

n

nd

F

-LRB-

-RRB-

Equation

can

be

rewritten

to

express

injection

rate

-LRB-

m

-RRB-

as

a

function

of

Tc

and

other

parameters

as

m

-LRB-

Tc

F

nd

-RRB-

d

cFd

-LRB-

-LRB-

n

-RRB-

-LRB-

d

-RRB-

F

-LRB-

Tc

F

nd

-RRB-

d

-RRB-

-LRB-

-RRB-

Comparing

Performance

of

the

Model

with

Simulations

We

validated

the

above

model

through

simulations

against

a

wide

range

of

clustered

systems

with

varying

inter-cluster

topology

and

cluster

size

with

variety

of

workload

parameters

It

was

found

that

the

model

closely

predicted

the

simulated

performance

For

example

a

comparison

of

simulation

results

with

analytical

prediction

from

the

above

model

are

depicted

in

Fig.

In

Fig.

-LRB-

a

-RRB-

as

message

length

F

is

varied

from

flits

in

a

-LRB-

-RRB-

system

the

model

closely

matches

the

simulation

results

Similar

matches

were

obtained

with

varying

cluster

sizes

from

through

as

depicted

in

Fig

-LRB-

b

-RRB-

Our

clustered

simulation

testbed

models

star-clusters

routes

messages

through

the

inter-cluster

network

in

dimension-order

and

generates

statistics

such

as

average

message

latency

and

observed

message

rates

Each

simulation

was

run

until

the

confidence

interval

of

a

data

point

was

within

of

its

mean

In

the

next

section

we

demonstrate

the

use

of

the

above

model

in

deriving

good

configurations

Average

Message

Latency

-LRB-

T

-RRB-

in

cycles

Throughput

per

processor

-LRB-

messages/cycles

-RRB-

anal

F

sim

F

anal

F

sim

F

anal

F

sim

F

Average

Message

Latency

-LRB-

T

-RRB-

in

cycles

Throughput

-LRB-

messages/cycles

-RRB-

inter-cluster

F

anal

c

sim

c

anal

c

sim

c

anal

c

sim

c

Figure

Comparing

the

analytical

latency-throughput

model

with

simulations

for

a

-RRB-

system

with

message

length

being

varied

from

F

to

flits

and

b

-RRB-

inter-cluster

network

with

cluster

size

being

varied

from

c

to

while

message

length

is

kept

fixed

at

F

flits

Solid

lines

correspond

to

model

predictions

Design

Framework

Methodology

Let

us

put

together

all

the

components

of

our

supply-demand

framework

as

discussed

earlier

For

a

given

system

size

our

goal

is

to

derive

the

best

configuration

This

configuration

should

satisfy

packaging

and

demand

constraints

in

the

most

cost-effective

manner

We

formulate

this

as

a

search

problem

of

selecting

the

best

configuration

from

amongst

the

various

configurations

possible

in

the

design

space

The

problem

is

solved

in

three

phases

Phase

I

Deriving

Design-Feasible

Configurations

The

set

of

configurations

satisfying

all

packaging

constraints

are

derived

in

this

phase

These

packaging

constraints

as

discussed

earlier

are

ffl

All

board

sizes

are

less

than

a

maximum

board

size

-LRB-

b

bmax

-RRB-

ffl

Pinout

from

a

board

is

bounded

by

board

size

and

pinout

density

-LRB-

P

Pb

where

Pb

pp

pb

under

periphery

pinout

technology

and

Pb

psb

under

surface

pinout

technology

-RRB-

ffl

Intended

channel

width

should

be

supportable

by

channel

width

technology

-LRB-

W

W

where

W

and

-RRB-

ffl

Pinout

from

a

router

is

less

than

maximum

supportable

pinout

R.

For

a

given

set

of

packaging

parameters

a

bmax

pp

ps

W

R

and

desired

system

size

N

the

design-feasible

solutions

can

be

easily

derived

through

a

computer-aided

search

We

refer

to

this

set

of

design

feasible

configurations

as

the

supply

side

Phase

II

Deriving

Good

Configurations

The

set

of

good

configurations

is

a

subset

of

the

design-feasible

configurations

satisfying

the

demanded

performance

requirements

The

demanded

performance

requirements

refer

to

the

upper

bound

on

average

network

latency

-LRB-

Tmax

cycles

-RRB-

and

a

minimum

average

throughput

-LRB-

bits/cycle

-RRB-

as

discussed

earlier

in

Sec.

For

ease

of

discussion

we

represent

the

demanded

minimum

average

throughput

in

terms

of

L

messages/cycle

where

L

denotes

the

length

of

a

message

in

bits

For

each

design-feasible

configuration

using

Eqn

we

derive

the

value

of

maximum

average

throughput

that

can

be

sustained

with

average

message

latency

being

less

than

Tmax

Such

a

maximum

value

of

average

throughput

is

denoted

by

mmax

messages/cycle

A

design-feasible

system

configuration

offering

mmax

L

meets

both

latency

and

throughput

demands

on

performance

and

is

therefore

marked

as

a

good

configuration

Any

member

from

the

set

of

good

configurations

can

be

used

to

build

a

machine

under

packaging

constraints

while

offering

the

desired

performance

Phase

III

Deriving

the

Best

Configuration

The

final

phase

of

the

problem

is

to

select

the

best

configuration

from

the

set

of

good

configurations

Cost-effectiveness

offered

maximum

throughput

and

scope

for

future

scalability

are

three

important

considerations

for

deciding

the

best

configuration

For

each

good

configuration

our

framework

provides

information

such

as

exact

board

size

used

number

of

inter-board

connectors

required

and

total

volume

of

wires

Such

information

can

be

used

by

a

system

designer

to

estimate

the

cost

of

a

given

configuration

However

a

cost

comparison

is

dependent

on

the

exact

cost

model

used

Thus

the

search

for

the

configuration

with

lowest

cost

may

not

be

a

very

clear-cut

process

An

alternative

approach

is

to

select

the

good

configuration

offering

the

maximum

value

of

mmax

as

the

best

configuration

Another

factor

considered

is

the

potential

of

a

configuration

for

future

scalability

to

larger

sizes

to

meet

higher

computing

needs

Let

us

analyze

this

scalability

aspect

in

choosing

the

best

configuration

For

example

let

us

consider

the

design-problem

of

building

a

system

with

processors

Once

a

configuration

-LRB-

say

n

c

b

and

W

-RRB-

is

chosen

to

fabricate

a

machine

these

system

parameters

are

fixed

Such

a

system

can

be

scaled

to

a

larger

size

by

adding

more

processors

to

it

However

to

have

a

homogeneous

system

the

values

of

the

parameters

n

b

c

and

W

must

be

maintained

Otherwise

the

system

requires

complete

redesign

and

refabrication

Let

-LRB-

n

b

c

W

-RRB-

denote

the

set

of

all

n

b

c

and

W

values

that

represent

good

configurations

to

build

a

system

of

N

processors

Similarly

let

-LRB-

n

b

c

W

-RRB-

denote

such

a

set

for

system

size

of

N

processors

The

intersection

set

-LRB-

n

b

c

W

-RRB-

T

-LRB-

n

b

c

W

-RRB-

therefore

denotes

good

configurations

under

both

N

and

processors

Such

configurations

can

be

used

to

build

a

system

with

processors

which

can

be

scaled

up

to

processors

while

still

meeting

demanded

performance

We

emphasize

on

such

scalability

in

our

framework

to

derive

best

configurations

Applying

the

Framework

In

this

section

we

illustrate

the

above

three

phase

process

by

deriving

the

best

configuration

to

design

a

system

with

N

ss

processors

for

a

representative

set

of

packaging

and

technological

parameters

The

packaging

parameters

used

are

bmax

pp

and

R

A

demanded

performance

of

Tmax

cycles

bits/cycle

and

a

message

length

of

L

bits

is

assumed

Deriving

Design-Feasible

Configurations

Table

summarizes

various

design-feasible

configurations

obtained

through

the

framework

for

three

different

values

of

W

and

The

results

are

organized

in

columns

with

respect

to

number

of

clusters

per

board

-LRB-

-RRB-

For

example

column

in

Table

summarizes

these

solutions

with

one

cluster

per

board

-LRB-

-RRB-

Similarly

columns

and

in

Table

present

solutions

obtained

with

allowable

number

of

clusters

per

board

of

and

respectively

Each

entry

is

either

a

single

cluster

size

or

a

range

of

cluster

sizes

Such

a

cluster

size

coupled

with

the

value

of

dimension

-LRB-

n

-RRB-

indicated

on

the

row

of

the

entry

represents

a

feasible

configuration

For

example

the

entry

of

c

in

column

and

the

row

corresponding

to

W

and

n

represents

a

feasible

configuration

Blank

entries

in

the

table

indicate

that

there

was

no

valid

configuration

for

that

inter-cluster

dimensionality

pinout

technology

channel

width

and

board

area

For

a

given

value

of

W

the

inter-cluster

dimensionality

was

varied

up

to

a

maximum

dictated

by

the

router

pinout

constraint

of

R

as

discussed

in

Sec.

From

Table

we

observe

that

for

thinner

supportable

channel

width

-LRB-

W

-RRB-

configurations

with

lower

dimensionality

are

not

design-feasible

-LRB-

hence

not

depicted

in

the

table

-RRB-

This

limits

wastage

of

board

resources

as

discussed

in

Sec.

Similarly

for

wider

channel

width

-LRB-

W

-RRB-

configurations

with

higher

dimensionality

are

not

design-feasible

in

order

to

remain

within

the

router

pinout

limit

From

Table

it

can

also

be

observed

that

for

wider

channel

width

technologies

-LRB-

W

and

-RRB-

clustered

organizations

-LRB-

c

-RRB-

are

feasible

while

flat

organizations

are

not

Note

that

the

above

representation

of

design-feasible

solutions

is

independent

of

the

total

processors

in

the

system

Given

a

system

size

of

N

processors

the

above

representation

of

feasible

configurations

can

be

expanded

as

explained

in

the

paragraph

below

In

the

remaining

discussion

we

continue

Table

Valid

Ranges

of

Cluster

Size

-LRB-

c

-RRB-

from

Packaging

Constraints

are

Shown

for

a

Periphery

Pinout

Density

of

pp

and

Three

Different

Supportable

Channel

Widths

of

W

and

Inter-Cluster

Dimensionality

-LRB-

n

-RRB-

is

Varied

up

to

a

Maximum

Dictated

by

the

Router

Pinout

Constraint

of

R

Number

of

Clusters/Board

-LRB-

-RRB-

is

Varied

from

Largest

Board

Capacity

-LRB-

b

-RRB-

is

bounded

by

bmax

n

cluster

size

-LRB-

c

-RRB-

W

W

W

the

illustration

of

the

framework

for

only

one

channel

width

of

W

The

framework

can

be

similarly

applied

to

other

values

of

W

For

each

of

the

design-feasible

configuration

in

Table

we

first

derive

the

exact

inter-cluster

topology

to

realize

a

system

with

N

ss

processors

For

example

consider

the

entry

in

Table

corresponding

to

n

c

and

W

The

desired

number

of

clusters

in

this

system

with

processors

is

Let

us

consider

selecting

a

-LRB-

n

-RRB-

inter-cluster

topology

to

interconnect

clusters

The

closest

configuration

offering

inter-cluster

is

The

resultant

system

configuration

has

clusters

or

processors

which

is

more

than

the

desired

number

of

Without

loss

of

generality

we

allow

such

deviations

in

system

size

up

to

reasonable

limits

Similarly

other

configurations

corresponding

to

W

are

also

expanded

All

design-feasible

configurations

corresponding

to

W

in

Table

are

presented

in

the

left

half

of

Table

Deriving

Good

Configurations

From

among

the

design-feasible

configurations

presented

in

the

left

half

of

Table

we

can

derive

a

set

of

good

configurations

Let

us

assume

a

demanded

performance

of

Tmax

cycles

bits/cycle

and

a

message

length

of

L

bits

This

leads

to

L

messages/cycle

For

each

configuration

in

Table

the

value

of

maximum

throughput

sustainable

-LRB-

mmax

-RRB-

while

maintaining

average

message

latency

less

than

Tmax

cycles

is

shown

next

to

it

These

values

are

derived

using

Eqn

Some

of

the

values

for

very

low

radix

systems

were

derived

through

simulation

experiments

This

is

because

Agrawal

's

model

-LSB-

-RSB-

on

which

our

model

is

based

does

not

hold

for

very

low

radix

configurations

The

performance

plots

depicting

the

average

message

latency

versus

average

throughput

for

all

feasible

configurations

to

build

a

N

ss

processor

system

are

shown

in

Fig.

These

were

obtained

through

actual

simulation

experiments

These

plots

demonstrate

similar

comparative

trends

between

configurations

as

derived

by

the

analytical

model

The

configurations

in

Table

offering

mmax

L

are

good

configurations

and

depicted

in

boldface

For

example

the

configurations

and

were

derived

to

be

good

Table

Design

Feasible

Configurations

to

Build

Systems

with

N

ss

and

Processors

under

the

Packaging

Parameters

pp

W

R

and

bmax

Good

Configurations

are

Shown

in

Boldface

and

these

Offer

mmax

L

with

Tmax

cycles

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

N

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

N

Average

Message

Latency

-LRB-

T

-RRB-

in

cycles

Average

Throughput

per

Processor

-LRB-

m

-RRB-

messages/cycles

Design-Feasible

Configurations

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Comparing

the

performance

obtained

through

simulation

of

the

design-feasible

configurations

shown

in

Table

to

build

a

system

with

N

ss

processors

Deriving

the

Best

Configuration

Let

us

first

consider

the

best

configuration

derived

based

on

selecting

the

good

configuration

offering

the

maximum

value

of

mmax

From

Table

this

best

configuration

is

-LRB-

-RRB-

with

a

mmax

messages/cycle

Next

let

us

analyze

the

best

configuration

with

potential

to

scale

up

to

a

system

size

of

N

ss

processors

The

design-feasible

configurations

and

the

good

configurations

were

also

derived

for

the

larger

system

size

with

N

ss

processors

for

a

similar

set

of

packaging

and

performance

parameters

These

are

shown

in

the

right

half

of

Table

For

a

set

of

packaging

parameters

the

set

of

design-feasible

configurations

-LRB-

in

terms

of

n

and

c

-RRB-

are

similar

irrespective

of

size

of

the

system

being

designed

However

the

size

of

the

inter-cluster

needs

to

be

larger

to

accommodate

more

clusters

For

example

consider

the

-LRB-

-RRB-

feasible

configuration

shown

in

first

row

of

Table

to

build

a

N

ss

system

This

configuration

needs

to

be

scaled

to

a

larger

-LRB-

-RRB-

feasible

configuration

to

build

a

system

with

N

ss

processors

However

from

Result

we

know

that

the

inter-cluster

bisection

size

per

processor

does

not

scale

linearly

with

system

size

This

is

also

reflected

in

a

fall

in

the

value

of

supportable

throughput

mmax

derived

using

Eqn

from

to

messages/cycle

as

shown

in

Table

Similarly

for

other

configurations

a

fall

in

the

offered

value

for

mmax

was

observed

with

increasing

system

size

It

can

be

observed

that

only

one

configuration

is

capable

of

meeting

the

demanded

performance

of

L

messages/cycle

to

build

a

N

ss

processor

system

This

leads

us

to

conclude

that

-LRB-

-RRB-

is

the

only

good

configuration

to

build

a

system

with

processors

The

intersection

of

this

set

with

the

set

of

good

configurations

for

a

system

with

processors

again

leads

us

to

the

as

the

best

configuration

This

configuration

has

potential

to

scale

up

to

processors

while

still

offering

the

minimal

demanded

performance

In

the

above

example

it

is

a

coincidence

that

the

best

configuration

derived

from

maximum

throughput

and

scope

for

scalability

turned

out

to

be

same

Similar

to

the

above

illustration

our

framework

can

be

applied

to

derive

the

best

configuration

under

different

sets

of

packaging

and

demand

parameters

Considering

the

above

example

as

a

base

case

we

also

studied

the

impact

of

varying

different

packaging

and

demand

parameters

on

design

Further

results

on

best

configurations

are

derived

by

choosing

the

good

configuration

offering

maximum

value

of

mmax

Impact

of

Varying

Packaging

and

Demand

Parameters

on

the

Design

Process

In

this

section

we

analyze

the

impact

of

changing

various

packaging

and

demand

parameters

on

the

set

of

design-feasible

and

good

configurations

Processor

and

Interconnect

Technology

For

a

given

set

of

packaging

technologies

the

set

of

design-feasible

solutions

is

fixed

However

the

set

of

good

configurations

is

dependent

on

the

demanded

values

of

Tmax

and

In

Section

it

was

indicated

that

the

value

of

depends

on

the

relative

advancements

in

processor

and

interconnect

technologies

We

analyzed

the

impact

of

varying

on

the

set

of

good

configurations

Table

shows

the

same

design-feasible

configurations

as

shown

in

Table

to

build

a

system

with

N

ss

processors

The

left

half

of

Table

depicts

the

good

configurations

for

a

lower

value

of

corresponding

to

L

messages/cycle

The

right

half

of

Table

similarly

depicts

the

good

configurations

for

a

higher

value

of

corresponding

to

L

messages/cycle

In

this

example

it

was

observed

that

for

a

L

messages/cycle

all

design-feasible

configurations

are

good

-LRB-

depicted

in

boldface

-RRB-

However

for

the

higher

value

of

L

messages/cycle

only

the

and

configurations

are

good

Under

both

values

of

the

best

configuration

based

on

maximum

value

of

offered

mmax

is

This

best

configuration

is

depicted

in

Table

preceded

by

a

sign

Table

Impact

of

Demanded

Throughput

-LRB->=-RRB-

on

the

Set

of

Good

Configurations

to

Design

a

System

with

N

ss

Processors

Feasible

and

Good

Configurations

are

Shown

for

Two

Values

of

and

Packaging

Parameters

of

pp

W

R

and

bmax

are

Assumed

Maximum

Average

Throughput

mmax

is

Derived

for

Tmax

cycles

A

Best

Configuration

is

Preceded

by

a

sign

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

Increasing

Supported

Channel

Width

We

studied

the

impact

of

wider

channel

width

of

W

while

designing

a

system

with

N

ss

processors

The

other

parameters

were

maintained

at

pp

R

and

bmax

The

resultant

feasible

configurations

are

shown

in

Table

The

performance

plots

depicting

the

average

message

latency

versus

average

throughput

for

these

configurations

obtained

through

actual

simulation

are

shown

in

Figure

The

simulation

results

again

conform

to

the

latency-throughput

trends

indicated

by

the

analytical

model

Let

us

compare

these

configurations

with

those

obtained

in

Table

for

a

similar

design

problem

with

W

It

can

be

observed

that

higher

dimensional

systems

-LRB-

n

-RRB-

feasible

earlier

under

W

are

no

longer

feasible

under

W

Similarly

for

a

given

inter-cluster

dimension

under

both

channel

widths

the

cluster

size

is

larger

with

W

For

example

the

configuration

is

feasible

with

W

while

configurations

with

larger

cluster

sizes

and

become

feasible

with

W

These

results

confirm

the

observations

in

Result

The

best

configuration

in

this

case

was

derived

to

be

-LRB-

-RRB-

Table

Impact

of

an

Increased

Channel

Width

Technology

-LRB-

W

-RRB-

on

the

Set

of

Feasible

Configurations

to

Design

a

System

with

N

ss

Processors

Feasible

and

Good

Configurations

are

Shown

for

a

W

Other

Packaging

Parameters

of

pp

R

and

bmax

are

Assumed

Maximum

Average

Throughput

mmax

is

Derived

for

Tmax

cycles

Good

configurations

are

Depicted

in

Boldface

for

mmax

L

The

Best

Configuration

is

Preceded

by

a

sign

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

pp

W

Average

Message

Latency

-LRB-

T

-RRB-

in

cycles

Average

Throughput

per

Processor

-LRB-

m

-RRB-

in

messages/cycles

Design-Feasible

Configurations

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Comparing

the

performance

obtained

through

simulation

of

the

design-feasible

configurations

shown

in

Table

to

build

a

system

with

N

ss

processors

Increasing

Board

Pinout

Density

As

observed

in

Table

in

Section

an

increase

in

board

pinout

density

makes

higher

dimensional

inter-cluster

networks

feasible

Similarly

smaller

cluster

sizes

also

become

feasible

with

fixed

inter-cluster

dimensionality

These

impacts

of

increasing

pinout

density

of

pp

and

were

analyzed

while

designing

a

system

with

N

ss

processors

The

other

parameters

were

maintained

at

values

used

in

Sec.

R

W

and

bmax

The

resultant

feasible

configurations

obtained

are

shown

in

Table

We

observed

to

be

a

feasible

configuration

under

pp

in

Table

With

increasing

pinout

density

pp

the

feasibility

set

shifted

to

support

a

smaller

cluster

size

a

configuration

as

shown

in

the

first

row

of

Table

With

even

higher

pinout

density

pp

a

topology

can

not

utilize

the

increased

pinout

effectively

because

the

cluster

size

can

not

be

smaller

than

one

Thus

a

configuration

is

no

longer

feasible

and

therefore

not

depicted

in

Table

Although

higher

dimensional

-LRB-

n

-RRB-

systems

become

supportable

with

increased

pinout

density

the

maximum

router

pinout

-LRB-

R

-RRB-

constrains

systems

with

n

from

being

feasible

The

best

configurations

derived

with

pp

and

were

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

Table

Impact

of

Increasing

Pinout

-LRB-

pp

-RRB-

on

the

Set

of

Good

Configurations

to

Design

a

System

with

N

ss

Processors

Feasible

and

Good

Configurations

are

Shown

for

Two

Pinout

Densities

of

pp

and

Other

Packaging

Parameters

of

W

R

and

bmax

are

Assumed

Maximum

Average

Throughput

mmax

is

Derived

for

Tmax

cycles

Good

configurations

are

Depicted

in

Boldface

for

mmax

L

A

Best

Configuration

is

Preceded

by

a

sign

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

pp

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

pp

Increasing

the

Maximum

Router

Pinout

The

impact

of

increasing

maximum

router

pinout

from

R

and

was

observed

while

designing

a

system

with

N

ss

processors

The

other

parameters

were

maintained

at

pp

W

and

bmax

The

resultant

feasible

configurations

are

shown

in

Table

The

feasible

configurations

derived

for

a

similar

design

problem

with

R

were

presented

in

Table

By

comparing

Tables

and

it

can

be

observed

that

the

maximum

dimensionality

of

feasible

configurations

increased

from

n

to

as

R

was

raised

from

to

Larger

dimensional

systems

were

also

associated

with

larger

cluster

sizes

configurations

with

c

and

and

configuration

with

c

These

results

confirm

the

observation

in

Sec.

The

best

configurations

derived

with

R

and

were

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

Table

Impact

of

Increased

Router

Pinout

-LRB-

R

-RRB-

on

the

Set

of

Feasible

Configurations

to

Design

a

System

with

N

ss

Processors

Feasible

and

Good

Configurations

are

Shown

for

Two

Values

of

R

and

Other

Packaging

Parameters

of

pp

W

and

bmax

are

Assumed

Maximum

Average

Throughput

mmax

is

Derived

for

Tmax

cycles

Good

configurations

are

Depicted

in

Boldface

for

mmax

L

A

Best

Configuration

is

Preceded

by

a

sign

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

R

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

R

Maximum

Board

Capacity

We

observed

the

impact

of

lowering

the

maximum

board

capacity

to

bmax

Table

shows

minimum

board

sizes

required

to

build

the

feasible

configurations

shown

earlier

in

Table

With

reduced

maximum

board

size

bmax

the

earlier

configurations

offering

b

are

no

longer

feasible

The

feasible

configurations

with

b

are

depicted

in

Table

preceded

by

a

sign

In

this

case

the

configuration

offering

maximum

value

of

mmax

remains

feasible

even

with

smaller

maximum

board

size

This

leads

us

to

-LRB-

-RRB-

as

the

best

configuration

Table

Impact

of

Reduced

Maximum

Board

Size

-LRB-

bmax

-RRB-

on

the

Set

of

Feasible

Configurations

to

Design

a

System

with

N

ss

Processors

Feasible

Configurations

from

Table

for

a

bmax

are

Shown

Along

with

the

Required

Board

Size

in

the

System

A

Lower

bmax

Restricts

the

Set

of

Feasible

Configurations

The

New

Feasible

Configurations

are

Shown

Preceded

by

a

Sign

and

a

Best

Configuration

is

Preceded

by

a

sign

Internet

cluster

Maximum

Minimum

Board

topology

size

Average

Size

Required

-LRB-

c

-RRB-

Throughput

-LRB-

b

-RRB-

-LRB-

mmax

-RRB-

N

Surface

Pinout

Technology

In

Fig.

in

Sec.

it

was

indicated

that

surface

pinout

technology

has

potential

to

scale

better

with

board

capacity

and

offer

higher

pinout

than

periphery

technology

The

impact

of

surface

pinout

technology

on

design

was

found

to

be

similar

to

that

observed

with

higher

board

pinout

density

Higher

dimensional

networks

supporting

smaller

clusters

become

more

feasible

Table

shows

the

feasible

configurations

under

a

surface

pinout

density

of

ps

W

R

and

bmax

Comparing

these

configurations

to

those

shown

in

Table

for

a

similar

design

problem

under

periphery

pinout

density

of

pp

it

can

be

observed

that

surface

pinout

technology

leads

to

systems

with

smaller

clusters

From

Table

we

have

two

best

configurations

in

this

case

offering

the

same

maximum

value

of

mmax

Table

Feasible

and

Good

Configurations

to

Design

a

System

with

N

ss

Processors

Packaging

Parameters

of

ps

bmax

W

and

R

are

Assumed

Maximum

Average

Throughput

mmax

is

Derived

for

Tmax

cycles

Good

configurations

are

Depicted

in

Boldface

for

mmax

L

A

Best

Configuration

is

Preceded

by

a

sign

Internet

cluster

Maximum

topology

size

Average

-LRB-

c

-RRB-

Throughput

-LRB-

mmax

-RRB-

ps

W

Conclusions

and

Future

Work

In

this

paper

we

have

presented

a

comprehensive

supply-demand

framework

to

design

large

multiprocessor

systems

by

taking

into

account

advancements

in

packaging

processor

and

interconnection

technologies

The

framework

explores

the

design

space

of

flat

k-ary

n-cube

topologies

and

their

clustered

variations

-LRB-

k-ary

n-cube

cluster-c

-RRB-

to

derive

design-feasible/best

configurations

The

elegance

of

this

framework

lies

in

its

parameterized

representation

of

different

technologies

and

then

deriving

best

system

configuration

for

any

set

of

technologies

and

constraints

while

considering

practical

design

aspects

like

maximum

board

area

maximum

available

pinout

fixed

channel

width

scalability

etc.

The

significance

of

this

framework

lies

in

its

generality

which

has

never

been

taken

into

consideration

by

earlier

researchers

All

proposed

earlier

works

have

centered

around

fixed

technology

and

packaging

constraints

This

generalized

framework

can

be

applied

to

a

wide

variety

of

technological

parameters

and

constraints

for

years

to

come

Using

this

framework

the

following

design

guidelines

have

been

obtained

for

building

large

multiprocessor

systems

with

k-ary

n-cube

cluster-c

organizations

A

two-level

clustered

architecture

widens

the

design

space

in

terms

of

alternative

configurations

possible

to

build

a

system

with

a

given

number

of

processors

Flat

configurations

may

not

be

design-feasible

under

all

packaging

technologies

On

the

other

hand

clustered

configurations

demonstrate

higher

potential

in

offering

design-feasible

configurations

This

indicates

that

clustered

systems

would

be

a

dominant

trend

in

building

future

systems

For

a

wide

range

of

technological

parameters

it

is

shown

that

best

configurations

are

achieved

with

processors

per

cluster

and

inter-cluster

interconnection

This

research

has

emphasized

on

the

inter-cluster

network

and

cluster

size

while

designing

clustered

systems

In

the

next

phase

of

this

research

we

are

investigating

on

the

design

of

the

intra-cluster

interconnection

and

the

cluster

interface

The

cluster

interface

may

be

augmented

by

adding

extra

functional

capacities

to

implement

various

communication

and

synchronization

operations

more

efficiently

It

will

be

interesting

to

identify

the

nature

and

extent

of

these

extra

capacities

to

be

provided

as

a

function

of

the

cluster

size

and

their

impact

on

overall

system

performance

References

-LSB-

-RSB-

S.

Abraham

and

K.

Padmanabhan

Constraint

based

evaluation

of

multicomputer

networks

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

A.

Agrawal

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

Oct

-LSB-

-RSB-

A.

Asthana

H.

Jagdish

and

B.

Mathews

Impact

of

Advanced

VLSI

packaging

on

the

design

of

a

large

parallel

computer

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Scalable

Architectures

with

k-ary

n-cube

cluster-c

organization

In

Proc

of

the

Symposium

of

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Large

Hierarchical

Multiprocessor

Systems

under

Pro

cessor

Interconnection

and

Packaging

Advancements

In

Proc

of

the

Int

l

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Large

Hierarchical

Multiprocessor

Systems

under

Processor

Interconnection

and

Packaging

Advancements

Technical

Report

OSU-CIS

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Jan

-LSB-

-RSB-

D.

Carlson

The

Mesh

with

a

Global

Mesh

a

Flexible

High-speed

Organization

for

Parallel

Computation

In

Proc

of

the

Int

Conf

on

Supercomputer

Systems

IEEE

Comp

Soc

Press

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

W.

J.

Dally

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Trans

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

S.

Dandamudi

and

D.

Eager

Hierarchical

Interconnection

Networks

for

Multicomputer

Systems

IEEE

Trans

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Performance

of

Hierarchical

Systems

with

wiring

constraints

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Impact

of

Wiring

Constraints

on

Hierarchical

Network

Performance

In

Proc

of

the

Int

Parallel

Processing

Symposium

Mar

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

A.

W.

Wilson

Jr.

Hierarchical

Cache/Bus

Architecture

for

Shared

Memory

Multiprocessors

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

A.

Louri

and

H.

Sung

Optical

Interconnects

for

High-Speed

Interchip

and

Interboard

Communications

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

M.

D.

Noakes

D.

A.

Wallach

and

W.

J.

Dally

The

J-Machine

Multicomputer

An

Architectural

Evaluation

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

K.

Padmanabhan

Effective

Architectures

for

Data

Access

in

a

Shared

Memory

Hierarchy

Jour

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

D.

K.

Panda

and

D.

Basak

Issues

in

Designing

Scalable

Systems

with

k-ary

n-cube

clusterc

Organization

In

Proc

of

the

First

International

Workshop

on

Parallel

Processing

India

pages

-LCB-

-LSB-

-RSB-

D.

A.

Patterson

Observations

in

Massive

Parallelism

Trends

and

Predictions

for

to

Technical

Report

DIMACS

Sept

-LSB-

-RSB-

M.

T.

Raghunath

Interconnection

Network

Design

Based

on

Packaging

Considerations

PhD

thesis

U.

C.

Berkeley

Nov

-LSB-

-RSB-

M.

T.

Raghunath

and

A.

Ranade

Designing

interconnection

networks

for

multi-level

packaging

In

Proc

of

the

Supercomputing

pages

-LCB-

-LSB-

-RSB-

E.

Rothberg

J.

P.

Singh

and

A.

Gupta

Working

Sets

Cache

Sizes

and

Node

Granularity

Issues

for

Large-Scale

Multiprocessors

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

A.

A.

Sawchuk

B.

K.

Jenkins

C.

S.

Raghavendra

and

A.

Varma

Optical

Crossbar

Networks

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

S.

L.

Scott

and

J.

R.

Goodman

The

impact

of

pipelined

channels

on

k-ary

n-cube

networks

IEEE

Trans

on

Parallel

and

Distributed

Systems

pages

-LCB-

Jan

-LSB-

-RSB-

Seraphim

Lasky

and

Li

Eds

Principles

of

Electronic

Packaging

McGraw

Hill

-LSB-

-RSB-

Tummala

and

Rymaszewski

Microelectronics

Packaging

Handbook

Van

Nostrand

Reinhold

multiprocessor system design
packaging constraints
scalable systems

clustered architectures
hierarchical organization
k-ary n-cube interconnection
parallel architectures