Designing

Clustered

Multiprocessor

Systems

under

Packaging

and

Technological

Advancements

Debashis

Basak

and

Dhabaleswar

K.

Panda

Department

of

Computer

and

Information

Science

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

fbasak,pandag@cis.ohio-state.edu

Abstract

Packaging

technologies

impose

various

physical

constraints

on

bisection

bandwidth

pinout

and

channel

width

of

a

system

whereas

processor

and

interconnect

technologies

lead

to

certain

demanded

throughput

on

network

bisection

Earlier

studies

in

literature

have

proposed

hierarchical

and

clustered

interconnections

by

considering

the

effect

of

limited

packaging

constraints

Pinout

technologies

and

capacity

of

packaging

modules

have

been

ignored

often

leading

to

configurations

which

are

not

design-feasible

In

this

paper

we

solve

this

design

problem

by

proposing

a

new

supplydemand

optimization

framework

This

generalized

framework

uses

parameterized

representation

of

processor

board

area

pinout

technologies

-LRB-

periphery

or

surface

-RRB-

channel

width

and

channel

speed

The

family

of

flat

k-ary

n-cube

topologies

and

their

clustered

variations

-LRB-

k-ary

n-cube

cluster-c

-RRB-

are

evaluated

to

derive

optimal

configurations

which

can

lead

to

cost-effective

design

of

scalable

parallel

systems

using

wormhole-routing

The

analysis

identifies

processor

board

area

channel

width

and

pinout

density

as

critical

parameters

The

study

indicates

that

cluster-based

parallel

systems

can

deliver

better

performance

with

lower

cost

It

is

predicted

that

optimal

configurations

for

future

systems

will

be

cluster-based

-LRB-

processors

per

cluster

-RRB-

with

inter-cluster

interconnection

This

framework

is

quite

general

to

capture

technological

trends

of

future

years

The

framework

is

validated

by

the

design

solutions

of

current

machines

using

contemporary

technologies

research

is

supported

by

the

National

Science

Foundation

Grant

A

preliminary

version

of

this

paper

has

appeared

in

the

proceedings

of

the

International

Conference

on

Parallel

Processing

Contents

Introduction

Two-Level

Clustering

with

k-ary

n-cube

cluster-c

Organization

Demand

on

Network

Bisection

Bandwidth

Effect

of

Processor

Speed

Effect

of

Interconnect

Speed

Characterizing

Demanded

Bisection

Size

Modeling

and

Parameterization

of

Packaging

Technologies

Hierarchical

Packaging

Model

Parameterizing

Packaging

Technologies

Processor

board

technology

Pinout

Technologies

Channel

Width

Technology

Under-Utilization

of

Board

Area

and

Pinout

Capacity

Impact

of

Packaging

on

Architectural

Parameters

Offered

Inter-Cluster

Channel

Width

Supporting

a

Fixed

Inter-Cluster

Channel

Width

Realizing

Designs

with

Under

Utilization

Offered

Inter-Cluster

Bisection

Impact

of

board

size

Indirect

impact

of

cluster

size

Deriving

Optimal

System

Configurations

Demand

vs

Supply

An

Optimization

Problem

Design

Choices

under

Various

Technologies

Deciding

on

Good

Configurations

Deriving

the

Best

Configuration

Applying

the

Framework

to

Contemporary

Design

Trends

Summary

of

Best

Scalable

Configurations

Conclusions

and

Future

Work

Introduction

Rapid

developments

in

the

field

of

processor

interconnect

and

packaging

technologies

make

the

task

of

efficient

design

of

large

multiprocessor

systems

a

difficult

one

-LSB-

-RSB-

Design

guidelines

to

derive

the

most

optimal

interconnection

have

to

take

into

account

changes

in

these

technologies

to

yield

best

results

Several

previous

studies

have

considered

packaging

constraints

-LSB-

-RSB-

while

selecting

the

best

system

configuration

These

studies

include

Dally

's

-LSB-

-RSB-

analysis

of

k-ary

n-cube

interconnection

under

a

VLSI

model

with

constant

bisection

bandwidth

and

Abraham

and

Padmanabhan

's

study

-LSB-

-RSB-

under

a

constant

pinout

from

a

processing

node

Agrawal

's

-LSB-

-RSB-

analysis

of

the

same

class

of

networks

considers

three

different

constraints

constant

bisection

width

constant

channel

width

and

constant

pinout

and

is

based

on

a

more

comprehensive

model

of

node

and

wire

delays

However

Ranade

-LSB-

-RSB-

and

Yew

-LSB-

-RSB-

argued

that

neither

Dally

's

VLSI

model

with

limited

bisection

bandwidth

nor

the

limited

pinout

model

as

proposed

in

-LSB-

-RSB-

is

adequate

for

very

large

systems

Both

models

confine

to

only

one

level

of

packaging

hierarchy

whereas

large

systems

typically

employ

several

levels

of

packaging

It

was

demonstrated

in

-LSB-

-RSB-

that

considering

two-level

hierarchical/clustered

systems

widen

the

scope

of

the

design-space

The

architectural

levels

can

be

chosen

to

closely

match

the

packaging

hierarchy

leading

to

better

designs

A

variety

of

two-level

hierarchical

configurations

have

been

proposed

by

researchers

in

the

past

to

build

scalable

systems

Though

a

larger

number

of

levels

might

be

more

general

it

is

commonly

believed

that

most

systems

in

the

near

future

will

fit

in

two

hierarchies

Moreover

the

techniques

which

work

for

two-level

hierarchies

can

be

easily

extended

to

design

more

levels

Examples

of

previous

work

in

this

area

include

two-level

systems

based

on

hypercube

and

other

network

topologies

-LSB-

-RSB-

MINs

and

n-hop

networks

-LSB-

-RSB-

two-level

systems

with

k-ary

n-cube

topologies

-LSB-

-RSB-

and

combination

of

bus

and

mesh/hypercube

networks

-LSB-

-RSB-

However

most

of

these

analyses

did

not

take

into

account

packaging

and

interconnecting

constraints

Guidelines

developed

in

-LSB-

-RSB-

were

based

only

on

fixed

board

sizes

with

fixed

pinouts

and

did

not

consider

changes

in

board

sizes

and

alternate

pinout

technologies

The

impact

of

the

changes

in

processor

and

interconnect

technologies

in

relation

to

packaging

technology

was

also

not

studied

Analogously

in

our

analysis

we

focus

on

two-level

clustered

architectures

based

on

a

k-ary

n-cube

cluster-c

-LSB-

-RSB-

interconnection

These

are

clustered

extensions

of

the

flat

k-ary

n-cube

systems

Our

design

model

is

more

comprehensive

and

flexible

in

terms

of

considering

packaging

and

interconnect

constraints

Among

other

things

we

allow

varying

board

sizes

only

reasonable

channel

widths

and

flexible

pinout

from

board

which

depends

on

its

size

System

design

under

alternate

pinout

technologies

like

periphery

and

surface

is

also

explored

The

increase

in

level

of

integration

with

time

in

terms

of

smaller

chips

larger

wiring

boards

and

wiring

and

pinout

densities

is

also

captured

and

the

impact

of

each

studied

on

system

design

In

this

paper

we

propose

a

new

framework

as

shown

in

Fig.

for

designing

and

developing

clustered

architectures

We

consider

a

packaging

technology

with

two

levels

the

lower

level

made

of

chips/multi-chip

modules

and

the

upper

level

made

up

of

processor

wiring

boards

We

analyze

the

problem

of

designing

an

efficient

system

with

N

processors

using

two-levels

Theoretically

a

large

number

of

alternate

configurations

are

possible

to

build

such

a

system

The

configurations

which

conform

to

the

packaging

restrictions

like

maximum

board

size

the

pinout

technology

being

used

peripheral

or

surface

and

the

pinout

densities

are

defined

by

us

as

design-feasible

configurations

It

is

clear

that

only

the

design-feasible

configurations

can

translate

into

real

machines

Each

of

the

design-feasible

configurations

offer

a

level

of

performance

depending

on

its

architectural

characteristics

Referring

to

the

set

of

design-feasible

configurations

and

their

performance

characteristics

as

what

the

packaging

technology

offers

we

call

this

the

supply

side

On

the

other

hand

to

sustain

processor

performance

in

terms

of

throughput

places

a

demand

on

the

size

of

the

required

bisection

bandwidth

-LSB-

-RSB-

of

the

system

The

design-feasible

configurations

which

offer

performance

greater

than

what

is

demanded

are

defined

as

good

configurations

Among

the

good

choices

the

one

which

provides

desired

performance

at

optimal

cost

is

defined

as

the

best

configuration

Our

goal

is

to

derive

this

best

topology

Simulation

modeling

is

also

employed

to

determine

exact

performance

to

decide

between

very

close

good

candidates

Processor

board

Pinout

Applications

Communication

locality

volume

Packaging

Technology

Architecture

characterestics

technology

technology

Size

and

topology

of

each

level

Bisection

bandwidth

and

channel

width

in

each

hierarchy

Maximum

board

size

Pinout

density

Surface

technology

Periphery

technology

Bisection

bandwidth

demanded

by

processors

in

each

hierarchy

match

determines

relationship

Processor

Speeds

Interconnection

Technology

in

each

Maximum

channel

widths

Commmunication

Link

speeds

SUPPLY

DEMAND

System

size

number

of

hierarchies

hierarchy

Good

configurations

Best

Configuration

Design

Guideline

Framework

Simulation

-LRB-

if

multiple

good

choices

-RRB-

Design-Feasible

configurations

-LRB-

Supply

Demand

-RRB-

Traffic

Figure

A

supply-demand

optimization

framework

for

designing

and

developing

scalable

architectures

under

varying

processor

interconnection

and

packaging

technologies

The

paper

is

organized

as

follows

In

Sec.

we

present

the

two-level

k-ary

n-cube

cluster-c

architecture

used

in

later

discussion

Processor

speeds

communication

link

speeds

and

the

per

formance

demand

are

discussed

in

Sec.

In

Sec.

we

discuss

the

trends

in

growth

of

processor

board

sizes

alternate

pinout

technologies

and

channel

width

technology

In

Sec.

expressions

for

offered

channel

width

and

bisection

bandwidth

and

the

design-feasible

configurations

under

different

packaging

and

processor

and

interconnect

technologies

are

derived

In

Sec.

we

derive

the

good

configurations

and

discuss

important

considerations

in

choosing

the

best

configuration

Finally

concluding

remarks

and

future

work

is

presented

in

Sec.

Two-Level

Clustering

with

k-ary

n-cube

cluster-c

Organization

Many

current

parallel

systems

like

the

CRAY

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

and

the

Stanford

DASH

-LSB-

-RSB-

are

taking

a

two-level

clustering

approach

Recently

we

have

introduced

a

new

k-ary

n-cube

cluster-c

organization

-LSB-

-RSB-

to

capture

this

upcoming

trend

in

building

scalable

parallel

systems

In

this

organization

the

lower

level

consists

of

kn

clusters

of

processors

These

clusters

are

interconnected

by

a

higher

level

direct

k-ary

n-cube

network

-LRB-

also

referred

to

as

inter-cluster

network

or

internet

-RRB-

Each

cluster

consists

of

c

processors

leading

to

a

total

of

N

-LRB-

kn

c

-RRB-

processors

in

the

system

This

interconnection

achieves

two

main

design

objectives

a

-RRB-

direct

internet

provides

easy

scalability

and

b

-RRB-

processor

clusters

allow

easy

exploitation

of

communication

locality

Figure

shows

the

overall

configuration

of

such

a

system

Intercluster

Network

Intra-cluster

Network

Cluster

Interface

-LRB-

CI

-RRB-

k-ary

n-cube

c-processor

Clusters

Bus/MIN/direct

Figure

Two-level

clustering

with

k-ary

n-cube

cluster-c

organization

The

interconnection

within

a

cluster

-LRB-

also

referred

to

as

intra-cluster

network

or

intranet

-RRB-

can

be

chosen

as

bus/MIN/star

network/direct

network

as

shown

in

Fig.

Each

cluster

is

connected

to

the

rest

of

the

system

through

a

cluster

interface

The

main

task

of

the

cluster

interface

is

to

handle

the

volume

of

communication

to/from

the

cluster

Other

functionalities

may

be

added

to

the

cluster

interface

to

efficiently

implement

various

communication

synchronization

and

cachecoherence

operations

to

enhance

overall

system

performance

-LSB-

-RSB-

However

such

discussion

is

beyond

the

scope

of

this

paper

The

memory

in

the

system

is

distributed

physically

across

the

clusters

to

increase

overall

memory

bandwidth

Organization

of

memory

within

a

cluster

is

left

as

an

open

choice

depending

on

the

size

of

cluster

and

its

configuration

The

exact

nature

of

this

distribution

is

not

critical

to

the

analysis

presented

in

this

paper

Therefore

without

loss

of

generality

we

assume

the

cluster

a

-RRB-

direct-network

based

c

-RRB-

bus-based

d

-RRB-

star-based

b

-RRB-

MIN-based

MIN

Cluster

interconnections

connections

to

other

clusters

CI

CI

CI

CI

processor-memory

unit

router

CI

cluster

interface

Figure

Four

possible

cluster

interconnections

under

k-ary

n-cube

cluster-c

organization

memory

to

be

distributed

uniformly

amongst

the

processors

in

the

cluster

Architectural

Alternatives

To

build

an

N-processor

system

there

are

vast

number

of

possible

alternatives

The

degrees

of

freedom

are

number

of

processors

-LRB-

size

-RRB-

in

each

cluster

and

topologies

of

the

two

levels

-LRB-

intercluster

and

intra-cluster

-RRB-

Let

us

consider

designing

a

system

with

N

processors

This

system

can

be

designed

with

clusters

of

processors

each

clusters

of

processors

each

and

so

on

Note

that

for

a

given

system

size

fixing

the

size

of

one

level

automatically

determines

the

size

of

the

other

level

Having

fixed

the

size

of

each

level

there

is

still

freedom

to

vary

the

topology

in

each

level

For

example

in

a

system

with

clusters

with

processors

each

the

topologies

can

be

internet

with

bus-based

clusters

internet

with

MIN-based

clusters

and

so

on

Our

objective

is

to

select

the

configuration

which

at

a

minimum

cost

offers

best

system

performance

It

is

easy

to

observe

that

the

flat

k-ary

n-cube

can

be

derived

as

a

special

case

of

the

k-ary

n-cube

cluster-c

family

by

choosing

cluster

size

c

Since

inter-cluster

interconnections

are

more

expensive

than

intra-cluster

interconnections

-LSB-

-RSB-

in

this

paper

we

primarily

emphasize

on

internet

and

cluster

size

In

the

following

sections

we

develop

a

supply-demand

optimization

framework

to

derive

optimal

k-ary

n-cube

cluster-c

organizations

-LRB-

c

-RRB-

Table

provides

a

summary

of

the

main

symbols

and

notations

used

in

this

paper

Demand

on

Network

Bisection

Bandwidth

In

this

section

we

first

analyze

the

impact

of

processor

speed

on

the

rate

of

injection

of

traffic

into

the

interconnection

network

We

then

show

the

impact

of

communication

link

technology

on

the

bisection

bandwidth

of

a

system

The

demanded

bisection

size

of

the

system

to

support

a

given

processor

speed

and

communication

link

speed

technologies

is

then

presented

Table

Summary

of

Important

Symbols

and

Notations

Used

a

Unit

area

to

hold

a

processor

chip

associated

memory

and

router

logic

b

Area

of

a

board

in

units

of

a

-LRB-

number

of

processors

in

a

board

-RRB-

b

c

Number

of

clusters

in

a

board

c

Size

of

each

cluster

n

Dimensionality

of

inter-cluster

network

pp

Pinout

density

under

periphery

pinout

technology

ps

Pinout

density

under

surface

pinout

technology

t

Time

to

transfer

a

bit

across

a

wire

-LRB-

indicates

communication

link

technology

-RRB-

ub

Percentage

utilization

of

board

area

b

up

Percentage

utilization

of

available

board

pinout

Pb

Bb

Bisection

size

of

a

board

-LRB-

number

of

wires

-RRB-

D

Average

number

of

bits

injected

by

a

processor

per

second

into

the

network

-LRB-

indicates

processor

technology

-RRB-

N

Total

number

of

processors

in

a

system

N

N

c

Total

number

of

clusters

in

system

Nboards

Total

number

of

processor

boards

used

in

a

system

-LRB-

N

b

-RRB-

Pb

Total

pincount

from

a

board

-LRB-

depends

on

pinout

technology

-RRB-

W

Inter-cluster

channel

width

-LRB-

number

of

wires

-RRB-

Wb

Maximum

value

of

W

constrained

by

board

bisection

Bb

Wp

Maximum

value

of

W

constrained

by

board

pinout

Pb

dl

Demanded

bisection

size

in

the

intra-cluster

-LRB-

lower

-RRB-

level

network

du

Demanded

bisection

size

in

the

inter-cluster

-LRB-

upper

-RRB-

level

network

sl

Offered

-LRB-

supplied

-RRB-

bisection

size

in

the

intracluster

-LRB-

lower

-RRB-

level

network

su

Offered

-LRB-

supplied

-RRB-

bisection

size

in

the

intercluster

-LRB-

upper

-RRB-

level

network

Dt

-LRB-

reflects

both

processor

and

communication

link

technologies

-RRB-

Effect

of

Processor

Speed

With

advances

in

processor

technology

the

rate

at

which

processors

can

execute

instructions

keeps

going

up

As

a

general

observation

in

applications

an

amount

of

computation

is

associated

with

a

quota

of

necessary

communication

expressed

as

the

communication

to

computation

ratio

-LRB-

r

-RRB-

-LSB-

-RSB-

Intuitively

it

is

clear

that

with

increase

in

processing

speeds

computations

take

less

time

This

requires

the

associated

communications

to

also

occur

faster

leading

to

greater

demands

on

the

network

bandwidth

With

a

given

processor

technology

let

us

define

D

bits/sec

as

the

rate

of

traffic

which

each

processor

injects

into

the

network

An

increase

in

processing

power

leads

to

a

higher

value

of

D

A

typical

estimate

of

the

range

of

communication

to

computation

ratio

-LRB-

r

-RRB-

as

suggested

in

-LSB-

-RSB-

is

r

to

Flop/byte

For

r

this

implies

that

a

processor

would

inject

traffic

into

network

at

the

rate

of

or

Effect

of

Interconnect

Speed

The

bisection

size

of

a

network

is

defined

as

the

minimum

number

of

wires

that

need

to

be

cut

in

order

to

divide

the

network

into

two

equal

parts

-LSB-

-RSB-

It

limits

the

number

of

bits

that

can

cross

from

one

half

to

another

half

of

the

network

For

example

in

a

k-ary

n-cube

interconnection

with

W-bit

wide

channels

the

bisection

size

can

be

derived

as

-LSB-

-RSB-

Let

t

be

defined

as

the

cycle

time

the

time

to

transfer

a

bit

across

a

wire

in

seconds

In

other

words

a

set

of

W

parallel

wires

can

transfer

W

bits

simultaneously

in

time

t

The

bisection

size

of

a

network

corresponds

to

a

bisection

bandwidth

of

-LRB-

t

-RRB-

bits/sec

A

reduction

in

t

allows

more

data

to

be

sent

across

any

wire

in

a

given

time

and

increases

the

bisection

bandwidth

for

the

same

bisection

size

Characterizing

Demanded

Bisection

Size

Given

that

each

processor

injects

messages

into

its

cluster

network

at

D

bits/sec

let

us

derive

the

expressions

for

minimal

demanded

bisection

bandwidth

in

the

internet

and

intranet

to

sustain

the

associated

communication

requirement

Represent

the

traffic

rate

to

be

supported

across

the

bisection

in

the

internet

-LRB-

upper-level

network

-RRB-

as

Bdu

and

that

in

each

intranet

-LRB-

lower-level

network

-RRB-

as

Bdl

Given

that

the

probability

of

a

generated

request

not

going

outside

a

lower-level

network

as

p

the

effective

injection

rate

of

a

cluster

into

the

inter-cluster

network

can

be

derived

as

Dclus

cD

-LRB-

p

-RRB-

bits/sec

This

leads

to

a

combined

total

injected

traffic

of

-LRB-

DclusN

c

-RRB-

into

the

internet

Under

uniform

traffic

assumption

a

generated

message

by

a

processor

is

equally

probable

to

be

destined

for

any

other

processor

in

the

system

Thus

p

can

be

derived

as

c

N

which

approaches

for

large

systems

with

small

cluster

sizes

It

also

indicates

that

on

average

half

the

messages

are

destined

to

clusters

on

the

other

half

of

the

system

and

thus

require

to

cross

the

network

bisection

Thus

the

minimal

bisection

bandwidth

demanded

in

the

inter-cluster

network

can

be

derived

as

Bdu

DclusN

c

D

-LRB-

p

-RRB-

N

DN

-LRB-

-RRB-

All

expressions

in

this

section

have

been

simplified

by

assuming

p

A

localized

traffic

model

would

yield

-LRB-

p

c

N

-RRB-

leading

to

lesser

demanded

bisection

Bdu

However

while

designing

a

generalpurpose

machine

without

prior

knowledge

of

the

nature

of

the

applications

to

be

executed

on

it

an

uniform

traffic

model

is

considered

more

representative

-LSB-

-RSB-

The

demand

on

bisection

bandwidth

inside

a

cluster

can

be

divided

into

three

components

shown

as

flows

and

in

Fig.

The

flow

corresponds

to

the

local

traffic

across

bisection

and

is

given

by

cDp

The

flow

cD

-LRB-

p

-RRB-

is

the

fraction

that

goes

to

the

upper

level

across

the

bisection

and

the

flow

cD

-LRB-

p

-RRB-

is

a

similar

contribution

coming

from

the

upper

level

that

crosses

the

bisection

This

leads

to

a

minimal

bisection

bandwidth

demand

in

the

intra-cluster

network

as

Bdl

cD

-LRB-

p

-RRB-

cD

-LRB-

-RRB-

Lower-level

networks

each

interconnecting

c

processors

Cluster

Bdu

ND

-LRB-

-RRB-

Bdl

cD

-LRB-

-RRB-

Interface

-LRB-

interconnecting

N/c

lower

networks

-RRB-

Upper

level

Network

Figure

The

demanded

bisection

bandwidths

in

the

inter-cluster

-LRB-

upper-level

-RRB-

network

and

inside

each

cluster

-LRB-

lower-level

-RRB-

network

under

an

uniform

traffic

pattern

The

above

expressions

Bdl

and

Bdu

give

the

minimal

transfer

rate

to

be

supported

at

the

network

bisections

of

the

upper

and

lower

level

networks

respectively

The

actual

size

of

these

bisections

in

terms

of

number

of

wires

to

support

the

rate

also

depends

on

the

speed

of

the

communication

links

Let

us

define

the

minimal

bisection

size

in

the

upper-level

network

as

du

Let

tu

be

the

cycle

time

in

the

inter-cluster

network

This

indicates

du

Bdutu

Observe

that

an

increase

in

the

value

of

D

captures

the

advancement

in

processor

speed

while

a

decrease

in

the

value

of

tu

captures

that

of

link

speeds

To

capture

both

these

technological

advancements

together

we

introduce

a

parameter

u

Dtu

This

leads

to

du

Dtu

-LRB-

p

-RRB-

N

u

-LRB-

p

-RRB-

N

uN

-LRB-

-RRB-

Analogously

we

define

the

bisection

size

to

handle

the

traffic

rate

Bdl

across

a

lower-level

cluster

network

as

dl

Bdl

tl

where

tl

is

the

cycle

time

in

the

cluster

network

Representing

l

Dtl

we

can

write

dl

c

l

-LRB-

p

-RRB-

c

l

-LRB-

-RRB-

For

the

remaining

discussion

in

this

section

we

treat

u

and

l

as

same

and

refer

to

it

as

simply

It

is

to

be

noted

that

if

processor

and

link

speeds

increase

in

the

same

proportion

then

does

not

change

This

leads

to

no

change

in

the

required

du

and

dl

To

derive

representative

values

of

in

current

and

future

systems

we

consider

predictions

made

by

Patterson

-LSB-

-RSB-

As

shown

in

Table

the

future

values

of

will

broadly

lie

in

the

range

of

Table

Sample

values

of

for

various

representative

combinations

of

processor

and

network

speeds

Processor

Network

channel

ns

demand

on

n/w

D

-LRB-

Mb/s

-RRB-

cycle

time

t

-LRB-

ns

-RRB-

l

Dt

ns

ns

ns

ns

To

support

larger

sized

clusters

the

intra-cluster

bisection

bandwidth

should

scale

linearly

with

size

c

Since

packaging

constraints

are

less

rigid

in

lower

hierarchies

it

is

possible

to

provide

thicker

channel/buses

for

higher

bandwidth

-LSB-

-RSB-

Hence

in

this

paper

we

focus

on

minimum

bisection

size

in

the

upper

level

du

The

demanded

bisection

is

parameterized

with

respect

to

thus

capturing

advances

in

both

processor

and

interconnection

speeds

In

the

following

section

we

compare

this

demanded

bisection

size

with

that

being

offered

by

the

packaging

technology

Modeling

and

Parameterization

of

Packaging

Technologies

The

packaging

of

high

performance

systems

has

an

extreme

impact

on

their

performance

It

is

quite

often

the

case

that

system

level

design

is

dictated

by

available

packaging

technology

We

first

present

a

model

to

capture

hierarchical

packaging

schemes

We

then

focus

on

a

two-level

packaging

with

the

first

level

as

clusters

inside

a

board

and

the

second

as

across

the

boards

Current

trends

in

board

and

pinout

technologies

are

investigated

and

a

general

scheme

is

presented

to

parameterize

the

architectural

characteristics

with

respect

to

these

technologies

Hierarchical

Packaging

Model

Some

earlier

models

proposed

to

capture

packaging

constraints

like

the

VLSI

model

with

limited

bisection

bandwidth

as

proposed

by

Dally

-LSB-

-RSB-

and

the

limited

pincount

model

as

proposed

by

Agrawal

-LSB-

-RSB-

deal

with

only

one

level

of

packaging

Large

parallel

machines

on

the

other

hand

typically

employ

several

levels

of

packaging

There

are

inherent

technological

constraints

that

limit

VLSI

die

sizes

board

areas

number

of

boards

per

card-cage

and

number

of

card-cages

per

cabinet

etc.

Hence

multi-level

hierarchies

are

inevitable

for

packaging

large

parallel

systems

consisting

of

thousands

of

processors

Similar

to

the

characterization

of

multi-level

packaging

hierarchies

in

-LSB-

-RSB-

we

model

the

packaging

hierarchy

as

consisting

of

l

levels

l

A

packaging

module

in

level-i

is

referred

to

as

a

mi

Each

mi

contains

some

number

of

mi

modules

and

these

in

turn

contain

mi

modules

and

so

on

We

treat

processing

elements

memory

units

and

routing

nodes

as

the

lowest

level

modules

A

module

mi

is

associated

with

a

property

vector

propi

-LRB-

dimsi

Bi

Pi

ti

-RRB-

which

describes

its

various

technological

capacities

These

are

as

follows

ffl

dimsi

the

range

of

sizes

and

dimensions

that

a

packaging

module

mi

can

support

This

parameter

is

used

to

determine

other

geometric

characteristics

like

cross-section

area

volume

and

periphery

ffl

Bi

the

bisection

bandwidth

available

in

a

module

mi

The

cross-section

size

and

the

crosssection

wire-density

at

level-i

decide

the

available

Bi

which

is

the

maximum

bisection

bandwidth

available

to

the

sub-topology

of

mi

modules

that

can

be

contained

within

a

mi

module

ffl

Pi

the

pincount

available

with

a

mi

module

These

pins

are

used

to

interconnect

various

modules

at

level-i

to

comprise

a

sub-topology

to

be

contained

within

a

mi

module

This

parameter

Pi

depends

on

the

pinout

technology

physical

dimensions

of

mi

and

the

pinout

densities

supportable

at

mi

In

Sec.

we

analyze

different

pinout

technologies

and

their

impact

on

our

design

guidelines

ffl

ti

the

time

to

transfer

a

bit

over

a

level-i

wire

This

generalized

packaging

model

can

capture

and

characterize

a

wide

range

of

packaging

hierarchies

and

technologies

by

appropriately

choosing

the

parameters

With

respect

to

our

two-level

k-ary

n-cube

cluster-c

organization

shown

in

Fig.

we

consider

a

two-level

packaging

hierarchy

The

lower

level

modules

consists

of

processors

inside

a

cluster

The

dimensions

of

this

clustermodule

depends

on

the

size

of

the

cluster

and

the

level

of

processor-memory

integration

Boards

represent

the

upper-level

modules

Depending

on

the

cluster

size

and

board

area

multiple

clusters

may

be

put

onto

a

single

board

The

inter-cluster

network

may

therefore

be

partly

intra-board

and

partly

across

boards

We

do

not

allow

a

single

cluster

to

span

across

multiple

boards

In

this

research

we

study

the

interplay

of

the

cluster-modules

and

board-modules

on

overall

system

design

as

a

function

of

cluster

size

board

size

and

technological

advances

Parameterizing

Packaging

Technologies

In

this

section

we

discuss

current

trends

in

board

and

pinout

technologies

Then

we

present

a

general

scheme

in

which

architectural

characteristics

like

bisection

bandwidth

and

channel

width

are

parameterized

with

respect

to

the

board

and

pinout

technologies

Processor

board

technology

Processor

boards

can

not

be

arbitrarily

large

in

size

Their

size

is

restricted

by

electrical

mechanical

and

board

fabrication

constraints

Let

us

define

the

area

required

by

a

processor

chip

its

local

memory

and

a

per-processor

fraction

of

the

network

and

other

interface

logic

to

be

a

We

treat

this

area

as

a

square

with

a

perimeter

of

and

measure

all

board

sizes

in

terms

of

a.

Given

a

board

of

size

-LRB-

ba

-RRB-

we

can

accommodate

up

to

b

processors

on

it

This

can

represent

b

c

clusters

with

c

processors

in

each

cluster

For

sake

of

simplicity

in

further

discussions

we

drop

the

area

unit

a

and

refer

to

a

board

area

of

-LRB-

ba

-RRB-

as

simply

b.

Over

the

years

board

sizes

have

grown

in

physical

dimensions

We

now

have

systems

like

MIT

J-machine

-LSB-

-RSB-

which

put

processors

on

a

board

However

the

J-machine

uses

fine-grained

nodes

-LSB-

-RSB-

with

integrated

processor

and

router

organization

The

memory

provided

per

processor

is

also

small

compared

to

other

contemporary

commercial

systems

and

research

prototypes

The

current

trend

in

building

large

parallel

systems

is

to

use

off-the-shelf

commodity

processors

and

to

provide

a

few

megabyte

of

memory

per

processor

Hence

for

illustrating

our

framework

we

consider

board

sizes

to

hold

a

maximum

of

processors

To

build

a

system

with

N

processors

we

need

a

total

board

area

N

However

board

area

being

precious

-LSB-

-RSB-

we

suggest

using

a

total

board

area

N

All

boards

used

in

a

system

are

assumed

to

be

of

the

same

size

and

b

The

total

number

of

boards

used

in

designing

an

N-processor

system

is

defined

as

Nboards

N

b.

Another

important

consideration

in

the

design

problem

is

the

on-board

bisection

bandwidth

Bb

This

parameter

reflects

the

maximum

number

of

wires

that

can

be

laid

across

the

bisection

of

the

board

Multi-layered

wiring

is

commonly

employed

to

support

a

higher

bisection

bandwidth

density

Pinout

Technologies

The

pin-count

Pb

out

of

a

board

has

a

direct

influence

on

the

data

volume

that

can

flow

in/out

of

a

given

processor

board

Currently

two

different

types

of

technologies

are

being

employed

by

the

computer

industry

a

-RRB-

Peripheral

pinouts

This

is

the

traditional

technology

-LSB-

-RSB-

Only

the

periphery

of

the

board

is

used

for

external

connections

Thus

if

the

area

of

the

board

is

A

the

peripheral

pinout

is

limited

by

O

-LRB-

pA

-RRB-

Let

us

define

parameter

pp

to

be

the

pin-count

that

can

be

supported

from

a

perimeter

of

length

-LRB-

corresponding

to

the

perimeter

of

a

square

board

area

a

-RRB-

using

this

technology

This

implies

that

a

board

area

of

-LRB-

ja

-RRB-

would

have

a

peripheral

pinout

of

Pb

pppj

b

-RRB-

Surface

pinouts

In

this

technology

the

surface

of

the

board

is

utilized

for

external

connections

Representative

examples

are

electronic

interconnections

using

elastomeric

connectors

-LSB-

-RSB-

and

optical

interconnections

-LSB-

-RSB-

Under

this

technology

the

pin-count

from

a

board

of

area

A

is

O

-LRB-

A

-RRB-

Let

us

define

parameter

ps

to

be

the

pin-count

that

can

be

supported

on

a

given

board

area

a

using

surface

pinout

technology

This

implies

that

a

board

area

of

-LRB-

ja

-RRB-

would

have

a

surface

pin-count

of

Pb

jps

It

is

natural

to

expect

that

the

future

trend

will

be

towards

a

mixed

pinout

technology

employing

a

convenient

combination

of

surface

and

peripheral

pinouts

The

pincount

under

mixed

technology

may

be

expressed

as

a

function

f

-LRB-

pp

ps

-RRB-

of

the

periphery

and

surface

pinout

densities

The

exact

nature

of

this

function

is

difficult

to

predict

as

it

depends

on

specific

fabrication

advancements

However

reasonable

suggestions

may

be

a

max

function

or

a

weighted

mean

Without

loss

of

generality

in

this

paper

we

present

our

results

with

respect

to

the

peripheral

and

surface

technologies

The

results

for

a

mixed

pinout

technology

can

be

derived

by

simple

extension

Figure

shows

the

growths

of

pincount

with

board

area

under

the

respective

surface

and

peripheral

technologies

for

different

relationships

between

ps

and

pp

The

vertical

clipping

line

in

each

graph

reflects

that

the

board

size

can

not

grow

continuously

and

is

limited

by

some

maximum

size

Board

size

-LRB-

A

-RRB-

Pincount

O

-LRB-

A

-RRB-

surface

O

-LRB-

A

-RRB-

peripheral

Board

size

-LRB-

A

-RRB-

O

-LRB-

A

-RRB-

surface

O

-LRB-

A

-RRB-

peripheral

technology

technology

ja

technology

technology

ja

ps

pp

ps

pp

increase

in

board

size

Pincount

Figure

Growth

of

pincount

with

board

area

under

peripheral

and

surface

technologies

shown

for

two

different

relationships

between

pp

and

ps

The

parameters

pp

and

ps

reflect

the

pincount

that

can

be

supported

from

a

unit

board

area

a

under

periphery

and

surface

pinout

technologies

respectively

As

expected

for

large

board

sizes

the

surface

technology

always

supersedes

the

peripheral

technology

However

if

ps

pp

then

for

smaller

board

sizes

the

peripheral

technology

can

beat

the

surface

technology

Representative

values

of

pp

and

ps

derived

from

the

current

design

trends

of

CRAY

-LSB-

-RSB-

and

J-machine

-LSB-

-RSB-

are

and

respectively

Future

values

of

these

parameters

will

depend

on

the

advances

made

in

the

respective

technologies

To

cover

a

wide

spectrum

of

possibilities

and

show

the

impact

under

different

situations

we

also

use

projected

values

of

pp

and

ps

Channel

Width

Technology

Most

current

machines

have

data

channels

This

corresponds

to

a

channel

width

of

W

ss

including

control

acknowledgment

and

parity

wires

Many

factors

like

path-width

inside

routers

and

connector

technology

restrict

channel

widths

from

being

arbitrarily

large

In

the

near

future

it

is

expected

that

technologies

would

allow

channels

to

carry

and

data

-LSB-

-RSB-

corresponding

to

W

ss

and

respectively

Larger

channel

widths

would

make

effective

message

lengths

shorter

leading

to

lower

message

latencies

and

lesser

contention

in

routing

-LSB-

-RSB-

thus

yielding

better

system

performance

In

this

paper

we

consider

design

under

all

three

channel

widths

Under-Utilization

of

Board

Area

and

Pinout

Capacity

It

is

not

always

necessary

to

either

fill

up

a

processor

board

to

its

maximum

capacity

or

to

utilize

all

the

pins

coming

out

from

it

In

our

k-ary

n-cube

cluster-c

organization

consider

the

N

N

c

clusters

to

be

interconnected

by

a

generalized

k-ary

n-cube

topology

with

different

radix

in

each

dimension

This

leads

to

N

x

x.

x

kn

The

interconnection

is

assumed

to

be

laid

out

across

the

boards

Consider

a

sub-topology

of

the

inter-cluster

network

with

clusters

requiring

an

area

of

cfl

to

be

placed

on

a

board

Let

P

wires

be

required

to

connect

the

chosen

sub-topology

to

other

clusters

outside

the

board

This

requires

b

cfl

and

Pb

P

A

perfect

matching

of

b

cfl

and

Pb

P

where

both

board

capacity

and

pinout

are

fully

used

is

usually

difficult

to

achieve

On

choosing

an

optimal

board

size

with

capacity

b

cfl

may

imply

Pb

P

leading

to

under-utilization

of

pin-capacity

Defining

up

P

Pb

this

under-utilization

factor

can

be

expressed

as

-LRB-

up

-RRB-

Similarly

choosing

a

board

with

optimal

pincount

Pb

P

may

lead

to

-LRB-

b

cfl

-RRB-

which

reflects

board

area

under-utilization

or

under-population

of

board

estate

Defining

ub

cfl

b

the

factor

of

under-population

can

be

expressed

as

-LRB-

ub

-RRB-

It

is

to

be

noted

that

under-population

of

board

area

allows

fewer

nodes

to

share

the

pincount

from

a

board

making

wider

channels

possible

The

disadvantage

is

clearly

the

wastage

of

precious

board

estate

leading

to

larger

number

of

boards

greater

system

volume

and

higher

cost

In

the

discussion

that

follows

we

assume

board

area

to

be

fully

utilized

unless

mentioned

otherwise

Underutilization

of

pin-capacity

is

used

if

larger

board

sizes

are

required

to

fit

the

desired

topology

but

the

resulting

larger

pincount

can

not

be

utilized

with

wider

channels

as

discussed

in

sec.

It

is

also

possible

to

choose

a

design

where

under-utilization

of

both

board

and

pinout-capacity

is

used

This

is

obviously

not

required

because

we

can

always

select

a

board

size

where

one

of

the

capacities

is

optimally

utilized

but

not

the

other

Such

a

design

rule

can

be

captured

as

-LRB-

ub

-RRB-

-LRB-

up

-RRB-

Impact

of

Packaging

on

Architectural

Parameters

In

this

section

we

determine

possible

channel

widths

and

bisection

bandwidths

to

design

k-ary

n-cube

cluster-c

systems

for

a

given

board

pinout

and

channel

width

technologies

From

the

expressions

we

derive

in

this

section

for

arbitrary

cluster

sizes

the

results

for

flat

architectures

can

be

obtained

by

choosing

c

Offered

Inter-Cluster

Channel

Width

Let

the

sub-topology

of

the

inter-cluster

network

N

N

c

x

x

kn

being

placed

on

a

board

be

-LRB-

x

x

x

bn

-RRB-

where

bi

ki

and

some

bi

's

may

be

to

depict

that

only

one

processor

exists

on

the

board

along

that

dimension

This

implies

that

no

inter-cluster

dimension

can

be

fully

contained

within

a

board

This

is

reasonable

to

expect

for

large

system

sizes

Figure

shows

an

example

inter-cluster

interconnection

-LRB-

N

x

x

-RRB-

being

placed

on

boards

each

with

a

capacity

for

holding

four

processor

clusters

The

sub-topology

placed

on

each

board

is

x

x

Let

the

number

of

rows

in

dimension

i

of

the

sub-topology

be

defined

as

ri

This

can

processor

cluster

interconnection

N

inter-cluster

b

sub-topology

on

a

board

Figure

An

example

inter-cluster

interconnection

being

placed

on

boards

each

with

a

capacity

of

holding

processor

clusters

The

sub-topology

placed

on

each

board

is

be

derived

as

ri

bi

Note

that

inter-board

channels

go

out/come

in

from

the

two

ends

of

each

row

leading

to

inter-board

channels

along

dimension

i.

Hence

the

total

number

of

interboard

channels

from

all

n

dimensions

is

derived

as

-LRB-

bi

-RRB-

To

support

a

channel

width

of

W

a

total

pincount

of

-LRB-

bi

-RRB-

is

required

from

each

board

Thus

the

maximum

supportable

inter-board

channel

width

from

pinout

restrictions

denoted

by

Wp

is

derived

as

Wp

Pb

-LRB-

bi

-RRB-

-LRB-

-RRB-

Let

us

now

consider

the

required

on-board

bisection

size

to

support

W

width

channels

in

the

-LRB-

x

x.

x

bn

-RRB-

sub-topology

on

a

board

Clearly

the

available

board

bisection

should

be

able

to

support

the

largest

bisection

in

the

sub-topology

The

largest

bisection

in

the

sub-topology

is

the

one

orthogonal

to

the

smallest

dimension

However

a

smallest

dimension

j

with

bj

implying

only

one

processor

along

that

dimension

does

not

require

any

on-board

bisection

Thus

the

largest

bisection

occurs

orthogonal

to

dimension

j

such

that

bj

is

the

smallest

dimension

not

equal

to

one

There

are

bj

nodes

nearer

to

this

bisection

and

assuming

bidirectional

channels

we

have

bj

channels

crossing

this

bisection

With

channel

width

of

W

it

requires

bj

wires

across

this

bisection

With

a

board

bisection

Bb

the

maximum

supportable

intra-board

channel

width

from

board

bisection

restrictions

Wb

is

given

by

Wb

Bb

bj

Bbbj

-LRB-

-RRB-

Assuming

that

the

sub-topology

on

the

board

is

regular

i.e.

-LRB-

bi

n

-RRB-

this

can

be

simplified

to

Wb

Bb

-LRB-

n

-RRB-

n

-LRB-

-RRB-

Similarly

Eqn

can

be

simplified

to

Wp

Pb

-LRB-

n

-RRB-

n

Pb

-LRB-

b

c

-RRB-

-LRB-

n

-RRB-

n

-LRB-

-RRB-

Thus

the

channel

width

supported

in

a

system

is

restricted

by

two

constraints

Wp

from

pinout

and

Wb

from

board

bisection

constraints

respectively

This

can

potentially

lead

to

different

inter-board

and

intra-board

channel

widths

For

designing

an

architecture

with

uniform

intercluster

channel

width

we

choose

W

min

-LRB-

Wp

Wb

-RRB-

With

multi-layered

interconnection

it

is

reasonable

to

assume

on-board

connection

density

to

be

higher

than

off-board

density

This

leads

to

channel

width

being

limited

by

the

pinout

constraint

resulting

in

W

Wp

Unless

otherwise

specified

we

therefore

use

W

and

Wp

interchangeably

In

the

following

discussion

we

assume

full

utilization

of

board

area

and

pinout

resources

Design

with

under-utilization

is

analyzed

later

From

the

expression

for

Wp

in

Eqn

it

can

be

observed

that

the

channel

width

is

determined

by

board

size

cluster

size

pinout

technology

from

board

and

dimensionality

of

the

inter-cluster

network

It

does

not

depend

on

the

total

system

size

For

a

given

pinout

technology

and

board

size

let

us

consider

the

impact

of

varying

the

inter-cluster

dimensionality

while

keeping

cluster

size

fixed

In

order

to

support

larger

dimensional

networks

we

need

more

channels

This

leads

to

thinner

channels

with

a

fixed

pinout

from

a

board

Similarly

for

a

given

pinout

technology

and

board

size

decreasing

the

cluster

size

while

keeping

inter-cluster

dimensionality

fixed

also

leads

to

thinner

channels

A

smaller

cluster

size

results

in

larger

number

of

clusters

on

board

This

implies

that

pinout

from

the

board

gets

shared

among

more

clusters

leading

to

thinner

channels

We

summarize

these

as

Observation

For

a

given

pinout

technology

-LRB-

Pb

-RRB-

and

board

size

-LRB-

b

-RRB-

the

offered

channel

width

falls

with

ffl

Increase

in

the

inter-cluster

network

dimensionality

-LRB-

n

-RRB-

while

keeping

cluster

size

-LRB-

c

-RRB-

fixed

or

ffl

Fall

in

the

cluster

size

-LRB-

c

-RRB-

while

keeping

the

inter-cluster

dimensionality

-LRB-

n

-RRB-

fixed

Figure

shows

the

trend

as

cluster

size

is

increased

for

different

inter-cluster

dimensionalities

In

the

figure

each

board

is

assumed

to

be

large

enough

to

hold

exactly

one

cluster

-LRB-

-RRB-

and

channel

widths

up

to

lines

have

only

been

shown

For

a

given

cluster

size

offered

channel

width

drops

with

increasing

inter-cluster

dimensionality

It

can

be

observed

that

the

drop

in

channel

width

is

more

significant

at

lower

dimensions

-LRB-

from

to

-RRB-

than

at

higher

dimensions

-LRB-

from

to

-RRB-

Therefore

for

reasonably

higher

dimensions

the

channel

widths

are

expected

to

be

close

Similar

results

are

also

obtained

for

For

a

given

message

size

a

wider

channel

reduces

the

effective

message

length

in

the

inter-cluster

network

This

has

a

potential

to

reduce

message

contention

leading

to

smaller

message

latency

Thus

it

is

desirable

to

choose

the

largest

channel

width

supportable

by

technology

at

a

given

time

as

discussed

in

sec.

Channel

width

-LRB-

W

-RRB-

cluster

size

-LRB-

c

-RRB-

a

-RRB-

Surface

Pinout

Tech

and

b

Channel

width

-LRB-

W

-RRB-

cluster

size

-LRB-

c

-RRB-

b

-RRB-

Periphery

Pinout

Tech

and

b

Figure

Offered

channel

width

as

a

function

of

cluster

size

and

inter-cluster

dimensionalities

Board

size

is

not

fixed

and

chosen

to

hold

one

cluster

with

c

processors

-LRB-

-RRB-

Two

different

pinout

technologies

are

considered

a

-RRB-

surface

pinout

technology

with

ps

and

b

-RRB-

periphery

pinout

technology

with

pp

Let

us

consider

the

impact

of

board

size

on

channel

width

W

Under

periphery

pinout

technology

Pb

pp

pb

Using

b

c

we

can

simplify

Eqn

as

W

ppc

-LRB-

n

-RRB-

n

-LRB-

n

-RRB-

-LRB-

-RRB-

n

Thus

for

a

given

cluster

size

and

inter-cluster

dimensionality

n

an

increase

in

board

size

b

leads

to

a

fall

in

channel

width

The

channel

width

in

a

inter-cluster

network

remains

constant

while

that

in

an

network

rises

sharply

This

is

shown

in

Fig.

-LRB-

b

-RRB-

for

pp

and

a

cluster

size

of

Under

surface

pinout

technology

with

Pb

psb

a

similar

simplification

leads

to

W

psc

-LRB-

n

-RRB-

n

Thus

for

any

given

cluster

size

and

inter-cluster

dimensionality

an

increase

in

board

size

b

leads

to

a

rise

in

channel

width

However

this

rise

is

not

very

significant

for

higher

dimensions

Such

trends

are

shown

in

Fig.

-LRB-

a

-RRB-

for

ps

and

a

cluster

size

of

This

leads

to

Observation

Under

periphery

pinout

technology

keeping

cluster

size

c

fixed

and

dimensionality

of

inter-cluster

network

n

an

increase

in

board

size

results

in

channel

width

to

fall

However

under

surface

pinout

technology

for

all

inter-cluster

dimensionalities

it

leads

to

a

rise

in

channel

width

Channel

width

-LRB-

W

-RRB-

Board

size

-LRB-

b

-RRB-

a

-RRB-

Surface

Pinout

Tech

Fixed

cluster

size

Channel

width

-LRB-

W

-RRB-

Board

size

-LRB-

b

-RRB-

b

-RRB-

Periphery

Pinout

Tech

Fixed

cluster

size

Figure

Channel

width

as

a

function

of

board

size

for

different

inter-cluster

dimensionalities

under

a

-RRB-

surface

pinout

technology

with

ps

and

b

-RRB-

periphery

pinout

technology

with

pp

Cluster

size

is

kept

fixed

at

c

Supporting

a

Fixed

Inter-Cluster

Channel

Width

Most

of

the

prior

studies

on

system

design

while

proposing

guidelines

under

different

constraints

like

constant

bisection

bandwidth

-LSB-

-RSB-

did

not

impose

any

restrictions

on

the

values

that

the

channel

width

can

take

while

satisfying

other

constraints

Supporting

an

arbitrary

channel

width

is

not

easy

because

it

has

impact

on

router

design

Agrawal

realized

the

importance

of

this

constraint

and

analyzed

system

designs

under

fixed

channel

width

-LSB-

-RSB-

For

a

simpler

interface

design

the

width

of

data

lines

in

a

channel

is

expected

to

maintain

an

integral

relationship

with

that

of

processor

and

memory

which

are

typically

in

multiples

of

a

byte

In

Eqn

we

showed

that

W

-LRB-

Wp

-RRB-

gets

determined

once

other

system

parameters

like

n

b

and

c

get

specified

Given

a

channel

width

technology

with

W

-LRB-

W

-RRB-

an

obvious

design

objective

while

selecting

values

for

n

b

and

c

is

to

ensure

that

the

offered

channel

width

-LRB-

W

-RRB-

is

equal

to

the

supportable

channel

width

Together

with

Eqn

this

leads

us

to

the

following

set

of

observations

on

the

cluster

size

that

can

be

placed

on

a

board

while

other

parameters

like

inter-cluster

dimensionality

pinout

density

and

channel

width

are

varied

Observation

ffl

For

a

fixed

channel

width

-LRB-

W

-RRB-

board

size

-LRB-

b

-RRB-

and

a

pinout

technology

an

increase

in

intercluster

network

dimensionality

-LRB-

n

-RRB-

leads

to

an

increase

in

the

cluster

size

-LRB-

c

-RRB-

resulting

in

fewer

clusters

on

the

same

board

area

ffl

For

a

fixed

channel

width

-LRB-

W

-RRB-

board

size

-LRB-

b

-RRB-

and

inter-cluster

dimensionality

-LRB-

n

-RRB-

an

increase

in

pinout

density

-LRB-

pp

or

ps

and

hence

Pb

-RRB-

leads

to

a

fall

in

the

cluster

size

-LRB-

c

-RRB-

resulting

in

more

clusters

on

the

same

board

area

ffl

Increasing

the

channel

width

-LRB-

W

-RRB-

leads

to

larger

cluster

size

-LRB-

c

-RRB-

while

other

parameters

are

kept

constant

This

leads

to

fewer

clusters

on

a

given

board

area

While

maintaining

a

fixed

channel

width

-LRB-

W

-RRB-

it

may

also

be

desired

to

maintain

a

given

fixed

number

of

clusters

-LRB-

b

c

-RRB-

on

each

board

Under

such

a

condition

supporting

larger

clusters

also

implies

using

larger

boards

to

maintain

a

fixed

Let

us

consider

the

effect

of

increasing

board

size

-LRB-

or

equivalently

cluster

size

-RRB-

on

the

inter-cluster

dimensionality

in

Eqn

while

maintaining

a

fixed

W

and

Larger

board

area

allows

more

pinout

per

board

thus

leading

to

an

increase

in

Pb

This

implies

that

n

should

also

increase

to

maintain

the

same

channel

width

This

can

be

stated

as

the

following

observation

Observation

For

fixed

number

of

clusters

on

a

board

-LRB-

-RRB-

and

supportable

channel

width

-LRB-

W

-RRB-

an

increase

in

board

-LRB-

cluster

-RRB-

size

leads

to

an

increase

in

the

dimensionality

of

the

inter-cluster

network

Let

us

consider

the

impact

of

pinout

technology

on

supporting

a

fixed

channel

width

Under

periphery

pinout

technology

as

discussed

earlier

we

need

to

satisfy

W

ppc

-LRB-

n

-RRB-

n

-LRB-

n

-RRB-

-LRB-

-RRB-

n

Thus

for

a

given

inter-cluster

dimensionality

n

and

pinout

density

-LRB-

pp

-RRB-

supporting

larger

cluster

size

requires

an

increase

in

the

size

of

the

boards

used

For

n

cluster

size

can

not

be

varied

because

changing

board

sizes

has

no

impact

Similarly

for

n

board

size

can

not

be

varied

because

changing

cluster

size

has

no

impact

Under

surface

pinout

technology

we

need

to

satisfy

W

psc

-LRB-

n

-RRB-

n

Thus

for

a

given

n

and

pinout

density

-LRB-

ps

-RRB-

supporting

larger

cluster

sizes

requires

a

fall

in

the

size

of

the

boards

used

For

n

board

size

can

not

be

varied

because

changing

cluster

size

has

no

impact

The

above

interplay

can

be

summarized

as

the

following

observation

Observation

In

supporting

a

fixed

channel

width

-LRB-

W

-RRB-

in

an

inter-cluster

network

with

a

given

dimensionality

-LRB-

n

-RRB-

for

periphery

pinout

technology

with

n

supporting

a

larger

cluster

size

requires

an

increase

in

board

size

In

comparison

under

surface

pinout

technology

with

n

supporting

a

larger

cluster

size

requires

a

fall

in

board

size

Realizing

Designs

with

Under

Utilization

The

expression

in

Eqn

yields

a

channel

width

W

assuming

full

utilization

of

board

area

and

pinout

resources

However

while

building

a

real

machine

it

is

unlikely

that

both

board

area

and

pinout

resources

get

fully

utilized

To

make

the

study

more

realistic

and

interesting

we

extend

the

expression

in

Eqn

to

allow

for

under-utilization

of

board

and

pinout

capacities

as

discussed

in

sec

Now

Eqn

can

be

rewritten

as

W

Pbup

-LRB-

b

c

-RRB-

-LRB-

n

-RRB-

n

W

up

ub

-LRB-

-RRB-

where

Pbup

is

the

utilized

pincount

from

board

is

the

actual

number

of

clusters

placed

on

a

board

-LRB-

out

of

the

maximum

b

c

-RRB-

and

W

is

the

channel

width

value

obtained

assuming

full-utilization

of

resources

We

assume

a

reasonable

bound

on

both

under-utilizations

such

that

up

ub

umin

for

some

umin

closer

to

Thus

observations

made

earlier

assuming

full-utilization

-LRB-

umin

-RRB-

continue

to

hold

For

illustrative

purposes

in

this

paper

we

choose

umin

Given

a

pinout

density

and

a

channel

width

technology

we

formulate

the

following

search

problem

to

determine

-LRB-

n

b

c

-RRB-

tuples

which

satisfy

the

following

inequalities

Allowing

only

under-utilization

of

pinout

we

have

up

and

ub

This

leads

to

W

W

-LRB-

ub

up

-RRB-

-RRB-

W

W

-LRB-

-RRB-

Allowing

only

under-utilization

of

board

area

we

have

ub

and

up

This

leads

to

W

W

-LRB-

ub

up

-RRB-

-RRB-

W

W

-LRB-

-RRB-

Figure

shows

the

plots

in

Fig.

with

such

under-utilization

tolerance

lines

which

provide

flexibility

of

around

the

channel

widths

of

and

The

configurations

which

fall

within

these

tolerance

boundaries

are

valid

or

design-feasible

configurations

We

summarize

the

solutions

to

this

search

problem

in

Table

for

surface

pinout

technology

of

ps

and

in

Table

for

periphery

pinout

technology

of

pp

In

both

tables

three

different

supportable

channel

width

technologies

corresponding

to

W

and

and

different

allowable

number

of

clusters

per

board

are

considered

In

the

examples

shown

in

Fig.

and

Tables

and

we

ensure

that

the

maximum

board

size

-LRB-

b

-RRB-

does

not

exceed

which

was

assumed

in

sec.

as

the

largest

board

size

Blank

entries

in

the

tables

indicate

that

there

is

no

valid

configuration

for

that

intercluster

dimensionality

pinout

technology

channel

width

and

board

area

Note

that

the

majority

of

the

valid

configurations

fall

under

or

Solutions

become

fewer

for

larger

values

of

In

fact

for

the

technological

parameters

chosen

no

configurations

are

possible

for

-LRB-

the

columns

corresponding

to

have

therefore

not

been

considered

in

these

tables

-RRB-

This

suggests

that

Channel

width

-LRB-

W

-RRB-

cluster

size

-LRB-

c

-RRB-

a

-RRB-

Surface

Pinout

Tech

and

b

Channel

width

-LRB-

W

-RRB-

cluster

size

-LRB-

c

-RRB-

b

-RRB-

Periphery

Pinout

Tech

and

b

Figure

Valid

configurations

offering

channel

width

-LRB-

W

-RRB-

closer

to

or

with

a

reasonable

-LRB-

-RRB-

under-utilization

of

board

or

pinout

resources

Two

different

pinout

technologies

are

considered

a

-RRB-

surface

with

ps

and

b

-RRB-

periphery

with

pp

most

current

and

near

future

systems

will

have

or

clusters

on

each

board

With

increase

in

pinout

densities

under

both

surface

and

periphery

technologies

it

may

also

be

observed

that

the

number

of

valid

configurations

with

increase

This

suggests

that

future

systems

with

higher

pinout

densities

will

be

able

to

better

support

to

clusters

on

each

board

From

Table

note

that

for

any

given

value

of

ps

-LRB-

-RRB-

W

-LRB-

-RRB-

and

-LRB-

-RRB-

as

cluster

size

c

is

increased

from

to

the

required

dimensionality

of

the

inter-cluster

network

also

rises

from

to

This

confirms

Obs

Next

consider

the

board

sizes

required

for

configurations

under

surface

pinout

technology

corresponding

to

ps

W

n

with

two

different

cluster

sizes

c

and

The

configuration

with

the

smaller

cluster

size

-LRB-

c

-RRB-

and

requires

boards

of

size

b

while

the

one

with

larger

cluster

size

-LRB-

c

-RRB-

and

requires

a

smaller

b

Under

periphery

pinout

technology

a

different

trend

is

noted

In

Table

the

board

sizes

required

for

configurations

corresponding

to

ps

W

n

with

two

different

cluster

sizes

c

and

are

b

and

respectively

Therefor

as

cluster

size

increases

so

does

board

size

This

confirms

the

observations

made

in

Obs

Offered

Inter-Cluster

Bisection

Now

let

us

consider

the

impact

of

packaging

on

offered

inter-cluster

bisection

Given

the

intercluster

network

as

a

-LRB-

x

x.

x

kn

-RRB-

mesh/torus

and

a

fixed

channel

width

the

size

of

the

inter-cluster

bisection

can

be

computed

in

the

following

manner

A

-LRB-

x

x.

x

kn

-RRB-

mesh/torus

can

have

various

bisections

which

divide

this

network

into

two

halves

For

example

in

a

torus

with

x

y

and

z

dimensions

we

can

have

three

possible

bisections

one

orthogonal

to

x

dimension

along

yz

plane

and

so

on

We

are

interested

in

the

size

of

the

smallest

bisection

in

the

system

Table

Valid

Ranges

of

Cluster

Size

-LRB-

c

-RRB-

from

Packaging

Constraints

Surface

pinout

technology

with

ps

and

is

assumed

Supported

channel

widths

are

W

and

Inter-cluster

dimensionality

-LRB-

n

-RRB-

varies

from

Number

of

clusters/board

-LRB-

-RRB-

is

varied

from

Maximum

board

size

-LRB-

b

-RRB-

is

assumed

to

be

ps

ps

n

c

n

c

W

W

W

because

it

maximally

constrains

the

performance

of

the

system

under

random

traffic

Clearly

the

smallest

bisection

has

to

be

orthogonal

to

the

dimension

having

the

largest

radix

given

by

kmax

maxi

n

-LRB-

ki

-RRB-

The

number

of

nodes

in

such

a

bisection

is

given

by

N

kmax

Assume

each

of

these

nodes

is

connected

to

another

across

this

bisection

using

bidirectional

channels

of

width

W

This

leads

to

the

offered

bisection

size

su

as

W

kmax

wires

The

presence

of

wrap-around

channels

in

torus

doubles

this

number

This

leads

to

su

W

kmax

W

-LRB-

N

c

-RRB-

n

-LRB-

N

c

-RRB-

-LRB-

n

-RRB-

nW

-LRB-

N

c

-RRB-

-LRB-

n

-RRB-

n

Pb

-LRB-

n

-RRB-

n

Nboards

-LRB-

n

-RRB-

nPb

n

-LRB-

-RRB-

which

is

simplified

by

assuming

the

inter-cluster

network

is

regular

i.e.

-LRB-

kmax

-LRB-

N

c

-RRB-

n

-RRB-

and

using

Eqn

to

replace

W

and

Nboards

N

b.

Table

Valid

Ranges

of

Cluster

Size

-LRB-

c

-RRB-

from

Packaging

Constraints

Periphery

pinout

technology

with

pp

and

is

assumed

Supported

channel

widths

are

W

and

Inter-cluster

dimensionality

-LRB-

n

-RRB-

varies

from

Number

of

clusters/board

-LRB-

-RRB-

is

varied

from

Maximum

board

size

-LRB-

b

-RRB-

is

assumed

to

be

pp

pp

n

c

n

c

W

W

W

Impact

of

board

size

Figure

shows

the

plots

of

offered

bisection

bandwidth

B

in

the

inter-cluster

network

with

respect

to

different

pinout

technologies

as

board

size

is

increased

Under

periphery

pinout

technology

with

Pb

pp

pb

and

Nboards

N

b

Eqn

reduces

to

su

-LRB-

N

b

-RRB-

-LRB-

n

-RRB-

npppb

n

N

-LRB-

n

-RRB-

npp

b

-LRB-

n

-RRB-

-LRB-

-RRB-

n

Thus

for

a

given

system

size

-LRB-

N

-RRB-

and

inter-cluster

network

dimensionality

-LRB-

n

-RRB-

the

size

of

the

bisection

falls

with

increase

in

board

size

for

n

This

can

be

explained

by

the

fact

that

the

total

number

of

board

pincount

reduces

as

board

sizes

are

increased

For

n

the

value

of

su

remains

fixed

and

for

n

the

value

rises

slowly

The

key

point

is

that

under

this

pinout

technology

it

is

ideal

to

work

with

smaller

boards

as

they

offer

higher

bisection

sizes

and

hence

performance

Under

surface

pinout

technology

with

Pb

psb

Eqn

reduces

to

B

-LRB-

N

b

-RRB-

-LRB-

n

-RRB-

npsb

n

N

-LRB-

n

-RRB-

n

n

Thus

we

see

here

that

for

a

given

system

size

-LRB-

N

-RRB-

and

inter-cluster

network

dimensionality

-LRB-

n

-RRB-

the

size

of

the

bisection

increases

for

n

These

observations

can

be

summarized

as

Bisection

size

of

inter-cluster

network

Board

size

-LRB-

b

-RRB-

a

-RRB-

Periphery

Pinout

Tech

Bisection

size

of

inter-cluster

network

Board

size

-LRB-

b

-RRB-

b

-RRB-

Surface

Pinout

Tech

Figure

Offered

inter-cluster

bisection

su

in

a

system

of

processors

as

a

function

of

board

size

and

different

inter-cluster

dimensionalities

under

two

different

pinout

technologies

a

-RRB-

periphery

pinout

with

pp

and

b

-RRB-

surface

pinout

with

ps

Observation

Under

periphery

pinout

technology

for

a

given

system

size

-LRB-

N

-RRB-

keeping

the

dimensionality

of

inter-cluster

network

-LRB-

n

-RRB-

fixed

at

a

value

greater

than

an

increase

in

board

size

-LRB-

b

-RRB-

leads

to

a

fall

in

the

inter-cluster

bisection

size

However

under

surface

pinout

technology

as

board

size

-LRB-

b

-RRB-

is

increased

bisection

size

increases

for

all

inter-cluster

dimensionalities

Next

let

us

define

Vw

as

the

volume

-LRB-

number

-RRB-

of

inter-board

wires

required

to

interconnect

a

desired

inter-cluster

configuration

This

parameter

Vw

can

be

derived

as

NboardsPb

where

Nboards

N

b

Pb

is

the

pinout

from

each

board

and

each

wire

connects

two

pins

Under

periphery

pinout

technology

Vw

can

be

simplified

as

Npp

while

under

surface

pinout

technology

it

simplifies

to

From

these

two

expressions

we

conclude

Observation

For

a

given

system

size

the

volume

of

inter-board

wires

falls

with

an

increase

in

board

size

under

periphery

pinout

technology

while

under

surface

pinout

technology

it

remains

fixed

Indirect

impact

of

cluster

size

From

Eqn

note

that

su

-LRB-

N

c

-RRB-

-LRB-

n

-RRB-

nW

Thus

keeping

channel

width

W

fixed

at

W

an

increase

in

the

cluster

size

c

-LRB-

maintaining

W

W

may

require

board

size

to

be

varied

also

-RRB-

causes

the

offered

bisection

to

fall

Fig.

shows

this

for

a

system

size

N

processors

and

a

given

channel

width

W

and

different

inter-cluster

dimensionality

-LRB-

n

-RRB-

Offered

bisection

lines

cluster

size

-LRB-

c

-RRB-

W

Figure

Offered

inter-cluster

bisection

su

as

a

function

of

cluster

size

and

inter-cluster

dimensionalities

while

maintaining

a

channel

width

W

for

a

processor

system

Note

that

for

a

given

pinout

technology

the

board

sizes

are

not

fixed

and

changed

with

cluster

size

to

maintain

W

Deriving

Optimal

System

Configurations

Demand

vs

Supply

An

Optimization

Problem

For

a

given

system

size

our

goal

is

to

derive

the

optimal

configuration

This

configuration

should

satisfy

packaging

and

bisection

demand

constraints

and

offer

best

performance

We

formulate

this

as

an

optimization

problem

of

selecting

the

best

configuration

from

amongst

the

various

configurations

The

problem

is

solved

in

three

phases

In

the

first

phase

we

derive

the

set

of

design-feasible

configurations

which

adhere

to

the

packaging

constraints

as

discussed

earlier

ffl

Maximum

board

size

constraint

-LRB-

b

-RRB-

ffl

Pinout

constraint

from

board

-LRB-

Pb

psb

under

surface

pinout

technology

or

Pb

pp

pb

under

periphery

pinout

technology

-RRB-

ffl

Channel

width

constraint

-LRB-

W

min

-LRB-

Wp

Wb

-RRB-

-RRB-

-LRB-

W

-RRB-

System

configurations

which

are

possible

to

be

synthesized

under

the

above

constraints

are

sorted

out

in

this

process

This

was

studied

in

sec.

and

valid

or

design-feasible

configurations

derived

under

various

technological

parameters

in

Table

and

For

each

design-feasible

configuration

we

then

compute

the

offered

bisection

in

the

inter-cluster

-LRB-

su

-RRB-

and

intra-cluster

networks

-LRB-

sl

-RRB-

The

set

of

design

feasible

configurations

and

their

offered

characteristics

are

referred

by

us

as

the

supply

side

The

demand

side

refers

to

the

demanded

bisection

sizes

du

and

dl

in

the

inter-cluster

and

intra-cluster

networks

of

such

design-feasible

configurations

derived

earlier

in

sec.

on

the

basis

of

processor

and

interconnect

speeds

The

second

phase

of

the

optimization

problem

sorts

out

the

design-feasible

configurations

which

meet

the

demanded

bisections

i.e.

ffl

Inter-cluster

bisection

constraint

su

du

ffl

Intra-cluster

bisection

constraint

sl

dl

The

Inter-cluster

Bisection

Constraint

states

that

the

offered

inter-cluster

bisection

from

packaging

constraints

should

be

greater

than

the

demanded

inter-cluster

bisection

from

processor

and

interconnect

speed

considerations

Using

Eqn

and

Eqn

in

the

above

inequality

we

obtain

the

following

-LRB-

N

c

-RRB-

-LRB-

n

-RRB-

nW

N

-LRB-

-RRB-

bisection

su

with

respect

to

cluster

size

c

We

augment

that

plot

by

a

horizontal

line

as

shown

in

Fig.

The

horizontal

line

corresponds

to

the

demanded

bisection

size

du

of

N

as

derived

in

Sec.

A

representative

value

of

is

assumed

here

yielding

du

for

a

system

with

N

processors

It

is

very

clear

that

only

those

configurations

above

the

du

line

meet

the

bisection

demand

In

Table

we

summarize

the

range

of

cluster

sizes

which

meet

the

demanded

bisection

bandwidth

under

for

each

combination

of

system

size

-LRB-

N

and

-RRB-

channel

width

technology

-LRB-

W

and

-RRB-

and

inter-cluster

dimensionality

-LRB-

n

to

-RRB-

Offered

bisection

lines

cluster

size

-LRB-

c

-RRB-

W

Figure

Offered

inter-cluster

bisection

su

as

a

function

of

cluster

size

and

inter-cluster

dimensionalities

while

maintaining

a

channel

width

W

The

demanded

bisection

bandwidth

corresponding

to

is

shown

as

a

horizontal

line

The

intra-cluster

bisection

constraint

states

that

the

offered

intra-cluster

bisection

from

packaging

constraints

should

be

greater

than

the

demanded

intra-cluster

bisection

from

processor

and

interconnect

speed

considerations

We

have

not

considered

any

limitations

or

restrictions

on

the

size

and

scalability

of

the

intra-cluster

network

in

this

work

and

assume

the

intra-cluster

bisection

demand

to

be

satisfied

However

in

bus-based

clusters

the

size

of

clusters

do

get

limited

by

the

bus-bandwidth

Contemporary

bus-based

clusters

support

up

to

processors

-LSB-

-RSB-

Future

bus

technology

is

expected

to

support

larger

sized

clusters

In

current

technology

large

clusters

with

higher

intra-cluster

bisection

can

be

supported

by

using

MIN-based

or

direct

network-based

clusters

The

second

phase

of

the

optimization

leads

us

to

the

set

of

good

configurations

The

final

optimization

step

is

to

select

the

best

configuration

from

the

good

configurations

Next

we

take

up

an

example

to

show

how

to

derive

the

set

of

good

configurations

and

the

best

configuration

under

a

given

set

of

current

technological

parameters

Table

Range

of

Cluster

Sizes

Satisfying

Bisection

Demands

for

Different

System

Sizes

-LRB-

N

and

-RRB-

Channel

Widths

-LRB-

W

and

-RRB-

and

Inter-cluster

Dimensionalities

-LRB-

n

-RRB-

The

Maximum

Cluster

Size

is

Restricted

to

Cluster

Sizes

N

N

N

n

n

n

W

n

n

n

n

n

n

n

n

W

n

n

n

n

n

n

n

n

W

n

n

n

n

n

Design

Choices

under

Various

Technologies

Deciding

on

Good

Configurations

Tables

and

provided

a

summary

of

the

design-feasible

configurations

which

satisfy

the

packaging

constraints

-LRB-

including

the

channel

width

constraint

-RRB-

under

surface

and

periphery

pinout

technologies

respectively

Table

provided

a

range

of

configurations

which

satisfy

the

inter-cluster

bisection

constraint

The

intersection

of

these

two

tables

identify

good

configurations

Tables

and

show

the

offered

good

configurations

for

a

system

size

of

N

processors

under

surface

and

periphery

pinout

technologies

respectively

Similar

tables

can

be

generated

for

other

system

sizes

of

N

and

processors

Table

Good

Ranges

of

Cluster

Size

-LRB-

c

-RRB-

from

Packaging

and

Bisection

Demand

Constraints

Surface

pinout

technology

with

ps

and

is

assumed

Supported

channel

widths

are

W

and

Inter-cluster

dimensionality

-LRB-

n

-RRB-

varies

from

Number

of

clusters/board

-LRB-

-RRB-

is

varied

from

Maximum

board

size

-LRB-

b

-RRB-

is

assumed

to

be

ps

ps

n

c

n

c

W

W

W

Note

that

most

design-feasible

configurations

in

Tables

and

meet

the

bisection

demands

corresponding

to

a

system

size

of

N

processors

and

and

qualify

as

good

configurations

in

Tables

and

However

for

larger

system

sizes

with

greater

processor

demand

on

network

it

can

be

shown

that

fewer

design-feasible

configurations

would

meet

the

higher

bisection

demands

and

qualify

as

good

configurations

Table

Good

Ranges

of

Cluster

Size

-LRB-

c

-RRB-

from

Packaging

and

Bisection

Demand

Constraints

Periphery

pinout

technology

with

pp

and

is

assumed

Supported

channel

widths

are

W

and

Inter-cluster

dimensionality

-LRB-

n

-RRB-

varies

from

Number

of

clusters/board

-LRB-

-RRB-

is

varied

from

Maximum

board

size

-LRB-

b

-RRB-

is

assumed

to

be

pp

pp

n

c

n

c

W

W

W

Deriving

the

Best

Configuration

For

any

given

set

of

technological

parameters

the

set

of

good

configurations

narrow

down

the

design-choice

space

Using

any

of

these

good

configurations

it

is

possible

to

actually

fabricate

a

machine

under

packaging

constraints

and

such

a

machine

is

guaranteed

to

offer

more

than

or

equal

the

desired

bisection

bandwidth

However

there

are

two

other

important

considerations

while

selecting

a

configuration

It

should

offer

low

average

message

latency

and

it

should

have

low

cost

Thus

among

the

good

configurations

we

are

interested

in

the

best

configuration

-LSB-

-RSB-

which

provides

best

performance

at

lowest

cost

From

Tables

and

it

is

to

be

noted

that

under

most

combinations

of

pinout

density

and

channel

width

technology

we

are

left

with

around

good

choices

from

which

we

have

to

select

the

best

one

This

search

however

is

not

a

very

clear-cut

process

Unknown

and

inestimable

factors

like

exact

link

cost

connector

cost

cluster

technology

delay

due

to

longer

wires

and

other

packaging

costs

make

this

a

hard

decision

The

scalability

of

a

configuration

to

larger

sizes

is

also

very

important

In

this

section

we

use

a

combined

qualitative

quantitative

and

simulation

approach

to

derive

the

best

configurations

We

first

demonstrate

the

design

choices

under

the

current

technologies

of

channel

width

-LRB-

W

-RRB-

surface

pinout

density

-LRB-

ps

-RRB-

and

periphery

pinout

density

-LRB-

pp

-RRB-

Each

entry

in

Table

corresponding

to

a

good

configuration

can

be

expanded

to

show

detailed

characteristics

about

the

connector

and

wiring

costs

together

with

its

potential

for

future

scalability

Table

shows

such

detailed

characteristics

For

example

the

entry

corresponding

to

n

ps

W

and

cluster

size

of

in

Table

gets

expanded

to

the

first

entry

in

Table

The

desired

number

of

clusters

in

this

system

with

processors

should

be

ss

However

the

closest

internet

topology

is

having

clusters

This

leads

to

an

actual

system

size

of

processors

which

is

larger

than

the

desired

size

of

by

We

allow

such

deviations

in

system

size

up

to

The

good

configurations

which

yield

actual

system

sizes

outside

this

range

are

dropped

The

actual

bisection

size

offered

by

this

configuration

is

-LRB-

obtained

using

Eqn

-RRB-

The

average

distance

to

be

traveled

in

the

internet

is

obtained

as

ni

-LSB-

-RSB-

The

total

number

of

internet

channels

can

be

obtained

as

-LRB-

N

c

-RRB-

yielding

a

value

of

for

this

configuration

This

is

a

measure

of

the

number

of

inter-cluster

connectors

required

Connector

cost

being

quite

high

-LSB-

-RSB-

a

natural

design

guideline

would

be

to

opt

for

configurations

which

require

less

number

of

connectors

The

board

size

of

used

to

place

the

clusters

in

this

configuration

is

the

same

as

the

cluster

size

Therefore

only

one

cluster

is

placed

on

each

board

The

number

of

inter-board

wires

which

is

a

measure

of

the

volume

of

wiring

is

derived

as

-LRB-

based

on

the

discussion

in

Sec.

-RRB-

Other

entries

in

Table

are

similarly

expanded

Table

shows

similar

characteristics

for

entries

in

Table

under

periphery

pinout

of

pp

Both

Tables

and

are

split

into

three

parts

for

ease

of

presentation

a

-RRB-

one

cluster

is

put

on

one

board

implying

i.e.

the

board

size

is

chosen

to

fit

the

cluster

size

b

-RRB-

two

clusters

are

put

on

a

board

leading

to

and

c

-RRB-

three

or

more

clusters

are

put

on

a

board

leading

to

The

average

distance

to

be

traveled

in

the

internet

is

a

measure

of

the

average

latency

a

message

incurs

While

choosing

the

best

configuration

a

design

objective

should

be

to

choose

a

configuration

that

offers

low

average

message

latency

Since

messages

can

travel

across

clusters

this

average

latency

measure

should

include

both

the

inter-cluster

network

delay

as

well

as

the

delay

incurred

inside

the

cluster

However

not

having

restricted

our

study

to

a

particular

intra-cluster

topology

we

primarily

emphasize

on

the

inter-cluster

delay

Let

us

consider

the

scalability

aspect

in

choosing

a

good

configuration

While

building

a

system

with

processors

once

a

good

configuration

-LRB-

say

n

c

b

and

W

-RRB-

is

chosen

to

build

it

these

parameters

get

fixed

Now

assume

that

it

is

required

to

scale

this

system

to

processors

To

maintain

a

homogeneous

system

the

addition

of

new

processors

to

the

system

should

be

done

while

keeping

the

parameters

n

b

c

and

W

fixed

to

their

prior

values

Otherwise

the

system

requires

complete

redesign

and

refabrication

Let

-LRB-

n

b

c

W

-RRB-

denote

the

set

of

all

n

b

c

and

W

values

that

represent

good

configurations

to

build

a

system

of

N

processors

Similarly

let

-LRB-

n

b

c

W

-RRB-

and

-LRB-

n

b

c

W

-RRB-

denote

such

sets

for

system

sizes

of

N

and

processors

respectively

The

intersection

set

-LRB-

n

b

c

W

-RRB-

T

-LRB-

n

b

c

W

-RRB-

therefore

denotes

good

configurations

under

both

N

and

processors

This

implies

that

these

configurations

used

to

build

a

system

with

processors

can

be

scaled

up

to

processors

Similarly

the

intersection

set

-LRB-

n

b

c

W

-RRB-

T

-LRB-

n

b

c

W

-RRB-

T

-LRB-

n

b

c

W

-RRB-

represents

configurations

possible

for

systems

with

and

processors

Therefore

such

configurations

can

be

scaled

up

to

processors

The

last

column

in

Tables

and

shows

the

maximum

system

size

to

which

the

configuration

being

considered

can

be

scaled

Note

that

not

all

good

configurations

for

processors

scale

up

to

or

processors

Choosing

the

best

configuration

A

-RRB-

Surface

pinout

O

-LRB-

A

-RRB-

technology

In

this

technology

irrespective

of

internet

configuration

the

total

volume

of

inter-board

wires

-LRB-

total

number

of

pins

-RRB-

is

fixed

Larger

boards

offer

greater

bisection

bandwidth

under

this

technology

Thus

a

reasonable

approach

is

to

go

for

larger

board

sizes

for

any

given

internet

dimensionality

To

decide

among

the

good

configurations

we

should

also

keep

in

mind

factors

like

minimal

latency

connector

costs

cost

of

inter-board

wires

and

extent

of

future

scalability

Let

us

consider

the

good

configurations

under

ps

and

W

technology

in

Table

We

chose

the

with

c

as

the

best

topology

as

it

offers

reasonably

high

bisection

size

-LRB-

recall

that

for

a

system

of

size

processors

demanded

bisection

size

is

-RRB-

It

also

offers

low

average

latency

with

low

connector

cost

while

being

scalable

to

around

processors

B

-RRB-

Under

Peripheral

Pinout

O

-LRB-

pA

-RRB-

technology

Under

this

technology

the

total

volume

of

inter-board

wires

in

the

system

falls

with

increasing

board

size

This

decreasing

system

volume

could

translate

to

a

fall

in

packaging

costs

Consider

the

good

configurations

under

pp

and

W

technology

in

Table

Note

the

configurations

are

scalable

up

to

only

processors

For

scalability

beyond

we

need

to

choose

amongst

the

higher

dimensional

systems

The

with

c

and

with

c

under

and

with

c

under

provide

such

scalability

while

offering

reasonably

high

bisection

bandwidth

at

low

connector

and

inter-board

wire

volume

Amongst

these

three

configurations

the

under

is

clearly

better

than

the

under

as

it

offers

similar

characteristics

with

lower

inter-board

volume

of

wires

We

therefore

consider

the

two

configurations

with

c

and

and

with

c

and

Simulation

modeling

was

used

to

decide

the

best

configuration

by

comparing

the

offered

throughput

versus

average

message

latency

For

this

experiment

bus-based

clusters

were

assumed

We

observed

as

shown

in

Fig.

that

at

high

system

loads

the

cluster-bus

became

a

greater

bottleneck

in

the

system

with

larger

clusters

Message

latencies

in

both

systems

at

lower

latencies

were

comparable

These

observations

lead

us

to

conclude

that

the

with

c

will

be

the

best

system

configuration

Avg

latency

of

a

message

-LRB-

cycles

-RRB-

c

c

Throughput

per

node

in

l

bits/cycle

Periphery

Technology

pinout

density

pp

x

x

x

xx

Comparing

two

close

good

configurations

Figure

Comparing

the

performance

of

two

very

close

good

choices

under

periphery

pinout

technology

-LRB-

pp

-RRB-

and

channel

width

-LRB-

W

data

lines

-RRB-

Wormhole

routing

with

message

length

bits

and

message

start

up

time

of

sec

-LRB-

cycles

-RRB-

were

used

Applying

the

Framework

to

Contemporary

Design

Trends

From

Tables

and

the

following

interesting

comments

can

be

made

Cray

example

Employing

the

technology

combination

of

pp

and

W

the

Cray

uses

boards

of

size

containing

clusters

-LRB-

nodes

-RRB-

each

of

size

interconnected

through

a

inter-cluster

network

-LSB-

-RSB-

Our

framework

derives

this

configuration

as

the

first

entry

in

Table

under

This

may

be

treated

as

a

validation

of

our

framework

in

predicting

real

system

design

Also

note

the

first

entry

in

Table

under

This

predicts

that

if

in

a

similar

system

if

boards

of

size

are

used

instead

of

then

by

placing

clusters

on

each

board

the

volume

of

inter-board

wiring

in

a

Cray

system

could

be

reduced

from

to

i.e

almost

halved

leading

to

a

much

less

expensive

system

Flat

systems

are

not

always

possible

Our

framework

is

general

enough

to

allow

flat

configurations

-LRB-

c

-RRB-

However

it

is

interesting

to

note

that

while

designing

system

under

some

technology

combination

it

may

happen

that

the

flat

configuration

is

not

possible

at

all

because

it

is

not

a

good

configuration

This

can

be

seen

in

Table

where

of

the

good

configurations

offer

c

Higher

inter-cluster

dimensionality

may

be

required

Table

Comparing

Various

Performance

and

Cost

Characteristics

of

Good

Configurations

to

Design

a

System

with

N

ss

Processors

Under

Current

Surface

Pinout

Technology

-LRB-

ps

-RRB-

and

Channel

Width

-LRB-

W

-RRB-

Maximum

Cluster

size

and

Board

Size

are

limited

to

Processors

Internet

cluster

Actual

Actual

Average

Number

of

Largest

Volume

Number

of

topology

size

c

number

bisection

distance

internet

board

-LRB-

number

-RRB-

processors

of

processors

size

traveled

in

channels

size

of

inter-board

up

to

which

internet

-LRB-

connectors

-RRB-

usable

wires

scalable

Number

of

clusters

on

a

board

Number

of

clusters

on

a

board

Number

of

clusters

on

a

board

Table

Comparing

Various

Performance

and

Cost

Characteristics

of

Good

Configurations

to

Design

a

System

with

N

ss

Processors

Under

Current

Periphery

Pinout

Technology

-LRB-

pp

-RRB-

and

Channel

Width

-LRB-

W

-RRB-

Maximum

Cluster

size

and

Board

Size

are

limited

to

Processors

Internet

cluster

Actual

Actual

Average

Number

of

Largest

Volume

Number

of

topology

size

c

number

bisection

distance

internet

board

-LRB-

number

-RRB-

processors

of

processors

size

traveled

in

channels

size

of

inter-board

up

to

which

internet

-LRB-

connectors

-RRB-

usable

wires

scalable

Number

of

clusters

on

a

board

Number

of

clusters

on

a

board

Number

of

clusters

on

a

board

It

has

been

recently

demonstrated

that

that

it

is

easier

to

build

lower

dimensional

systems

with

interconnections

-LSB-

-RSB-

Higher

dimensional

networks

require

longer

wires

which

may

lead

to

higher

wiring

costs

and

packing

effort

However

it

is

possible

that

system

design

under

a

given

set

of

packaging

constraints

and

technologies

may

not

offer

any

good

configuration

with

interconnection

For

example

consider

the

design

problem

of

a

scalable

system

with

N

processors

under

current

technologies

of

pp

and

W

Let

the

packaging

constraints

be

such

that

we

can

only

place

one

cluster

on

each

board

-LRB-

-RRB-

The

possible

good

configurations

are

shown

in

Table

under

Note

that

no

good

configuration

is

possible

to

build

such

a

system

with

a

interconnect

Under

such

constraints

a

system

designer

would

therefore

be

forced

to

choose

a

higher

dimensional

system

Summary

of

Best

Scalable

Configurations

We

extended

our

previous

analysis

to

predict

best

system

configurations

for

other

channel

width

and

pinout

density

parameters

reflecting

near

future

technologies

Combinations

of

channel

width

technologies

-LRB-

W

and

-RRB-

were

considered

in

conjunction

with

surface

pinout

technologies

of

-LRB-

ps

and

-RRB-

and

periphery

pinout

technologies

of

-LRB-

pp

and

-RRB-

We

first

studied

the

problem

of

choosing

the

best

configurations

while

restricting

design

to

low-dimensional

-LRB-

-RRB-

inter-cluster

networks

with

small

cluster

sizes

-LRB-

c

-RRB-

Table

shows

such

best

configurations

to

build

a

system

with

size

of

N

processors

under

various

technologies

A

best

system

configuration

depicted

using

the

shorthand

-LRB-

-RRB-

represents

a

-LRB-

-RRB-

system

Other

entries

can

be

similarly

expanded

to

the

k-ary

n-cube

cluster-c

notation

Note

that

the

availability

of

larger

channel

width

under

future

technology

has

potential

to

go

for

larger

cluster

sizes

under

a

given

surface

or

peripheral

pinout

density

However

an

increase

in

pinout

density

without

growth

in

channel

width

will

force

systems

with

smaller

cluster

sizes

We

also

observe

that

a

future

periphery

pinout

density

of

pp

yields

best

system

choices

similar

to

those

by

a

future

surface

pinout

density

of

ps

This

suggests

that

although

theoretically

surface

pinout

has

potential

to

eject

periphery

pinout

technology

as

shown

in

sec.

the

state

of

the

art

is

such

that

even

if

surface

pinout

technology

were

to

improve

times

while

periphery

pinout

technology

only

grew

twice

they

would

still

be

comparable

We

next

considered

the

design

of

systems

with

processors

which

are

scalable

up

to

processors

under

the

same

technologies

In

this

case

we

did

not

restrict

ourselves

to

only

lowdimensional

networks

with

small

cluster

sizes

Table

shows

the

obtained

best

configurations

Note

that

interestingly

all

these

configurations

use

inter-cluster

networks

and

cluster

sizes

from

These

are

reasonable

values

and

it

is

easy

to

comprehend

that

such

systems

will

be

available

in

the

near

future

The

entry

corresponding

to

a

technology

of

ps

and

W

is

empty

because

no

system

configuration

with

processors

scales

up

to

processors

Table

Summary

of

Best

Configurations

Under

Various

Combinations

of

Surface/Periphery

Pinout

and

Channel

Width

Technologies

to

Build

a

System

of

N

Processors

while

Designing

for

Low

Dimensional

-LRB-

-RRB-

Internets

and

Cluster

Size

-LRB-

c

-RRB-

The

first

value

in

each

entry

is

the

dimension

of

the

internet

the

second

is

the

cluster

size

of

the

best

configuration

Surface

Pinout

Technology

ps

ps

W

W

W

W

W

W

Periphery

Pinout

Technology

pp

pp

W

W

W

W

W

W

Table

Summary

of

Best

Configurations

Under

Various

Combinations

of

Surface/Periphery

Pinout

and

Channel

Width

Technologies

to

Build

a

System

of

N

Processors

which

is

Scalable

Up

to

Processors

The

first

value

in

each

entry

is

the

dimension

of

the

internet

the

second

is

the

cluster

size

of

the

best

configuration

Surface

Pinout

Technology

ps

ps

W

W

W

W

W

W

Periphery

Pinout

Technology

pp

pp

W

W

W

W

W

W

Conclusions

and

Future

Work

In

this

paper

we

have

presented

a

comprehensive

supply-demand

framework

to

design

large

multiprocessor

systems

by

taking

into

account

advancements

in

packaging

processor

and

interconnection

technologies

The

elegance

of

this

framework

lies

in

its

parameterized

representation

of

different

technologies

and

then

deriving

an

optimal

system

configuration

for

any

set

of

technologies

and

constraints

while

considering

practical

design

aspects

like

maximum

board

area

maximum

available

pinout

fixed

channel

width

scalability

etc.

The

significance

of

it

lies

in

its

generality

which

has

never

been

taken

into

consideration

by

earlier

researchers

All

proposed

earlier

works

have

centered

around

fixed

topology

technology

or

packaging

constraint

This

generalized

framework

can

be

applied

to

a

wide

variety

of

technological

parameters

and

constraints

for

years

to

come

The

proposed

framework

has

been

validted

with

design

trends

of

contemporary

systems

like

Cray

Using

this

framework

the

following

design

guidelines

have

also

been

obtained

for

building

large

multiprocessor

systems

with

k-ary

n-cube

cluster-c

organizations

Using

a

representative

set

of

current

technological

parameters

and

low

dimensional

networks

a

system

can

be

optimally

built

as

a

-LRB-

-RRB-

system

under

surface

pinout

technology

and

as

a

-LRB-

-RRB-

system

under

periphery

pinout

technology

Designing

optimal

systems

with

high

scalability

under

various

representative

future

technologies

leads

to

inter-cluster

networks

with

cluster

sizes

between

Availability

of

larger

channel

width

under

future

technology

has

potential

to

go

for

larger

cluster

sizes

under

a

given

surface

or

peripheral

pinout

technology

However

an

increase

in

pinout

densities

without

growth

in

channel

width

will

force

systems

with

smaller

cluster

sizes

Clustered

systems

deliver

better

performance

than

flat

systems

under

most

technologies

It

is

therefore

expected

that

clustered

systems

would

be

a

dominant

trend

in

building

future

systems

This

research

has

emphasized

on

the

inter-cluster

network

and

cluster

size

while

designing

clustered

systems

In

the

next

phase

of

this

research

we

are

investigating

on

the

design

of

the

intra-cluster

interconnection

and

the

cluster

interface

The

cluster

interface

may

be

augmented

by

adding

extra

functional

capacities

to

implement

various

communication

and

synchronization

operations

more

efficiently

It

will

be

interesting

to

identify

the

nature

and

extent

of

these

extra

capacities

to

be

provided

as

a

function

of

the

cluster

size

and

their

impact

on

overall

system

performance

This

will

make

our

understanding

of

clustered

systems

more

complete

References

-LSB-

-RSB-

S.

Abraham

and

K.

Padmanabhan

Constraint

based

evaluation

of

multicomputer

networks

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

A.

Agrawal

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

Oct

-LSB-

-RSB-

A.

Asthana

H.

Jagdish

and

B.

Mathews

Impact

of

Advanced

VLSI

packaging

on

the

design

of

a

large

parallel

computer

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Scalable

Architectures

with

k-ary

n-cube

cluster-c

organization

In

Proc

of

the

Symposium

of

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Large

Hierarchical

Multiprocessor

Systems

under

Pro

cessor

Interconnection

and

Packaging

Advancements

In

Proc

of

the

Int

l

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Large

Hierarchical

Multiprocessor

Systems

under

Processor

Interconnection

and

Packaging

Advancements

Technical

Report

OSU-CIS

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Jan

-LSB-

-RSB-

D.

Carlson

The

Mesh

with

a

Global

Mesh

a

Flexible

High-speed

Organization

for

Parallel

Computation

In

Proc

of

the

Int

Conf

on

Supercomputer

Systems

IEEE

Comp

Soc

Press

-LSB-

-RSB-

A.

A.

Chien

and

J.

H.

Kim

Planar

Adaptive

Routing

Low

cost

adaptive

networks

for

multiprocessors

In

Proc

Ann.Int.Symp.on

Comp.Arch

pages

-LCB-

-LSB-

-RSB-

L.

Choi

and

A.

A.

Chien

Integrating

Networks

and

Memory

Hierarchies

in

a

Multicomputer

Node

Architecture

In

Proceedings

of

the

International

Parallel

Processing

Symposium

pages

-LCB-

April

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

W.

J.

Dally

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Transactions

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

S.

Dandamudi

and

D.

Eager

Hierarchical

Interconnection

Networks

for

Multicomputer

Systems

IEEE

Transactions

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Performance

of

Hierarchical

Systems

with

wiring

constraints

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Impact

of

Wiring

Constraints

on

Hierarchical

Network

Performance

In

Proc

of

the

Int

Parallel

Processing

Symposium

Mar

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

A.

W.

Wilson

Jr.

Hierarchical

Cache/Bus

Architecture

for

Shared

Memory

Multiprocessors

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

A.

Louri

and

H.

Sung

Optical

Interconnects

for

High-Speed

Interchip

and

Interboard

Communications

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

M.

D.

Noakes

D.

A.

Wallach

and

W.

J.

Dally

The

J-Machine

Multicomputer

An

Architectural

Evaluation

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

K.

Padmanabhan

Effective

Architectures

for

Data

Access

in

a

Shared

Memory

Hierarchy

Jour

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

D.

K.

Panda

and

D.

Basak

Issues

in

Designing

Scalable

Systems

with

k-ary

n-cube

clusterc

Organization

In

Proc

of

the

First

International

Workshop

on

Parallel

Processing

India

-LSB-

-RSB-

D.

A.

Patterson

Observations

in

Massive

Parallelism

Trends

and

Predictions

for

to

Technical

Report

DIMACS

Sept

-LSB-

-RSB-

M.

T.

Raghunath

Interconnection

Network

Design

Based

on

Packaging

Considerations

PhD

thesis

U.

C.

Berkeley

Nov

-LSB-

-RSB-

M.

T.

Raghunath

and

A.

Ranade

Designing

interconnection

networks

for

multi-level

packaging

In

Proc

of

the

Supercomputing

pages

-LCB-

-LSB-

-RSB-

E.

Rothberg

J.

P.

Singh

and

A.

Gupta

Working

Sets

Cache

Sizes

and

Node

Granularity

Issues

for

Large-Scale

Multiprocessors

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

A.

A.

Sawchuk

B.

K.

Jenkins

C.

S.

Raghavendra

and

A.

Varma

Optical

Crossbar

Networks

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

Seraphim

Lasky

and

Li

Eds

Principles

of

Electronic

Packaging

McGraw

Hill

-LSB-

-RSB-

Tummala

and

Rymaszewski

Microelectronics

Packaging

Handbook

Van

Nostrand

Reinhold

clustered architectures
packaging constraints
pinout technology
scalable systems

hierarchical organization
k-ary n-cube interconnection
multiprocessor system design