Benefits

of

Processor

Clustering

in

Designing

Large

Parallel

Systems

When

and

How

D.

Basak

D.

K.

Panda

and

M.

Banikazemi

Technical

Report

Benefits

of

Processor

Clustering

in

Designing

Large

Parallel

Systems

When

and

How

D.

Basak

D.

K.

Panda

M.

Banikazemi

Dept.

of

Computer

and

Information

Science

Dept.

of

Electrical

Engineering

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

fbasak,panda,banikazeg@cis.ohio-state.edu

Contact

Author

D.

K.

Panda

Abstract

Scaling

the

size

of

parallel

systems

while

maintaining

the

system

performance

is

an

important

problem

A

default

scaling

approach

by

using

larger

networks

to

interconnect

more

processors

works

only

up

to

a

limited

extent

and

the

derived

configurations

using

this

approach

are

not

cost-effective

Recent

advents

in

VLSI

and

packaging

technologies

now

offer

multiple

processors

on

a

single

multichip

module

This

offers

potential

to

design

larger

systems

by

replacing

each

processor

in

a

base

system

by

a

processor-cluster

Similarly

advents

in

the

interconnection

network

technology

now

provide

the

ability

to

use

wider

and

faster

network

channels

In

this

paper

we

analyze

the

scaling

of

systems

by

taking

into

account

current

interconnection

and

processor-integration

technolgies

The

impact

of

software

overhead

to

send

and

receive

messages

in

parallel

system

on

its

scalability

is

also

demonstrated

Our

results

indicate

that

systems

with

high

message

overhead

costs

can

be

scaled

in

a

cost-effective

manner

by

employing

larger

clusters

as

computing

nodes

This

can

be

achieved

without

changing

the

network

until

the

messaging

costs

are

balanced

A

system

with

balanced

messaging

costs

can

be

further

scaled

in

a

more

cost-effective

manner

by

using

larger

clusters

and

wider

network

channels

research

is

supported

in

part

by

NSF

Grant

and

an

Ohio

State

University

Presidential

Fellowship

Contents

Introduction

Scaling

a

Given

System

to

a

Larger

Configuration

Increasing

the

network

size

to

interconnect

more

nodes

Wider

Network

Channels

to

Maintain

System

Performance

Impact

of

Messaging

Overheads

on

Scaling

Large

Configurations

with

Processor

Clustering

Clustered

Organizations

Scaling

by

using

larger

processor-clusters

while

maintaining

network

topology

Architectural

Issues

in

Clustered

Organizations

Performance

Bottlenecks

Messaging

Overheads

Injection/Consumption

Channels

Network

Bisection

Scope

of

Clustering

in

Designing

Large

Systems

Scaling

Systems

with

High

Messaging

Overheads

Scaling

Systems

with

Low

Messaging

Overheads

Unified

Approach

to

Scale

a

given

System

With

Limits

on

Maximum

Cluster

Size

and

Channel

Width

Simulation

Experiments

and

Evaluations

Impact

of

Message

Processing

Overhead

on

System

Performance

Impact

of

clustering

to

derive

balanced

configurations

while

maintaining

base

network

Alternatives

to

Build

a

Given

Sized

System

with

a

fixed

network

channel

width

Scaling

Systems

with

Low

Message

Overhead

and

Wider

Channel

Technology

Conclusion

Introduction

The

ability

to

scale

parallel

systems

is

an

important

problem

to

meet

the

growing

computational

needs

Currently

many

commercial

and

research

parallel

systems

are

using

mesh/torus

interconnections

Their

popularity

mainly

stems

from

the

simplicity

of

fabrication

of

these

topologies

The

default

approach

to

scale

systems

is

to

increase

the

interconnection

network

size

to

connect

more

computing

nodes

A

problem

generally

encountered

while

building

larger

systems

is

that

the

network

may

have

limited

scalability

It

is

well

known

that

the

bisection

of

mesh/torus

interconnections

do

not

scale

linearly

with

size

The

bisection

of

a

network

defined

as

the

minimal

number

of

wires

to

be

cut

to

divide

it

into

two

equal

halves

is

indicative

of

the

peak

message

throughput

supportable

in

the

system

As

the

network

topology

size

is

increased

to

interconnect

more

processors

the

bisection

size

of

the

system

does

not

scale

linearly

and

leads

to

a

fall

in

the

supportable

throughput

per

processor

Thus

the

default

scaling

approach

is

not

good

In

systems

build

in

the

recent

past

by

using

advanced

interconnection

networks

with

and

bit

channels

and

channel

cycle

times

of

ns

the

offered

bandwidth

for

inter-processor

communication

is

very

high

This

coupled

with

the

recent

trends

in

routing

mechanisms

like

fast

packet

switching

virtual-cut

through

and

wormhole

routing

-LSB-

-RSB-

has

led

to

systems

offering

very

small

point-to-point

network

delay

in

the

range

of

less

than

a

microsecond

However

in

most

of

these

systems

the

fast

communication

subsystem

remains

under-utilized

This

happens

because

in

point-to-point

communication

the

sending

and

receiving

of

a

message

is

associated

with

processing

overheads

from

packetization

checksum

computing

and

possibly

memory

copying

Such

overheads

usually

in

the

range

of

few

to

tens

of

microseconds

can

severely

limit

the

rate

at

which

messages

can

be

sent

or

received

from

a

processor

This

can

lead

to

much

of

the

communication

bandwidth

offered

by

the

expensive

fast

interconnect

to

remain

unutilized

This

indicates

the

potential

to

use

multiple

processors

as

a

node

to

utilize

the

network

bandwidth

effectively

This

leads

to

a

challenge

whether

architectural

features

for

a

modern

parallel

systems

can

be

determined

so

that

the

systems

can

be

scaled

in

size

while

utilizing

communication

bandwidth

of

an

interconnection

network

effectively

With

advancements

in

VLSI

and

packaging

technologies

it

has

become

cost-effective

to

integrate

multiple

processing

elements

into

a

chip

or

a

board

-LSB-

-RSB-

This

is

leading

to

the

development

of

parallel

systems

using

such

processor-clusters

as

building

blocks

instead

of

single

processors

-LSB-

-RSB-

Similarly

there

has

been

advancements

in

the

area

of

interconnection

technologies

leading

to

networks

with

wider

and

faster

channels

-LSB-

-RSB-

becoming

feasible

These

new

trends

of

larger

processorclusters

with

wider

interconnection

technology

offer

potential

to

provide

a

more

cost-effective

scaling

of

systems

while

utilizing

the

communication

bandwidth

effectively

In

this

paper

we

take

such

an

approach

and

analyze

the

advantages

and

scope

of

scaling

systems

by

using

processor

clustering

Previous

research

on

clustered

systems

have

focused

mostly

on

proposing

and

proving

different

interconnection

topologies

Examples

of

hierarchical

configurations

proposed

by

researchers

in

the

last

decade

to

build

scalable

systems

using

processor

clusters

include

cluster

of

processors

with

buses

and

MINs

-LSB-

-RSB-

local

and

global

meshes

-LSB-

-RSB-

and

two-level

systems

based

on

hypercube

and

other

network

topologies

-LSB-

-RSB-

Other

researchers

have

studied

the

design

problem

under

very

realistic

packaging

constraints

-LSB-

-RSB-

and

proved

that

under

such

conditions

clustering

becomes

more

useful

However

these

works

do

not

demonstrate

the

merits

of

clustering

in

scaling

systems

Although

some

recent

systems

like

the

-LSB-

-RSB-

DASH

-LSB-

-RSB-

are

being

built

based

on

the

clustering

approach

there

is

no

formal

study

to

demonstrate

when

and

how

such

systems

can

be

scaled

using

processor

clustering

In

this

paper

we

demonstrate

the

potential

of

clustering

in

designing

balanced

and

costeffective

systems

along

two

directions

Systems

with

high

messaging

overheads

can

have

poor

network

utilization

In

this

case

we

show

that

processor

clustering

can

be

used

to

share

network

resources

This

leads

to

effective

exploitation

of

the

communication

bandwidth

of

the

interconnection

network

resulting

in

larger

as

well

as

balanced

and

more

cost-effective

systems

Along

the

second

direction

we

show

that

the

advantages

of

clustering

can

be

also

extended

to

design

large

systems

when

messaging

overheads

are

low

Building

larger

systems

is

proposed

in

this

case

by

the

use

of

larger

clusters

while

maintaining

the

interconnection

network

topology

The

performance

is

maintained

by

using

proportionately

wider

network

channels

We

demonstrate

that

as

long

as

cluster

size

and

channel

width

can

grow

under

technological

constraints

such

approach

offers

solutions

to

design

cost-effective

systems

while

maintaining

performance

The

study

presented

in

this

paper

is

particularly

relevant

from

the

point

of

view

of

a

system

architect

designing

larger

system

configurations

By

exploiting

the

advantages

of

using

larger

clusters

and

wider

interconnect

technology

a

system

designer

can

offer

more

balanced

and

costeffective

configurations

Such

trends

can

be

observed

in

the

design

of

Cray

system

-LSB-

-RSB-

The

results

derived

in

this

paper

provide

detailed

insight

into

this

trend

which

can

be

used

by

system

architects

to

build

larger

and

cost-effective

parallel

systems

under

technological

constraints

The

paper

is

organized

as

follows

In

Sec.

we

present

the

problems

encountered

while

scaling

systems

to

larger

sizes

An

alternative

scaling

approach

using

processor-clusters

is

presented

in

Section

In

Sec.

we

briefly

discuss

the

various

potential

bottlenecks

affecting

the

communication

performance

in

a

parallel

system

Design

guidelines

to

organize

large

systems

in

a

balanced

and

cost-effective

manner

are

presented

in

Sec.

In

Sec.

we

present

simulation

results

to

verify

such

guidelines

Finally

concluding

remarks

and

future

work

are

presented

in

Sec.

Scaling

a

Given

System

to

a

Larger

Configuration

A

parallel

system

consists

of

large

number

of

processors

interconnected

through

a

network

Typical

topologies

for

the

interconnection

are

meshes/tori

hypercubes

multi-stage

interconnection

networks

fat-trees

Many

currently

proposed

systems

like

the

Intel

Paragon

use

mesh/torus

topology

To

meet

the

increasing

demand

on

computing

bandwidth

parallel

systems

need

to

be

scaled

to

larger

configurations

supporting

more

processors

In

this

section

we

illustrate

how

two

default

approaches

can

be

used

for

scaling

and

emphasize

their

limitations

Increasing

the

network

size

to

interconnect

more

nodes

This

is

the

default

approach

to

scale

systems

as

shown

through

the

illustration

in

Figure

-LRB-

a

-RRB-

An

example

system

with

N

processors

organized

as

a

mesh

with

channels

of

width

W

Let

this

system

be

scaled

by

a

factor

of

g

times

to

a

processor

system

organized

as

a

larger

mesh

All

other

architectural

parameters

in

the

system

like

network

channel

width

-LRB-

W

-RRB-

channel

speed

are

kept

fixed

It

is

desirable

that

performance

in

a

system

defined

by

the

sustainable

throughput

per

processor

and

the

average

network

latency

to

achieve

it

be

maintained

while

such

scaling

is

attained

However

it

can

be

observed

in

Fig.

-LRB-

a

-RRB-

that

the

size

of

the

bisection

denoted

as

B

does

not

scale

linearly

with

system

size

In

the

case

of

mesh

it

scales

by

a

factor

of

only

pg

from

in

the

system

to

in

the

system

The

achievable

throughput

per

processor

under

uniform

traffic

denoted

as

is

proportional

to

the

size

of

the

network

bisection

divided

by

the

number

of

processors

in

the

system

-LRB-

B

N

-RRB-

Thus

in

a

mesh

falls

by

a

factor

of

pg

as

the

system

size

is

scaled

by

a

factor

of

g

This

leads

to

degradation

in

performance

with

scaling

Wider

Network

Channels

to

Maintain

System

Performance

To

overcome

the

above

problem

and

maintain

performance

with

scaling

it

is

clear

that

the

bisection

capacity

needs

to

be

scaled

linearly

This

can

be

achieved

by

using

wider

channels

in

the

network

For

example

consider

the

system

in

Fig.

-LRB-

b

-RRB-

Instead

of

having

channels

of

width

W

channels

of

width

xW

are

now

used

With

x

the

bisection

size

of

the

system

is

leading

to

a

similar

as

in

the

base

system

With

improvements

in

router

and

interconnection

technology

wider

channels

in

the

network

have

become

a

viable

option

A

system

architect

can

thus

exploit

such

technologies

to

design

larger

systems

W

Wcomputing

node

with

processor

router

W

W

xW

Base

system

with

N

processors

in

a

mesh

Maintaining

Channel

Width

W

Using

Wider

Channels

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Bisection

size

B

B

B

Figure

An

example

showing

the

scaling

of

a

base

system

with

processors

to

a

processor

system

by

using

a

larger

network

The

channel

width

in

the

scaled

system

as

compared

to

the

base

system

may

be

a

-RRB-

kept

same

or

b

-RRB-

increased

by

a

given

factor

x.

Impact

of

Messaging

Overheads

on

Scaling

The

performance

attainable

in

a

given

system

is

determined

by

various

factors

like

the

network

bisection

size

as

discussed

above

However

other

factors

like

message

processing

overheads

can

also

constrain

the

maximum

performance

obtained

from

a

system

We

discuss

these

issues

in

greater

detail

in

Section

For

now

let

us

consider

the

impact

of

messaging

overheads

in

a

simplistic

manner

Messaging

overheads

are

generally

related

to

software

processing

costs

incurred

during

sending

and

receiving

a

message

Such

costs

can

restrict

the

maximum

rate

of

sending

messages

from

a

processor

We

denote

such

a

limit

on

the

throughput

due

to

messaging

overheads

as

sw

The

value

of

sw

is

independent

of

system

size

Let

us

denote

the

constraint

on

throughput

from

network

bisection

as

discussed

earlier

as

nw

;p

The

overall

throughput

in

the

system

can

now

be

determined

as

min

-LRB-

sw

nw

;p

-RRB-

Let

us

consider

the

impact

of

relative

values

of

the

parameters

sw

and

nw

;p

on

system

scaling

For

the

system

in

Fig.

-LRB-

a

-RRB-

let

sw

messages

sec

and

nw

;p

messages

sec

This

leads

to

a

value

of

messages

sec

By

scaling

the

system

to

a

configuration

while

maintaining

channel

width

the

value

of

nw

;p

falls

by

a

factor

of

two

to

messages

sec

However

the

overall

system

throughput

in

this

case

remains

at

messages

sec

By

using

an

analytical

model

similar

to

the

one

presented

in

-LSB-

-RSB-

to

model

contention

in

wormhole

routed

systems

it

can

be

easily

demonstrated

that

such

scaling

is

achieved

with

only

a

marginal

increase

in

average

message

Thus

such

a

system

can

be

scaled

without

degradation

in

system

throughput

to

an

size

without

using

wider

network

channels

However

let

us

now

consider

a

different

set

of

values

for

sw

and

nw

;p

Let

sw

messages

sec

and

nw

;p

messages

sec

in

the

system

leading

to

a

messages

sec

As

before

the

parameter

nw

;p

reduces

to

messages

sec

when

the

system

is

scaled

to

configuration

However

this

leads

to

the

overall

throughput

to

degrade

to

messages

sec

In

such

a

system

wider

channels

are

necessary

to

maintain

system

performance

Large

Configurations

with

Processor

Clustering

With

advancements

in

VLSI

and

packaging

technologies

it

has

become

cost-effective

to

integrate

multiple

processing

elements

into

a

chip

or

board

Thus

computing

nodes

having

more

than

one

processor

on

a

single

multi-chip

module

or

processor-board

are

becoming

increasingly

available

These

are

also

referred

to

as

processor-clusters

Many

current

parallel

systems

like

the

CRAY

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

and

the

Stanford

DASH

-LSB-

-RSB-

are

using

such

clustered

organizations

to

build

systems

In

this

section

we

introduce

clustered

organizations

indicate

that

such

organizations

have

potential

to

support

larger

systems

in

a

more

cost-effective

manner

and

then

discuss

some

architectural

details

of

such

organizations

Clustered

Organizations

Recently

we

have

introduced

a

new

k-ary

n-cube

cluster-c

organization

-LSB-

-RSB-

to

capture

the

upcoming

trend

in

building

scalable

parallel

systems

In

this

organization

the

lower

level

consists

of

kn

clusters

of

processors

These

clusters

are

interconnected

by

a

higher

level

direct

k-ary

n-cube

network

-LRB-

also

referred

to

as

inter-cluster

network

-RRB-

Each

cluster

consists

of

c

processors

leading

to

a

total

of

N

-LRB-

kn

c

-RRB-

processors

in

the

system

Figure

shows

the

overall

configuration

of

such

a

system

Intercluster

Network

Intra-cluster

Network

ClusterInterface

-LRB-

CI

-RRB-

k-ary

n-cube

c-processor

Clusters

Bus/MIN/direct

star

Figure

Two-level

clustering

with

k-ary

n-cube

cluster-c

organization

network

contention

with

wormhole

routing

is

not

the

objective

of

this

paper

Without

presenting

such

details

here

we

refer

interested

readers

to

relevant

treatises

on

analytical

modeling

in

-LSB-

-RSB-

Scaling

by

using

larger

processor-clusters

while

maintaining

network

topology

Large

processor

clusters

can

be

used

to

scale

systems

as

shown

in

Fig

An

example

system

with

N

processors

organized

as

a

mesh

with

channels

of

width

W

is

scaled

times

to

a

processor

system

by

using

processor-clusters

with

processors

The

network

topology

is

maintained

as

a

mesh

In

Fig.

-LRB-

a

-RRB-

the

channel

width

is

maintained

at

W

the

same

as

in

the

base

system

Thus

the

bisection

size

of

the

network

in

the

scaled

system

is

also

kept

the

same

However

the

number

of

processors

in

the

system

is

four

times

larger

This

leads

to

a

value

of

nw

;p

in

the

scaled

clustered

system

to

be

of

the

value

in

the

base

system

In

Fig.

-LRB-

b

-RRB-

the

channel

width

in

the

clustered

system

is

increased

by

a

factor

of

x

This

leads

to

the

value

of

the

parameter

nw

;p

in

such

a

system

to

be

as

in

the

base

system

For

x

the

value

of

nw

;p

in

the

scaled

system

can

be

maintained

as

in

the

base

system

As

discussed

in

Section

the

relative

values

of

sw

and

nw

;p

have

impact

on

the

nature

of

scaling

For

a

base

system

with

sw

nw

;p

the

network

is

under-utilized

Thus

larger

system

sizes

can

be

supported

as

shown

in

Fig.

-LRB-

a

-RRB-

by

using

larger

clusters

until

the

network

is

effectively

used

For

systems

with

sw

nw

;p

the

network

is

not

under-utilized

Thus

any

increase

in

system

size

by

using

larger

clusters

has

to

be

coupled

with

increase

in

channel

width

to

maintain

performance

as

shown

in

Fig.

-LRB-

b

-RRB-

The

above

discussion

indicates

that

using

processor-clusters

offers

an

alternate

way

to

scale

systems

to

larger

sizes

In

this

approach

the

network

topology

can

be

maintained

leading

to

more

compact

interconnections

at

lower

cost

We

discuss

these

aspects

in

detail

in

Section

computing

node

with

processor

router

WBase

system

with

N

processors

in

a

mesh

W

W

W

xW

computing

node

with

processors

Scaling

Computing

Node

Size

to

to

have

N

processors

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Bisection

size

B

B

B

Maintaining

Channel

Width

W

Using

Wider

Channels

Figure

An

example

showing

the

scaling

of

a

base

system

with

processors

to

a

processor

system

by

using

larger

computing

nodes

with

processors

each

The

channel

width

in

the

scaled

system

as

compared

to

the

base

system

may

be

a

-RRB-

kept

same

or

b

-RRB-

increased

by

a

given

factor

Architectural

Issues

in

Clustered

Organizations

Before

going

to

detailed

performance

analysis

with

clustered

organizations

let

us

discuss

some

architectural

issues

in

designing

clustered

organizations

We

use

these

architectural

features

in

our

analysis

In

this

paper

we

assume

the

popular

wormhole

mechanism

to

route

messages

in

the

intercluster

network

In

wormhole

routing

a

message

is

divided

into

flits

-LSB-

-RSB-

which

are

then

pipelined

over

a

path

in

the

network

established

by

the

header

flit

This

offers

a

low-cost

low-latency

and

scalable

communication

sub-system

The

interconnection

within

a

cluster

-LRB-

also

referred

to

as

intra-cluster

network

or

intranet

-RRB-

can

be

chosen

as

bus/MIN/star

network/direct

network

as

shown

in

Fig.

The

merits

and

demerits

of

each

of

these

organizations

is

presented

in

-LSB-

-RSB-

Although

the

study

presented

in

this

paper

can

be

extended

to

include

other

cluster

organizations

in

this

paper

we

assume

clusters

to

be

starconnected

as

shown

in

Fig.

-LRB-

d

-RRB-

Each

processor

in

a

star

cluster

is

connected

to

a

cluster

interface

-LRB-

CI

-RRB-

It

is

to

be

noted

that

star-based

cluster

organization

with

c

is

used

in

the

system

-LSB-

-RSB-

a

-RRB-

direct-network

based

c

-RRB-

bus-based

d

-RRB-

star-basedb

-RRB-

MIN-basedMIN

Cluster

interconnections

connections

toother

clusters

CI

CI

CI

CI

processor-memory

unit

routerCI

cluster

interface

consumption

channel

injectionchannel

Figure

Four

possible

cluster

interconnections

under

k-ary

n-cube

cluster-c

organization

The

CI

is

connected

to

the

rest

of

the

system

through

a

network

router

The

main

task

of

the

cluster

interface

is

to

handle

the

volume

of

communication

to/from

the

cluster

Other

functionalities

of

the

CI

may

include

efficient

implementation

of

various

communication

synchronization

and

cache-coherence

operations

-LSB-

-RSB-

However

such

discussion

is

beyond

the

scope

of

this

paper

The

links

connecting

a

CI

to

the

local

router

are

referred

to

as

injection

and

consumption

channels

-LSB-

-RSB-

An

injection

channel

is

used

to

send

out

-LRB-

inject

-RRB-

a

message

into

the

network

Similarly

a

consumption

channel

is

used

to

receive

-LRB-

consume

-RRB-

an

incoming

message

destined

to

this

cluster

An

injection

or

consumption

channel

is

reserved

by

the

header

flit

of

a

message

and

held

reserved

till

the

tail

flit

releases

it

Currently

most

multiprocessor

systems

provide

only

one

injection

and

one

consumption

channel

at

each

router

Without

loss

of

generality

and

for

deriving

realistic

design

guidelines

in

the

remaining

discussion

we

assume

the

same

It

is

easy

to

observe

that

the

flat

k-ary

n-cube

can

be

derived

as

a

special

case

of

the

k-ary

n-cube

cluster-c

family

by

choosing

cluster

size

c

In

our

analysis

we

denote

the

width

of

the

channels

in

the

network

and

injection/consumption

channels

as

W

In

the

remaining

discussion

we

represent

a

k-ary

n-cube

cluster-c

system

with

channel

width

W

using

a

compact

notation

-LRB-

kn

c

W

-RRB-

For

example

a

inter-cluster

network

with

clusters

of

four

processors

each

and

channel

width

of

bytes

is

represented

as

-LRB-

-RRB-

or

-LRB-

-RRB-

Performance

Bottlenecks

Let

us

consider

the

process

of

message

transfer

from

a

sender

processor

in

one

cluster

to

a

destination

processor

in

another

cluster

as

shown

in

Fig.

-LRB-

a

-RRB-

Before

being

sent

out

a

message

requires

software

processing

at

the

sender

This

is

followed

by

injection

of

the

message

into

the

network

via

an

injection

channel

-LSB-

-RSB-

The

message

then

traverses

across

the

network

to

the

router

associated

with

the

destination

cluster

where

it

transferred

to

the

destination

processor

via

a

consumption

channel

It

then

incurs

a

software

processing

overhead

at

the

destination

processor

before

being

ready

to

be

used

Such

processing

of

a

message

can

therefore

be

divided

into

five

stages

as

shown

ProcessorSource

consumptionchannels

injectionchannels

ProcessorDestination

intermediate

routers

destinationrouter

messagingoverheads

consumptionchannels

injectionchannelsmessagingoverheads

Interconnection

Network

source

router

Source

cluster

Destination

cluster

messageprocessingatsource

injectionchannel

inter-connectionnetwork

message

-LRB-

a

-RRB-

-LRB-

b

-RRB-

processor

consump-channeltion

messageprocessingatdestinationprocessor

Figure

The

process

of

a

message

transfer

from

a

source

processor

to

a

destination

processor

across

different

clusters

in

Fig.

-LRB-

b

-RRB-

Clearly

the

bandwidth

available

at

each

of

these

stages

restricts

the

overall

rate

at

which

messages

can

be

handled

in

the

system

Let

us

derive

the

sustainable

throughput

allowed

by

each

of

these

stages

The

actual

throughput

achieved

in

the

system

is

determined

by

the

most

restrictive

constraint

This

analysis

helps

us

in

estimating

relative

performance

of

different

system

configurations

and

predict

the

impact

of

scaling

on

a

given

system

performance

An

accurate

simulation

modeling

is

used

later

to

verify

the

guidelines

established

here

and

in

the

next

section

Messaging

Overheads

While

processing

a

message

at

the

sender

a

software

latency

is

involved

which

includes

the

overhead

of

packetization

checksum

computing

and

possibly

memory

copying

A

similar

software

overhead

is

also

incurred

at

the

receiver

Without

loss

of

generality

in

this

paper

we

lump

both

overheads

onto

the

sender

side

and

represent

them

as

a

single

processing

overhead

delay

denoted

as

tsw?sec

In

the

absence

of

specialized

hardware

to

perform

these

operations

the

local

processor

has

to

incur

these

overheads

Thus

it

restricts

the

fastest

rate

at

which

messages

can

be

dispatched

from

a

processor

This

is

denoted

as

sw

where

we

have

sw

tsw

-LRB-

messages

sec

-RRB-

-LRB-

-RRB-

Representative

values

of

tsw

of

some

current

machines

are

shown

in

Table

For

systems

with

high

value

of

tsw

as

is

the

case

with

many

of

the

recent

machines

like

IBM/SP

the

achievable

message

injection

can

be

quite

limited

For

example

the

maximum

message

injection

rate

on

the

IBM/SP

can

be

only

messages

sec

considering

byte

messages

Table

Message

Start-up

Overheads

in

Several

Current

Commercial

and

Research

Multiprocessors

Machine

Network

Channel

Cycle

tsw

Width

-LRB-

W

-RRB-

ns

cycles

bits

Hypercube

-LRB-

sec

-RRB-

FatTree

-LRB-

sec

-RRB-

Dash

Mesh

-LRB-

sec

-RRB-

IBM/SP

MIN

-LRB-

sec

-RRB-

J-Machine

Mesh

-LRB-

sec

-RRB-

Monsoon

Butterfly

-LRB-

sec

-RRB-

Injection/Consumption

Channels

A

message

to

be

sent

into

the

network

from

a

processor

is

assigned

an

injection

channel

which

is

held

until

the

last

flit

of

the

message

is

released

into

the

network

Thus

for

a

message

of

length

L

bytes

or

L

W

flits

and

a

system

channel

cycle

time

of

tc

the

time

to

inject

a

message

is

derived

as

tc

-LRB-

L

W

-RRB-

The

maximum

rate

at

which

messages

can

be

injected

from

a

cluster

using

a

single

injection

channel

is

therefore

tc

-LRB-

L

W

-RRB-

messages

sec

Assuming

this

injection

bandwidth

is

equally

shared

by

the

c

processors

inside

a

cluster

the

maximum

injection

rate

available

to

each

processor

denoted

as

CI

;p

is

given

by

CI

;p

-LRB-

ctc

-LRB-

L

W

-RRB-

-RRB-

-LRB-

-RRB-

Similarly

the

maximum

rate

of

consumption

per

processor

can

be

derived

to

be

identical

to

CI

;p

Network

Bisection

A

small

network

bisection

limits

the

maximal

achievable

system

performance

Let

us

denote

the

bisection

of

a

network

as

B

It

is

the

minimum

number

of

network

channels

to

be

cut

so

as

to

divide

the

network

into

two

equal

halves

For

a

clustered

k-ary

n-cube

cluster-c

system

the

bisection

of

the

k-ary

n-cube

inter-cluster

network

is

derived

-LSB-

-RSB-

as

B

-LRB-

-RRB-

assuming

a

channel

width

of

W

bytes

Let

the

channel

cycle

time

in

the

network

be

denoted

as

tc

sec

Thus

W

bytes

can

be

sent

across

a

network

channel

in

tc

sec

Similarly

B

bytes

can

be

sent

across

the

network

bisection

in

the

same

duration

The

maximum

rate

of

sending

messages

across

the

bisection

is

therefore

B

Tc

bytes

sec

Under

uniform

traffic

where

each

processor

sends

messages

to

other

processors

with

equal

probability

given

a

sending

rate

of

messages

sec

at

each

processor

the

number

of

messages

crossing

the

bisection

can

be

derived

as

N

where

N

knc

is

the

number

of

processors

in

the

system

This

corresponds

to

NL

bytes

crossing

the

bisection

every

sec

where

L

denotes

the

message

length

in

the

system

in

bytes

Since

this

has

to

be

less

than

the

bisection

capacity

it

leads

to

NL

B

Tc

-LRB-

-RRB-

From

the

above

inequality

the

maximum

injection

at

each

processor

denoted

by

us

as

B

;p

can

be

derived

as

B

;p

-LRB-

NLTc

-RRB-

messages

sec

-LRB-

-RRB-

For

a

k-ary

n-cube

inter-cluster

network

this

can

be

simplified

as

B

;p

-LRB-

NLTc

-RRB-

-LRB-

kncLTc

-RRB-

-LRB-

kcLTc

-RRB-

leading

us

to

B

;p

-LRB-

kcLTc

-RRB-

messages

sec

-LRB-

-RRB-

By

comparing

Equations

and

we

can

derive

CI

;p

B

;p

to

be

k

Thus

for

systems

with

k

CI

;p

B

;p

indicates

that

bisection

constraint

is

more

restrictive

than

injection/consumption

constraint

on

throughput

In

future

discussion

when

considering

inter-cluster

network

with

k

we

consider

only

B

;p

It

is

also

to

be

noted

that

typically

due

to

contention

in

interconnection

networks

the

average

throughput

achievable

per

processor

is

only

a

fraction

of

B

;p

This

fraction

represents

the

utilization

of

the

network

and

usually

for

a

given

set

of

system

parameters

like

toruses

this

number

needs

to

be

adjusted

by

a

factor

of

two

to

B

channels

due

to

the

presence

of

wrap-around

links

message

length

routing

adaptivity

-LSB-

-RSB-

traffic

patterns

and

virtual

channel

flow

control

-LSB-

-RSB-

techniques

it

remains

almost

unchanged

with

system

size

-LSB-

-RSB-

We

denote

this

achievable

fractional

throughput

as

nw

;p

Later

we

use

accurate

simulation

modeling

of

contention

to

determine

such

network

throughput

in

presence

of

contention

Thus

we

have

nw

;p

B

pr

r

-LRB-

kcLTc

-RRB-

messages

sec

-LRB-

-RRB-

where

r

denotes

the

utilization

fraction

of

the

network

From

Equation

it

may

be

observed

that

keeping

other

parameters

fixed

if

the

radix

k

of

the

inter-cluster

network

were

to

be

doubled

the

rate

nw

;p

would

fall

by

half

This

implies

that

if

a

system

with

a

kxk

interconnection

kxk

c

W

is

scaled

four

times

to

c

W

by

increasing

the

interconnection

to

the

value

of

nw

;p

falls

by

half

Similarly

if

the

cluster

size

-LRB-

c

-RRB-

is

increased

nw

;p

falls

as

c

This

leads

us

to

the

following

observation

Observation

For

a

-LRB-

k?k

c

W

-RRB-

system

supporting

a

message

injection

rate

per

processor

of

nw

;p

if

the

system

is

scaled

four

times

to

-LRB-

W

-RRB-

the

supported

nw

;p

falls

by

half

Similarly

if

the

system

is

scaled

twice

to

-LRB-

k?k

c

W

-RRB-

by

using

two

times

larger

clusters

the

supported

nw

;p

falls

by

half

Given

a

system

we

characterize

it

by

its

nw

;p

and

sw

parameters

In

systems

with

high

messaging

overheads

we

expect

sw

nw

;p

Thus

in

such

systems

the

throughput

is

throttled

by

messaging

overheads

which

leads

to

network

resources

to

remain

under-utilized

Similarly

in

systems

with

sw

nw

;p

the

network

is

effectively

used

but

the

processors

capacity

to

inject

faster

is

throttled

A

system

with

sw

ss

nw

;p

is

defined

to

be

balanced

Clearly

in

a

balanced

system

sw

nw

;p

This

indicates

that

both

the

communication

processing

and

network

resources

in

the

system

are

used

effectively

leading

to

cost-optimal

systems

Systems

not

balanced

are

referred

to

as

unbalanced

In

the

next

section

we

carry

out

a

study

in

two

phases

to

design

such

balanced

systems

using

processor-clusters

In

the

first

phase

we

consider

balancing

systems

which

are

unbalanced

-LRB-

with

sw

nw

;p

-RRB-

by

using

larger

clusters

In

the

second

phase

we

consider

scaling

balanced

systems

to

larger

balanced

systems

Scope

of

Clustering

in

Designing

Large

Systems

In

Section

we

indicated

that

clustered

systems

can

offer

more

cost-effective

approach

to

scale

systems

In

this

section

we

analyze

this

in

more

detail

for

systems

with

both

high

and

low

messaging

overheads

Scaling

Systems

with

High

Messaging

Overheads

In

systems

with

messaging

overheads

sw

nw

;p

the

achievable

maximum

throughput

min

-LRB-

sw

nw

;p

-RRB-

per

processor

is

governed

by

sw

resulting

in

network

resources

to

remain

under-utilized

Thus

scaling

can

be

achieved

without

having

to

use

wider

network

channels

Let

us

consider

an

example

of

a

base

system

with

processors

in

a

flat

configuration

and

channel

width

W

bytes

as

shown

in

Fig.

-LRB-

a

-RRB-

This

system

is

represented

as

in

the

shorthand

notation

discussed

earlier

Let

sw

messages

sec

and

nw

;p

messages

The

network

resources

in

this

system

are

utilized

only

up

to

This

suggests

that

a

larger

system

with

more

processors

can

be

supported

on

the

same

network

while

utilizing

the

network

resources

more

effectively

Such

a

system

with

configuration

is

shown

in

Fig.

-LRB-

b

-RRB-

From

Equation

we

know

that

nw

;p

c.

Thus

in

configuration

-LRB-

b

-RRB-

value

of

nw

;p

reduces

to

messages

sec

This

leads

to

a

match

with

the

value

of

sw

resulting

in

a

balanced

system

Thus

for

the

same

inter-cluster

network

cost

in

this

example

we

can

support

a

twice

larger

system

In

Sec.

we

demonstrate

that

this

is

also

achieved

with

very

marginal

increase

in

average

message

latency

Let

us

investigate

an

alternative

solution

to

build

a

processor

system

by

scaling

the

network

to

a

mesh

while

using

clusters

of

size

one

The

resultant

configuration

of

is

shown

in

Fig

-LRB-

c

-RRB-

In

such

a

system

also

nw

;p

reduces

to

messages

sec

Thus

this

system

offers

similar

throughput

as

the

clustered

system

However

the

network

used

in

the

configuration

being

larger

we

expect

this

system

to

have

higher

interconnection

costs

This

indicates

that

the

clustered

approach

is

more

cost-effective

The

above

approach

of

scaling

systems

using

larger

clusters

can

be

generalized

into

the

following

result

Result

Given

a

base

system

-LRB-

kn-c

W

-RRB-

with

N

knc

processors

and

f

nw

;p

sw

s.t.

f

a

larger

system

of

size

Nf

delivering

similar

performance

can

be

built

by

using

clusters

of

size

cf

and

keeping

the

inter-cluster

network

size

-LRB-

kn

-RRB-

and

channel

width

-LRB-

W

-RRB-

fixed

where

f

f

Thus

scaling

by

using

larger

clusters

while

maintaining

the

same

network

works

until

sw

nw

;p

When

a

scaled

balanced

configuration

with

sw

ss

nw

;p

is

reached

the

above

approach

is

no

longer

applicable

Further

scaling

while

not

modifying

the

network

leads

to

a

degradation

in

system

performance

high/low

messaging

overheads

we

do

not

refer

to

tsw

in

absolute

terms

Rather

we

refer

to

the

relationship

between

sw

and

nw

;p

-RRB-

A

high/low

value

of

sw

nw

;p

indicates

low/high

messaging

overheads

assume

that

this

value

nw

;p

is

derived

by

taking

network

contention

into

account

system

W

bytes

cluster

-LRB-

c

-RRB-

processor

router

system

N

processors

N

processors

lnw

p

messages/msec

lnw

p

messages/msec

lsw

messages/msec

in

all

systems

-LRB-

a

-RRB-

-LRB-

b

-RRB-

thus

l

messages/msec

in

all

systems.l

min

-LRB-

lnw

p

lsw

-RRB-

cluster

-LRB-

c

-RRB-

W

bytes

lnw

p

messages/msec

W

bytes

systemN

processors

-LRB-

c

-RRB-

Figure

An

example

to

demonstrate

scaling

in

systems

with

high

message

overheads

-LRB-

sw

nw

;p

-RRB-

while

maintaining

the

average

throughput

in

the

system

A

base

configuration

-LRB-

a

-RRB-

flat

system

with

c

can

be

scaled

to

a

balanced

configuration

-LRB-

b

-RRB-

using

larger

clusters

-LRB-

c

-RRB-

or

configuration

-LRB-

c

-RRB-

using

a

larger

network

with

c

The

channel

width

is

maintained

at

W

bytes

in

all

three

configurations

Scaling

Systems

with

Low

Messaging

Overheads

Let

us

consider

the

problem

of

scaling

a

base

system

with

low

messaging

overheads

with

sw

nw

;p

In

such

a

system

the

network

is

not

under-utilized

and

the

achieved

throughput

per

processor

is

derived

as

nw

;p

In

the

scaled

configurations

it

is

desired

to

maintain

the

same

performance

To

achieve

this

we

need

to

increase

the

network

bandwidth

as

we

scale

the

system

size

In

Sec.

we

indicated

that

this

can

be

achieved

by

using

wider

network

channels

We

compare

two

alternate

techniques

to

scale

such

a

given

system

a

-RRB-

using

larger

clusters

with

wider

network

channels

while

keeping

interconnection

topology

fixed

and

b

-RRB-

using

larger

interconnection

network

with

wider

channels

while

keeping

cluster

size

fixed

These

are

illustrated

through

examples

in

Fig.

Let

us

consider

an

example

base

system

configuration

with

processors

as

shown

in

Fig.

-LRB-

a

-RRB-

Let

the

system

be

balanced

with

sw

nw

;p

messages

sec

The

system

size

can

be

increased

to

processors

by

using

four

times

larger

clusters

-LRB-

c

-RRB-

but

this

leads

to

a

fall

in

nw

;p

to

messages

sec

from

Equation

The

nw

;p

can

be

maintained

at

messages

sec

by

increasing

the

channel

width

in

the

system

four

times

to

W

bytes

leading

to

a

system

as

shown

in

Fig.

-LRB-

b

-RRB-

An

alternative

way

to

scale

the

above

system

is

by

increasing

the

network

to

mesh

thus

allowing

more

clusters

to

be

connected

This

is

first

done

while

maintaining

cluster

size

and

channel

width

in

the

system

However

from

Observation

we

know

that

the

bisection

of

a

system

does

not

scale

linearly

with

its

size

There

is

a

resultant

fall

in

the

value

of

nw

;p

to

messages

sec

system

with

W

bytes

N

processors

lnw

p

messages/msec

lsw

messages/msec

W

bytes

N

processors

system

with

lnw

p

messages/msec

W

bytes

system

with

lnw

p

messages/msec

Inter-cluster

wiring

cost

wires

Inter-cluster

wiring

cost

wires

Inter-cluster

wiring

cost

-LRB-

-RRB-

W

wires

wires

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

N

processors

processor

cluster

-LRB-

c

-RRB-

cluster

-LRB-

c

-RRB-

router

Figure

An

example

to

demonstrate

the

scaling

of

a

base

system

with

low

messaging

overheads

while

maintaining

the

average

throughput

in

the

system

A

configuration

with

processors

is

scaled

to

a

size

of

processors

in

two

ways

-RRB-

by

increasing

cluster

size

and

channel

width

to

yield

a

system

and

-RRB-

by

increasing

network

size

and

channel

width

to

yield

a

system

The

is

more

cost-effective

in

terms

of

inter-cluster

wiring

costs

This

fall

in

throughput

can

be

compensated

by

doubling

the

network

channel

width

to

W

bytes

The

resulting

system

configuration

of

is

shown

in

Fig.

-LRB-

c

-RRB-

Both

configurations

derived

above

and

maintain

the

throughput

of

the

base

system

This

leads

us

to

an

interesting

question

given

that

either

system

can

be

built

which

one

of

these

would

be

more

cost-effective

and

deliver

lower

average

message

latency

To

compare

costs

here

we

use

the

metric

of

interconnect

wiring

costs

-LSB-

-RSB-

It

has

been

demonstrated

in

-LSB-

-RSB-

that

such

costs

in

the

inter-cluster

network

constitute

a

significant

fraction

of

the

total

system

costs

We

determine

these

cost

figures

for

both

systems

by

first

computing

the

total

number

of

channels

in

the

inter-cluster

network

-LRB-

-LRB-

k

-RRB-

-RRB-

This

is

multiplied

by

the

channel

width

-LRB-

W

bytes

wires

-RRB-

to

yield

the

total

number

of

connecting

wires

as

-LRB-

k

-RRB-

W

These

figures

are

shown

in

Fig.

below

the

respective

configurations

It

may

be

observed

that

the

system

offers

almost

half

inter-cluster

wiring

cost

over

the

configuration

This

leads

us

to

conclude

that

using

larger

clusters

offers

more

cost-effective

design

Similarly

assuming

that

intra-cluster

message

delays

are

low

the

smaller

inter-cluster

network

in

the

system

with

a

smaller

diameter

is

expected

to

offer

lower

average

message

latency

This

leads

us

to

the

following

result

Result

A

base

system

with

-LRB-

kn-c

W

-RRB-

configuration

and

N

knc

processors

and

low

messaging

overheads

-LRB-

nw

;p

sw

-RRB-

can

be

scaled

to

a

larger

system

size

with

Ng

processors

while

maintaining

the

average

throughput

in

the

system

by

using

a

-RRB-

Larger

clusters

of

size

cg

and

channel

width

gW

to

yield

a

-LRB-

kn-cg

gW

-RRB-

configuration

or

b

-RRB-

Larger

interconnect

-LRB-

n

-RRB-

n

with

channel

width

nW

to

yield

a

-LRB-

-LRB-

n

-RRB-

n-c

nW

-RRB-

configuration

However

alternative

-LRB-

a

-RRB-

using

larger

clusters

and

smaller

inter-cluster

network

offers

a

more

wiring

cost-effective

system

with

lower

average

message

latency

It

may

be

noted

that

besides

the

two

approaches

to

scale

a

base

system

to

a

larger

size

as

discussed

above

there

is

a

whole

gamut

of

hybrid

approaches

These

use

a

combination

of

increasing

cluster

size

network

size

and

channel

width

to

build

larger

systems

For

example

consider

scaling

a

kn-c

W

configuration

with

N

processors

to

a

system

with

Ng

processors

Let

g

We

first

scale

the

base

system

times

by

using

larger

clusters

and

a

corresponding

larger

channel

width

leading

us

to

a

system

with

configuration

We

then

use

a

larger

network

with

larger

channel

width

to

scale

the

resultant

system

further

times

to

lead

us

to

the

final

system

with

-LRB-

n

-RRB-

n

W

configuration

For

a

inter-cluster

network

-LRB-

n

-RRB-

this

simplifies

to

-LRB-

-RRB-

x

-LRB-

-RRB-

W

Let

us

compare

the

inter-cluster

wiring

costs

of

such

final

configurations

obtained

with

different

values

of

and

Using

a

cost

expression

similar

to

the

one

used

earlier

this

can

be

derived

as

-LRB-

-RRB-

-LRB-

-RRB-

W

It

can

be

shown

that

this

is

minimized

when

g

and

This

matches

the

intuition

of

using

the

smallest

inter

cluster

network

with

the

largest

possible

cluster

size

We

demonstrate

later

that

under

technological

constraints

it

may

not

be

always

possible

to

scale

systems

by

choosing

larger

clusters

In

such

a

case

configurations

similar

to

those

derived

above

will

be

used

to

scale

systems

Unified

Approach

to

Scale

a

given

System

In

this

subsection

we

present

an

unified

framework

to

scale

a

given

base

system

to

a

larger

size

by

integrating

the

approaches

presented

in

Sections

and

We

assume

no

restriction

on

cluster

size

and

channel

width

in

this

analysis

In

the

next

subsection

we

consider

the

problem

of

design

under

realistic

limits

on

cluster

size

and

channel

width

Let

us

consider

the

problem

of

scaling

an

arbitrary

system

-LRB-

with

high

or

low

messaging

overheads

-RRB-

to

a

larger

size

while

maintaining

the

achieved

performance

at

each

processor

If

the

base

system

is

unbalanced

with

high

messaging

overheads

we

scale

the

system

as

suggested

in

Result

by

using

larger

clusters

while

keeping

the

inter-cluster

network

unchanged

This

step

can

be

repeated

until

the

system

becomes

balanced

At

this

point

the

given

network

is

used

most

optimally

Scaling

beyond

this

by

using

larger

clusters

leads

to

a

fall

in

throughput

This

happens

due

to

the

network

becoming

a

bottleneck

Further

scaling

of

such

a

balanced

system

can

be

achieved

as

suggested

in

Result

by

coupling

the

use

of

larger

clusters

with

wider

network

channels

to

maintain

the

network

throughput

Given

a

base

system

which

is

balanced

to

begin

with

-LRB-

low

messaging

overheads

-RRB-

it

can

be

scaled

further

in

a

similar

fashion

as

suggested

in

Result

Let

us

illustrate

the

above

unified

framework

through

an

example

of

scaling

a

base

system

with

configuration

with

processors

to

a

system

with

processors

We

assume

the

parameters

sw

messages

sec

and

nw

;p

messages

sec

Note

that

this

base

system

is

the

same

as

the

one

presented

earlier

in

Fig.

-LRB-

a

-RRB-

The

given

base

system

is

not

balanced

and

the

average

system

throughput

is

min

-LRB-

sw

nw

;p

-RRB-

messages

sec

Similar

to

the

step

in

the

example

in

Fig.

-LRB-

b

-RRB-

we

first

employ

larger

clusters

of

size

two

to

lead

to

a

balanced

system

configuration

of

The

resultant

system

has

processors

and

sw

nw

;p

messages

sec

Note

that

this

system

configuration

also

corresponds

to

the

base

system

used

in

the

example

in

Fig.

In

that

example

it

was

demonstrated

that

scaling

a

configuration

to

processors

while

maintaining

a

throughput

of

messages

sec

can

be

achieved

in

the

most

cost-effective

manner

by

employing

a

configuration

Thus

using

a

combination

of

approaches

suggested

in

Sections

and

we

scale

an

unbalanced

system

to

a

balanced

system

while

maintaining

the

performance

of

messages

sec

With

Limits

on

Maximum

Cluster

Size

and

Channel

Width

The

design

of

a

real

system

is

constrained

by

the

contemporary

technologies

which

dictate

parameters

like

network

channel

cycle

time

processor

speed

etc.

Similarly

packaging

constraints

like

maximum

board

size

pinout

limitations

system

bisection

size

put

bounds

on

the

maximum

channel

width

-LRB-

Wmax

-RRB-

and

maximum

cluster

size

-LRB-

cmax

-RRB-

-LSB-

-RSB-

Design

under

such

constraints

has

been

analyzed

by

us

in

detail

and

presented

in

-LSB-

-RSB-

It

was

shown

that

the

design

process

is

sensitive

to

technological

and

packaging

advancements

In

this

paper

without

going

into

such

details

we

capture

similar

limitations

on

design

by

imposing

maximum

limits

on

the

inter-cluster

channel

width

-LRB-

W

Wmax

-RRB-

and

the

cluster

size

-LRB-

c

cmax

-RRB-

In

this

subsection

we

first

demonstrate

the

impact

of

such

constraints

on

the

scalability

of

a

given

base

system

following

the

guidelines

established

earlier

Alternate

design

configurations

are

then

analyzed

to

overcome

such

scalability

problems

Let

us

consider

the

problem

of

scaling

a

base

system

with

configuration

In

the

last

subsection

we

demonstrated

that

such

a

configuration

with

sw

messages

sec

and

nw

;p

messages

sec

can

be

scaled

to

a

size

of

processors

by

using

a

configuration

Further

scaling

to

a

size

of

processors

can

be

similarly

achieved

through

a

configuration

However

assuming

limits

on

cluster

size

and

channel

width

of

cmax

and

Wmax

respectively

such

a

configuration

is

not

feasible

to

be

build

This

is

because

the

required

channel

width

of

bytes

violates

the

channel

width

constraint

Therefore

to

build

a

processor

system

we

need

to

investigate

other

alternatives

In

Sec.

we

demonstrated

that

allowing

a

combination

of

increasing

cluster

size

and

intercluster

network

size

can

lead

to

a

whole

gamut

of

solutions

to

scale

a

system

Amongst

these

the

solutions

with

larger

cluster

sizes

are

more

cost-effective

However

if

a

solution

with

the

largest

cluster

size

is

not

feasible

we

suggest

using

the

next

feasible

solution

with

the

largest

cluster

size

For

example

let

us

consider

different

alternatives

to

build

a

processor

system

starting

from

the

balanced

system

with

configuration

with

processors

We

need

to

further

scale

the

system

by

a

factor

of

which

can

be

represented

as

a

product

of

two

factors

as

and

Based

on

the

discussion

in

Sec.

let

us

assume

the

first

of

these

factors

as

the

scaling

of

the

cluster

size

and

the

second

as

the

scaling

of

the

inter-cluster

network

The

corresponding

configurations

having

approximately

processors

are

derived

as

and

All

these

configurations

can

provide

a

throughput

of

ss

messages

sec

However

for

Wmax

only

the

last

two

of

these

configurations

are

feasible

Based

on

Result

among

the

feasible

configurations

the

one

with

the

largest

cluster

size

has

lower

cost

This

leads

us

to

conclude

that

the

configuration

is

the

most

cost-effective

feasible

solution

to

build

a

processor

system

Simulation

Experiments

and

Evaluations

To

verify

the

guideline

established

in

earlier

sections

we

performed

accurate

simulation

modeling

on

an

event

driven

simulated

cluster

testbed

developed

in

CSIM

The

testbed

models

dimensionorder

wormhole

routing

and

implements

flit-level

simulation

and

allows

for

demand

multiplexing

of

virtual

channels

on

physical

channels

The

simulator

has

flexibility

for

varying

inter-cluster

network

size

cluster

size

channel

width

message

length

and

timing

parameters

Experiments

were

carried

out

on

clustered

configurations

with

uniform

traffic.Each

simulation

experiment

was

carried

out

for

a

sufficient

number

of

runs

to

ensure

that

the

confidence

interval

was

within

of

the

mean

Impact

of

Message

Processing

Overhead

on

System

Performance

We

studied

the

impact

of

increasing

message

overhead

costs

on

system

performance

Figure

shows

network

latency

versus

throughput

for

three

different

flat

system

configurations

and

The

message

length

was

kept

fixed

at

bits

and

the

channel

width

was

bits

or

bytes

The

message

overhead

cost

was

varied

over

a

wide

range

of

representative

values

of

tsw

and

sec

These

values

of

tsw

correspond

to

sw

and

messages

sec

respectively

We

also

carried

out

experiments

with

tsw

sec

to

determine

nw

;p

supported

by

the

underlying

network

when

messaging

overhead

costs

are

negligible

For

the

configuration

a

value

of

nw

;p

messages

sec

was

observed

Similar

values

of

nw

;p

observed

in

different

system

configurations

are

summarized

in

Table

From

the

results

presented

for

a

system

in

Fig.

it

may

be

observed

that

achievable

throughput

is

determined

by

max

-LRB-

sw

nw

;p

-RRB-

With

higher

message

overheads

of

and

sec

the

performance

in

the

system

is

curtailed

by

the

value

of

sw

resulting

in

the

network

resources

to

remain

highly

underutilized

However

with

low

message

overhead

cost

of

sec

the

performance

was

bounded

by

nw

;p

This

confirms

the

observation

in

Section

Table

Summary

of

Values

of

the

Maximum

Throughput

Supported

by

the

Interconnection

Network

-LRB-

nw

;p

messages

sec

-RRB-

in

Different

System

Configurations

These

Values

were

Derived

from

Simulation

Experiments

Assuming

Messaging

Overhead

Costs

to

be

Negligible

-LRB-

tsw

sec

-RRB-

The

Channel

Width

was

Kept

Fixed

at

W

bytes

and

L

bits

Used

The

Cluster

Size

was

Varied

from

Processors

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

us

us

us

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

us

us

us

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

us

us

us

us

Figure

Impact

of

message

overhead

cost

-LRB-

tsw

-RRB-

on

achieved

system

performance

Latency

vs.

throughput

plots

for

and

flat

configurations

with

different

message

overhead

cost

of

tsw

and

sec

are

shown

The

maximum

throughput

corresponding

to

tsw

sec

in

each

system

shows

the

corresponding

value

of

nw

;p

Impact

of

clustering

to

derive

balanced

configurations

while

maintaining

base

network

Figure

shows

the

impact

of

clustering

to

design

larger

systems

while

maintaining

the

same

base

network

Latency

versus

throughput

results

with

uniform

traffic

are

presented

for

a

and

base

networks

with

channel

width

of

bytes

For

the

network

the

cluster

size

in

the

system

was

varied

from

c

processors

resulting

in

system

size

to

be

varied

from

N

processors

Similarly

on

the

network

the

cluster

size

was

varied

from

c

to

yield

systems

of

size

N

processors

All

experiments

were

performed

with

message

overhead

costs

being

and

sec

respectively

For

a

base

network

with

high

message

overhead

cost

-LRB-

sec

-RRB-

it

was

observed

that

clusters

with

processors

can

be

supported

at

each

node

of

the

network

while

supporting

a

system

throughput

of

messages

sec

Note

that

in

such

a

system

the

throughput

per

processor

is

bounded

by

sw

messages

sec

Thus

even

with

very

large

cluster

sizes

this

maximal

throughput

is

achieved

Further

this

is

done

without

appreciable

degradation

in

average

message

latency

This

indicates

that

a

cluster

size

of

up

to

can

be

easily

supported

on

a

base

network

to

yield

a

system

of

maximum

size

processors

This

confirms

the

discussion

in

section

where

it

was

claimed

that

clusters

of

up

to

nw

;p

sw

processors

can

be

supported

on

a

given

base

network

For

a

base

network

with

W

bytes

-LRB-

-RRB-

the

value

of

nw

;p

ss

messages

sec

This

follows

from

the

discussion

in

the

last

subsection

leading

to

nw

;p

sw

This

implies

that

up

to

processors

can

be

supported

per

cluster

while

maintaining

the

throughput

of

messages

sec

Hence

the

observed

results

with

processors

per

cluster

From

Figure

it

can

be

observed

that

with

tsw

sec

cluster

sizes

up

to

processors

can

be

supported

while

maintaining

the

maximal

achievable

throughput

in

this

case

being

messages

sec

For

very

low

message

overhead

the

supportable

cluster

size

reduces

For

example

it

may

be

observed

that

with

tsw

sec

-LRB-

sw

messages

sec

-RRB-

the

network

can

only

support

clusters

with

or

processor

This

indicates

that

in

such

a

system

the

network

is

not

under-utilized

and

hence

larger

clusters

can

not

be

supported

without

degradation

in

performance

as

shown

in

Figure

for

a

system

with

tsw

In

such

with

low

message

overhead

costs

the

system

throughput

falls

proportionately

to

the

cluster

size

confirming

Observation

Similar

trends

as

above

were

observed

with

the

base

network

From

the

previous

subsection

a

base

network

with

W

bytes

can

support

nw

;p

ss

messages

sec

Thus

for

tsw

and

sec

-LRB-

corresponding

to

sw

and

messages

sec

-RRB-

the

maximal

cluster

sizes

supportable

on

the

respective

systems

are

nw

;p

sw

and

These

are

verified

through

the

experimental

results

presented

for

the

base

network

in

Figure

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

c

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

c

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

c

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

Figure

Determining

the

largest

system

size

that

can

be

supported

without

degradation

in

performance

on

a

given

base

network

for

different

message

overhead

costs

Latency

vs.

throughput

for

a

base

network

with

varying

cluster

size

c

and

network

with

c

are

shown

Three

different

message

overhead

costs

of

tsw

and

sec

were

used

Alternatives

to

Build

a

Given

Sized

System

with

a

fixed

network

channel

width

The

alternatives

to

design

a

larger

system

using

different

base

networks

while

assuming

a

given

fixed

channel

width

technology

were

also

evaluated

For

example

we

considered

designing

a

system

with

processors

as

a

and

The

channel

width

was

kept

fixed

at

bytes

in

all

three

systems

The

objective

was

to

derive

the

alternative

under

a

given

message

overhead

cost

which

offered

the

maximal

achievable

throughput

in

the

most

cost-effective

manner

The

latency

versus

throughput

results

were

derived

with

three

different

message

overheads

of

tsw

and

sec

for

all

three

configurations

These

results

are

presented

in

Fig.

For

systems

with

high

message

overhead

of

tsw

sec

the

achievable

throughput

is

bounded

by

sw

in

all

three

configurations

This

is

because

the

corresponding

values

of

nw

;p

in

the

three

systems

as

derived

in

Table

and

messages

sec

respectively

are

greater

than

sw

From

Fig.

it

can

be

also

observed

that

the

configuration

offers

similar

throughput

with

lowest

latency

while

also

providing

the

most

wiring

cost-effective

inter-cluster

network

This

leads

us

to

conclude

that

with

large

cluster

size

is

the

optimal

configuration

to

build

a

processor

system

with

high

message

overhead

cost

For

a

lower

message

start

up

cost

of

tsw

sec

-LRB-

sw

-RRB-

the

and

configurations

with

nw

;p

-RRB-

demonstrate

potential

to

offer

higher

performance

than

the

configuration

with

nw

;p

-RRB-

This

is

verified

in

Fig.

In

this

case

the

configuration

is

derived

as

the

optimal

configuration

With

very

low

message

overheads

of

tsw

sec

-LRB-

sw

-RRB-

the

value

of

nw

;p

sw

is

observed

in

all

three

configurations

Thus

the

configuration

offers

the

maximum

throughput

as

verified

in

Fig.

leading

us

to

conclude

that

at

very

low

message

overhead

cost

large

cluster

sizes

do

not

lead

to

optimal

systems

In

general

for

a

given

channel

width

technology

the

cluster

size

to

be

used

to

build

a

system

of

given

size

is

sensitive

to

the

value

of

the

message

overhead

cost

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Figure

Deriving

the

optimal

cluster

size

to

design

a

system

with

processors

for

three

different

message

overhead

costs

tsw

and

sec

The

latency

versus

throughput

plots

are

shown

for

each

combination

of

system

configuration

and

message

cost

The

channel

width

was

W

bytes

in

all

systems

Larger

cluster

sizes

are

optimal

when

message

overhead

costs

are

higher

Scaling

Systems

with

Low

Message

Overhead

and

Wider

Channel

Technology

Next

we

considered

scaling

systems

with

very

low

messaging

overhead

costs

In

such

systems

the

achievable

performance

is

determined

by

the

interconnection

network

As

the

system

size

is

in

creased

the

network

becomes

more

of

a

bottleneck

leading

to

lower

values

of

nw

;p

and

hence

poor

scalability

We

studied

the

scaling

of

such

systems

by

using

larger

clusters

while

proportionately

using

wider

network

channels

Figure

shows

the

latency

versus

throughput

results

with

uniform

traffic

and

tsw

sec

while

scaling

two

base

systems

and

to

larger

configurations

respectively

It

was

observed

that

as

the

system

size

is

made

larger

the

performance

is

maintained

resulting

in

ideal

scaling

This

verifies

Result

In

some

cases

specially

in

the

configurations

an

improvement

in

performance

was

observed

as

system

size

was

increased

This

can

be

explained

on

the

basis

of

message

length

in

terms

of

flits

It

becomes

shorter

as

the

channel

width

in

the

network

is

increased

This

leads

to

lower

contention

with

wormhole

routing

and

hence

higher

throughput

and

lower

message

latency

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Figure

Impact

on

system

performance

as

a

base

system

is

scaled

to

larger

sizes

by

using

larger

clusters

and

proportionately

larger

channel

widths

Latency

versus

throughput

with

uniform

traffic

while

scaling

two

base

systems

and

are

shown

The

problem

of

designing

a

processor

system

was

reexamined

assuming

the

scope

to

use

wider

channels

Figure

shows

the

latency

versus

throughput

results

with

uniform

traffic

and

tsw

sec

for

four

different

configurations

to

design

an

approximately

processor

system

and

We

observed

that

the

configuration

with

largest

cluster

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Figure

Deriving

the

optimal

cluster

size

to

design

a

system

with

processors

with

low

messaging

overhead

costs

tsw

sec

The

latency

versus

throughput

plots

are

shown

for

three

different

system

configuration

Configurations

using

larger

clusters

and

smaller

network

can

use

wider

channels

while

still

offering

lower

interconnect

cost

Such

configurations

are

derived

as

optimal

size

and

channel

width

has

potential

to

offer

best

throughput

with

lowest

latency

Configurations

with

smaller

cluster

size

and

channel

width

offer

lower

throughput

with

higher

latency

This

demonstrates

that

cluster

and

channel

width

technologies

allowing

system

configurations

designed

using

larger

clusters

with

wider

channels

offers

higher

performance

With

constraints

on

the

maximum

channel

width

and

cluster

size

the

configuration

using

the

largest

cluster

size

should

be

chosen

For

example

with

maximum

channel

width

being

Wmax

bytes

the

optimal

configuration

is

the

Conclusion

In

this

paper

we

demonstrated

a

new

approach

to

scale

parallel

systems

by

taking

into

account

recent

advancements

in

interconnection

and

processor-cluster

technologies

The

impact

of

message

overhead

cost

on

scaling

system

size

has

been

analyzed

Our

results

indicate

that

systems

with

high

messaging

overheads

can

be

scaled

in

a

cost-effective

manner

by

employing

larger

clusters

as

computing

nodes

This

can

be

achieved

without

changing

the

network

until

the

messaging

costs

are

balanced

A

system

with

balanced

messaging

costs

can

be

further

scaled

by

using

larger

clusters

and

wider

network

channel

As

processor

clustering

is

beginning

to

be

used

in

some

parallel

systems

like

Cray

and

Intel

Paragon

the

results

derived

in

this

paper

provide

a

strong

foundation

for

architects

to

analyze

the

benefits

of

processor

clustering

This

shows

when

and

how

a

given

system

can

take

advantage

of

processor

clustering

to

build

a

larger

and

cost-effective

system

while

utilizing

the

communication

bandwidth

effectively

In

this

research

we

have

emphasized

on

starbased

cluster

organization

In

the

next

phase

we

are

extending

the

analysis

to

other

cluster

organizations

like

bus

MIN

and

direct

networks

References

-LSB-

-RSB-

A.

Agrawal

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

Oct

-LSB-

-RSB-

A.

Asthana

H.

Jagdish

and

B.

Mathews

Impact

of

Advanced

VLSI

packaging

on

the

design

of

a

large

parallel

computer

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

S.

Balakrishnan

and

D.

K.

Panda

Impact

of

Multiple

Consumption

Channels

on

Wormhole

Routed

k-ary

n-cube

Networks

In

Proc

of

the

Int

Parallel

Processing

Symp.

pages

-LCB-

April

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Scalable

Architectures

with

k-ary

n-cube

cluster-c

organization

In

Proc

of

the

Symposium

of

Parallel

and

Distributed

Processing

pages

-LCB-

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Large

Hierarchical

Multiprocessor

Systems

under

Pro

cessor

Interconnection

and

Packaging

Advancements

In

Proc

of

the

Int

l

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

Y.

M.

Boura

and

C.

R.

Das

Modeling

virtual

channel

flow

control

in

hypercubes

In

Proc

of

the

High-Performance

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

Carlson

The

Mesh

with

a

Global

Mesh

a

Flexible

High-speed

Organization

for

Parallel

Computation

In

Proc

of

the

Int

Conf

on

Supercomputer

Systems

IEEE

Comp

Soc

Press

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

D.

J.

Kuck

et

al.

The

Cedar

System

and

an

Initial

Performance

Study

In

Proc

of

the

Int

l

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

W.

J.

Dally

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Trans

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

W.

J.

Dally

Virtual-channel

flow

control

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

S.

Dandamudi

and

D.

Eager

Hierarchical

Interconnection

Networks

for

Multicomputer

Systems

IEEE

Trans

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Performance

of

Hierarchical

Systems

with

wiring

constraints

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

L.

M.

Ni

and

P.

K.

McKinley

A

survey

of

wormhole

routing

techniques

in

direct

networks

IEEE

Computer

-LCB-

Feb

-LSB-

-RSB-

K.

Padmanabhan

Effective

Architectures

for

Data

Access

in

a

Shared

Memory

Hierarchy

Jour

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

D.

K.

Panda

and

D.

Basak

Issues

in

Designing

Scalable

Systems

with

k-ary

n-cube

clusterc

Organization

In

Proc

of

the

First

International

Workshop

on

Parallel

Processing

India

pages

-LCB-

-LSB-

-RSB-

D.

A.

Patterson

Observations

in

Massive

Parallelism

Trends

and

Predictions

for

to

Technical

Report

DIMACS

Sept

-LSB-

-RSB-

M.

T.

Raghunath

and

A.

Ranade

Designing

interconnection

networks

for

multi-level

packaging

In

Proc

of

the

Supercomputing

pages

-LCB-

interconnection networks
messaging overheads
scalability

clustered architectures
hierarchical architectures
parallel architectures