Alleviating

Consumption

Channel

Bottleneck

in

Wormhole-Routed

k-ary

n-cube

Systems

Debashis

Basak

and

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

fbasak,pandag@cis.ohio-state.edu

Abstract

This

paper

identifies

performance

degradation

in

wormhole

routed

k-ary

n-cube

networks

due

to

limited

number

of

router-to-processor

consumption

channels

at

each

node

Many

recent

research

in

wormhole

routing

have

advocated

the

advantages

of

adaptive

routing

and

virtual

channel

flow

control

schemes

to

deliver

better

network

performance

This

paper

indicates

that

the

advantages

associated

with

these

schemes

can

not

be

realized

with

limited

consumption

capacity

To

alleviate

such

performance

bottleneck

a

new

network

interface

design

using

multiple

consumption

channels

is

proposed

To

match

virtual

multiplexing

on

network

channels

we

also

propose

each

consumption

channel

to

support

multiple

virtual

consumption

channels

The

impact

of

message

arrival

rate

at

a

node

on

the

required

number

of

consumption

channel

is

studied

analytically

It

is

shown

that

wormhole

networks

with

higher

routing

adaptivity

dimensionality

degree

of

hot-spot

traffic

and

number

of

virtual

lanes

have

to

take

advantage

of

multiple

consumption

channels

to

deliver

better

performance

The

interplay

between

system

topology

routing

algorithm

number

of

virtual

lanes

messaging

overheads

and

communication

traffic

is

studied

through

simulation

to

derive

the

effective

number

of

consumption

channels

required

in

a

system

Using

the

on-going

technological

trend

it

is

shown

that

wormhole-routed

systems

can

use

up

to

consumption

channels

per

node

to

deliver

better

system

performance

research

is

supported

in

part

by

NSF

Grant

Faculty

Early

Career

Development

Award

MIP

and

an

Ohio

State

University

Presidential

Fellowship

A

preliminary

version

of

this

paper

-LSB-

-RSB-

has

been

presented

in

International

Parallel

Processing

Symposium

Contents

Introduction

Designing

efficient

consumption

channel

interface

Typical

interface

with

single

consumption

channel

Consumption

collision

and

domino-blocking

effect

Design

with

multiple

consumption

channels

Supporting

virtual

consumption

channels

Number

of

physical

consumption

channels

supportable

by

router

pinout

Analytical

model

for

estimating

consumption

collision

delay

Estimating

probability

of

consumption

collision

Estimating

collision

delay

Parameters

affecting

collision

delay

and

benefits

from

more

consumption

channels

Applying

the

model

to

determine

required

number

of

physical

consumption

channels

Impact

of

routing

balanced

versus

unbalanced

rates

Impact

of

message

length

Impact

of

network

dimensionality

Constraints

affecting

maximum

message

arrival

rate

and

number

of

physical

consumption

channels

Network

bisection

Non-uniform

traffic

Adaptivity

and

virtual

multiplexing

on

network

channels

Messaging

overheads

Interface

to

processor

bandwidth

Summary

Simulation

experiments

and

results

Simulation

Set-up

Impact

of

routing

adaptivity

Impact

of

message

length

Impact

of

system

dimensionality

Impact

of

system

size

with

uniform

traffic

Impact

of

more

virtual

channels

Impact

of

messaging

overheads

Impact

of

I-to-P

bandwidth

All-to-one

traffic

Uniform

and

Non-uniform

traffic

Summary

and

conclusion

Introduction

The

wormhole-routing

switching

technique

often

referred

to

as

wormhole

routing

is

becoming

the

trend

in

building

parallel

systems

due

to

its

inherent

advantages

like

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

Cray

-LSB-

-RSB-

Ncube

J

Machine

-LSB-

-RSB-

and

Stanford

DASH/FLASH

-LSB-

-RSB-

are

representative

systems

falling

into

this

category

In

wormhole-switching

technique

the

flow

of

header

flit

triggers

the

movement

of

the

remaining

flits

in

a

message

If

the

header

flit

gets

blocked

due

to

unavailability

of

a

network

channel

the

remaining

flits

are

blocked

in

the

network

Such

blocking

besides

leading

to

under-utilization

of

channels

may

block

other

messages

in

the

network

Though

the

wormhole-routing

was

initially

proposed

with

deterministic

routing

-LSB-

-RSB-

a

number

of

adaptive

routing

schemes

have

been

proposed

recently

by

many

researchers

-LSB-

-RSB-

to

reduce

such

message

blocking

in

the

network

Similarly

the

use

of

multiple

virtual

networks

-LRB-

or

lanes

-RRB-

with

Virtual

Channel

Flow

Control

-LRB-

VCFC

-RRB-

has

been

proposed

-LSB-

-RSB-

in

literature

to

be

used

with

either

deterministic

or

adaptive

routing

These

schemes

emphasize

reducing

message

blocking

or

contention

in

the

network

However

once

a

worm

reaches

a

destination

node

it

needs

to

be

consumed

-LRB-

taken

out

from

network

-RRB-

-LSB-

-RSB-

Thus

after

the

header

flit

of

a

worm

reaches

its

destination

the

movement

of

its

subsequent

flits

in

the

network

depends

on

the

rate

at

which

the

message

gets

consumed

This

movement

depends

on

the

consumption

capacity

available

at

the

destination

node-network

interface

which

can

be

characterized

by

number

of

consumption

channels

for

message

consumption

If

the

worm

gets

blocked

or

slowed

down

due

to

limited

consumption

channels

then

the

subsequent

flits

of

the

worm

remain

in

the

network

for

additional

time

leading

to

wastage

of

network

bandwidth

In

-LSB-

-RSB-

it

was

first

reported

that

wormhole

systems

can

undergo

severe

performance

degradation

due

to

limited

number

of

router-to-processor

consumption

channels

The

results

were

based

on

experimentation

on

the

deterministic

wormhole

routed

system

and

suggested

using

multiple

consumption

channels

in

future

wormhole

systems

Thus

a

good

network

design

must

consider

its

routing

scheme

together

with

the

consumption

capacity

available

at

a

node

to

evaluate

its

overall

performance

Most

contemporary

systems

provide

only

one

consumption

channel

per

processor

Under

traffic

conditions

when

multiple

messages

are

destined

to

a

node

within

a

short

duration

a

single

consumption

channel

can

become

a

bottleneck

For

example

many

parallel

applications

exhibit

non-uniform

traffic

with

hot-spots

-LSB-

-RSB-

both

in

distributed-memory

and

distributed-sharedmemory

paradigms

Such

traffic

leads

to

more

messages

being

destined

towards

some

nodes

in

the

system

Even

with

uniform

traffic

at

higher

rate

of

communication

there

is

more

probability

that

some

nodes

receive

multiple

messages

during

a

small

interval

of

time

For

both

types

of

traffic

if

multiple

messages

reaching

a

destination

node

can

not

be

consumed

at

full

speed

due

to

limited

consumption

capacity

they

get

blocked

in

the

network

surrounding

the

destination

node

Such

blocking

can

lead

to

domino-blocking

-LRB-

tree-saturation

-RRB-

-LSB-

-RSB-

in

the

network

which

may

affect

the

propagation

of

other

messages

in

the

system

This

has

potential

to

degrade

performance

in

terms

of

lower

throughput

and

higher

latency

Adaptive

routing

and

VCFC

schemes

aim

to

achieve

higher

network

performance

by

alleviating

message

blocking

in

the

network

However

with

a

closer

look

it

can

be

observed

that

these

schemes

drive

more

messages

to

nodes

leading

to

increasing

demand

on

the

consumption

capacity

at

a

node

In

the

absence

of

sufficient

consumption

capacity

at

a

node

the

advantages

of

these

schemes

become

somewhat

limited

Over

the

years

multiple

virtual

channels

per

network

channel

has

been

proposed

to

provide

deadlock-freedom

-LSB-

-RSB-

routing

adaptivity

-LSB-

-RSB-

and

VCFC

-LSB-

-RSB-

Contemporary

systems

are

adopting

such

multiple

virtual

channels/lanes

on

network

channels

-LSB-

-RSB-

In

such

systems

a

consumption

channel

is

required

to

support

virtual

multiplexing

Otherwise

virtual

multiplexing

on

the

network

channels

causing

slowdown

in

propagation

of

messages

can

lead

to

messages

arriving

at

their

destinations

at

a

speed

slower

than

the

full

network

channel

speed

This

can

lead

to

situations

where

a

message

having

a

consumption

channel

is

consumed

slowly

-LRB-

not

utilizing

the

full

consumption

channel

bandwidth

-RRB-

while

another

message

is

blocked

on

the

consumption

channel

Thus

a

good

system

design

should

incorporate

virtual

consumption

channels

to

take

advantages

of

the

benefits

of

adaptivity

and

VCFC

schemes

The

above

discussion

suggests

that

an

architect

must

provide

sufficient

consumption

capacity

as

well

as

virtual

consumption

channels

to

alleviate

consumption

channel

bottleneck

problem

and

obtain

better

performance

However

there

is

no

systematic

study

of

this

problem

in

the

literature

to

provide

necessary

design

guidelines

to

an

architect

In

this

paper

we

take

on

such

challenges

and

propose

a

new

network

interface

design

with

multiple

consumption

channels

from

a

router

to

its

processor

To

avoid

wastage

of

consumption

channel

bandwidth

due

to

message

slowdown

by

virtual

multiplexing

in

the

network

we

propose

using

matching

number

of

virtual

lanes

on

each

consumption

channel

A

message

arriving

at

a

destination

node

chooses

a

free

consumption

channel

gets

consumed

and

buffered

at

the

interface

A

direct

memory

access

-LRB-

DMA

-RRB-

then

transfers

the

message

to

the

processor-memory

over

the

local

bus

In

designing

such

a

network

interface

the

main

question

needs

to

be

solved

is

How

many

consumption

channels

should

we

provide

in

a

given

system

Clearly

the

blocking

effect

on

a

consumption

channel

-LRB-

defined

by

us

as

consumption

collision

-RRB-

reduces

with

increasing

number

of

channels

As

a

naive

solution

one

can

provide

physical

consumption

channels

per

node

each

having

v

virtual

where

n

network

dimensionality

and

v

number

of

virtual

lanes

per

network

channel

In

such

a

system

at

most

messages

can

arrive

simultaneously

at

a

node

and

terms

virtual

lane

and

virtual

channel

are

used

interchangeably

in

this

paper

thus

having

a

total

of

virtual

consumption

channels

can

handle

this

worst

case

However

this

naive

solution

requiring

physical

channels

is

an

overkill

and

may

not

be

realizable

due

to

limited

router

pinout

constraint

Thus

the

challenge

is

to

determine

how

many

physical

consumption

channels

are

effective

as

well

as

realizable

to

deliver

best

performance

In

order

to

derive

this

effective

number

in

this

paper

we

consider

several

factors

like

network

bisection

messaging

overheads

interface-to-processor

bandwidth

system

dimensionality

routing

adaptivity

and

virtual

networks

in

an

integrated

manner

These

factors

determine

the

rate

of

messages

arriving

at

a

node

First

we

develop

an

analytical

consumption

collision

model

to

estimate

consumption

collision

delay

based

on

the

message

arrival

rates

Using

this

model

we

analyze

the

interplay

between

the

above

parameters

and

the

number

of

consumption

channels

Using

this

analysis

we

derive

guidelines

and

trends

for

using

effective

number

of

consumption

channels

under

different

values

of

the

above

factors

Finally

through

detailed

flit-level

simulation

we

verify

these

guidelines

for

a

range

of

system

sizes

system

dimensionalities

routing

adaptivity

messaging

overheads

interface-to-processor

bandwidths

and

number

of

virtual

networks

The

overall

study

indicates

that

consumption

channels

per

node

need

to

be

used

on

current

and

future

generation

wormhole

k-ary

n-cube

systems

to

alleviate

the

consumption

channel

bottleneck

It

is

also

demonstrated

that

these

consumption

channels

are

realizable

under

current

and

future

router

pinout

constraints

The

paper

is

organized

as

follows

Section

introduces

the

new

network

interface

design

with

more

consumption

channels

and

virtual

multiplexing

on

consumption

channels

to

match

network

channel

multiplexing

Section

characterizes

various

aspects

of

the

consumption

bottleneck

and

presents

a

contention

model

for

analyzing

the

benefits

from

more

consumption

channels

The

impacts

of

routing

algorithm

virtual

channels

topology

messaging

overheads

and

communication

traffic

on

demanded

number

of

consumption

channels

are

analyzed

in

Sections

and

Section

presents

simulation

experiments

and

results

to

derive

the

effective

number

of

consumption

channels

under

different

system

parameters

Concluding

remarks

are

presented

in

Section

Designing

efficient

consumption

channel

interface

Multicomputers

with

direct

networks

supporting

wormhole

switching

have

become

a

popular

trend

in

building

large

parallel

systems

Most

commonly

used

direct

network

topologies

are

meshes/tori

belonging

to

the

class

of

generalized

k-ary

n-cube

interconnections

-LSB-

-RSB-

These

are

n-dimensional

grid

structures

with

kn

nodes

along

dimensions

n

respectively

The

values

of

kn

may

be

same

or

different

Figure

shows

a

two

dimensional

mesh

-LRB-

a

-RRB-

system

Each

node

in

the

system

comprises

of

a

router

and

a

processor

The

routers

are

interconnected

as

a

mesh

Each

processor

is

connected

to

its

router

so

that

it

can

inject

messages

to

other

nodes

and

consume

messages

destined

to

it

In

this

study

we

assume

the

interprocessor

messages

to

traverse

across

the

interconnection

network

using

wormhole-routing

-LSB-

-RSB-

In

such

routing

a

message

is

transferred

as

a

train

of

flow

control

digits

-LRB-

flits

-RRB-

A

flit

is

typically

the

size

of

the

network

channel

width

Only

the

header

flit

has

routing

information

Each

router

only

dedicates

a

few

flit

buffers

for

a

message

With

the

header

getting

blocked

due

to

contention

the

remaining

flits

in

the

network

are

blocked

in

place

Many

current

systems

built

in

industry

and

academia

like

the

Cray

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

and

Stanford

FLASH

-LSB-

-RSB-

employ

such

wormhole-switched

mesh/torus

interconnection

networks

with

organization

consumption

channel

injection

channel

physical

channels

-LRB-

with

v

virtual

into

router

from

router

ROUTER

Processor

CacheMemory

local

bus

fromprocessor

processor

router

channels

demand

multiplexed

on

each

-RRB-

DMA

I/O

engineBusinterface

a

messagebeing

consumed

a

messagewaiting

to

beconsumedmessagebuffers

a

continuingmessage

Figure

A

two-dimensional

mesh

of

size

-LRB-

-RRB-

A

typical

router-processor

interface

is

illustrated

for

a

k-ary

n-cube

system

with

one

consumption

channel

per

node

Typical

interface

with

single

consumption

channel

Let

us

analyze

the

typical

router

processor

interface

as

shown

in

Fig

A

router

can

receive

incoming

messages

in

either

direction

on

any

of

the

n

dimensions

leading

to

incoming

physical

channels

into

the

router

Similarly

a

message

may

need

to

be

routed

on

any

of

the

outgoing

physical

channels

from

the

router

Depending

on

the

routing

scheme

and

virtual

channel

flow

control

used

a

set

of

v

virtual

channels

may

be

demand-multiplexed

on

each

physical

channel

Besides

these

network

channels

a

single

consumption

channel

is

used

for

the

local

processor

to

consume

messages

destined

to

it

from

the

network

Similarly

a

single

injection

channel

is

used

for

the

local

processor

to

inject

messages

into

the

network

With

a

single

consumption

channel

a

processor

is

constrained

to

consuming

only

one

message

at

a

time

For

example

it

can

be

observed

in

Fig.

that

while

a

message

-LRB-

shown

in

gray

-RRB-

is

being

consumed

another

message

-LRB-

shown

in

black

-RRB-

destined

to

be

also

consumed

at

the

same

node

is

forced

to

wait

Let

us

consider

the

process

of

message

consumption

in

more

detail

A

message

being

consumed

is

removed

from

the

network

along

a

consumption

channel

stored

temporarily

at

the

interface

in

a

message

buffer

and

then

transferred

over

a

local

bus

-LSB-

-RSB-

to

the

processor-memory

In

the

past

programmed

I/O

was

used

-LSB-

-RSB-

to

transfer

bytes

of

a

message

from

the

interface

to

the

processor-memory

Clearly

this

leads

to

low

communication

bandwidth

with

the

processor

becoming

the

bottleneck

It

also

results

in

lack

of

overlap

between

computation

and

communication

In

recent

systems

DMA-based

message

consumption/injection

-RRB-

is

used

-LSB-

-RSB-

for

transferring

messages

between

the

interface

and

processor-memory

As

shown

in

Fig.

the

local

processor-memory

bus

bandwidth

is

shared

between

DMA

operations

to

write

consumed

messages

messages

to

be

injected

memory

read/write

operations

due

to

processor-cache

subsystem

and

other

I/O

Thus

only

a

fraction

of

the

bus

bandwidth

may

be

available

to

DMA

for

writing

consumed

messages

from

interface

to

processor-memory

-LRB-

I-to-P

-RRB-

Let

Bmem

denote

such

available

I-to-P

bandwidth

The

buffering

of

messages

at

the

interface

helps

to

isolate

the

network

from

the

vagaries

of

the

processor-memory

bus

and

vice-versa

With

the

processor-memory

bus

being

busy

with

other

operations

messages

can

still

be

removed

from

the

network

and

buffered

at

the

interface

The

number

of

message

buffers

at

the

interface

depends

on

the

message

arrival

rate

message

length

and

the

available

I-to-P

bandwidth

To

have

smaller

buffers

a

limit

is

typically

put

on

the

message

transfer

size

in

the

network

Larger

messages

are

divided

into

such

smaller

messages

packets

before

being

transferred

over

the

network

For

example

on

the

and

IBM

such

transfer

size

on

the

network

is

approximately

and

bytes

-LSB-

-RSB-

respectively

Earlier

research

-LSB-

-RSB-

has

demonstrated

that

providing

a

small

amount

of

memory

corresponding

to

a

few

message

packet

buffers

is

sufficient

to

approximate

infinite

buffering

performance

In

the

eventuality

that

all

message

packet

buffers

are

full

further

consumption

is

stalled

and

incoming

messages

forced

to

block

in

the

network

From

here

on

in

this

paper

we

refer

to

a

message

packet

as

simply

a

message

Consumption

collision

and

domino-blocking

effect

A

single

consumption

channel

can

be

a

bottleneck

to

system

performance

We

illustrate

such

bottleneck

through

an

example

and

demonstrate

the

alleviating

impact

of

multiple

consumption

channels

on

performance

The

following

definitions

are

useful

in

the

following

discussion

Definition

A

message

whose

header

flit

has

reached

its

destination

node

is

defined

as

a

terminating

message

Definition

An

event

when

a

terminating

message

gets

blocked

in

the

network

due

to

unavailability

of

consumption

channels

is

defined

as

a

consumption

collision

Definition

The

increase

in

latency

of

a

terminating

message

due

to

consumption

collision

is

defined

as

consumption

delay

Figure

shows

an

example

of

two

terminating

messages

and

arriving

simultaneously

at

destination

node

in

a

mesh

along

the

y

and

y

links

Let

node

have

exactly

one

consumption

channel

A

single

consumption

channel

forces

these

two

messages

to

be

consumed

one

at

a

time

Let

the

consumption

channel

service

first

with

a

service

time

denoted

by

Tc

Thus

encounters

a

consumption

collision

and

a

consumption

delay

of

Tc

Wormhole

routing

forces

the

flits

of

to

be

held

in

place

while

holding

onto

network

channels

As

secondary

impact

a

third

message

destined

to

a

different

node

gets

blocked

on

a

network

channel

held

by

Although

not

shown

in

the

figure

we

can

consider

a

fourth

message

being

blocked

on

and

so

on

We

refer

to

such

an

event

as

a

domino-blocking

effect

Such

blocking

leads

to

higher

average

message

latency

and

lower

message

throughput

due

to

wastage

of

network

resources

The

probability

of

domino-blocking

effect

increases

with

network

load

Having

more

consumption

channels

-LRB-

two

for

this

example

-RRB-

can

reduce

-LRB-

or

eliminate

-RRB-

consumption

delay

leading

to

alleviating

of

the

dominoblocking

effect

This

leads

us

to

the

following

observation

Observation

With

more

consumption

channels

the

possibility

of

a

consumption

collision

is

reduced

This

has

two

impacts

a

direct

impact

in

lowering

the

latency

of

the

message

being

consumed

and

an

indirect

impact

on

the

gain

in

overall

network

performance

-LRB-

throughput

-RRB-

by

alleviating

the

wasteful

effects

of

domino-blocking

xy

Figure

An

example

of

consumption

collision

in

a

mesh

with

one

consumption

channel

under

dimension-order

-LRB-

x-y

-RRB-

routing

Two

messages

and

are

destined

towards

the

node

A

third

message

destined

to

is

blocked

on

Design

with

multiple

consumption

channels

Based

on

the

above

observation

in

this

paper

we

propose

supporting

multiple

consumption

channels

at

a

node

to

alleviate

the

consumption

bottleneck

Let

C

denote

the

number

of

consumption

channels

-LRB-

also

referred

to

as

physical

consumption

channels

-RRB-

being

supported

at

a

node

Figure

shows

a

router-processor

interface

supporting

multiple

physical

consumption

channels

per

node

consumptionchannels

into

router

from

router

ROUTER

both

messages

can

be

consumed

at

the

same

time

Figure

A

router-processor

interface

supporting

multiple

consumption

channels

per

node

As

discussed

earlier

a

naive

solution

to

alleviate

the

consumption

bottleneck

is

to

provide

physical

consumption

channels

in

an

n-dimensional

system

to

handle

the

worst

case

message

arrival

However

this

naive

solution

may

be

an

overkill

and

may

not

be

realizable

due

to

limited

router

pinout

constraint

as

discussed

in

more

detail

in

Sec.

In

most

cases

a

much

lesser

number

may

be

sufficient

to

deliver

best

system

performance

The

required

number

of

consumption

channels

depends

on

the

complex

interplay

of

various

factors

including

the

traffic

pattern

routing

adaptivity

interconnection

dimensionality

network

bisection

size

messaging

overheads

and

degree

of

virtual

channel

multiplexing

on

network

channels

Our

objective

is

to

analyze

this

interplay

and

derive

the

minimum

number

of

required

consumption

channels

for

best

performance

The

following

questions

are

addressed

in

this

paper

ffl

When

does

having

more

than

one

physical

consumption

channel

lead

to

better

performance

ffl

What

are

the

factors

contributing

towards

more

demand

for

consumption

channels

ffl

What

is

the

effective

number

of

consumption

channels

beyond

which

returned

increase

in

gained

performance

does

not

merit

more

channels

The

above

issues

are

discussed

in

more

detail

in

the

following

sections

In

this

paper

we

focus

on

the

design

of

consumption

channel

interface

It

is

to

be

noted

that

a

message

in

a

wormhole

network

may

also

get

blocked

at

its

source

node

due

to

limited

number

of

injection

channels

-LSB-

-RSB-

Under

high

network

load

a

message

may

hold

onto

an

injection

channel

longer

because

of

its

slower

propagation

due

to

network

contention

In

such

a

situation

it

seems

that

having

additional

injection

channels

may

help

to

inject

waiting

messages

out

of

the

node

However

this

may

be

of

limited

use

because

an

injected

message

can

get

blocked

on

a

busy

network

channel

Thus

providing

more

injection

capacity

may

not

lead

to

better

system

performance

This

indicates

that

the

injection

channel

bottleneck

is

less

critical

than

the

consumption

channel

bottleneck

However

to

be

fair

to

the

injection

side

and

to

derive

balanced

design

guidelines

we

assume

the

number

of

injection

channels

at

a

node

being

equal

to

the

number

of

consumption

channels

Supporting

virtual

consumption

channels

To

avoid

the

wasteful

effects

of

consumption

channel

bandwidth

caused

by

message

slowdown

due

to

virtual

multiplexing

in

the

network

a

good

design

must

provide

matching

number

of

virtual

consumption

channels

on

each

physical

consumption

channel

Let

us

consider

an

enhanced

interface

design

to

support

multiple

physical

consumption

channels

each

supporting

virtual

channels

to

match

degree

of

network

channel

multiplexing

Design

implementation

of

virtual

channels

has

been

studied

in

detail

in

-LSB-

-RSB-

Assuming

routers

already

supporting

v

virtual

channels

on

each

network

channel

it

is

straight-forward

to

extend

such

design

to

also

support

v

virtual

consumption

channels

on

each

physical

consumption

channel

as

shown

in

Fig.

Each

virtual

consumption

channel

has

message

buffers

associated

with

Other

network/consumption

physical

channels

Flit

buffers

Router

Consumption

Interface

Message

buffers

consumptionchannelphysical

fromswitch

Other

physical

consumption

channels

toDMA

Figure

Supporting

multiple

virtual

consumption

channels

on

a

physical

consumption

channel

On

the

router

side

the

virtual

channel

implementation

on

a

router

network

channel

is

extended

to

a

consumption

channel

On

the

interface

side

each

virtual

consumption

channel

has

message

buffers

associated

with

it

it

These

serve

the

same

purpose

of

intermediate

buffering

as

the

message

buffers

shown

earlier

in

Fig.

With

multiple

virtual

channels

per

physical

channel

messages

along

a

virtual

channel

come

at

a

slower

rate

Thus

the

buffering

required

per

virtual

channel

need

not

be

very

large

Based

on

preliminary

experimentation

in

this

study

we

associate

only

two

buffers

per

virtual

consumption

channel

While

one

buffer

is

full

and

waiting

to

be

written

to

processor-memory

by

DMA

the

other

buffer

is

used

to

gather

the

next

message

along

the

virtual

channel

Experimentation

with

higher

number

of

message

buffers

yielded

similar

performance

Number

of

physical

consumption

channels

supportable

by

router

pinout

Current

technology

supports

up

to

to

pins

-LSB-

-RSB-

per

router

The

pinout

restricts

the

number

of

channels

and

channel

width

that

can

be

supported

from

a

router

-LSB-

-RSB-

Although

a

very

high

number

of

consumption

channels

is

clearly

not

possible

let

us

derive

an

estimate

on

reasonable

number

of

consumption

channels

supportable

by

contemporary

router

pinout

technology

Let

R

denote

the

required

router

pinout

in

a

k-ary

n-cube

system

with

W

bytes

wide

data

channels

In

a

system

with

bidirectional

channels

the

number

of

network

channels

is

For

supporting

C

consumption

channels

per

router

and

assuming

equal

number

of

injection

channels

the

total

number

of

channels

to

be

supported

per

router

is

derived

as

C

As

discussed

earlier

let

v

be

the

degree

of

virtual

multiplexing

in

the

system

The

number

of

control

bits

required

per

physical

channel

for

identifying

the

virtual

channel

being

send

or

acknowledged

is

v

bits

-LSB-

-RSB-

For

a

system

with

W

bytes

-LRB-

bits

-RRB-

and

v

bits

are

required

per

channel

leading

to

R

-LRB-

-RRB-

Thus

a

router

pinout

of

can

support

up

to

C

in

a

network

-LRB-

n

-RRB-

-LRB-

-RRB-

For

network

thinner

data

channels

or

fewer

virtual

channels

a

higher

C

can

be

supported

This

leads

us

to

conclude

that

it

is

reasonable

to

expect

contemporary

router

pinout

technology

to

support

up

to

physical

consumption

channel

per

router

In

the

following

sections

we

analyze

various

factors

affecting

the

effective

number

of

consumption

channels

required

for

alleviating

the

bottleneck

Analytical

model

for

estimating

consumption

collision

delay

In

this

section

we

present

a

contention

model

to

estimate

the

probability

of

consumption

collision

and

consumption

delay

at

a

node

for

a

given

number

of

incoming

network

channels

message

rates

on

these

channels

and

number

of

physical

consumption

channels

Our

model

is

based

on

an

analysis

by

Dally

for

estimating

wormhole

contention

delay

in

-LSB-

-RSB-

To

keep

the

model

simple

we

do

not

consider

the

impact

of

virtual

channels

here

Such

impact

is

discussed

in

a

later

section

Trends

based

on

the

model

are

validated

with

accurate

simulation

modeling

in

Sec.

Let

us

consider

a

node

with

m

incoming

network

channels

into

a

router

with

a

-RRB-

one

consumption

channel

and

b

-RRB-

multiple

consumptions

channels

as

shown

in

Fig.

For

convenience

the

following

is

a

summary

of

notation

used

in

this

analysis

network

channelconsumption

channel

a

-RRB-

Single

consumption

channel

b

-RRB-

Multiple

-LRB-

C

-RRB-

consumption

channelsC

Tc

router

TcTc

Tc

service

time

of

a

message

li

terminating

message

rate

on

channel

i

Figure

A

simplified

model

for

estimating

probability

of

consumption

collision

and

consumption

delay

with

a

-RRB-

single

consumption

channel

and

b

-RRB-

multiple

consumption

channels

m

The

number

of

incoming

network

channels

C

The

number

of

physical

consumption

channels

at

each

node

i

The

rate

of

arrival

of

terminating

messages

along

incoming

network

channel

i

in

messages/cycle

Total

message

arrival

rate

at

a

node

in

messages/cycle

L

Message

length

in

bytes

W

Channel

width

in

bytes

Tc

Service

time

seen

by

a

message

at

a

consumption

channel

di

Duty

factor

of

a

network

channel

with

respect

to

terminating

messages

pi

Probability

of

a

terminating

message

along

channel

i

encountering

a

consumption

collision

Ti

Expected

consumption

delay

seen

by

a

terminating

message

along

channel

i.

T

Expected

consumption

delay

seen

by

a

terminating

message

along

any

channel

This

is

a

weighted

average

of

delays

along

all

incoming

channels

We

make

the

following

assumptions

in

our

model

-RRB-

All

message

packets

are

of

length

L.

-RRB-

All

consumption

channels

have

the

same

message

service

rate

Tc

L

W

cycles

Consider

an

incoming

network

channel

i

with

arrival

rate

of

terminating

messages

i

An

arriving

terminating

message

reserves

a

consumption

channel

and

is

consumed

in

Tc

cycles

Thus

the

incoming

network

channel

is

busy

servicing

a

terminating

message

for

a

period

of

Tc

every

i

cycles

leading

to

a

duty

factor

of

di

iTc

Clearly

by

definition

di

We

denote

mj

j

as

the

total

terminating

message

rate

at

a

node

The

total

bandwidth

along

C

consumption

channels

imposes

the

constraint

C

Tc

In

a

later

section

we

demonstrate

the

constraining

impact

of

other

factors

on

the

message

arrival

rate

at

a

node

Estimating

probability

of

consumption

collision

Let

us

compute

pi

the

probability

of

a

terminating

message

along

channel

i

encountering

a

collision

delay

We

first

compute

pi

for

a

node

having

one

consumption

channel

denoted

by

and

then

generalize

it

to

C

consumption

channels

denotes

by

pCi

The

probability

of

collision

is

the

single

consumption

channel

being

busy

handling

a

message

along

another

channel

Based

on

the

analysis

in

-LSB-

-RSB-

the

probability

of

such

an

event

is

the

total

duty

factor

on

other

channels

leading

to

di

di

dl

Tc

-LRB-

i

i

m

-RRB-

-LRB-

-RRB-

In

general

with

C

consumption

channels

the

probability

of

a

consumption

collision

along

channel

i

is

that

of

all

consumption

channels

being

busy

with

messages

along

other

channels

With

C

m

every

incoming

channel

has

a

separate

consumption

channel

leading

to

-LRB-

pmi

-RRB-

For

C

m

pi

is

the

summation

of

probabilities

of

the

-LRB-

m

C

-RRB-

possible

ways

a

message

along

channel

i

can

collide

with

messages

along

the

other

m

network

channels

on

C

consumption

channels

This

probability

is

derived

as

pCi

djC

-LRB-

Tc

-RRB-

C

jC

-LRB-

-RRB-

where

the

summation

is

over

all

terms

obtained

as

-LRB-

jq

ffl

-LSB-

m

-RSB-

i

-RRB-

and

any

such

term

jC

has

for

Equation

can

be

observed

to

be

a

special

case

of

the

above

equation

obtained

with

C

By

comparing

pCi

for

different

values

of

C

leads

us

to

Observation

For

a

given

set

of

message

arrival

rates

along

m

channels

-LRB-

m

-RRB-

the

probability

of

consumption

collision

along

any

channel

i

-LRB-

pCi

-RRB-

reduces

as

the

number

of

consumption

channels

C

is

increased

from

to

m.

For

C

m

-LRB-

pi

-RRB-

This

observation

supports

the

intuition

that

having

more

consumption

channels

reduces

consumption

contention

This

leads

to

lower

expected

collision

delay

as

shown

below

Estimating

collision

delay

Given

that

we

have

a

collision

let

us

compute

the

expected

collision

delay

along

a

channel

i

The

waiting

time

has

a

uniform

distribution

from

to

Tc

With

no

queuing

at

the

switch

the

average

waiting

time

is

half

the

consumption

service

time

Tc

Thus

expected

consumption

collision

delay

seen

by

a

terminating

message

along

channel

i

Ti

is

Ti

pCi

Tc

-LRB-

-RRB-

The

collision

delay

seen

by

a

message

entering

this

node

to

be

consumed

denoted

by

T

is

given

as

a

weighted

average

of

delays

along

all

channels

T

-LRB-

>=-RRB-

lj

-LRB-

Ti

i

-RRB-

-LRB-

-RRB-

Tc

-LRB-

lj

-LRB-

pCi

i

-RRB-

-RRB-

-LRB-

-RRB-

A

higher

value

of

collision

delay

leads

to

greater

chance

of

other

messages

in

the

network

blocking

on

resources

held

by

consumption

blocked

message

aggravating

the

impact

of

a

domino-blocking

effect

Based

on

Observation

and

the

above

expression

we

conclude

that

higher

number

of

consumption

channels

reduce

consumption

delay

and

hence

alleviate

the

problems

from

the

domino-blocking

effect

leading

to

higher

network

performance

Parameters

affecting

collision

delay

and

benefits

from

more

consumption

channels

Using

the

expression

for

pCi

derived

earlier

in

Eqn

and

applying

it

to

Eqn

leads

us

to

the

following

expression

for

T

in

a

node

with

C

consumption

channels

T

-LRB-

>=-RRB-

-LRB-

Tc

-RRB-

-LRB-

Tc

-RRB-

C

-LRB-

C

-RRB-

-LRB-

jC

jC

-RRB-

-LRB-

-RRB-

where

the

summation

is

over

all

terms

obtained

as

q

-LRB-

jq

ffl

-LSB-

m

-RSB-

-RRB-

and

any

such

term

jC

has

for

Eqn

leads

us

to

Observation

Given

a

fixed

total

terminating

message

rate

at

a

node

-LRB-

mj

j

-RRB-

the

expected

collision

delay

T

is

minimized

when

j

for

some

j

i

and

j

for

j

i.

Similarly

T

is

maximized

when

message

rates

are

balanced

i

m.

Thus

the

maximum

collision

delay

denoted

by

Tmax

for

given

C

Tc

m

and

-LRB-

assuming

balanced

rate

on

channels

-RRB-

is

derived

as

Tmax

-LRB-

-LRB-

Tc

m

-RRB-

C

-LRB-

C

-RRB-

-LRB-

mC

-RRB-

-RRB-

-LRB-

>=-RRB-

-LRB-

-RRB-

The

parameter

Tmax

being

a

function

of

C

m

and

Tc

can

be

expressed

as

Tmax

-LRB-

C

m

Tc

-RRB-

For

C

m

it

can

be

shown

that

the

expression

-LRB-

Tmax

-LRB-

C

m

Tc

-RRB-

Tmax

-LRB-

C

m

Tc

-RRB-

-RRB-

is

an

increasing

function

with

respect

to

m

and

Tc

Thus

at

higher

values

of

these

parameters

the

decrease

in

consumption

delay

with

more

consumption

channels

is

expected

to

be

higher

Based

on

the

above

discussion

we

make

the

following

summary

on

expected

trends

on

benefits

from

more

consumption

channels

Observation

Benefits

from

more

consumption

channels

at

a

node

is

significant

at

higher

values

of

collision

delay

Collision

delay

is

higher

with

higher

message

rate

into

a

node

message

rates

being

balanced

along

incoming

channels

higher

network

dimensionality

-LRB-

m

-RRB-

and

longer

messages

-LRB-

Tc

L

W

-RRB-

In

the

following

section

we

consider

the

impact

of

the

above

factors

in

more

detail

Applying

the

model

to

determine

required

number

of

physical

consumption

channels

In

this

section

we

consider

the

impact

of

balanced

versus

unbalanced

message

rates

longer

message

length

and

system

dimensionality

on

the

benefits

from

more

consumption

channels

Impact

of

routing

balanced

versus

unbalanced

rates

The

routing

algorithm

used

to

route

messages

in

a

network

can

lead

to

messages

arriving

at

a

node

along

different

incoming

channels

in

a

balanced

or

unbalanced

manner

This

can

affect

the

demanded

number

of

consumption

channels

at

a

node

Let

us

first

consider

dimension-order

routing

as

discussed

in

the

example

in

Fig.

It

routes

messages

on

a

minimal

path

in

order

of

increasing

dimensions

For

a

given

destination

node

let

us

consider

the

directions

along

which

messages

from

other

nodes

enter

here

as

determined

by

routing

restrictions

The

directions

are

identified

as

i

i

n.

By

considering

all

cases

it

can

be

shown

in

a

k-ary

n-cube

system

the

number

of

messages

entering

the

destination

node

along

direction

i

is

-LRB-

bk

cki

-RRB-

while

-LRB-

dk

eki

-RRB-

messages

enter

along

direction

i

The

total

possible

messages

being

-LRB-

kn

-RRB-

the

relative

message

arrival

rate

along

direction

i

is

derived

as

-LRB-

bk

cki

-RRB-

-LRB-

kn

-RRB-

ss

bk

c

kn?i

Thus

for

a

given

total

message

termination

rate

at

a

node

being

the

message

rate

along

direction

i

denoted

here

as

i

is

obtained

as

i

-LRB-

b

k

c

kn?i

-RRB-

-LRB-

-RRB-

Similarly

message

rates

can

be

obtained

along

other

dimensions

i

n.

From

the

above

equation

a

sharp

imbalance

can

be

observed

in

the

message

rates

along

different

dimensions

Most

messages

enter

along

the

highest

dimensions

-LRB-

n

-RRB-

A

more

adaptive

routing

is

expected

to

lead

to

more

balanced

message

arrival

rates

at

a

node

In

this

case

for

a

given

total

message

termination

rate

at

a

node

being

the

message

rate

along

direction

i

is

obtained

as

i

-LRB-

-RRB-

Figure

depicts

the

impact

of

having

more

consumption

channels

on

expected

collision

delay

The

plots

show

collision

delay

-LRB-

T

-RRB-

versus

message

rate

-LRB->=-RRB-

at

a

node

in

a

-LRB-

-RRB-

system

Figure

-LRB-

a

-RRB-

is

derived

based

on

Eqn

and

message

rates

from

dimension-order

routing

as

in

Eqn

Similarly

Fig.

-LRB-

b

-RRB-

is

derived

based

on

Eqns

and

message

rates

from

a

more

adaptive

routing

as

in

Eqn

A

message

length

of

bytes

and

channel

width

of

bytes

leading

to

Tc

cycles

is

assumed

The

maximum

allowable

message

rate

is

derived

by

ensuring

the

duty

cycle

of

any

channel

not

to

exceed

From

the

plots

it

can

be

observed

that

for

maintaining

a

small

average

collision

delay

-LRB-

let

us

consider

cycles

-RRB-

the

number

of

consumption

channels

required

increases

with

message

rate

By

comparing

the

plots

in

Fig.

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

we

observe

that

more

consumption

channels

offer

higher

benefits

with

balanced

rates

This

leads

us

to

the

following

observation

Average

collision

delay

-LRB-

cycles

-RRB-

Message

arrival

rate

at

a

node

-LRB-

msgs/cycle

-RRB-

a

-RRB-

Dimension-order

routing

n

Tc

cycles

C

C

C

Average

collision

delay

-LRB-

cycles

-RRB-

Message

arrival

rate

at

a

node

-LRB-

msgs/cycle

-RRB-

b

-RRB-

Adaptive

routing

n

Tc

cycles

C

C

C

C

Figure

Demonstrating

impact

of

more

physical

consumption

channels

at

a

node

in

a

-LRB-

-RRB-

network

system

Routing

assumed

is

a

-RRB-

dimension-order

leading

to

unbalanced

traffic

along

incoming

channels

and

b

-RRB-

adaptive

leading

to

more

balanced

traffic

along

incoming

channels

Observation

Routing

algorithms

with

adaptivity

offering

balanced

message

rates

along

incoming

channels

can

gain

from

higher

number

of

consumption

channels

than

non-adaptive

-LRB-

dimensionorder

-RRB-

routing

In

Sec.

we

compare

fully

adaptive

-LSB-

-RSB-

routing

against

dimension-order

to

determine

the

impact

of

routing

on

the

effective

number

of

consumption

channels

Impact

of

message

length

Figure

depicts

the

impact

of

more

consumption

channels

in

a

system

with

two

different

message

length

of

and

bytes

respectively

It

can

be

observed

that

with

the

longer

message

length

-LRB-

L

bytes

-RRB-

values

of

average

collision

delay

are

much

higher

For

example

with

L

bytes

and

a

message

arrival

rate

of

messages/cycle

-LRB-

or

bytes/cycle

-RRB-

the

collision

delay

is

cycles

with

C

For

L

bytes

a

similar

arrival

rate

of

bytes/cycle

occurs

at

messages/cycle

-LRB-

obtained

as

-RRB-

With

C

the

collision

delay

at

this

message

rate

is

much

higher

cycles

Similarly

it

can

be

observed

that

the

drop

in

collision

delay

with

another

consumption

channel

-LRB-

C

-RRB-

is

also

much

more

significant

with

L

-LRB-

cycles

as

compared

to

with

L

-RRB-

This

indicates

that

a

system

with

longer

messages

can

gain

more

from

higher

number

of

channels

Average

collision

delay

-LRB-

cycles

-RRB-

Message

arrival

rate

at

a

node

-LRB-

msgs/cycle

-RRB-

n

Tc

cycles

-LRB-

L

W

-RRB-

C

C

C

C

Average

collision

delay

-LRB-

cycles

-RRB-

Message

arrival

rate

at

a

node

-LRB-

msgs/cycle

-RRB-

n

Tc

cycles

-LRB-

L

W

-RRB-

C

C

C

C

Figure

Demonstrating

impact

of

more

physical

consumption

channels

at

a

node

in

a

-LRB-

-RRB-

network

system

with

longer

messages

Message

lengths

assumed

are

L

and

bytes

respectively

Impact

of

network

dimensionality

Figure

depicts

the

impact

of

more

consumption

channels

in

two

systems

with

different

dimensionality

The

plots

are

derived

based

on

Eqn

and

are

shown

separately

for

a

node

in

a

and

network

system

respectively

As

discussed

earlier

an

n-dimensional

system

requires

at

most

physical

consumption

channels

Thus

a

system

can

use

up

to

four

consumption

channels

while

the

system

can

use

up

to

six

At

and

beyond

these

respective

number

of

consumption

channels

the

expected

collision

delay

drops

to

zero

based

on

Obs

From

the

plots

it

can

be

observed

that

for

the

same

message

rate

the

decrease

in

consumption

delay

with

more

consumption

channels

is

higher

in

the

network

indicating

higher

dimensional

systems

can

gain

more

from

higher

number

of

channels

Average

collision

delay

-LRB-

cycles

-RRB-

Message

arrival

rate

at

a

node

-LRB-

msgs/cycle

-RRB-

n

Tc

cycles

C

C

C

C

Average

collision

delay

-LRB-

cycles

-RRB-

Message

arrival

rate

at

a

node

-LRB-

msgs/cycle

-RRB-

n

Tc

cycles

C

C

C

C

C

C

Figure

Demonstrating

maximum

impact

of

more

physical

consumption

channels

at

a

node

in

two

systems

with

different

dimensionality

and

respectively

From

the

above

discussion

it

is

clear

that

two

major

factors

dictate

the

expected

benefits

from

more

consumption

channels

These

are

a

-RRB-

maximum

possible

message

rate

in

a

system

and

b

-RRB-

the

balance

among

the

arrival

rates

along

different

incoming

channels

The

maximum

message

rate

or

in

a

system

is

bounded

by

other

constraints

such

as

network

bisection

size

sustained

the

paper

we

have

used

message

rate

and

throughput

interchangeably

available

bandwidth

between

interface

and

processor

messaging

overheads

for

sending/receiving

messages

traffic

patterns

and

virtual

channel

flow

control

Constraints

affecting

maximum

message

arrival

rate

and

number

of

physical

consumption

channels

In

this

section

we

analyze

the

impact

of

the

factors

mentioned

above

on

the

maximum

message

arrival

rate

at

a

node

Network

bisection

A

small

network

bisection

limits

the

maximal

achievable

system

performance

under

uniform

traffic

Let

B

denote

the

bisection

size

of

a

network

It

is

the

minimum

number

of

network

channels

to

be

cut

so

as

to

divide

the

network

into

two

equal

halves

The

bisection

size

of

a

k-ary

n-cube

network

is

derived

-LSB-

-RSB-

as

B

-LRB-

-RRB-

where

W

as

defined

earlier

denotes

the

channel

width

in

bytes

Thus

at

most

B

bytes

can

be

sent

across

the

network

bisection

per

cycle

For

messages

of

length

L

bytes

the

same

bandwidth

translates

to

B

L

L

messages/cycle

Under

uniform

traffic

half

the

generated

messages

in

the

system

need

to

cross

the

network

bisection

Thus

for

a

message

sending

rate

of

messages/cycle

at

each

of

the

kn

nodes

in

the

system

the

number

of

messages

traveling

across

the

bisection

every

cycle

is

derived

as

kn

Since

this

has

to

be

less

than

the

bisection

size

it

leads

to

-LRB-

-LRB-

kn

-RRB-

-LRB-

L

-RRB-

-RRB-

The

maximum

message

rate

from

bisection

constraint

denoted

by

B

is

B

-LRB-

-RRB-

-LRB-

kL

-RRB-

messages/cycle

-LRB-

-RRB-

This

leads

us

to

the

following

observation

Observation

Keeping

channel

width

-LRB-

W

-RRB-

and

message

length

-LRB-

L

-RRB-

fixed

in

a

k-ary

n-cube

system

increasing

network

size

by

using

larger

radix

-LRB-

k

-RRB-

leads

to

a

fall

in

maximum

message

rate

-LRB-

B

-RRB-

supportable

by

network

bisection

Depending

on

the

system

size

-LRB-

k

-RRB-

it

may

happen

that

in

some

configurations

B

is

constrained

to

be

very

small

This

leads

to

probability

of

consumption

collision

also

being

small

In

such

case

more

consumption

channels

may

not

offer

significant

benefit

in

performance

toruses

this

number

needs

to

be

adjusted

by

a

factor

of

two

to

B

channels

due

to

the

presence

of

wrap-around

links

It

should

be

noted

that

the

parameter

B

is

only

an

upper

limit

on

the

maximum

achievable

throughput

in

a

system

The

actual

maximum

achievable

throughput

is

typically

lower

due

to

network

contention

Similar

observations

continue

to

hold

with

such

throughput

also

-LSB-

-RSB-

Network

contention

depends

on

various

factors

including

virtual

channel

flow

control

routing

adaptivity

and

message

length

In

Sec

we

study

these

aspects

in

detail

through

simulation

experiments

We

demonstrate

that

in

contemporary

systems

even

up

to

processors

the

achievable

value

of

throughput

is

high

enough

to

merit

more

consumption

channels

Non-uniform

traffic

In

a

traffic

scenario

hot-spots

occur

when

some

nodes

in

the

system

receive

more

than

their

fair-share

of

messages

Let

us

characterize

non-uniform

traffic

by

choosing

a

fraction

fh

of

the

nodes

-LRB-

N

-RRB-

in

the

system

randomly

as

hot

nodes

Each

processor

in

the

system

generates

messages

at

a

given

rate

and

the

messages

are

directed

towards

a

hot

node

with

a

probability

controlled

by

a

parameter

d

defining

the

degree

of

hotness

of

traffic

The

parameter

d

can

reflect

various

degrees

of

hot

traffic

A

value

of

d

corresponds

to

uniform

traffic

while

higher

values

-LRB-

d

-RRB-

correspond

to

non-uniform

traffic

As

the

traffic

becomes

more

non-uniform

-LRB-

increasing

d

-RRB-

the

network

bisection

becomes

lesser

constraint

on

achievable

non-uniform

throughput

Thus

at

the

hot

nodes

message

arrival

rate

much

greater

than

B

can

be

expected

In

such

a

situation

the

number

of

incoming

network

channels

into

a

hot

node

becomes

the

dominant

constraint

on

performance

With

higher

message

rate

the

number

of

consumption

channels

required

to

alleviate

the

consumption

bottleneck

is

expected

to

be

higher

This

leads

us

to

the

following

observation

Observation

Higher

number

of

consumption

channels

are

expected

to

be

useful

in

alleviating

the

consumption

bottleneck

under

hot-spot

traffic

than

under

uniform

traffic

Adaptivity

and

virtual

multiplexing

on

network

channels

In

Sec.

we

observed

one

impact

of

adaptive

routing

in

offering

more

balanced

message

arrival

rates

The

other

impact

of

adaptivity

is

in

offering

higher

network

throughput

-LSB-

-RSB-

by

allowing

a

message

to

circumvent

points

of

congestion

Similar

impact

is

also

achieved

with

virtual

channel

flow

control

-LRB-

VCFC

-RRB-

-LSB-

-RSB-

which

allows

an

active

message

to

bypass

blocked

messages

on

a

channel

leading

to

lower

blocking

and

better

network

utilization

This

leads

to

the

following

observation

terms

non-uniform

traffic

and

hot-spot

traffic

are

used

interchangeably

in

this

paper

Observation

Higher

adaptivity

and

virtual

channels

lead

to

higher

network

throughput

and

higher

maximum

message

rate

For

sustaining

such

higher

message

rate

at

the

consumption

interface

more

consumption

channels

can

be

beneficial

Messaging

overheads

In

point-to-point

communication

a

message

is

associated

with

processing

overheads

from

procedure

calls

protection

checking

checksum

computing

and

possibly

memory

copying

Let

ts

cycles/message

denote

such

overhead

The

maximum

message

handling

rate

from

such

overhead

denoted

by

sw

is

derived

as

sw

ts

messages/cycle

-LRB-

-RRB-

Such

overheads

usually

in

the

range

of

few

hundreds

to

thousands

of

cycles

-LSB-

-RSB-

can

severely

limit

the

rate

at

which

messages

can

be

sent

or

received

from

a

node

-LSB-

-RSB-

For

example

a

messaging

overhead

of

cycles

limits

maximum

message

rate

per

node

to

messages/cycle

This

can

lead

to

much

of

the

communication

bandwidth

offered

by

the

expensive

fast

interconnect

to

remain

unutilized

With

low

maximum

message

rate

per

node

the

probability

of

consumption

delay

can

be

small

leading

to

limited

benefits

from

higher

number

of

consumption

channels

Several

research

studies

are

ongoing

-LSB-

-RSB-

for

designing

messaging

protocols

and

hardware

to

reduce

such

overheads

In

the

near

future

we

expect

efficient

protocols

to

offer

ts

in

the

range

of

few

tens

to

hundreds

of

cycles

Similarly

in

hardware

based

implementations

of

Distributed

Shared

Memory

-LRB-

DSM

-RRB-

systems

with

small

messages

for

cache

coherence

the

value

of

ts

is

expected

to

be

small

In

Sec.

we

study

the

impact

of

such

overhead

on

the

requirement

of

consumption

channels

We

demonstrate

that

even

with

reasonably

high

messaging

overhead

ts

cycles

more

than

one

consumption

channel

can

yield

significant

gain

in

performance

Interface

to

processor

bandwidth

Each

physical

consumption

channel

can

remove

messages

from

the

network

into

the

interface

message

buffers

at

the

rate

of

Bphy

W

bytes/cycle

where

W

is

the

channel

width

as

defined

earlier

With

C

physical

consumption

channels

this

rate

increases

proportionally

to

WC

bytes/cycle

However

this

may

not

be

the

rate

that

can

be

sustained

at

the

consumption

interface

Messages

at

the

interface

buffers

are

transferred

by

DMA

to

the

processor

memory

over

the

local

bus

As

discussed

earlier

in

Sec.

only

a

fraction

of

the

bus

bandwidth

-LRB-

I-to-P

bandwidth

-RRB-

is

available

to

DMA

operations

for

writing

consumed

messages

The

parameter

Bmem

bytes/cycle

refers

to

such

I-to-P

bandwidth

for

message

consumption

A

cycle

here

refers

to

the

network

cycle

time

and

not

the

cycle

time

of

the

local

bus

Clearly

a

small

value

of

Bmem

can

restrict

the

maximum

sustainable

message

rate

in

the

system

For

example

in

a

system

with

Bmem

bytes/cycle

-LRB-

for

a

cycle

time

of

sec

this

corresponds

to

MBytes/sec

-RRB-

and

message

length

bytes

the

maximum

message

rate

is

messages/cycle

Having

limited

I-to-P

bandwidth

also

raises

an

interesting

question

Is

there

a

need

to

have

more

than

Bmem

Bphy

consumption

channels

in

a

system

even

if

this

number

is

very

small

Let

us

define

the

parameter

Cm

to

denote

the

number

Bmem

Bphy

It

can

be

observed

that

by

using

only

Cm

channels

the

I-to-P

bandwidth

can

be

fully

utilized

However

for

small

value

of

Cm

-LRB-

or

-RRB-

the

probability

of

collision

on

the

consumption

channels

is

high

leading

to

high

collision

delays

High

consumption

delay

leads

degradation

of

performance

due

to

domino-blocking

effect

Only

in

traffic

patterns

like

all-to-one

where

all

messages

in

the

system

are

destined

to

a

single

node

the

domino-blocking

effect

can

not

hurt

the

performance

of

messages

destined

to

other

nodes

-LRB-

since

there

are

no

such

messages

-RRB-

Under

such

a

pattern

having

Cm

consumption

channels

is

sufficient

even

if

this

number

is

small

-LRB-

or

-RRB-

However

under

other

traffic

patterns

like

uniform

and

nonuniform

where

domino-blocking

effect

leads

to

overall

degradation

of

performance

having

more

than

Cm

consumption

channels

can

be

beneficial

especially

when

Cm

is

a

small

number

like

or

This

leads

to

Observation

For

small

values

of

Cm

having

more

than

Cm

physical

consumption

channels

does

not

lead

to

better

completion

time

for

an

all-to-one

traffic

pattern

However

the

performance

of

uniform

and

non-uniform

traffic

pattern

can

improve

with

more

than

Cm

channels

Summary

It

can

be

observed

that

the

need

for

more

than

one

physical

consumption

channel

is

dependent

on

the

interplay

of

all

the

above

factors

routing

algorithm

message

length

traffic

pattern

network

bisection

messaging

overhead

interface-to-processor

bandwidth

and

network

dimensionality

With

all

these

factors

determining

consumption

performance

it

becomes

a

challenging

problem

to

determine

How

many

physical

consumption

channels

need

to

be

provided

per

node

to

derive

the

best

performance

Intuitively

with

each

added

channel

we

expect

to

get

back

diminishing

returns

in

terms

of

gained

performance

Thus

we

are

interested

in

a

minimum

or

effective

number

of

channels

beyond

which

added

performance

gained

-LRB-

in

terms

of

reduction

in

latency

or

increase

in

throughput

-RRB-

is

less

than

some

specified

percentage

of

the

performance

attained

with

one

consumption

channel

Without

loss

of

generality

in

this

paper

we

choose

this

number

as

Earlier

studies

-LSB-

-RSB-

have

proposed

contention

models

to

determine

latency

versus

throughput

performance

in

wormhole-switched

k-ary

n-cube

networks

However

these

studies

focused

mostly

on

dimension-order

routing

and

assumed

no

consumption

collision

at

the

destination

node

-LRB-

consumption

assumption

-LSB-

-RSB-

-RRB-

In

this

study

our

focus

is

on

designing

the

consumption

interface

and

studying

the

impact

of

various

factors

on

such

design

The

large

number

of

parameters

being

studied

makes

it

difficult

to

come

up

with

a

tractable

integrated

contention

model

for

the

network

together

with

the

consumption

interface

in

a

wormhole

system

Thus

in

this

paper

we

establish

trends

and

derive

guidelines

through

simulation

by

considering

the

whole

gamut

of

interplay

discussed

above

Simulation

experiments

and

results

In

this

section

we

validate

the

guidelines

proposed

in

earlier

sections

using

simulation

and

derive

effective

number

of

consumption

channels

required

under

different

network

and

system

parameters

like

routing

message

length

system

size

messaging

overheads

I-to-P

bandwidth

traffic

patterns

and

degree

of

virtual

multiplexing

Simulation

Set-up

A

flit-level

wormhole-routed

simulator

WORMULSim

was

used

to

model

and

evaluate

the

impact

of

multiple

consumption

channels

for

different

topologies

routing

and

traffic

pattern

This

simulator

is

designed

using

CSIM

-LSB-

-RSB-

It

accurately

models

flit-level

transfer

corresponding

to

multiple

virtual

channels

using

demand

multiplexing

over

network

and

consumption

channels

Similarly

DMA

operations

over

the

local

bus

for

transferring

messages

are

also

modeled

Without

loss

of

generality

a

network

cycle

time

of

sec

was

assumed

Deterministic

-LSB-

-RSB-

and

fully-adaptive

-LSB-

-RSB-

routing

schemes

with

variable

number

of

virtual

channels

-LRB-

lanes

-RRB-

were

used

Four

different

system

configurations

and

were

examined

Three

different

traffic

models

were

considered

uniform

traffic

hot-spot

traffic

and

all-to-one

pattern

In

each

experiment

the

number

of

physical

consumption

channels

per

node

was

varied

from

to

Each

physical

consumption

channel

supported

v

virtual

consumption

channels

where

v

is

the

degree

of

multiplexing

on

network

channels

Each

simulation

experiment

was

carried

out

for

a

sufficient

number

of

runs

to

ensure

that

the

confidence

interval

was

within

of

the

mean

Three

different

values

of

I-to-P

bandwidth

corresponding

to

Cm

and

were

used

Impact

of

message

length

was

observed

for

two

different

values

of

L

and

bytes

Messaging

overhead

of

tsw

to

cycles

was

examined

A

channel

width

W

bytes

was

assumed

in

these

experiments

Impact

of

routing

adaptivity

Figure

shows

the

latency

-LRB-

microsecs

-RRB-

versus

throughput

per

node

-LRB-

messages/microsec

-RRB-

for

dimensionorder

routing

and

fully

adaptive

-LSB-

-RSB-

routing

schemes

respectively

on

a

system

The

top

row

of

plots

depict

the

results

for

uniform

traffic

while

the

bottom

row

of

plots

depict

results

with

hot-spot

traffic

A

message

length

of

bytes

was

assumed

From

Fig.

it

can

be

observed

that

under

uniform

traffic

with

dimension-order

routing

the

maximum

achieved

throughput

with

one

consumption

channel

per

node

-LRB-

C

-RRB-

is

only

messages

sec

It

is

interesting

to

note

that

with

adaptive

routing

and

C

exactly

the

same

maximum

throughput

is

attained

This

indicates

that

the

expected

gains

in

performance

are

inhibited

by

the

consumption

bottleneck

With

more

consumption

channels

-LRB-

C

to

-RRB-

increased

throughput

is

attained

under

both

routing

schemes

However

with

dimension-order

the

maximum

throughput

attained

with

C

is

limited

to

messages

sec

This

indicates

that

C

is

effective

for

this

system

With

adaptive

routing

up

to

C

channels

are

beneficial

leading

to

much

higher

maximum

throughput

of

messages

sec

These

results

validate

Obs

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

C

C

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully-adaptive

Uniform

L

bytes

C

C

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully-adaptive

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

C

Figure

Impact

of

routing

adaptivity

and

number

of

consumption

channels

on

performance

in

a

system

Both

uniform

and

non-uniform

traffic

were

considered

Similar

trends

were

also

observed

with

hot-spot

traffic

as

depicted

in

the

bottom

row

of

plots

in

Fig.

The

message

arrival

rates

at

hot

nodes

is

much

higher

with

hot-traffic

-LRB-

the

x-axis

in

the

plots

represent

message

generation

rate

the

message

arrival

rate

at

hot

nodes

is

d

times

higher

-RRB-

As

with

uniform

traffic

with

C

the

maximum

throughput

achieved

was

same

-LRB-

messages

sec

-RRB-

under

both

routing

schemes

indicating

consumption

bottleneck

inhibiting

gains

from

adaptivity

With

more

consumption

channels

the

maximum

throughput

with

dimension-order

and

adaptive

routing

increased

to

and

respectively

Such

message

rates

were

achieved

at

C

and

under

dimension-order

and

fully-adaptive

routing

respectively

This

confirms

that

higher

number

of

consumption

channels

are

more

beneficial

with

adaptive

routing

From

Fig.

it

can

also

be

observed

that

for

a

system

with

a

given

routing

more

consumption

channels

are

beneficial

with

hot-spot

traffic

compared

to

uniform

traffic

This

validates

Obs

Impact

of

message

length

Figure

shows

latency

versus

throughput

obtained

with

uniform

traffic

on

a

system

with

dimension-order

routing

Plots

are

separately

shown

for

message

length

of

L

and

bytes

respectively

From

Fig.

it

can

be

observed

that

with

L

there

is

no

gain

in

throughput

by

using

C

over

C

channels

However

with

longer

message

length

L

bytes

there

is

a

gain

of

in

throughput

with

C

This

indicates

that

with

higher

message

length

higher

number

of

consumption

channels

can

be

beneficial

validating

Obs

Similar

trends

were

also

observed

with

non-uniform

traffic

adaptive

routing

and

other

system

configurations

These

results

are

not

presented

here

in

order

to

limit

the

size

of

the

paper

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

torus

dim-order

Uniform

L

bytes

C

C

C

C

Figure

Impact

of

message

length

and

number

of

consumption

channels

on

uniform

traffic

performance

in

a

system

Two

different

message

length

of

L

and

bytes

respectively

were

considered

Impact

of

system

dimensionality

Figure

shows

latency

versus

throughput

obtained

with

with

non-uniform

traffic

on

two

systems

of

different

dimensionality

and

From

Fig.

it

can

be

observed

that

C

and

are

effective

in

the

and

systems

respectively

This

validates

Obs

that

higher

number

of

consumption

channels

can

be

beneficial

in

higher

dimensional

system

Impact

of

system

size

with

uniform

traffic

In

Obs

we

derived

that

achievable

uniform

message

rate

decreases

with

system

size

due

to

network

bisection

constraint

Reduced

message

rates

in

larger

systems

can

make

higher

number

of

consumption

channels

less

beneficial

Figure

shows

latency

versus

throughput

results

with

uniform

traffic

on

two

system

sizes

of

and

and

two

system

sizes

of

and

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully-adaptive

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully-adaptive

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

C

Figure

Impact

of

dimensionality

and

number

of

consumption

channels

on

system

performance

Two

different

systems

and

were

considered

From

Fig.

it

can

be

observed

that

C

is

beneficial

even

in

and

large

sized

systems

However

the

relative

gain

obtained

is

smaller

as

compared

to

smaller

systems

of

the

same

dimensionality

For

example

it

can

be

observed

from

Fig.

that

in

the

larger

system

the

maximum

message

throughput

is

messages

sec

with

C

This

is

much

lower

than

the

corresponding

messages

sec

obtained

on

the

system

Similarly

messages

sec

obtained

with

C

on

the

system

is

much

lower

than

corresponding

messages

sec

obtained

on

the

system

Thus

the

relative

gain

by

using

C

on

the

system

is

only

-LRB-

lower

than

corresponding

gain

of

on

the

system

-RRB-

The

above

results

indicate

that

under

uniform

traffic

benefits

from

more

consumption

channels

do

reduce

with

larger

system

size

However

the

benefits

are

significant

to

merit

them

even

in

reasonably

large

system

sizes

Similar

trends

were

observed

on

systems

and

as

shown

in

top

row

of

plots

in

Fig.

Although

not

reported

here

with

hot-spot

traffic

more

consumption

channels

were

derived

to

be

beneficial

even

in

large

system

sizes

Impact

of

more

virtual

channels

Figure

shows

the

latency

versus

throughput

plots

on

a

system

The

number

of

virtual

lanes

is

varied

from

v

and

The

following

observations

can

be

made

from

Fig.

With

C

the

maximum

achievable

message

rate

is

constrained

by

messages

sec

in

all

three

systems

This

indicates

that

the

consumption

bottleneck

with

a

single

channel

limits

the

benefits

from

higher

number

of

virtual

lanes

The

effective

number

of

channels

in

the

three

systems

are

C

and

respectively

Thus

with

v

increasing

from

to

one

more

consumption

channel

is

required

to

attain

higher

message

rate

However

with

v

no

further

gain

is

achieved

with

more

channels

This

is

because

benefits

from

more

virtual

lanes

level

off

at

this

point

These

results

validate

Obs

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

C

C

C

C

Figure

Impact

of

more

consumption

channels

on

uniform

traffic

performance

on

two

system

sizes

of

and

and

two

system

sizes

of

and

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

v

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

torus

dim-order

v

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

v

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

C

Figure

Impact

of

more

virtual

lanes

and

consumption

channels

on

performance

in

a

system

Three

different

values

of

v

and

were

considered

Impact

of

messaging

overheads

Figure

shows

the

latency

versus

throughput

for

a

system

with

uniform

traffic

The

messaging

overhead

was

varied

from

tsw

and

cycles

The

corresponding

constraint

on

maximum

throughput

per

node

is

and

messages

sec

-LRB-

as

stated

earlier

cycle

sec

-RRB-

From

Fig.

it

can

be

observed

that

the

maximum

achievable

throughput

with

tsw

cycles

is

messages

sec

In

this

example

the

network

bisection

constraint

does

not

allow

performance

to

be

as

high

as

messages

sec

In

this

system

C

is

effective

For

tsw

and

the

maximum

achieved

throughput

of

and

messages

sec

respectively

match

the

predicted

message

rate

from

messaging

overhead

constraint

At

these

lower

message

arrival

rates

C

offers

no

gain

in

throughput

over

C

However

benefits

in

terms

of

lower

message

latency

is

obtained

Such

benefits

lead

us

to

conclude

that

C

is

effective

even

in

systems

with

high

message

overheads

Although

not

reported

here

these

benefits

were

derived

to

be

higher

with

longer

messages

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

Tsw

us

-LRB-

cycles

-RRB-

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

Tsw

us

-LRB-

cycles

-RRB-

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

dim-order

Uniform

L

bytes

Tsw

us

-LRB-

cycles

-RRB-

C

C

C

C

Figure

Impact

of

messaging

overheads

on

maximum

message

rate

and

effective

number

of

consumption

channels

Uniform

traffic

performance

is

shown

for

a

configuration

with

three

different

messaging

overheads

tsw

and

cycles

Impact

of

I-to-P

bandwidth

The

available

bus

bandwidth

from

the

interface

to

processor-memory

for

message

consumption

-LRB-

Cm

-RRB-

restricts

the

maximum

sustained

message

rate

As

demonstrated

in

Sec.

this

is

critical

when

expected

message

rates

are

high

as

in

an

all-to-one

communication

pattern

Here

we

discuss

results

derived

for

three

different

patterns

all-to-one

pattern

uniform

traffic

and

non-uniform

traffic

Results

were

derived

for

a

range

of

values

of

Cm

and

The

value

of

corresponds

to

a

very

high

I-to-P

bandwidth

Such

high

value

of

I-to-P

bandwidth

reflects

systems

where

it

is

not

a

limitation

to

maximum

performance

Thus

experiments

with

Cm

demonstrate

the

maximum

value

of

effective

number

of

consumption

channels

required

for

a

system

with

I-to-P

bandwidth

not

being

a

bottleneck

All-to-one

traffic

Figure

shows

the

all-to-one

completion

time

in

and

systems

From

Fig.

it

can

be

observed

that

on

a

system

and

Cm

the

lowest

completion

time

is

attained

with

C

Higher

number

of

consumption

channels

offer

no

better

performance

This

leads

us

to

conclude

that

C

is

the

effective

number

of

channels

for

an

all-to-one

pattern

with

Cm

Similar

result

was

also

observed

on

a

system

with

Cm

Similarly

with

Cm

the

effective

number

of

consumption

channels

were

observed

to

be

C

in

both

the

systems

These

results

validate

Obs

that

in

systems

with

low

values

of

Cm

or

having

higher

C

does

not

lead

to

better

performance

with

all-to-one

traffic

For

high

value

of

Cm

the

performance

is

not

constrained

by

I-to-P

bandwidth

It

can

be

observed

from

Fig.

that

C

and

are

effective

on

the

and

systems

respectively

Similar

trends

were

also

obtained

with

smaller

systems

and

and

different

message

length

These

results

indicate

that

having

C

or

is

adequate

for

achieving

best

performance

even

under

very

hot-spot

traffic

Pattern

Completion

Time

Physical

Consumption

Channels

Per

Node

-LRB-

C

-RRB-

Fully-adaptive

L

bytes

Cm

Cm

Cm

Pattern

Completion

Time

Physical

Consumption

Channels

Per

Node

-LRB-

C

-RRB-

Fully-adaptive

L

bytes

Cm

Cm

Cm

Figure

Impact

of

increasing

I-to-P

bandwidth

-LRB-

Cm

-RRB-

and

consumption

channels

-LRB-

C

-RRB-

on

the

completion

time

of

an

all-to-one

pattern

in

a

and

system

Three

different

values

of

Cm

and

are

considered

Uniform

and

Non-uniform

traffic

In

Obs

it

was

derived

that

under

uniform

traffic

more

than

one

consumption

channel

may

be

necessary

to

deliver

better

performance

even

with

limited

I-to-P

bandwidth

-LRB-

Cm

-RRB-

This

was

validated

in

our

experiments

Figure

depicts

the

latency

versus

throughput

plots

with

uniform

traffic

on

a

system

and

Cm

and

As

discussed

in

Obs

the

trends

with

uniform

traffic

are

unlike

the

results

with

all-to-one

pattern

From

Fig.

it

can

be

observed

that

C

is

effective

even

with

a

low

value

of

Cm

The

plots

with

Cm

are

very

similar

to

those

with

Cm

indicating

that

with

high

I-to-P

bandwidth

there

is

no

further

gain

in

uniform

performance

It

can

be

observed

that

C

is

effective

even

with

Cm

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully-adaptive

Cm

Uniform

L

bytes

C

C

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully-adaptive

Cm

Uniform

L

bytes

C

C

C

C

C

C

Figure

Impact

of

increasing

I-to-P

bandwidth

-LRB-

Cm

-RRB-

and

consumption

channels

-LRB-

C

-RRB-

on

system

performance

with

uniform

traffic

Trends

with

non-uniform

traffic

were

in

between

those

obtained

with

uniform

traffic

and

all-to-one

pattern

Figure

depicts

the

latency

versus

throughput

plots

with

a

hot-spot

traffic

on

a

system

and

Cm

and

From

Fig.

it

can

be

observed

that

with

hot-spot

traffic

the

maximum

message

rate

is

limited

by

the

value

of

Cm

As

Cm

was

increased

the

maximum

message

rate

achievable

-LRB-

with

higher

number

of

consumption

channels

-RRB-

also

increased

With

Cm

and

the

effective

number

of

consumption

channels

are

and

respectively

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully

adaptive

Cm

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully

adaptive

Cm

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

Average

message

latency

-LRB-

T

-RRB-

in

microsecs

Throughput

-LRB-

msgs/microsec

-RRB-

fully

adaptive

Cm

Hot-spot

-LRB-

d

-RRB-

L

bytes

C

C

C

C

C

Figure

Impact

of

increasing

I-to-P

bandwidth

-LRB-

Cm

-RRB-

and

consumption

channels

-LRB-

C

-RRB-

on

system

performance

with

non-uniform

traffic

Summary

and

conclusion

In

this

paper

we

have

identified

performance

degradation

in

wormhole

k-ary

n-cube

networks

due

to

limited

number

of

router-to-processor

consumption

channels

at

each

node

It

is

shown

that

such

degradation

not

only

exists

for

hot-spot

traffic

but

also

for

uniform

traffic

in

systems

using

deterministic/adaptive

routing

with

multiple

virtual

lanes

Contrary

to

previous

results

in

the

literature

we

have

shown

that

the

advantages

of

adaptive

routing

and

virtual

lanes

are

very

much

limited

by

using

only

one

consumption

channel

per

node

We

have

proposed

a

new

network

interface

design

with

multiple

consumption

channels

and

virtual

multiplexing

to

alleviate

such

consumption

bottleneck

Our

study

indicates

that

multiple

consumption

channels

are

required

at

each

node

to

realize

the

full

benefits

of

routing

adaptivity

and

virtual

channel

flow

control

It

is

also

shown

that

higher

number

of

consumption

channels

are

required

to

alleviate

hot-spot

effects

in

nonuniform

patterns

By

examining

the

interplay

between

various

system

technological

and

traffic

parameters

we

have

established

design

guidelines

that

two

consumption

channels

are

useful

in

most

cases

Three

to

four

consumption

channels

have

shown

to

be

useful

for

systems

with

higher

adaptivity

low

messaging

overheads

high

I-to-P

bandwidth

and

hot

spot

traffic

We

have

also

shown

that

supporting

consumption

channels

is

realizable

under

current

router

technology

Thus

current

and

future

wormhole

k-ary

n-cube

systems

can

achieve

better

performance

by

using

multiple

consumption

channels

Acknowledgments

The

authors

are

grateful

to

anonymous

referees

for

their

valuable

comments

which

have

helped

us

in

improving

the

quality

of

this

paper

significantly

The

authors

are

also

grateful

to

Dr.

Craig

Stunkel

from

IBM

T.

J.

Watson

Research

Center

for

valuable

discussion

on

the

IBM

router-processor

interface

design

Additional

Information

A

number

of

related

papers

and

technical

reports

are

available

electronically

through

the

home

page

of

Parallel

Architecture

and

Communication

-LRB-

PAC

-RRB-

research

group

The

URL

is

Alternatively

the

papers

can

be

obtained

by

anonymous

ftp

from

ftp.cis.ohio-state

edu

cd

pub/communication

References

-LSB-

-RSB-

S.

Abraham

and

K.

Padmanabhan

Performance

of

the

Direct

Binary

n-cube

Network

for

Multiprocessors

IEEE

Transactions

on

Computers

pages

-LCB-

July

-LSB-

-RSB-

A.

Agrawal

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Oct

-LSB-

-RSB-

K.

V.

Anjan

and

T.

M.

Pinkston

An

efficient

fully

adaptive

deadlock

recovery

scheme

Disha

In

Proc

of

the

Int

Symp

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

S.

Balakrishnan

and

D.

K.

Panda

Impact

of

Multiple

Consumption

Channels

on

Wormhole

Routed

k-ary

n-cube

Networks

In

Proc

of

the

Int

Parallel

Processing

Symp.

pages

-LCB-

April

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Clustered

Multiprocessor

Systems

under

Packaging

and

Technological

Advancements

To

appear

in

IEEE

Trans

on

Parallel

and

Distributed

Systems

currently

available

at

-LSB-

-RSB-

M.

A.

Blumrich

and

K.

Li

et

al.

Virtual

Memory

Mapped

Network

Interface

for

the

SHRIMP

Multicomputer

In

Proc

of

the

Int

Symp

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

R.

V.

Boppana

and

S.

Chalasani

A

Comparison

of

Adaptive

Wormhole

Routing

Algorithms

In

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

Y.

M.

Boura

and

C.

R.

Das

Efficient

fully

adaptive

wormhole

routing

in

n-dimensional

meshes

In

Proc

of

the

Int

Conference

on

Distributed

Computing

Systems

pages

-LCB-

-LSB-

-RSB-

A.

A.

Chien

and

J.

H.

Kim

Planar

Adaptive

Routing

Low

cost

adaptive

networks

for

multiprocessors

In

Proc

Annual

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

S.

Chittor

and

R.

Enbody

Performance

Evaluation

of

Mesh-Connected

Wormhole-Routed

Net

works

for

Interprocessor

Communication

in

Multicomputers

In

Proceedings

of

the

Supercomputing

New

York

pages

-LCB-

Nov

-LSB-

-RSB-

L.

D.

Coster

N.

Dewulf

and

C.-T

Ho

Efficient

multi-packet

multicast

algorithms

on

meshes

with

wormhole

and

dimension-ordered

routing

In

Int

Conf

on

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

W.

J.

Dally

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Trans

on

Computers

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

W.

J.

Dally

Virtual-channel

flow

control

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

Deadlock-free

message

routing

in

multiprocessor

interconnection

networks

IEEE

Trans

on

Computers

-LCB-

May

-LSB-

-RSB-

S.

Dandamudi

and

D.

Eager

Hot-spot

contention

in

binary

hypercube

networks

IEEE

Trans

on

Computers

-LCB-

Feb

-LSB-

-RSB-

J.

J.

Dongarra

and

T.

Dunigan

Message-Passing

Performance

of

Various

Computers

Available

at

Aug

-LSB-

-RSB-

J.

Duato

A

New

Theory

of

Deadlock-Free

Adaptive

Routing

in

Wormhole

N

etworks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

T.

Eicken

D.

E.

Culler

S.

C.

Goldstein

and

K.

E.

Schauser

Active

Messages

a

Mechanism

for

Integrated

Communication

and

Computation

In

The

Annual

International

Symposium

on

Computer

Architecure

pages

-LCB-

May

-LSB-

-RSB-

C.

Stunkel

et

al.

The

High-Performance

Switch

In

Proc

of

the

Scalable

High-Performance

Computing

Conference

pages

-LCB-

-LSB-

-RSB-

C.

Stunkel

et

al.

The

High-Performance

Switch

IBM

Systems

Journal

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

J.

Kuskin

et

al.

The

stanford

flash

multiprocessor

In

Proc

of

the

Int

Symp

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

M.

Snir

et

al.

The

communication

software

and

parallel

environment

of

the

IBM

IBM

Systems

Journal

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

V.

Karamcheti

and

A.

A.

Chien

Software

Overhead

in

Messaging

Layers

Where

Does

the

Time

Go

In

Proc

of

the

Int

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pages

-LCB-

-LSB-

-RSB-

V.

Karamcheti

and

A.

A.

Chien

A

Comparison

of

Architectural

Support

for

Messaging

on

the

TMC

and

the

Cray

In

Proc

of

the

Int

Symp

on

Computer

Architecture

-LSB-

-RSB-

T.

Lang

and

L.

Kurisaki

Nonuniform

traffic

spots

-LRB-

nuts

-RRB-

in

multistage

interconnection

networks

Jour

of

Parallel

and

Distributed

Computing

-LCB-

-LSB-

-RSB-

X.

Lin

P.

K.

McKinley

and

L.

M.

Ni

Deadlock-free

multicast

wormhole

routing

in

mesh

multicomputers

IEEE

Trans

on

Parallel

and

Distributed

Systems

To

appear

-LSB-

-RSB-

D.

H.

Linder

and

J.

C.

Harden

An

adaptive

and

fault

tolerant

wormhole

routing

strategy

for

k-ary

n-cubes

IEEE

Trans

on

Computers

-LRB-

-RRB-

-LCB-

Jan.

-LSB-

-RSB-

L.

M.

Ni

and

P.

K.

McKinley

A

survey

of

wormhole

routing

techniques

in

direct

networks

IEEE

Computer

-LCB-

Feb

-LSB-

-RSB-

M.

D.

Noakes

D.

A.

Wallach

and

W.

J.

Dally

The

J-Machine

Multicomputer

An

Architectural

Evaluation

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

G.

F.

Pfister

and

V.

A.

Norton

Hot-spot

Contention

and

Combining

in

Multistage

Interconnection

Networks

IEEE

Transactions

on

Computers

-LCB-

-LSB-

-RSB-

G.

D.

Pifarre

L.

Gravano

S.

A.

Felperin

and

J.

Sanz

Fully

Adaptive

Minimal

Deadlock-Free

Routing

in

Hypercubes

Meshes

and

Other

Networks

Algorithms

and

Simulations

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

S.

Ramany

and

D.

Eager

The

Interaction

between

Virtual

Channel

Flow

Control

and

Adaptive

Routing

in

Wormhole

Networks

In

International

Conference

on

Supercomputing

pages

-LCB-

-LSB-

-RSB-

H.D.

Schwetman

Using

CSIM

to

Model

Complex

Systems

In

Proceedings

of

the

Winter

Simulation

Conference

pages

-LCB-

-LSB-

-RSB-

C.

B.

Stunkel

D.

G.

Shea

B.

Abali

M.

M.

Denneeau

P.

H.

Hochschild

D.

J.

Joseph

B.

J.

Nathanson

M.

Tsao

and

P.

R.

Varker

Architecture

and

Implementation

of

Vulcan

In

Proceedings

of

the

Int

l

Parallel

Processing

Symposium

pages

-LCB-

-LSB-

-RSB-

R.

Suaya

and

G.

Birtwistle

VLSI

and

Parallel

Computation

chapter

Morgan

Kaufman

-LSB-

-RSB-

P.

C.

Yew

N.

F.

Tzeng

and

D.

H.

Lawrie

Distributed

Hot-spot

Addressing

in

Large-scale

Multiprocessor

IEEE

Transactions

on

Computers

pages

-LCB-

Apr

adaptive routing
consumption channel
deterministic routing
interprocessor communication
performance evaluation
virtual channel
wormhole routing

hot-spot traffic
k-ary n-cube
parallel computer architecture