How

Much

Does

Network

Contention

Affect

Distributed

Shared

Memory

Performance

Donglai

Dai

and

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

E-mail

fdai,pandag@cis.ohio-state.edu

Contact

Author

Dhabaleswar

K.

Panda

Abstract

Many

research

results

in

recent

years

have

focused

on

the

design

of

distributed

shared

memory

-LRB-

DSM

-RRB-

systems

However

most

of

these

results

are

centered

around

either

careful

design

of

node

controllers

or

cache

coherence

protocols

While

evaluating

these

designs

simplified

models

of

networks

-LRB-

constant

network

latency

or

average

latency

based

on

the

network

size

-RRB-

are

typically

used

Such

models

completely

ignore

network

contention

and

thus

do

not

capture

the

latency

of

remote

memory

references

accurately

While

this

approach

may

seem

reasonable

for

designing

better

node

architectures

or

coherency

protocols

it

serves

poorly

for

understanding

and

solving

the

network

latency

bottleneck

problem

In

order

to

help

network

designers

to

design

better

networks

for

DSM

systems

in

this

paper

we

focus

on

two

major

goals

-RRB-

to

estimate

the

impact

of

network

link

contention

and

network

interface

contention

on

the

overall

performance

of

DSM

applications

and

-RRB-

to

study

the

impact

of

critical

architectural

parameters

on

network

contention

We

achieve

these

goals

by

evaluating

a

set

of

applications

from

the

widely

used

suite

on

a

closely

integrated

processor

and

network

simulator

for

a

CC-NUMA

system

The

simulator

models

the

processor

cache

and

memory

access

references

at

instruction

level

and

models

the

network

at

a

flit

transfer

level

A

series

of

three

network

models

are

proposed

to

isolate

network

link

contention

and

network

interface

contention

For

an

wormhole

system

the

study

indicates

that

network

contention

can

degrade

performance

up

to

Out

of

this

up

to

is

caused

by

network

interface

contention

alone

The

study

indicates

that

network

contention

becomes

dominant

for

DSM

systems

using

small

caches

wide

cache

line

sizes

low

degrees

of

associativity

high

processing

node

speeds

high

memory

speeds

low

network

speeds

or

small

network

widths

This

research

is

supported

in

part

by

NSF

Grant

and

NSF

Career

Award

Contents

Introduction

Modeling

Network

Contention

in

DSMs

CC-NUMA

and

its

Network

Contention

Existing

Network

Models

Used

in

Popular

DSM

Projects

Methodology

for

Evaluating

Network

Contention

No-Contention

Network

Model

-LRB-

NCM

-RRB-

Interface

Only

Network

Model

-LRB-

NIM

-RRB-

Detailed

Network

Model

-LRB-

DNM

-RRB-

Simulation

Environment

Simulated

Architecture

Implementing

Network

Models

No-Contention

Network

Model

-LRB-

NCM

-RRB-

Interface

Only

Network

Model

-LRB-

NIM

-RRB-

Detailed

Network

Model

-LRB-

DNM

-RRB-

Applications

Simulation

Results

and

Discussion

Network

Contention

in

Baseline

System

Impact

of

Cache

Design

on

Network

Contention

Effect

of

Cache

Size

per

Processing

Node

Effect

of

Cache

Line

Size

Effect

of

Cache

Associativity

Impact

of

Memory

Design

on

Network

Contention

Effect

of

Memory

Response

Time

Effect

of

Memory

Bus

Width

Effect

of

Node

Speed

Impact

of

Network

Design

on

Network

Contention

Effect

of

Network

Speed

Effect

of

Network

Flit

Width

Conclusion

Introduction

Tremendous

amount

of

effort

has

been

spent

in

recent

years

in

designing

and

developing

distributed

shared

memory

-LRB-

DSM

-RRB-

multiprocessors

While

many

hardware

and

software

challenges

encountered

in

small

to

moderate

scale

DSM

systems

have

been

studied

and

solved

research

remains

to

be

done

for

machines

which

can

be

efficiently

scaled

beyond

a

few

dozen

processors

Long

memory

latency

has

been

a

lead

impediment

to

achieving

the

full

performance

potential

of

such

systems

The

fundamental

reason

for

the

long

memory

latency

is

the

physical

distribution

of

the

computation

and

data

in

DSMs

Besides

the

speed

gap

between

a

processor

and

memory

two

dominant

factors

in

such

long

latency

are

-RRB-

overhead

of

the

cache

coherence

protocol

and

-RRB-

delays

in

the

underlying

communication

network

Previous

research

in

DSMs

has

mostly

concentrated

on

techniques

useful

for

designing

better

node

architectures

protocol

controllers

efficient

coherence

protocols

at

different

levels

in

the

system

or

applications

etc.

Almost

all

of

these

evaluations

are

based

on

the

assumption

of

a

point-to-point

communication

network

with

some

fixed

-LRB-

constant

or

a

function

of

the

network

diameter

-RRB-

latency

Representative

examples

include

various

memory

consistency

models

-LSB-

-RSB-

data

pre-fetching

-LSB-

-RSB-

data

forwarding/updating

-LSB-

-RSB-

remote

get/put

operations

-LSB-

-RSB-

integrated

or

decoupled

protocol

controllers

-LSB-

-RSB-

estimating

accuracy

vs.

performance

in

simulating

DSM

systems

-LSB-

-RSB-

software

DSM

systems

-LSB-

-RSB-

and

explicit

communication

primitives

-LSB-

-RSB-

Research

towards

reducing

network

latency

has

been

largely

left

to

the

-LRB-

interconnection

-RRB-

network

community

However

most

recently

several

papers

-LSB-

-RSB-

have

reported

that

network

latency

is

becoming

a

key

architectural

bottleneck

in

designing

large

scale

DSM

systems

after

integrating

some

of

the

above

techniques

Under

a

closer

examination

it

can

be

observed

that

network

latency

contains

two

components

minimal

communication

latency

and

contention

delays

Minimal

communication

latency

is

largely

defined

by

the

state

of

technology

in

circuit

speed

and

switch

design

Various

network

contention

delays

are

caused

by

limited

resources

like

queue

sizes

injection

channels

consumption

channels

link

contention

router

contention

etc

in

the

system

Current

and

next

generation

networks

promise

to

exploit

performance

aggressively

by

using

different

kinds

of

adaptive

routing

-LSB-

-RSB-

and

collective

communication

primitives

-LSB-

-RSB-

in

addition

to

higher

speed

The

essence

behind

these

communication

innovations

is

to

reduce

traffic

volume

and

network

contention

In

order

to

exploit

the

potential

performance

provided

by

such

new

networks

in

DSM

systems

some

major

design

issues

like

directory

organization

coherence

protocol

structure

of

protocol

engines

etc.

need

to

be

re-investigated

Other

issues

like

memory

consistency

models

data

pre-fetching/forwarding/updating

and

so

on

need

to

be

integrated

into

the

resulting

designs

Our

previous

research

-LSB-

-RSB-

has

applied

one

such

cost-effective

collective

communication

mechanism

i.e.

multidestination

message

passing

to

reduce

the

overhead

of

invalidations

in

wormhole

routed

DSM

systems

Despite

the

evidence

of

the

above

success

some

important

questions

remain

unclear

about

the

potential

performance

gain

by

employing

these

advanced

network

communication

mechanisms

in

DSM

systems

given

the

state-of-the-art

network

technology

In

this

paper

we

take

on

this

challenge

and

study

the

problem

of

network

contention

in

DSM

systems

Specifically

we

aim

to

answer

the

following

five

questions

ffl

For

machines

with

more

than

a

dozen

processors

how

much

does

network

contention

affects

the

performance

of

DSM

systems

when

running

realistic

applications

ffl

There

are

many

design

choices

for

cache

structure

on

each

node

How

does

each

design

parameter

i.e.

cache

size

line

size

set

associativity

etc.

affect

the

network

contention

ffl

Memory

controllers

have

become

increasingly

complicated

in

modern

node

designs

Without

loss

of

generality

we

consider

the

two

most

important

design

parameters

memory

response

delay

and

memory

bus

width

What

is

the

effect

of

each

of

them

on

the

network

contention

ffl

As

a

common

trend

processor

speeds

increase

much

sooner

than

the

speeds

of

any

other

components

in

a

computer

system

What

impact

does

the

increase

of

processor

speed

have

on

the

network

contention

ffl

In

contrast

to

increasing

the

processor

speed

a

network

with

smaller

minimum

communication

latency

or

wider

links

can

be

built

at

a

higher

cost

To

what

extent

will

such

a

design

reduce

network

contention

In

order

to

understand

the

network

contention

in

DSM

systems

and

to

answer

the

above

questions

we

first

classify

the

types

of

potential

network

contention

limited

queue/buffer

sizes

in

a

network

interface

-LRB-

sending

and

receiving

buffer

contention

-RRB-

injection

channel

contention

consumption

channel

contention

virtual

channel

and

physical

link

contention

etc.

After

reviewing

some

existing

network

models

used

in

popular

DSM

projects

we

describe

a

methodology

consisting

of

three

network

models

-LRB-

no-contention

network

interface

contention

only

network

and

detailed

network

-RRB-

to

isolate

and

evaluate

the

effect

of

different

types

of

network

contention

occurring

at

the

network

interface

and

within

the

network

respectively

The

principle

of

this

methodology

can

be

used

to

investigate

one

or

more

types

of

network

contention

in

a

DSM

system

We

perform

an

in-depth

quantitative

analysis

on

network

contention

for

a

CC-NUMA

architecture

using

a

wormhole

-LSB-

-RSB-

mesh

interconnection

network

We

consider

a

system

architecture

similar

to

the

FLASH

system

-LSB-

-RSB-

Our

simulator

models

the

processor

cache

and

memory

access

references

at

an

instruction

level

and

models

the

network

at

a

flit

transfer

level

The

evaluations

are

based

on

four

representative

applications

ported

from

the

suite

-LSB-

-RSB-

For

an

system

the

results

show

that

network

contention

can

degrade

overall

performance

of

DSM

applications

by

up

to

Out

of

this

less

than

is

caused

by

network

interface

contention

implying

that

the

contention

inside

the

network

alone

is

a

dominant

component

of

the

increase

in

network

latency

This

suggests

that

detailed

modeling

of

network

components

is

essential

to

estimate

accurate

performance

of

DSM

applications

We

also

evaluate

the

effect

of

network

contention

when

a

set

of

major

architectural

parameters

for

a

DSM

system

are

varied

These

parameters

include

cache

size

cache

line

size

cache

set

associativity

processing

node

speed

memory

speed

memory

bus

width

network

speed

and

network

width

It

is

shown

that

smaller

caches

larger

cache

lines

lower

set

associativity

higher

processing

node

speeds

higher

memory

speeds

lower

network

speeds

and

narrower

networks

can

significantly

increase

the

effect

of

network

contention

on

application

performance

These

results

indicate

that

variations

of

these

architectural

parameters

need

to

be

evaluated

together

with

detailed

modeling

of

network

components

to

derive

realistic

guidelines

for

designing

new

generation

DSM

systems

The

study

also

suggests

that

there

exists

great

potential

to

improve

the

performance

of

current

and

future

generation

DSM

systems

by

developing

techniques

for

reducing

network

contention

The

rest

of

the

paper

is

organized

as

follows

Section

characterizes

components

of

network

contention

in

DSM

systems

Section

provides

a

methodology

for

evaluating

network

contention

Details

of

simulation

environment

are

discussed

in

Section

Simulation

results

and

discussions

are

presented

in

Section

Concluding

remarks

are

made

in

Section

Modeling

Network

Contention

in

DSMs

Distributed

shared

memory

systems

generally

fall

into

three

categories

hardware

software

and

hybrid

The

fundamental

difference

between

these

categories

is

that

data

coherence

is

maintained

by

hardware

software

or

a

combination

of

the

two

All

of

them

solve

the

coherence

problem

based

on

similar

directory-based

protocols

In

this

section

we

present

our

study

using

a

particular

hardware

cache

coherent

DSM

with

a

wormhole

interconnection

network

-LRB-

commonly

referred

to

as

CC-NUMA

cache

coherent

non-uniform

memory

access

multiprocessor

-RRB-

as

an

example

However

the

principle

can

be

applied

to

other

DSMs

as

well

CC-NUMA

and

its

Network

Contention

A

typical

cache

coherent

DSM

machine

consists

of

a

set

of

identical

processing

nodes

connected

by

an

interconnection

network

Each

node

contains

a

processor

with

its

cache

a

portion

of

globally

shared

memory

a

node

controller

and

a

network

interface

Memory

is

organized

as

blocks

-LRB-

lines

-RRB-

and

each

node

is

the

home

node

to

the

memory

blocks

it

contains

A

directory

entry

is

associated

with

each

block

to

keep

track

of

the

current

sharing

information

of

the

block

in

the

system

The

node

controller

serves

as

the

central

control

point

within

a

node

and

enforces

the

coherence

protocol

by

handling

and

synchronizing

all

data

movement

into

or

out

of

the

node

based

on

the

directory

contents

The

network

interface

sends/receives

messages

into/from

the

interconnection

network

via

a

router

In

order

to

avoid

deadlocks

usually

two

separate

-LRB-

physical

or

virtual

-RRB-

networks

are

used

one

for

request

messages

and

the

other

for

response

messages

For

more

information

about

request

and

response

messages

and

the

associated

deadlock

scenarios

the

reader

is

requested

to

refer

to

-LSB-

-RSB-

Let

us

consider

the

transmission

of

a

message

from

one

node

to

another

as

illustrated

in

Fig.

Various

resources

are

required

for

this

to

succeed

First

space

must

be

available

for

the

message

to

be

constructed

in

the

sending

buffer

at

the

sender

's

network

interface

Such

space

may

not

be

available

because

of

a

number

of

reasons

such

as

-RRB-

disparity

between

the

processing

rates

of

the

node

controller

and

the

network

interface

-RRB-

link

contention

between

messages

in

some

part

of

the

network

etc.

Sending

buffer

contention

is

said

to

occur

when

a

message

is

blocked

due

to

the

lack

of

such

space

in

the

sending

buffer

nodecontroller

Network

contention

virtual

channel

contention

physical

link

contention

networkinterfacenetwork

receiving

buffer

contentionconsumption

channel

contention

sending

buffer

contentioninjection

channel

contention

Figure

Various

types

of

network

contention

in

a

DSM

system

Once

the

message

has

been

generated

an

injection

channel

must

be

obtained

to

copy

the

message

flit

by

flit

into

the

associated

router

Due

to

the

limited

number

of

injection

channels

a

network

interface

has

-LRB-

or

in

current

generation

DSM

machines

-RRB-

a

message

may

have

to

wait

for

the

injection

channels

because

they

are

being

used

for

transmitting

other

messages

Such

a

scenario

is

referred

to

as

injection

channel

contention

The

entire

message

moves

flit

by

flit

in

a

pipelined

fashion

from

the

source

to

its

destination

after

it

has

been

injected

into

the

network

Flits

hold

links

between

neighboring

routers

and

the

associated

buffers

in

the

routers

Handshaking

flow

control

signals

are

used

along

each

link

to

advance

the

flits

As

the

header

flit

of

the

message

passes

through

each

router

along

its

path

the

router

must

make

a

routing

decision

and

reserve

the

corresponding

outgoing

link

for

the

remaining

flits

If

the

outgoing

link

is

not

available

the

message

will

be

blocked

in

the

network

This

is

known

as

link

contention

-LRB-

or

virtual

channel

contention

if

multiple

virtual

channels

share

a

single

physical

link

-RRB-

In

case

multiple

virtual

channels

share

a

physical

link

if

more

than

one

virtual

channel

has

flits

ready

for

transfer

the

physical

link

is

multiplexed

among

the

channels

-LRB-

demandmultiplexing

-RRB-

This

effectively

reduces

the

moving

rate

of

each

message

involved

We

refer

to

this

scenario

as

physical

link

contention

After

the

header

flit

of

the

message

arrives

at

the

router

associated

with

the

destination

a

consumption

channel

in

the

network

interface

of

the

destination

node

must

be

obtained

to

copy

the

message

flit

by

flit

from

the

router

into

the

network

interface

Again

due

to

the

limited

number

of

consumption

channels

a

network

interface

has

-LRB-

or

in

current

generation

DSM

machines

-RRB-

a

message

may

be

blocked

because

the

consumption

channels

are

being

used

by

other

messages

This

is

known

as

consumption

channel

contention

Consumption

channel

contention

can

cause

considerably

increase

in

the

latency

of

a

message

under

hot-spot

traffic

Finally

a

receiving

buffer

in

the

destination

's

network

interface

must

be

obtained

to

assemble

the

entire

message

before

it

can

be

delivered

to

the

node

controller

If

there

is

no

receiving

buffer

available

the

message

will

again

be

blocked

while

holding

the

consumption

channel

and

other

links/virtual

channels

in

the

network

This

is

known

as

receiving

buffer

contention

It

is

clear

that

the

communication

delay

of

a

message

can

be

significantly

affected

by

any

of

the

above

types

of

contention

However

due

to

the

temporal

nature

of

network

contention

it

is

usually

very

difficult

to

measure

its

effect

on

a

real

DSM

machine

Even

if

it

were

possible

to

collect

data

on

the

network

contention

in

a

real

DSM

machine

it

is

impossible

to

compare

the

results

from

two

different

machines

This

is

because

of

-LRB-

i

-RRB-

the

differences

in

theirs

design

and

implementation

and

-LRB-

ii

-RRB-

the

fact

that

different

machines

have

different

networks

that

affect

the

types

and

mixes

of

contention

even

when

everything

else

is

the

same

Some

analytical

contention

models

have

been

used

before

These

models

often

require

parameters

like

average

number

of

messages

generated

per

node

average

length

of

a

message

average

distance

a

message

travels

etc.

to

predict

the

corresponding

network

latency

From

a

DSM

system

perspective

a

major

flaw

of

such

an

analytical

model

is

that

it

does

not

consider

the

crucial

dependency

between

computation

and

communication

in

an

application

This

leaves

simulation

as

the

only

plausible

approach

for

estimating

the

effect

of

various

types

of

network

contention

Existing

Network

Models

Used

in

Popular

DSM

Projects

Due

to

the

complexity

of

modeling

a

network

most

of

the

research

work

in

DSMs

has

ignored

network

contention

partially

or

entirely

The

original

WWT

-LSB-

-RSB-

is

a

parallel

DSM

simulator

developed

by

the

Wisconsin

Wind

Tunnel

research

group

In

this

simulator

a

constant

network

latency

of

processor

cycles

is

assigned

for

every

message

independent

of

the

length

of

the

message

the

distance

traveled

and

other

traffic

in

the

network

A

DSM

simulator

used

in

the

Stanford

FLASH

-LSB-

-RSB-

research

group

models

the

network

with

slightly

higher

accuracy

network

interfaces

are

modeled

in

this

simulator

The

network

latency

of

every

message

is

calculated

based

on

the

length

of

the

message

and

half

of

the

diameter

of

the

network

The

Typhoon

-LSB-

-RSB-

simulator

from

the

same

Wisconsin

Wind

Tunnel

group

models

the

network

interface

and

assigns

processor

cycles

to

every

message

as

the

delay

of

the

message

inside

the

network

of

these

DSM

simulators

has

modeled

the

channel

contention

or

physical

link

contention

Our

results

in

Section

show

that

these

types

of

contention

inside

the

network

dominate

the

overall

network

contention

When

network

latency

is

not

a

performance

bottleneck

such

an

approach

is

valid

and

it

has

helped

the

community

discover

many

valuable

techniques

and

mechanisms

for

DSM

node

design

However

when

the

network

becomes

a

bottleneck

this

approach

fails

to

offer

the

designers

insight

into

the

reasons

for

this

bottleneck

network

contention

is

one

of

the

most

important

of

these

reasons

Isolating

the

impact

of

the

various

types

of

network

contention

helps

an

architect

identify

the

design

choices

for

improving

network

performance

such

as

routing

adaptivity

network

support

for

collective

communication

etc.

In

next

section

we

propose

some

modifications

to

these

network

models

which

can

provide

us

useful

information

about

the

various

types

of

network

contention

Methodology

for

Evaluating

Network

Contention

In

this

section

we

describe

our

methodology

for

isolating

and

evaluating

the

impact

of

various

types

of

network

contention

that

occur

in

a

DSM

system

The

basic

idea

is

to

construct

a

series

of

network

models

Each

of

these

simulates

the

network

at

an

increasing

level

of

detail

and

is

driven

by

the

same

DSM

node

and

memory

simulator

while

collecting

the

statistical

data

using

exactly

the

same

input

Comparing

the

predicted

performance

metrics

of

any

two

such

DSM

simulators

the

difference

tells

us

the

effect

of

types

of

contention

which

are

modeled

by

the

more

sophisticated

simulator

For

this

study

our

main

goal

is

to

evaluate

the

overall

performance

impact

of

various

types

of

network

contention

Secondarily

we

want

to

examine

the

performance

impact

of

only

those

kinds

of

contention

that

occur

inside

the

network

interfaces

Once

the

overall

performance

impact

of

various

types

of

network

contention

and

the

impact

of

network

interface

contention

have

been

determined

the

impact

of

the

kinds

of

contention

occurring

inside

the

network

alone

can

be

inferred

To

achieve

these

goals

we

construct

a

series

of

three

network

models

as

described

below

No-Contention

Network

Model

-LRB-

NCM

-RRB-

This

model

is

an

enhancement

of

the

network

model

used

in

the

original

WWT

simulator

-LSB-

-RSB-

It

assumes

-RRB-

infinite

number

of

sending

buffers

-RRB-

infinite

number

of

injection

channels

-RRB-

infinite

number

of

consumption

channels

-RRB-

infinite

number

of

receiving

buffers

and

-RRB-

no

traffic

interference

inside

the

network

These

assumptions

guarantee

that

no

network

contention

can

ever

occur

during

any

communication

Unlike

the

WWT

in

our

model

each

message

is

delayed

by

exactly

the

total

time

for

construction

network

propagation

and

consumption

The

construction

time

and

consumption

time

vary

depending

on

the

length

of

the

message

The

network

propagation

time

is

a

function

of

both

the

length

and

the

distance

traveled

by

the

message

Interface

Only

Network

Model

-LRB-

NIM

-RRB-

The

NIM

model

is

an

enhancement

of

the

network

model

used

in

the

FLASH

project

-LSB-

-RSB-

The

NIM

model

simulates

detailed

management

of

the

limited

number

of

sending

buffers

receiving

buffers

injection

channels

and

consumption

channels

The

NIM

model

still

assumes

no

traffic

interference

inside

the

network

Such

a

model

captures

the

types

of

contention

occurring

within

a

network

interface

For

every

message

if

there

is

no

contention

within

a

network

interface

the

message

is

delayed

in

this

model

by

exactly

the

same

amount

of

time

as

in

the

NCM

model

It

is

clear

that

any

difference

in

the

predicted

performance

between

this

model

and

the

NCM

model

is

caused

by

contention

occurring

inside

the

network

interfaces

during

application

execution

Detailed

Network

Model

-LRB-

DNM

-RRB-

The

DNM

model

simulates

every

possible

type

of

contention

in

the

network

It

simulates

the

detailed

management

of

transmission

links

and

intermediate

switches

-LRB-

at

a

flit

level

-RRB-

in

additional

to

modeling

the

network

interface

The

flow

control

mechanism

used

by

the

network

must

be

modeled

as

closely

as

possible

For

example

the

handshaking

signals

in

an

asynchronous

network

must

be

modeled

For

every

message

that

is

not

involved

in

contention

inside

the

network

the

message

is

delayed

by

exactly

the

same

amount

of

time

as

in

the

NIM

model

Such

a

model

captures

all

types

of

contention

within

the

network

interfaces

and

every

part

of

network

It

is

clear

that

any

difference

in

the

predicted

performance

between

this

model

and

the

NCM

model

is

caused

by

contention

occurring

in

the

entire

network

-LRB-

including

the

network

interfaces

-RRB-

Any

difference

in

the

predicted

performance

between

this

model

and

the

NIM

model

is

caused

by

contention

occurring

inside

network

alone

-LRB-

excluding

the

network

interfaces

-RRB-

The

relations

among

the

above

three

network

models

is

shown

in

Fig.

It

is

to

be

noted

that

this

methodology

is

general

and

applicable

for

evaluating

the

impact

of

network

contention

in

any

type

of

DSM

system

hardware

software

and

hybrid

In

following

section

we

describe

in

much

more

detail

the

specific

DSM

simulation

implementation

which

we

used

for

collecting

all

the

results

presented

in

this

paper

NCMNIMDNM

Network

contention

Network

interface

contention

Contention-free

Figure

Relations

among

the

network

models

-LRB-

NCM

NIM

DNM

-RRB-

Simulation

Environment

In

order

to

apply

above

methodology

for

evaluating

the

impact

of

network

contention

in

a

DSM

system

we

simulate

a

machine

with

an

architecture

and

coherence

protocol

similar

to

the

FLASH

machine

-LSB-

-RSB-

but

with

some

important

differences

described

later

In

this

section

we

first

describe

the

important

details

of

the

simulated

processing

node

and

each

of

the

network

models

that

are

used

and

then

briefly

discuss

the

applications

in

our

experiments

We

describe

a

baseline

configuration

whose

architectural

parameters

have

the

set

of

values

defined

in

this

section

In

next

section

when

we

study

the

effect

of

each

important

architectural

parameter

on

network

contention

we

will

specify

the

differences

from

this

baseline

configuration

Simulated

Architecture

The

hardware

cache

coherent

multiprocessor

we

simulate

is

illustrated

in

Fig.

The

machine

has

processing

nodes

The

processor

in

each

processing

node

is

assumed

to

be

a

MHz

single-issue

microprocessor

with

a

perfect

instruction

cache

and

a

KB

set

associative

data

cache

with

a

line

size

of

bytes

The

cache

is

assumed

to

operate

in

dual-port

mode

using

write-back

and

write-allocate

policies

The

instruction

latencies

issue

rules

and

memory

interface

are

modeled

based

on

the

DLX

design

-LSB-

-RSB-

The

memory

bus

is

assumed

to

be

bytes

wide

On

a

memory

block

access

the

first

word

of

the

block

is

returned

in

processor

cycles

-LRB-

ns

-RRB-

The

successive

words

in

the

block

follow

in

a

pipelining

fashion

The

machine

is

assumed

to

be

using

a

full-mapped

invalidation-based

three-state

directory

coherence

protocol

-LSB-

-RSB-

The

node

simulator

models

the

internal

structures

and

the

queuing

and

contention

at

the

node

controller

main

memory

and

cache

The

sizes

of

the

queues

between

the

internal

structures

and

the

appropriate

actions

taken

if

a

queue

is

full

are

shown

in

Table

Following

the

assumptions

in

-LSB-

-RSB-

the

node

controller

incurs

a

small

fixed

occupancy

for

purely

generating

a

request

message

into

the

network

The

same

occupancy

is

also

incurred

for

receiving

a

response

message

from

the

network

The

node

controller

at

the

home

node

of

a

remote

request/response

incurs

a

higher

fixed

occupancy

because

data

-LRB-

in

most

cases

-RRB-

and

directory

information

must

be

retrieved

and

manipulated

In

such

a

case

the

memory

access

is

assumed

to

proceed

in

parallel

with

the

node

controller

The

node

controller

also

incurs

extra

occupancy

per

invalidation

sent

In

a

scenario

where

the

processor

of

a

node

has

a

load/store

miss

to

a

clean

block

whose

home

node

is

the

node

itself

we

assume

the

memory

module

can

be

pipelined

with

the

cache

Thus

the

retrieve

of

the

block

the

filling

of

the

cache

line

and

the

manipulation

of

the

directory

information

are

overlapped

Table

shows

the

memory

hierarchy

parameters

and

node

controller

occupancy

delays

used

in

our

simulated

architecture

Table

Default

sizes

of

various

queues

in

a

simulated

node

controller

Queue

Size

Action

if

full

Incoming

network

queue

messages

Messages

blocked

in

the

network

Outgoing

network

queue

messages

Node

controller

stalls

Memory

request

queue

request

Node

controller

stalls

Incoming

processor

queue

messages

Processor

stalls

Outgoing

processor

queue

messages

Node

controller

stalls

The

major

difference

between

our

simulated

architecture

and

the

FLASH

architecture

are

as

follows

sequential

memory

consistency

instead

of

the

release

memory

consistency

used

in

the

Router

links

fromother

routers

links

toother

routers

Node

NetworkInterface

Memory

NodeControllerCacheProcessor

CommunicationSubstrate

Figure

Simulated

node

architecture

and

interconnection

network

FLASH

architecture

is

enforced

by

the

coherence

protocol

Therefore

at

most

one

outstanding

request

can

be

issued

by

a

processor

A

load/store

miss

stalls

the

processor

until

the

first

word

of

data

is

returned

Implementing

Network

Models

We

assume

the

processing

nodes

in

the

machine

are

connected

with

a

pair

of

virtual

networks

sharing

a

physical

wormhole

network

-LSB-

-RSB-

The

physical

network

is

assumed

to

operate

at

a

frequency

of

MHz

The

method

adopted

for

modeling

the

network

by

each

of

the

network

simulators

is

described

in

detail

in

following

three

subsections

No-Contention

Network

Model

-LRB-

NCM

-RRB-

The

NCM

network

simulator

is

quite

simple

It

takes

a

message

from

the

node

controller

at

the

source

node

and

informs

the

node

controller

at

the

destination

node

that

a

message

has

arrived

after

a

time

delay

equal

to

the

calculated

network

latency

of

the

message

The

network

latency

of

a

message

is

calculated

as

latency

Toutgoing

-LRB-

Trout

Tlink

-RRB-

distance

-LRB-

Tsw

Tlink

-RRB-

length

Tincoming

where

distance

is

the

number

of

hops

from

the

source

to

the

destination

and

length

is

the

length

of

the

message

in

flits

Toutgoing

is

the

time

delay

for

a

message

to

be

constructed

by

the

network

interface

Trout

is

the

time

needed

for

a

router

to

make

a

routing

decision

based

on

its

knowledge

about

the

network

and

the

requirement

of

a

message

contained

in

the

header

flit

Tlink

is

the

time

for

any

flit

of

a

message

to

propagate

over

a

single

physical

link

Tsw

is

the

time

for

a

non-header

flit

of

a

message

to

pass

through

a

router

The

parameters

used

in

our

simulation

for

these

network

delays

are

listed

in

Table

However

for

a

simple

three-state

directory-based

coherence

protocol

such

an

implementation

will

cause

memory

inconsistency

due

to

the

non-FIFO

delivery

between

a

source

and

destination

pair

Following

the

method

described

in

-LSB-

-RSB-

we

modeled

additional

end-to-end

flow

control

to

guarantee

the

FIFO

property

of

message

delivery

between

each

pair

of

nodes

It

is

to

be

noted

that

the

incoming/outgoing

network

queue

size

at

a

network

interface

listed

in

Table

is

ignored

in

the

NCM

network

simulator

-LRB-

i.e.

these

queues

are

assumed

to

be

of

infinite

length

as

described

in

section

-RRB-

Interface

Only

Network

Model

-LRB-

NIM

-RRB-

This

simulator

takes

into

account

detailed

modeling

and

timing

delays

associated

with

structures

such

as

the

sending

buffers

receiving

buffers

injection

channels

and

consumption

channels

within

the

network

interface

For

a

node

to

send

a

message

a

space

in

the

sending

buffer

must

be

reserved

for

Toutgoing

time

to

construct

the

message

Once

the

message

is

constructed

if

there

is

an

injection

channel

available

it

is

reserved

for

-LRB-

Tlink

Tsw

-RRB-

length

amount

of

time

to

pump

the

entire

message

into

the

network

At

exactly

-LRB-

Tlink

Trout

-RRB-

distance

time

after

the

header

flit

of

the

message

is

injected

into

the

network

at

the

network

interface

of

the

source

node

a

consumption

channel

and

a

space

in

the

receiving

buffer

at

the

network

interface

of

the

destination

node

is

reserved

for

-LRB-

Tlink

Tsw

-RRB-

length

time

Once

the

message

has

been

consumed

entirely

into

the

receiving

buffer

the

node

controller

at

the

receiving

node

is

informed

about

its

arrival

after

Tincoming

amount

of

time

In

case

any

of

the

above

reservations

can

not

succeed

-LRB-

because

of

other

messages

to

the

network

-RRB-

the

message

can

not

move

forward

and

continues

to

hold

the

resources

it

has

acquired

It

is

to

be

noted

that

such

a

network

simulator

guarantees

the

FIFO

property

of

message

delivery

between

each

pair

of

nodes

if

there

is

only

one

injection

and

one

consumption

channel

at

each

network

interface

Detailed

Network

Model

-LRB-

DNM

-RRB-

The

implementation

of

the

DNM

network

simulator

is

the

most

sophisticated

one

It

models

the

internal

structures

such

as

data

links

channel

buffers

signal

lines

etc.

within

and

between

the

routers

in

addition

to

the

network

interface

It

accurately

simulates

the

mechanisms

for

wormhole

switching

such

as

distributed

routing

book-keeping

on

channel

status

and

flit-level

asynchronous

ready/empty

handshaking

For

detailed

information

about

wormhole

routing

the

reader

is

requested

to

refer

to

-LSB-

-RSB-

In

this

simulator

when

a

message

is

first

injected

into

the

network

the

header

flit

of

the

message

reserves

the

channels

in

the

routers

for

the

remaining

flits

of

the

message

while

moving

forward

along

its

path

from

the

source

to

the

destination

When

the

header

moves

into

a

router

a

delay

of

Trout

is

incurred

for

the

routing

decision

If

the

-LRB-

desired

-RRB-

outgoing

channel

-LRB-

decided

by

the

routing

scheme

-RRB-

is

available

the

header

flit

reserves

that

channel

and

reaches

the

next

router

after

a

time

elapse

of

Tlink

Each

non-header

flit

of

the

message

can

move

forward

only

when

its

immediate

predecessor

has

already

moved

as

detected

using

the

asynchronous

handshaking

signals

Every

non-header

flit

incurs

a

latency

of

Tsw

at

every

router

and

Tlink

at

every

link

on

the

path

to

the

message

destination

The

tail

flit

of

the

message

releases

the

previous

channel

when

it

leaves

each

router

When

two

virtual

channels

over

the

same

physical

link

are

both

transmitting

messages

the

physical

link

is

demand-multiplexed

resulting

in

slowdown

of

both

messages

It

is

to

be

noted

that

such

a

network

simulator

guarantees

the

FIFO

property

of

message

delivery

between

each

pair

of

nodes

because

of

the

simulated

dimension

order

routing

scheme

Table

summarizes

the

default

values

for

all

important

parameters

used

in

the

simulation

Table

shows

the

contention-free

latency

of

a

typical

memory

miss

to

a

neighboring

home

node

To

illustrate

how

the

parameters

in

Table

get

translated

into

the

times

in

Table

we

show

the

time

breakdown

of

contention-free

remote

read

miss

to

a

neighboring

home

node

in

Table

These

memory

latencies

are

comparable

to

the

latencies

presented

in

-LSB-

-RSB-

reflecting

the

accuracy

of

our

simulations

Applications

We

use

four

applications

Barnes

LU

Radix

and

Water

in

our

simulation

evaluation

All

these

are

real

applications

and

challenging

computational

kernels

ported

from

the

widely

used

suite

Table

gives

the

basic

traffic

pattern

and

input

sizes

for

the

various

applications

These

applications

are

well-structured

and

heavily

optimized

for

DSM

machines

For

more

information

about

the

applications

the

reader

is

requested

to

refer

to

-LSB-

-RSB-

Barnes

is

an

application

representative

of

the

class

of

hierarchical

N-body

methods

used

in

astrophysics

electrostatics

and

plasma

physics

among

others

In

this

application

a

globally

shared

space-motion-mass

tree

-LRB-

its

primary

data

structure

-RRB-

is

continually

reconstructed

and

repartitioned

at

every

time-step

Particles

are

assigned

to

each

processor

and

accessed

through

pointer

arrays

The

communication

pattern

in

Barnes

is

hierarchical

and

irregular

LU

is

the

computation

kernel

of

lower

and

upper

matrix

decomposition

It

represents

a

wide

range

of

dense

linear

algebra

applications

The

n?n

input

matrix

of

double

precision

numbers

is

converted

into

to

a

N?N?B?B

matrix

for

purposes

of

reducing

communication

and

load

balancing

The

elements

assigned

to

the

same

processor

are

allocated

contiguously

to

improve

spatial

locality

The

communication

pattern

is

one-to-many

and

largely

regular

Table

Default

system

parameters

used

in

simulation

Memory

Hierarchy

Parameters

Processor

frequency

Cache

access

cycle

Cache

line

size

-LRB-

L

-RRB-

bytes

Cache

set

associativity

Cache

size

per

node

Kbytes

Memory

word

width

-LRB-

W

-RRB-

bytes/cycle

Memory

response

delay

cycles

Cache

block

fill

time

L/W

cycles

Memory

block

access

time

L/W

cycles

Node

Controller

Occupancy

Directory

check

cycles

Directory

check

and

update

cycles

Each

invalidation

request

cycles

Message

forward

cycles

Network

Interface

Parameters

Outgoing

message

startup

-LRB-

Toutgoing

-RRB-

cycles

Incoming

message

dispatch

-LRB-

Tincoming

-RRB-

cycles

Control

message

size

bytes

Data

message

size

bytes

Injection

channels

per

node

Consumption

channels

per

node

Network

Parameters

Network

frequency

Channel

width

Flit

size

bytes

Link

Propagation

-LRB-

Tlink

-RRB-

network

cycle

Router

switch

delay

-LRB-

Tsw

-RRB-

network

cycle

Routing

time

-LRB-

Trout

-RRB-

network

cycles

Physical

network

Virtual

networks

Virtual

channels

per

virtual

network

Table

Contention-free

latencies

of

typical

memory

miss

to

neighboring

nodes

in

cycles

Memory

Home

Directory

Inv

or

Access

Location

State

CpyBack

Latency

local

not

cached

remote

not

cached

Load

remote

-LRB-

-RRB-

home

dirty

remote

-LRB-

-RRB-

remote

dirty

local

not

cached

local

remote

shared

Store

remote

remote

shared

remote

-LRB-

-RRB-

home

shared

remote

-LRB-

-RRB-

remote

shared

remote

-LRB-

-RRB-

remote

dirty

Table

Latency

break

down

of

a

clean

read-miss

to

neighboring

node

in

cycles

Sub-operation

Latency

Cache-miss

to

request

in

the

network

Request

transmit

time

-LRB-

bytes

-RRB-

Request

at

the

home

node

to

output

header

transmit

Data

return

in

network

-LRB-

bytes

-RRB-

Response

arrival

to

beginning

of

cache

fill

Cache

fill

Total

Table

Applications

and

default

problem

sizes

Communication

Application

Description

Pattern

Problem

Size

Barnes

Hierarchical

N-body

irregular

hierarchical

particles

galaxy

simulation

time

steps

LU

Blocked

dense

one-to-many

doubles

linear

algebra

regular

blocks

Radix

Integer

radix

sort

all-to-all

irregular

keys

radix

max

Water

N-body

molecular

near

neighbor

molecules

dynamics

simulation

irregular

time

steps

Radix

is

the

integer

radix

sort

kernel

a

major

task

in

data

base

applications

It

iteratively

creates

local

histograms

by

permuting

keys

and

by

merging

them

into

a

global

histogram

The

permutation

phase

requires

depending

on

the

key

values

all-to-all

irregular

communication

-LRB-

in

the

worst

case

-RRB-

W

ater

is

an

N-body

molecular

dynamics

application

It

evaluates

forces

and

potentials

in

a

system

of

water

molecules

over

a

period

of

time

The

main

data

structure

is

an

array

of

the

molecules

in

the

system

It

adopts

a

predictor-corrector

method

-LRB-

O

-LRB-

-RRB-

algorithm

-RRB-

to

compute

the

forces

and

potentials

in

each

time-step

Irregular

communication

occurs

when

the

cell

list

is

updated

which

in

turn

is

caused

by

the

movement

of

molecules

into

and

out

of

cells

Simulation

Results

and

Discussion

We

now

present

the

results

of

our

simulations

for

the

applications

described

above

In

Section

we

evaluate

the

contention

within

the

network

interface

and

inside

the

network

Sections

and

examine

the

impact

of

alternate

designs

of

cache

and

memory

on

network

contention

Section

investigates

the

impact

of

processing

node

speed

Finally

we

evaluate

the

impact

of

different

designs

of

the

network

itself

in

Section

For

convenience

of

reference

we

identify

the

three

DSM

simulators

described

in

the

previous

section

by

the

network

model

each

one

uses

All

results

are

reported

using

the

total

execution

time

normalized

to

that

time

predicted

by

the

NCM

simulator

for

the

same

application

Each

such

time

is

further

broken

down

into

four

categories

processor

computation

time

-LRB-

Computation

-RRB-

read

stall

time

-LRB-

Read

-RRB-

write

stall

time

-LRB-

Write

-RRB-

and

synchronization

stall

time

-LRB-

Sync

-RRB-

Network

Contention

in

Baseline

System

Figure

shows

the

execution

time

of

each

application

using

the

the

NCM

NIM

and

DNM

simulator

respectively

in

the

DSM

system

using

the

baseline

configuration

As

expected

network

contention

slows

down

every

application

However

the

exact

effect

depends

on

application

ranging

from

-LRB-

in

Water

-RRB-

to

-LRB-

in

Radix

-RRB-

Radix

incurs

such

high

network

contention

because

of

the

all-to-all

irregular

communication

occurring

during

its

histogram

merge

phase

It

is

clear

that

communication

contention

enlarges

the

stalling

time

of

each

category

in

every

application

The

results

show

that

contention

within

the

network

interface

alone

only

accounts

for

a

small

portion

of

total

communication

contention

less

than

in

our

experiments

In

other

words

contention

inside

the

network

is

the

dominant

cause

of

higher

network

latency

This

observation

remains

true

for

all

our

experiments

Normalized

Execution

TimeSynchWriteReadComputationNCMNIMDNMBarnesNCMNIMDNMLUNCMNIMDNMRadixN

CMNIMDNMWater

Figure

Breakdown

of

application

overall

execution

time

on

the

baseline

system

Impact

of

Cache

Design

on

Network

Contention

Let

us

now

examine

the

impact

of

alternate

design

choices

of

processing

node

cache

on

the

network

contention

Sections

and

respectively

look

at

this

issue

by

changing

the

cache

size

cache

line

size

and

set

associativity

Effect

of

Cache

Size

per

Processing

Node

Figure

depicts

the

execution

time

breakdowns

for

the

experiments

with

and

cache

respectively

Two

observations

are

noteworthy

First

as

the

cache

size

increases

the

network

contention

in

each

application

reduces

to

a

constant

amount

-LRB-

approximately

-RRB-

This

is

because

when

cache

is

large

enough

the

largest

working

set

for

an

application

can

fit

into

the

cache

completely

Our

results

show

that

even

in

such

a

case

the

slowdown

caused

by

communication

contention

ranges

from

-LRB-

in

Water

-RRB-

to

-LRB-

in

Radix

-RRB-

This

also

shows

that

our

baseline

configuration

gives

us

a

rather

conservative

estimate

of

the

network

Second

when

the

cache

size

decreases

the

network

contention

becomes

more

significant

The

primary

reason

for

this

is

the

extra

traffic

in

the

network

caused

by

cache

capacity

misses

This

is

true

even

when

the

smallest

working

set

of

an

application

still

fits

in

the

cache

Effect

of

Cache

Line

Size

Figure

shows

the

execution

times

for

the

experiments

with

cache

line

size

of

and

bytes

respectively

in

the

same

KB

cache

As

the

cache

line

grows

larger

the

network

contention

in

all

application

shows

the

bath-tub

behavior

i.e.

first

decreasing

to

a

pollution

point

and

then

increasing

significantly

The

surprising

discovery

from

these

results

is

that

the

pollution

point

is

relatively

small

around

or

bytes

per

line

for

three

out

of

the

four

applications

The

reason

for

this

is

as

follows

When

the

cache

line

increases

the

reply

messages

which

contain

data

grow

longer

The

length

increase

in

this

type

of

message

causes

increase

in

the

time

required

for

transmitting

them

thus

longer

time

is

spent

on

the

links

and

routers

causing

more

network

contention

Such

an

effect

not

only

slows

down

a

message

of

this

type

but

also

increases

the

latency

of

messages

of

other

types

Similar

to

a

uniprocessor

system

longer

cache

line

exploits

spatial

locality

existing

in

an

application

at

the

cost

of

reducing

the

benefits

of

temporal

locality

The

effect

of

data

pre-fetching

because

of

longer

cache

line

can

reduce

the

number

of

cache

misses

However

the

increase

in

cache

line

size

does

not

reduce

is

because

the

baseline

configuration

assumes

a

cache

size

large

enough

to

hold

the

largest

working

set

of

any

of

the

applications

we

used

This

represents

a

system

that

generates

very

little

cache-capacity

miss

traffic

and

therefore

underestimates

network

contention

Normalized

Execution

Normalized

Execution

Normalized

Execution

Normalized

Execution

Figure

Effect

of

cache

size

per

node

on

network

contention

the

number

of

messages

by

proportion

after

when

the

lines

size

is

long

enough

In

DSM

this

leads

to

an

increase

in

overall

volume

of

network

traffic

because

of

unnecessary

data

pre-fetching

results

in

more

network

contention

and

the

network

latency

increases

explosively

Such

an

effect

can

be

observed

by

examining

the

difference

between

the

results

of

NCM

and

DNM

simulators

for

Radix

the

proportion

of

the

network

latency

in

the

overall

execution

time

grows

from

for

a

byte

cache

line

to

for

a

byte

cache

line

and

to

for

a

byte

cache

line

The

reason

that

Radix

shows

this

effect

earlier

than

other

applications

is

because

the

communication

to

computation

ratio

is

higher

in

this

application

because

of

the

histogram

merging

performed

Normalized

Execution

Normalized

Execution

Normalized

Execution

Normalized

Execution

Figure

Effect

of

cache

line

size

on

network

contention

Effect

of

Cache

Associativity

The

execution

times

of

the

applications

when

cache

is

organized

as

direct-mapped

and

associative

are

shown

in

Fig.

When

the

set

associativity

is

equal

to

or

greater

than

the

proportion

of

network

contention

remains

almost

constant

in

each

application

When

the

cache

is

organized

as

direct-mapped

the

proportion

of

network

contention

increases

in

all

applications

because

the

cache

mapping

conflicts

increase

the

network

traffic

Our

experiments

show

that

a

-LRB-

Barnes

-RRB-

-LRB-

LU

-RRB-

-LRB-

Radix

-RRB-

-LRB-

Water

-RRB-

performance

degradation

occurs

for

a

direct-mapped

cache

of

size

Normalized

Execution

Normalized

Execution

Normalized

Execution

Normalized

Execution

Figure

Effect

of

cache

associativity

on

network

contention

In

summary

network

contention

is

sensitive

to

the

design

choices

of

cache

in

a

processing

node

This

is

especially

true

for

cache

line

size

selection

Overall

alternative

cache

designs

other

than

our

baseline

configuration

tend

to

increase

the

impact

of

network

contention

in

a

DSM

system

Impact

of

Memory

Design

on

Network

Contention

Let

us

now

evaluate

the

impact

of

alternative

design

choices

of

the

memory

module

in

the

processing

node

on

the

network

contention

We

focus

on

the

two

primary

memory

metrics

memory

response

time

-LRB-

in

Section

-RRB-

and

memory

bus

width

-LRB-

in

Section

-RRB-

Effect

of

Memory

Response

Time

Memory

response

time

is

the

time

interval

from

the

issue

of

memory

access

command

to

the

time

when

the

first

word

in

a

memory

block

is

available

for

use

The

experimental

results

shown

in

Fig.

are

for

different

response

time

of

processor

cycles

As

much

as

performance

degradation

occurs

in

in

Barnes

in

LU

in

Radix

and

in

Water

for

a

memory

response

time

of

processor

clock

cycles

As

the

memory

responds

faster

the

network

contention

increases

slightly

in

all

applications

because

the

request

and

reply

messages

are

generated

in

a

shorter

time

resulting

in

greater

traffic

congestion

Effect

of

Memory

Bus

Width

Memory

bus

width

decides

the

amount

of

time

needed

for

finishing

an

access

to

a

memory

block

after

the

first

word

is

available

The

experimental

results

shown

in

Fig.

are

for

different

memory

bus

widths

of

and

bytes

In

these

experiments

we

use

a

block

-LRB-

line

-RRB-

size

of

bytes

As

the

memory

bus

becomes

wider

the

network

contention

barely

changes

in

each

application

because

of

the

assumption

of

memory

pipelining

Generally

speaking

when

the

memory

module

becomes

faster

in

a

node

the

network

contention

increases

Between

memory

response

time

and

memory

bus

width

the

former

has

a

stronger

impact

on

the

network

contention

Effect

of

Node

Speed

In

this

section

we

examine

the

impact

of

node

speed

on

the

network

contention

Our

system

is

assumed

to

use

an

integrated

node

controller

Therefore

when

the

node

speed

increases

Normalized

Execution

Normalized

Execution

LU

Normalized

Execution

Normalized

Execution

Figure

Effect

of

memory

response

time

on

network

contention

Normalized

Execution

bytesBarnes

Normalized

Execution

bytesLU

Normalized

Execution

bytesRadix

Normalized

Execution

bytesWater

Figure

Effect

of

memory

bus

width

on

network

contention

all

parts

in

a

node

-LRB-

processor

cache

memory

node

controller

and

network

interface

-RRB-

are

faster

The

results

in

Fig.

show

that

the

network

contention

increases

considerably

in

most

application

Specifically

the

performance

degradation

because

of

network

contention

worsens

from

to

in

Barnes

to

in

LU

to

in

Radix

to

in

Water

as

the

node

speed

increases

from

to

The

reason

for

this

is

again

that

the

network

is

stressed

in

trying

to

cope

with

more

traffic

generated

by

the

faster

processor

in

a

given

amount

of

time

Normalized

Execution

Normalized

Execution

Normalized

Execution

Normalized

Execution

Figure

Effect

of

node

speed

on

network

contention

Impact

of

Network

Design

on

Network

Contention

In

this

section

we

examine

the

impact

on

the

network

contention

with

different

network

designs

Effect

of

Network

Speed

Not

surprisingly

from

the

results

in

Fig.

the

network

contention

is

significantly

worse

in

a

slower

network

-LRB-

e.g.

LANs

in

DSMs

based

on

networks

of

workstations

-RRB-

than

that

in

a

faster

network

Our

results

show

that

performance

degradation

changes

from

to

in

Barnes

to

in

LU

to

in

Radix

to

in

Water

as

the

network

speed

changes

from

to

Effect

of

Network

Flit

Width

As

expected

from

the

results

in

Fig.

the

network

contention

is

significantly

worse

in

a

network

with

narrow

links

than

in

a

network

with

wider

links

For

example

the

performance

degradation

increases

from

to

in

Barnes

to

in

LU

to

in

Radix

and

to

in

Water

as

the

network

width

reduces

from

to

bits

In

general

a

more

powerful

network

does

reduce

the

network

contention

However

considering

the

commonly

used

narrow

links

and

slower

networks

from

a

DSM

perspective

network

contention

is

an

important

factor

for

the

large

network

latency

in

DSM

systems

Conclusion

In

this

paper

we

have

studied

the

impact

of

network

contention

on

the

performance

of

four

representative

applications

-LRB-

from

the

benchmark

suite

-RRB-

on

a

DSM

system

By

implementing

three

separate

network

models

we

have

isolated

the

impact

of

the

contention

in

the

network

interface

the

contention

in

the

network

alone

and

the

overall

network

contention

We

ran

the

four

representative

applications

on

these

three

network

models

and

studied

the

impact

of

the

various

types

of

contention

while

different

architectural

parameters

were

varied

The

architectural

parameters

that

were

varied

are

cache

size

cache

line

size

cache

set

associativity

processing

node

speed

memory

speed

memory

bus

width

network

speed

and

network

width

Our

results

show

that

the

impact

of

network

contention

on

overall

application

performance

is

considerable

Furthermore

the

major

component

of

this

network

contention

occurs

inside

the

network

alone

-LRB-

injection

channel

contention

consumption

channel

contention

virtual

channel

contention

and

physical

channel

contention

-RRB-

If

network

contention

is

taken

into

account

application

performance

can

differ

by

as

much

as

when

compared

to

the

corresponding

Normalized

Execution

Normalized

Execution

Normalized

Execution

Normalized

Execution

Figure

Effect

of

network

speed

on

network

contention

Normalized

Execution

bitsBarnes

Normalized

Execution

bitsLU

Normalized

Execution

bitsRadix

Normalized

Execution

bitsWater

Figure

Effect

of

flit

width

on

network

contention

performance

under

models

that

do

not

take

any

sort

of

network

contention

into

account

When

compared

with

models

that

take

network

interface

contention

into

account

-LRB-

and

assume

contention

free

transmission

within

the

network

-RRB-

application

performance

can

differ

by

as

much

as

These

results

clearly

demonstrate

the

importance

of

modeling

all

types

of

network

contention

in

general

and

contention

within

the

network

alone

in

particular

while

designing

DSM

systems

Our

study

also

shows

that

the

various

architectural

parameters

can

have

considerable

impact

on

the

effect

of

network

contention

on

overall

application

performance

Smaller

caches

larger

cache

lines

lower

set

associativity

higher

processing

node

speeds

higher

memory

speeds

lower

network

speeds

and

narrower

networks

can

significantly

increase

this

effect

of

network

contention

on

application

performance

These

results

show

that

changes

in

any

of

these

parameters

of

a

DSM

system

can

have

a

much

greater

impact

on

overall

DSM

performance

if

studied

in

conjunction

with

methods

for

reducing

the

amount

of

network

contention

-LRB-

such

as

routing

adaptivity

hardware

support

for

collective

communications

etc.

-RRB-

Alternatively

a

study

of

methods

to

reduce

network

contention

can

result

in

possible

changes

in

these

architectural

parameters

so

that

the

system

can

deliver

similar

-LRB-

or

better

-RRB-

performance

at

a

lower

cost

Acknowledgment

We

want

to

thank

Rajeev

Sivaram

for

having

interesting

discussion

on

this

topic

and

for

providing

significant

feedback

during

the

preparation

of

this

manuscript

References

-LSB-

-RSB-

C.

Amza

A.

L.

Cox

and

et

al.

Treadmarks

Shared

memory

computing

on

networks

of

workstations

IEEE

Computer

-LRB-

-RRB-

-LCB-

February

-LSB-

-RSB-

D.

C.

Burger

and

D.

A.

Wood

Accuracy

vs.

Performance

in

Parallel

Simulation

of

Interconnection

Networks

In

Proceedings

of

the

International

Symposium

on

Parallel

Processing

April

-LSB-

-RSB-

D.

Dai

and

D.

K.

Panda

Reducing

Cache

Invalidation

Overheads

in

Wormhole

DSMs

Using

Multidestination

Message

Passing

In

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

Donglai

Dai

and

D.

K.

Panda

Efficient

Schemes

for

Limited

Directory-Based

DSMs

Using

Multidestination

Message

Passing

Technical

Report

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Nov

-LSB-

-RSB-

W.

J.

Dally

Virtual

Channel

Flow

Control

IEEE

Transactions

on

Parallel

and

Distributed

Systems

pages

-LCB-

Mar

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

The

Torus

Routing

Chip

Distributed

Computing

-LCB-

-LSB-

-RSB-

J.

Duato

A

New

Theory

of

Deadlock-Free

Adaptive

Routing

in

Wormhole

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

A.

Agarwal

et

al.

The

MIT

Alewife

Machine

Architecture

and

Performance

In

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

C.

Holt

et

al.

The

Effects

of

Latency

Occupancy

and

Bandwidth

in

Distributed

Shared

Memory

Multiprocessors

Technical

Report

Stanford

University

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Directory-Based

Cache

Coherence

Protocol

for

the

DASH

Multiprocessor

In

Proceedings

of

the

Annual

Symposium

on

Computer

Architecture

pages

-LCB-

May

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

March

-LSB-

-RSB-

J.

Kuskin

et

al.

The

Stanford

FLASH

Multiprocessor

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

S.

C.

Woo

et

al.

The

Programs

Chracterization

and

Methodological

Considerations

In

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

S.

K.

Reinhardt

et

al.

The

Wisconsin

Wind

Tunnel

Virtual

Prototyping

of

Parallel

Computers

In

ACM

SIGMETRICS

International

Conference

on

Measurement

and

Modeling

of

Computer

Systems

pages

-LCB-

May

-LSB-

-RSB-

C.

J.

Glass

and

L.

Ni

Maximally

Fully

Adaptive

Routing

in

Meshes

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

M.

Heinrich

J.

Kuskin

and

Others

The

performance

impact

of

flexibility

in

the

stanford

flash

multiprocessor

In

The

sixth

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

-LRB-

ASPLOS-VI

-RRB-

San

Jose

CA

October

-LSB-

-RSB-

J.

L.

Hennessy

and

D.

Patterson

Computer

Architecture

A

Quantitative

Approach

Morgan

Kaufmann

-LSB-

-RSB-

C.

Holt

J.

P.

Singh

and

J.

Hennessy

Application

and

Architectural

Bottlenecks

in

Large

Scale

Distributed

Shared

Memory

Machines

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

May

-LSB-

-RSB-

Yousef

Khalidi

and

Umakishore

Ramachandran

An

Implementation

of

Distributed

Shared

Memory

Software

-LCB-

Practice

and

Experience

-LRB-

-RRB-

-LCB-

May

-LSB-

-RSB-

J.

H.

Kim

and

A.

A.

Chien

An

Evaluation

of

Planar-Adaptive

Routing

-LRB-

PAR

-RRB-

In

Proceedings

of

the

Symposium

on

Parallel

and

Distributed

Pro

cessing

pages

-LCB-

-LSB-

-RSB-

D.

A.

Koufaty

X.

Chen

D.

K.

Poulsen

and

J.

Torrellas

Data

Forwarding

in

Scalable

Shared-Memory

Multiprocessors

In

ACM

International

Conference

on

Supercomputing

pages

-LCB-

-LSB-

-RSB-

D.

Lenoski

J.

Laudon

T.

Joe

D.

Nakahira

L.

Stevens

A.

Gupta

and

J.

Hennessy

The

DASH

Prototype

Logic

Overhead

and

Performance

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Jan

-LSB-

-RSB-

K.

Li

and

P.

Hudak

Memory

Coherence

in

Shared

Virtual

Memory

Systems

ACM

Transactions

on

Computer

Systems

November

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Mar

-LSB-

-RSB-

T.

Mowry

and

A.

Gupta

Tolerating

Latency

Through

Software-Controlled

Prefetching

in

Shared-Memory

Multiprocessor

Journal

of

Parallel

and

Distributed

Computing

June

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

W.

Oed

Massively

Parallel

Processor

System

Cray

Cray

Research

GmbH

-LSB-

-RSB-

G.

D.

Pifarre

L.

Gravano

S.

A.

Felperin

and

J.

Sanz

Fully

Adaptive

Minimal

DeadlockFree

Routing

in

Hypercubes

Meshes

and

Other

Networks

Algorithms

and

Simulations

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Xiaohan

Qin

and

Jean-Loup

Baer

On

the

Use

and

Performance

of

Explicit

Communication

Primitives

in

Cache-coherent

Multiprocessor

Systems

In

Proceedings

of

the

Third

International

Symposium

on

High

Performance

Computer

Architecture

February

-LSB-

-RSB-

S.

K.

Reinhardt

J.

R.

Larus

and

D.

A.

Wood

Tempest

and

Typhoon

User-Level

Shared

Memory

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

April

-LSB-

-RSB-

S.

K.

Reinhardt

R.

W.

Pfile

and

D.

A.

Wood

Decoupled

Hardware

Support

for

Distributed

Shared

Memory

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

May

-LSB-

-RSB-

J.

H.

Upadhyay

V.

Varavithya

and

P.

Mohapatra

Efficient

and

Balanced

Adaptive

Routing

in

Two-Dimensional

Meshes

In

International

Symposium

on

High

Performance

Computer

Architecture

pages

-LCB-

cache coherence
distributed shared memory systems
interconnection networks
meshes
wormhole routing

directory-based protocols
parallel architecture