MULTIPROCESSOR

EMULATION

WITH

RPM

EARLY

EXPERIENCE

Michel

Dubois

Alain

Gefflaut

Jaeheon

Jeong

Adrian

Moga

and

Koray

ner

Department

of

Electrical

Engineering

Systems

University

of

Southern

California

Los

Angeles

CA

IRISA

Campus

de

Beaulieu

Rennes

Cedex

France

-LCB-

dubois

gefflaut

jaeheonj

moga

oner

-RCB-

@paris

usc.edu

CENG

Technical

Report

Abstract

Field-Programmable

Gate

Arrays

is

an

emerging

technology

which

promises

easy

hardware

reconfigurability

by

software

at

low

cost

Entire

systems

can

be

built

in

which

some

parts

are

easily

programmable

Such

systems

are

flexible

hardware

platforms

or

emulators

which

are

then

tailored

to

implement

various

architectures

The

performance

of

these

architectures

can

be

compared

on

the

same

hardware

substrate

Besides

having

a

large

speedup

advantage

over

software

simulation

the

emulator

is

a

detailed

hardware

implementation

of

the

architecture

including

I/O

on

which

complex

software

systems

can

be

run

without

code

instrumentation

and

it

is

a

more

convincing

proof

of

concept

On

the

other

hand

it

is

much

more

cost-effective

than

a

full-fledged

prototype

We

have

built

a

multiprocessor

emulator

called

RPM

Rapid

Prototyping

engine

for

Multiprocessor

systems

RPM

can

emulate

various

configurations

of

shared-memory

and

message-passing

systems

The

bandwidth

and

latency

of

various

components

can

be

easily

modified

to

match

various

processor

memory

and

interconnect

technologies

In

this

paper

we

present

the

modeling

methodology

the

performance

collection

mechanism

the

calibration

of

the

emulator

as

well

as

our

first

results

obtained

for

the

emulator

of

a

cachecoherent

non

uniform

memory

access

multiprocessor

-LRB-

CC-NUMA

-RRB-

INTRODUCTION

There

are

currently

many

competing

ideas

to

implement

multiprocessor

systems

The

virtues

of

message-passing

versus

shared-memory

are

still

being

debated

-LSB-

-RSB-

-LSB-

-RSB-

Among

shared-memory

systems

many

variants

ranging

from

Virtual

Shared-Memory

systems

-LSB-

-RSB-

or

networks

of

workstations

to

cache-based

multiprocessors

with

hardware-enforced

coherence

-LSB-

-RSB-

are

under

intense

scrutiny

in

the

research

community

Several

projects

both

in

industry

and

academia

are

currently

evaluating

new

ideas

in

multiprocessor

systems

by

building

prototypes

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

In

other

projects

abstracted

machine

prototypes

are

developed

on

software

simulation

platforms

-LSB-

-RSB-

-LSB-

-RSB-

Both

approaches

have

their

advantages

and

shortcomings

Hardware

prototyping

is

extremely

costly

It

requires

engineering

skills

and

a

technical

environment

not

always

available

in

an

academic

or

even

an

industrial

laboratory

setting

A

prototype

verifies

only

one

-LRB-

or

a

few

-RRB-

design

point

and

takes

years

to

build

This

takes

place

in

the

backdrop

of

the

relentless

rate

of

technical

improvements

in

the

computer

architecture

field

Processor

technology

is

improving

at

a

dizzying

pace

New

ideas

in

multiprocessor

architectures

spring

up

flourish

and

fade

away

in

the

span

of

a

couple

of

years

As

a

result

prototypes

are

usually

obsolete

by

the

time

they

are

built

or

they

have

to

keep

adapting

incorporating

new

ideas

as

they

come

along

and

the

hardware

is

never

really

built

The

lure

of

prototypes

is

that

they

are

extremely

realistic

and

attempt

to

exploit

the

current

technology

as

much

as

possible

One

problem

that

has

plagued

prototypes

however

is

the

fact

that

they

are

hard

to

observe

Even

if

a

large

number

of

hardware

monitors

are

included

to

try

to

cover

many

possibilities

there

are

always

limits

as

to

what

can

be

observed

once

the

hardware

has

been

frozen

Software

simulations

may

range

from

detailed

cycle-by-cycle

simulations

of

a

target

which

is

very

slow

to

an

abstracted

simulation

which

can

go

all

the

way

to

a

simple

trace

in

which

no

machine

is

simulated

but

instructions

for

different

processors

are

simply

interleaved

and

some

events

such

as

cache

misses

are

counted

Abstracted

simulations

of

the

latter

type

can

be

very

fast

and

yield

useful

information

about

program

behavior

However

the

more

abstracted

the

simulation

the

less

convincing

it

becomes

Parameters

are

guessed

without

an

actual

implementa

tion

effects

are

neglected

without

really

verifying

the

validity

of

the

approximation

and

entire

parts

of

the

machine

are

sometimes

removed

from

the

simulation

to

simplify

it

There

is

a

tradeoff

in

software

simulations

between

realistic

simulation

speeds

and

realism

and

accuracy

To

solve

this

dilemma

more

efficient

techniques

are

being

developed

to

speed

up

software

simulation

including

parallelization

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Hardware

emulation

the

approach

adopted

in

RPM

-LRB-

Rapid

Prototyping

engine

for

Multiprocessors

-RRB-

is

an

intermediate

approach

between

software

simulation

and

hardware

prototyping

RPM

is

a

common

hardware

platform

or

substrate

on

top

of

which

various

multiprocessor

architectures

can

be

implemented

and

compared

The

parameters

of

the

hardware

substrate

are

also

variable

so

that

various

technological

environments

can

be

reproduced

The

performance

methodology

relies

on

three

key

ideas

multi-cycle

pclock

emulation

-LRB-

each

pclock

or

processor

clock

of

the

target

machine

is

emulated

by

several

clocks

of

the

emulator

-RRB-

time

scaling

-LRB-

the

absolute

speeds

of

all

components

are

different

in

the

target

and

in

the

emulator

but

their

relative

speeds

are

the

same

-RRB-

and

count

memory

-LRB-

a

programmable

event

histogramming

mechanism

-RRB-

The

overall

architecture

of

RPM

and

the

details

of

the

hardware

design

can

be

found

in

-LSB-

-RSB-

and

-LSB-

-RSB-

This

paper

relates

our

initial

experience

with

our

first

emulator

a

Cache-Coherent

NonUniform

Memory

Access

-LRB-

CC-NUMA

-RRB-

architecture

In

the

following

we

will

briefly

describe

the

hardware

substrate

of

RPM

and

the

allocation

of

resources

for

our

first

emulation

in

Section

Then

in

Section

we

show

how

the

methodology

has

been

successfully

applied

in

a

concrete

case

The

calibration

and

debugging

of

the

emulator

is

a

difficult

task

explained

in

Section

We

show

the

results

obtained

on

systems

of

different

sizes

and

with

different

application

data

set

sizes

and

then

conclude

in

Sections

and

OVERVIEW

OF

RPM

AND

THE

CC-NUMA

EMULATION

Hardware

Substrate

RPM

is

made

of

nine

SPARC

processors

connected

to

a

Futurebus

backplane

and

is

currently

clocked

at

MHz

Normally

one

processor

acts

as

an

I/O

node

The

single

chip

processors

have

both

an

integer

and

a

floating-point

pipelines

and

the

execution

of

floating

operations

are

overlapped

with

the

execution

of

integer

instructions

They

have

no

on-chip

cache

and

therefore

all

instruction

fetches

and

data

accesses

are

observable

The

block

diagram

of

each

processor

board

is

shown

in

Figure

Figure

Block

Diagram

of

RPM?s

Processor

Node

-LSB-

-RSB-

The

CPU

is

an

LSI

Logic

SPARC

IU/FPU

The

first-level

cache

is

built

with

Xilinx

FPGAs

-LRB-

-RRB-

connected

to

Mbytes

of

SRAM

SIMMs

-LRB-

-RRB-

the

second-level

cache

is

built

with

Xilinx

FPGAs

-LRB-

-RRB-

connected

to

of

SRAM

-LRB-

-RRB-

and

the

main

memory

-LRB-

-RRB-

is

made

of

of

DRAM

The

main

memory

controller

is

made

of

Xilinx

FPGAs

plus

an

off-the-shelf

DRAM

controller

-LRB-

Cypress

-RRB-

The

delay

unit

is

built

with

a

FIFO

controlled

by

one

AMD

MACH

chip

The

FIFO

-LRB-

kbytes

-RRB-

contains

blocks

and

messages

which

are

sent

to

the

bus

interface

after

a

programmable

delay

depending

on

the

target

machine?s

interconnect

latencies

and

packet

size.The

FutureBus

chip

set

comes

from

Newbridge

and

National

Semiconductors

and

it

includes

bus

transceivers

plus

the

Newbridge

LIFE

chip

The

SCSI

interface

of

the

I/O

board

is

attached

to

the

SCSI

bus

of

a

SPARC

station

II

which

currently

serves

as

an

I/O

server

and

console

for

RPM

The

serial

interface

can

connect

to

a

terminal

and

is

used

mostly

for

debugging

purposes

The

board

size

is

SI

LOCAL

BUS

SCSI

BUS

DU

FUTUREBUS

BUS

INTERFACE

SUN

Workstation

ASCII

TERMINAL

DU

Delay

Unit

SI

SCSI

Interface

Each

processor

board

has

a

Mbyte

first-level

memory

-LRB-

SRAM

-RRB-

an

Mbyte

secondlevel

memory

-LRB-

SRAM

-RRB-

and

Mbytes

of

main

memory

-LRB-

DRAM

-RRB-

In

a

typical

emulation

a

fraction

of

these

memories

emulates

the

target

system?s

caches

and

memories

and

the

rest

is

used

for

performance

collection

and

for

the

emulation

of

special

registers

and

buffers

-LRB-

such

as

a

T.L.B.

-RRB-

The

first-level

memory

controller

drives

the

emulation

of

each

pclock

Typically

at

the

start

of

each

pclock

emulation

it

resumes

the

processor

execution

for

one

cycle

receive

the

next

processor

access

-LRB-

if

any

-RRB-

blocks

the

processor

and

then

emulates

the

memory

access

-LRB-

in

one

or

several

pclocks

-RRB-

The

number

of

clocks

per

pclock

can

be

easily

changed

and

it

depends

on

the

complexity

of

the

pclock

emulation

Each

controller

is

implemented

with

two

FPGAs

-LRB-

Field

Programmable

Gate

Arrays

-RRB-

The

emulation

of

each

pclock

is

implemented

by

a

combination

of

control

in

the

FPGAs

and

memory

space

in

the

RAMs

attached

to

the

memory

controllers

The

Delay

Unit

-LRB-

DU

-RRB-

is

a

programmable

chip

which

emulates

variable

interconnection

delays

Messages

that

are

sent

out

of

the

processor

node

are

stored

in

the

Delay

Unit

for

an

amount

of

time

that

is

programmable

If

L

is

the

length

of

a

packet

then

the

delay

is

equal

to

where

a

is

a

fixed

delay

per

packet

and

b

is

the

time

per

word

transfer

The

FutureBus

interconnection

is

wide

and

transfers

one

word

every

for

a

total

bandwidth

of

Mbytes

per

second

The

main

memory

speed

can

be

modified

by

suspending

memory

requests

using

interleaving

registers

as

described

in

-LSB-

-RSB-

after

the

memory

controller

has

received

and

decoded

a

request

packet

it

suspends

the

request

for

a

programmable

time

it

then

resumes

the

request

and

sends

out

the

messages

resulting

from

the

transaction

During

this

entire

time

the

memory

appears

busy

The

SCSI

interface

and

serial

port

allow

every

board

to

be

configured

as

an

I/O

node

Memories

of

all

boards

can

be

accessed

in

Test

mode

or

in

Emulation

mode

In

Test

mode

-LRB-

mostly

used

for

booting

and

debugging

-RRB-

the

memories

on

all

boards

are

all

part

of

a

contiguous

address

space

and

every

location

can

be

accessed

randomly

To

configure

RPM

into

a

target

multiprocessor

the

FPGA

programs

of

all

three

controllers

are

written

in

VHDL

compiled

and

then

a

b

L

downloaded

through

the

I/O

node

as

part

of

the

booting

procedure

Once

the

FGPAs

are

programmed

the

machine

is

in

test

mode

Memories

are

initialized

the

code

and

data

of

the

application

are

downloaded

into

RPM?s

main

memory

and

the

machine

then

switches

to

emulation

mode

where

memory

accesses

are

restricted

by

the

addressing

mechanisms

of

the

target

machine

An

Emulator

for

Small-Scale

CC-NUMAs

Today?s

successful

multiprocessors

are

mostly

small-scale

systems

-LRB-

to

processors

-RRB-

with

shared

memory

and

cache

coherence

enforced

by

a

snooping

protocol

on

one

or

multiple

buses

However

as

the

speed

of

uniprocessors

keeps

increasing

it

becomes

more

and

more

difficult

to

support

more

than

a

few

processors

on

buses

As

an

alternative

to

snooping

industry

and

academia

are

exploring

other

point-to-point

interconnect

such

as

rings

-LSB-

-RSB-

or

crossbars

to

connect

ultra-fast

processors

in

a

small

number

-LRB-

to

-RRB-

Our

first

emulator

is

a

CC-NUMA

under

strong

ordering

of

shared-memory

accesses

-LSB-

-RSB-

Strong

order

is

still

a

requirement

of

many

commercial

systems

For

example

even

if

the

SPARC

-LRB-

RMO

-RRB-

memory

model

is

very

weak

the

current

version

of

Solaris

still

assumes

a

strong

memory

order

-LRB-

close

to

sequential

consistency

-RRB-

and

disables

latency-hiding

hardware

optimizations

-LSB-

-RSB-

Protocol

The

protocol

is

write-invalidate

with

a

directory

organization

based

on

Censier

and

Feautrier?s

design

-LSB-

-RSB-

Besides

the

presence

bits

and

the

modify

bit

per

block

we

have

added

some

transient

state

bits

to

support

concurrent

transactions

on

different

directory

entries

A

miss

request

first

goes

to

the

home

memory

Before

returning

a

copy

of

the

block

the

home

memory

must

sometimes

invalidate

copies

-LRB-

write

miss

in

the

presence

of

multiple

remote

copies

-RRB-

or

obtain

the

dirty

copy

from

the

owner

-LRB-

miss

in

the

presence

of

a

dirty

remote

copy

-RRB-

This

latter

transaction

takes

four

traversals

of

the

interconnect

-LRB-

requester-to-home

home-to-dirty

dirty-to-home

and

home-torequester

-RRB-

After

invalidations

are

sent

by

the

memory

controller

the

block

state

is

set

to

transient

and

the

presence

bits

indicate

which

copies

have

pending

invalidations

While

invalidations

are

propagated

the

memory

controller

is

released

and

any

other

request

to

the

block

is

nacked

by

the

home

node

As

invalidations

are

acknowledged

the

memory

controller

resets

the

presence

bits

once

all

acknowledgements

have

been

received

the

controller

completes

the

transaction

and

sets

the

block

to

a

stable

state

More

details

on

this

protocol

can

be

found

in

-LSB-

-RSB-

Table

RPM

Hardware

Latencies

-LRB-

no

conflict

-RRB-

Miss

is

the

time

to

receive

a

packet

fetch

the

block

from

DRAM

update

the

directory

and

send

a

packet

back

to

the

requester

Send

k

Invalidations

is

the

time

to

send

k

invalidation

packets

Ack

Invalidations

is

the

time

to

receive

an

acknowledgement

for

one

invalidation

and

update

the

directory

Receive

Write

Back

is

the

time

to

receive

a

block

packet

store

the

block

in

DRAM

and

possibly

forward

the

block

to

the

requester

Get

Block

from

Dirty

is

the

time

to

receive

a

packet

from

the

requester

check

the

directory

and

send

a

packet

to

the

dirty

node

Send

Dirty

Block

to

Requester

is

the

time

to

receive

the

block

copy

from

the

dirty

node

and

send

it

to

the

requester

with

ownership

-LRB-

memory

is

not

updated

-RRB-

Nack

is

the

time

to

receive

a

packet

and

negatively

acknowledge

it

Resource

Event

Basic

Hardware

-LRB-

clocks

-RRB-

processors

-LRB-

pclocks

-RRB-

FLC

Data

Read

-LRB-

single

-RRB-

or

Instruction

Hit

Data

Write

Hit

-LRB-

single

-RRB-

Data

Read

Hit

-LRB-

double

-RRB-

Data

Write

Hit

-LRB-

double

-RRB-

Fill

from

SLC

SLC

Hit

from

FLC

Hit

from

Bus

Interface

Miss

Detection

SLC

fill

SLC

restart

Internal

Bus

Request

Packet

4 1/2

Data

Packet

DU

and

FutureBus

Request

Packet

Data

Packet

Memory

Miss

-LRB-

Tmiss

-RRB-

Send

k

Invalidations

-LRB-

Tinv

-RRB-

-LRB-

k

-RRB-

-LRB-

k

-RRB-

Ack

Invalidation

-LRB-

Tack

-RRB-

Receive

Write

Back

-LRB-

Twb

-RRB-

Get

Block

from

Dirty

-LRB-

Tmd

-RRB-

Send

Dirty

Block

to

Requester

-LRB-

Tmr

-RRB-

Nack

-LRB-

Tnack

-RRB-

Architectural

Parameters

The

first-level

cache

is

a

direct-mapped

write-through

cache

with

a

block

size

of

bytes

The

second-level

cache

is

a

direct-mapped

write-back

cache

with

byte

blocks

The

emulated

memory

size

on

each

board

is

The

basic

hardware

latencies

listed

in

Table

-LRB-

second

column

-RRB-

are

the

number

of

cycles

needed

for

the

emulation

They

were

obtained

by

counting

cycles

in

the

RPM

hardware

They

are

the

lowest

possible

latencies

for

the

CC

NUMA

emulation

The

latencies

of

second

level

cache

read

misses

comprises

three

terms

on-board

delays

memory

delay

interconnect

delays

From

Table

the

following

equations

are

applicable

for

the

latencies

-LRB-

in

clocks

-RRB-

of

read

misses

in

the

absence

of

conflicts

SLC

read

miss

from

Local

Home

Memory

SLC

read

miss

from

Remote

Home

Memory

-LRB-

hops

-RRB-

SLC

read

miss

from

Dirty

Node

-LRB-

hops

-RRB-

Figure

Timings

of

a

Read

Miss

with

Fill

from

a

Dirty

Node

-LRB-

no

conflict

-RRB-

Tmis

s

Tmiss

a

-LRB-

-RRB-

b

Tmd

Twb

a

-LRB-

-RRB-

b

On

Board

-LRB-

Request

-RRB-

DU

and

Futurebus

On

Board

-LRB-

Request

-RRB-

Miss

Request

to

Dirty

Node

On

Board

-LRB-

Request

-RRB-

DU

and

Futurebus

On

Board

-LRB-

Request

-RRB-

SLC

Access

On

Board

-LRB-

Block

-RRB-

DU

and

Futurebus

On

Board

-LRB-

Block

-RRB-

Receive

Write

Back

On

Board

-LRB-

Block

-RRB-

DU

and

Futurebus

On

Board

-LRB-

Block

-RRB-

SLC

Fill

Requester

Node

Home

Node

Dirty

Node

Home

Node

Requester

Node

a

Tmd

a

a

Twb

a

Figure

shows

the

decomposition

of

the

latencies

of

a

read

miss

on

a

dirty

copy

The

basic

hardware

latencies

for

read

misses

-LRB-

a

and

b

-RRB-

are

clocks

-LRB-

local

-RRB-

clocks

-LRB-

home

-RRB-

and

clocks

-LRB-

dirty

-RRB-

There

are

small

fluctuations

in

these

latencies

because

some

timings

-LRB-

such

as

accesses

to

the

DRAM

FutureBus

and

the

second

level

cache

-RRB-

may

vary

slightly

However

these

simple

estimates

are

suprisingly

accurate

as

we

will

see

in

section

Software

Environment

We

have

developed

a

software

environment

for

RPM

to

run

applications

using

the

ANL

programming

macros

to

express

and

manage

parallelism

In

particular

the

SPLASH

and

sets

of

benchmarks

-LSB-

-RSB-

are

written

in

such

a

way

The

original

SPLASH

benchmarks

employ

the

UNIX

FORK

model

in

which

parallel

processes

communicate

exclusively

through

variables

that

are

explicitely

allocated

in

globally-shared

memory

Our

emulator

provides

a

natural

support

for

this

model

by

running

the

master

process

on

the

I/O

board

and

slave

processes

on

the

others

The

addressing

space

of

each

process

consists

of

a

private

space

hosting

local

data

bss

stack

and

local

heap

and

a

shared

space

for

the

executable

code

and

the

global

heap

The

emulator

provides

access

to

other

boards

private

locations

through

special

alternate

space

transfers

thus

supporting

the

forking

mechanism

Synchronization

between

parallel

processes

is

based

on

shared-memory

locks

and

the

atomic

testandset

operation

A

hardware

barrier

is

also

available

in

the

emulator

but

is

only

used

by

system

routines

Some

benchmarks

are

written

for

the

SPROC

model

in

which

all

processes

-LRB-

now

threads

-RRB-

share

the

same

addressing

space

In

particular

threads

can

communicate

through

named

global

variables

stored

in

the

data

segment

We

made

minor

modifications

to

these

applications

and

retrofitted

them

to

the

FORK

model

As

shown

in

Figure

applications

for

the

emulator

are

compiled

using

the

standard

procedures

and

UNIX

tools

for

a

workstation

First

most

of

the

library

functions

are

statically

linked

directly

from

SUN?s

standard

C

and

math

libraries

Some

of

the

functions

-LRB-

e.g.

sbrk

-LRB-

-RRB-

getpagesize

-LRB-

-RRB-

exit

-LRB-

-RRB-

abort

-LRB-

-RRB-

setjmp

-LRB-

-RRB-

-RRB-

had

to

be

rewritten

however

They

are

part

of

libcx.a

System-specific

functions

such

as

device

access

and

hardware

barrier

synchronization

are

provided

by

the

librpm.a

library

Figure

Software

Environment

to

Run

the

SPLASH

Benchmarks

on

RPM

Second

we

had

to

provide

system

support

for

I/O

functions

A

special

operating

system

trap

handler

-LRB-

-RRB-

was

written

to

support

this

task

It

intercepts

the

system

call

parameters

and

passing

them

over

a

communication

line

to

a

server

running

on

the

SUN

workstation

acting

as

the

emulator?s

console

The

server

performs

the

request

and

cooperates

with

the

system

call

stub

running

on

the

emulator?s

I/O

board

for

possible

data

transfers

Upon

completion

the

server

passes

a

return

code

to

the

stub

which

is

delivered

to

the

application

as

a

result

of

the

system

call

The

whole

mechanism

is

transparent

to

the

application

and

the

appearence

of

a

UNIX

kernel

is

entire

Third

a

new

C

runtime

library

-LRB-

-RRB-

had

to

be

developed

to

take

care

of

the

initialization

and

termination

of

processes

Once

compiled

applications

can

be

downloaded

into

the

shared-memory

of

the

emulator

at

a

fixed

address

To

start

them

control

is

passed

from

the

emulator?s

monitor

to

the

application

entry

point

after

switching

from

test

mode

to

emulation

mode

The

slave

boards

block

immediately

awaiting

commands

from

the

master

board

After

some

initialization

including

relocation

gcc

ld

water.C

water.H

lib/libc

a

lib/libm

a

librpm.a

libcx.a

RPM

exec

water.c

water.h

water.o

WATER

SPLASH

benchmarks

ANL

macros

of

data

into

the

private

space

the

master

board

proceeds

through

the

sequential

part

of

the

application

Eventually

it

forks

a

number

of

slave

processes

and

they

all

stop

at

a

hardware

barrier

to

start

collecting

statistics

before

beginning

the

parallel

section

A

similar

barrier

is

entered

upon

termination

of

a

process

and

then

execution

returns

to

the

monitor

routine

PERFORMANCE

METHODOLOGY

Multi-cycle

Pclock

Emulation

Each

pclock

of

the

CC-NUMA

emulator

is

executed

in

eight

clocks

The

corresponding

reduction

in

emulation

speed

is

compensated

by

three

advantages

First

we

can

emulate

complex

mechanisms

without

sacrificing

flexibility

Second

the

latencies

of

the

basic

hardware

are

independent

of

the

latencies

of

the

target

even

if

a

large

number

of

clocks

are

needed

to

move

packets

on

the

board

and

on

the

bus

and

to

emulate

memories

and

complex

directories

the

target

may

still

have

very

low

latencies

when

expressed

in

pclocks

Finally

the

bandwidth

of

the

FutureBus

is

not

a

constraint

in

RPM

If

processors

executed

at

the

rate

of

one

clock

per

pclock

the

bus

would

be

a

serious

bottleneck

Time

Scaling

Latencies

In

RPM

the

latencies

of

main

memory

and

of

the

interconnect

can

be

easily

changed

We

can

emulate

systems

with

various

processor

memory

and

interconnect

technologies

The

simple

rule

is

that

latencies

measured

in

pclocks

must

be

the

same

in

the

target

and

in

the

emulator

For

example

if

in

the

target

system

processors

are

clocked

at

-LRB-

pclock

-RRB-

interconnect

delay

is

and

DRAM

access

time

is

then

the

interconnect

delay

must

be

pclocks

-LRB-

or

clocks

-RRB-

and

each

DRAM

access

must

take

pclocks

-LRB-

or

clocks

-RRB-

in

RPM

The

third

column

of

Table

shows

the

latencies

for

a

target

systems

with

processors

and

DRAM

access

times

The

values

for

memory/directory

accesses

assume

that

the

directory

is

built

in

fast

SRAMs

that

the

DRAMs

can

fetch

an

entire

block

in

one

cycle

and

that

the

memory

controller

can

be

clocked

at

the

speed

of

the

processor

and

works

in

parallel

with

the

DRAM

access

The

values

of

pclocks

and

pclocks

-LRB-

obtained

by

setting

a

clocks

and

b

clocks

-RRB-

for

the

latencies

of

the

target

interconnect

were

chosen

from

the

hypernode

of

the

Convex

Examplar

which

is

an

processor

cluster

connected

by

a

crossbar

switch

-LSB-

-RSB-

From

the

numbers

in

Table

and

the

expressions

for

the

read

miss

latencies

in

the

target

the

number

of

pclocks

to

service

a

SLC

read

miss

in

the

CC-NUMA

emulation

is

pclocks

if

serviced

by

the

home

locally

pclocks

if

serviced

by

the

home

remotely

and

pclocks

if

serviced

by

a

dirty

node

Bandwidths

Besides

latency

bandwidth

is

also

a

critical

performance

parameter

for

any

hardware

resource

Many

protocol

mechanisms

such

as

updates

and

prefetches

work

wonders

when

memories

and

interconnects

are

modelled

with

infinite

bandwidth

but

perform

poorly

in

a

real

system

because

of

the

limited

bandwidth

-LSB-

-RSB-

The

bandwidth

of

the

memories

and

directories

in

our

CC-NUMA

emulator

is

directly

related

to

their

access

latencies

since

we

assume

that

memory

on

each

board

is

not

interleaved

However

we

have

included

a

mechanism

in

the

RPM

hardware

to

emulate

memory

interleaving

-LRB-

up

to

on

each

board

-RRB-

and

thus

increase

the

memory

bandwidth

for

given

memory

access

latencies

This

mechanism

is

based

on

a

set

of

interleaving

registers

and

was

described

in

-LSB-

-RSB-

The

current

width

of

the

FutureBus

is

-LRB-

extensible

to

bits

-RRB-

and

its

bandwidth

is

bytes

per

clock

or

bytes

per

pclock

With

a

target

pclock

rate

of

this

corresponds

to

an

interconnect

bandwidth

of

about

Gbytes

per

second

in

the

target

system

This

is

a

huge

practically

infinite

bandwidth

which

increases

proportionnaly

to

the

target?s

pclock

rate

and

this

explains

why

we

have

observed

very

low

interconnect

traffic

in

our

experiments

-LRB-

see

section

-RRB-

The

FutureBus

bandwidth

can

be

modified

by

changing

the

bus

clock

rate

or

the

bus

width

and

by

padding

packets

with

extra

bytes

It

is

not

difficult

to

hold

the

bus

for

longer

than

necessary

on

each

bus

transfer

in

effect

adding

dummy

bytes

to

packets

as

they

reach

the

FutureBus

interface

Count

Memory

The

primary

two

mechanisms

to

collect

performance

statistics

in

RPM

are

memory-mapped

counters

-LRB-

implemented

in

the

FPGAs

-RRB-

and

memory-mapped

event-count

memory

-LRB-

implemented

in

memory

-RRB-

Memory-mapped

counters

are

simple

counters

that

can

be

read/written

as

any

memory

location

Such

counters

can

measure

the

total

execution

time

the

utilization

of

the

processors

and

the

controllers

and

can

count

events

So

far

the

only

counter

we

have

implemented

in

the

FPGAs

is

a

pclock

counter

in

the

first-level

cache

controller

of

every

processor

to

record

the

total

execution

time

of

each

processor

Event-count

memories

consist

of

a

set

of

counters

implemented

in

the

SRAMs

of

the

caches

and

the

DRAM

of

main

memory

Events

are

mapped

to

memory

addresses

and

at

the

occurrence

of

a

given

event

the

value

stored

at

the

corresponding

memory

location

is

incremented

Therefore

to

implement

event-count

memories

it

is

necessary

to

have

some

extra

hardware

to

read

a

counter

from

count

memory

increment

its

value

and

write

the

updated

value

back

to

memory

Only

one

count-memory

location

can

be

accessed

in

each

pclock

We

associate

a

basic

event

to

each

bit

of

an

address

in

count

memory

For

example

one

address

bit

may

be

associated

with

the

event

hit/miss

one

address

bit

may

be

associated

with

the

event

read/write

and

one

address

bit

may

be

associated

with

the

event

instruction/data

At

the

occurrence

of

a

data

read

miss

the

signal

hit/miss

indicates

a

miss

the

signal

read/write

indicates

a

write

and

the

signal

instruction/data

indicates

a

data

access

and

the

corresponding

count

memory

location

is

incremented

Consequently

the

total

number

of

data-read-misses

is

accumulated

at

the

memory

location

addressed

by

these

three

basic

events

-LRB-

data

read

and

miss

-RRB-

the

total

number

of

data-reads

is

obtained

by

adding

the

contents

of

the

locations

data-read-misses

and

data

read-hits

at

the

end

of

the

emulation

Figure

Control

for

Count

Memory

This

logic

is

present

in

each

cache

controller

and

in

the

main

memory

controller

The

memory

address

is

provided

either

by

the

controller

or

by

the

event

count

logic

The

event

count

logic

stores

the

address

of

the

event

-LRB-

or

combination

of

events

-RRB-

occurring

in

the

current

cycle

in

the

address

latch

this

address

concatenated

with

the

base

address

of

the

count

memory

accesses

the

memory

during

the

performance

cycles

The

value

in

the

memory

location

is

loaded

into

a

counter

the

counter

is

incremented

and

the

value

is

stored

back

into

the

same

memory

location

In

the

first-level

cache

an

event

counter

is

incremented

at

each

pclock

as

part

of

the

pclock

emulation

whereas

in

the

controllers

of

the

second-level

cache

and

of

the

main

memory

an

event

counter

is

incremented

on

each

request

to

the

controller

The

hardware

in

each

controller

to

implement

count

memory

is

shown

in

Figure

Figure

displays

the

composition

of

addresses

to

the

event

counters

in

the

three

memories

obtained

by

merging

hardware

signals

corresponding

to

RAM

Interface

Address

Counter

Memory

State

Machine

Event

Count

Logic

Mux

Data

Control

-LRB-

as

ds

r/w

-RRB-

Controller

-LRB-

FLCC

SLCC

MC

-RRB-

Events

Base

address

Latch

Address

Count

Memory

various

occurrences

and

resource

states

Figure

Addresses

Used

to

Access

Count

Memory

in

MC

SLC

and

FLC

The

main

memory

contains

M

counters

-LRB-

a

-RRB-

for

each

request

type

each

request

source

and

each

global

state

of

the

block

there

are

counters

one

for

every

possible

distribution

of

the

copies

among

the

processors

The

second-level

cache

count

memory

contains

counters

-LRB-

b

-RRB-

the

location

of

the

event

counter

depends

on

the

request

type

the

source

of

the

request

-LRB-

bus

or

local

processor

plus

the

ID

of

the

processor

-RRB-

the

state

of

the

block

in

the

cache

the

state

of

the

replaced

block

and

the

value

of

the

hit/miss

line

Finally

the

count

memory

of

the

first-level

cache

contains

K

counters

-LRB-

c

-RRB-

detecting

various

data

or

instruction

cycles

in

the

processor

Each

processor

board

has

three

sets

of

event

counters

At

the

end

of

an

emulation

run

all

processors

store

their

counters

in

their

main

memory

in

parallel

then

the

I/O

processor

uploads

all

counters

to

the

SUN

host

where

the

events

are

summed

up

and

combined

to

obtain

the

counts

of

meaningful

aggregate

events

Presence

Bits

-LRB-

-RRB-

Directory

Entry

State

-LRB-

-RRB-

Requester

ID

-LRB-

-RRB-

Request

Type

-LRB-

-RRB-

COUNT

MEMORY

ADDRESS

LATCH

-LRB-

MC

-RRB-

Requester

ID

-LRB-

-RRB-

Request

Type

-LRB-

-RRB-

COUNT

MEMORY

ADDRESS

LATCH

-LRB-

SLC

-RRB-

SLC

State

-LRB-

-RRB-

Victim

Status

-LRB-

-RRB-

Hit/Miss

Source

-LRB-

Proc

Bus

-RRB-

COUNT

MEMORY

ADDRESS

LATCH

-LRB-

FLC

-RRB-

Stalled/Busy

Data/Instruction

Emulation/Test

busy

cycle

Double/Single

Word

Replacement

Invalidation

Spin

wait

Acquire

Release

Prefetch

Shared/Private

Data

Read/Write

Hit/Miss

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

VERIFICATION

CALIBRATION

AND

PERFORMACE

DEBUGGING

Once

an

architecture

emulator

is

built

it

is

very

important

to

verify

that

the

characteristics

of

the

architecture

such

as

cache

sizes

cache

organizations

and

access

times

are

correct

according

to

the

original

specifications

defined

for

the

target

system

For

an

emulator

built

on

top

of

RPM

it

is

also

necessary

to

verify

that

the

performance

counters

recorded

during

executions

are

accurate

In

this

section

we

present

the

three

steps

used

to

calibrate

and

verify

the

correctness

of

the

CC

NUMA

emulator

In

a

first

step

we

verify

the

performance

counters

and

access

latencies

using

a

set

of

small

programs

each

program

repeats

a

specific

and

predictable

memory

access

pattern

In

a

second

step

we

verify

the

architecture

characteristics

of

the

emulator

using

a

method

derived

from

-LSB-

-RSB-

Finally

we

verify

the

emulator

by

comparing

results

obtained

with

a

simulator

and

the

results

obtained

with

the

emulator

running

some

of

the

applications

Checking

Performance

Counters

and

Latencies

To

test

and

verify

the

count-memory

mechanism

we

have

developed

a

set

of

simple

programs

each

of

which

is

a

simple

loop

exercising

a

particular

memory

access

pattern

The

statistic

collection

is

performed

only

during

the

execution

of

the

loop

As

the

behavior

of

each

of

these

loops

is

perfectly

known

the

value

reported

by

the

counters

can

be

predicted

and

then

verified

Figure

gives

a

simple

example

of

a

loop

where

each

read

access

misses

in

the

first-level

cache

but

hits

in

the

second-level

cache

In

a

first

step

the

second

level

cache

is

filled

with

data

by

reading

an

array

The

main

loop

is

then

executed

and

generates

a

miss

for

each

read

access

With

such

a

simple

program

we

were

able

to

verify

the

counters

for

the

number

of

read

accesses

in

the

FLC

the

number

of

read

misses

in

the

FLC

the

number

of

read

accesses

received

in

the

second-level

cache

and

the

number

of

read

hits

in

the

second-level

cache

More

than

of

these

small

programs

were

used

to

address

all

the

possible

situations

that

can

happen

during

the

execution

of

a

real

program

Additionally

this

approach

has

been

used

to

verify

the

timing

characteristics

of

the

emulator

Consider

the

previous

example

The

number

of

pclocks

recorded

in

read

misses

divided

by

the

global

number

of

read

misses

gives

a

good

approximation

of

the

time

to

service

a

read

miss

in

the

second-level

cache

Figure

Example

of

one

of

the

Programs

Used

to

Verify

the

Performance

Counters

This

strategy

has

been

extremely

valuable

to

identify

and

correct

a

large

number

of

bugs

in

the

performance

collection

in

particular

in

the

first-level

cache

where

a

counter

is

updated

in

each

pclock

and

where

a

simple

error

in

the

address

of

a

counter

leads

to

incorrect

results

As

an

example

an

inconsistency

was

discovered

between

the

number

of

misses

issued

from

first-level

cache

and

the

number

of

requests

received

by

the

second-level

cache

for

double

read

or

write

accesses

To

fix

this

bug

we

had

to

add

new

counters

for

double

accesses

Verifying

the

Architecture

Parameters

To

verify

that

the

emulator

respects

the

specifications

of

the

target

architecture

Saavedra?s

technique

described

in

-LSB-

-RSB-

and

based

on

micro-benchmarks

has

been

applied

to

the

emulator

int

-LRB-

-RRB-

-LCB-

a

-LRB-

int

-RRB-

-LRB-

MBYTES

-RRB-

for

-LRB-

i

i

-LRB-

MBYTES/LINE

SIZE

-RRB-

i

-RRB-

b

a

-LSB-

i

-RSB-

return

-LRB-

a

-RRB-

-RCB-

main

-LRB-

-RRB-

-LCB-

Fill

first

level

cache

a

-LRB-

-RRB-

for

-LRB-

i

i

-LRB-

MBYTES/L

SIZE

-RRB-

i

-RRB-

-LCB-

b

a

-LSB-

i

-RSB-

-RCB-

-RCB-

The

second

level

cache

is

filled

with

data

Turn

on

performance

counter

collection

The

array

is

read

again

Each

read

misses

in

the

FLC

but

hits

in

the

SLC

Turn

off

performance

counter

collection

The

micro-benchmark

is

made

of

multiple

loops

In

each

loop

the

processor

reads

every

S-th

-LRB-

Stride

-RRB-

element

of

an

array

of

size

R

After

each

iteration

the

stride

is

multiplied

by

until

it

reaches

When

S

reaches

R

is

increased

and

S

is

reset

to

its

initial

value

For

each

value

of

R

and

S

we

first

run

the

micro-benchmark

with

the

read

access

and

then

we

run

it

again

with

the

read

access

replaced

by

a

noop

The

times

are

measured

using

the

pclock

counter

provided

by

the

RPM

hardware

in

the

first-level

cache

controller

The

difference

between

the

execution

times

of

these

two

loops

divided

by

the

number

of

iterations

yields

the

average

read

access

time

A

diagram

representing

the

average

time

to

read

a

single

element

for

different

values

of

R

and

S

is

obtained

On

the

diagram

each

curve

corresponds

to

a

value

of

R

The

same

benchmark

with

write

accesses

could

be

used

to

measure

the

timings

of

write

accesses

Figure

Performance

profile

of

the

CC-NUMA

emulator

K

K

K

K

K

M

Stride

-LRB-

bytes

-RRB-

Average

Execution

Time

-LRB-

pclocks

-RRB-

FLC

Size

SLC

Size

FLC

Line

Size

Bytes

SLC

Line

Size

bytes

pclocks

SLC

Access

Time

pclocks

Local

Memory

Access

Time

pclocks

Remote

Memory

Access

Time

R

R

R

Different

regimes

are

observed

on

a

graph

resulting

from

the

execution

of

a

micro-benchmark

For

example

when

the

size

of

R

is

inferior

to

the

size

of

the

first-level

cache

all

elements

accessed

in

the

loop

fit

in

the

first-level

cache

and

the

average

read

time

as

a

function

of

S

is

a

constant

as

shown

by

the

bottom

curve

of

Figure

By

varying

the

values

of

S

and

R

other

regimes

where

the

read

access

hits

or

misses

in

the

second-level

cache

are

observed

These

different

regimes

allow

us

to

observe

experimentally

the

cache

size

organization

and

latency

times

through

various

features

of

the

diagrams

as

pointed

out

in

the

figure

A

more

detail

description

of

these

regimes

and

of

the

way

to

interpret

the

graphs

can

be

found

in

-LSB-

-RSB-

Figure

shows

the

graph

obtained

for

the

CC-NUMA

emulator

From

this

figure

we

can

infer

that

the

first-level

cache

is

direct-mapped

and

that

its

size

is

we

can

also

verify

that

the

second-level

cache

is

direct

mapped

with

a

size

of

and

that

the

lines

in

both

caches

are

bytes

Access

times

are

confirmed

as

follows

A

read

hit

to

first

level

cache

takes

pclock

A

read

miss

in

the

first-level

cache

that

hits

in

the

second

level

cache

takes

pclocks

A

read

miss

in

the

first

and

second-evel

caches

takes

pclocks

if

it

is

serviced

from

the

local

home

memory

and

pclocks

if

it

is

serviced

from

a

remote

home

memory

These

experimental

timings

are

more

accurate

than

the

timings

estimated

by

counting

cycles

as

we

did

in

section

However

the

two

sets

of

estimates

are

in

close

agreement

Adding

the

first-level

cache

access

time

the

second-level

cache

miss

detection

time

the

second-level

cache

restart

time

and

the

firstlevel

cache

fill

time

-LRB-

a

total

of

pclocks

-RRB-

to

the

second-level

cache

miss

times

computed

in

section

we

find

a

total

of

pclocks

-LRB-

vs.

-RRB-

for

a

read

access

with

a

local

second-level

cache

miss

and

pclocks

-LRB-

vs.

-RRB-

for

a

read

access

with

a

remote

second-level

cache

miss

The

technique

employed

in

this

cache

parameter

verification

is

applicable

to

set-associative

caches

as

well

-LSB-

-RSB-

RPMsim

RPMsim

is

an

approximate

simulator

of

the

CC-NUMA

emulator

Luiz

Barroso

wrote

this

simu

lator

in

C

using

the

CSIM

package

-LSB-

-RSB-

and

the

Cache-Mire

SPARC

interpreter

-LSB-

-RSB-

It

consists

of

several

modules

implementing

the

processor

first

and

second-evel

caches

memory

controller

main

memory

and

the

shared

bus

The

processor

module

is

a

simulator

for

the

instruction

set

of

the

SPARC

architecture

It

has

no

detailed

simulation

of

execution

pipelines

and

it

issues

a

memory

cycle

upon

every

invocation

including

instruction

fetches

The

fact

that

we

do

not

simulate

the

pipeline

introduces

some

error

in

the

simulation

because

in

the

real

machine

instructions

do

not

always

complete

in

one

processor

cycle

Moreover

some

cycles

are

cycles

cycles

are

mostly

due

to

annuled

instructions

in

branch

delay

slots

Moreover

of

the

extended

precision

instruction

and

a

few

double

precision

instructions

are

implemented

in

the

simulator

as

they

do

not

appear

in

the

code

generated

by

the

GCC

compiler

However

some

of

them

may

occur

in

math

library

functions

linked

together

with

the

application

To

handle

the

situation

these

functions

have

been

implemented

as

application

traps

to

the

simulator

The

benefit

is

a

shorter

simulation

time

to

the

detriment

of

accuracy

Because

these

functions

make

heavy

use

of

registers

and

have

few

memory

accesses

the

caching

behavior

of

the

application

is

not

severely

affected

but

the

application

execution

time

may

be

affected

The

timing

parameters

of

the

simulated

system

are

embedded

in

the

cache

memory

controller

and

bus

modules

These

parameters

attempt

to

closely

approximate

the

values

of

the

emulated

architecture

but

there

are

several

limitations

and

simplifications

Double

write

accesses

are

simulated

as

two

individual

back-to-back

accesses

and

incur

the

latency

of

a

write-through

twice

In

the

real

machine

double

writes

are

treated

differently

because

the

tag

matching

in

the

secondlevel

cache

is

done

just

once

hence

the

overall

latency

is

smaller

More

importantly

the

simulated

clock

is

the

processor

clock

and

all

the

simulated

delays

have

fixed

average

values

In

the

emulator

the

system

clock

has

a

frequency

eight

times

larger

and

some

events

and

arbitrations

do

not

happen

synchronously

with

the

processor

clock

Hence

latencies

have

small

fluctuations

in

the

real

case

Other

minor

inaccuracies

are

introduced

in

the

simulation

by

the

absence

of

main

memory

refresh

cycles

in

the

operation

of

the

memory

controller

and

of

other

particularities

of

its

hardware

implementation

We

will

report

on

our

simulation

experiments

at

the

end

of

section

Emulation

Results

In

this

section

we

present

some

of

the

performance

data

collected

on

the

CC-NUMA

emulator

and

do

some

preliminary

comparisons

with

simulation

Emulation

Speed

RPM

currently

emulates

the

execution

of

target

pclocks

per

second

at

the

MHz

clock

rate

Assuming

that

every

instruction

executes

in

one

pclock

this

corresponds

to

a

peak

emulation

rate

of

instructions

per

second

and

per

RPM

board

The

emulation

rate

of

RPM

is

strongly

affected

by

the

characteristics

of

the

programs

it

executes

RPM

approaches

its

peak

emulation

rate

for

programs

with

little

memory

activity

and

with

very

high

computation-to-communication

ratios

As

more

processors

are

added

in

RPM

the

effective

emulation

rate

increases

an

RPM?s

emulation

reaches

a

peak

emulation

rate

of

million

instructions

per

second

Figure

-LRB-

b

-RRB-

shows

how

the

emulation

rate

of

RPM

is

affected

by

increasing

the

number

of

processors

All

programs

except

are

taken

from

the

benchmark

suite

-LSB-

-RSB-

The

selected

data

sets

are

larger

than

the

default

values

for

Cholesky

and

FFT

and

smaller

for

Barnes-Hut

FMM

Radiosity

Radix

Raytrace

and

Volrend

The

best

emulation

rate

is

achieved

for

Volrend

in

which

RPM

reaches

nearly

million

emulated

instructions

per

second

which

is

of

RPM?s

peak

emulation

rate

and

the

worst

case

is

for

with

emulated

instructions

per

second

We

observe

that

the

programs

with

high

algorithmic

speedups

such

as

Volrend

Waternsquared

Raytrace

Barnes-Hut

Radix

perform

very

well

on

RPM

too

By

contrast

programs

with

low

speedups

such

as

and

FFT

perform

poorly

Figure

CC-NUMA

Emulation

Performance

-LRB-

a

-RRB-

Emulation

Times

-LRB-

b

-RRB-

Emulation

rates

-LRB-

c

-RRB-

Worst-case

Memory

Controller

Utilization

-LRB-

board

-RRB-

-LRB-

d

-RRB-

FutureBus

Utilization

Number

of

Processors

Emulation

Time

-LRB-

minutes

-RRB-

Number

of

Processors

Emulation

Rate

-LRB-

Million

Instructions/sec

-RRB-

Number

of

Processors

Memory

Controller

Utilization

-LRB-

Board

-RRB-

Number

of

Processors

FutureBus

Utilization

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

-LRB-

-RRB-

Radix

-LRB-

K

-RRB-

FFT

-LRB-

K

-RRB-

LU

-LRB-

x

-RRB-

Ocean

-LRB-

x

-RRB-

Cholesky

-LRB-

-RRB-

Water

-LRB-

-RRB-

Raytrace

-LRB-

teapot

-RRB-

Volrend

-LRB-

-RRB-

FMM

-LRB-

K

-RRB-

Barnes-Hut

-LRB-

-RRB-

Radiosity

-LRB-

Test

-RRB-

There

are

some

exceptions

For

example

FMM

has

a

very

high

speedup

but

its

emulation

rate

on

RPM

is

not

that

good

This

is

because

of

the

huge

number

of

local

double

write

operations

FMM

executes

Similar

to

FMM

Radiosity

also

has

a

good

speedup

but

mediocre

emulation

speed

Basically

Radiosity

spends

on

the

average

of

its

execution

time

on

instruction

stalls

and

of

its

execution

on

data

read

stalls.In

the

current

implementation

of

RPM

local

data

and

code

are

stored

in

consecutive

boards

starting

with

board

For

programs

with

very

small

data

set

sizes

this

causes

two

problems

-RRB-

the

memory

controller

on

board

becomes

a

hot

spot

and

reaches

very

high

utilization

values

and

-RRB-

programs

with

very

high

local

data

and

instruction

misses

will

have

very

high

instruction

and

data

-LRB-

read/write

-RRB-

stall

times

In

Radiosity

we

observe

both

problems

the

utilization

of

the

memory

controller

on

board

reaches

and

the

number

of

instruction

and

data

read

misses

is

high

Although

LU

has

normally

worse

speedup

than

FFT

it

achieves

a

very

good

emulation

rate

The

main

reason

for

this

is

that

the

work

is

not

uniformly

distributed

among

processors

in

LU

When

a

processor

is

waiting

for

other

processors

-LRB-

busy

waiting

-RRB-

it

stays

in

a

tight

loop

with

a

single

data

access

which

hits

most

of

the

time

During

this

time

RPM

is

close

to

its

peak

emulation

rate

Ocean

has

normally

better

speedup

than

FFT

but

its

emulation

rate

is

lower

than

FFTs

Ocean

exhibits

a

large

local

data

traffic

-LSB-

-RSB-

and

in

the

current

configuration

of

RPM

the

local

data

of

boards

other

than

board

are

remote

We

can

also

see

from

Figure

-LRB-

c

-RRB-

that

the

utilization

of

the

memory

controller

on

board

reaches

in

Ocean

By

contrast

Cholesky

has

lower

speedup

but

still

achieves

better

emulation

rate

on

RPM

This

is

mostly

because

of

Cholesky?s

smaller

data

set

size

and

lower

data

traffic

Figure

-LRB-

a

-RRB-

shows

the

emulation

times

of

all

benchmarks

The

emulation

times

vary

from

to

minutes

on

one

RPM

processor

and

from

to

minutes

on

eight

processors

FMM

takes

the

most

time

and

Barnes-Hut

comes

second

This

graph

gives

a

feel

for

the

speed

of

RPM

Figure

-LRB-

c

-RRB-

displays

the

utilizations

of

the

memory

controller

on

board

The

memory

controller

utilizations

are

calculated

in

the

following

way

We

can

divide

all

messages

that

are

received

and

send

by

a

memory

controller

into

five

types

-RRB-

read

miss

requests

-RRB-

write

miss

requests

-RRB-

ownership

requests

-RRB-

write

backs

due

to

replacement

and

-RRB-

negative

acknowledg

ment

Message

types

and

are

consumed

by

the

memory

controller

and

they

do

not

generate

any

further

messages

but

message

types

may

generate

secondary

messages

if

the

block

is

shared

or

if

it

is

dirty

in

another

processor

cache

To

find

the

utilization

of

the

memory

controller

we

first

calculate

the

busy

time

of

the

memory

controller

by

summing

up

the

times

the

memory

controller

takes

to

process

specific

request

types

including

the

secondary

messages

and

their

replies

For

example

the

total

time

a

memory

controller

spends

processing

read

misses

is

calcu

lated

in

the

following

way

Tmiss

-LRB-

Tmd

Twb

-RRB-

where

is

the

number

of

read

misses

for

which

the

memory

copy

is

clean

when

the

request

is

received

and

is

the

number

of

read

misses

for

which

the

memory

copy

is

stale

In

these

experiments

Radiosity

has

a

very

large

data

set

size

and

its

instruction

miss

rate

is

higher

than

all

other

benchmarks

which

in

turn

results

in

up

to

instruction

stall

time

on

an

system

This

high

instruction

miss

rate

may

be

partially

explained

by

the

nondetermistic

behaviour

of

Radiosity

There

may

also

be

some

trashing

between

data

and

instruction

accesses

has

the

worst

behavior

over

all

In

the

memory

controller

utilization

of

board

reaches

on

processors

Due

to

the

large

number

of

messages

generated

by

the

memory

controller

becomes

a

hot

spot

and

the

read

and

write

stall

times

increase

with

each

additional

processor

When

we

implement

page

level

memory

interleaving

in

RPM

we

expect

to

reduce

the

utilizations

of

the

memory

controllers

Even

though

it

is

not

as

efficient

as

it

can

be

the

current

RPM

configuration

is

also

an

actual

implementation

of

a

shared

memory

multiprocessor

and

nicely

illustrates

the

hot

spot

problem

on

such

machines

Figure

-LRB-

d

-RRB-

shows

the

FutureBus

utilization

This

utilization

is

computed

from

the

event

counts

using

a

similar

procedure

as

for

the

memory

controllers

The

FutureBus

is

very

much

underutilized

Therefore

we

can

safely

assume

that

we

have

an

infinite

bandwidth

interconnection

for

all

practical

purposes

In

order

to

emulate

limited

bandwidth

systems

we

can

artificially

increase

the

size

of

each

packet

sent

to

the

FutureBus

Varying

Data

Set

Sizes

We

have

also

examined

the

effects

of

increased

data

set

sizes

for

Barnes-Hut

Water-nsquared

Ocean

Radix

and

FFT

on

a

system

For

Barnes-Hut

we

have

used

and

particles

For

Water-nsquared

the

number

of

molecules

were

changed

from

up

to

We

ran

Radix

with

and

integers

with

a

radix

of

For

FFT

and

points

are

used

We

simulated

Ocean

with

and

grid

sizes

The

memory

used

by

these

benchmarks

are

and

MB

for

Barnes-Hut

and

MB

for

Water-nsquared

and

MB

for

Radix

and

MB

for

FFT

and

and

MB

for

Ocean

When

increasing

the

data

set

sizes

we

were

limited

by

three

factors

-RRB-

Execution

time

-RRB-

Memory

size

-RRB-

Completion

of

the

program

execution

Among

the

selected

benchmarks

Radix

Ocean

and

FFT

are

memory-bound

On

a

RPM

we

have

MB

memory

and

for

these

three

benchmarks

we

could

not

run

bigger

problems

because

of

our

memory

limitations

Water-nsquared

and

Barnes-Hut

are

compute-bound

programs

and

we

are

currently

limited

by

their

execution

time

In

the

current

RPM

configuration

we

are

limited

to

about

two

hours

of

execution

time

because

of

the

resolution

of

the

event

counters

in

count

memory

In

the

near

future

we

plan

to

interrupt

the

IO

processor

periodically

every

two

hours

to

freeze

the

whole

system

upload

all

the

counters

to

the

host

Sun

SPARC

Workstation

and

resume

the

program

We

expect

this

mechanism

to

add

very

little

distortion

to

the

collected

performance

statistics

-LRB-

it

takes

a

couple

of

pclocks

to

stop

all

processors

-RRB-

while

increasing

the

tolerance

to

crashes

We

had

enough

memory

and

event-counter

bits

to

run

Barnes-Hut

with

particles

but

unfortunately

the

system

crashed

after

running

for

one

hour

We

are

currently

fixing

this

problem

Figure

and

Figure

show

the

total

and

the

normalized

execution

times

of

the

five

benchmarks

on

four

processors

The

run

for

Ocean

took

the

longest

time

about

one

hour

and

minutes

on

RPM

The

busy

time

is

the

total

execution

time

including

instruction

executions

when

all

data

read/write

accesses

hit

in

the

First

Level

Cache

The

Busy

time

is

the

sum

of

all

the

times

when

the

processor

can

not

execute

a

useful

instruction

because

of

an

pipeline

interlock

or

a

unused

branch

delay

slot

The

write

stall

read

stall

swap

stall

and

instruction

stall

times

are

the

times

during

which

a

processor

is

blocked

pending

the

completion

of

a

write

a

read

a

swap

and

an

instruction

fetch

in

the

memory

system

The

spin

time

is

the

total

synchronization

time

excluding

the

swap

stall

time

Figure

Total

Execution

Times

of

the

Five

Benchmarks

on

a

System

Barnes-Hut

and

Water-nsquared

have

very

high

processor

utilizations

and

their

memory

behavior

is

not

affected

by

the

increased

data

set

sizes

because

their

working

set

sizes

are

much

Time

-LRB-

Giga

Pclocks

-RRB-

Barnes

Water-nsquared

FFT

Ocean

Radix

Busy

busy

Write

stall

Read

stall

Inst

stall

Swap

stall

Spin

smaller

than

MB

and

therefore

they

fit

in

our

second-level

cache

easily

Pipeline

overheads

dominate

The

memory

behavior

of

Radix

is

also

unaffected

by

data

set

size

increases

This

result

can

be

explained

mainly

by

the

communication-to-computation

ratio

of

Radix

This

ratio

is

constant

as

long

as

the

number

of

processors

is

fixed

One

interesting

point

about

Radix

is

the

large

amount

of

time

it

spends

in

Busy

cycles

A

large

number

of

branch

delay

slots

are

not

utilized

In

this

case

processor

overheads

dominate

memory

overheads

Figure

Normalized

Execution

Time

of

the

Five

Benchmarks

on

a

System

FFT

is

another

benchmark

whose

memory

behavior

is

unaffected

by

increasing

the

data

set

sizes

In

FFT

the

most

important

data

set

size

is

one

row

of

the

matrix

-LSB-

-RSB-

which

fits

our

second

level

cache

On

top

of

that

the

communication-to-computation

ratio

decreases

logarithmically

with

increasing

data

set

sizes

We

have

run

FFT

with

three

data

set

sizes

points

-LRB-

MB

-RRB-

Time

Barnes

Water-nsquared

FFT

Ocean

Radix

Busy

busy

Write

stall

Read

stall

Inst

stall

Swap

stall

Spin

points

-LRB-

MB

-RRB-

and

points

-LRB-

MB

-RRB-

The

changes

in

communication-to-computation

ratio

from

points

to

points

and

from

points

to

points

are

and

respectively

We

see

the

slight

effect

of

the

reduced

communication-to-computation

rate

in

terms

of

increasing

busy

times

for

larger

data

sets

Ocean

is

the

only

benchmark

which

shows

a

dramatic

difference

in

its

memory

behavior

as

the

data

set

size

is

increased

Ocean

has

a

large

working

set

size

which

does

not

fit

into

our

MB

second-level

cache

for

the

x

grid

size

-LRB-

It

is

possible

that

the

working

set

size

for

the

x

grid

does

not

fit

in

our

second-level

cache

as

well

-RRB-

As

the

data

set

size

increases

the

miss

rates

in

the

second-level

cache

increases

from

to

and

then

to

As

reported

in

-LSB-

-RSB-

the

local

data

traffic

increases

drastically

with

the

data

set

size

for

configurations

and

a

bytes

block

size

Since

the

memory

in

RPM

is

allocated

contiguously

starting

from

board

some

data

which

is

supposed

to

be

local

to

a

processor

may

be

allocated

in

another

board?s

memory

which

increases

the

service

time

of

the

read/write

misses

to

local

data

as

well

as

the

data

traffic

Simulation

Experiments

Table

shows

some

comparisons

between

RPM

emulations

and

RPMsim

simulations

All

the

experiments

reported

in

Table

have

been

conducted

for

four

processors

clocked

at

The

simulations

were

performed

on

a

SUN

with

Mbytes

of

memory

-LRB-

a

MIPS

machine

-RRB-

The

time

values

correspond

to

the

parallel

section

of

the

benchmark

only

The

average

emulation

speedup

is

close

to

and

it

should

more

than

double

when

eight

processors

are

used

instead

of

four

Also

this

speedup

should

double

when

we

reach

the

clock

for

which

the

boards

were

designed

Applications

with

high

busy

times

-LRB-

Barnes-Hut

Volrend

Raytrace

Water

-RRB-

achieve

large

emulation

speedups

These

speedups

are

given

here

as

an

indication

In

fact

more

efficient

simulation

approaches

could

cut

the

simulation

time

Also

the

simulation

could

be

parallelized

or

run

on

faster

machines

Still

even

taking

these

factors

into

consideration

the

speedups

are

quite

impres

sive

The

simulation

time

depends

heavily

on

the

level

of

simulation

details

The

emulation

speedups

would

be

much

higher

if

the

emulation

was

compared

to

cycle-by-cycle

simulation

The

simulation

accuracy

ranges

from

surprisingly

good

-LRB-

Barnes-Hut

Cholesky

FFT

LU

Raytrace

Water

-RRB-

to

moderate

-LRB-

Radix

Volrend

-RRB-

and

bad

-LRB-

Radiosity

-RRB-

In

the

emulation

Radix

spends

to

of

its

time

in

busy

cycles

which

are

unaccounted

for

in

the

simulation

Volrend

makes

heavy

use

of

mathematical

library

functions

which

are

not

simulated

Hence

the

simulated

execution

times

are

too

optimistic

Finally

Radiosity

seems

to

have

followed

different

convergence

paths

to

the

solution

in

the

simulation

and

in

the

emulation

Clearly

we

will

have

to

include

pipeline

effects

in

the

simulation

and

simulate

more

of

the

mathematical

library

functions

if

we

want

to

improve

the

accuracy

of

the

simulator

Table

Comparison

of

Emulation

with

Simulation

CONCLUSION

AND

FUTURE

WORK

Multiprocessor

emulation

is

an

alternative

to

prototyping

and

software

simulation

In

this

paper

we

have

related

our

initial

experience

with

emulation

technology

based

on

field-programmable

gate

arrays

for

the

performance

evaluation

of

multiprocessor

systems

RPM

is

a

configurable

hardware

platform

or

substrate

on

top

of

which

various

multiprocessor

architectures

can

be

imple

Benchmark

Parameters

Emulation

time

-LRB-

sec

-RRB-

Simulation

time

-LRB-

sec

-RRB-

Emulation

speedup

Parallel

run

time

-LRB-

clk

-RRB-

Simulated

parallel

run

time

-LRB-

clk

-RRB-

Simulation

accuracy

-LRB-

-RRB-

barnes

cholesky

fft

50 6444 128

lu

radiosity

test

848 105083

radix

int

raytrace

teapot

volrend

water-nsq

343 303 56498

mented

in

hardware

Because

the

controllers

are

made

of

FPGAs

architecture

parameters

as

well

as

performance

measurement

hardware

can

be

easily

changed

The

emulation

and

performance

methodologies

are

based

on

multi-cycle

pclock

emulation

for

flexibility

time

scaling

and

eventcount

memory

We

have

described

the

methodologies

and

illustrated

them

with

our

first

CC

NUMA

emulation

on

RPM

Currently

we

have

a

complete

software

environment

to

run

parallel

scientific

programs

based

on

the

ANL

macros

By

using

Saavedra?s

micro-benchmark

approach

we

are

able

to

determine

the

cache

parameters

and

the

latencies

of

various

accesses

We

were

surprised

of

how

close

these

experimental

values

were

to

the

estimated

values

based

on

hardware

cycle

counts

since

some

hardware

components

do

not

always

have

fixed

delays

The

micro-benchmarks

have

been

an

extremely

valuable

tool

to

gain

confidence

in

the

validity

of

the

design

and

the

accuracy

of

the

latency

estimates

In

RPM

every

single

event

is

emulated

including

instruction

fetches

and

pipeline

effects

We

have

seen

in

our

emulations

that

these

often

neglected

effects

may

in

some

cases

dominate

other

performance

overheads

Clearly

there

are

some

obvious

improvements

to

the

CC-NUMA

hardware

One

is

to

add

a

memory

mapping

table

in

the

first

level

cache

to

relocate

data

and

instructions

to

the

physical

memory

in

a

flexible

way

Another

easy

improvement

is

to

overcome

the

limitations

on

the

emulation

time

due

to

the

limited

size

of

the

event

counters

by

uploading

the

counters

periodically

Obviously

we

also

need

to

improve

the

reliability

of

the

RPM

simulator

so

that

we

can

rely

on

it

for

debugging

the

performance

measurement

on

RPM

We

are

currently

developing

a

COMA

emulator

-LSB-

-RSB-

so

that

we

can

compare

it

with

the

CC-NUMA

To

port

system

software

on

RPM

requires

that

we

add

hardware

support

for

virtual

memory

and

that

we

boost

the

I/O

capabilities

of

RPM

We

have

also

to

develop

the

time

scaling

methodology

for

I/O

processing

if

we

want

to

make

interesting

system-level

measurements

on

RPM

Acknowledgments

Funding

for

this

work

has

been

provided

by

the

National

Science

Foundation

under

Grant

Alain

Gefflaut

is

funded

by

INRIA/IRISA

-LRB-

France

-RRB-

Besides

the

authors

several

individuals

have

contributed

to

the

project

In

particular

we

want

to

thank

Per

Stenstr?m

from

Lund

University

-LRB-

Sweden

-RRB-

and

Massoud

Pedram

from

EE-Systems

U.S.C.

Luiz

Barroso

Jacqueline

Chame

Sasan

Iman

and

Krishnan

Ramamurthy

participated

in

the

design

of

the

RPM

emulator

Through

their

University

Program

several

companies

helped

reduce

the

cost

of

the

hardware

and

software

needed

for

the

project

These

companies

are

Advanced

Micro

Devices

Synopsis

Viewlogic

Axil

Workstations

and

Xilinx

Finally

John

Granacki

from

ISI

offered

the

services

of

EZFAB

which

is

part

of

the

ARPA-sponsored

Systems

Assembly

Project

REFERENCES

-LSB-

-RSB-

Agarwal

A.

Bianchini

R.

Chaiken

D.

Johnson

K.L.

Kranz

D.

Kubiatowicz

J.

Lim

B.

H.

Mackenzie

K.

Yeung

D.

The

MIT

Alewife

Machine

Architecture

and

Performance

Proceedings

of

the

International

Symposium

on

Computer

Architecture

June

-LSB-

-RSB-

Arnold

J.

Buell

D.

and

Davis

E.

Proceedings

of

the

Annual

Symposium

on

Parallel

Algorithms

and

Architectures

pp

June

-LSB-

-RSB-

Barroso

L.A.

and

Dubois

M.

Performance

Evaluation

of

the

Slotted-Ring

Multiprocessor

IEEE

Transactions

on

Computers

Vol

No.

July

-LSB-

-RSB-

Barroso

L.A.

Iman

S.

Jeong

J.

ner

K.

Ramamurthy

K.

and

Dubois

M.

RPM

A

Rapid

Prototyping

Engine

for

Multiprocessor

Systems

IEEE

Computer

pp.

February

-LSB-

-RSB-

Blumrich

M.A.

Li

K.

Alpert

R.

Dubnicki

C.

Felten

E.

and

Sandberg

J.

Virtual

Memory

Mapped

Network

Interface

for

the

SHRIMP

Multicomputer

Proceedings

of

the

International

Symposium

on

Computer

Architecture

April

-LSB-

-RSB-

Brorsson

M.

Dahlgren

F.

Nilsson

H.

and

Stenstr?m

P.

The

CacheMire

Test

Bench

A

Flexible

and

Effective

Approach

for

Simulation

of

Multiprocessors

In

Proceedings

of

the

Annual

Simulation

Symposium

pp.

March

-LSB-

-RSB-

Catanzaro

B.

Multiprocessor

System

Architectures

Prentice-Hall

-LSB-

-RSB-

Censier

L.

and

Feautrier

P.

A

New

Solution

to

Coherence

Problems

in

Multicache

Systems

IEEE

Transactions

on

Computers

Vol

No.

December

-LSB-

-RSB-

Convex

Examplar

Architecture

Convex

Press

-LSB-

-RSB-

Davis

H.

Goldschmidt

S.R.

and

Hennessy

J.

Multiprocessor

Simulation

and

Tracing

Using

Tango

Proceedings

of

the

Parallel

Processing

Conference

pp

August

-LSB-

-RSB-

Dubois

M.

Scheurich

C.

and

Briggs

F.A.

Synchronization

Coherence

and

Event

Ordering

in

Multiprocessors

IEEE

Computer

Vol

No.

pp.

February

-LSB-

-RSB-

Dubois

M.

Skeppstedt

J.

and

Stenstr?m

P.

Essential

Misses

and

Data

Traffic

in

Coherence

Protocols

Journal

of

Parallel

and

Distributed

Computing

October

-LSB-

-RSB-

Fujimoto

R.M.

Parallel

Discrete

Event

Simulation

Communications

of

the

ACM

Vol

No.

pp.

October

-LSB-

-RSB-

Hagersten

E.

Landin

A.

and

Haridi

S.

DDM

A

Cache-Only

Memory

Architecture

IEEE

Computer

Vol

No.

pp.

September

-LSB-

-RSB-

Kuck

D.

et

al.

The

Cedar

System

and

an

Initial

Performance

Study

Proceedings

of

the

International

Symposium

on

Computer

Architecture

May

-LSB-

-RSB-

Kuskin

J.

et

al.

The

Stanford

FLASH

Multiprocessor

Proceedings

of

the

International

Symposium

on

Computer

Architecture

April

-LSB-

-RSB-

Lenoski

D.

Laudon

J.

Gharachorloo

K.

Weber

W.-D.

Gupta

A.

Hennessy

J.

Horowitz

M.

and

Lam

M.S.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pp.

Vol

No.

March

-LSB-

-RSB-

Li

K.

and

Hudak

P.

Memory

Coherence

in

Shared

Virtual

Memory

Systems

ACM

Transactions

on

Computer

Systems

Vol

No.

pp

November

-LSB-

-RSB-

Muller

H.L.

Stallard

P.W.

Warren

D.

and

Raina

S.

Parallel

Evaluation

of

a

Parallel

Architecture

by

means

of

Calibrated

Emulation

Proceedings

of

the

Int

Parallel

Processing

Symposium

pp.

April

-LSB-

-RSB-

Noakes

M.D.

Wallach

D.A.

and

Dally

W.J.

The

J-Machine

Multicomputer

An

Architectural

Evaluation

Proceedings

of

the

International

Symposium

on

Computer

Architecture

May

-LSB-

-RSB-

Nowatzyk

A.

Aybay

G.

Browne

M.

Kelly

E.

Parkin

M.

Radke

B.

and

Vishin

S.

The

Scalable

Shared-Memory

Multiprocessor

Proceedings

of

the

International

Conference

on

Parallel

Processing

pp

August

-LSB-

-RSB-

ner

K.

Barroso

L.A.

Iman

S.

Jeong

J.

Ramamurthy

K.

and

Dubois

M.

The

Design

of

RPM

An

FPGA-based

Multiprocessor

Emulator

Proceedings

of

the

ACM

International

Symposium

on

Field-Programmable

Gate

Arrays

June

-LSB-

-RSB-

Pong

F.

Stenstr?m

P.

and

Dubois

M.

An

Integrated

Methodology

for

the

Verification

of

Directory-based

Cache

Protocols

Proceedings

of

the

International

Conference

on

Parallel

Processing

pp

August

-LSB-

-RSB-

Reinhardt

S.

Hill

M.D.

Larus

J.R.

Lebeck

A.R.

Lewis

J.C.

and

Wood

D.A.

The

Wisconsin

Wind

Tunnel

Virtual

Prototyping

of

Parallel

Computers

Proceedings

of

the

ACM

Sigmetrics

Conf

on

Measurements

and

Modeling

of

Computer

Systems

pp.

May

-LSB-

-RSB-

Saavedra

R.H.

Gaines

R.S.

and

Carlton

M.J.

Micro

Benchmark

Analysis

of

the

Proceedings

of

Supercomputing

Portland

November

-LSB-

-RSB-

Schwetman

CSIM

A

C-based

Process-oriented

simulation

Language

Proceedings

of

Winter

Simulation

Conference

pp

-LSB-

-RSB-

Stenstr?m

P.

A

Survey

of

Cache

Coherence

Schemes

for

Multiprocessors

IEEE

Computer

Vol

No.

pp.

June

-LSB-

-RSB-

Trimberger

S.

A

Reprogrammable

Gate

Array

and

Applications

Proceedings

of

the

IEEE

Vol

No.

pp.

July

-LSB-

-RSB-

Woo

S.C.

Ohara

M.

Torrie

E.

Singh

J.P.

and

Gupta

A.

The

Programs

Characterization

and

Methodological

Considerations

Proceedings

of

the

International

Symposium

on

Computer

Architecture

June

caches
coma
emulation
field programmable gate arrays
multiprocessor
sequential consistency