LIPLaboratoire

de

l'Informatique

du

Parall?lisme

Ecole

Normale

Sup?rieure

de

Lyon

Institut

IMAG

Unit

de

recherche

associ?e

au

CNRS

n

Dataflow

dot

product

on

networks

of

heterogeneous

digit-serial

arithmetic

units

Jean

Duprat

Mario

Fiallos

Aguilar

March

Research

Report

No

Ecole

Normale

Sup?rieure

de

Lyon

All?e

d'Italie

Lyon

Cedex

France

T?l?phone

T?l?copieur

Adresses

lectroniques

lip@lip.ens-lyon.fr

-LRB-

uucp

-RRB-

Dataflow

dot

product

on

networks

of

heterogeneous

digit-serial

arithmetic

units

Jean

Duprat

Mario

Fiallos

Aguilar

March

Abstract

In

this

paper

we

deal

with

a

new

high

precision

computation

of

the

dot

product

The

key

idea

is

to

use

hundreds

of

digit-serial

arithmetic

units

that

allow

a

massive

digitlevel

pipelining

Parallel

discrete-event

simulations

performed

on

a

memory-distributed

massively

parallel

computer

show

that

with

a

limited

number

of

arithmetic

units

the

computation

of

dot

product

when

performed

using

a

classical

algorithmic

technique

-LRB-

i.e.

serial

cumulative

multiplications

-RRB-

is

almost

as

fast

as

the

case

where

an

optimal

divide-and-conquer

algorithmic

technique

is

used

Interconnection

networks

for

both

algorithmic

techniques

are

considered

R?esum?e

Ce

document

d?ecrit

un

produit

scalaire

a

haute

precision

L'id

ee

principale

est

d'utiliser

plusieurs

centaines

d'unit

es

arithm?etiques

permettant

le

pipeline

au

niveau

du

chiffre

Des

simulations

parall?eles

d'

ev?enements

discrets

faites

sur

des

machines

parall?eles

a

m?emoire

distribu?ee

montrent

que

lorsque

le

produit

scalaire

est

calcul?e

avec

un

nombre

fixe

d'unit

es

un

ordononnancement

de

multiplications

cumulatifs

est

presque

aussi

rapide

qu'un

ordononnancement

divide-and-conquer

Les

r?eseaux

d'interconnection

pour

les

deux

techniques

sont

aussi

present?es

Mots-cl

es

flot

de

donn?ees

produit

scalaire

pipeline

massive

calcul

en-ligne

Dataflow

dot

product

on

networks

of

heterogeneous

digit-serial

arithmetic

units

Jean

Duprat

and

Mario

Fiallos

Aguilary

Laboratoire

de

l'Informatique

du

Parall?elisme

-LRB-

LIP

-RRB-

Ecole

Normale

Sup?erieure

de

Lyon

All?ee

d'Italie

Lyon

Cedex

France

mfiallos@lip.ens-lyon.fr

duprat@lip.ens-lyon.fr

Introduction

Matrix

and

vector

operations

based

on

dot

product

computation

occur

frequently

in

engineering

and

scientific

applications

A

lot

of

work

has

been

performed

in

order

to

obtain

better

algorithms

and

efficient

implementations

on

parallel

computers

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Unfortunately

in

computations

of

arithmetic

algorithms

that

deal

with

the

approximation

of

real

numbers

by

floating-point

representations

inaccurate

calculations

and

representations

lead

to

completely

wrong

results

These

errors

are

produced

by

cancellation

and

truncation

of

the

floating-point

numbers

-LSB-

-RSB-

-LSB-

-RSB-

A

computer

that

allows

the

size

of

operands

and

results

to

be

large

enough

to

compute

according

to

the

needs

of

accuracy

potentially

resolves

these

problems

However

as

high

accuracy

is

achieved

using

very-long

precision

arithmetic

the

representation

of

numbers

needs

a

lot

of

bits

typically

several

thousands

It

is

more

practical

to

carry

all

these

bits

serially

than

in

parallel

In

digit

on-line

mode

of

computation

-LSB-

-RSB-

-LSB-

-RSB-

the

operands

and

the

results

flow

through

the

arithmetic

operators

or

units

-LRB-

aus

-RRB-

serially

digit

by

digit

starting

with

the

most

significant

allowing

a

digit-level

pipelining

This

paper

deals

with

the

digit

on-line

mode

computation

of

dot

product

Floating-point

digit

online

adders

and

multipliers

are

used

to

compute

with

a

maximum

accuracy

of

digits

Two

different

algorithmic

techniques

for

computing

the

dot

product

are

studied

The

first

one

is

the

divide-and-conquer

technique

-LRB-

a

technique

frequently

used

in

parallel

machines

-RRB-

The

second

consists

basically

in

computing

the

dot

product

using

cumulative

multiplications

a

technique

frequently

used

in

SISD

computers

A

comparison

of

the

two

algorithmic

techniques

is

performed

using

analytical

methods

and

parallel

discrete-event

simulation

on

MasPar

On-line

and

dataflow

modes

of

computation

As

stated

above

in

digit

on-line

computation

the

operands

and

the

results

flow

between

arithmetic

units

serially

most

significant

digit

first

-LRB-

MSD

-RRB-

Similarly

LSD

means

least

significant

digit

A

consequence

of

this

flow

is

the

need

of

a

redundant

number

system

-LSB-

-RSB-

In

such

systems

addition

This

work

is

part

of

a

project

called

CARESSE

which

is

partially

supported

by

the

PRC

Architectures

Nouvelles

de

Machines

of

the

French

Minist?ere

de

la

Recherche

et

de

la

Technologie

and

the

Centre

National

de

la

Recherche

Scientifique

ySupported

by

CNPq

and

Universidade

Federal

do

Cear?a

Brazil

ON-LINE

AND

DATAFLOW

MODES

OF

COMPUTATION

is

carry

free

and

can

be

performed

in

parallel

or

in

any

serial

mode

-LRB-

MSD

or

LSD

-RRB-

The

most

usual

arithmetic

operations

can

be

calculated

in

MSD

mode

too

Digit

on-line

arithmetic

is

the

combination

of

MSD

and

redundant

number

system

An

interesting

implementation

of

a

carry-free

redundant

system

is

the

Borrow

Save

notation

BS

for

short

In

BS

the

iith

digit

xi

of

a

number

x

is

represented

by

two

bits

x

i

and

x?i

with

xi

x

i

x?i

Then

has

two

representations

-LRB-

-RRB-

and

-LRB-

-RRB-

The

digit

is

represented

by

-LRB-

-RRB-

and

the

digit

is

represented

by

-LRB-

-RRB-

A

BS

floating-point

number

x

with

n

digits

of

mantissa

and

p

digits

of

exponent

is

represented

by

x

where

mx

Pni

and

ex

Pp

i

exi

In

our

system

the

exponents

and

the

mantissas

circulate

in

digit

on-line

mode

exponent

first

The

digit

on-line

systems

are

characterized

by

their

delay

that

is

the

number

ffi

such

that

p

digits

of

the

result

are

deduced

from

p

ffi

digits

of

the

input

operands

When

successive

digit

on-line

operations

are

performed

in

digit

pipelined

mode

the

resulting

delay

will

be

the

sum

of

the

individual

delays

of

operations

and

communications

and

the

computation

of

large

numerical

jobs

can

be

executed

in

an

efficient

manner

We

will

assume

that

any

communication

has

a

delay

of

See

figure

As

we

can

see

from

figure

the

computations

in

digit

on-line

mode

can

be

described

as

a

data

a

cd

b

ith

digit

of

a

b

c

d

-LRB-

i

-RRB-

th

digits

of

the

products

-LRB-

i

-RRB-

th

digits

of

the

inputs

Figure

Digit-level

pipelining

in

digit

on-line

arithmetic

dependence

graph

or

dataflow

graph

DFG

These

graphs

consist

of

nodes

which

indicate

operations

executed

on

arithmetic

units

and

edges

from

one

node

to

another

node

which

indicate

the

flow

of

data

between

them

A

nodal

operation

can

be

executed

only

when

the

required

information

a

digit

from

all

the

input

edges

is

received

Typically

a

nodal

operation

requires

one

or

two

operands

and

produces

one

result

Once

that

the

node

has

been

activated

and

the

computations

related

to

the

input

digits

inside

the

arithmetic

unit

performed

-LRB-

i.

e.

the

node

has

fired

-RRB-

the

output

digit

is

passed

to

the

destination

nodes

This

process

is

repeated

until

all

nodes

have

been

activated

and

the

final

result

obtained

Of

course

more

than

one

node

can

be

fired

simultaneously

Pseudo-normalization

In

classical

binary

floating-point

representation

a

number

is

said

normalized

if

its

mantissa

belongs

to

-LSB-

-LSB-

or

-RSB-

-RSB-

Normalization

of

numbers

leads

to

more

accurate

representations

and

consequently

results

In

BS

representation

to

check

if

a

number

is

normalized

may

be

necessary

to

examine

all

its

digits

For

this

reason

we

replace

the

concept

of

normalized

numbers

by

that

of

pseudo-normalized

numbers

A

number

is

said

pseudo-normalized

if

its

mantissa

belongs

to

-LSB-

-LSB-

or

-RSB-

-RSB-

It

is

very

easy

to

ensure

that

a

number

is

pseudo-normalized

it

suffices

to

forbid

a

mantissa

beginning

by

or

This

pseudo-normalization

is

performed

serially

In

the

next

sections

we

will

describe

briefly

the

two

digit

on-line

floating-point

arithmetic

units

-LRB-

aus

-RRB-

used

in

the

computation

of

dot

product

FULLY

DIGIT

ON-LINE

FLOATING-POINT

ADDER

Fully

digit

on-line

floating-point

adder

As

in

classical

floating-point

adders

the

fully

digit

on-line

adder

-LRB-

addflo

for

short

-RRB-

-LSB-

-RSB-

-LSB-

-RSB-

performs

three

basic

operations

exponent

calculation

mantissa

alignment

and

mantissa

calculation

Figure

shows

the

different

blocks

of

the

adder

ffl

A

serial

maximizer

computes

the

maximum

of

the

two

exponents

ffl

A

serial

aligner

performs

the

mantissa

alignment

with

a

shift

register

using

the

difference

between

the

exponents

ffl

A

serial

adder

calculates

the

sum

of

the

aligned

mantissas

ffl

A

synchronizer

is

used

to

guarantee

that

only

an

unavoidable

carry

appearing

in

the

sum

of

the

mantissas

will

provoke

the

truncation

of

the

mantissa

and

the

incrementation

of

the

exponent

of

the

result

maximizer

delayincremeteroverflow

detector

serialadder

alignernormalizer

carry

detector

subtracter

counterdelayoutputxye

eeemmyxxyxy

Figure

The

on-line

floating-point

adder

The

synchronizer

-LRB-

normalizer

carry

detector

in

figure

-RRB-

must

test

the

carry

digit

and

the

more

significant

digit

of

the

mantissas

sum

When

these

digits

are

equal

to

or

respectively

the

incrementation

of

the

exponent

and

the

truncation

of

the

mantissa

-LRB-

the

last

digit

is

lost

-RRB-

are

performed

When

these

digits

are

equal

to

or

the

exponent

is

not

modified

and

by

substituting

by

or

by

when

necessary

the

carry

digit

of

the

sum

is

The

operation

performed

by

the

synchronizer

is

different

from

the

systematic

incrementation

of

the

exponent

during

all

addition

proposed

by

Tu

-LSB-

-RSB-

In

this

last

case

the

truncation

of

the

mantissa

of

the

result

leads

to

a

needless

loss

of

information

In

the

solution

we

have

proposed

-LSB-

-RSB-

the

incrementation

is

performed

only

when

necessary

After

the

tests

of

the

first

two

bits

of

the

mantissas

sum

the

exponent

is

incremented

iff

the

modified

carry

digit

is

not

Since

the

decision

of

incrementing

or

not

can

be

made

when

the

last

digit

of

the

exponent

result

is

outputting

the

incrementer

-LRB-

see

figure

-RRB-

the

on-line

delay

of

addflo

ffiadd

becomes

The

digit

pair

is

used

to

transmit

a

and

the

pair

for

a

non

significant

transmission

so

that

the

synchronization

is

insured

automatically

Fully

digit

on-line

floating-point

multiplier

The

floating

point

multiplier

consists

of

three

different

parts

-LSB-

-RSB-

-LSB-

-RSB-

ffl

A

serial

adder

for

the

exponents

ffl

A

serial

multiplier

for

the

mantissas

ffl

A

synchronizer

which

ensures

that

if

two

input

numbers

are

pseudo-normalized

the

output

will

be

pseudo-normalized

too

COMPUTING

DOT

PRODUCT

IN

DIGIT

ON-LINE

MODE

e

e

e

e

e

m

m

m

m

m

e

e

e

e

m

m

m

m

m

-LRB-

cycles

-RRB-

inputoperands

maximum

delayincrementer

overflow

detector

serialadderpseudonormalization

x

x

x

x

x

x

x

x

x

x

max

max

max

max

max

max

max

max

max

max

inc

inc

inc

inc

inc

inc

ove

ove

ove

ove

ove

adm

adm

adm

adm

adm

adm

n

adm

adm

adm

adm

adm

delaymax

output

digit

of

the

maximum

calculation

operator

delay

delay

of

the

computed

maximum

iadm

output

digit

of

the

mantissas

adder

iinc

output

digit

of

the

incrementer

ove

output

digit

of

the

overflow

n

p

e

z

e

n

i

i

y

y

y

y

y

y

y

y

y

y

n

Figure

Synchronization

in

the

on-line

floating-point

adder

The

serial

multiplier

is

described

in

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

The

serial

adder

and

the

pseudo-normalizer

are

similar

to

the

floating-point

adder

ones

and

will

not

present

here

-LSB-

-RSB-

-LSB-

-RSB-

Since

the

two

input

mantissas

belong

to

-LSB-

-LSB-

-LRB-

in

absolute

value

-RRB-

the

product

belongs

to

-LSB-

-LSB-

thus

if

we

want

maintain

the

result

pseudo-normalized

it

is

necessary

to

shift

up

the

mantissa

up

to

positions

to

the

right

This

kind

of

normalization

requires

the

dynamic

subtraction

of

or

to

the

exponent

of

the

result

To

generate

the

final

exponent

the

last

two

digits

of

the

exponent

are

controlled

by

the

three

digits

of

the

mantissa

product

The

normalizer

contains

a

decrementer

followed

by

an

overflow

detector

which

is

similar

to

the

one

of

the

adder

The

digit

on-line

delay

of

the

multiplier

ffimul

is

The

internal

synchronization

is

similar

to

the

adfflo

case

Computing

dot

product

in

digit

on-line

mode

With

the

adder

and

the

multiplier

presented

in

the

last

two

sections

we

envisage

computing

the

dot

product

of

two

vectors

in

a

massive

digit-pipelined

mode

The

dot

product

C

of

two

vectors

A

ang

and

B

bng

is

given

by

C

nX

i

aibi

-LRB-

-RRB-

The

first

fast

mode

of

computation

arises

immediately

compute

the

products

in

parallel

first

and

after

use

adders

to

perform

operations

of

reduction

following

a

divide-and-conquer

technique

See

figure

The

resulting

dataflow

graph

is

a

complete

binary

tree

CBT

for

short

with

levels

We

can

also

try

to

compute

dot

product

by

cumulative

multiplications

-LRB-

see

figure

-RRB-

we

compute

the

first

product

This

result

is

added

then

to

the

next

using

an

adder

and

so

on

until

the

final

result

is

reached

The

DFG

resulting

is

a

linear

array

of

operators

-LRB-

LA

-RRB-

graph

can

be

defined

also

as

a

strictly

binary

tree

with

n

levels

COMPUTING

DOT

PRODUCT

IN

DIGIT

ON-LINE

MODE

digits

addition

product

Complete

binary

tree

of

operators

Linear

array

of

operators

Figure

Two

resulting

graphs

for

dot

product

computation

Three

cases

arise

when

the

dot

product

is

performed

in

digit

on-line

mode

ffl

The

number

of

arithmetic

units

is

greater

than

the

number

of

operations

to

perform

and

a

minimum

delay

is

obtained

ffl

The

number

of

arithmetic

units

is

less

than

the

number

of

operations

to

perform

but

reusing

the

idle

operators

it

is

possible

to

compute

with

minimum

delay

ffl

The

number

of

arithmetic

units

is

less

than

the

number

of

operations

to

perform

and

though

the

idle

operators

are

reused

as

soon

as

possible

it

is

not

possible

to

compute

with

minimum

delay

Note

that

in

the

last

two

cases

a

scheduling

policy

must

be

used

Computing

dot

products

with

a

number

of

aus

larger

or

equal

than

n

With

ffimul

ffiadd

and

assuming

that

any

communication

has

a

delay

of

it

is

easy

to

find

that

the

minimum

delay

for

computing

a

dot

product

using

the

divide-and-conquer

technique

-LRB-

see

fig.

-RRB-

is

ffiCBT

ffimul

-LRB-

ffiadd

-RRB-

n

n

-LRB-

-RRB-

Similarly

the

minimum

on-line

delay

for

the

computation

of

the

dot

product

using

cumulative

multiplications

is

ffiCM

ffimul

-LRB-

ffiadd

-RRB-

n

-LRB-

-RRB-

As

we

stated

the

two

cases

above

are

not

realistics

Reusing

the

aus

to

compute

with

minimum

delay

The

problem

of

reusing

operators

is

unavoidable

in

a

real

machine

where

the

number

of

them

can

not

be

grown

indefinitely

However

it

is

possible

to

reuse

the

aus

to

obtain

a

minimum

delay

On

a

not

digit

on-line

computer

-LRB-

i.

e.

it

receives

all

digits

of

the

operands

in

parallel

-RRB-

reusing

is

simple

For

the

complete

binary

tree

this

can

be

achieved

with

n

adders

and

n

multipliers

In

a

similar

way

it

is

possible

to

reduce

the

number

of

operators

to

compute

in

digit

on-line

mode

But

here

the

situation

is

more

complex

because

as

the

numbers

are

transmitted

serially

digit

by

digit

the

predecessor

operators

may

be

computing

the

last

or

some

middle

digits

of

the

result

number

whose

other

digits

are

being

consumed

by

the

successor

nodes

and

then

the

predecessors

can

not

be

reused

until

them

have

produced

the

last

digit

of

the

result

A

similar

situation

occurs

in

the

LA

of

operators

We

will

present

first

the

case

for

the

complete

binary

tree

graph

and

after

the

case

for

the

linear

array

of

operators

COMPUTING

DOT

PRODUCT

IN

DIGIT

ON-LINE

MODE

Reusing

aus

in

the

tree

It

is

not

possible

to

reach

the

minimum

ffiCBT

without

using

n

multipliers

The

problem

is

to

know

how

many

adders

are

necessary

to

compute

with

minimum

delay

having

n

multipliers

We

note

ui

as

the

level

on

the

CBT

of

operators

-LRB-

the

multipliers

are

at

level

-RRB-

and

L

the

number

of

digits

used

to

code

the

numbers

It

is

possible

to

reuse

only

the

first

level

of

adders

The

beginning

and

ending

time

of

adders

-LRB-

tbega

tenda

-RRB-

and

multipliers

-LRB-

tbegm

tendm

-RRB-

are

tbegm

-LRB-

-RRB-

tendm

ffimul

L

L

-LRB-

-RRB-

tbega

-LRB-

ui

-RRB-

ffiadd

ffimul

ui

-LRB-

-RRB-

tenda

tbega

ffiadd

L

L

-LRB-

-RRB-

The

adders

that

have

not

begun

their

first

digit

computation

when

the

ones

at

the

second

level

of

the

CBT

have

finished

are

those

with

uj

d

-LRB-

L

-RRB-

-RRB-

e

-LRB-

-RRB-

For

example

if

L

and

n

the

number

of

adders

saved

by

reusing

is

only

For

CBT

searching

for

the

minimum

digit

on-line

delay

is

not

realistic

Reusing

aus

in

the

linear

array

If

we

enumerate

the

adders

of

the

LA

from

the

left

to

right

as

an

and

the

multipliers

as

and

mn

-LRB-

see

figure

-RRB-

we

can

compute

the

beginning

and

ending

time

of

the

operators

as

following

tbegai

ffimul

-LRB-

ffiadd

-RRB-

i

-LRB-

-RRB-

tendai

tbegai

ffiadd

L

L

-LRB-

-RRB-

tbegmi

tbegai

ffimul

-LRB-

-RRB-

tendmi

tbegai

L

L

-LRB-

-RRB-

-LRB-

-RRB-

For

the

first

adder

L

It

can

be

reused

at

time

tbegre

L.

Then

we

look

for

the

adders

whose

beginning

time

are

greater

or

equal

than

tbegre

tbegrea

L

-LRB-

-RRB-

L

-RRB-

i

d

-LRB-

L

-RRB-

We

note

that

i

is

the

subindex

that

identifies

each

adder

For

the

other

aus

the

situation

is

similar

These

results

indicate

that

computation

of

dot

product

with

minimum

delay

can

be

performed

by

using

a

number

of

operators

which

depends

only

on

the

length

of

the

format

of

the

number

L

and

not

on

the

number

of

products

n.

Table

summarizes

the

number

of

operators

required

to

compute

with

minimum

delay

as

function

of

the

number

of

digits

used

to

represent

the

numbers

-LRB-

for

adders

d

-LRB-

L

-RRB-

and

for

multipliers

d

-LRB-

L

-RRB-

-RRB-

COMPUTING

DOT

PRODUCT

IN

DIGIT

ON-LINE

MODE

Number

of

digits

Adders

needed

Multiplier

needed

512 130 131

1024 258 259

2048 514 515

4096 1026 1027

8192 2050 2051

Table

Number

of

operators

to

compute

with

CM

technique

with

minimum

delay

A

first

comparison

between

the

techniques

and

networks

The

CM

technique

permits

the

computation

of

dot

product

with

a

number

of

aus

that

is

independent

of

the

number

of

products

to

be

performed

This

contrasts

with

the

divide-and-conquer

technique

where

the

computation

depends

on

both

the

length

of

operands

L

and

the

dimension

of

the

dot

product

n

It

is

true

that

the

digit

on-line

delay

of

the

LA

is

greater

than

the

delay

using

the

divide-and-conquer

technique

but

it

is

interesting

to

investigate

what

will

happen

in

a

real-world

situation

where

the

number

of

operators

will

be

limited

In

the

next

sections

we

will

show

that

the

advantages

of

computing

with

the

divide-and-conquer

method

may

vanish

when

the

number

of

operators

is

limited

and

the

number

of

digits

to

represent

the

numbers

increases

Moreover

it

is

easy

to

see

that

as

the

operators

must

be

reused

the

computation

of

dot

products

by

using

cumulative

multiplications

can

be

performed

on

a

ring

of

operators

instead

of

on

a

LA.

That

is

the

output

of

the

last

adder

of

the

computation

is

feedback

to

one

of

the

inputs

of

the

first

adder

See

figure

From

now

we

suppose

that

the

computation

of

dot

products

using

cumulative

multiplications

will

be

performed

on

a

ring

of

aus

Note

also

that

the

divide-and-conquer

technique

suggests

immediately

to

use

a

set

of

CBT

interconnected

aus

product

addition

Ring

of

operators

Figure

Computing

dot-product

using

a

ring

of

operators

Computing

dot

product

with

a

greater

delay

than

the

minimum

We

suppose

now

that

we

have

less

aus

than

necessary

to

compute

with

minimum

delay

To

compute

the

dot-product

we

adopt

a

two

phases

scheduling

algorithm

The

first

phase

is

the

assignment

of

a

priority

number

for

each

task

Priorities

are

in

decreasing

order

The

second

phase

schedules

the

tasks

according

to

their

priority

number

and

the

number

of

available

aus

As

there

are

two

types

of

aus

adders

and

multipliers

the

scheduling

can

be

performed

independently

and

in

parallel

for

each

type

We

will

present

this

scheduling

algorithm

with

more

details

in

section

Let

us

present

first

the

host

computer

where

the

scheduling

algorithm

and

the

dot

product

were

performed

PARALLEL

SIMULATION

AND

THE

HOST

MACHINE

Parallel

simulation

and

the

host

machine

We

use

discrete-event

parallel

simulation

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

in

our

work

In

the

discrete

approach

to

system

simulation

state

changes

are

represented

by

a

series

of

discrete

changes

or

events

at

specific

instants

of

time

In

our

case

the

events

are

the

input

and

output

of

digits

of

the

aus

MasPar

the

host

computer

of

the

simulation

is

a

SIMD

massively

parallel

computer

-LSB-

-RSB-

-LSB-

-RSB-

In

all

processors

change

state

in

a

simple

predictable

fashion

The

parallelism

in

MasPar

is

achieved

from

the

execution

of

single

operation

simultaneously

across

a

large

set

of

data

In

it

is

easy

to

determine

the

program

state

because

all

processes

are

either

active

or

inactive

and

the

full

synchronization

guarantees

that

the

value

each

processor

retrieves

is

correct

The

processors

-LRB-

PEs

-RRB-

are

interconnected

by

an

xnet

toroidal

neighborhood

mesh

and

a

global

multistage

crossbar

router

network

The

programming

language

used

is

MPL

a

superset

of

C

that

includes

commands

for

the

data-parallel

programming

mode

The

key

idea

to

simulate

several

aus

on

is

to

map

to

several

PEs

several

aus

processes

It

is

possible

to

map

several

aus

of

the

same

or

different

types

to

each

PE

but

all

the

processors

would

simultaneously

simulate

the

same

type

of

operator

since

is

a

SIMD

computer

Description

of

the

performed

simulation

We

have

performed

both

the

scheduling

algorithm

and

the

computations

of

dot-product

using

the

parallel

facilities

of

The

simulation

can

be

viewed

as

a

finite

succession

of

two

different

steps

computation

and

communication

In

fact

due

to

the

data-parallel

programming

model

of

MasPar

problems

of

synchronization

between

the

different

arithmetic

units

are

easily

solved

The

computations

are

performed

in

one

type

of

operator

at

a

time

Static

or

dynamic

scheduling

may

be

applied

to

our

problem

We

use

dynamic

scheduling

As

the

digit

on-line

delay

of

the

adders

and

multipliers

is

fixed

static

scheduling

seems

more

natural

at

first

But

in

digit

on-line

mode

of

computation

there

are

some

arithmetic

operations

as

the

division

that

can

not

be

computed

with

a

constant

digit

on-line

delay

and

consequently

the

static

scheduling

can

not

be

used

However

in

both

cases

of

scheduling

the

results

will

be

identical

Some

other

features

of

the

simulation

are

ffl

The

event

list

is

partitioned

or

distributed

on

the

PEs

In

fact

each

PE

has

a

variable

called

-LRB-

priority

-RRB-

that

contains

its

priority

relatively

to

the

other

tasks

of

the

same

type

ffl

It

exist

a

global

counter

for

counting

the

number

of

cycles

used

to

perform

the

computations

and

local

counters

to

describe

the

state

of

the

operator

The

local

counters

are

used

to

control

the

computational

progress

on

the

node

they

belong

to

The

global

and

local

counters

always

progress

forward

ffl

The

time

is

advanced

according

to

the

production

of

the

next

event

That

is

after

one

step

of

computation

and

communication

the

time

is

incremented

in

one

unit

ffl

Using

the

data-parallel

paradigm

it

is

guaranteed

that

the

simulated

computation

time

of

each

node

that

produces

output

digit

is

less

than

the

virtual

or

simulated

receive

time

of

the

node

that

consumes

the

output

digit

Simulation

of

the

fully-digit

on-line

floating-point

operators

Each

node

of

the

simulated

DFG

performs

its

discrete-event

simulation

by

repeatedly

processing

the

inputs

performing

some

computation

and

outputting

its

results

In

our

simulation

a

BS

digit

is

represented

by

two

bits

The

floating-point

BS

format

chosen

may

have

from

to

digits

for

DESCRIPTION

OF

THE

PERFORMED

SIMULATION

the

mantissa

and

from

to

digits

for

the

exponent

Control

of

each

arithmetic

unit

process

is

assumed

by

a

status

variable

The

process

works

like

a

global

automaton

which

controls

local

ones

-LRB-

maximum

overflow

detector

and

pseudo-normalizer

etc

-RRB-

and

circuits

-LRB-

serial

adder

and

incrementer

etc

-RRB-

-LSB-

-RSB-

Mappings

It

is

necessary

to

map

the

tree

-LSB-

-RSB-

and

the

linear

array

of

operators

on

the

mesh

of

The

mappings

of

the

tree

and

of

the

linear

array

of

processors

DFGs

for

a

dot

product

are

shown

in

figures

respectively

Figure

Mapping

of

a

tree

on

a

mesh

Figure

Mapping

of

a

linear

array

on

a

mesh

The

simulation

of

the

interconnection

network

From

figures

and

we

see

that

the

communication

distances

are

short

In

order

to

take

advantages

of

this

the

networks

are

simulated

using

the

static

mappings

With

this

mode

of

simulation

we

will

activate

a

number

of

operators

less

than

or

equal

to

the

number

of

available

operators

The

result

is

that

the

communication

distances

will

be

kept

short

and

the

computation

will

be

performed

fast

Note

also

that

at

this

level

of

abstraction

the

simulation

of

the

linear

array

of

operators

and

the

ring

are

equivalent

The

scheduling

algorithm

and

its

simulation

From

now

the

terms

task

and

node

will

be

used

as

synonymous

A

flag

will

be

used

to

indicate

when

an

operator

has

been

scheduled

Counter

C

will

store

the

number

of

iterations

of

the

algorithm

An

information

table

will

contain

the

beginning

and

ending

time

of

computation

for

each

operator

in

PERFORMANCE

OF

THE

TECHNIQUES

AND

NETWORKS

the

computation

-LRB-

tbeg

tend

-RRB-

If

the

two

predecessors

of

a

node

have

produced

valid

digits

then

we

will

say

that

the

node

is

ready

We

present

the

scheduling

algorithm

applied

to

the

CBT

The

case

for

the

LA

of

operators

is

similar

The

algorithm

can

be

stated

as

follows

Assigns

priorities

to

the

nodes

that

represent

the

additions

from

one

side

to

the

other

of

the

CBT

beginning

at

the

first

level

of

adders

and

ending

at

the

level

of

the

root

The

node

with

priority

has

the

highest

priority

and

the

node

n

the

lowest

one

In

a

similar

way

assign

priorities

to

the

the

nodes

that

represent

the

multiplications

Set

counter

C

to

zero

As

long

as

there

are

nodes

to

be

scheduled

do

the

following

-LRB-

a

-RRB-

For

each

type

of

task

determine

the

number

of

ready

nodes

Scheduled

the

maximum

number

of

ready

tasks

according

to

the

number

of

available

operators

of

the

type

-LRB-

b

-RRB-

Set

in

the

beginning

time

tbeg

of

the

arithmetic

units

selected

in

the

last

item

to

the

value

of

C.

Compute

for

each

node

its

tend

too

-LRB-

c

-RRB-

Wait

computations

of

the

cycle

to

be

performed

-LRB-

d

-RRB-

Return

to

the

group

of

available

operators

whose

which

interval

of

computation

have

expired

-LRB-

e

-RRB-

Increment

C.

End

As

one

of

the

inputs

of

an

operator

may

be

delayed

in

relation

to

the

other

a

synchronization

must

be

provided

Latches

are

used

to

delay

the

input

that

is

ready

first

The

scheduling

algorithm

was

performed

using

MPL

and

the

parallel

facilities

of

The

scheduling

were

performed

in

a

type

of

operator

at

a

time

but

using

data-parallel

statements

Performance

of

the

techniques

and

networks

In

order

to

compare

the

networks

we

adopt

the

following

measures

of

performance

Number

of

cycles

means

the

number

of

necessary

cycles

to

perform

the

computations

In

fact

the

number

of

cycles

is

equal

to

the

digit

on-line

delay

length

of

the

operands

The

speed-up

of

computing

with

n

operators

of

each

type

is

defined

as

the

ratio

of

the

number

of

necessary

cycles

to

compute

with

operator

of

each

type

and

the

number

of

necessary

cycles

to

compute

with

n

operators

of

each

type

Efficiency

is

the

ratio

of

the

speed-up

and

the

number

of

operators

used

Finally

traces

show

how

the

utilization

of

the

different

operators

along

the

time

are

outputs

differents

from

PERFORMANCE

OF

THE

TECHNIQUES

AND

NETWORKS

Number

of

cycles

to

perform

computations

The

figures

and

show

that

when

the

number

of

multiplications

of

the

dot

product

is

fixed

the

cumulative

multiplications

technique

when

performed

on

the

ring

has

performances

comparable

to

those

of

the

binary

tree

When

the

number

of

available

operators

begins

to

increase

the

performance

of

the

binary

tree

as

can

be

expected

begins

to

be

better

10 100 1000

Cycles

Number

of

arithmetic

units

tree

ring

Figure

Number

of

cycles

needed

to

perform

a

dot

product

with

L

10 100 1000

Cycles

Number

of

arithmetic

units

tree

ring

Figure

Number

of

cycles

needed

to

perform

a

dot

product

with

L

PERFORMANCE

OF

THE

TECHNIQUES

AND

NETWORKS

Speed-up

The

figures

and

show

the

speed-up

obtained

for

the

tree

and

the

ring

respectively

In

the

tree

the

speed-up

is

better

when

the

number

of

aus

is

a

power

of

For

the

ring

the

speed-up

reaches

a

maximum

value

relatively

fast

50 100 150 200

Speed-up

Number

of

arithmetic

units

L

L

L

Figure

Speed-up

on

the

tree

for

a

dot

product

50 100 150 200

Speed-up

Number

of

arithmetic

units

L

L

L

Figure

Speed-up

on

the

ring

for

a

dot

product

PERFORMANCE

OF

THE

TECHNIQUES

AND

NETWORKS

Efficiency

In

the

tree

-LRB-

fig.

-RRB-

the

efficiency

is

better

when

the

number

of

aus

is

a

power

of

The

efficiency

for

the

ring

is

self

explanatory

-LRB-

see

fig

-RRB-

50 100 150 200

Effciency

Number

of

arithmetic

units

L

L

L

Figure

Efficiency

on

the

tree

for

a

dot

product

50 100 150 200

Effciency

Number

of

arithmetic

units

L

L

L

Figure

Efficiency

on

the

ring

for

a

dot

product

PERFORMANCE

OF

THE

TECHNIQUES

AND

NETWORKS

Traces

of

adders

and

multipliers

From

figure

we

see

that

the

peak

value

of

the

adders

used

for

the

tree

is

reached

a

number

of

times

equal

to

the

ratio

of

n

-LRB-

the

dimension

of

the

dot

product

-RRB-

per

number

of

available

aus

For

the

ring

-LRB-

figs.

and

-RRB-

the

maximum

number

of

multipliers

and

adders

is

reached

fast

and

maintained

practically

constant

until

the

end

of

computation

100 200 300 400

500 600 700

Operatorsused

Cycles

aus

aus

aus

Figure

Traces

of

utilization

of

multipliers

on

the

tree

for

a

dot

product

with

L

100 200 300 400

500 600 700

Operatorsused

Cycles

aus

aus

aus

Figure

Trace

of

utilization

of

adders

on

the

tree

for

a

dot

product

with

L

CONCLUDING

REMARKS

AND

FUTURE

WORK

200 400 600 800

Operators

used

Cycles

aus

aus

aus

Figure

Traces

ofutilization

of

multipliers

on

the

ring

for

a

dot

product

with

L

200 400 600 800

Operators

used

Cycles

aus

aus

aus

Figure

Traces

of

utilization

of

adders

on

the

ring

for

a

dot

product

with

L

Concluding

remarks

and

future

work

Here

we

have

described

a

heterogeneous

computer

made

up

of

digit

on-line

adders

and

multipliers

working

on

the

dot

product

problem

We

have

described

the

simulation

of

the

machine

on

a

massively

parallel

computer

the

MasPar

The

main

conclusion

is

that

due

to

the

natural

pipeline

at

digit

level

in

digit

on-line

mode

linear

arrays

have

performances

very

near

of

binary

trees

when

the

dimension

of

the

problem

is

large

compared

to

the

number

of

arithmetic

operators

This

phenomena

is

augmented

by

the

fact

that

working

at

the

digit

level

the

dimension

of

the

problem

is

the

product

of

the

number

of

inputs

and

the

length

of

the

number

Another

interesting

fact

is

the

ability

of

a

ring

of

digit

on-line

REFERENCES

arithmetic

units

that

with

a

reasonable

number

of

them

may

perform

high

precision

calculus

with

large

numbers

Other

numerical

computations

are

under

study

This

includes

polynomial

evaluation

and

the

Gauss

elimination

algorithm

to

solve

linear

equations

We

are

working

in

a

project

to

simulate

and

to

build

a

digit

on-line

machine

called

CARESSE

the

french

abbreviation

of

Serial

Redundant

Scientific

Computer

that

will

made

of

heterogeneous

digit

on-line

arithmetic

units

A

VLSI

prototype

of

the

multiplier

has

been

projected

and

tested

References

-LSB-

-RSB-

A.

Avizienis

Signed-digit

number

representations

for

fast

parallel

arithmetic

IRE

Transactions

on

Electronic

Computers

pp

-LCB-

-LSB-

-RSB-

J.C.

Bajard

Evaluation

de

fonctions

dans

des

Syst?emes

Redondantes

d'ecriture

des

Nombres

PhD

thesis

Ecole

Normale

Sup?erieure

de

Lyon

February

-LSB-

-RSB-

J.C.

Bajard

and

J.M.

Muller

On-line

power

series

In

International

Conference

on

Signal

Processing

Applications

and

Technology

Boston

USA

-LSB-

-RSB-

J.

Bezivin

and

H.Imbert

Adapting

a

simulation

language

to

a

distributed

environment

In

International

conference

on

distributed

computing

system

pages

-LCB-

IEEE

-LSB-

-RSB-

P.

Bjorstad

F.

Manne

T.

Sorevik

and

M.

Vajtersic

Efficient

matrix

multiplication

on

simd

computers

SIAM

Journal

of

Matrix

Anal

Appl.

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

E.

Debenedictis

S.

Ghosh

and

M.

YU

A

novel

algorithm

for

discrete-event

simulation

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

J.

Duprat

and

M.

Fiallos

On

the

simulation

of

pipelining

of

fully

digit

on-line

floating-point

adder

networks

on

massively

parallel

computers

In

Second

Joint

Conference

on

Vector

and

Parallel

Processing

Lecture

Notes

in

Computer

Science

pages

-LCB-

Springer-Verlag

September

-LSB-

-RSB-

J.

Duprat

M.

Fiallos

J.

M.

Muller

and

H.

J.

Yeh

Delays

of

on-line

floating-point

operators

in

borrow

save

notation

In

Algorithms

and

Parallel

VLSI

Architectures

II

pages

-LCB-

Noth

Holland

-LSB-

-RSB-

J.

Duprat

J.

M.

Muller

S.

Kla

and

J.C.

Bajard

Some

operators

for

radix

on-line

computations

Journal

of

Parallel

and

Distributed

Computing

To

Appear

-LSB-

-RSB-

M.D.

Ercegovac

On-line

arithmetic

an

overview

In

SPIE

editor

SPIE

Real

Time

Signal

Processing

VII

pages

pp

-LCB-

-LSB-

-RSB-

M.D.

Ercegovac

and

K.S

Trivedi

On-line

algorithms

for

division

and

multiplication

IEEE

Trans

Comp.

-LRB-

-RRB-

pp

-LCB-

-LSB-

-RSB-

C.T.

Evens

M.

Gargeya

and

T.

Leonard

Structure

of

a

distributed

simulation

system

In

International

conference

on

distributed

computing

systems

pages

-LCB-

IEEE

-LSB-

-RSB-

R.

Fujimoto

Parallel

discrete

event

simulation

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

October

-LSB-

-RSB-

A.

Guyot

Y.

Herreros

and

J.

M.

Muller

Janus

an

on-line

multiplier/divider

for

manipulating

large

numbers

In

Symposium

on

Computer

Arithmetic

pages

-LCB-

IEEE

Computer

Society

Press

-LSB-

-RSB-

E.

Horowitz

and

A.

Zorat

The

binary

tree

as

an

interconnection

network

Applications

to

multiprocessor

systems

and

vlsi

IEEE

Transactions

on

Computers

-LRB-

-RRB-

-LCB-

April

REFERENCES

-LSB-

-RSB-

K.

Hwang

and

Y.

Cheng

Partitioned

matrix

algorithms

for

vlsi

arithmetic

systems

IEEE

Transactions

on

computers

-LRB-

-RRB-

-LCB-

December

-LSB-

-RSB-

H.

V.

Jagadish

and

T.

Kailath

A

family

of

new

efficient

arrays

for

matrix

multiplication

IEEE

Transactions

on

computers

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

D.R.

Jefferson

Virtual

time

ACM

Transactions

on

Programming

Languages

and

Systems

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

J.

P.

Katoen

Simulation

of

doom

a

loosely

coupled

multiprocessor

system

Master

's

thesis

Computer

Science

Department

of

the

University

of

Twente

November

-LSB-

-RSB-

J.

kent

Peacecock

j.

Wong

and

E.

Manning

Distributed

simulation

using

a

network

of

processors

Computer

Networks

-LCB-

-LSB-

-RSB-

C.

King

W.

Chou

and

L.

Ni

Pipelined

data-parallel

algorithms

Part

design

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

October

-LSB-

-RSB-

C.

King

W.

Chou

and

L.

Ni

Pipelined

data-parallel

algorithms

concept

and

modeling

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

October

-LSB-

-RSB-

Andrewas

Knoefl

Fast

hardware

units

for

the

computation

of

accurate

dot

products

In

P.

Kornerup

and

D.

Matula

editors

Symposium

on

Computer

Arithmetic

IEEE

IEEE

Computer

Society

Press

June

-LSB-

-RSB-

L.

Lamport

Time

clocks

and

the

ordering

of

events

in

a

distributed

system

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

MasPar

Computer

Corporation

MasPar

Parallel

Application

Language

-LRB-

MPL

-RRB-

User

Guide

-LSB-

-RSB-

J.

Misra

Distributed

discrete-event

simulation

Computer

Surveys

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

Jean-Michel

Muller

and

Philippe

Francois

Faut-il

faire

confiance

aux

ordinateurs

Rapport

de

recherche

Laboratorire

de

L'Informatique

du

Parall?elisme

ENSL

-LSB-

-RSB-

J.M.

Muller

Arithmetique

des

Ordinateurs

Masson

-LSB-

-RSB-

J.

Nickolls

The

design

of

the

maspar

A

cost

effective

massively

parallel

computer

In

IEEE

editor

IEEE

Compcon

Spring

pages

pp

-LCB-

-LSB-

-RSB-

S.

K.

Reinhardt

M.

D.

Hill

J.

R.

Larus

A.

Lebecks

J.

C.

Lewis

and

D.

A.

Wood

The

winsconsin

wind

tunnel

Virtual

prototyping

of

parallel

computers

In

ACM

SIGMETRICS

Conference

May

-LSB-

-RSB-

T.

R.

Stiemerling

Design

and

Simulation

of

an

MIMD

Shared

memory

multiprocessor

with

interleaved

instruction

streams

PhD

thesis

Department

of

Computer

Science

University

of

Edinburgh

November

-LSB-

-RSB-

P.

K.

Tu

On-line

Arithmetic

Algorithms

for

Efficient

Implementation

PhD

thesis

Computer

Science

Department

UCLA

O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

digit on-line computation
massive pipelining