Parallel

Sparse

LU

Factorization

with

Partial

Pivoting

on

Distributed

Memory

Architectures

Cong

Fu

Xiangmin

Jiao

and

Tao

Yang

Department

of

Computer

Science

University

of

California

Santa

Barbara

CA

Abstract

Gaussian

elimination

based

sparse

LU

factorization

with

partial

pivoting

is

important

to

many

scientific

applications

but

it

is

still

an

open

problem

to

develop

a

high

performance

sparse

LU

code

on

distributed

memory

machines

The

main

difficulty

is

that

partial

pivoting

operations

make

structures

of

L

and

U

factors

unpredictable

beforehand

This

paper

presents

an

approach

called

S

for

parallelizing

this

problem

on

distributed

memory

machines

The

S

approach

adopts

static

symbolic

factorization

to

avoid

run-time

control

overhead

incorporates

L/U

supernode

partitioning

and

amalgamation

strategies

to

improve

caching

performance

and

exploits

irregular

task

parallelism

embedded

in

sparse

LU

using

asynchronous

computation

scheduling

The

paper

discusses

and

compares

the

algorithms

using

and

data

mapping

schemes

and

presents

experimental

studies

on

and

The

performance

results

for

a

set

of

nonsymmetric

benchmark

matrices

are

very

encouraging

and

S

has

achieved

up

to

GFLOPS

on

nodes

Introduction

Sparse

matrix

factorization

is

the

key

to

solving

a

sparse

system

of

linear

equations

If

a

matrix

is

symmetric

and

positive

definite

Cholesky

factorization

can

be

used

for

which

fast

sequential

and

parallel

algorithms

have

been

developed

in

-LSB-

-RSB-

However

in

many

applications

the

associated

equation

systems

involve

nonsymmetric

matrices

and

pivoting

may

be

necessary

to

maintain

numerical

stability

for

such

nonsymmetric

linear

systems

-LSB-

-RSB-

Because

pivoting

operations

interchange

rows

based

on

the

numerical

values

of

matrix

elements

during

the

elimination

process

it

is

impossible

to

predict

the

precise

structures

of

L

and

U

factors

without

actually

performing

the

numerical

factorization

The

adaptive

and

irregular

nature

of

sparse

LU

data

structures

makes

an

efficient

implementation

of

this

algorithm

very

hard

even

on

a

modern

sequential

machine

with

memory

hierarchies

There

are

several

approaches

that

can

be

used

for

solving

nonsymmetric

systems

One

approach

is

the

unsymmetric-pattern

multi-frontal

method

-LSB-

-RSB-

that

uses

elimination

graphs

to

model

irregular

parallelism

and

guide

the

parallel

computation

if

the

pivoting

sequence

can

be

known

prior

to

numerical

factorization

Another

approach

-LSB-

-RSB-

is

to

restructure

a

sparse

matrix

into

a

bordered

block

upper

triangular

form

and

use

a

special

pivoting

technique

which

preserves

the

structure

and

maintains

numerical

stability

at

acceptable

levels

It

is

implemented

on

Illinois

Cedar

multiprocessors

based

on

Aliant

shared

memory

clusters

This

paper

focuses

on

parallelization

issues

for

a

given

matrix

ordering

with

partial

pivoting

to

maintain

numerical

stability

Parallelization

of

sparse

LU

with

partial

pivoting

is

also

studied

in

-LSB-

-RSB-

on

a

shared

memory

machine

Their

approaches

overestimate

the

nonzero

fill-ins

by

using

a

static

symbolic

LU

factorization

so

that

the

dynamic

variation

of

LU

data

structures

is

avoided

They

have

obtained

good

speedups

for

up

to

processors

on

a

Sequent

machine

and

further

work

is

needed

to

assess

the

performance

of

the

sequential

code

As

far

as

we

know

there

are

no

published

results

for

parallel

sparse

LU

on

popular

commercial

distributed

memory

machines

such

as

Intel

Paragon

IBM

TMC

and

Meiko

One

difficulty

in

the

parallelization

of

sparse

LU

on

these

machines

is

how

to

utilize

a

sophisticated

uniprocessor

architecture

The

design

of

a

sequential

algorithm

must

take

advantage

of

caching

which

makes

some

previously

proposed

techniques

less

effective

On

the

other

hand

a

parallel

implementation

must

utilize

the

fast

communication

mechanisms

available

on

these

machines

It

is

easy

to

get

speedups

by

comparing

a

parallel

code

to

a

sequential

code

which

does

not

fully

exploit

the

uniprocessor

capability

but

it

is

not

as

easy

to

parallelize

a

highly

optimized

sequential

code

One

such

sequential

code

is

SuperLU

-LSB-

-RSB-

which

uses

a

supernode

approach

to

conduct

sequential

sparse

LU

with

partial

pivoting

The

supernode

partitioning

makes

it

possible

to

perform

most

of

numerical

updates

using

level

dense

matrix-vector

multiplications

and

therefore

to

better

exploit

memory

hierarchies

They

perform

symbolic

factorization

and

generate

supernodes

on

the

fly

as

the

factorization

proceeds

UMFPACK

is

another

competitive

sequential

code

for

this

problem

and

neither

SuperLU

nor

UMFPACK

is

always

better

than

the

other

-LSB-

-RSB-

is

a

code

for

sparse

matrices

with

symmetric

patterns

All

of

them

are

regarded

as

of

high

quality

and

deliver

excellent

megaflop

performance

In

this

paper

we

focus

on

the

performance

analysis

and

comparison

with

SuperLU

code

since

the

structure

of

our

code

is

closer

to

that

of

SuperLU

In

this

paper

we

present

an

approach

called

S

that

considers

the

following

key

strategies

together

in

parallelizing

the

sparse

LU

algorithm

-RRB-

adopt

a

static

symbolic

factorization

scheme

to

eliminate

the

data

structure

variation

caused

by

dynamic

pivoting

-RRB-

identify

data

regularity

from

the

sparse

structure

obtained

by

the

symbolic

factorization

scheme

so

that

efficient

dense

operations

can

be

used

to

perform

most

of

computation

-RRB-

develop

scheduling

techniques

for

exploiting

maximum

irregular

parallelism

and

reducing

memory

requirements

for

solving

large

problems

We

observe

that

on

most

current

commodity

processors

with

memory

hierarchies

a

highly

optimized

subroutine

usually

outperforms

a

subroutine

in

implementing

the

same

numerical

operations

-LSB-

-RSB-

We

can

afford

to

introduce

some

extra

operations

in

re-designing

the

LU

algorithm

so

that

the

new

algorithm

is

easy

to

be

parallelized

but

the

sequential

performance

of

this

code

is

still

competitive

to

the

current

best

sequential

code

We

use

the

static

symbolic

factorization

technique

first

proposed

in

-LSB-

-RSB-

to

predict

the

worst

possible

structures

of

L

and

U

factors

without

knowing

the

actual

numerical

values

then

we

develop

a

L/U

supernode

partitioning

technique

to

identify

dense

structures

in

both

L

and

U

factors

and

maximize

the

use

of

level

subroutines

for

these

dense

structures

We

also

incorporate

a

supernode

amalgamation

-LSB-

-RSB-

technique

to

increase

the

granularity

of

the

computation

In

exploiting

irregular

parallelism

in

the

re-designed

sparse

LU

algorithm

we

have

experimented

with

two

mapping

methods

one

of

which

uses

data

mapping

and

the

other

uses

data

mapping

One

advantage

of

using

data

mapping

is

that

the

corresponding

LU

algorithm

can

be

easily

modeled

by

directed

acyclic

task

graphs

-LRB-

DAGs

-RRB-

Graph

scheduling

techniques

and

efficient

run-time

support

are

available

to

schedule

and

execute

DAG

parallelism

-LSB-

-RSB-

Scheduling

and

executing

DAG

parallelism

is

a

difficult

job

because

parallelism

in

sparse

problems

is

irregular

and

execution

must

be

asynchronous

The

important

optimizations

are

overlapping

computation

with

communication

balancing

processor

loads

and

eliminating

unnecessary

communication

overhead

Graph

scheduling

can

do

an

excellent

job

in

exploiting

irregular

parallelism

but

it

leads

to

extra

memory

space

per

node

to

achieve

the

best

performance

Also

the

data

mapping

can

only

expose

limited

parallelism

Due

to

these

restrictions

we

have

also

examined

a

data

mapping

method

and

an

asynchronous

execution

scheme

which

exploits

parallelism

under

memory

constraints

We

have

implemented

our

sparse

LU

algorithms

and

conducted

experiments

with

a

set

of

nonsymmetric

benchmark

matrices

on

and

Our

experiments

show

that

our

approach

is

quite

effective

in

delivering

good

performance

in

terms

of

high

megaflop

numbers

In

particular

the

code

outperforms

the

current

code

when

processors

have

sufficient

memory

But

the

code

has

more

potential

to

solve

larger

problems

and

produces

higher

megaflop

numbers

The

rest

of

the

paper

is

organized

as

follows

Section

gives

the

problem

definition

Section

describes

structure

prediction

and

L/U

supernode

partitioning

for

sparse

LU

factorization

Section

describes

program

partitioning

and

data

mapping

schemes

Section

addresses

the

asynchronous

computation

scheduling

and

execution

Section

presents

the

experimental

results

Section

concludes

the

paper

Preliminaries

Given

a

nonsingular

matrix

A

it

can

be

factorized

into

two

matrices

L

and

U

using

Gaussian

Elimination

with

partial

pivoting

as

shown

in

Figure

The

elimination

steps

are

controlled

by

loop

index

k.

For

elements

manipulated

at

step

k

we

use

i

for

row

indexing

and

j

for

column

indexing

This

convention

will

be

used

through

the

rest

of

this

paper

During

each

step

of

the

elimination

process

a

row

interchange

may

be

needed

to

maintain

numerical

stability

The

result

of

LU

factorization

process

can

be

expressed

by

PA

LU

where

L

is

a

unit

lower

triangular

matrix

U

is

a

upper

triangular

matrix

and

P

is

a

permutation

matrix

which

contains

the

row

interchange

information

The

solution

of

a

linear

system

Ax

b

can

hence

be

solved

by

two

triangular

solvers

Ly

P

b

and

Ux

y

The

triangular

solvers

are

much

less

time

consuming

than

the

Gaussian

-LRB-

-RRB-

for

k

to

n

-LRB-

-RRB-

Find

m

such

that

jamkj

maxk

i

nfjaikjg

-LRB-

-RRB-

if

amk

then

A

is

singular

stop

-LRB-

-RRB-

Swap

row

k

with

row

m

-LRB-

-RRB-

for

i

k

to

n

with

aik

-LRB-

-RRB-

aik

aik

akk

-LRB-

-RRB-

endfor

-LRB-

-RRB-

for

j

k

to

n

with

akj

-LRB-

-RRB-

for

i

k

to

n

with

aik

-LRB-

-RRB-

aij

aij

aik

akj

-LRB-

-RRB-

endfor

-LRB-

-RRB-

endfor

-LRB-

-RRB-

endfor

Figure

Sparse

Gaussian

elimination

with

partial

pivoting

for

LU

factorization

elimination

process

Caching

behavior

plays

an

important

role

in

achieving

good

performance

for

scientific

computations

To

better

exploit

memory

hierarchy

in

modern

architectures

supernode

partitioning

is

an

important

technique

to

exploit

the

regularity

of

sparse

matrix

computations

and

utilize

BLAS

routines

to

speed

up

the

computation

It

has

been

successfully

applied

to

Cholesky

factorization

-LSB-

-RSB-

The

difficulty

for

the

nonsymmetric

factorization

is

that

supernode

structure

depends

on

pivoting

choices

during

the

factorization

thus

can

not

be

determined

in

advance

SuperLU

performs

symbolic

factorization

and

identifies

supernodes

on

the

fly

It

also

maximizes

the

use

of

BLAS

level

operations

to

improve

the

caching

performance

of

sparse

LU

However

it

is

challenging

to

parallelize

SuperLU

on

distributed

memory

machines

Using

the

precise

pivoting

information

at

each

elimination

step

can

certainly

optimize

data

space

usage

reduce

communication

and

improve

load

balance

but

such

benefits

could

be

offset

by

high

run-time

control

and

communication

overhead

The

strategy

of

static

data

structure

prediction

in

-LSB-

-RSB-

is

valuable

in

avoiding

dynamic

symbolic

factorization

identifying

the

maximum

data

dependence

patterns

and

minimizing

dynamic

control

overhead

We

will

use

this

static

strategy

in

our

S

approach

But

the

overestimation

does

introduce

extra

fill-ins

and

some

unnecessary

operations

in

the

numerical

factorization

We

observe

that

in

SuperLU

-LSB-

-RSB-

the

DGEMV

routine

-LRB-

the

level

dense

matrix

vector

multiplication

-RRB-

accounts

for

to

of

the

floating

point

operations

-LRB-

excluding

the

symbolic

factorization

part

-RRB-

It

is

also

a

fact

that

routine

DGEMM

-LRB-

matrix-matrix

multiplication

-RRB-

is

usually

much

faster

than

and

routines

-LSB-

-RSB-

On

with

a

matrix

of

size

DGEMM

can

achieve

MFLOPS

while

DGEMV

only

reaches

MFLOPS

Thus

the

key

idea

of

our

approach

is

that

if

we

could

find

a

way

to

maximize

the

use

of

DGEMM

after

using

static

symbolic

factorization

even

with

overestimated

nonzeros

and

extra

numerical

operations

the

overall

code

performance

could

still

be

competitive

to

SuperLU

which

mainly

uses

DGEMV

Storage

prediction

and

dense

structure

identification

Storage

prediction

The

purpose

of

symbolic

factorization

is

to

obtain

structures

of

L

and

U

factors

Since

pivoting

sequences

are

not

known

until

the

numerical

factorization

the

only

way

to

allocate

enough

storage

space

for

the

fill-ins

generated

in

the

numerical

factorization

phase

is

to

overestimate

Given

a

sparse

matrix

A

with

a

zero-free

diagonal

a

simple

solution

is

to

use

the

Cholesky

factor

Lc

of

ATA

It

has

been

shown

that

the

structure

of

Lc

can

be

used

as

an

upper

bound

for

the

structures

of

L

and

U

factors

regardless

of

the

choice

of

the

pivot

row

at

each

step

-LSB-

-RSB-

But

it

turns

out

that

this

bound

is

not

very

tight

It

often

substantially

overestimates

the

structures

of

the

L

and

U

factors

-LRB-

refer

to

Table

-RRB-

We

consider

another

method

instead

-LSB-

-RSB-

The

basic

idea

is

to

statically

consider

all

possible

pivoting

choices

at

each

step

The

space

is

allocated

for

all

the

possible

nonzeros

that

would

be

introduced

by

any

pivoting

sequence

that

could

occur

during

the

numerical

factorization

We

summarize

the

symbolic

factorization

method

briefly

as

follows

The

nonzero

structure

of

a

row

is

defined

as

a

set

of

column

indices

at

which

nonzeros

or

fill-ins

are

present

in

the

given

n

n

matrix

A

Since

the

nonzero

pattern

of

each

row

will

change

as

the

factorization

proceeds

we

use

Rki

to

denote

the

structure

of

row

i

after

step

k

of

the

factorization

and

Ak

to

denote

the

structure

of

the

matrix

A

after

step

k.

And

akij

denotes

the

element

aij

in

Ak

Notice

that

the

structures

of

each

row

or

the

whole

matrix

cover

the

structures

of

both

L

and

U

factors

In

addition

during

the

process

of

symbolic

factorization

we

assume

that

no

exact

numerical

cancellation

occurs

Thus

we

have

Rki

f

j

j

akij

is

structurally

nonzerog

We

also

define

the

set

of

candidate

pivot

rows

at

step

k

as

follows

Pk

f

i

j

i

k

and

ak

ik

is

structurally

nonzerog

We

assume

that

akk

is

always

a

nonzero

For

any

nonsingular

matrix

which

does

not

have

a

zero-free

diagonal

it

is

always

possible

to

permute

the

rows

of

the

matrix

so

that

the

permuted

matrix

has

a

zero-free

diagonal

-LSB-

-RSB-

Though

the

symbolic

factorization

does

work

on

a

matrix

that

contains

zero

entries

in

the

diagonal

it

is

not

preferable

because

it

makes

the

overestimation

too

generous

The

symbolic

factorization

process

will

iterate

n

steps

and

at

step

k

for

each

row

i

Pk

its

structure

will

be

updated

as

Rki

Rk

i

-LSB-

-LRB-

-LSB-

j

i

Rk

j

k

-RRB-

Essentially

the

structure

of

each

candidate

pivot

row

at

step

k

will

be

replaced

by

the

union

of

the

structures

of

all

the

candidate

pivot

rows

except

those

column

indices

less

than

k

In

this

way

it

is

guaranteed

that

the

resulting

structure

An

will

be

able

to

accommodate

the

fill-ins

introduced

by

any

possible

pivot

sequence

A

simple

example

in

Figure

demonstrates

the

whole

process

This

symbolic

factorization

is

applied

after

an

ordering

is

performed

on

the

matrix

A

to

reduce

fill-ins

The

ordering

we

are

currently

using

is

the

multiple

minimum

degree

ordering

for

ATA

We

Nonzero

Fill-in

A

k

k

k

Figure

The

first

steps

of

the

symbolic

factorization

on

a

sample

sparse

matrix

The

structure

remains

unchanged

at

steps

and

also

permute

the

rows

of

the

matrix

using

a

transversal

obtained

from

Duff

's

algorithm

-LSB-

-RSB-

to

make

A

have

a

zero-free

diagonal

The

transversal

can

often

help

reduce

fill-ins

-LSB-

-RSB-

We

have

tested

the

storage

impact

of

overestimation

for

a

number

of

nonsymmetric

testing

matrices

from

various

sources

The

results

are

listed

in

Table

The

fourth

column

in

the

table

is

original

number

of

nonzeros

and

the

fifth

column

measures

the

symmetry

of

the

structure

of

the

original

matrix

The

bigger

the

symmetry

number

is

the

more

nonsymmetric

the

original

matrix

is

A

unit

symmetry

number

indicates

a

matrix

is

symmetric

but

all

matrices

have

nonsymmetric

numerical

values

We

have

compared

the

number

of

nonzeros

obtained

by

the

static

approach

and

the

number

of

nonzeros

obtained

by

SuperLU

as

well

as

that

of

the

Cholesky

factor

of

ATA

for

these

matrices

The

results

in

Table

show

that

the

overestimation

usually

leads

to

less

than

extra

nonzeros

than

SuperLU

scheme

does

Extra

nonzeros

do

imply

additional

computational

cost

For

example

one

has

to

either

check

if

a

symbolic

nonzero

is

an

actual

nonzero

during

a

numerical

factorization

or

directly

perform

arithmetic

operations

which

could

be

unnecessary

If

we

can

aggregate

these

element-wise

floating

point

operations

and

maximize

the

use

of

subroutines

the

sequential

code

performance

will

still

be

competitive

Even

the

fifth

column

of

Table

shows

that

the

floating

operations

from

the

overestimating

approach

can

be

as

high

as

times

the

results

in

Section

will

show

that

actual

ratios

of

running

times

are

much

less

Thus

it

is

necessary

and

beneficial

to

identify

dense

structures

in

a

sparse

matrix

after

the

static

symbolic

factorization

It

should

be

noted

that

there

are

some

cases

that

static

symbolic

factorization

leads

to

excessive

overestimation

especially

when

matrices

have

a

nearly

dense

row

or

column

For

example

memplus

matrix

-LSB-

-RSB-

is

such

a

case

The

static

scheme

produces

times

as

many

nonzeros

as

SuperLU

does

In

such

situation

we

have

to

resort

to

an

alternative

approach

For

another

matrix

-LSB-

-RSB-

the

static

scheme

produces

times

as

many

nonzeros

as

SuperLU

does

But

our

code

can

still

produce

GFLOPS

for

it

on

nodes

of

This

paper

focuses

on

the

development

of

a

high

performance

parallel

code

when

overestimation

ratios

are

not

too

high

factor

entries

Matrix

Identifier

Order

jAj

jA

AT

j

jAj

Static

SuperLU

ATA

f

lop

-LRB-

S

-RRB-

f

lop

-LRB-

SuperLU

-RRB-

goodwin

7320 324772

inaccura

Table

Testing

matrices

and

their

statistics

L/U

supernode

partitioning

and

dense

structure

identifica

tion

Supernode

partitioning

is

a

commonly

used

technique

to

improve

the

caching

performance

of

sparse

code

-LSB-

-RSB-

For

a

symmetric

sparse

matrix

a

supernode

is

defined

as

a

group

of

consecutive

columns

that

have

nested

structure

in

the

L

factor

of

the

matrix

Excellent

performance

has

been

achieved

in

-LSB-

-RSB-

using

supernode

partitioning

for

Cholesky

factorization

However

the

above

definition

is

not

directly

applicable

to

sparse

LU

with

nonsymmetric

matrices

A

good

analysis

for

defining

unsymmetric

supernodes

in

an

L

factor

is

available

in

-LSB-

-RSB-

Notice

that

the

supernodes

may

need

to

be

further

broken

into

smaller

ones

to

fit

into

cache

and

to

expose

more

parallelism

For

the

SuperLU

approach

after

L

supernode

partitioning

there

are

no

regular

dense

structures

in

a

U

factor

that

could

make

it

possible

to

use

routines

-LRB-

see

Figure

-LRB-

a

-RRB-

-RRB-

However

in

the

S

approach

there

are

dense

columns

-LRB-

or

subcolumns

-RRB-

in

a

U

factor

that

we

can

identify

after

the

static

symbolic

factorization

-LRB-

see

Figure

-LRB-

b

-RRB-

-RRB-

The

U

partitioning

strategy

is

explained

as

follows

After

an

L

supernode

partition

has

been

obtained

on

a

sparse

matrix

A

i.e.

a

set

of

column

blocks

with

possible

different

block

sizes

the

same

partition

is

applied

to

the

rows

of

the

matrix

to

further

break

each

supernode

into

submatrices

Now

each

off-diagonal

submatrix

in

the

L

part

is

either

a

dense

block

or

contains

dense

blocks

Furthermore

the

following

theorem

identifies

dense

structure

patterns

in

U

factors

This

is

the

key

to

maximizing

the

use

of

subroutines

in

our

algorithm

In

the

following

theorem

we

show

that

the

L/U

partitioning

strategy

is

successful

and

there

is

a

rich

set

of

dense

structures

to

exploit

The

following

notations

will

be

used

through

the

rest

of

L

U

ii

ij

L

U

ii

ij

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

An

illustration

of

dense

structures

in

a

U

factor

in

the

SuperLU

approach

-LRB-

b

-RRB-

Dense

structures

in

a

U

factor

in

the

S

approach

the

paper

ffl

The

L

and

U

partitioning

divides

the

columns

of

A

into

N

column

blocks

and

the

rows

of

A

into

N

row

blocks

so

that

the

whole

matrix

is

divided

into

N?N

submatrices

For

submatrices

in

the

U

factor

we

denote

them

as

Uij

for

i

j

N

For

submatrices

in

the

L

factor

we

denote

them

as

Lij

for

j

i

N

If

i

j

Lii

denotes

the

diagonal

submatrix

We

use

Aij

to

denote

a

submatrix

when

it

is

not

necessary

to

distinguish

between

L

and

U

factors

ffl

Define

S

-LRB-

i

-RRB-

as

the

starting

column

-LRB-

or

row

-RRB-

number

of

the

i-th

column

-LRB-

or

row

-RRB-

block

For

convenience

we

define

S

-LRB-

N

-RRB-

n

ffl

A

subcolumn

-LRB-

or

subrow

-RRB-

is

a

column

-LRB-

or

row

-RRB-

in

a

submatrix

For

simplicity

we

use

a

global

column

-LRB-

or

row

-RRB-

index

to

denote

a

subcolumn

-LRB-

or

subrow

-RRB-

in

a

submatrix

For

example

by

subcolumn

k

in

the

submatrix

block

Uij

it

means

the

subcolumn

in

this

submatrix

with

the

global

column

index

k

where

S

-LRB-

j

-RRB-

k

S

-LRB-

j

-RRB-

Similarly

we

use

aij

to

indicate

an

individual

nonzero

element

based

on

global

indices

A

compound

structure

in

L

or

U

is

a

submatrix

a

subcolumn

or

a

subrow

ffl

A

compound

structure

is

nonzero

if

it

contains

at

least

one

nonzero

element

or

fill-in

We

use

Aij

to

indicate

that

block

Aij

is

nonzero

Notice

that

an

algorithm

only

needs

to

operate

on

nonzero

compound

structures

A

compound

structure

is

structurally

dense

if

all

of

its

elements

are

nonzeros

or

fill-ins

In

the

following

we

will

not

differentiate

between

nonzero

and

fill-in

entries

They

are

all

considered

as

nonzero

elements

Theorem

Given

a

sparse

matrix

A

with

a

zero-free

diagonal

after

the

above

static

symbolic

factorization

and

L/U

supernode

partitioning

are

performed

on

A

each

nonzero

submatrix

in

the

U

factor

of

A

contains

only

structurally

dense

subcolumns

Proof

Recall

that

Pk

is

the

set

of

candidate

pivot

rows

at

symbolic

factorization

step

k.

Given

a

supernode

spanning

from

column

k

to

k

s

from

its

definition

and

the

fact

that

after

step

k

the

static

symbolic

factorization

will

only

affect

the

nonzero

patterns

in

submatrix

ak

n

k

n

and

A

has

a

zero-free

diagonal

we

have

Pk

oe

Pk

oe

oe

Pk

s

and

Pk

s

OE

Notice

at

each

step

k

the

final

structures

of

row

i

-LRB-

i

Pk

-RRB-

are

updated

by

the

symbolic

factorization

procedure

as

Rki

Rk

i

-LSB-

-LRB-

-LSB-

j

i

Rk

j

k

-RRB-

For

the

structure

of

a

row

i

where

k

i

k

s

we

are

only

interested

in

nonzero

patterns

of

the

U

part

-LRB-

excluding

the

part

belonging

to

Lkk

-RRB-

We

call

this

partial

structure

as

URi

Thus

for

i

Pk

URki

-LSB-

URk

j

It

can

be

seen

that

after

the

k-th

step

updating

URki

URkk

for

i

Pk

Knowing

that

the

structure

of

row

k

is

unchanged

after

step

k

we

only

need

to

prove

that

URkk

URk

k

URk

s

k

s

as

shown

below

Then

we

can

infer

that

the

nonzero

structures

of

rows

from

k

to

k

s

are

same

and

subcolumns

at

the

U

part

are

either

structurally

dense

or

zero

Now

since

Pk

oe

Pk

and

Pk

OE

it

is

clear

that

URk

k

-LSB-

URkj

-LSB-

URkk

URkk

Similarly

we

can

show

that

URk

s

k

s

URk

s

k

s

URkk

The

above

theorem

shows

that

the

L/U

partitioning

can

generate

a

rich

set

of

structurally

dense

subcolumns

or

even

structurally

dense

submatrices

in

a

U

factor

We

also

further

incorporate

this

result

with

supernode

amalgamation

in

Section

and

our

experiments

indicate

that

more

than

of

numerical

updates

is

performed

by

the

routine

DGEMM

in

S

which

shows

the

effectiveness

of

the

L/U

partitioning

method

Figure

demonstrates

the

result

of

a

supernode

partitioning

on

a

sample

sparse

matrix

One

can

see

that

all

the

submatrices

in

the

upper

triangular

part

of

the

matrix

only

contain

structurally

dense

subcolumns

Nonzero

Fill-in

Figure

An

example

of

L/U

supernode

partitioning

Based

on

the

above

theorem

we

can

further

show

a

structural

relationship

between

two

submatrices

in

the

same

supernode

column

block

which

will

be

useful

in

implementing

our

algorithm

to

detect

nonzero

structures

efficiently

for

numerical

updating

Corollary

Given

two

nonzero

submatrices

Uij

where

i

j

and

if

subcolumn

k

in

Uij

is

structurally

dense

then

subcolumn

k

in

is

also

structurally

dense

Proof

The

corollary

is

illustrated

in

Figure

Since

is

nonzero

there

must

be

a

structurally

dense

subrow

in

This

will

lead

to

a

nonzero

element

in

the

subcolumn

k

in

because

the

subcolumn

k

of

Uij

is

structurally

dense

According

to

Theorem

subcolumn

k

in

is

structurally

dense

i

i

j

L

U

U

i

ji'i

ij

i

Figure

An

illustration

for

Corollary

Corollary

Given

two

nonzero

submatrices

Uij

where

i

j

and

is

nonzero

If

Uij

is

structurally

dense

must

be

structurally

dense

Proof

That

is

straightforward

using

Corollary

Supernode

amalgamation

For

most

tested

sparse

matrices

the

average

size

of

a

supernode

after

L/U

partitioning

is

very

small

about

to

columns

This

results

in

very

fine

grained

tasks

Amalgamating

small

supernodes

can

lead

to

great

performance

improvement

for

both

parallel

and

sequential

sparse

codes

because

it

can

improve

caching

performance

and

reduce

interprocessor

communication

overhead

There

could

be

many

ways

to

amalgamate

supernodes

-LSB-

-RSB-

The

basic

idea

is

to

relax

the

restriction

that

all

the

columns

in

a

supernode

must

have

exactly

the

same

nonzero

structure

below

diagonal

The

amalgamation

is

usually

guided

by

a

supernode

elimination

tree

A

parent

could

be

merged

with

its

children

if

the

merging

does

not

introduce

too

many

extra

zero

entries

into

a

supernode

Row

and

column

permutations

are

needed

if

the

parent

is

not

consecutive

with

its

children

However

a

column

permutation

introduced

by

the

above

amalgamation

method

could

undermine

the

correctness

of

the

static

symbolic

factorization

We

have

used

a

simpler

approach

that

does

not

require

any

permutation

This

approach

only

amalgamates

consecutive

supernodes

if

their

nonzero

structures

only

differ

by

a

small

number

of

entries

We

can

control

the

maximum

allowed

differences

by

an

amalgamation

factor

r

Our

experiments

show

that

when

r

is

in

the

range

of

it

gives

the

best

performance

for

the

tested

matrices

and

leads

to

improvement

on

the

execution

times

of

the

sequential

code

The

reason

is

that

by

getting

bigger

supernodes

we

are

getting

larger

dense

structures

although

there

may

be

a

few

zero

entries

in

them

and

we

are

taking

more

advantage

of

kernels

Notice

that

after

applying

the

supernode

amalgamation

the

dense

structures

identified

in

the

Theorem

are

not

strictly

dense

any

more

We

call

them

almost-dense

structures

and

can

still

use

the

result

of

Theorem

with

a

minor

revision

That

is

summarized

in

the

following

corollary

All

the

results

presented

in

Section

are

obtained

using

this

amalgamation

strategy

Corollary

Given

a

sparse

matrix

A

if

supernode

amalgamation

is

applied

to

A

after

the

static

symbolic

factorization

and

L/U

supernode

partitioning

are

performed

on

A

each

nonzero

submatrix

in

the

U

factor

of

A

contains

only

almost-structurally-dense

subcolumns

Program

partitioning

task

dependence

and

processor

mapping

After

dividing

a

sparse

matrix

A

into

submatrices

using

the

L/U

supernode

partitioning

we

need

to

partition

the

LU

code

accordingly

and

define

coarse

grained

tasks

that

manipulate

on

partitioned

dense

data

structures

Program

partitioning

Column

block

partitioning

follows

supernode

structures

Typically

there

are

two

types

of

tasks

One

is

F

actor

-LRB-

k

-RRB-

which

is

to

factorize

all

the

columns

in

the

k-th

column

block

including

finding

the

pivoting

sequence

associated

with

those

columns

The

other

is

Update

-LRB-

k

j

-RRB-

which

is

to

apply

the

pivoting

sequence

derived

from

F

actor

-LRB-

k

-RRB-

to

the

j-th

column

block

and

modify

the

j-th

column

block

using

the

k-th

column

block

where

k

j

and

Ukj

Instead

of

performing

the

row

interchange

to

the

right

part

of

the

matrix

right

after

each

pivoting

search

a

technique

called

delayed-pivoting

is

used

-LSB-

-RSB-

In

this

technique

the

pivoting

sequence

is

held

until

the

factorization

of

the

k-th

column

block

is

completed

Then

the

pivoting

sequence

is

applied

to

the

rest

of

the

matrix

i.e.

interchange

rows

Delayed-pivoting

is

important

especially

to

the

parallel

algorithm

because

it

is

equivalent

to

aggregating

multiple

small

messages

into

a

larger

one

Here

the

owner

of

the

k-th

column

block

sends

the

column

block

packed

together

with

the

pivoting

information

to

other

processors

An

outline

of

the

partitioned

sparse

LU

factorization

algorithm

with

partial

pivoting

is

described

in

Figure

The

code

of

F

actor

-LRB-

k

-RRB-

is

summarized

in

Figure

It

uses

and

BLAS

subroutines

The

computational

cost

of

the

numerical

factorization

is

mainly

dominated

by

Update

-LRB-

-RRB-

tasks

The

function

of

task

Update

-LRB-

k

j

-RRB-

is

presented

in

Figure

The

lines

-LRB-

-RRB-

and

-LRB-

-RRB-

are

using

dense

matrix

multiplications

We

use

directed

acyclic

task

graphs

-LRB-

DAGs

-RRB-

to

model

irregular

parallelism

arising

in

this

partitioned

sparse

LU

program

Previous

work

on

exploiting

task

parallelism

for

sparse

Cholesky

factorization

has

used

elimination

trees

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

which

is

a

good

way

to

expose

the

available

parallelism

because

the

input

matrix

is

symmetric

and

pivoting

is

not

involved

For

sparse

LU

an

elimination

tree

of

ATA

does

not

directly

reflect

the

available

parallelism

Dynamically

created

DAGs

have

been

used

for

modeling

parallelism

and

guiding

run-time

execution

in

a

nonsymmetric

multi-frontal

method

-LSB-

-RSB-

-LRB-

-RRB-

for

k

to

N

-LRB-

-RRB-

Perform

task

F

actor

-LRB-

k

-RRB-

-LRB-

-RRB-

for

j

k

to

N

with

Ukj

-LRB-

-RRB-

Perform

task

Update

-LRB-

k

j

-RRB-

-LRB-

-RRB-

endfor

-LRB-

-RRB-

endfor

Figure

A

partitioned

sparse

LU

factorization

with

partial

pivoting

-LRB-

-RRB-

F

actor

-LRB-

k

-RRB-

-LRB-

-RRB-

for

m

S

-LRB-

k

-RRB-

to

S

-LRB-

k

-RRB-

-LRB-

-RRB-

Find

the

pivoting

row

t

in

column

m

-LRB-

-RRB-

Interchange

row

t

and

row

m

of

the

column

block

k

-LRB-

-RRB-

Scale

column

m

and

update

rest

of

columns

in

this

column

block

-LRB-

-RRB-

endfor

Figure

The

description

of

task

F

actor

-LRB-

k

-RRB-

Given

the

task

definitions

in

Figures

and

we

can

define

the

structure

of

a

sparse

LU

task

graph

in

the

following

ffl

There

are

N

tasks

F

actor

-LRB-

k

-RRB-

where

k

N

ffl

There

is

a

task

Update

-LRB-

k

j

-RRB-

if

Ukj

where

k

j

N

For

a

dense

matrix

there

will

be

a

total

of

N

-LRB-

N

-RRB-

updating

tasks

ffl

There

is

a

dependence

edge

from

F

actor

-LRB-

k

-RRB-

to

task

Update

-LRB-

k

j

-RRB-

where

k

j

and

Ukj

ffl

There

is

a

dependence

from

Update

-LRB-

k

-RRB-

to

F

actor

-LRB-

-RRB-

where

k

and

there

exists

no

task

Update

-LRB-

t

-RRB-

such

that

k

t

ffl

There

is

a

dependence

from

Update

-LRB-

k

j

-RRB-

to

Update

-LRB-

j

-RRB-

where

k

and

there

exists

no

task

Update

-LRB-

t

j

-RRB-

such

that

k

t

Figure

-LRB-

a

-RRB-

shows

the

nonzero

pattern

of

the

partitioned

matrix

shown

in

Figure

Figure

-LRB-

b

-RRB-

is

the

corresponding

task

dependence

graph

data

mapping

In

the

data

mapping

all

submatrices

from

both

L

and

U

part

of

the

same

column

block

will

reside

in

the

same

processor

Column

blocks

are

mapped

to

processors

in

a

is

not

always

necessary

Here

we

take

a

conservative

approach

to

simplify

implementation

Our

experience

with

Cholesky

factorization

with

commuting

Update

-LRB-

-RRB-

tasks

-LSB-

-RSB-

indicates

that

the

loss

of

performance

is

not

substantial

about

in

average

when

graph

scheduling

is

used

-LRB-

-RRB-

Update

-LRB-

k

j

-RRB-

-LRB-

-RRB-

Interchange

rows

according

to

the

pivoting

sequence

-LRB-

-RRB-

Let

Lt

be

the

lower

triangular

part

of

Lkk

-LRB-

-RRB-

if

the

submatrix

Ukj

is

dense

-LRB-

-RRB-

Ukj

L

Ukj

-LRB-

-RRB-

else

for

each

dense

subcolumn

cu

of

Ukj

-LRB-

-RRB-

cu

L

cu

-LRB-

-RRB-

endfor

-LRB-

-RRB-

-LRB-

-RRB-

for

each

nonzero

submatrix

Aij

i

k

in

column

block

j

-LRB-

-RRB-

if

the

submatrix

Ukj

is

dense

-LRB-

-RRB-

Aij

Aij

Lik

Ukj

-LRB-

-RRB-

else

for

each

dense

subcolumn

cu

of

Ukj

-LRB-

-RRB-

Let

cb

be

the

corresponding

dense

subcolumn

of

Aij

-LRB-

-RRB-

cb

cb

Lik

cu

-LRB-

-RRB-

endfor

-LRB-

-RRB-

endfor

Figure

A

description

of

task

Update

-LRB-

k

j

-RRB-

cyclic

manner

or

based

on

other

scheduling

techniques

such

as

graph

scheduling

Tasks

are

assigned

based

on

owner-compute

rule

i.e.

tasks

that

modify

the

same

column

block

are

assigned

to

the

same

processor

that

owns

the

column

block

One

disadvantage

of

this

mapping

is

that

it

serializes

the

computation

in

a

single

F

actor

-LRB-

k

-RRB-

or

Update

-LRB-

k

j

-RRB-

task

In

other

words

a

single

F

actor

-LRB-

k

-RRB-

or

Update

-LRB-

k

j

-RRB-

task

will

be

performed

by

one

processor

But

this

mapping

strategy

has

an

advantage

that

both

pivot

searching

and

subrow

interchange

can

be

done

locally

without

any

communication

Another

advantage

is

that

parallelism

modeled

by

the

above

dependence

structure

can

be

effectively

exploited

using

graph

scheduling

techniques

F

-LRB-

-RRB-

F

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

F

-LRB-

-RRB-

U

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

-LRB-

b

-RRB-

-LRB-

a

-RRB-

Figure

-LRB-

a

-RRB-

The

nonzero

pattern

for

the

example

matrix

in

Figure

-LRB-

b

-RRB-

The

dependence

graph

derived

from

the

partitioning

result

For

convenience

F

-LRB-

-RRB-

is

used

to

denote

F

actor

-LRB-

-RRB-

U

-LRB-

-RRB-

is

used

to

denote

Update

-LRB-

-RRB-

data

mapping

In

the

literature

mapping

has

been

shown

more

scalable

than

for

sparse

Cholesky

-LSB-

-RSB-

However

there

are

several

difficulties

to

apply

the

block-oriented

mapping

to

the

case

of

sparse

LU

factorization

even

the

static

structure

is

predicted

Firstly

pivoting

operations

and

row

interchanges

require

frequent

and

well-synchronized

interprocessor

communication

when

submatrices

in

the

same

column

block

are

assigned

to

different

processors

Effective

exploitation

of

limited

irregular

parallelism

in

the

case

requires

a

highly

efficient

asynchronous

execution

mechanism

and

a

delicate

message

buffer

management

Secondly

it

is

difficult

to

utilize

and

schedule

all

possible

irregular

parallelism

from

sparse

LU

Lastly

how

to

manage

a

low

space

complexity

is

another

issue

since

exploiting

irregular

parallelism

to

a

maximum

degree

may

need

more

buffer

space

Our

algorithm

uses

a

simple

standard

mapping

function

In

this

scheme

p

available

processors

are

viewed

as

a

two

dimensional

grid

p

pr

pc

A

nonzero

submatrix

block

Aij

-LRB-

could

be

an

L

block

or

a

U

block

-RRB-

is

assigned

to

processor

Pi

mod

pr

j

mod

pc

The

data

mapping

is

considered

more

scalable

than

data

mapping

because

it

enables

parallelization

of

a

single

F

actor

-LRB-

k

-RRB-

or

Update

-LRB-

k

j

-RRB-

task

on

pr

processors

We

will

discuss

how

parallelism

is

exploited

using

asynchronous

schedule

execution

Parallelism

exploitation

Scheduling

and

run-time

support

for

methods

We

discuss

how

sparse

LU

tasks

are

scheduled

and

executed

so

that

parallel

time

can

be

minimized

-LSB-

-RSB-

uses

a

dynamic

load

balancing

algorithm

on

a

shared

memory

machine

For

distributed

memory

machines

dynamic

and

adaptive

load

balancing

works

well

for

problems

with

very

coarse

grained

computations

but

it

is

still

an

open

problem

to

balance

the

benefits

of

dynamic

scheduling

with

the

run-time

control

overhead

since

task

and

data

migration

cost

is

too

expensive

for

sparse

problems

with

mixed

granularities

We

use

task

dependence

graphs

to

guide

scheduling

and

have

investigated

two

types

of

scheduling

schemes

ffl

Compute-ahead

scheduling

-LRB-

CA

-RRB-

This

is

to

use

block-cyclic

mapping

of

tasks

with

a

compute-ahead

execution

strategy

which

is

demonstrated

in

Figure

This

idea

has

been

used

to

speed

up

parallel

dense

factorizations

-LSB-

-RSB-

It

executes

the

numerical

factorization

layer

by

layer

based

on

the

current

submatrix

index

The

parallelism

is

exploited

for

concurrent

updating

In

order

to

overlap

computation

with

communication

the

F

actor

-LRB-

k

-RRB-

task

is

executed

as

soon

as

F

actor

-LRB-

k

-RRB-

and

Update

-LRB-

k

k

-RRB-

-LRB-

if

exists

-RRB-

finish

so

that

the

pivoting

sequence

and

column

block

k

for

the

next

layer

can

be

communicated

as

early

as

possible

ffl

Graph

scheduling

We

order

task

execution

within

each

processor

using

the

graph

scheduling

algorithms

in

-LSB-

-RSB-

The

basic

optimizations

are

balancing

processor

loads

and

overlapping

computation

with

communication

to

hide

communication

latency

These

are

done

by

utilizing

global

dependence

structures

and

critical

path

information

-LRB-

-RRB-

if

column

block

is

local

-LRB-

-RRB-

Perform

task

F

actor

-LRB-

-RRB-

-LRB-

-RRB-

Broadcast

column

block

and

the

pivoting

sequence

-LRB-

-RRB-

for

k

to

N

-LRB-

-RRB-

if

column

block

k

is

local

-LRB-

-RRB-

if

Uk

k

-LRB-

-RRB-

Receive

column

block

k

and

the

pivoting

choices

-LRB-

-RRB-

Interchange

rows

according

to

the

pivoting

sequence

-LRB-

-RRB-

Perform

task

Update

-LRB-

k

k

-RRB-

-LRB-

-RRB-

Perform

task

F

actor

-LRB-

k

-RRB-

-LRB-

-RRB-

Broadcast

column

block

k

and

the

pivoting

sequence

-LRB-

-RRB-

for

j

k

to

N

with

Ukj

-LRB-

-RRB-

if

column

block

j

is

local

-LRB-

-RRB-

if

column

block

k

has

not

been

received

-LRB-

-RRB-

Receive

column

block

k

and

the

pivoting

choices

-LRB-

-RRB-

Interchange

rows

according

to

the

pivoting

sequence

-LRB-

-RRB-

Perform

task

Update

-LRB-

k

j

-RRB-

-LRB-

-RRB-

endfor

-LRB-

-RRB-

endfor

Figure

The

code

using

compute-ahead

schedule

Graph

scheduling

has

been

shown

effective

in

exploiting

irregular

parallelism

for

other

applications

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Graph

scheduling

should

outperform

the

CA

scheduling

for

sparse

LU

because

it

does

not

have

a

constraint

in

ordering

F

actor

-LRB-

-RRB-

tasks

We

demonstrate

this

point

using

the

LU

task

graph

in

Figure

For

this

example

the

Gantt

charts

of

the

CA

schedule

and

the

schedule

derived

by

our

graph

scheduling

algorithm

are

listed

in

Figure

It

is

assumed

that

each

task

has

a

computation

weight

and

each

edge

has

communication

weight

It

is

easy

to

see

that

our

scheduling

approach

produces

a

better

result

than

the

CA

schedule

If

we

look

at

the

CA

schedule

carefully

we

can

see

that

the

reason

is

that

CA

can

look

ahead

only

one

step

so

that

the

execution

of

task

F

actor

-LRB-

-RRB-

is

placed

after

Update

-LRB-

-RRB-

On

the

other

hand

the

graph

scheduling

algorithm

detects

that

F

actor

-LRB-

-RRB-

can

be

executed

before

Update

-LRB-

-RRB-

which

leads

to

better

overlap

of

communication

with

computation

However

the

implementation

of

the

CA

algorithm

is

much

easier

since

the

efficient

execution

of

a

sparse

task

graph

schedule

requires

a

sophisticated

run-time

system

to

support

asynchronous

communication

protocols

We

have

used

the

RAPID

run-time

system

-LSB-

-RSB-

for

the

parallelization

of

sparse

LU

using

graph

scheduling

The

key

optimization

is

to

use

Remote

Memory

Access

-LRB-

RMA

-RRB-

to

communicate

a

data

object

between

two

processors

It

does

not

incur

any

copying/buffering

during

a

data

transfer

since

low

communication

overhead

is

critical

for

sparse

code

with

mixed

granularities

RMA

is

available

in

modern

multi-processor

architectures

such

as

-LSB-

-RSB-

-LSB-

-RSB-

and

Meiko

-LSB-

-RSB-

Since

the

RMA

directly

writes

data

to

a

remote

address

it

is

possible

that

the

content

at

the

remote

address

is

still

being

used

by

other

tasks

and

then

the

execution

at

the

remote

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

F

-LRB-

-RRB-

U

-LRB-

-RRB-

F

-LRB-

-RRB-

U

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

-LRB-

b

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

U

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

F

-LRB-

-RRB-

-LRB-

a

-RRB-

Figure

-LRB-

a

-RRB-

A

schedule

derived

by

our

graph

scheduling

algorithm

-LRB-

b

-RRB-

A

compute-ahead

schedule

For

convenience

F

-LRB-

-RRB-

is

used

to

denote

F

actor

-LRB-

-RRB-

U

-LRB-

-RRB-

is

used

to

denote

Update

-LRB-

-RRB-

processor

could

be

incorrect

Thus

for

a

general

computation

a

permission

to

write

the

remote

address

needs

to

be

obtained

before

issuing

a

remote

write

However

in

the

RAPID

system

this

hand-shaking

process

is

avoided

by

a

carefully

designed

task

communication

protocol

-LSB-

-RSB-

This

property

greatly

reduces

task

synchronization

cost

As

shown

in

-LSB-

-RSB-

the

RAPID

sparse

code

can

deliver

more

than

of

the

speedup

predicted

by

the

scheduler

on

In

addition

using

RAPID

system

greatly

reduces

the

amount

of

implementation

work

to

parallelize

sparse

LU

Asynchronous

execution

for

the

code

As

we

discussed

previously

data

mapping

can

not

expose

parallelism

to

a

maximum

extent

Another

issue

is

that

a

time-efficient

schedule

may

not

be

space-efficient

Specifically

to

support

concurrency

among

multiple

updating

stages

in

both

RAPID

and

CA

code

multiple

buffers

are

needed

to

keep

pivoting

column

blocks

of

different

stages

on

each

processor

Therefore

for

a

given

problem

the

per

processor

space

complexity

of

the

codes

could

be

as

high

as

O

-LRB-

-RRB-

where

is

the

space

complexity

for

a

sequential

algorithm

For

sparse

LU

each

processor

in

the

worst

case

may

need

a

space

for

holding

the

entire

matrix

The

RAPID

system

-LSB-

-RSB-

also

needs

extra

memory

space

to

hold

dependence

structures

Based

on

the

above

observation

our

goal

for

the

code

is

to

reduce

memory

space

requirement

while

exploiting

a

reasonable

amount

of

parallelism

so

that

it

can

solve

large

problem

instances

in

an

efficient

way

In

this

section

we

present

an

asynchronous

algorithm

which

can

substantially

overlap

multi-stages

of

updating

but

its

memory

requirement

is

much

smaller

than

that

of

methods

Figures

shows

the

main

control

of

the

algorithm

in

an

SPMD

coding

style

Figure

shows

the

SPMD

code

for

F

actor

-LRB-

k

-RRB-

which

is

executed

by

processors

of

column

k

mod

pc

Recall

that

the

algorithm

uses

block-cyclic

data

mapping

and

the

coordinates

for

the

processor

that

owns

submatrix

Aij

are

-LRB-

i

mod

pr

j

mod

pc

-RRB-

Also

we

divide

the

function

of

Update

-LRB-

-RRB-

-LRB-

in

Figure

-RRB-

into

two

parts

ScaleSwap

-LRB-

-RRB-

which

does

scaling

and

delayed

row

interchange

for

submatrix

Ak

N

k

N

as

shown

in

Figure

Update

-LRB-

-RRB-

which

does

submatrix

updating

as

shown

in

Figure

In

all

-LRB-

-RRB-

Let

-LRB-

my

rno

my

cno

-RRB-

be

the

coordinates

of

this

processor

-LRB-

-RRB-

if

my

cno

then

Perform

task

F

actor

-LRB-

-RRB-

-LRB-

-RRB-

for

k

to

N

-LRB-

-RRB-

Perform

ScaleSwap

-LRB-

k

-RRB-

-LRB-

-RRB-

if

my

cno

-LRB-

k

-RRB-

mod

pc

-LRB-

-RRB-

Perform

Update

-LRB-

k

k

-RRB-

-LRB-

-RRB-

Perform

F

actor

-LRB-

k

-RRB-

-LRB-

-RRB-

for

j

k

to

N

-LRB-

-RRB-

if

my

cno

j

mod

pc

-LRB-

-RRB-

Perform

Updatej

-LRB-

k

j

-RRB-

-LRB-

-RRB-

endfor

-LRB-

-RRB-

endfor

Figure

The

SPMD

code

of

asynchronous

code

figures

the

statements

which

involve

interprocessor

communication

are

marked

with

It

can

be

seen

that

the

computation

flow

of

this

code

is

still

controlled

by

the

pivoting

tasks

F

actor

-LRB-

k

-RRB-

The

order

of

execution

for

F

actor

-LRB-

k

-RRB-

k

N

is

sequential

but

Update

-LRB-

-RRB-

tasks

where

most

of

the

computation

comes

from

can

execute

in

parallel

among

all

processors

The

asynchronous

parallelism

comes

from

two

levels

First

a

single

stage

of

tasks

Update

-LRB-

k

k

N

-RRB-

can

be

executed

concurrently

on

all

processors

In

addition

different

stages

of

Update

-LRB-

-RRB-

tasks

from

Update

-LRB-

k

k

N

-RRB-

and

Update

-LRB-

N

-RRB-

where

k

can

also

be

overlapped

The

idea

of

compute-ahead

scheduling

is

also

incorporated

i.e.

F

actor

-LRB-

k

-RRB-

is

executed

as

soon

as

Update

-LRB-

k

k

-RRB-

finishes

Some

detailed

explanation

for

pivoting

scaling

and

swapping

is

given

below

In

line

-LRB-

-RRB-

of

Figure

the

whole

subrow

is

communicated

when

each

processor

reports

its

local

maximum

to

Pk

mod

pr

k

mod

pc

i.e.

the

processor

that

owns

the

Lkk

block

Let

m

be

the

current

global

column

number

on

which

the

pivoting

is

conducted

then

without

further

synchronization

processor

Pk

mod

pr

k

mod

pc

can

locally

swap

subrow

m

with

subrow

t

which

contains

the

selected

pivoting

element

This

shortens

the

waiting

time

to

conduct

further

updating

with

a

little

more

communication

volume

However

in

line

-LRB-

-RRB-

processor

Pk

mod

pr

k

mod

pc

must

send

the

original

subrow

m

to

the

owner

of

subrow

t

for

swapping

and

the

selected

subrow

t

to

other

processors

as

well

for

updating

In

F

actor

-LRB-

-RRB-

tasks

synchronizations

take

place

at

lines

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

when

each

processor

reports

its

local

maximum

to

Pk

mod

pr

k

mod

pc

and

Pk

mod

pr

k

mod

pc

broadcasts

the

subrow

containing

global

maximum

along

the

processor

column

For

task

ScaleSwap

-LRB-

-RRB-

the

main

role

is

to

scale

Uk

k

N

and

perform

delayed

row

interchanges

for

remaining

submatrices

Ak

N

k

N

We

examine

the

degree

of

parallelism

exploited

in

this

algorithm

by

determining

number

of

updating

stages

that

can

be

overlapped

Using

this

information

we

can

also

determine

the

extra

buffer

space

needed

per

processor

to

execute

this

algorithm

correctly

We

define

the

stage

overlapping

degree

-LRB-

-RRB-

F

actor

-LRB-

k

-RRB-

-LRB-

-RRB-

for

m

S

-LRB-

k

-RRB-

to

S

-LRB-

k

-RRB-

-LRB-

-RRB-

Find

out

local

maximum

element

of

column

m

-LRB-

-RRB-

if

my

rno

k

mod

pr

-LRB-

-RRB-

Send

the

subrow

within

column

block

k

containing

the

local

maximum

to

processor

Pk

mod

pr

k

mod

pc

-LRB-

-RRB-

if

this

processor

owns

Lkk

-LRB-

-RRB-

Collect

all

local

maxima

and

find

the

pivot

row

t

-LRB-

-RRB-

Broadcast

the

subrow

t

within

column

block

k

along

this

processor

column

and

interchange

subrow

t

and

subrow

m

if

necessary

-LRB-

-RRB-

Scale

local

entries

of

column

m

-LRB-

-RRB-

Update

local

subcolumns

from

column

m

to

S

-LRB-

k

-RRB-

-LRB-

-RRB-

endfor

-LRB-

-RRB-

Multicast

the

pivot

sequence

along

this

processor

row

-LRB-

-RRB-

if

this

processor

owns

Lkk

then

Multicast

Lkk

along

this

processor

row

-LRB-

-RRB-

Multicast

the

part

of

nonzero

blocks

in

Lk

N

k

owned

by

this

processor

along

this

processor

row

Figure

Parallel

execution

of

F

actor

-LRB-

k

-RRB-

for

the

asynchronous

code

-LRB-

-RRB-

ScaleSwap

-LRB-

k

-RRB-

-LRB-

-RRB-

if

my

cno

k

mod

pc

then

receive

the

pivot

sequence

from

Pmy

rno

k

mod

pc

-LRB-

-RRB-

for

m

S

-LRB-

k

-RRB-

to

S

-LRB-

k

-RRB-

-LRB-

-RRB-

if

This

processor

own

a

part

of

row

m

or

the

pivot

row

t

for

column

m

-LRB-

-RRB-

Interchange

nonzero

parts

of

row

t

and

row

m

owned

by

this

processor

-LRB-

-RRB-

endfor

-LRB-

-RRB-

if

my

rno

k

mod

pr

-LRB-

-RRB-

if

my

cno

k

mod

pc

then

receive

Lkk

from

Pmy

rno

k

mod

pc

-LRB-

-RRB-

Scale

nonzero

blocks

in

Uk

k

N

owned

by

this

processor

-LRB-

-RRB-

Multicast

the

scaling

results

along

this

processor

column

-LRB-

-RRB-

if

my

cno

k

mod

pc

then

receive

Lk

N

k

from

Pk

mod

pr

my

cno

-LRB-

-RRB-

if

my

rno

k

mod

pr

then

receive

Uk

k

N

from

Pmy

rno

k

mod

pc

Figure

Task

ScaleSwap

-LRB-

k

-RRB-

for

the

asynchronous

code

-LRB-

-RRB-

Update

-LRB-

k

j

-RRB-

-LRB-

-RRB-

if

j

mod

pc

my

cno

-LRB-

-RRB-

for

i

k

to

N

-LRB-

-RRB-

if

i

mod

pr

my

rno

-LRB-

-RRB-

Update

Aij

using

Lik

and

Ukj

-LRB-

-RRB-

endfor

Figure

Update

-LRB-

k

j

-RRB-

for

the

asynchronous

code

for

updating

tasks

as

maxfjk

j

There

exist

tasks

Update

-LRB-

k

-RRB-

and

Update

-LRB-

-RRB-

executed

concurrently.g

Here

Update

-LRB-

k

-RRB-

denotes

a

set

of

Update

-LRB-

k

j

-RRB-

tasks

where

k

j

N

Theorem

For

the

asynchronous

algorithm

on

p

processors

where

p

and

p

pr

pc

the

reachable

upper

bound

of

overlapping

degree

is

pc

among

all

processors

and

the

reachable

upper

bound

of

overlapping

degree

within

a

processor

column

is

min

-LRB-

pr

pc

-RRB-

Proof

We

will

use

the

following

facts

in

proving

the

theorem

ffl

Fact

F

actor

-LRB-

k

-RRB-

is

executed

at

processors

with

column

number

k

mod

pc

Processors

on

this

column

are

synchronized

When

a

processor

completes

F

actor

-LRB-

k

-RRB-

this

processor

can

still

do

Update

-LRB-

k

-RRB-

as

shown

in

Figure

but

all

Update

-LRB-

k

t

-RRB-

tasks

belonging

to

this

processor

where

t

must

have

been

completed

on

this

processor

ffl

Fact

ScaleSwap

-LRB-

k

-RRB-

is

executed

at

processors

with

row

number

k

mod

pc

When

a

processor

completes

ScaleSwap

-LRB-

k

-RRB-

all

Update

-LRB-

k?t

-RRB-

tasks

belonging

to

this

processor

where

t

must

have

been

completed

on

this

processor

Part

First

we

show

that

Update

-LRB-

-RRB-

tasks

can

be

overlapped

to

a

degree

of

pc

among

all

processors

When

pc

it

is

trivial

based

on

Fact

When

pc

we

can

imagine

a

scenario

in

which

all

processors

in

column

have

just

finished

task

F

actor

-LRB-

k

-RRB-

and

some

of

them

are

still

working

on

Update

-LRB-

k

-RRB-

tasks

and

processors

in

column

could

go

ahead

and

execute

Update

-LRB-

k

-RRB-

tasks

After

processors

in

column

finish

Update

-LRB-

k

k

-RRB-

task

they

will

execute

F

actor

-LRB-

k

-RRB-

Then

after

finishing

Update

-LRB-

k

-RRB-

tasks

processors

in

column

could

execute

Update

-LRB-

k

k

-RRB-

task

and

start

F

actor

-LRB-

k

-RRB-

task

and

so

on

Finally

processors

in

column

pc

could

execute

F

actor

-LRB-

k

pc

-RRB-

and

then

start

Update

-LRB-

k

pc

-RRB-

At

this

moment

processors

in

column

may

be

still

working

on

Update

-LRB-

k

-RRB-

Thus

the

overlapping

degree

is

pc

Now

we

will

show

by

contradiction

that

the

maximum

overlapping

degree

is

pc

Assume

that

at

some

moment

there

exist

two

updating

stages

being

executed

concurrently

Update

-LRB-

k

-RRB-

and

Update

-LRB-

-RRB-

where

k

pc

Then

F

actor

-LRB-

-RRB-

must

have

been

completed

Without

loss

of

generality

assuming

that

processors

in

column

execute

F

actor

-LRB-

-RRB-

then

according

to

Fact

all

Update

-LRB-

j

-RRB-

tasks

where

j

should

be

completed

before

this

moment

Since

block

cyclic

mapping

is

used

it

is

easy

to

see

each

processor

column

has

performed

one

of

the

F

actor

-LRB-

j

-RRB-

tasks

where

pc

j

Then

Update

-LRB-

pc

-RRB-

should

be

completed

on

all

processors

Then

for

any

concurrent

stage

Update

-LRB-

k

-RRB-

k

must

satisfy

k

pc

which

is

a

contradiction

Part

First

we

show

that

overlapping

degree

min

-LRB-

pr

pc

-RRB-

can

be

achieved

within

a

processor

column

For

the

convenience

of

illustration

we

consider

a

scenario

in

which

all

delayed

row

interchanges

in

ScaleSwap

-LRB-

-RRB-

take

place

locally

without

any

communication

within

a

processor

column

Therefore

there

is

no

interprocessor

synchronization

going

on

within

a

processor

column

except

in

F

actor

-LRB-

-RRB-

tasks

Assuming

pr

pc

we

can

imagine

at

some

moment

processors

in

column

have

completed

F

actor

-LRB-

s

-RRB-

and

has

just

finished

ScaleSwap

-LRB-

s

-RRB-

and

starts

executing

Update

-LRB-

s

-RRB-

where

s

mod

pr

and

s

mod

pc

Then

processors

in

column

will

execute

Update

-LRB-

s

s

-RRB-

and

then

F

actor

-LRB-

s

-RRB-

after

which

can

start

ScaleSwap

-LRB-

s

-RRB-

and

then

Update

-LRB-

s

-RRB-

Following

this

reasoning

after

Update

-LRB-

s

pr

s

pr

-RRB-

and

F

actor

-LRB-

s

pr

-RRB-

have

been

finished

on

processors

of

column

pr

Ppr

could

complete

previous

Update

-LRB-

-RRB-

tasks

and

ScaleSwap

-LRB-

s

pr

-RRB-

and

start

Update

-LRB-

s

pr

-RRB-

Now

may

be

still

working

on

Update

-LRB-

s

-RRB-

Thus

the

overlapping

degree

is

pr

If

pr

pc

obviously

the

above

reasoning

will

stop

when

processors

of

column

pc

finish

Update

-LRB-

s

pc

s

pc

-RRB-

and

F

actor

-LRB-

s

pc

-RRB-

In

that

case

when

Ppc

is

to

start

Update

-LRB-

s

pc

-RRB-

the

processor

Ppr

could

be

still

working

on

Update

-LRB-

s

-RRB-

because

of

the

compute

ahead

scheduling

Hence

the

overlapping

degree

is

pc

Now

we

need

to

show

that

the

upper

bound

of

overlapping

degree

within

a

processor

column

is

min

-LRB-

pr

pc

-RRB-

We

have

already

shown

in

the

proof

of

Part

that

the

overall

overlapping

degree

is

less

than

pc

so

is

the

overlapping

degree

within

a

processor

column

To

prove

it

is

also

less

than

pr

we

can

use

the

similar

proof

as

that

for

part

except

using

ScaleSwap

-LRB-

k

-RRB-

to

replace

F

actor

-LRB-

k

-RRB-

and

using

Fact

instead

of

Fact

Knowing

degree

of

overlapping

is

important

in

determining

the

amount

of

memory

space

needed

to

accommodate

those

communication

buffers

on

each

processor

for

supporting

asynchronous

execution

Buffer

space

is

additional

to

data

space

needed

to

distribute

the

original

matrix

There

are

four

types

of

communication

that

needs

buffering

Pivoting

along

a

processor

column

-LRB-

lines

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

in

Figure

-RRB-

which

includes

communicating

pivot

positions

and

multicasting

pivot

rows

We

call

the

buffer

for

this

purpose

Pbuffer

Multicasting

along

a

processor

row

-LRB-

line

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

in

Figure

-RRB-

The

communicated

data

includes

Lkk

local

nonzero

blocks

in

Lk

N

k

and

pivoting

sequences

We

call

the

buffer

for

this

purpose

Cbuffer

Row

interchange

within

a

processor

column

-LRB-

line

-LRB-

-RRB-

in

Figure

-RRB-

We

call

this

buffer

Ibuffer

Multicasting

along

a

processor

column

-LRB-

line

-LRB-

-RRB-

in

Figure

-RRB-

The

data

includes

local

nonzero

blocks

of

a

row

panel

We

call

the

buffer

Rbuffer

Here

we

assume

that

pr

pc

because

based

on

our

experimental

results

setting

pr

pc

always

leads

to

better

performance

Thus

the

overlapping

degree

of

Update

-LRB-

-RRB-

tasks

within

a

processor

row

is

at

most

pc

and

the

overlapping

degree

within

a

processor

column

is

at

most

pr

Then

we

need

pc

separate

Cbuffer

's

for

overlapping

among

different

columns

and

pr

separate

Rbuffer

's

for

overlapping

among

different

rows

We

estimate

the

size

of

each

Cbuffer

and

Rbuffer

as

follows

Assuming

that

the

sparsity

ratio

of

a

given

matrix

is

s

after

fill-in

and

the

maximum

block

size

is

BSIZE

each

Cbuffer

is

of

size

C

maxfspace

for

local

nonzero

blocks

of

Lk

N

k

k

Ng

ss

n

BSIZE

s

pr

Similarly

each

Rbuffer

is

of

size

R

maxfspace

for

local

nonzero

blocks

of

Uk

k

N

k

Ng

ss

n

BSIZE

s

pc

We

ignore

the

buffer

size

for

Pbuffer

and

Ibuffer

because

they

are

very

small

-LRB-

the

size

of

Pbuffer

is

only

about

BSIZE

BSIZE

and

the

size

of

Ibuffer

is

about

s

n

pc

-RRB-

Thus

the

total

buffer

space

needed

for

the

asynchronous

execution

is

C

pc

R

-LRB-

pr

-RRB-

ss

n

BSIZE

s

-LRB-

pc

pr

pr

pc

-RRB-

Notice

that

the

sequential

space

complexity

In

practice

we

set

pc

pr

Therefore

the

buffer

space

complexity

for

each

processor

is

n

BSIZE

s

or

BSIZE

n

which

is

very

small

for

a

large

matrix

For

all

the

benchmark

matrices

we

have

tested

the

buffer

space

is

less

than

K

words

Given

a

sparse

matrix

if

the

matrix

data

is

evenly

distributed

onto

p

processors

the

total

memory

requirement

per

processor

is

=p

O

-LRB-

-RRB-

considering

n

AE

p

and

n

AE

BSIZE

This

leads

us

to

conclude

that

the

asynchronous

algorithm

is

very

space

scalable

Experimental

studies

Our

experiments

were

originally

conducted

on

a

distributed

memory

machine

at

San

Diego

Supercomputing

Center

Each

node

of

the

includes

a

DEC

Alpha

-LRB-

-RRB-

processor

with

Mbytes

of

memory

The

size

of

the

internal

cache

is

Kbytes

per

processor

The

matrix-matrix

multiplication

routine

DGEMM

can

achieve

MFLOPS

and

the

matrixvector

multiplication

routine

DGEMV

can

reach

MFLOPS

These

numbers

are

obtained

assuming

all

the

data

is

in

cache

and

using

cache

read-ahead

optimization

on

and

the

matrix

block

size

is

chosen

as

The

communication

network

of

the

is

a

torus

Cray

provides

a

shared

memory

access

library

called

shmem

which

can

achieve

Mbytes/s

bandwidth

and

s

communication

overhead

using

shmem

put

-LRB-

-RRB-

primitive

-LSB-

-RSB-

We

have

used

shmem

put

-LRB-

-RRB-

for

the

communications

in

all

the

implementations

We

have

also

conducted

experiments

on

a

newly

acquired

at

San

Diego

Supercomputing

Center

Each

node

has

a

clock

rate

of

MHZ

an

internal

cache

second

level

cache

and

Mbytes

main

memory

The

peak

bandwidth

between

nodes

is

reported

as

Mbytes/s

and

the

peak

round

trip

communication

latency

is

about

to

s

-LSB-

-RSB-

We

have

observed

that

when

block

size

is

DGEMM

achieves

MFLOPS

while

DGEMV

reaches

MFLOPS

We

have

used

block

size

in

our

experiments

since

if

the

block

size

is

too

large

the

available

parallelism

will

be

reduced

In

this

section

we

mainly

report

results

on

In

some

occasions

that

the

absolute

performance

is

concerned

we

also

list

the

results

on

to

see

how

our

approach

scales

when

the

underline

architecture

is

upgraded

All

the

results

are

obtained

on

unless

explicitly

stated

In

calculating

the

MFLOPS

achieved

by

our

parallel

algorithms

we

do

not

include

extra

floating

point

operations

introduced

by

the

overestimation

We

use

the

following

formula

Achieved

MFLOPS

Operation

count

obtained

from

SuperLU

Parallel

time

of

our

algorithm

on

or

The

operation

count

for

a

matrix

is

reported

by

running

SuperLU

code

on

a

SUN

workstation

with

large

memory

since

SuperLU

code

can

not

run

for

some

large

matrices

on

a

single

or

node

due

to

memory

constraint

We

also

compare

the

S

sequential

code

with

SuperLU

to

make

sure

that

the

code

using

static

symbolic

factorization

is

not

too

slow

and

will

not

prevent

the

parallel

version

from

delivering

high

megaflops

Impact

of

static

symbolic

factorization

on

sequential

performance

We

study

if

the

introduction

of

extra

nonzero

elements

by

the

static

factorization

substantially

affects

the

time

complexity

of

numerical

factorization

We

compare

the

performance

of

the

S

sequential

code

with

SuperLU

code

performance

in

Table

for

those

matrices

from

Table

that

can

be

executed

on

a

single

or

node

We

also

introduce

two

other

matrices

to

show

how

well

the

method

works

for

larger

matrices

and

denser

matrices

One

of

the

two

matrices

is

which

is

truncated

from

because

of

the

current

sequential

implementation

is

not

able

to

handle

the

entire

matrix

due

to

memory

constraint

and

the

other

one

is

Though

the

static

symbolic

factorization

introduces

a

lot

of

extra

computation

as

shown

in

Table

the

performance

of

S

after

L/U

partitioning

is

consistently

competitive

to

that

of

highly

optimized

SuperLU

The

absolute

single

node

performance

that

has

been

achieved

by

the

S

approach

on

both

and

is

consistently

in

the

range

of

of

the

highest

DGEMM

performance

for

those

matrices

of

small

or

medium

sizes

Considering

the

fact

that

sparse

codes

usually

suffer

poor

cache

reuse

this

performance

is

reasonable

In

addition

the

amount

of

computation

for

the

testing

matrices

in

Table

is

small

ranging

from

to

million

double

precision

floating

operations

Since

the

characteristic

of

the

S

approach

is

to

explore

more

dense

structures

and

utilize

Matrix

S

Approach

SuperLU

Exec

Time

Ratio

Seconds

Mflops

Seconds

Mflops

S

SuperLU

goodwin

Table

Sequential

performance

S

versus

SuperLU

A

implies

the

data

is

not

available

due

to

insufficient

memory

kernels

better

performance

is

expected

on

larger

or

denser

matrices

This

is

verified

on

a

matrix

For

even

larger

matrices

such

as

we

can

not

run

S

on

one

node

but

as

shown

later

the

code

can

achieve

MFLOPS

per

node

on

processors

Notice

that

the

megaflops

performance

per

node

for

sparse

Choleksy

reported

in

-LSB-

-RSB-

on

nodes

is

around

MFLOPS

which

is

also

a

good

indication

that

S

single-node

performance

is

satisfactory

We

present

a

quantitative

analysis

to

explain

why

S

can

be

competitive

to

SuperLU

Assume

the

speed

of

kernel

is

second

f

lop

and

the

speed

of

kernel

is

second

f

lop

The

total

amount

of

numerical

updates

is

C

f

lops

for

SuperLU

and

C

f

lops

for

the

S

Apparently

C

C.

For

simplicity

we

ignore

the

computation

from

the

scaling

part

within

each

column

because

it

contributes

very

little

to

the

total

execution

time

Hence

we

have

TSuperLU

Tsymbolic

C

and

-LRB-

-RRB-

TS

-LRB-

ae

-RRB-

C

ae

C

-LRB-

-RRB-

where

Tsymbolic

is

the

time

spent

on

dynamic

symbolic

factorization

in

the

SuperLU

approach

ae

is

the

percentage

of

the

numerical

updates

that

are

performed

by

DGEMM

in

S

Let

be

the

ratio

of

symbolic

factorization

time

to

numerical

factorization

time

in

SuperLU

then

we

simplify

Equation

-LRB-

-RRB-

to

the

following

TSuperLU

-LRB-

-RRB-

C

-LRB-

-RRB-

We

estimate

that

ss

for

the

tested

matrices

based

on

the

results

in

-LSB-

-RSB-

In

-LSB-

-RSB-

we

have

also

measured

ae

as

approximately

ae

ss

The

ratios

of

the

number

of

floating

point

operations

performed

in

S

and

SuperLU

for

the

tested

matrices

are

available

in

Table

In

average

the

value

of

C

is

We

plug

in

these

typical

parameters

in

Equation

and

and

we

have

TS

TSuperLU

ss

-LRB-

-RRB-

-LRB-

-RRB-

For

s

f

lop

and

s

f

lop

Then

we

can

get

TS

TSuperLU

ss

For

s

f

lop

and

s

f

lop

And

we

get

TS

TSuperLU

ss

These

estimations

are

close

to

the

ratios

obtained

in

Table

The

discrepancy

is

caused

by

the

fact

that

the

submatrix

sizes

of

supernodes

are

non-uniform

which

leads

to

different

caching

performance

If

submatrices

are

of

uniform

sizes

we

expect

our

prediction

is

more

accurate

For

instance

in

the

dense

case

C

is

exactly

The

ratio

TS

TSuperLU

is

calculated

as

for

and

for

which

are

almost

the

same

as

the

ratios

listed

in

Table

The

above

analysis

shows

that

using

as

much

as

possible

makes

S

competitive

to

SuperLU

Suppose

in

a

machine

that

DGEMM

outperforms

DGEMV

substantially

and

the

ratio

of

the

computation

that

is

performed

by

DGEMM

is

high

enough

S

could

be

faster

than

SuperLU

for

some

matrices

The

last

two

entries

in

Table

have

already

shown

this

Parallel

performance

of

codes

In

this

subsection

we

report

a

set

of

experiments

conducted

to

examine

the

overall

parallel

performance

of

codes

the

effectiveness

of

scheduling

and

supernode

amalgamation

Overall

Performance

We

list

the

MFLOPS

numbers

of

the

RAPID

code

obtained

on

various

number

of

processors

for

several

testing

matrices

in

Table

-LRB-

A

entry

implies

the

data

is

not

available

due

memory

constraint

same

below

-RRB-

We

know

that

the

megaflops

of

DGEMM

on

is

about

times

as

large

as

that

on

and

the

RAPID

code

after

using

a

upgraded

machine

is

speeded

up

about

times

in

average

which

is

satisfactory

For

the

same

machine

the

performance

of

the

RAPID

code

increases

when

the

number

of

processors

increases

and

speedups

compared

to

the

pure

S

sequential

code

-LRB-

if

applicable

-RRB-

can

reach

up

to

on

nodes

and

on

nodes

From

to

nodes

the

performance

gain

is

small

except

for

matrices

goodwin

and

which

are

much

larger

problems

than

the

rest

The

reason

is

that

those

small

tested

matrices

do

not

have

enough

amount

of

computation

and

parallelism

to

saturate

a

large

number

of

processors

when

the

elimination

process

proceeds

toward

the

end

It

is

our

belief

that

better

and

more

scalable

performance

can

be

obtained

on

larger

matrices

But

currently

the

available

memory

on

each

node

of

or

limits

the

problem

size

that

can

be

solved

with

the

current

version

of

the

RAPID

code

Effectiveness

of

Graph

Scheduling

We

compare

the

performance

of

CA

code

with

RAPID

code

in

Figure

The

Y

axis

is

PTRAP

ID

PTCA

where

PT

stands

for

parallel

time

For

and

processors

in

certain

cases

the

compute-ahead

code

is

slightly

faster

than

the

RAPID

code

But

for

the

number

of

processors

more

than

the

RAPID

code

runs

faster

The

Matrix

P

P

P

P

P

P

goodwin

Table

Absolute

performance

-LRB-

MFLOPS

-RRB-

of

the

RAPID

code

more

processors

involved

the

bigger

the

performance

gap

tends

to

be

The

reason

is

that

for

a

small

number

of

processors

there

are

sufficient

tasks

making

all

processors

busy

and

the

computeahead

schedule

performs

well

while

the

RAPID

code

suffers

a

certain

degree

of

system

overhead

For

a

larger

number

of

processors

schedule

optimization

becomes

important

since

there

is

limited

parallelism

to

exploit

Comparison

of

the

RAPID

Code

with

the

CA

Code

#proc

-LRB-

RAPID

-RRB-

PT

-LRB-

CA

-RRB-

x

o

Comparison

of

the

RAPID

Code

with

the

CA

Code

#proc

-LRB-

RAPID

-RRB-

PT

-LRB-

CA

-RRB-

o

x

goodwin

Figure

Impact

of

different

scheduling

strategies

on

code

approach

Effectiveness

of

supernode

amalgamation

We

have

examined

how

effective

our

supernode

amalgamation

strategy

is

using

the

RAPID

code

Let

PTa

and

PT

be

the

parallel

time

with

and

without

supernode

amalgamation

respectively

The

parallel

time

improvement

ratios

-LRB-

PTa

PT

-RRB-

on

for

several

testing

matrices

are

listed

in

Table

and

similar

results

on

are

in

-LSB-

-RSB-

Apparently

the

supernode

amalgamation

has

brought

significant

improvement

due

to

the

increase

of

supernode

size

which

implies

an

increase

of

the

task

granularities

This

is

important

to

obtaining

good

parallel

performance

-LSB-

-RSB-

Matrix

P

P

P

P

P

P

Table

Parallel

time

improvement

obtained

by

supernode

amalgamation

code

performance

As

mentioned

before

our

code

exploits

more

parallelism

but

maintains

a

lower

space

complexity

and

has

much

more

potential

to

solve

large

problems

We

show

the

absolute

performance

obtained

for

some

large

matrices

on

in

Table

Since

some

matrices

can

not

fit

for

a

small

number

of

processors

we

only

list

results

on

or

more

processors

The

maximum

absolute

performance

achieved

on

nodes

of

is

GFLOPS

which

is

translated

to

MFLOPS

per

node

For

nodes

the

per-node

performance

is

MFLOPS

P

P

P

Matrix

Time

-LRB-

Sec

-RRB-

Mflops

Time

-LRB-

Sec

-RRB-

Mflops

Time

-LRB-

Sec

-RRB-

Mflops

goodwin

Table

Performance

results

of

the

code

for

large

matrices

on

Table

shows

the

performance

numbers

on

for

the

code

We

have

achieved

up

to

GFLOPS

on

nodes

For

nodes

megaflops

on

are

from

to

times

as

large

as

that

on

Again

considering

that

DGEMM

megaflops

on

is

about

times

as

large

as

that

on

our

code

performance

after

using

a

upgraded

machine

is

good

Notice

that

codes

can

not

solve

the

last

six

matrices

of

Table

For

those

matrices

solvable

using

both

RAPID

and

codes

we

compare

the

average

parallel

time

differences

by

computing

the

ratio

PTRAP

ID

and

the

result

is

in

Figure

The

RAPID

code

achieves

better

performance

because

it

uses

sophisticated

graph

scheduling

technique

to

guide

the

mapping

of

column

blocks

and

ordering

of

tasks

which

results

in

better

overlapping

of

communication

with

computation

The

performance

difference

is

larger

for

the

matrices

listed

in

the

left

of

Figure

compared

to

the

right

of

Figure

We

partially

explain

the

reason

by

analyzing

load

balance

Matrix

P

P

P

P

P

Time

Mflops

Time

Mflops

Time

Mflops

Time

Mflops

Time

Mflops

goodwin

inaccura

Table

Performance

results

of

asynchronous

algorithm

on

All

times

are

in

seconds

factors

of

the

RAPID

code

and

the

code

in

Figure

The

load

balance

factor

is

defined

as

worktotal

-LRB-

P

workmax

-RRB-

-LSB-

-RSB-

Here

we

only

count

the

work

from

the

updating

part

because

it

is

the

major

part

of

the

computation

The

code

has

better

load

balance

which

can

make

up

for

the

impact

of

lacking

of

efficient

task

scheduling

This

is

verified

by

Figure

and

Figure

One

can

see

that

when

the

load

balance

factor

of

the

code

is

close

to

that

of

the

RAPID

code

-LRB-

e.g.

-RRB-

the

performance

of

the

RAPID

code

is

much

better

than

the

code

when

the

load

balance

factor

of

the

code

is

significantly

better

than

that

of

the

RAPID

code

-LRB-

e.g.

and

-RRB-

the

performance

differences

are

smaller

Comparison

of

the

RAPID

Code

with

the

Code

#proc

-LRB-

RAPID

-RRB-

PT

-LRB-

-RRB-

x

o

Comparison

of

the

RAPID

Code

with

the

Code

#proc

-LRB-

RAPID

-RRB-

PT

-LRB-

-RRB-

o

x

goodwin

Figure

Performance

improvement

of

RAPID

over

code

PTRAP

ID

Synchronous

versus

asynchronous

code

Using

a

global

barrier

in

the

code

at

each

elimination

step

can

simplify

the

implementation

but

it

can

not

overlap

computations

among

different

updating

stages

We

have

compared

parallel

time

reductions

between

the

asynchronous

code

and

the

synchronous

code

for

some

testing

matrices

in

Table

It

shows

that

asynchronous

design

improves

performance

significantly

especially

on

large

number

of

processors

on

It

demonstrates

the

importance

of

exploiting

parallelism

using

asynchronous

execution

The

experiment

Load

balance

comparison

of

RAPID

v.s.

#proc

load

balance

factor

o

x

RAPID

Load

balance

comparison

of

RAPID

v.s.

#proc

load

balance

factor

o

x

RAPID

Figure

Comparison

of

load

balance

factors

of

RAPID

code

and

code

data

on

is

in

-LSB-

-RSB-

Matrix

P

P

P

P

P

P

goodwin

Table

Performance

improvement

of

asynchronous

code

over

synchronous

code

Concluding

remarks

In

this

paper

we

present

an

approach

for

parallelizing

sparse

LU

factorization

with

partial

pivoting

on

distributed

memory

machines

The

major

contribution

of

this

paper

is

that

we

integrate

several

techniques

together

such

as

static

symbolic

factorization

scheduling

for

asynchronous

parallelism

L/U

supernode

partitioning

techniques

to

effectively

identify

dense

structures

and

maximize

the

use

of

subroutines

in

the

algorithm

design

Using

these

ideas

we

are

able

to

exploit

more

data

regularity

for

this

open

irregular

problem

and

achieve

up

to

GFLOPS

on

nodes

This

is

the

highest

known

performance

for

this

challenging

problem

and

the

previous

record

was

GFLOPS

on

shared

memory

machines

-LSB-

-RSB-

The

comparison

results

show

that

the

code

has

a

better

scalability

than

codes

because

mapping

exposes

more

parallelism

with

a

carefully

designed

buffering

scheme

The

scheduling

and

execution

techniques

for

the

code

are

simple

and

are

not

competitive

to

graph

scheduling

if

there

is

sufficient

memory

It

is

an

open

problem

to

develop

advanced

scheduling

techniques

that

better

exploit

parallelism

for

sparse

LU

factorization

with

partial

pivoting

There

are

other

issues

which

are

related

to

this

work

and

need

to

be

further

studied

for

example

alternative

for

parallel

sparse

LU

based

on

Schur

complements

-LSB-

-RSB-

and

static

estimation

and

parallelism

exploitation

for

sparse

QR

-LSB-

-RSB-

It

should

be

noted

that

the

static

symbolic

factorization

could

fail

to

be

practical

if

the

input

matrix

has

a

nearly

dense

row

because

it

will

lead

to

an

almost

complete

fill-in

of

the

whole

matrix

Fortunately

this

is

not

the

case

in

most

of

matrices

we

have

tested

Therefore

our

approach

is

applicable

to

a

wide

range

of

problems

It

is

an

open

problem

to

develop

a

parallel

LU

algorithm

which

consistently

delivers

good

performance

for

all

sparse

matrices

on

distributed

memory

machines

Acknowledgment

This

work

is

supported

by

NSF

and

a

startup

fund

from

UCSB

We

would

like

to

thank

Xiaoye

Li

and

Jim

Demmel

for

helpful

discussion

and

providing

us

their

testing

matrices

and

SuperLU

code

Cleve

Ashcraft

Tim

Davis

Apostolos

Gerasoulis

Esmond

Ng

Ed

Rothberg

Rob

Schreiber

Horst

Simon

Chunguang

Sun

Kathy

Yelick

and

anonymous

referees

for

their

valuable

comments

References

-LSB-

-RSB-

C.

Ashcraft

and

R.

Grimes

The

Influence

of

Relaxed

Supernode

Partitions

on

the

Multifrontal

Method

ACM

Transactions

on

Mathematical

Software

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

C.

Ashcraft

R.

Grimes

J.

Lewis

B.

Peyton

and

H.

Simon

Progress

in

Sparse

Matrix

Methods

for

Large

Sparse

Linear

Systems

on

Vector

Supercomputers

International

Journal

of

Supercomputer

Applications

-LCB-

-LSB-

-RSB-

T.

Davis

User

's

guide

for

the

Unsymmetric-pattern

Multifrontal

Package

-LRB-

UMFPACK

-RRB-

Technical

Report

Computer

and

Information

Sciences

Department

University

of

Florida

June

-LSB-

-RSB-

T.

Davis

Personal

Communication

-LSB-

-RSB-

T.

Davis

and

I.

S.

Duff

An

Unsymmetric-pattern

Multifrontal

Method

for

Sparse

LU

factorization

SIAM

Matrix

Analysis

Applications

January

-LSB-

-RSB-

J.

Demmel

Numerical

Linear

Algebra

on

Parallel

Processors

Lecture

Notes

for

NSF-CBMS

Regional

Conference

in

the

Mathematical

Sciences

June

To

be

published

as

a

book

by

SIAM

-LSB-

-RSB-

J.

Demmel

S.

Eisenstat

J.

Gilbert

X.

Li

and

J.

Liu

A

Supernodal

Approach

to

Sparse

Partial

Pivoting

Technical

Report

UC

Berkeley

September

-LSB-

-RSB-

J.

Dongarra

J.

Du

Croz

S.

Hammarling

and

R.

Hanson

An

Extended

Set

of

Basic

Linear

Algebra

Subroutines

ACM

Trans

on

Mathematical

Software

-LCB-

-LSB-

-RSB-

I.

S.

Duff

On

Algorithms

for

Obtaining

a

Maximum

Transversal

ACM

Transactions

on

Mathematical

Software

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

I.

S.

Duff

Personal

Communication

-LSB-

-RSB-

S.

Eisenstat

and

J.

W.

H.

Liu

Structural

Representations

of

Schur

Complements

in

Sparse

Matrices

volume

of

Graph

Theory

and

Sparse

Matrix

Computation

-LRB-

Edited

by

Alan

George

and

John

R.

Gilbert

and

Joseph

W.H.

Liu

-RRB-

pages

-LCB-

Springer-Verlag

New

York

-LSB-

-RSB-

C.

Fu

X.

Jiao

and

T.

Yang

A

Comparison

of

and

Data

Mapping

for

Sparse

LU

Factorization

with

Partial

Pivoting

In

Proc

of

Eighth

SIAM

Conference

on

Parallel

Processing

for

Scientific

Computing

March

-LSB-

-RSB-

C.

Fu

and

T.

Yang

Efficient

Run-time

Support

for

Irregular

Task

Computations

with

Mixed

Granularities

In

Proceedings

of

IEEE

International

Parallel

Processing

Symposium

pages

-LCB-

Hawaii

April

-LSB-

-RSB-

C.

Fu

and

T.

Yang

Run-time

Compilation

for

Parallel

Sparse

Matrix

Computations

In

Proceedings

of

ACM

International

Conference

on

Supercomputing

pages

-LCB-

Philadelphia

May

-LSB-

-RSB-

C.

Fu

and

T.

Yang

Sparse

LU

Factorization

with

Partial

Pivoting

on

Distributed

Memory

Machines

In

Proceedings

of

ACM/IEEE

Supercomputing

Pittsburgh

November

-LSB-

-RSB-

K.

Gallivan

B.

Marsolf

and

H.

Wijshoff

The

Parallel

Solution

of

Nonsymmetric

Sparse

Linear

Systems

using

H

Reordering

and

an

Associated

Factorization

In

Proc

of

ACM

International

Conference

on

Supercomputing

pages

-LCB-

Manchester

July

-LSB-

-RSB-

A.

George

and

E.

Ng

Symbolic

Factorization

for

Sparse

Gaussian

Elimination

with

Partial

Pivoting

SIAM

J.

Scientific

and

Statistical

Computing

-LRB-

-RRB-

-LCB-

November

-LSB-

-RSB-

A.

George

and

E.

Ng

Parallel

Sparse

Gaussian

Elimination

with

Partial

Pivoting

Annals

of

Operations

Research

-LCB-

-LSB-

-RSB-

A.

Gerasoulis

and

T.

Yang

On

the

Granularity

and

Clustering

of

Directed

Acyclic

Task

Graphs

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

G.

Golub

and

J.

M.

Ortega

Scientific

Computing

An

Introduction

with

Parallel

Computing

Compilers

Academic

Press

-LSB-

-RSB-

A.

Gupta

and

V.

Kumar

Highly

Scalable

Parallel

Algorithms

for

Sparse

Matrix

Factorization

Technical

Report

TR

Computer

Science

Univ.

of

Minnesota

August

-LSB-

-RSB-

S.

Hadfield

and

T.

Davis

A

Parallel

Unsymmetric-pattern

Multifrontal

Method

Technical

Report

Computer

and

Information

Sciences

Department

University

of

Florida

August

-LSB-

-RSB-

M.

Heath

E.

Ng

and

B.

Peyton

Parallel

Algorithms

for

Sparse

Linear

Systems

SIAM

Review

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

J.

W.

H.

Liu

Computational

Models

and

Task

Scheduling

for

Parallel

Sparse

Cholesky

Factorization

Parallel

Computing

-LCB-

-LSB-

-RSB-

P.

Raghavan

Distributed

Sparse

Gaussian

Elimination

and

Orthogonal

Factorization

SIAM

Journal

on

Scientific

Computing

-LRB-

-RRB-

-LCB-

November

-LSB-

-RSB-

E.

Rothberg

Exploiting

the

Memory

Hierarchy

in

Sequential

and

Parallel

Sparse

Cholesky

Factorization

PhD

thesis

Dept.

of

Computer

Science

Stanford

December

-LSB-

-RSB-

E.

Rothberg

and

R.

Schreiber

Improved

Load

Distribution

in

Parallel

Sparse

Cholesky

Factorization

In

Proc

of

Supercomputing

pages

-LCB-

November

-LSB-

-RSB-

S.

L.

Scott

Synchronization

and

Communication

in

the

Multiprocess

In

ASPLOS-VII

October

-LSB-

-RSB-

S.

L.

Scott

and

G.

M.

Thorson

The

Cray

Network

Adaptive

Routing

in

a

High

Performance

Torus

In

Proceedings

of

HOT

Interconnects

IV

August

-LSB-

-RSB-

T.

Stricker

J.

Stichnoth

D.

O'Hallaron

S.

Hinrichs

and

T.

Gross

Decoupling

Synchronization

and

Data

Transfer

in

Message

Passing

Systems

of

Parallel

Computers

In

Proc

of

International

Conference

on

Supercomputing

pages

-LCB-

Barcelona

July

-LSB-

-RSB-

C.

Sun

Parallel

Sparse

Orthogonal

Factorization

on

Distributed-memory

Multiprocessors

SIAM

Journal

on

Scientific

Computing

-LRB-

-RRB-

-LCB-

May

-LSB-

-RSB-

T.

Yang

and

A.

Gerasoulis

PYRROS

Static

Task

Scheduling

and

Code

Generation

for

Message-Passing

Multiprocessors

In

Proc

of

ACM

International

Conference

on

Supercomputing

pages

-LCB-

asynchronous computation scheduling
dense structures
irregular parallelism
partial pivoting
sparse gaussian elimination
sparse lu factorization