A

Supernodal

Approach

to

Sparse

Partial

Pivoting

James

W.

Demmel

Stanley

C.

Eisenstaty

John

R.

Gilbertz

Xiaoye

S.

Li

Joseph

W.

H.

Liux

July

Abstract

We

investigate

several

ways

to

improve

the

performance

of

sparse

LU

factorization

with

partial

pivoting

as

used

to

solve

unsymmetric

linear

systems

To

perform

most

of

the

numerical

computation

in

dense

matrix

kernels

we

introduce

the

notion

of

unsymmetric

supernodes

To

better

exploit

the

memory

hierarchy

we

introduce

unsymmetric

supernode-panel

updates

and

two-dimensional

data

partitioning

To

speed

up

symbolic

factorization

we

use

Gilbert

and

Peierls

's

depth-first

search

with

Eisenstat

and

Liu

's

symmetric

structural

reductions

We

have

implemented

a

sparse

LU

code

using

all

these

ideas

We

present

experiments

demonstrating

that

it

is

significantly

faster

than

earlier

partial

pivoting

codes

We

also

compare

performance

with

Umfpack

which

uses

a

multifrontal

approach

our

code

is

usually

faster

AMS

-LRB-

MOS

-RRB-

subject

classifications

Computing

Reviews

descriptors

G.

-LSB-

Numerical

Analysis

-RSB-

Numerical

Linear

Algebra

Linear

systems

-LRB-

direct

and

iterative

methods

-RRB-

Sparse

and

very

large

systems

Computer

Science

Division

University

of

California

Berkeley

CA

-LRB-

fdemmel,xiaoyeg@cs.berkeley.edu

-RRB-

The

research

of

these

authors

was

supported

in

part

by

NSF

grant

ASC

-LCB-

DOE

grant

DE

-LCB-

-LCB-

UT

Subcontract

No

from

ARPA

Contract

No

-LCB-

-LCB-

DOE

grant

DE

-LCB-

-LCB-

and

NSF

Infrastructure

grants

CDA

-LCB-

and

CDA

-LCB-

yDepartment

of

Computer

Science

Yale

University

P.O.

Box

New

Haven

CT

-LRB-

sce@cs.yale.edu

-RRB-

The

research

of

this

author

was

supported

in

part

by

NSF

grant

zXerox

Palo

Alto

Research

Center

Coyote

Hill

Road

Palo

Alto

California

-LRB-

gilbert@xerox.com

-RRB-

The

research

of

this

author

was

supported

in

part

by

the

Institute

for

Mathematics

and

Its

Applications

at

the

University

of

Minnesota

Copyright

c

by

Xerox

Corporation

All

rights

reserved

xDepartment

of

Computer

Science

York

University

North

York

Ontario

Canada

-LRB-

joseph@cs.yorku.ca

-RRB-

The

research

of

this

author

was

supported

in

part

by

the

Natural

Sciences

and

Engineering

Research

Council

of

Canada

under

grant

for

column

j

to

n

do

f

A

-LRB-

j

-RRB-

Symbolic

factor

determine

which

columns

of

L

will

update

f

for

each

updating

column

r

j

in

topological

order

do

Col-col

update

f

f

f

-LRB-

r

-RRB-

L

-LRB-

r

-RRB-

end

for

Pivot

interchange

f

-LRB-

j

-RRB-

and

f

-LRB-

k

-RRB-

where

jf

-LRB-

k

-RRB-

j

max

jf

-LRB-

j

n

-RRB-

j

Separate

L

and

U

U

-LRB-

j

j

-RRB-

f

-LRB-

j

-RRB-

L

-LRB-

j

n

j

-RRB-

f

-LRB-

j

n

-RRB-

Divide

L

-LRB-

j

-RRB-

L

-LRB-

j

-RRB-

L

-LRB-

j

j

-RRB-

Prune

symbolic

structure

based

on

column

j

end

for

Figure

LU

factorization

with

column-column

updates

Introduction

The

problem

of

solving

sparse

symmetric

positive

definite

systems

of

linear

equations

on

sequential

and

vector

processors

is

fairly

well

understood

Normally

the

solution

process

is

broken

into

two

phases

ffl

First

symbolic

factorization

to

determine

the

nonzero

structure

of

the

Cholesky

factor

ffl

Second

numeric

factorization

and

solution

Elimination

trees

-LSB-

-RSB-

and

compressed

subscripts

-LSB-

-RSB-

reduce

the

time

and

space

for

symbolic

factorization

to

a

low-order

term

Supernodal

-LSB-

-RSB-

and

multifrontal

-LSB-

-RSB-

elimination

allow

the

use

of

dense

vector

operations

for

nearly

all

of

the

floating-point

computation

thus

reducing

the

symbolic

overhead

in

numeric

factorization

to

a

low-order

term

Overall

the

megaflop

rates

of

modern

sparse

Cholesky

codes

are

nearly

comparable

to

those

of

dense

solvers

-LSB-

-RSB-

For

unsymmetric

systems

where

pivoting

is

required

to

maintain

numerical

stability

progress

has

been

less

satisfactory

Recent

research

has

concentrated

on

two

basic

approaches

submatrixbased

methods

and

column-based

-LRB-

or

row-based

-RRB-

methods

Submatrix

methods

typically

use

some

form

of

Markowitz

pivoting

in

which

each

stage

's

pivot

element

is

chosen

from

the

uneliminated

submatrix

by

criteria

that

attempt

to

balance

numerical

quality

and

preservation

of

sparsity

Recent

submatrix

codes

include

from

the

Harwell

subroutine

library

-LSB-

-RSB-

and

Davis

and

Duff

's

unsymmetric

multifrontal

code

Umfpack

-LSB-

-RSB-

Column

methods

by

contrast

typically

use

ordinary

partial

pivoting

The

pivot

is

chosen

from

the

current

column

according

to

numerical

considerations

alone

the

columns

may

be

preordered

before

factorization

to

preserve

sparsity

Figure

sketches

a

generic

left-looking

column

LU

factorization

Notice

that

the

bulk

of

the

numeric

computation

occurs

in

column-column

updates

or

to

use

Blas

terminology

-LSB-

-RSB-

in

sparse

Axpys

Column

methods

have

the

advantage

that

the

preordering

for

sparsity

is

completely

separate

from

the

factorization

just

as

in

the

symmetric

case

However

symbolic

factorization

can

not

be

methods

are

exactly

analogous

to

column

methods

and

codes

of

both

sorts

exist

We

will

use

column

terminology

those

who

prefer

rows

may

interchange

the

terms

throughout

the

paper

separated

from

numeric

factorization

because

the

nonzero

structures

of

the

factors

depend

on

the

numerical

pivoting

choices

Thus

column

codes

must

do

some

symbolic

factorization

at

each

stage

typically

this

amounts

to

predicting

the

structure

of

each

column

of

the

factors

immediately

before

computing

it

-LRB-

George

and

Ng

-LSB-

-RSB-

described

ways

to

obtain

upper

bounds

on

the

structure

of

the

factors

based

only

on

the

nonzero

structure

of

the

original

matrix

-RRB-

An

early

example

of

such

a

code

is

Sherman

's

Nspiv

-LSB-

-RSB-

-LRB-

which

is

actually

a

row

code

-RRB-

Gilbert

and

Peierls

-LSB-

-RSB-

showed

how

to

use

depth-first

search

and

topological

ordering

to

get

the

structure

of

each

factor

column

This

gives

a

column

code

that

runs

in

total

time

proportional

to

the

number

of

nonzero

floating-point

operations

unlike

the

other

partial

pivoting

codes

Eisenstat

and

Liu

-LSB-

-RSB-

designed

a

pruning

technique

to

reduce

the

amount

of

structural

information

required

for

the

symbolic

factorization

as

we

describe

further

in

Section

The

result

was

that

the

time

and

space

for

symbolic

factorization

were

in

practice

reduced

to

a

low

order

term

In

view

of

the

success

of

supernodal

techniques

for

symmetric

matrices

it

is

natural

to

consider

the

use

of

supernodes

to

enhance

the

performance

of

unsymmetric

solvers

One

difficulty

is

that

unlike

the

symmetric

case

supernodal

structure

can

not

be

determined

in

advance

but

rather

emerges

depending

on

pivoting

choices

during

the

factorization

In

this

paper

we

generalize

supernodes

to

unsymmetric

matrices

and

we

give

efficient

algorithms

for

locating

and

using

unsymmetric

supernodes

during

a

column-based

LU

factorization

We

describe

a

new

code

called

SuperLU

that

uses

depth-first

search

and

symmetric

pruning

to

speed

up

symbolic

factorization

and

uses

unsymmetric

supernodes

to

speed

up

numeric

factorization

The

rest

of

the

paper

is

organized

as

follows

Section

introduces

the

tools

we

use

unsymmetric

supernodes

panels

and

the

column

elimination

tree

Section

describes

the

supernodal

numeric

factorization

Section

describes

the

supernodal

symbolic

factorization

In

Section

we

present

experimental

results

we

benchmark

our

code

on

several

test

matrices

we

compare

its

performance

to

other

column

and

submatrix

codes

and

we

investigate

its

cache

behavior

in

some

detail

Finally

Section

presents

conclusions

and

open

questions

Unsymmetric

supernodes

The

idea

of

a

supernode

is

to

group

together

columns

with

the

same

nonzero

structure

so

they

can

be

treated

as

a

dense

matrix

for

storage

and

computation

Supernodes

were

originally

used

for

-LRB-

symmetric

-RRB-

sparse

Cholesky

factorization

the

first

published

results

are

by

Ashcraft

Grimes

Lewis

Peyton

and

Simon

-LSB-

-RSB-

In

the

factorization

A

LLT

a

supernode

is

a

range

-LRB-

r

s

-RRB-

of

columns

of

L

with

the

same

nonzero

structure

below

the

diagonal

that

is

L

-LRB-

r

s

r

s

-RRB-

is

full

lower

triangular

and

every

row

of

L

-LRB-

s

n

r

s

-RRB-

is

either

full

or

zero

-LRB-

Columns

of

Cholesky

supernodes

need

not

be

contiguous

but

we

will

consider

only

contiguous

supernodes

-RRB-

Ng

and

Peyton

-LSB-

-RSB-

analyzed

the

effect

of

supernodes

in

Cholesky

factorization

on

modern

uniprocessor

machines

with

memory

hierarchies

and

vector

or

superscalar

hardware

All

the

updates

from

columns

of

a

supernode

are

summed

into

a

dense

vector

before

the

sparse

update

is

performed

This

reduces

indirect

addressing

and

allows

the

inner

loops

to

be

unrolled

In

effect

a

sequence

of

column-column

updates

is

replaced

by

a

supernode-column

update

The

sup-col

update

can

be

implemented

using

a

call

to

a

standard

dense

matrix-vector

multiplication

kernel

This

idea

can

be

further

extended

to

supernode-supernode

updates

which

can

be

implemented

using

a

dense

matrix-matrix

kernel

This

can

reduce

memory

traffic

by

an

order

of

magnitude

use

Matlab

notation

for

integer

ranges

and

submatrices

r

s

or

-LRB-

r

s

-RRB-

is

the

vector

of

integers

-LRB-

r

r

s

-RRB-

If

I

and

J

are

vectors

of

integers

then

A

-LRB-

I

J

-RRB-

is

the

submatrix

of

A

with

rows

whose

indices

are

from

I

and

with

columns

whose

indices

are

from

J

A

-LRB-

J

-RRB-

abbreviates

A

-LRB-

n

J

-RRB-

nnz

-LRB-

A

-RRB-

denotes

the

number

of

nonzeros

in

A.

have

same

structureRows

have

same

structureDense

Figure

Four

possible

types

of

unsymmetric

supernodes

because

a

supernode

in

the

cache

can

participate

in

multiple

column

updates

Ng

and

Peyton

reported

that

a

sparse

Cholesky

algorithm

based

on

sup-sup

updates

typically

runs

to

times

as

fast

as

a

col-col

algorithm

Indeed

supernodes

have

become

a

standard

tool

in

sparse

Cholesky

factorization

-LSB-

-RSB-

To

sum

up

supernodes

as

the

source

of

updates

help

because

The

inner

loop

-LRB-

over

rows

-RRB-

has

no

indirect

addressing

-LRB-

Sparse

is

replaced

by

dense

-RRB-

The

outer

loop

-LRB-

over

columns

in

the

supernode

-RRB-

can

be

unrolled

to

save

memory

references

-LRB-

is

replaced

by

-RRB-

Supernodes

as

the

destination

of

updates

help

because

Elements

of

the

source

supernode

can

be

reused

in

multiple

columns

of

the

destination

supernode

to

reduce

cache

misses

-LRB-

is

replaced

by

-RRB-

Supernodes

in

sparse

Cholesky

can

be

determined

during

symbolic

factorization

before

the

numeric

factorization

begins

However

in

sparse

LU

the

nonzero

structure

can

not

be

predicted

before

numeric

factorization

so

we

must

identify

supernodes

on

the

fly

Furthermore

since

the

factors

L

and

U

are

no

longer

transposes

of

each

other

we

must

generalize

the

definition

of

a

supernode

Definition

of

unsymmetric

supernodes

We

considered

several

possible

ways

to

generalize

the

symmetric

definition

of

supernodes

to

unsymmetric

factorization

We

define

F

L

U

I

to

be

the

filled

matrix

containing

both

L

and

U

Same

row

and

column

structures

A

supernode

is

a

range

-LRB-

r

s

-RRB-

of

columns

of

L

and

rows

of

U

such

that

the

diagonal

block

F

-LRB-

r

s

r

s

-RRB-

is

full

and

outside

that

block

all

the

columns

of

L

in

the

range

have

the

same

structure

and

all

the

rows

of

U

in

the

range

have

the

same

structure

supernodes

make

it

possible

to

do

sup-sup

updates

realizing

all

three

benefits

median

mean

Table

Fraction

of

nonzeros

not

in

first

column

of

supernode

Same

column

structure

in

L

A

supernode

is

a

range

-LRB-

r

s

-RRB-

of

columns

of

L

with

full

triangular

diagonal

block

and

the

same

structure

below

the

diagonal

block

supernodes

allow

sup-col

updates

realizing

the

first

two

benefits

Same

column

structure

in

L

full

diagonal

block

in

U

A

supernode

is

a

range

-LRB-

r

s

-RRB-

of

columns

of

L

and

U

such

that

the

diagonal

block

F

-LRB-

r

s

r

s

-RRB-

is

full

and

below

the

diagonal

block

the

columns

of

L

have

the

same

structure

supernodes

allow

sup-col

updates

like

In

addition

if

the

storage

for

a

supernode

is

laid

out

as

for

a

two-dimensional

array

-LRB-

for

or

calls

-RRB-

supernodes

do

not

waste

any

space

in

the

diagonal

block

of

U

Same

column

structure

in

L

and

U

A

supernode

is

a

range

-LRB-

r

s

-RRB-

of

columns

of

L

and

U

with

identical

structure

-LRB-

Since

the

diagonal

is

nonzero

the

diagonal

block

must

be

full

-RRB-

supernodes

allow

sup-col

updates

and

also

simplify

storage

of

L

and

U

Supernodes

of

ATA

A

supernode

is

a

range

-LRB-

r

s

-RRB-

of

columns

of

L

corresponding

to

a

Cholesky

supernode

of

the

symmetric

matrix

ATA

supernodes

are

motivated

by

George

and

Ng

's

observation

-LSB-

-RSB-

that

-LRB-

with

suitable

representations

-RRB-

the

structures

of

L

and

U

in

the

unsymmetric

factorization

PA

LU

are

contained

in

the

structure

of

the

Cholesky

factor

of

ATA

In

unsymmetric

LU

these

supernodes

themselves

are

sparse

so

we

would

waste

time

and

space

operating

on

them

Thus

we

do

not

consider

them

further

Figure

is

a

schematic

of

definitions

through

Supernodes

are

only

useful

if

they

actually

occur

in

practice

The

occurrence

of

symmetric

supernodes

is

related

to

the

clique

structure

of

the

chordal

graph

of

the

Cholesky

factor

which

arises

because

of

fill

during

the

factorization

Unsymmetric

supernodes

seem

harder

to

characterize

but

they

also

are

related

to

dense

submatrices

arising

from

fill

We

measured

the

supernodes

according

to

each

definition

for

unsymmetric

matrices

from

the

Harwell-Boeing

sparse

matrix

library

-LSB-

-RSB-

under

various

column

orderings

Table

shows

for

each

definition

the

fraction

of

nonzeros

of

L

that

are

not

in

the

first

column

of

a

supernode

this

measures

how

much

row

index

storage

is

saved

by

using

supernodes

Corresponding

values

for

symmetric

supernodes

for

the

symmetric

Harwell-Boeing

structural

analysis

problems

usually

range

from

about

to

Larger

numbers

are

better

indicating

larger

supernodes

We

reject

supernodes

as

being

too

rare

to

make

up

for

the

simplicity

of

their

storage

scheme

supernodes

allow

updates

but

as

we

will

see

in

Section

we

can

get

most

of

their

cache

advantage

with

the

more

common

or

supernodes

by

using

sup-panel

updates

Thus

we

conclude

that

either

or

is

the

definition

of

choice

Our

code

uses

which

gives

slightly

larger

supernodes

than

at

a

small

extra

cost

in

storage

Figure

shows

a

sample

matrix

and

the

nonzero

structure

of

its

factors

with

no

pivoting

Using

definition

this

matrix

has

four

supernodes

and

For

example

in

columns

and

the

diagonal

blocks

of

L

and

U

are

full

and

the

columns

of

L

all

have

nonzeros

in

rows

and

By

definition

the

matrix

has

five

supernodes

and

Column

fails

to

join

as

a

supernode

because

is

zero

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

Original

matrix

A

Factors

F

L

U

I

Figure

A

sample

matrix

and

its

LU

factors

Diagonal

elements

and

are

zero

Storage

of

supernodes

A

standard

way

to

lay

out

storage

for

a

sparse

matrix

is

a

one-dimensional

array

of

nonzero

values

in

column

major

order

plus

integer

arrays

giving

row

numbers

and

column

starting

positions

We

use

this

layout

for

both

L

and

U

but

with

a

slight

modification

we

store

the

entire

square

diagonal

block

of

each

supernode

as

part

of

L

including

both

the

strict

lower

triangle

of

values

from

L

and

the

upper

triangle

of

values

from

U

We

store

this

square

block

as

if

it

were

completely

full

-LRB-

it

is

full

in

supernodes

but

its

upper

triangle

may

contain

zeros

in

supernodes

-RRB-

This

allows

us

to

address

each

supernode

as

a

two-dimensional

array

in

calls

to

Blas

routines

In

other

words

if

columns

-LRB-

r

s

-RRB-

form

a

supernode

then

all

the

nonzeros

in

F

-LRB-

r

n

r

s

-RRB-

are

stored

as

a

single

dense

two-dimensional

array

This

also

lets

us

save

some

storage

for

row

indices

only

the

indices

of

nonzero

rows

outside

the

diagonal

block

need

be

stored

and

the

structures

of

all

columns

within

a

supernode

can

be

described

by

one

set

of

row

indices

This

is

similar

to

the

effect

of

compressed

subscripts

in

the

symmetric

case

-LSB-

-RSB-

We

represent

the

part

of

U

outside

the

supernodal

blocks

with

a

standard

sparse

structure

the

values

are

stored

by

columns

with

a

companion

integer

array

the

same

size

to

store

row

indices

another

array

of

n

integers

indicates

the

start

of

each

column

Figure

shows

the

structure

of

the

factors

in

the

example

from

Figure

with

sk

denoting

a

nonzero

in

the

k-th

supernode

and

uk

denoting

a

nonzero

in

the

k-th

column

of

U

outside

the

supernodal

block

Figure

shows

the

storage

layout

-LRB-

We

omit

the

indexing

vectors

that

point

to

the

beginning

of

each

supernode

and

the

beginning

of

each

column

of

U

-RRB-

The

column

elimination

tree

Since

our

definition

requires

the

columns

of

a

supernode

to

be

contiguous

we

should

get

larger

supernodes

if

we

bring

together

columns

of

L

with

the

same

nonzero

structure

But

the

column

ordering

is

fixed

for

sparsity

before

numeric

factorization

what

can

we

do

In

symmetric

Cholesky

factorization

the

so-called

fundamental

supernodes

can

be

made

contiguous

by

permuting

the

matrix

-LRB-

symmetrically

-RRB-

according

to

a

postorder

on

its

elimination

tree

-LSB-

-RSB-

This

postorder

is

an

example

of

what

Liu

calls

an

equivalent

reordering

-LSB-

-RSB-

which

does

not

change

the

sparsity

of

the

factor

The

postordered

elimination

tree

can

also

be

used

to

locate

the

super

BBBBBBBBBBBBBBBB

CCCCCCCCCCCCCCCCA

Figure

Supernodal

structure

-LRB-

by

definition

-RRB-

of

the

factors

of

the

sample

matrix

BBB

CCCA

BBB

CCCA

BBBBB

CCCCCA

BBB

CCCA

Supernodal

blocks

-LRB-

stored

in

column-major

order

-RRB-

Off-supernodal

nonzeros

in

columns

of

U

Row

Subscripts

Figure

Storage

layout

for

factors

of

the

sample

matrix

using

supernodes

nodes

before

the

numeric

factorization

We

proceed

similarly

for

the

unsymmetric

case

Here

the

appropriate

analog

of

the

symmetric

elimination

tree

is

the

column

elimination

tree

or

column

etree

for

short

The

vertices

of

this

tree

are

the

integers

through

n

representing

the

columns

of

A

The

column

etree

of

A

is

the

-LRB-

symmetric

-RRB-

elimination

tree

of

the

column

intersection

graph

of

A

or

equivalently

the

elimination

tree

of

ATA

provided

there

is

no

cancellation

in

computing

ATA

See

Gilbert

and

Ng

-LSB-

-RSB-

for

complete

definitions

The

column

etree

can

be

computed

from

A

in

time

almost

linear

in

the

number

of

nonzeros

of

A

by

a

variation

of

an

algorithm

of

Liu

-LSB-

-RSB-

The

following

theorem

says

that

the

column

etree

represents

potential

dependencies

among

columns

in

LU

factorization

and

that

-LRB-

for

strong

Hall

matrices

-RRB-

no

stronger

information

is

obtainable

from

the

nonzero

structure

of

A.

Note

that

column

i

updates

column

j

in

LU

factorization

if

and

only

if

uij

Theorem

-LRB-

Column

Elimination

Tree

-RRB-

-LSB-

-RSB-

Let

A

be

a

square

nonsingular

possibly

unsymmetric

matrix

and

let

PA

LU

be

any

factorization

of

A

with

pivoting

by

row

interchanges

Let

T

be

the

column

elimination

tree

of

A.

If

vertex

i

is

an

ancestor

of

vertex

j

in

T

then

i

j.

If

lij

then

vertex

i

is

an

ancestor

of

vertex

j

in

T

If

uij

then

vertex

j

is

an

ancestor

of

vertex

i

in

T

Suppose

in

addition

that

A

is

strong

Hall

-LRB-

that

is

A

can

not

be

permuted

to

a

nontrivial

block

triangular

form

-RRB-

If

vertex

j

is

the

parent

of

vertex

i

in

T

then

there

is

some

choice

of

values

for

the

nonzeros

of

A

that

makes

uij

when

the

factorization

PA

LU

is

computed

with

partial

pivoting

Just

as

a

postorder

on

the

symmetric

elimination

tree

brings

together

symmetric

supernodes

we

expect

a

postorder

on

the

column

etree

to

bring

together

unsymmetric

supernodes

Thus

before

we

factor

the

matrix

we

compute

its

column

etree

and

permute

the

matrix

columns

according

to

a

postorder

on

the

tree

We

now

show

that

this

does

not

change

the

factorization

in

any

essential

way

Theorem

Let

A

be

a

matrix

with

column

etree

T

Let

ss

be

a

permutation

such

that

whenever

ss

-LRB-

i

-RRB-

is

an

ancestor

of

ss

-LRB-

j

-RRB-

in

T

we

have

i

j.

Let

P

be

the

permutation

matrix

such

that

ss

P

-LRB-

n

-RRB-

T

Let

A

PAPT

A

A

-LRB-

ss

ss

-RRB-

The

column

etree

T

of

A

is

isomorphic

to

T

in

particular

relabeling

each

node

i

of

T

as

ss

-LRB-

i

-RRB-

yields

T

Suppose

in

addition

that

A

has

an

LU

factorization

without

pivoting

A

L

U

Then

PT

LP

and

PT

U

P

are

respectively

unit

lower

triangular

and

upper

triangular

so

A

-LRB-

PT

LP

-RRB-

-LRB-

PT

U

P

-RRB-

is

also

an

LU

factorization

Remark

Liu

-LSB-

-RSB-

attributes

to

F.

Peters

a

result

similar

to

part

-LRB-

-RRB-

for

the

symmetric

positive

definite

case

concerning

the

Cholesky

factor

and

the

-LRB-

usual

symmetric

-RRB-

elimination

tree

Proof

Part

-LRB-

-RRB-

is

immediate

from

the

definition

of

P

Part

-LRB-

-RRB-

follows

from

Corollary

in

Liu

-LSB-

-RSB-

with

the

symmetric

structure

of

the

column

intersection

graph

of

our

matrix

A

taking

the

place

of

Liu

's

symmetric

matrix

A.

-LRB-

Liu

exhibits

the

isomorphism

explicitly

in

the

proof

of

his

Theorem

-RRB-

Now

we

prove

part

-LRB-

-RRB-

We

have

ass

-LRB-

i

-RRB-

ss

-LRB-

j

-RRB-

aij

for

all

i

and

j.

Write

L

PT

LP

and

U

PT

U

P

so

that

lss

-LRB-

i

-RRB-

ss

-LRB-

j

-RRB-

lij

and

uss

-LRB-

i

-RRB-

ss

-LRB-

j

-RRB-

uij

Then

A

LU

we

need

only

show

that

L

and

U

are

triangular

Consider

a

nonzero

uss

-LRB-

i

-RRB-

ss

-LRB-

j

-RRB-

of

U

In

the

triangular

factorization

A

L

U

element

uij

is

equal

to

uss

-LRB-

i

-RRB-

ss

-LRB-

j

-RRB-

and

is

therefore

nonzero

By

part

-LRB-

-RRB-

of

Theorem

then

j

is

an

ancestor

of

i

in

T

By

the

isomorphism

between

T

and

T

this

implies

that

ss

-LRB-

j

-RRB-

is

an

ancestor

of

ss

-LRB-

i

-RRB-

in

T

Then

it

follows

from

part

-LRB-

-RRB-

of

Theorem

that

ss

-LRB-

j

-RRB-

ss

-LRB-

i

-RRB-

Thus

every

nonzero

of

U

is

on

or

above

the

diagonal

so

U

is

upper

triangular

A

similar

argument

shows

that

every

nonzero

of

L

is

on

or

below

the

diagonal

so

L

is

lower

triangular

The

diagonal

elements

of

L

are

a

permutation

of

those

of

L

so

they

are

all

equal

to

Since

the

triangular

factors

of

A

are

just

permutations

of

the

triangular

factors

of

PAPT

they

have

the

same

sparsity

Indeed

they

require

the

same

arithmetic

to

compute

the

only

possible

difference

is

the

order

of

updates

If

addition

for

updates

is

commutative

and

associative

this

implies

that

with

partial

pivoting

-LRB-

i

j

-RRB-

is

a

legal

pivot

in

A

iff

-LRB-

ss

-LRB-

i

-RRB-

ss

-LRB-

j

-RRB-

-RRB-

is

a

legal

pivot

in

A

In

floating-point

arithmetic

the

different

order

of

updates

could

conceivably

change

the

pivot

sequence

Thus

we

have

the

following

corollary

Corollary

Let

ss

be

a

postorder

on

the

column

elimination

tree

of

A

let

be

any

permutation

matrix

and

let

be

the

permutation

matrix

with

ss

-LRB-

n

-RRB-

T

If

LU

is

an

LU

factorization

then

so

is

-LRB-

-RRB-

A

-LRB-

-RRB-

-LRB-

-RRB-

In

exact

arithmetic

the

former

is

an

LU

factorization

with

partial

pivoting

of

if

and

only

if

the

latter

is

an

LU

factorization

with

partial

pivoting

of

A

This

corollary

says

that

an

LU

code

can

permute

the

columns

of

its

input

matrix

by

postorder

on

the

column

etree

and

then

fold

the

column

permutation

into

the

row

permutation

on

output

Thus

our

SuperLU

code

has

the

option

of

returning

either

four

matrices

L

and

U

-LRB-

with

LU

-RRB-

or

just

the

three

matrices

and

which

are

a

row

permutation

and

two

triangular

matrices

The

advantage

of

returning

all

four

matrices

is

that

the

columns

of

each

supernode

are

contiguous

in

L

which

permits

the

use

of

a

supernodal

triangular

solve

for

the

forward-substitution

phase

of

a

linear

system

solver

The

supernodes

are

not

contiguous

in

Relaxed

supernodes

We

have

explored

various

ways

of

relaxing

the

denseness

of

a

supernode

We

experimented

with

both

and

supernodes

and

found

that

supernodes

-LRB-

those

with

only

nested

column

structures

in

L

-RRB-

give

slightly

better

performance

We

observe

that

for

most

matrices

the

average

size

of

a

supernode

is

only

about

to

columns

-LRB-

though

a

few

supernodes

are

much

larger

-RRB-

A

large

percentage

of

supernodes

consist

of

only

a

single

column

many

of

which

are

leaves

of

the

column

etree

Therefore

we

have

devised

a

scheme

to

merge

groups

of

columns

at

the

fringe

of

the

etree

into

artificial

supernodes

regardless

of

their

row

structures

A

parameter

r

controls

the

granularity

of

the

merge

Our

merge

rule

is

node

i

is

merged

with

its

parent

node

j

when

the

subtree

rooted

at

j

has

at

most

r

nodes

In

practice

for

column

j

to

n

do

f

A

-LRB-

j

-RRB-

Symbolic

factor

determine

which

supernodes

of

L

will

update

f

Determine

whether

j

belongs

to

the

same

supernode

as

j

for

each

updating

supernode

-LRB-

r

s

-RRB-

j

in

topological

order

do

Apply

supernode-column

update

to

column

j

f

-LRB-

r

s

-RRB-

L

-LRB-

r

s

r

s

-RRB-

f

-LRB-

r

s

-RRB-

f

-LRB-

s

n

-RRB-

f

-LRB-

s

n

-RRB-

L

-LRB-

s

n

r

s

-RRB-

f

-LRB-

r

s

-RRB-

end

for

Pivot

interchange

f

-LRB-

j

-RRB-

and

f

-LRB-

k

-RRB-

where

jf

-LRB-

k

-RRB-

j

max

jf

-LRB-

j

n

-RRB-

j

Separate

L

and

U

U

-LRB-

j

j

-RRB-

f

-LRB-

j

-RRB-

L

-LRB-

j

n

j

-RRB-

f

-LRB-

j

n

-RRB-

Divide

L

-LRB-

j

-RRB-

L

-LRB-

j

-RRB-

L

-LRB-

j

j

-RRB-

Prune

symbolic

structure

based

on

column

j

end

for

Figure

LU

factorization

with

supernode-column

updates

the

best

values

of

r

are

generally

between

and

and

yield

improvements

in

running

time

of

to

Artificial

supernodes

are

a

special

case

of

relaxed

supernodes

which

Ashcraft

and

Grimes

have

used

in

the

context

of

multifrontal

methods

for

symmetric

systems

-LSB-

-RSB-

Ashcraft

and

Grimes

allow

a

small

number

of

nonzeros

in

the

structure

of

any

supernode

thus

relaxing

the

condition

that

the

columns

must

have

strictly

nested

structures

It

would

be

possible

to

use

this

idea

in

the

unsymmetric

code

as

well

though

we

have

not

experimented

with

it

Relaxed

supernodes

could

be

constructed

either

on

the

fly

-LRB-

by

relaxing

the

nonzero

count

condition

described

in

Section

below

-RRB-

or

by

preprocessing

the

column

etree

to

identify

small

subtrees

that

we

would

merge

into

supernodes

Supernodal

numeric

factorization

Now

we

show

how

to

modify

the

column-column

algorithm

to

use

supernode-column

updates

and

supernode-panel

updates

This

section

describes

the

numerical

computation

involved

in

the

updates

Section

describes

the

symbolic

factorization

that

determines

which

supernodes

update

which

columns

and

also

the

detection

of

boundaries

between

supernodes

Supernode-column

updates

Figure

sketches

the

supernode-column

algorithm

The

only

difference

from

the

column-column

algorithm

is

that

all

the

updates

to

a

column

from

a

single

supernode

are

done

together

Consider

a

supernode

-LRB-

r

s

-RRB-

that

updates

column

j

The

coefficients

of

the

updates

are

the

values

from

a

segment

of

column

j

of

U

namely

U

-LRB-

r

s

j

-RRB-

The

nonzero

structure

of

such

a

segment

is

particularly

simple

all

the

nonzeros

are

contiguous

and

follow

all

the

zeros

-LRB-

as

proved

in

Corollary

below

-RRB-

Thus

if

k

is

the

index

of

the

first

nonzero

row

in

U

-LRB-

r

s

j

-RRB-

the

updates

to

column

j

from

supernode

-LRB-

r

s

-RRB-

come

from

columns

k

through

s

Since

the

supernode

is

stored

as

a

dense

matrix

these

updates

can

be

performed

by

a

dense

lower

triangular

solve

-LRB-

with

the

matrix

L

-LRB-

k

s

k

s

-RRB-

-RRB-

and

a

dense

matrix-vector

multiplication

-LRB-

with

the

matrix

L

-LRB-

s

n

k

s

-RRB-

-RRB-

As

described

in

Section

the

symbolic

phase

determines

the

value

of

k

that

is

the

position

of

the

first

nonzero

in

the

segment

U

-LRB-

r

s

j

-RRB-

The

advantages

of

using

supernode-column

updates

are

similar

to

those

in

the

symmetric

case

-LSB-

-RSB-

Efficient

matrix-vector

kernels

can

be

used

for

the

triangular

solve

and

matrixvector

multiply

Furthermore

all

the

updates

from

the

supernodal

columns

can

be

collected

in

a

dense

vector

before

doing

a

single

scatter

into

the

target

vector

This

reduces

the

amount

of

indirect

addressing

Supernode-panel

updates

We

can

improve

the

supernode-column

algorithm

further

on

machines

with

a

memory

hierarchy

by

changing

the

data

access

pattern

The

data

we

are

accessing

in

the

inner

loop

-LRB-

lines

-LCB-

of

Figure

-RRB-

include

the

destination

column

j

and

all

the

updating

supernodes

-LRB-

r

s

-RRB-

to

the

left

of

column

j.

Column

j

is

accessed

many

times

while

each

supernode

-LRB-

r

s

-RRB-

is

used

only

once

In

practice

the

number

of

nonzero

elements

in

column

j

is

much

less

than

that

in

the

updating

supernodes

Therefore

the

access

pattern

given

by

this

loop

provides

little

opportunity

to

reuse

cached

data

In

particular

the

same

supernode

-LRB-

r

s

-RRB-

may

be

needed

to

update

both

columns

j

and

j

But

when

we

factor

the

-LRB-

j

-RRB-

st

column

-LRB-

in

the

next

iteration

of

the

outer

loop

-RRB-

we

will

have

to

fetch

supernode

-LRB-

r

s

-RRB-

again

from

memory

instead

of

from

cache

-LRB-

unless

the

supernodes

are

small

compared

to

the

cache

-RRB-

To

exploit

memory

locality

we

factor

several

columns

-LRB-

say

w

of

them

-RRB-

at

a

time

in

the

outer

loop

so

that

one

updating

supernode

-LRB-

r

s

-RRB-

can

be

used

to

update

as

many

of

the

w

columns

as

possible

We

refer

to

these

w

consecutive

columns

as

a

panel

to

differentiate

them

from

a

supernode

the

row

structures

of

these

columns

may

not

be

correlated

in

any

fashion

and

the

boundaries

between

panels

may

be

different

from

those

between

supernodes

The

new

method

requires

rewriting

the

doubly

nested

loop

as

the

triple

loop

shown

in

Figure

This

is

analogous

to

loop

tiling

techniques

used

in

optimizing

compilers

to

improve

cache

behavior

for

two-dimensional

arrays

with

regular

stride

It

is

also

somewhat

analogous

to

the

supernode-supernode

updates

that

Ng

and

Peyton

-LSB-

-RSB-

and

Rothberg

and

Gupta

-LSB-

-RSB-

have

used

in

symmetric

Cholesky

factorization

The

structure

of

each

supernode-column

update

is

the

same

as

in

the

supernode-column

algorithm

For

each

supernode

-LRB-

r

s

-RRB-

to

the

left

of

column

j

if

ukj

for

some

r

k

s

then

uij

for

all

k

i

s.

Therefore

the

nonzero

structure

of

the

panel

of

U

consists

of

dense

column

segments

that

are

row-wise

separated

by

supernodal

boundaries

as

in

Figure

Thus

it

is

sufficient

for

the

symbolic

factorization

algorithm

to

record

only

the

first

nonzero

position

of

each

column

segment

As

detailed

in

Section

symbolic

factorization

is

applied

to

all

the

columns

in

a

panel

at

once

outside

the

numeric-factorization

loop

over

updating

supernodes

In

dense

factorization

the

entire

supernode-panel

update

in

lines

-LCB-

of

Figure

would

be

implemented

as

two

calls

a

dense

triangular

solve

with

w

right-hand

sides

followed

by

a

dense

matrix-matrix

multiply

In

the

sparse

case

this

is

not

possible

because

the

different

supernode-column

updates

begin

at

different

positions

k

within

the

supernode

and

the

submatrix

U

-LRB-

r

s

j

j

w

-RRB-

is

not

dense

Thus

the

sparse

supernode-panel

algorithm

still

calls

the

Blas

However

we

get

the

similar

cache

benefits

as

the

Blas

at

the

cost

of

doing

the

loop

reorganization

ourselves

Thus

we

sometimes

call

the

kernel

of

this

algorithm

a

method

In

the

double

loop

nest

-LRB-

-LCB-

-RRB-

the

ideal

circumstance

is

that

all

w

columns

in

the

panel

require

for

column

j

to

n

step

w

do

Symbolic

factor

determine

which

supernodes

will

update

any

of

L

-LRB-

j

j

w

-RRB-

for

each

updating

supernode

-LRB-

r

s

-RRB-

j

in

topological

order

do

for

column

jj

j

to

j

w

do

Apply

supernode-column

update

to

column

jj

end

for

end

for

Inner

factorization

apply

the

sup-col

algorithm

to

the

panel

end

for

j

n

J

W

j

j

row

j

sr

r

s

U

L

L

J

J

J

J

Figure

The

supernode-panel

algorithm

with

column-wise

blocking

J

j

updates

from

supernode

-LRB-

r

s

-RRB-

Then

this

supernode

will

be

used

w

times

before

it

is

forced

out

of

the

cache

There

is

a

trade-off

between

the

value

of

w

and

the

size

of

cache

For

this

scheme

to

work

efficiently

we

need

to

ensure

that

the

nonzeros

in

the

w

columns

do

not

cause

cache

thrashing

That

is

we

must

keep

w

small

enough

that

all

the

data

accessed

in

this

doubly

nested

loop

fit

in

cache

Otherwise

the

cache

cross-interference

between

the

source

supernode

and

the

destination

panel

can

offset

the

benefit

of

the

new

algorithm

Outer

and

inner

factorization

At

the

end

of

the

supernode-panel

update

-LRB-

line

-RRB-

columns

j

through

j

w

of

L

and

U

have

received

all

their

updates

from

columns

to

the

left

of

j

We

call

this

the

outer

factorization

What

remains

is

to

apply

updates

that

come

from

columns

within

the

panel

This

amounts

to

forming

the

LU

factorization

of

the

panel

itself

-LRB-

in

columns

-LRB-

j

j

w

-RRB-

and

rows

-LRB-

j

n

-RRB-

-RRB-

This

inner

factorization

is

performed

by

the

supernode-column

algorithm

almost

exactly

as

shown

in

Figure

The

inner

factorization

includes

a

columnwise

symbolic

factorization

just

as

in

the

supernode-column

algorithm

The

inner

factorization

also

includes

the

supernode

identification

partial

pivoting

and

symmetric

structure

reduction

for

the

entire

algorithm

Reducing

cache

misses

by

row-wise

blocking

Our

first

experiments

with

the

supernode-panel

algorithm

showed

speedups

for

some

medium-sized

problems

of

around

-LCB-

However

the

improvement

for

large

matrices

was

often

only

a

few

percent

We

now

study

the

reasons

and

remedies

for

this

To

implement

loops

-LRB-

-LCB-

-RRB-

of

Figure

we

first

expand

the

nonzeros

of

the

panel

columns

of

A

into

an

n

by

w

full

working

array

called

the

SPA

-LRB-

for

sparse

accumulator

-LSB-

-RSB-

-RRB-

This

allows

random

access

to

the

entries

of

the

active

panel

A

temporary

array

stores

the

results

of

the

Blas

operations

and

the

updates

are

scattered

into

the

SPA

At

the

end

of

panel

factorization

the

data

in

the

SPA

are

copied

into

storage

for

L

and

U

Although

increasing

the

panel

size

w

gives

more

opportunity

for

data

reuse

it

also

increases

the

size

of

the

active

data

set

that

must

fit

into

cache

The

supernode-panel

update

loop

accesses

the

following

data

ffl

the

nonzeros

in

the

updating

supernode

L

-LRB-

r

n

r

s

-RRB-

ffl

the

SPA

data

structure

consisting

of

an

n

by

w

full

array

and

a

temporary

store

of

size

n.

By

instrumenting

the

code

we

found

that

the

working

sets

of

large

matrices

are

much

larger

than

the

cache

size

Hence

cache

thrashing

limits

performance

We

experimented

with

a

scheme

suggested

by

Rothberg

-LSB-

-RSB-

in

which

the

SPA

has

only

as

many

rows

as

the

number

of

nonzero

rows

in

the

panel

-LRB-

as

predicted

by

symbolic

factorization

-RRB-

and

an

extra

indirection

array

of

size

n

is

used

to

address

the

SPA

Unfortunately

the

cost

incurred

by

double

indirection

is

not

negligible

and

this

scheme

was

not

as

effective

as

the

two-dimensional

blocking

method

we

now

describe

We

implemented

a

row-wise

blocking

scheme

on

top

of

the

column-wise

blocking

in

the

supernodepanel

update

The

blocking

adds

another

level

of

looping

between

the

two

loops

in

lines

and

of

Figure

This

partitions

the

supernodes

-LRB-

and

the

SPA

structure

-RRB-

into

block

rows

Then

each

block

row

of

the

updating

supernode

is

used

for

up

to

w

partial

matrix-vector

multiplies

which

are

pushed

all

the

way

through

into

the

SPA

before

the

next

block

row

of

the

supernode

is

accessed

The

active

data

set

accessed

in

the

inner

loops

is

thus

much

smaller

than

in

the

scheme

The

blocking

algorithm

is

organized

as

in

Figure

The

key

performance

gains

come

from

for

j

to

n

step

w

do

for

each

updating

supernode

-LRB-

r

s

-RRB-

j

in

topological

order

do

Apply

triangular

solves

to

A

-LRB-

r

s

j

j

w

-RRB-

using

L

-LRB-

r

s

r

s

-RRB-

for

each

row

block

B

in

L

-LRB-

s

n

r

s

-RRB-

do

for

jj

j

to

j

w

do

Multiply

B

U

-LRB-

r

s

jj

-RRB-

and

scatter

into

SPA

-LRB-

jj

-RRB-

end

for

end

for

end

for

end

for

Figure

The

supernode-panel

algorithm

with

blocking

the

loops

-LRB-

-LCB-

-RRB-

where

each

row

block

is

reused

as

much

as

possible

before

the

next

row

block

is

brought

into

the

cache

The

innermost

loop

is

still

a

dense-matrix

vector

multiply

performed

by

a

kernel

Combining

and

blocking

The

blocking

works

well

when

the

rectangular

supernodal

matrix

L

-LRB-

r

n

r

s

-RRB-

is

large

in

both

dimensions

If

all

of

L

-LRB-

r

n

r

s

-RRB-

can

fit

into

cache

then

the

row-wise

blocking

gives

no

benefit

but

still

incurs

overhead

for

setting

up

loop

variables

skipping

the

empty

loop

body

and

so

on

This

overhead

can

be

nearly

for

some

of

the

sparser

problems

in

our

test

suite

Thus

we

have

devised

a

hybrid

update

algorithm

that

uses

either

the

or

partitioning

scheme

depending

on

the

size

of

each

updating

supernode

The

decision

is

made

at

run-time

as

shown

in

Figure

The

overhead

is

limited

to

the

test

at

line

of

Figure

It

turns

out

that

this

hybrid

scheme

works

better

than

either

or

codes

for

many

problems

Therefore

this

is

the

algorithm

that

we

use

in

the

performance

analysis

in

Section

In

Section

we

will

discuss

in

more

detail

what

we

mean

by

large

in

the

test

on

line

Symbolic

factorization

Symbolic

factorization

is

the

process

that

determines

the

nonzero

structure

of

the

triangular

factors

L

and

U

from

the

nonzero

structure

of

the

matrix

A

This

in

turn

determines

which

columns

of

L

update

each

column

j

of

the

factors

-LRB-

namely

those

columns

r

for

which

urj

-RRB-

and

also

which

columns

of

L

can

be

combined

into

supernodes

Without

numeric

pivoting

the

complete

symbolic

factorization

can

be

performed

before

any

numeric

factorization

Partial

pivoting

however

requires

that

the

numeric

and

symbolic

factorizations

be

interleaved

The

supernode-column

algorithm

performs

symbolic

factorization

for

each

column

just

before

it

is

computed

as

described

in

Section

The

supernode-panel

algorithm

performs

symbolic

factorization

for

an

entire

panel

at

once

as

described

in

Section

for

j

to

n

step

w

do

for

each

updating

supernode

-LRB-

r

s

-RRB-

j

in

topological

order

do

if

-LRB-

r

s

-RRB-

is

large

then

use

blocking

Apply

triangular

solves

to

A

-LRB-

r

s

j

j

w

-RRB-

using

L

-LRB-

r

s

r

s

-RRB-

for

each

row

block

B

in

L

-LRB-

s

n

r

s

-RRB-

do

for

jj

j

to

j

w

do

Multiply

B

U

-LRB-

r

s

jj

-RRB-

and

scatter

into

SPA

-LRB-

jj

-RRB-

end

for

end

for

else

use

blocking

for

jj

j

to

j

w

do

Apply

triangular

solve

to

A

-LRB-

r

s

jj

-RRB-

Multiply

L

-LRB-

s

n

r

s

-RRB-

U

-LRB-

r

s

jj

-RRB-

and

scatter

into

SPA

-LRB-

jj

-RRB-

end

for

end

if

end

for

end

for

Figure

The

supernode-panel

algorithm

with

both

and

blocking

Column

depth-first

search

From

the

numeric

factorization

algorithm

it

is

clear

that

the

structure

of

column

F

-LRB-

j

-RRB-

depends

on

the

structure

of

column

A

-LRB-

j

-RRB-

of

the

original

matrix

and

on

the

structure

of

L

-LRB-

J

-RRB-

where

J

j

Indeed

F

-LRB-

j

-RRB-

has

the

same

structure

as

the

solution

vector

for

the

following

triangular

system

-LSB-

-RSB-

L

-LRB-

J

-RRB-

I

F

-LRB-

j

-RRB-

A

-LRB-

j

-RRB-

A

straightforward

way

to

compute

the

structure

of

F

-LRB-

j

-RRB-

from

the

structures

of

L

-LRB-

J

-RRB-

and

A

-LRB-

j

-RRB-

is

to

simulate

the

numerical

computation

A

less

expensive

way

is

to

use

the

following

characterization

in

terms

of

paths

in

the

directed

graph

of

L

-LRB-

J

-RRB-

For

any

matrix

M

the

notation

i

M

j

means

that

there

is

an

edge

from

i

to

j

in

the

directed

graph

of

M

that

is

mij

The

notation

i

M

=-RRB-

j

means

that

there

is

a

directed

path

from

i

to

j

in

the

directed

graph

of

M

Such

a

path

may

have

length

zero

that

is

i

M

=-RRB-

i

always

holds

Theorem

-LSB-

-RSB-

fij

is

nonzero

-LRB-

equivalently

i

F

j

-RRB-

if

and

only

if

i

L

-LRB-

J

-RRB-

=-RRB-

k

A

j

for

some

k

i

This

result

implies

that

the

symbolic

factorization

of

column

j

can

be

obtained

as

follows

Consider

the

nonzeros

in

A

-LRB-

j

-RRB-

as

a

subset

of

the

vertices

of

the

directed

graph

G

G

-LRB-

L

-LRB-

J

-RRB-

T

-RRB-

which

is

the

reverse

of

the

directed

graph

of

L

-LRB-

J

-RRB-

The

nonzero

positions

of

F

-LRB-

j

-RRB-

are

then

given

by

the

vertices

reachable

by

paths

from

this

set

in

G

We

use

the

graph

of

LT

here

because

of

the

convention

that

edges

are

directed

from

rows

to

columns

Since

L

is

actually

stored

by

columns

our

data

structure

gives

precisely

the

adjacency

information

for

G.

Therefore

we

can

determine

the

structure

of

column

j

of

L

and

U

by

traversing

G

from

the

set

of

starting

nodes

given

by

the

structure

of

A

-LRB-

j

-RRB-

The

traversal

of

G

determines

the

structure

of

U

-LRB-

j

-RRB-

which

in

turn

determines

the

columns

of

L

that

will

participate

in

updates

to

column

j

in

the

numerical

factorization

These

updates

must

be

applied

in

an

order

consistent

with

a

topological

ordering

of

G

We

use

depth-first

search

to

perform

the

traversal

which

makes

it

possible

to

generate

a

topological

order

-LRB-

specifically

reverse

postorder

-RRB-

on

the

nonzeros

of

U

-LRB-

j

-RRB-

as

they

are

located

-LSB-

-RSB-

Another

consequence

of

the

path

theorem

is

the

following

corollary

It

says

that

if

we

divide

each

column

of

U

into

segments

one

per

supernode

then

within

each

segment

the

column

of

U

just

consists

of

a

consecutive

sequence

of

nonzeros

Thus

we

need

only

keep

track

of

the

position

of

the

first

nonzero

in

each

segment

Corollary

Let

-LRB-

r

s

-RRB-

be

a

supernode

-LRB-

of

either

type

or

-RRB-

in

the

factorization

PA

LU

Suppose

ukj

is

nonzero

for

some

j

with

r

k

s

Then

uij

for

all

i

with

k

i

s.

Proof

Let

k

i

s

Since

ukj

we

have

k

L

-LRB-

J

-RRB-

=-RRB-

A

j

by

Theorem

Now

lik

is

in

the

diagonal

block

of

the

supernode

and

hence

is

nonzero

Thus

i

L

-LRB-

J

-RRB-

k

so

i

L

-LRB-

J

-RRB-

=-RRB-

A

j

whence

uij

is

nonzero

by

Theorem

Pruning

the

symbolic

structure

We

can

speed

up

the

depth-first

search

traversals

by

using

a

reduced

graph

in

place

of

G

the

reverse

of

the

graph

of

L

-LRB-

J

-RRB-

We

have

explored

this

idea

in

a

series

of

papers

-LSB-

-RSB-

Any

graph

H

can

be

substituted

for

G

provided

that

i

H

=-RRB-

j

if

and

only

if

i

G

=-RRB-

j

The

traversals

are

more

efficient

if

H

has

fewer

edges

but

any

gain

in

efficiency

must

be

traded

off

against

the

cost

of

computing

H

An

extreme

choice

of

H

is

the

elimination

dag

-LSB-

-RSB-

which

is

the

transitive

reduction

of

G

or

the

minimal

subgraph

of

G

that

preserves

paths

However

the

elimination

dag

is

expensive

to

compute

The

symmetric

reduction

-LSB-

-RSB-

is

a

subgraph

that

has

-LRB-

in

general

-RRB-

fewer

edges

than

G

but

more

edges

than

the

elimination

dag

and

that

is

much

less

expensive

to

compute

The

symmetric

reduction

takes

advantage

of

symmetry

in

the

structure

of

the

filled

matrix

F

if

F

is

completely

symmetric

it

is

just

the

symmetric

elimination

tree

The

symmetric

reduction

of

L

-LRB-

J

-RRB-

is

obtained

by

removing

all

nonzeros

lrs

for

which

ltsust

for

some

t

min

-LRB-

r

j

-RRB-

Eisenstat

and

Liu

-LSB-

-RSB-

give

an

efficient

method

to

compute

the

symmetric

reduction

during

symbolic

factorization

and

demonstrate

experimentally

that

it

significantly

reduces

the

total

factorization

time

with

an

algorithm

that

does

column-column

updates

Our

supernodal

code

uses

symmetric

reduction

to

speed

up

its

symbolic

factorization

The

example

in

Figure

illustrates

symmetric

reduction

in

the

presence

of

supernodes

We

use

S

to

represent

the

supernodal

structure

of

L

-LRB-

J

-RRB-

T

and

R

to

represent

the

symmetric

reduction

of

S

It

is

this

R

that

we

use

in

our

code

Note

that

the

edges

of

the

graph

of

R

are

directed

from

columns

of

L

to

rows

of

L.

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

Filled

matrix

Lower

triangle

Supernodal

Reduced

F

-LRB-

J

-RRB-

G

S

R

Figure

Supernodal

and

symmetrically

reduced

structures

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

BBBBBBBBBBBBBBBB

ffl

ffl

ffi

ffi

ffi

ffi

CCCCCCCCCCCCCCCCA

=-RRB-

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

Figure

One

step

of

symbolic

factorization

in

the

reduced

structure

In

the

figure

the

small

dot

indicates

an

entry

in

S

that

was

pruned

from

R

by

symmetric

reduction

The

-LRB-

-RRB-

entry

was

pruned

due

to

the

symmetric

nonzero

pair

-LRB-

-RRB-

and

-LRB-

-RRB-

The

figure

shows

the

current

state

of

the

reduced

structure

based

on

the

first

seven

columns

of

the

filled

matrix

It

is

instructive

to

follow

this

example

through

one

more

column

to

see

how

symbolic

factorization

is

carried

out

in

the

reduced

supernodal

structure

Consider

the

symbolic

step

for

column

Suppose

and

are

nonzero

The

other

nonzeros

in

column

of

the

factor

are

generated

by

paths

in

the

reduced

supernodal

structure

-LRB-

we

just

show

one

possible

path

for

each

nonzero

-RRB-

AT

R

AT

R

AT

R

R

AT

R

Figure

shows

the

reduced

supernodal

structure

before

and

after

column

In

column

of

A

the

original

nonzeros

are

shown

as

ffl

and

the

fill

nonzeros

are

shown

as

ffi

Once

the

structure

of

column

of

U

is

known

more

symmetric

reduction

is

possible

The

entry

is

pruned

due

to

the

symmetric

nonzeros

in

and

Also

is

pruned

by

and

Figure

shows

the

new

structure

The

supernodal

symbolic

factorization

relies

on

the

path

characterization

in

Theorem

and

on

the

path-preserving

property

of

symmetric

reduction

In

effect

we

use

the

modified

path

condition

i

AT

R

=-RRB-

j

on

the

symmetrically-reduced

supernodal

structure

R

of

L

-LRB-

J

-RRB-

T

Detecting

supernodes

Since

supernodes

consist

of

contiguous

columns

of

L

we

can

decide

at

the

end

of

each

symbolic

factorization

step

whether

the

new

column

j

belongs

to

the

same

supernode

as

column

j

For

supernodes

the

test

is

straightforward

During

symbolic

factorization

we

test

whether

L

-LRB-

j

-RRB-

L

-LRB-

j

-RRB-

-LRB-

where

the

containment

applies

to

the

set

of

nonzero

indices

-RRB-

At

the

end

of

the

symbolic

factorization

step

we

test

whether

nnz

-LRB-

L

-LRB-

j

-RRB-

-RRB-

nnz

-LRB-

L

-LRB-

j

-RRB-

-RRB-

Column

j

joins

column

j

's

supernode

if

and

only

if

both

tests

are

passed

supernodes

also

require

the

diagonal

block

of

U

to

be

full

To

check

this

it

suffices

to

check

whether

the

single

element

urj

is

nonzero

where

r

is

the

first

column

index

of

the

supernode

This

follows

from

Corollary

urj

implies

that

uij

for

all

r

i

j.

Indeed

we

can

even

omit

the

test

L

-LRB-

j

-RRB-

L

-LRB-

j

-RRB-

for

supernodes

If

urj

then

uj

j

which

means

that

column

j

updates

column

j

which

implies

L

-LRB-

j

-RRB-

L

-LRB-

j

-RRB-

Thus

we

have

proved

the

following

Theorem

Suppose

a

supernode

begins

with

column

r

and

extends

at

least

through

column

j

Column

j

belongs

to

this

supernode

if

and

only

if

urj

and

nnz

-LRB-

L

-LRB-

j

-RRB-

-RRB-

nnz

-LRB-

L

-LRB-

j

-RRB-

-RRB-

For

either

or

supernodes

it

is

straightforward

to

implement

the

relaxed

versions

discussed

in

Section

Also

since

the

main

benefits

of

supernodes

come

when

they

fit

into

the

cache

we

impose

a

maximum

size

for

a

supernode

Panel

depth-first

search

The

supernode-panel

algorithm

consists

of

an

outer

factorization

-LRB-

applying

updates

from

supernodes

to

the

active

panel

-RRB-

and

an

inner

factorization

-LRB-

applying

supernode-column

updates

within

the

active

panel

-RRB-

Each

has

its

own

symbolic

factorization

The

outer

symbolic

factorization

happens

once

per

panel

and

determines

two

things

-LRB-

-RRB-

a

single

column

structure

which

is

the

union

of

the

structures

of

the

panel

columns

and

-LRB-

-RRB-

which

supernodes

update

each

column

of

the

panel

and

in

what

order

This

is

the

information

that

the

supernode-panel

update

loop

in

Figure

needs

The

inner

symbolic

factorization

happens

once

for

each

column

of

the

panel

interleaved

column

by

column

with

the

inner

numeric

factorization

In

addition

to

determining

the

nonzero

structure

of

the

active

column

and

which

supernodes

within

the

panel

will

update

the

active

column

the

inner

symbolic

factorization

is

also

responsible

for

forming

supernodes

-LRB-

that

is

for

deciding

whether

the

active

column

will

start

a

new

supernode

-RRB-

and

for

symmetric

structural

pruning

The

inner

symbolic

factorization

is

therefore

exactly

the

supernode-column

symbolic

factorization

described

above

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

ffl

CCCCCCCCCCCCCCCCA

BBBBBBBBBBBBBBBB

ffl

ffl

ffl

ffi

ffi

ffl

ffl

ffi

ffi

ffi

ffl

ffi

ffl

CCCCCCCCCCCCCCCCA

Reduced

supernodal

R

A

-LRB-

-RRB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Figure

The

supernodal

directed

graph

corresponding

to

L

-LRB-

-RRB-

T

The

outer

symbolic

factorization

must

determine

the

structures

of

columns

j

to

j

w

i.e.

the

structure

of

the

whole

panel

and

also

a

topological

order

for

U

-LRB-

j

j

j

w

-RRB-

en

masse

To

this

end

we

developed

an

efficient

panel

depth-first

search

scheme

which

is

a

slight

modification

of

the

column

DFS

The

panel

depth-first

search

algorithm

maintains

a

single

postorder

DFS

list

for

all

w

columns

of

the

panel

Let

us

call

this

the

PO

list

which

is

initially

empty

The

algorithm

invokes

the

column

depth-search

procedure

for

each

column

from

j

to

j

w

In

the

column

DFS

each

time

the

search

backtracks

from

a

vertex

that

vertex

is

appended

to

the

PO

list

In

the

panel

DFS

however

the

vertex

is

appended

to

the

PO

list

only

if

it

is

not

already

on

the

list

This

gives

a

single

list

that

includes

every

position

that

is

nonzero

in

any

panel

column

just

once

and

the

entire

list

is

in

-LRB-

reverse

-RRB-

topological

order

Thus

the

order

of

updates

specified

by

the

list

is

acceptable

for

each

of

the

w

individual

panel

columns

We

illustrate

the

idea

in

Figure

using

the

sample

matrix

from

Figure

and

a

panel

of

width

two

The

first

seven

columns

have

been

factored

and

the

current

panel

consists

of

columns

and

In

the

panel

nonzeros

of

A

are

shown

as

ffl

and

fill

in

F

is

shown

as

ffi

The

depth-first

search

for

column

starts

from

vertices

and

After

that

search

is

finished

the

panel

postorder

list

is

PO

-LRB-

-RRB-

Now

the

depth-first

search

for

column

starts

from

vertices

and

-LRB-

not

since

is

the

representative

vertex

for

the

supernode

containing

column

-RRB-

This

DFS

only

appends

to

the

PO

list

because

is

already

on

the

list

Thus

the

final

list

for

this

panel

is

PO

-LRB-

-RRB-

The

postorder

list

of

column

is

-LRB-

-RRB-

and

the

postorder

list

of

column

is

-LRB-

-RRB-

which

are

simply

two

sublists

of

the

panel

PO

list

The

topological

order

is

the

reverse

of

PO

or

-LRB-

-RRB-

In

the

loop

of

line

of

Figure

we

follow

this

topological

order

to

schedule

the

updating

supernodes

and

perform

numeric

updates

to

columns

of

the

panel

Evaluation

In

this

section

we

apply

our

algorithms

to

practical

matrices

from

various

sources

We

will

compare

the

performance

of

SuperLU

our

supernode-panel

code

with

its

predecessors

and

with

other

approaches

to

sparse

LU

factorization

Matrix

Name

s

Rows

Nonzeros

Nonzeros/row

Memplus

Mcfe

Goodwin

7320 324772

Inaccura

1000 1000000

Table

Benchmark

matrices

Structural

symmetry

s

is

defined

to

be

the

fraction

of

the

nonzeros

matched

by

nonzeros

in

symmetric

locations

of

the

matrices

are

numerically

symmetric

Experimental

setup

Table

lists

matrices

with

some

characteristics

of

their

nonzero

structures

The

matrices

are

sorted

in

increasing

order

of

f

lops

nnz

-LRB-

F

-RRB-

the

ratio

of

the

number

of

floating

point

operations

to

the

number

of

nonzeros

nnz

-LRB-

F

-RRB-

in

the

factored

matrix

F

U

L

I

The

reason

for

this

order

will

be

described

in

more

detail

in

section

Some

of

the

matrices

are

from

the

Harwell-Boeing

collection

-LSB-

-RSB-

Most

of

the

larger

matrices

are

from

the

ftp

site

maintained

by

Tim

Davis

of

the

University

of

Florida

Those

matrices

are

as

follows

Goodwin

is

a

finite

element

matrix

in

a

nonlinear

solver

for

a

fluid

mechanics

problem

provided

by

Ralph

Goodwin

of

the

University

of

Illinois

at

Urbana-Champaign

Memplus

is

a

circuit

simulation

matrix

from

Steve

Hamm

of

Motorola

Inaccura

and

were

provided

by

Horst

Simon

of

Silicon

Graphics

is

from

a

fluid

structure

interaction

turbulence

problem

is

from

a

buckling

problem

for

a

container

model

comes

from

an

implicit

Euler

solver

for

an

unstructured

grid

in

a

flow

simulation

is

from

solving

a

coupled

nonlinear

PDE

system

in

a

-LRB-

uniform

mesh

-RRB-

semiconductor

device

simulation

as

provided

by

Song

Wang

of

the

University

of

New

South

Wales

Sydney

is

derived

from

a

direct

fully-coupled

method

for

solving

the

Navier-Stokes

equations

for

viscous

flow

calculations

provided

by

Wei

Shyy

of

the

University

of

Florida

is

an

unsymmetric

augmented

matrix

for

a

PDE

with

highly

varying

coefficients

-LSB-

-RSB-

is

a

dense

matrix

In

this

paper

we

do

not

address

the

performance

of

preordering

for

sparsity

Matrices

and

were

symmetrically

permuted

by

Matlab

's

symmetric

minimum

degree

ordering

on

A+AT

For

cis.ufl.edu

in

pub/umfpack/matrices

IBM

Fraction

of

time

Fraction

of

flops

Matrix

nnz

-LRB-

F

-RRB-

nnz

-LRB-

F

-RRB-

nnz

-LRB-

A

-RRB-

#flops

-LRB-

-RRB-

Seconds

Mflops

in

numeric

in

Dgemv

10 623806

11 453112

13 7635773

14 12785320

15 3109585

16 10016266

17 1000000

18 17631651

19 13287108

20 26714111

21 49687658

Table

Performance

of

SuperLU

on

an

IBM

all

other

matrices

the

columns

were

permuted

by

Matlab

's

minimum

degree

ordering

of

ATA

-LSB-

-RSB-

We

performed

the

numerical

experiments

on

an

IBM

The

CPU

clock

rate

is

MHz

The

processor

has

two

branch

units

two

fixed-point

units

and

two

floating-point

units

which

can

all

operate

in

parallel

if

there

are

no

dependencies

In

particular

each

FPU

can

perform

two

operations

-LRB-

a

multiply

and

an

add

or

subtract

-RRB-

at

every

cycle

Thus

the

peak

floating-point

performance

is

Mflops

The

data

cache

is

of

size

KB

with

lines

and

is

set-associative

with

LRU

replacement

policy

There

is

a

separate

KB

instruction

cache

The

size

of

the

main

memory

is

MB

The

SuperLU

algorithm

is

implemented

in

C

we

use

the

AIX

xlc

compiler

with

optimization

Performance

of

the

code

Table

presents

the

performance

of

the

SuperLU

code

on

this

system

All

floating

point

computations

are

in

double

precision

In

the

inner

loops

of

our

sparse

code

we

call

the

two

dense

routines

Dtrsv

-LRB-

triangular

solve

-RRB-

and

Dgemv

-LRB-

matrix-vector

multiply

-RRB-

provided

in

the

IBM

ESSL

library

-LSB-

-RSB-

whose

matrix-matrix

multiply

routine

-LRB-

Dgemm

-RRB-

achieves

about

Mflops

when

dimension

of

the

matrix

is

larger

than

-LSB-

-RSB-

In

our

sparse

algorithm

we

find

that

Dgemv

typically

accounts

for

more

than

of

the

floating-point

operations

As

shown

in

the

last

column

of

Table

this

percentage

is

higher

than

for

many

matrices

Our

measurements

reveal

that

for

typical

dimensions

arising

from

the

benchmark

matrices

Dgemv

achieves

roughly

Mflops

if

the

data

IBM

GP

GP-Mod

SupCol-F

SupCol-C

SuperLU

Matrix

xlf

xlf

xlf

xlc

xlc

Geometric

Mean

Table

Speedups

achieved

by

each

enhancement

over

the

GP

column-column

code

on

the

are

from

cache

If

the

data

are

fetched

from

main

memory

this

rate

can

drop

by

a

factor

of

or

The

Blas

speed

is

clearly

an

upper

bound

on

the

overall

factorization

rate

However

because

symbolic

manipulation

usually

takes

a

nontrivial

amount

of

time

this

bound

could

be

much

higher

than

the

actual

sparse

code

performance

Table

also

presents

the

percentage

of

the

total

execution

time

spent

in

numeric

computation

For

matrices

and

the

program

spent

less

than

of

its

time

in

the

numeric

part

Compared

to

the

others

these

two

matrices

are

sparser

have

less

fill

and

have

smaller

supernodes

so

our

supernodal

techniques

are

less

applicable

Matrix

is

also

highly

unsymmetric

which

makes

the

symmetric

structural

reduction

technique

less

effective

However

it

is

important

to

note

that

the

execution

times

for

these

two

matrices

are

small

For

larger

and

denser

matrices

such

as

-LCB-

we

achieve

between

and

Mflops

which

is

about

half

of

the

machine

peak

These

matrices

take

much

longer

to

factor

which

could

be

a

serious

bottleneck

in

an

iterative

simulation

process

Our

techniques

are

successful

in

reducing

the

solution

times

for

this

type

of

problem

For

a

dense

matrix

our

code

achieves

Mflops

This

compares

with

Mflops

reported

in

the

Lapack

manual

-LSB-

-RSB-

on

a

matrix

of

this

size

and

Mflops

reported

in

the

online

Linpack

benchmark

files

-LSB-

-RSB-

Sparc

GP

GP-Mod

SupCol-F

SupCol-C

SuperLU

Matrix

cc

cc

Geometric

Mean

Table

Speedups

achieved

by

each

enhancement

over

the

GP

column-column

code

on

a

Sparc

The

CPU

is

rated

at

MHz

and

there

is

a

MB

external

cache

This

system

does

not

provide

a

Blas

library

so

we

use

our

own

C

Blas

routines

Some

large

problems

could

not

be

tested

because

of

physical

memory

constraints

Comparison

with

previous

row

or

column

LU

algorithms

In

this

section

we

compare

the

performance

of

SuperLU

with

several

of

its

predecessors

including

the

partial

pivoting

code

by

Gilbert

and

Peierls

-LSB-

-RSB-

-LRB-

referred

to

as

GP

-RRB-

Eisenstat

and

Liu

's

improved

GP

code

that

incorporates

symmetric

reduction

-LSB-

-RSB-

-LRB-

referred

to

as

GP-Mod

-RRB-

and

two

versions

of

our

supernode-column

code

-LRB-

referred

to

as

SupCol-F

and

SupCol-C

-RRB-

GP

GP-Mod

and

SupCol-F

are

written

in

Fortran

SupCol-C

and

SuperLU

are

written

in

C

We

translated

SupCol-F

literally

into

C

to

produce

SupCol-C

no

changes

in

algorithms

or

data

structures

were

made

SupCol-F

SupCol-C

and

SuperLU

use

ESSL

Blas

-LRB-

Matlab

contains

a

C

implementation

of

GP

-LSB-

-RSB-

which

we

did

not

test

here

-RRB-

Tables

through

present

the

speedups

achieved

by

various

enhancements

over

the

original

GP

column-column

code

on

high-end

workstations

from

three

vendors

Thus

for

example

a

speedup

of

means

that

the

running

time

was

half

that

of

GP

The

numbers

in

the

last

row

of

each

table

are

obtained

by

averaging

the

speedups

in

the

corresponding

column

We

make

the

following

observations

about

the

results

on

the

IBM

ffl

Symmetric

structure

pruning

-LRB-

GP-Mod

-RRB-

is

very

effective

in

reducing

the

graph

search

time

This

significantly

decreases

the

symbolic

time

in

the

GP

code

It

achieves

speedup

in

all

problems

ffl

Supernodes

-LRB-

SupCol

-RRB-

restrict

the

search

to

the

supernodal

graph

and

allow

the

numeric

kernels

to

employ

dense

operations

The

effects

are

not

as

dramatic

as

the

pruning

technique

For

some

matrices

such

as

-LCB-

the

results

are

not

as

good

as

GP-Mod

This

is

DEC

Alpha

GP

GP-Mod

SupCol-F

SupCol-C

SuperLU

Matrix

cc

cc

Geometric

Mean

Table

Speedups

achieved

by

each

enhancement

over

the

GP

column-column

code

on

a

DEC

Alpha

The

CPU

is

rated

at

MHz

and

there

is

a

KB

external

cache

We

use

the

Blas

routines

from

DEC

's

DXML

library

Some

large

problems

could

not

be

tested

because

of

physical

memory

constraints

because

the

supernodes

are

often

small

especially

for

sparser

problems

ffl

Supernode-panel

updates

-LRB-

SuperLU

-RRB-

reduce

the

cache

miss

rate

and

exploit

dense

substructures

in

the

factor

F

For

problems

without

much

structure

the

gain

is

often

offset

by

various

overheads

However

the

benefits

become

evident

for

larger

or

denser

problems

ffl

The

Fortran

compiler

produces

slightly

faster

code

than

the

C

compiler

-LRB-

for

SupCol

-RRB-

on

both

the

IBM

and

DEC

machines

The

difference

is

about

to

for

small

problems

and

is

less

for

large

problems

where

most

of

the

time

is

spent

in

Blas

routines

We

have

seen

smaller

runtime

differences

between

the

codes

generated

by

the

SunOS

Fortran

and

C

compilers

on

the

Sun

Sparc

Matrix

is

an

exception

the

Fortran

code

is

about

faster

on

the

IBM

faster

on

the

Sun

Sparc

and

faster

on

the

DEC

Alpha

As

more

and

more

sophisticated

techniques

are

introduced

the

overhead

cost

of

the

code

is

also

increased

to

some

extent

This

overhead

can

show

up

prominently

in

small

problems

For

example

on

the

IBM

GP-Mod

works

better

than

any

of

the

subsequent

codes

for

problems

-LCB-

Comparison

with

other

approaches

to

LU

factorization

In

this

section

we

compare

our

supernode-panel

algorithm

with

other

popular

algorithms

to

solve

the

unsymmetric

linear

system

Ax

b

using

the

matrices

from

our

benchmark

suite

The

righthand

side

vector

b

is

constructed

so

that

the

solution

is

xi

i

n.

Matrix

number

-LRB-

a

-RRB-

time

-LRB-

UMF

-RRB-

time

-LRB-

LU

-RRB-

Matrix

number

-LRB-

b

-RRB-

fill

-LRB-

UMF

-RRB-

fill

-LRB-

LU

-RRB-

Matrix

number

-LRB-

c

-RRB-

mem

-LRB-

UMF

-RRB-

mem

-LRB-

LU

-RRB-

Matrix

number

-LRB-

d

-RRB-

condition

number

and

error

Figure

Comparison

of

SuperLU

algorithm

with

unsymmetric

multifrontal

algorithm

implemented

in

Umfpack

on

an

IBM

with

Blas

routines

from

the

ESSL

library

In

-LRB-

d

-RRB-

we

plot

the

estimated

condition

number

-LRB-

A

-RRB-

labeled

with

x

err

-LRB-

Umfpack

-RRB-

=-LRB-

-LRB-

A

-RRB-

ffl

-RRB-

labeled

with

o

and

err

-LRB-

SuperLU

-RRB-

=-LRB-

-LRB-

A

-RRB-

ffl

-RRB-

labeled

with

These

plots

do

not

show

Matrix

and

because

their

condition

numbers

are

larger

than

ffl

In

particular

we

consider

the

unsymmetric

multifrontal

algorithm

implemented

in

Umfpack

version

-LSB-

-RSB-

Umfpack

is

implemented

in

Fortran

so

we

use

the

AIX

xlf

compiler

with

optimization

and

link

with

the

IBM

ESSL

library

for

Blas

calls

We

use

the

parameter

settings

for

Umfpack

suggested

by

its

authors

-LSB-

-RSB-

Figure

compares

several

aspects

of

the

two

implementations

We

plot

the

ratio

of

each

individual

measure

from

Umfpack

to

that

of

our

SuperLU

code

The

memory

requirement

only

counts

the

amount

of

memory

actually

used

excluding

any

external

fragmentation

Neither

code

is

always

faster

than

the

other

For

six

problems

Umfpack

is

faster

than

SuperLU

by

at

most

a

factor

of

two

For

the

rest

of

problems

SuperLU

is

faster

than

Umfpack

For

seven

out

of

the

matrices

SuperLU

runs

more

than

twice

as

fast

as

Umfpack

Figure

-LRB-

d

-RRB-

compares

the

solution

accuracy

and

stability

of

the

two

approaches

without

using

iterative

refinement

SuperLU

consistently

delivers

more

accurate

solutions

because

Umfpack

uses

a

threshold

pivoting

strategy

to

trade

off

stability

and

sparsity

In

particular

for

stable

algorithms

we

expect

the

normalized

error

to

be

kx

k

-LRB-

A

-RRB-

ffl

-LRB-

-RRB-

But

very

frequently

this

error

from

Umfpack

is

much

larger

than

-LRB-

-RRB-

For

Matrix

the

backward

error

is

as

large

as

ffl

To

mitigate

the

instability

we

experimented

with

workingprecision

iterative

refinement

in

Umfpack

and

found

that

in

many

cases

one

iteration

can

reduce

the

backward

error

to

a

level

comparable

to

that

of

SuperLU

The

refinement

process

normally

takes

less

than

of

the

factorization

time

We

recommend

that

Umfpack

incorporate

this

inexpensive

technique

to

guarantee

backward

stability

The

iteration

can

be

terminated

when

the

scaled

residual

k

is

acceptable

say

less

than

machine

precision

ffl

In

our

SuperLU

software

equilibration

and

refinement

are

options

for

the

user

We

compute

the

componentwise

relative

backward

error

maxi

-LRB-

j

rji

-LRB-

jAjj

xj

jbj

-RRB-

i

-RRB-

and

stop

refining

when

ffl

or

does

not

decrease

by

at

least

a

factor

of

two

See

Arioli

Demmel

and

Duff

-LSB-

-RSB-

for

details

In

practice

most

of

the

test

matrices

take

only

one

or

two

refinement

steps

to

meet

this

criterion

The

time

for

SuperLU

does

not

include

the

time

to

reorder

the

columns

Umfpack

is

less

sensitive

to

the

initial

column

ordering

because

it

dynamically

permutes

the

columns

for

sparsity

Surprisingly

Figure

-LRB-

b

-RRB-

seems

to

suggest

that

for

large

matrices

the

dynamic

fill-reducing

approach

used

in

Umfpack

is

less

effective

than

the

minimum

degree

ordering

algorithms

Understanding

cache

behavior

and

parameters

In

this

subsection

we

analyze

the

behavior

of

SuperLU

in

detail

We

wish

to

understand

when

our

algorithm

is

significantly

faster

than

other

algorithms

We

would

like

performance-predicting

variable

-LRB-

s

-RRB-

that

depend

on

intrinsic

properties

of

the

problem

such

as

the

sparsity

structure

rather

than

algorithmic

details

and

machine

characteristics

We

begin

by

analyzing

the

speedups

of

our

enhanced

codes

over

the

base

GP

implementation

Figures

and

depict

the

speedups

and

the

characteristics

of

the

matrices

with

panel

size

w

How

much

cache

reuse

can

we

expect

As

discussed

in

Section

the

supernode-panel

algorithm

gets

its

primary

gains

from

improved

data

locality

by

reusing

a

cached

supernode

several

times

To

understand

how

much

cache

reuse

SuperLU

SupCol-C

GP-Mod

GP

Matrix

number

Speedups

on

a

Model

Figure

Speedups

of

each

enhancement

over

GP

code

on

the

we

can

hope

for

we

computed

two

statistics

ops-per-nz

and

ops-per-ref

After

defining

these

statistics

carefully

we

discuss

which

more

successfully

measures

reuse

Ops-per-nz

is

simply

calculated

as

f

lops

nnz

-LRB-

F

-RRB-

the

total

number

of

floating

point

operations

per

nonzero

in

the

filled

matrix

F

If

there

were

perfect

cache

behavior

i.e.

one

cache

miss

per

data

item

-LRB-

ignoring

the

effect

of

cache

line

size

-RRB-

then

ops-per-nz

would

exactly

measure

the

amount

of

work

per

cache

miss

In

reality

ops-per-nz

is

an

upper

bound

on

the

reuse

Note

that

ops-per-nz

depends

only

on

the

fact

that

we

are

performing

Gaussian

elimination

with

partial

pivoting

not

on

algorithmic

or

machine

details

Ops-per-nz

is

a

natural

measure

of

potential

data

reuse

because

it

has

long

been

used

to

distinguish

among

the

different

levels

of

Blas

In

contrast

ops-per-ref

provides

a

lower

bound

on

cache

reuse

and

does

depend

on

the

details

of

the

SuperLU

algorithm

Ops-per-ref

looks

at

each

supernode-panel

update

separately

and

assumes

that

all

the

associated

data

is

outside

the

cache

before

beginning

the

update

This

pessimistic

assumption

limits

the

potential

reuse

to

twice

the

panel

size

Now

we

define

ops-per-ref

more

carefully

Consider

a

single

update

from

supernode

-LRB-

r

s

-RRB-

to

panel

-LRB-

j

j

w

-RRB-

Depending

on

the

panel

's

nonzero

structure

each

entry

in

the

updating

supernode

is

used

to

update

from

to

w

panel

columns

Thus

each

entry

in

the

updating

supernode

participates

in

between

and

floating

point

operations

during

a

sup-panel

update

We

assume

that

the

supernode

entry

is

brought

into

cache

from

main

memory

exactly

once

for

the

entire

suppanel

update

if

it

is

used

at

all

Thus

during

a

single

sup-panel

update

each

entry

accessed

in

the

updating

supernode

accounts

for

between

and

operations

per

reference

The

ops-per-ref

statistic

is

the

average

of

this

number

over

all

entries

in

all

sup-panel

updates

It

measures

how

many

times

the

average

supernode

entry

is

used

each

time

it

is

brought

into

cache

from

main

memory

Ops-per-ref

ranges

from

to

with

larger

values

indicating

better

cache

use

If

there

is

little

correlation

between

the

row

structures

of

the

columns

in

each

panel

ops-per-ref

will

be

small

if

there

is

perfect

correlation

as

in

a

dense

matrix

it

will

be

close

to

Matrix

number

-LRB-

a

-RRB-

ops-per-nz

-LRB-

F

-RRB-

Matrix

number

-LRB-

b

-RRB-

ops-per-ref

Matrix

number

-LRB-

c

-RRB-

Flop

count

Matrix

number

-LRB-

d

-RRB-

nnz

-LRB-

F

-RRB-

Figure

Some

characteristics

of

the

matrices

Matrix

number

-LRB-

a

-RRB-

Mean

supernode

size

Matrix

number

-LRB-

b

-RRB-

Structural

symmetry

Matrix

number

-LRB-

c

-RRB-

Dimension

n

Matrix

number

-LRB-

d

-RRB-

Density

Figure

Some

intrinsic

properties

of

the

matrices

Now

we

describe

how

we

compute

the

average

ops-per-ref

for

the

entire

factorization

For

each

updating

supernode

-LRB-

r

s

-RRB-

and

each

panel

-LRB-

j

j

w

-RRB-

-LRB-

see

Figure

-RRB-

define

ksmin

min

j

jj

j

w

r

i

sfijA

-LRB-

i

jj

-RRB-

g

Then

nnz

-LRB-

L

-LRB-

r

n

ksmin

s

-RRB-

-RRB-

entries

of

the

supernode

are

referenced

in

the

sup-panel

update

The

dense

triangular

solve

in

column

jj

of

the

update

takes

-LRB-

s

ks

-RRB-

-LRB-

s

ks

-RRB-

flops

where

ks

minr

i

sfijA

-LRB-

i

jj

-RRB-

g

The

matrix-vector

multiply

uses

-LRB-

s

ks

-RRB-

nnz

-LRB-

L

-LRB-

s

n

s

-RRB-

-RRB-

flops

We

count

both

additions

and

multiplications

For

all

panel

updates

we

sum

the

memory

reference

counts

and

the

flop

counts

then

divide

the

second

sum

by

the

first

to

arrive

at

an

average

ops-perref

Now

we

compare

the

predictive

powers

of

ops-per-nz

-LRB-

Figure

-LRB-

a

-RRB-

-RRB-

and

ops-per-ref

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

in

predicting

speedup

-LRB-

Figure

-RRB-

The

superiority

of

ops-per-nz

is

evident

it

is

much

more

strongly

correlated

with

the

speedup

of

SuperLU

than

ops-per-ref

This

is

good

news

because

ops-per-nz

measures

the

best

case

reuse

and

ops-per-ref

the

worst

case

But

neither

statistic

captures

all

the

variation

in

the

performance

In

future

work

we

hope

to

use

a

hardware

monitor

to

measure

the

exact

cache

reuse

rate

-LRB-

This

data

could

also

be

obtained

from

a

simulator

but

the

matrices

we

are

interested

in

are

much

too

large

for

a

simulator

to

be

viable

-RRB-

How

large

are

the

supernodes

The

supernode

size

determines

the

size

of

the

matrix

to

be

passed

to

matrix-vector

multiply

and

other

routines

in

our

algorithm

Figure

-LRB-

a

-RRB-

shows

the

average

number

of

columns

in

the

supernodes

of

the

matrices

after

amalgamating

the

relaxed

supernodes

at

the

bottom

of

the

column

etree

The

average

size

is

usually

quite

small

More

important

than

average

size

is

the

distribution

of

supernode

sizes

In

sparse

Gaussian

elimination

more

fill

tends

to

occur

in

the

later

stages

Usually

there

is

a

large

percentage

of

small

supernodes

corresponding

to

the

fringe

of

the

column

etree

even

after

amalgamation

Larger

supernodes

appear

at

the

higher

levels

of

the

tree

In

Figure

we

plot

the

histograms

of

the

supernode

size

for

four

matrices

chosen

to

exhibit

a

wide

range

of

behavior

Matrix

has

supernodes

all

but

one

of

which

have

less

than

columns

the

single

large

supernode

with

columns

is

the

dense

submatrix

at

the

bottom

right

corner

of

F

Matrix

has

more

supernodes

distributed

over

a

wider

spectrum

it

has

supernodes

with

to

columns

This

matrix

gives

greater

speedups

over

the

non-supernodal

codes

Figure

also

plots

three

other

properties

of

each

matrix

structural

symmetry

dimension

and

density

of

them

have

any

significant

correlation

with

the

performance

The

effectiveness

of

symmetric

reduction

depends

on

F

being

structurally

symmetric

which

depends

on

the

choice

of

pivots

So

structural

symmetry

of

A

does

not

gives

any

useful

information

We

note

that

the

speedups

achieved

by

the

dense

problem

-LRB-

matrix

-RRB-

show

the

best

performance

gains

because

this

matrix

has

large

supernodes

and

exhibits

ideal

data

reuse

It

achieves

speedups

of

to

on

the

three

platforms

The

gains

for

any

sparse

matrix

should

be

smaller

than

this

Blocking

parameters

In

our

hybrid

blocking

algorithm

-LRB-

Figure

-RRB-

we

need

to

select

appropriate

values

for

the

parameters

that

describe

the

two-dimensional

data

blocking

panel

width

w

maximum

supernode

size

t

and

row

block

size

b

The

key

considerations

are

that

the

active

data

we

access

in

the

inner

loop

-LRB-

the

-LRB-

a

-RRB-

Matrix

rows

supernodes

Supernode

size

-LRB-

number

of

occurrences

-RRB-

-LRB-

b

-RRB-

Matrix

rows

supernodes

Supernode

size

-LRB-

number

of

occurrences

-RRB-

-LRB-

c

-RRB-

Matrix

rows

supernodes

Supernode

size

-LRB-

number

of

occurrences

-RRB-

-LRB-

d

-RRB-

Matrix

rows

supernodes

Supernode

size

-LRB-

number

of

occurrences

-RRB-

Figure

Distribution

of

supernode

size

for

four

matrices

The

number

at

the

bottom

of

each

bar

is

the

smallest

supernode

size

in

that

bin

The

mark

o

at

the

bottom

of

a

bin

indicates

zero

occurrences

Artificial

supernodes

of

granularity

r

are

used

-LRB-

see

Section

-RRB-

125 125 125

column

dimension

n

row

dimension

m

-LRB-

-RRB-

contour

of

DGEMV

-LRB-

m

n

-RRB-

MFLOPS

m

n

K

doubles

60 80 100 120

max

supernode

size

t

row

block

size

b

contour

of

working

set

size

KB

cache

size

o

-LRB-

-RRB-

w

w

Figure

-LRB-

a

-RRB-

Contour

plot

of

Dgemv

performance

-LRB-

b

-RRB-

Contour

plot

of

working

set

in

algorithm

working

set

-RRB-

should

fit

into

the

cache

and

that

the

matrices

presented

to

the

routine

Dgemv

should

be

the

sizes

and

shapes

for

which

that

routine

is

optimized

Here

we

describe

in

detail

the

methodology

we

used

to

choose

parameters

for

the

IBM

ffl

Dgemv

optimization

As

indicated

in

the

last

column

of

Table

the

majority

of

the

floating-point

operations

are

in

the

matrix-vector

multiply

The

dimensions

-LRB-

m

n

-RRB-

of

the

matrices

in

calls

to

Dgemv

vary

greatly

depending

on

the

supernode

dimensions

Very

often

the

supernode

is

a

tall

and

skinny

matrix

that

is

m

AE

n

We

measured

the

Dgemv

Mflops

rate

for

various

m

and

n

and

present

a

contour

plot

in

the

-LRB-

m

n

-RRB-

plane

in

Figure

-LRB-

a

-RRB-

Each

contour

represents

a

constant

Mflops

rate

The

dashed

curve

represents

mn

double

floats

or

a

cache

capacity

of

KB

In

the

optimum

region

we

achieve

more

than

Mflops

outside

this

region

performance

drops

either

because

the

matrices

exceed

the

cache

capacity

or

because

the

column

dimension

n

is

too

small

ffl

Working

set

By

studying

the

data

access

pattern

in

the

inner

loop

of

the

algorithm

lines

-LRB-

-LCB-

-RRB-

in

Figure

we

find

that

the

working

set

size

is

the

following

function

of

w

t

and

b

as

shown

in

Figure

WS

b

t

-LCB-

z

-RCB-

row

block

from

supernode

-LRB-

t

b

-RRB-

w

-LCB-

z

-RCB-

vectors

in

matrix-vector

multiply

b

w

-LCB-

z

-RCB-

part

of

SPA

structure

In

Figure

-LRB-

b

-RRB-

we

fix

two

w

values

and

plot

the

contour

lines

for

WS

in

the

-LRB-

t

b

-RRB-

plane

If

the

point

-LRB-

t

b

-RRB-

is

below

the

contour

curve

then

the

working

set

can

fit

in

a

cache

of

double

floats

or

kilobytes

Based

on

this

analysis

we

use

the

following

rules

to

set

the

parameters

J

J

L'

U

J

L

b

W

t

t

b

Figure

Parameters

of

the

working

set

in

the

algorithm

First

we

choose

w

the

width

of

the

panel

in

columns

Larger

panels

mean

more

reuse

of

cached

data

in

the

outer

factorization

but

also

mean

that

the

inner

factorization

-LRB-

by

the

sup-col

algorithm

-RRB-

must

be

applied

to

larger

matrices

We

find

empirically

that

the

best

choice

for

w

is

between

and

Performance

tends

to

degrade

for

larger

w.

Next

we

choose

b

the

number

of

rows

per

block

and

t

the

maximum

number

of

columns

in

a

supernode

Recall

that

b

and

t

are

upper

bounds

on

the

row

and

column

dimensions

of

the

call

to

Dgemv

We

choose

t

and

b

ss

which

guarantees

that

the

working

set

fits

in

cache

-LRB-

per

Figure

-LRB-

b

-RRB-

-RRB-

and

that

we

can

hope

to

be

near

the

optimum

region

of

Dgemv

performance

-LRB-

per

Figure

-LRB-

a

-RRB-

-RRB-

Recall

that

b

is

relevant

only

when

we

use

row-wise

blocking

that

is

when

the

test

if

-LRB-

r

s

-RRB-

is

large

succeeds

at

line

of

Figure

This

implies

that

the

scheme

adds

overhead

only

if

the

updating

supernode

is

small

In

the

actual

code

the

test

for

a

large

supernode

is

if

ncol

and

nrow

b

then

the

supernode

is

large

where

nrow

is

the

number

of

dense

rows

below

the

diagonal

block

of

the

supernode

ncol

is

the

number

of

actual

dense

columns

of

the

supernode

updating

the

panel

In

practice

this

choice

usually

gives

the

best

performance

The

best

choice

of

the

parameters

w

t

and

b

depends

on

the

machine

architecture

and

on

the

Blas

implementation

but

it

is

largely

independent

of

the

matrix

structure

Thus

we

do

not

expect

each

user

of

SuperLU

to

choose

values

for

these

parameters

Instead

our

library

code

provides

an

inquiry

function

that

returns

the

parameter

values

much

in

the

spirit

of

the

Lapack

environment

routine

Ilaenv

The

machine-independent

defaults

will

often

give

satisfactory

performance

The

methodology

we

have

described

here

for

the

can

serve

as

a

guide

for

users

who

want

to

modify

the

inquiry

function

to

give

optimal

performance

for

particular

computer

systems

Remarks

The

rest

of

the

package

In

addition

to

the

LU

factorization

kernel

described

in

this

paper

we

have

developed

a

suite

of

supporting

routines

to

solve

linear

systems

The

complete

SuperLU

package

includes

condition

number

estimation

iterative

refinement

of

solutions

and

componentwise

error

bounds

for

the

refined

solutions

These

are

all

based

on

the

dense

matrix

routines

in

Lapack

-LSB-

-RSB-

In

addition

SuperLU

includes

a

Matlab

mex-file

interface

so

that

our

factor

and

solve

routines

can

be

called

as

alternatives

to

those

built

into

Matlab

Effect

of

the

matrix

on

performance

The

supernodal

approach

reduces

both

symbolic

and

numeric

computation

time

But

unsymmetric

supernodes

tend

to

be

smaller

than

supernodes

in

symmetric

matrices

The

supernode-panel

method

is

most

effective

for

large

problems

with

enough

dense

submatrices

to

use

dense

block

operations

and

exploit

data

locality

In

this

regard

the

dense

example

illustrates

the

largest

possible

gains

Dense

blocks

are

necessary

for

top

performance

in

all

modern

factorization

algorithms

whether

left-looking

right-looking

multifrontal

or

any

other

style

Our

goal

has

been

to

develop

sparse

LU

software

that

works

well

for

problems

with

a

wide

range

of

characteristics

It

is

harder

to

achieve

high

flop

rates

on

problems

that

are

very

sparse

and

have

no

structure

to

exploit

it

is

easier

on

problems

that

are

denser

or

become

so

during

elimination

Fortunately

the

hard

matrices

by

this

definition

generally

take

many

fewer

floating

point

operations

than

the

easy

ones

and

hence

take

much

less

time

to

factor

Our

combination

of

and

blocking

techniques

gives

a

good

performance

compromise

for

all

the

problems

we

have

studied

and

with

particularly

good

performance

on

the

largest

problems

Effect

of

the

computer

system

on

performance

We

have

studied

several

characteristics

of

the

computing

platform

that

can

affect

the

overall

performance

including

the

speed

and

the

cache

size

Based

on

these

factors

we

can

systematically

make

a

good

choice

of

the

blocking

parameters

in

the

code

so

as

to

maximize

the

speed

of

the

numeric

kernel

Although

we

have

empirical

evidence

only

for

the

IBM

we

expect

this

methodology

to

be

applicable

to

other

systems

-LRB-

and

Blas

implementations

-RRB-

as

well

Possible

enhancements

We

are

considering

several

possible

enhancements

to

the

SuperLU

code

One

is

to

switch

to

a

dense

LU

code

at

a

late

stage

of

the

factorization

It

would

be

difficult

to

implement

this

in

a

supernode-column

code

because

that

code

is

strictly

left-looking

and

only

one

column

of

the

matrix

is

factored

at

a

time

However

this

would

be

much

easier

in

the

supernode-panel

code

At

the

time

we

decide

to

switch

we

simply

treat

the

rest

of

the

matrix

columns

-LRB-

say

d

of

them

-RRB-

as

one

panel

and

perform

the

panel

update

to

A

-LRB-

n

n?d

n

-RRB-

-LRB-

One

might

want

to

split

this

panel

up

for

better

cache

behavior

-RRB-

Then

the

reduced

matrix

at

the

bottom

right

corner

can

be

factored

by

calling

an

efficient

dense

code

for

example

from

Lapack

-LSB-

-RSB-

The

dense

code

does

not

spend

time

on

symbolic

structure

prediction

and

pruning

thus

streamlining

the

numeric

computation

We

believe

that

for

large

problems

the

final

dense

submatrix

will

be

big

enough

to

make

the

switch

beneficial

For

example

for

a

k

k

square

grid

problem

ordered

by

nested

dissection

the

dimension

of

the

final

dense

submatrix

is

k

k

for

a

k

k

k

cubic

grid

it

is

if

pivots

come

from

the

diagonal

To

enhance

SuperLU

's

performance

on

small

problems

it

would

be

possible

to

make

a

choice

at

runtime

whether

to

use

supernode-panel

supernode-column

or

column-column

updates

The

choice

would

depend

on

the

size

of

the

matrix

A

and

the

expected

properties

of

its

supernodes

it

might

be

based

on

an

efficient

symbolic

computation

of

the

density

and

supernode

distribution

of

the

Cholesky

factor

of

ATA

-LSB-

-RSB-

Could

we

make

supernode-panel

panel

updates

more

effective

by

improving

the

similarity

between

the

row

structures

of

the

columns

in

a

panel

We

believe

this

could

be

accomplished

with

a

more

sophisticated

column

permutation

strategy

We

could

partition

the

nodes

of

the

column

etree

into

connected

subtrees

grouping

together

nodes

that

have

common

descendants

-LRB-

and

therefore

the

potential

for

updates

from

the

same

supernodes

-RRB-

Then

the

overall

column

order

would

be

a

two-level

postorder

first

within

the

subtrees

-LRB-

panels

-RRB-

and

then

among

them

Again

it

might

be

possible

to

use

information

about

the

Cholesky

supernodes

of

ATA

to

guide

this

grouping

We

are

also

developing

a

parallel

sparse

LU

algorithm

based

on

SuperLU

In

this

context

we

target

large

problems

especially

those

too

big

to

be

solved

on

a

uniprocessor

system

Therefore

we

plan

to

parallelize

the

blocked

supernode-panel

algorithm

which

has

very

good

asymptotic

behavior

for

large

problems

The

block-oriented

layout

has

been

shown

to

scale

well

for

parallel

sparse

Cholesky

factorization

-LSB-

-RSB-

Acknowledgements

We

thank

Rob

Schreiber

and

Ed

Rothberg

for

very

helpful

discussions

during

the

development

of

SuperLU

Rob

pushed

us

to

find

a

way

to

do

the

supernode-panel

update

Ed

suggested

the

ops-per-ref

statistic

We

are

also

grateful

to

Tim

Davis

and

Steve

Vavasis

for

making

their

test

problems

available

to

us

The

Institute

for

Mathematics

and

Its

Applications

at

the

University

of

Minnesota

provided

the

fertile

environment

in

which

this

work

began

References

-LSB-

-RSB-

R.C.

Agarwal

F.G.

Gustavson

P.

Palkar

and

M.

Zubair

A

performance

analysis

of

the

subroutines

in

the

ESSL/LAPACK

call

conversion

interface

-LRB-

CCI

-RRB-

IBM

T.J.

Watson

Research

Center

Yorktown

Heights

-LSB-

-RSB-

E.

Anderson

et

al.

LAPACK

User

's

Guide

Second

Edition

SIAM

Philadelphia

-LSB-

-RSB-

M.

Arioli

J.

W.

Demmel

and

I.

S.

Duff

Solving

sparse

linear

systems

with

sparse

backward

error

SIAM

J.

Matrix

Anal

Appl.

-LRB-

-RRB-

-LCB-

April

-LSB-

-RSB-

C.

Ashcraft

and

R.

Grimes

The

influence

of

relaxed

supernode

partitions

on

the

multifrontal

method

ACM

Trans

Mathematical

Software

-LCB-

-LSB-

-RSB-

C.

Ashcraft

R.

Grimes

J.

Lewis

B.

Peyton

and

H.

Simon

Progress

in

sparse

matrix

methods

for

large

sparse

linear

systems

on

vector

supercomputers

Intern

J.

of

Supercomputer

Applications

-LCB-

-LSB-

-RSB-

T.

A.

Davis

User

's

guide

for

the

unsymmetric-pattern

multifrontal

package

-LRB-

UMFPACK

-RRB-

Technical

Report

Computer

and

Information

Sciences

Department

University

of

Florida

June

-LSB-

-RSB-

T.

A.

Davis

and

I.

S.

Duff

An

unsymmetric-pattern

multifrontal

method

for

sparse

LU

factorization

Technical

Report

RAL

Rutherford

Appleton

Laboratory

Chilton

Didcot

Oxfordshire

-LSB-

-RSB-

J.

J.

Dongarra

J.

Du

Croz

S.

Hammarling

and

R.

J.

Hanson

An

extended

set

of

basic

linear

algebra

subroutines

ACM

Trans

Mathematical

Software

-LCB-

-LCB-

-LSB-

-RSB-

I.

S.

Duff

R.

Grimes

and

J.

Lewis

Sparse

matrix

test

problems

ACM

Trans

Mathematical

Software

-LCB-

-LSB-

-RSB-

I.

S.

Duff

and

J.

K.

Reid

a

Fortran

code

for

direct

solution

of

sparse

unsymmetric

linear

systems

of

equations

Technical

Report

RAL

-LCB-

-LCB-

Rutherford

Appleton

Laboratory

Oxon

UK

-LSB-

-RSB-

I.S.

Duff

and

J.K.

Reid

The

multifrontal

solution

of

indefinite

sparse

symmetric

linear

equations

ACM

Trans

Mathematical

Software

-LCB-

-LSB-

-RSB-

S.

C.

Eisenstat

and

J.

W.

H.

Liu

Exploiting

structural

symmetry

in

sparse

unsymmetric

symbolic

factorization

SIAM

J.

Matrix

Analysis

and

Applications

-LCB-

-LSB-

-RSB-

S.

C.

Eisenstat

and

J.

W.

H.

Liu

Exploiting

structural

symmetry

in

a

sparse

partial

pivoting

code

SIAM

J.

Scientific

and

Statistical

Computing

-LCB-

-LSB-

-RSB-

J.

A.

George

and

E.

Ng

An

implementation

of

Gaussian

elimination

with

partial

pivoting

for

sparse

systems

SIAM

J.

Scientific

and

Statistical

Computing

-LCB-

-LSB-

-RSB-

J.

A.

George

and

E.

Ng

Symbolic

factorization

for

sparse

Gaussian

elimination

with

partial

pivoting

SIAM

J.

Scientific

and

Statistical

Computing

-LCB-

-LSB-

-RSB-

J.

R.

Gilbert

Predicting

structure

in

sparse

matrix

computations

SIAM

J.

Matrix

Analysis

and

Applications

-LCB-

-LSB-

-RSB-

J.

R.

Gilbert

and

J.

W.

H.

Liu

Elimination

structures

for

unsymmetric

sparse

LU

factors

SIAM

J.

Matrix

Analysis

and

Applications

-LCB-

-LSB-

-RSB-

J.

R.

Gilbert

C.

Moler

and

R.

Schreiber

Sparse

matrices

in

Matlab

Design

and

implementation

SIAM

J.

Matrix

Analysis

and

Applications

-LCB-

-LSB-

-RSB-

J.

R.

Gilbert

and

E.

Ng

Predicting

structure

in

nonsymmetric

sparse

matrix

factorizations

In

Alan

George

John

R.

Gilbert

and

Joseph

W.

H.

Liu

editors

Graph

Theory

and

Sparse

Matrix

Computation

Springer-Verlag

-LSB-

-RSB-

J.

R.

Gilbert

and

T.

Peierls

Sparse

partial

pivoting

in

time

proportional

to

arithmetic

operations

SIAM

J.

Scientific

and

Statistical

Computing

-LCB-

-LSB-

-RSB-

John

R.

Gilbert

Esmond

G.

Ng

and

Barry

W.

Peyton

An

efficient

algorithm

to

compute

row

and

column

counts

for

sparse

Cholesky

factorization

SIAM

J.

Matrix

Analysis

and

Applications

-LCB-

-LSB-

-RSB-

A.

Gupta

and

V.

Kumar

Optimally

scalable

parallel

sparse

cholesky

factorization

In

The

SIAM

Conference

on

Parallel

Processing

for

Scientific

Computing

pages

-LCB-

-LSB-

-RSB-

International

Business

Machines

Corporation

Engineering

and

Scientific

Subroutine

Library

Guide

and

Reference

Version

Release

Order

No

-LSB-

-RSB-

J.

W.

H.

Liu

The

role

of

elimination

trees

in

sparse

factorization

SIAM

J.

Matrix

Analysis

and

Applications

-LCB-

-LSB-

-RSB-

PDS

The

performance

database

server

May

-LSB-

-RSB-

E.

G.

Ng

and

B.

W.

Peyton

Block

sparse

Cholesky

algorithms

on

advanced

uniprocessor

computers

SIAM

J.

Scientific

and

Statistical

Computing

-LCB-

-LSB-

-RSB-

E.

Rothberg

and

A.

Gupta

Efficient

sparse

matrix

factorization

on

high-performance

workstations

-LCB-

exploiting

the

memory

hierarchy

ACM

Trans

Mathematical

Software

-LCB-

-LSB-

-RSB-

E.

Rothberg

and

A.

Gupta

An

evaluation

of

left-looking

right-looking

and

multifrontal

approaches

to

sparse

Cholesky

factorization

on

hierarchical-memory

machines

Int

J.

High

Speed

Computing

-LCB-

-LSB-

-RSB-

E.

E.

Rothberg

and

A.

Gupta

An

efficient

block-oriented

approach

to

parallel

sparse

cholesky

factorization

In

Supercomputing

pages

-LCB-

November

-LSB-

-RSB-

A.

H.

Sherman

On

the

efficient

solution

of

sparse

systems

of

linear

and

nonlinear

equations

PhD

thesis

Yale

University

-LSB-

-RSB-

A.

H.

Sherman

Algorithm

NSPIV

a

FORTRAN

subroutine

for

sparse

Gaussian

elimination

with

partial

pivoting

ACM

Trans

Mathematical

Software

-LCB-

-LSB-

-RSB-

H.

Simon

P.

Vu

and

C.

Yang

Performance

of

a

supernodal

general

sparse

solver

on

the

CRAY

Y-MP

GFLOPS

with

autotasking

Technical

Report

TR

Boeing

Computer

Services

-LSB-

-RSB-

S.

A.

Vavasis

Stable

finite

elements

for

problems

with

wild

coefficients

Technical

Report

-LCB-

Department

of

Computer

Science

Cornell

University

Ithaca

NY

To

appear

in

SIAM

J.

Numerical

Analysis

column elimination tree
partial pivoting
supernodes
unsymmetric linear systems