A

Randomized

Parallel

Sorting

Algorithm

With

an

Experimental

Study

David

R.

Helman

David

A.

Bader

Joseph

J?aJ?ay

Institute

for

Advanced

Computer

Studies

Department

of

Electrical

Engineering

University

of

Maryland

College

Park

MD

fhelman

dbader

josephg@umiacs.umd.edu

August

Abstract

Previous

schemes

for

sorting

on

general-purpose

parallel

machines

have

had

to

choose

between

poor

load

balancing

and

irregular

communication

or

multiple

rounds

of

all-to-all

personalized

communication

In

this

paper

we

introduce

a

novel

variation

on

sample

sort

which

uses

only

two

rounds

of

regular

all-to-all

personalized

communication

in

a

scheme

that

yields

very

good

load

balancing

with

virtually

no

overhead

Moreover

unlike

previous

variations

our

algorithm

efficiently

handles

the

presence

of

duplicate

values

without

the

overhead

of

tagging

each

element

with

a

unique

identifier

This

algorithm

was

implemented

in

Split-C

and

run

on

a

variety

of

platforms

including

the

Thinking

Machines

the

IBM

and

the

Cray

Research

We

ran

our

code

using

widely

different

benchmarks

to

examine

the

dependence

of

our

algorithm

on

the

input

distribution

Our

experimental

results

illustrate

the

efficiency

and

scalability

of

our

algorithm

across

different

platforms

In

fact

it

seems

to

outperform

all

similar

algorithms

known

to

the

authors

on

these

platforms

and

its

performance

is

invariant

over

the

set

of

input

distributions

unlike

previous

efficient

algorithms

Our

results

also

compare

favorably

with

those

reported

for

the

simpler

ranking

problem

posed

by

the

NAS

Integer

Sorting

-LRB-

IS

-RRB-

Benchmark

The

support

by

NASA

Graduate

Student

Researcher

Fellowship

No

is

gratefully

acknowledged

ySupported

in

part

by

NSF

grant

No

and

NSF

HPCC/GCAG

grant

No

Introduction

Sorting

is

arguably

the

most

studied

problem

in

computer

science

both

because

of

its

intrinsic

theoretical

importance

and

its

use

in

so

many

applications

Its

significant

requirements

for

interprocessor

communication

bandwidth

and

the

irregular

communication

patterns

that

are

typically

generated

have

earned

its

inclusion

in

several

parallel

benchmarks

such

as

NAS

-LSB-

-RSB-

and

SPLASH

-LSB-

-RSB-

Moreover

its

practical

importance

has

motivated

the

publication

of

a

number

of

empirical

studies

seeking

to

identify

the

most

efficient

sorting

routines

Yet

parallel

sorting

strategies

have

still

generally

fallen

into

one

of

two

groups

each

with

its

respective

disadvantages

The

first

group

using

the

classification

of

Li

and

Sevcik

-LSB-

-RSB-

is

the

single-step

algorithms

so

named

because

data

is

moved

exactly

once

between

processors

Examples

of

this

include

sample

sort

-LSB-

-RSB-

parallel

sorting

by

regular

sampling

-LSB-

-RSB-

and

parallel

sorting

by

overpartitioning

-LSB-

-RSB-

The

price

paid

by

these

single-step

algorithms

is

an

irregular

communication

scheme

and

difficulty

with

load

balancing

The

other

group

of

sorting

algorithms

is

the

multi-step

algorithms

which

include

bitonic

sort

-LSB-

-RSB-

column

sort

-LSB-

-RSB-

rotate

sort

-LSB-

-RSB-

hyperquicksort

-LSB-

-RSB-

flashsort

-LSB-

-RSB-

B-flashsort

-LSB-

-RSB-

smoothsort

-LSB-

-RSB-

and

Tridgell

and

Brent

's

sort

-LSB-

-RSB-

Generally

speaking

these

algorithms

accept

multiple

rounds

of

communication

in

return

for

better

load

balancing

and

in

some

cases

regular

communication

In

this

paper

we

present

a

novel

variation

on

the

sample

sort

algorithm

-LSB-

-RSB-

which

addresses

the

limitations

of

previous

implementations

We

exchange

the

single

step

of

irregular

communication

for

two

steps

of

regular

communication

In

return

we

reduce

the

problem

of

poor

load

balancing

because

we

are

able

to

sustain

a

very

high

oversampling

ratio

at

virtually

no

cost

Second

we

efficiently

accommodate

the

presence

of

duplicates

without

the

overhead

of

tagging

each

element

And

we

obtain

predictable

regular

communication

requirements

which

are

essentially

invariant

with

respect

to

the

input

distribution

Utilizing

regular

communication

has

become

more

important

with

the

advent

of

message

passing

standards

such

as

MPI

-LSB-

-RSB-

which

seek

to

guarantee

the

availability

of

very

efficient

-LRB-

often

machine

specific

-RRB-

implementations

of

certain

basic

collective

communication

routines

Our

algorithm

was

implemented

in

a

high-level

language

and

run

on

a

variety

of

platforms

including

the

Thinking

Machines

the

IBM

and

the

Cray

Research

We

ran

our

code

using

a

variety

of

benchmarks

that

we

identified

to

examine

the

dependence

of

our

algorithm

on

the

input

distribution

Our

experimental

results

are

consistent

with

the

theoretical

analysis

and

illustrate

the

scalability

and

efficiency

of

our

algorithm

across

different

platforms

In

fact

it

seems

to

outperform

all

similar

algorithms

known

to

the

authors

on

these

platforms

and

its

performance

is

indifferent

to

the

set

of

input

distributions

unlike

previous

efficient

algorithms

The

high-level

language

used

in

our

studies

is

Split-C

-LSB-

-RSB-

an

extension

of

C

for

distributed

memory

machines

The

algorithm

makes

use

of

MPI-like

communication

primitives

but

does

not

make

any

assumptions

as

to

how

these

primitives

are

actually

implemented

The

basic

data

transport

is

a

read

or

write

operation

The

remote

read

and

write

typically

have

both

blocking

and

nonblocking

versions

Also

when

reading

or

writing

more

than

a

single

element

bulk

data

transports

are

provided

with

corresponding

bulk

read

and

bulk

write

primitives

Our

collective

communication

primitives

described

in

detail

in

-LSB-

-RSB-

are

similar

to

those

of

the

MPI

-LSB-

-RSB-

the

IBM

POWERparallel

-LSB-

-RSB-

and

the

Cray

MPP

systems

-LSB-

-RSB-

and

for

example

include

the

following

transpose

bcast

gather

and

scatter

Brief

descriptions

of

these

are

as

follows

The

transpose

primitive

is

an

all-to-all

personalized

communication

in

which

each

processor

has

to

send

a

unique

block

of

data

to

every

processor

and

all

the

blocks

are

of

the

same

size

The

bcast

primitive

is

used

to

copy

a

block

of

data

from

a

single

source

to

all

the

other

processors

The

primitives

gather

and

scatter

are

companion

primitives

Scatter

divides

a

single

array

residing

on

a

processor

into

equal-sized

blocks

each

of

which

is

distributed

to

a

unique

processor

and

gather

coalesces

these

blocks

back

into

a

single

array

at

a

particular

processor

See

-LSB-

-RSB-

for

algorithmic

details

performance

analyses

and

empirical

results

for

these

communication

primitives

The

organization

of

this

paper

is

as

follows

Section

presents

our

computation

model

for

analyzing

parallel

algorithms

Section

describes

in

detail

our

improved

sample

sort

algorithm

Finally

Section

describes

our

data

sets

and

the

experimental

performance

of

our

sorting

algorithm

The

Parallel

Computation

Model

We

use

a

simple

model

to

analyze

the

performance

of

our

parallel

algorithms

Our

model

is

based

on

the

fact

that

current

hardware

platforms

can

be

viewed

as

a

collection

of

powerful

processors

connected

by

a

communication

network

that

can

be

modeled

as

a

complete

graph

on

which

communication

is

subject

to

the

restrictions

imposed

by

the

latency

and

the

bandwidth

properties

of

the

network

We

view

a

parallel

algorithm

as

a

sequence

of

local

computations

interleaved

with

communication

steps

where

we

allow

computation

and

communication

to

overlap

We

account

for

communication

costs

as

follows

Assuming

no

congestion

the

transfer

of

a

block

consisting

of

m

contiguous

words

between

two

processors

takes

-LRB-

o

oem

-RRB-

time

where

o

is

the

latency

of

the

network

and

oe

is

the

time

per

word

at

which

a

processor

can

inject

or

receive

data

from

the

network

Note

that

the

bandwidth

per

processor

is

inversely

proportional

to

oe

We

assume

that

the

bisection

bandwidth

is

sufficiently

high

to

support

block

permutation

routing

amongst

the

p

processors

at

the

rate

of

In

particular

for

any

subset

of

q

processors

a

block

permutation

amongst

the

q

processors

takes

-LRB-

o

oem

-RRB-

time

where

m

is

the

size

of

the

largest

block

Using

this

cost

model

we

can

evaluate

the

communication

time

Tcomm

-LRB-

n

p

-RRB-

of

an

algorithm

as

a

function

of

the

input

size

n

the

number

of

processors

p

and

the

parameters

o

and

oe

The

coefficient

of

o

gives

the

total

number

of

times

collective

communication

primitives

are

used

and

the

coefficient

of

oe

gives

the

maximum

total

amount

of

data

exchanged

between

a

processor

and

the

remaining

processors

This

communication

model

is

close

to

a

number

of

similar

models

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

that

have

recently

appeared

in

the

literature

and

seems

to

be

well-suited

for

designing

parallel

algorithms

on

current

high

performance

platforms

We

define

the

computation

time

Tcomp

as

the

maximum

time

it

takes

a

processor

to

perform

all

the

local

computation

steps

In

general

the

overall

performance

Tcomp

Tcomm

involves

a

tradeoff

between

Tcomp

and

Tcomm

In

many

cases

it

is

possible

to

minimize

both

Tcomp

and

Tcomm

simultaneously

and

sorting

is

such

a

case

A

New

Sample

Sort

Algorithm

Consider

the

problem

of

sorting

n

elements

equally

distributed

amongst

p

processors

where

we

assume

without

loss

of

generality

that

p

divides

n

evenly

The

idea

behind

sample

sort

is

to

find

a

set

of

p

splitters

to

partition

the

n

input

elements

into

p

groups

indexed

from

up

to

p

such

that

every

element

in

the

ith

group

is

less

than

or

equal

to

each

of

the

elements

in

the

-LRB-

i

-RRB-

th

group

for

i

p

Then

the

task

of

sorting

each

of

the

p

groups

can

be

turned

over

to

the

correspondingly

indexed

processor

after

which

the

n

elements

will

be

arranged

in

sorted

order

The

efficiency

of

this

algorithm

obviously

depends

on

how

evenly

we

divide

the

input

and

this

in

turn

depends

on

how

well

we

choose

the

splitters

One

way

to

choose

the

splitters

is

by

randomly

sampling

the

input

elements

at

each

processor

hence

the

name

sample

sort

Previous

versions

of

sample

sort

-LSB-

-RSB-

have

randomly

chosen

s

samples

from

the

np

elements

at

each

processor

routed

these

ps

samples

to

a

single

processor

sorted

them

at

that

processor

and

then

selected

every

sth

element

as

a

splitter

Each

processor

Pi

then

performs

a

binary

search

on

these

splitters

for

each

of

its

input

values

and

then

uses

the

results

to

route

the

values

to

the

appropriate

destination

after

which

local

sorting

is

done

to

complete

the

sorting

process

The

first

difficulty

with

this

approach

is

the

work

involved

in

gathering

and

sorting

the

samples

A

larger

value

of

s

results

in

better

load

balancing

but

it

also

increases

the

overhead

The

second

difficulty

is

that

no

matter

how

the

routing

is

scheduled

there

exist

inputs

that

give

rise

to

large

variations

in

the

number

of

elements

destined

for

different

processors

and

this

in

turn

results

in

an

inefficient

use

of

the

communication

bandwidth

Moreover

such

an

irregular

communication

scheme

can

not

take

advantage

of

the

regular

communication

primitives

proposed

under

the

MPI

standard

-LSB-

-RSB-

The

final

difficulty

with

the

original

approach

is

that

duplicate

values

are

accommodated

by

tagging

each

item

with

a

unique

value

-LSB-

-RSB-

This

of

course

doubles

the

cost

of

both

memory

access

and

interprocessor

communication

In

our

solution

we

incur

no

overhead

in

obtaining

samples

from

each

processor

and

in

sorting

these

samples

to

identify

the

splitters

Because

of

this

very

high

oversampling

we

are

able

to

replace

the

irregular

routing

with

exactly

two

calls

to

our

transpose

primitive

and

in

addition

we

are

able

to

efficiently

accommodate

the

presence

of

duplicates

without

resorting

to

tagging

The

pseudocode

for

our

algorithm

is

as

follows

ffl

Step

-LRB-

-RRB-

Each

processor

Pi

-LRB-

i

p

-RRB-

randomly

assigns

each

of

its

np

elements

to

one

of

p

buckets

With

high

probability

no

bucket

will

receive

more

than

elements

where

is

a

constant

to

be

defined

later

ffl

Step

-LRB-

-RRB-

Each

processor

Pi

routes

the

contents

of

bucket

j

to

processor

Pj

for

-LRB-

i

j

p

-RRB-

Since

with

high

probability

no

bucket

will

receive

more

than

elements

this

is

equivalent

to

performing

a

transpose

operation

with

block

size

ffl

Step

-LRB-

-RRB-

Each

processor

Pi

sorts

at

most

np

np

values

received

in

Step

-LRB-

-RRB-

using

an

appropriate

sequential

sorting

algorithm

For

integers

we

use

the

radix

sort

algorithm

whereas

for

floating

point

numbers

we

use

the

merge

sort

algorithm

ffl

Step

-LRB-

-RRB-

From

its

sorted

list

of

np

np

elements

processor

selects

each

j

th

element

as

Splitter

-LSB-

j

-RSB-

for

-LRB-

j

p

-RRB-

By

default

Splitter

-LSB-

p

-RSB-

is

the

largest

value

allowed

by

the

data

type

used

Additionally

for

each

Splitter

-LSB-

j

-RSB-

binary

search

is

used

to

determine

the

values

FracL

-LSB-

j

-RSB-

and

FracR

-LSB-

j

-RSB-

which

are

respectively

the

fractions

of

the

total

number

of

elements

at

processor

with

the

same

value

as

Splitter

-LSB-

j

-RSB-

and

Splitter

-LSB-

j

-RSB-

which

also

lie

between

index

-LRB-

j

-RRB-

and

index

j

inclusively

ffl

Step

-LRB-

-RRB-

Processor

broadcasts

the

Splitter

FracL

and

FracR

arrays

to

the

other

p

processors

ffl

Step

-LRB-

-RRB-

Each

processor

Pi

uses

binary

search

on

its

sorted

local

array

to

define

for

each

of

the

p

splitters

a

subsequence

Sj

The

subsequence

associated

with

Splitter

-LSB-

j

-RSB-

contains

all

those

values

which

are

greater

than

Splitter

-LSB-

j

-RSB-

and

less

than

Splitter

-LSB-

j

-RSB-

as

well

as

FracL

-LSB-

j

-RSB-

and

FracR

-LSB-

j

-RSB-

of

the

total

number

of

elements

in

the

local

array

with

the

same

value

as

Splitter

-LSB-

j

-RSB-

and

Splitter

-LSB-

j

-RSB-

respectively

ffl

Step

-LRB-

-RRB-

Each

processor

Pi

routes

the

subsequence

associated

with

Splitter

-LSB-

j

-RSB-

to

processor

Pj

for

-LRB-

i

j

p

-RRB-

Since

with

high

probability

no

sequence

will

contain

more

than

elements

where

is

a

constant

to

be

defined

later

this

is

equivalent

to

performing

a

transpose

operation

with

block

size

ffl

Step

-LRB-

-RRB-

Each

processor

Pi

merges

the

p

sorted

subsequences

received

in

Step

-LRB-

-RRB-

to

produce

the

ith

column

of

the

sorted

array

Note

that

with

high

probability

no

processor

has

received

more

than

np

elements

where

is

a

constant

to

be

defined

later

We

can

establish

the

complexity

of

this

algorithm

with

high

probability

that

is

with

probability

-LRB-

n?ffl

-RRB-

for

some

positive

constant

ffl

But

before

doing

this

we

need

to

establish

the

results

of

the

following

lemmas

Lemma

At

the

completion

of

Step

-LRB-

-RRB-

the

number

of

elements

in

each

bucket

is

at

most

with

high

probability

for

any

and

n

ln

n

Proof

The

probability

that

exactly

elements

are

placed

in

a

particular

bucket

in

Step

-LRB-

-RRB-

is

given

by

the

binomial

distribution

b

-LRB-

s

r

q

-RRB-

r

s

qs

-LRB-

q

-RRB-

r?s

-LRB-

-RRB-

where

s

r

np

and

q

Using

the

following

Chernoff

bound

-LSB-

-RSB-

for

estimating

the

tail

of

a

binomial

distribution

X

s

-LRB-

ffl

-RRB-

rq

b

-LRB-

s

r

q

-RRB-

e

-LRB-

-RRB-

the

probability

that

a

particular

bucket

will

contain

at

least

elements

can

be

bounded

by

e

-LRB-

-RRB-

-LRB-

-RRB-

Hence

the

probability

that

any

of

the

buckets

contains

at

least

elements

can

be

bounded

by

-LRB-

-RRB-

-LRB-

-RRB-

and

Lemma

follows

Lemma

At

the

completion

of

Step

-LRB-

-RRB-

the

total

number

of

elements

received

by

processor

which

comprise

the

set

of

samples

from

which

the

splitters

are

chosen

is

at

most

np

with

high

probability

for

any

and

n

ln

n

Proof

The

probability

that

processor

receives

exactly

np

elements

is

given

by

the

binomial

distribution

b

np

n

Using

the

Chernoff

bound

for

estimating

the

tail

of

a

binomial

distribution

the

probability

that

processor

receives

at

least

np

elements

can

be

bounded

by

e

-LRB-

-RRB-

and

Lemma

follows

Lemma

For

each

Splitter

-LSB-

j

-RSB-

where

-LRB-

j

p

-RRB-

let

SEj

and

SSj

be

respectively

the

sets

of

input

elements

and

samples

that

are

both

equal

in

value

to

Splitter

-LSB-

j

-RSB-

and

let

jSSj

j

j

Then

with

high

probability

no

SEj

will

contain

more

than

Mj

np

elements

where

Mj

-LRB-

j

-RRB-

j

-LRB-

-RRB-

Proof

The

set

of

input

elements

SEj

xjljg

can

have

more

than

Mj

np

members

only

if

j

or

less

members

are

selected

to

be

samples

from

the

set

xj

-LRB-

Mj

np

-RRB-

g

which

is

the

set

composed

of

the

first

Mj

np

members

in

SEj

However

since

each

element

of

is

independently

chosen

to

be

a

sample

with

probability

the

probability

of

this

event

occurring

is

given

by

X

s

j

np

b

s

Mj

n

p

p

-LRB-

-RRB-

Using

the

following

Chernoff

type

bound

-LSB-

-RSB-

for

estimating

the

head

of

a

binomial

distribution

X

s

fflrq

b

-LRB-

s

r

q

-RRB-

e

-LRB-

ffl

-RRB-

-LRB-

-RRB-

where

s

j

r

Mj

np

and

q

it

follows

that

the

probability

that

a

set

SEj

among

the

p

sets

of

input

elements

has

more

than

Mj

np

is

bounded

by

p

X

i

e

jMj

Mjn

-LRB-

-RRB-

Using

the

fact

that

n

ln

n

it

is

easy

to

show

that

the

above

sum

can

be

bounded

by

n?ffl

for

some

ffl

and

Mj

-LRB-

j

-RRB-

j

-LRB-

-RRB-

The

bound

of

Lemma

will

also

hold

if

we

include

the

subsets

of

elements

and

samples

whose

values

fall

strictly

between

two

consecutive

splitters

Lemma

At

the

completion

of

Step

-LRB-

-RRB-

the

number

of

elements

received

by

each

processor

is

at

most

np

with

high

probability

for

any

-LRB-

without

duplicates

-RRB-

and

n

ln

n

Proof

Let

Q

be

the

set

of

input

elements

to

be

sorted

by

our

algorithm

let

R

be

the

set

of

samples

of

Step

-LRB-

-RRB-

at

processor

with

cardinality

np

and

and

let

S

be

the

subset

of

R

associated

with

Splitter

-LSB-

j

-RSB-

which

we

define

to

be

the

samples

in

R

with

indices

-LRB-

j

-RRB-

through

j

inclusively

Let

np

and

be

respectively

the

number

of

elements

in

Q

R

and

S

with

value

equal

to

Splitter

-LSB-

j

-RSB-

let

and

be

respectively

the

number

of

elements

in

Q

R

and

S

with

values

greater

than

Splitter

-LSB-

j

-RSB-

but

less

than

Splitter

-LSB-

j

-RSB-

and

let

np

and

be

respectively

the

number

of

elements

in

Q

R

and

S

with

value

equal

to

Splitter

-LSB-

j

-RSB-

According

to

Step

-LRB-

-RRB-

of

our

algorithm

processor

Pj

will

receive

FracL

-LSB-

j

-RSB-

FracR

-LSB-

j

-RSB-

n

p

n

p

-LRB-

-RRB-

elements

To

compute

the

upper

bound

np

on

this

expression

we

first

use

Lemma

to

bound

each

Qi

np

giving

us

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

n

p

-LRB-

-RRB-

Rearranging

this

expression

we

get

s

-LRB-

-RRB-

s

n

p

-LRB-

-RRB-

Clearly

this

expression

is

maximized

for

and

Substituting

these

values

and

rearranging

once

again

we

get

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

n

p

-LRB-

-RRB-

Since

this

expression

is

maximized

for

Since

Lemma

guarantees

that

with

high

probability

Lemma

follows

with

high

probability

for

Alternatively

if

there

are

no

duplicates

we

can

show

that

the

bound

follows

with

high

probability

for

Lemma

If

the

set

of

input

elements

is

arbitrarily

partitioned

into

at

most

subsets

each

of

size

Xi

np

-LRB-

i

-RRB-

with

high

probability

at

the

conclusion

of

Step

-LRB-

-RRB-

no

processor

will

receive

more

than

Yi

elements

from

any

particular

subset

for

Yi

-LRB-

Xi

pXi

-RRB-

and

n

ln

n

Proof

The

probability

that

exactly

Yi

elements

are

sent

to

a

particular

processor

by

the

conclusion

of

Step

-LRB-

-RRB-

is

given

by

the

binomial

distribution

b

-LRB-

Yi

Xinp

-RRB-

Using

the

Chernoff

bound

for

estimating

the

tail

of

a

binomial

distribution

the

probability

that

from

M

possible

subsets

any

processor

will

receive

at

least

Yi

elements

can

be

bounded

by

M

X

i

pe

YiXi

Xin

-LRB-

-RRB-

and

Lemma

follows

for

M

Lemma

The

number

of

elements

exchanged

by

any

two

processors

in

Step

-LRB-

-RRB-

is

at

most

with

high

probability

for

any

-LRB-

without

duplicates

-RRB-

and

n

ln

n

Proof

Let

U

be

the

set

of

input

elements

to

be

sorted

by

our

algorithm

let

V

be

the

set

of

elements

held

by

intermediate

processor

Pi

after

Step

-LRB-

-RRB-

and

let

W

be

the

set

of

elements

held

by

destination

processor

Pj

after

Step

-LRB-

-RRB-

Let

np

and

np

be

respectively

the

number

of

elements

in

U

V

and

W

with

values

equal

to

Splitter

-LSB-

j

-RSB-

let

np

and

np

be

respectively

the

number

of

elements

in

U

V

and

W

with

values

greater

than

Splitter

-LSB-

j

-RSB-

but

less

than

Splitter

-LSB-

j

-RSB-

and

let

np

and

np

be

respectively

the

number

of

elements

in

U

V

and

W

with

values

equal

to

Splitter

-LSB-

j

-RSB-

According

to

Step

-LRB-

-RRB-

of

our

algorithm

intermediate

processor

Pi

will

send

FracL

-LSB-

j

-RSB-

FracR

-LSB-

j

-RSB-

n

-LRB-

-RRB-

elements

to

processor

Pj

To

compute

the

upper

bound

on

this

expression

we

first

use

Lemma

to

bound

each

Vk

giving

us

FracL

-LSB-

j

-RSB-

FracR

-LSB-

j

-RSB-

n

-LRB-

-RRB-

Notice

that

since

destination

processor

Pj

receives

respectively

FracL

-LSB-

j

-RSB-

and

FracR

-LSB-

j

-RSB-

of

the

elements

at

each

intermediate

processor

with

values

equal

to

Splitter

-LSB-

j

-RSB-

and

Splitter

-LSB-

j

-RSB-

it

follows

that

FracL

-LSB-

j

-RSB-

and

FracR

-LSB-

j

-RSB-

Hence

we

can

rewrite

the

expression

above

as

n

-LRB-

-RRB-

Rearranging

this

expression

we

get

s

s

n

-LRB-

-RRB-

Clearly

this

expression

is

maximized

for

and

Substituting

these

values

and

rearranging

we

get

n

-LRB-

-RRB-

Since

this

expression

is

maximized

for

Since

Lemma

guarantees

that

with

high

probability

Lemma

follows

with

high

probability

for

Alternatively

if

there

are

no

duplicates

we

can

show

that

the

bound

follows

with

high

probability

for

With

these

bounds

on

the

values

of

and

the

analysis

of

our

sample

sort

algorithm

is

as

follows

Steps

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

involve

no

communication

and

are

dominated

by

the

cost

of

the

sequential

sorting

in

Step

-LRB-

-RRB-

and

the

merging

in

Step

-LRB-

-RRB-

Sorting

integers

using

radix

sort

requires

O

np

time

whereas

sorting

floating

point

numbers

using

merge

sort

requires

O

np

log

np

time

Step

-LRB-

-RRB-

requires

O

np

log

p

time

if

we

merge

the

sorted

subsequences

in

a

binary

tree

fashion

Steps

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

call

the

communication

primitives

transpose

bcast

and

transpose

respectively

The

analysis

of

these

primitives

in

-LSB-

-RSB-

shows

that

with

high

probability

these

three

steps

require

Tcomm

-LRB-

n

p

-RRB-

o

-LRB-

p

-RRB-

oe

Tcomm

-LRB-

n

p

-RRB-

-LRB-

o

-LRB-

p

-RRB-

oe

-RRB-

and

Tcomm

-LRB-

n

p

-RRB-

o

-LRB-

p

-RRB-

oe

respectively

Hence

with

high

probability

the

overall

complexity

of

our

sample

sort

algorithm

is

given

-LRB-

for

floating

point

numbers

-RRB-

by

T

-LRB-

n

p

-RRB-

Tcomp

-LRB-

n

p

-RRB-

Tcomm

-LRB-

n

p

-RRB-

O

n

p

log

n

o

n

p

oe

-LRB-

-RRB-

for

n

ln

n

Clearly

our

algorithm

is

asymptotically

optimal

with

very

small

coefficients

But

a

theoretical

comparison

of

our

running

time

with

previous

sorting

algorithms

is

difficult

since

there

is

no

consensus

on

how

to

model

the

cost

of

the

irregular

communication

used

by

the

most

efficient

algorithms

Hence

it

is

very

important

to

perform

an

empirical

evaluation

of

an

algorithm

using

a

wide

variety

of

benchmarks

as

we

will

do

next

Performance

Evaluation

Our

sample

sort

algorithm

was

implemented

using

Split-C

-LSB-

-RSB-

and

run

on

a

variety

of

machines

and

processors

including

the

Cray

Research

the

IBM

and

the

Thinking

Machines

For

every

platform

we

tested

our

code

on

eight

different

benchmarks

each

of

which

had

both

a

integer

version

-LRB-

on

the

Cray

-RRB-

and

a

double

precision

floating

point

number

-LRB-

double

-RRB-

version

Sorting

Benchmarks

Our

eight

sorting

benchmarks

are

defined

as

follows

in

which

n

and

p

are

assumed

for

simplicity

to

be

powers

of

two

and

MAXD

the

maximum

value

allowed

for

doubles

is

approximately

Uniform

-LSB-

U

-RSB-

a

uniformly

distributed

random

input

obtained

by

calling

the

C

library

random

number

generator

random

-LRB-

-RRB-

This

function

which

returns

integers

in

the

range

to

is

seeded

by

each

processor

Pi

with

the

value

-LRB-

i

-RRB-

For

the

double

data

type

we

normalize

the

integer

benchmark

values

by

first

subtracting

the

value

and

then

scaling

the

result

by

MAXD

Gaussian

-LSB-

G

-RSB-

a

Gaussian

distributed

random

input

approximated

by

adding

four

calls

to

random

-LRB-

-RRB-

and

then

dividing

the

result

by

four

For

the

double

data

type

we

normalize

the

integer

benchmark

values

in

the

manner

described

for

-LSB-

U

-RSB-

Zero

-LSB-

Z

-RSB-

a

zero

entropy

input

created

by

setting

every

value

to

a

constant

such

as

zero

Bucket

Sorted

-LSB-

B

-RSB-

an

input

that

is

sorted

into

p

buckets

obtained

by

setting

the

first

elements

at

each

processor

to

be

random

numbers

between

and

p

the

second

elements

at

each

processor

to

be

random

numbers

between

p

and

p

and

so

forth

For

the

double

data

type

we

normalize

the

integer

benchmark

values

in

the

manner

described

for

-LSB-

U

-RSB-

g-Group

-LSB-

g-G

-RSB-

an

input

created

by

first

dividing

the

processors

into

groups

of

consecutive

processors

of

size

g

where

g

can

be

any

integer

which

partitions

p

evenly

If

we

index

these

groups

in

consecutive

order

from

up

to

pg

then

for

group

j

we

set

the

first

npg

elements

to

be

random

numbers

between

-LRB-

j

-RRB-

g

mod

p

p

and

-LRB-

j

-RRB-

g

mod

p

p

the

second

npg

elements

at

each

processor

to

be

random

numbers

between

-LRB-

j

-RRB-

g

mod

p

p

and

-LRB-

j

-RRB-

g

mod

p

p

and

so

forth

For

the

double

data

type

we

normalize

the

integer

benchmark

values

in

the

manner

described

for

-LSB-

U

-RSB-

Staggered

-LSB-

S

-RSB-

created

as

follows

if

the

processor

index

i

is

less

than

or

equal

to

then

we

set

all

np

elements

at

that

processor

to

be

random

numbers

between

-LRB-

-RRB-

p

and

-LRB-

-RRB-

p

Otherwise

we

set

all

np

elements

to

be

random

numbers

between

-LRB-

p

-RRB-

p

and

-LRB-

p

-RRB-

p

For

the

double

data

type

we

normalize

the

integer

benchmark

values

in

the

manner

described

for

-LSB-

U

-RSB-

Deterministic

Duplicates

-LSB-

DD

-RSB-

an

input

of

duplicates

in

which

we

set

all

np

elements

at

each

of

the

first

processors

to

be

log

n

all

np

elements

at

each

of

the

next

processors

to

be

log

and

so

forth

At

processor

Pp

we

set

the

first

elements

to

be

log

np

the

next

elements

to

be

log

and

so

forth

Randomized

Duplicates

-LSB-

RD

-RSB-

an

input

of

duplicates

in

which

each

processor

fills

an

array

T

with

some

constant

number

range

-LRB-

range

is

for

our

work

-RRB-

of

random

values

between

and

-LRB-

range

-RRB-

whose

sum

is

S

The

first

T

-LSB-

-RSB-

S

np

values

of

the

input

are

then

set

to

a

random

value

between

and

-LRB-

range

-RRB-

the

next

T

-LSB-

-RSB-

S

np

values

of

the

input

are

then

set

to

another

random

value

between

and

-LRB-

range

-RRB-

and

so

forth

We

selected

these

eight

benchmarks

for

a

variety

of

reasons

Previous

researchers

have

used

the

Uniform

Gaussian

and

Zero

benchmarks

and

so

we

too

included

them

for

purposes

of

comparison

But

benchmarks

should

be

designed

to

illicit

the

worst

case

behavior

from

an

algorithm

and

in

this

sense

the

Uniform

benchmark

is

not

appropriate

For

example

for

n

AE

p

one

would

expect

that

the

optimal

choice

of

the

splitters

in

the

Uniform

benchmark

would

be

those

which

partition

the

range

of

possible

values

into

equal

intervals

Thus

algorithms

which

try

to

guess

the

splitters

might

perform

misleadingly

well

on

such

an

input

In

this

respect

the

Gaussian

benchmark

is

more

telling

But

we

also

wanted

to

find

benchmarks

which

would

evaluate

the

cost

of

irregular

communication

Thus

we

wanted

to

include

benchmarks

for

which

an

algorithm

which

uses

a

single

phase

of

routing

would

find

contention

difficult

or

even

impossible

to

avoid

A

naive

approach

to

rearranging

the

data

would

perform

poorly

on

the

Bucket

Sorted

benchmark

Here

every

processor

would

try

to

route

data

to

the

same

processor

at

the

same

time

resulting

in

poor

utilization

of

communication

bandwidth

This

problem

might

be

avoided

by

an

algorithm

in

which

at

each

processor

the

elements

are

first

grouped

by

destination

and

then

routed

according

to

the

specifications

of

a

sequence

of

p

destination

permutations

Perhaps

the

most

straightforward

way

to

do

this

is

by

iterating

over

the

possible

communication

strides

But

such

a

strategy

would

perform

poorly

with

the

g-Group

benchmark

for

a

suitably

chosen

value

of

g

In

this

case

using

stride

iteration

those

processors

which

belong

to

a

particular

group

all

route

data

to

the

same

subset

of

g

destination

processors

This

subset

of

destinations

is

selected

so

that

when

the

g

processors

route

to

this

subset

they

choose

the

processors

in

exactly

the

same

order

producing

contention

and

possibly

stalling

Alternatively

one

can

synchronize

the

processors

after

each

permutation

but

this

in

turn

will

reduce

the

communication

bandwidth

by

a

factor

of

pg

In

the

worst

case

scenario

each

processor

needs

to

send

data

to

a

single

processor

a

unique

stride

away

This

is

the

case

of

the

Staggered

benchmark

and

the

result

is

a

reduction

of

the

communication

bandwidth

by

a

factor

of

p.

Of

course

one

can

correctly

object

that

both

the

g-Group

benchmark

and

the

Staggered

benchmark

have

been

tailored

to

thwart

a

routing

scheme

which

iterates

over

the

possible

strides

and

that

another

sequences

of

permutations

might

be

found

which

performs

better

This

is

possible

but

at

the

same

time

we

are

unaware

of

any

single

phase

deterministic

algorithm

which

could

avoid

an

equivalent

challenge

Finally

the

Deterministic

Duplicates

and

the

Randomized

Duplicates

benchmarks

were

included

to

assess

the

performance

of

the

algorithms

in

the

presence

of

duplicate

values

Experimental

Results

For

each

experiment

the

input

is

evenly

distributed

amongst

the

processors

The

output

consists

of

the

elements

in

non-descending

order

arranged

amongst

the

processors

so

that

the

elements

at

each

processor

are

in

sorted

order

and

no

element

at

processor

Pi

is

greater

than

any

element

at

processor

Pj

for

all

i

j.

Two

variations

were

allowed

in

our

experiments

First

radix

sort

was

used

to

sequentially

sort

integers

whereas

merge

sort

was

used

to

sort

double

precision

floating

point

numbers

-LRB-

doubles

-RRB-

Second

different

implementations

of

the

communication

primitives

were

allowed

for

each

machine

Wherever

possible

we

tried

to

use

the

vendor

supplied

implementations

In

fact

IBM

does

provide

all

of

our

communication

primitives

as

part

of

its

machine

specific

Collective

Communication

Library

-LRB-

CCL

-RRB-

-LSB-

-RSB-

and

MPI

As

one

might

expect

they

were

faster

than

the

high

level

Split-C

implementation

Size

-LSB-

U

-RSB-

-LSB-

G

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

B

-RSB-

-LSB-

S

-RSB-

-LSB-

Z

-RSB-

-LSB-

DD

-RSB-

-LSB-

RD

-RSB-

Table

I

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

a

variety

of

integer

benchmarks

on

a

Cray

Size

-LSB-

U

-RSB-

-LSB-

G

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

B

-RSB-

-LSB-

S

-RSB-

-LSB-

Z

-RSB-

-LSB-

DD

-RSB-

-LSB-

RD

-RSB-

Table

II

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

a

variety

of

integer

benchmarks

on

a

IBM

Size

-LSB-

U

-RSB-

-LSB-

G

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

B

-RSB-

-LSB-

S

-RSB-

-LSB-

Z

-RSB-

-LSB-

DD

-RSB-

-LSB-

RD

-RSB-

Table

III

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

a

variety

of

double

benchmarks

on

a

Cray

Tables

I

II

III

and

IV

display

the

performance

of

our

sample

sort

as

a

function

of

input

distribution

for

a

variety

of

input

sizes

In

each

case

the

performance

is

essentially

independent

of

the

input

distribution

These

tables

present

results

obtained

on

a

node

Cray

and

a

node

IBM

results

obtained

from

the

TMC

validate

this

claim

as

well

Because

of

this

independence

the

remainder

of

this

section

will

only

discuss

the

performance

of

our

sample

sort

on

the

single

benchmark

-LSB-

U

-RSB-

The

results

in

Tables

V

and

VI

together

with

their

graphs

in

Figure

examine

the

scalability

of

our

sample

sort

as

a

function

of

machine

size

Results

are

shown

for

the

the

and

the

Bearing

in

mind

that

these

graphs

are

log-log

plots

they

show

that

for

a

fixed

input

size

n

the

execution

time

scales

almost

inversely

with

the

number

of

processors

p

While

this

is

certainly

the

expectation

of

our

analytical

model

for

doubles

it

might

at

first

appear

to

exceed

our

prediction

of

an

O

np

log

p

computational

complexity

for

integers

However

the

appearance

of

an

inverse

relationship

is

still

quite

reasonable

when

we

note

that

for

values

of

p

between

and

log

p

varies

Size

-LSB-

U

-RSB-

-LSB-

G

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

B

-RSB-

-LSB-

S

-RSB-

-LSB-

Z

-RSB-

-LSB-

DD

-RSB-

-LSB-

RD

-RSB-

Table

IV

Total

execution

time

-LRB-

in

seconds

-RRB-

for

required

to

sort

a

variety

of

double

benchmarks

on

a

IBM

Sample

Sorting

of

Integers

-LSB-

U

-RSB-

Number

of

Processors

Machine

CRAY

IBM

TMC

Table

V

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

integers

on

a

variety

of

machines

and

processors

using

the

-LSB-

U

-RSB-

benchmark

A

hyphen

indicates

that

particular

platform

was

unavailable

to

us

Sample

Sorting

of

Doubles

-LSB-

U

-RSB-

Number

of

Processors

Machine

CRAY

IBM

TMC

Table

VI

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

doubles

on

a

variety

of

machines

and

processors

using

the

-LSB-

U

-RSB-

benchmark

A

hyphen

indicates

that

particular

platform

was

unavailable

to

us

by

only

a

factor

of

Moreover

this

O

np

log

p

complexity

is

entirely

due

to

the

merging

in

Step

-LRB-

-RRB-

and

in

practice

Step

-LRB-

-RRB-

never

accounts

for

more

than

of

the

observed

execution

time

Note

that

the

complexity

of

Step

could

be

reduced

to

O

-LRB-

np

-RRB-

for

integers

using

radix

sort

but

the

resulting

execution

time

would

in

most

cases

be

slower

The

graphs

in

Figure

examine

the

scalability

of

our

sample

sort

as

a

function

of

problem

size

for

differing

numbers

of

processors

They

show

that

for

a

fixed

number

of

processors

there

is

an

almost

linear

dependence

between

the

execution

time

and

the

total

number

of

elements

n

While

this

is

certainly

the

expectation

of

our

analytic

model

for

integers

it

might

at

first

appear

to

exceed

our

prediction

of

a

O

np

log

n

computational

complexity

for

floating

point

values

However

this

appearance

of

a

linear

relationship

is

still

quite

reasonable

when

we

consider

that

for

the

range

of

values

shown

log

n

differs

by

only

a

factor

of

Next

the

graphs

in

Figure

examine

the

relative

costs

of

the

eight

steps

in

our

sample

sort

Results

are

shown

for

both

a

node

and

a

node

using

both

the

integer

and

the

double

versions

of

the

-LSB-

U

-RSB-

benchmark

Notice

that

for

n

integers

the

sequential

sorting

and

merging

performed

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

consume

approximately

of

the

execution

time

on

the

and

approximately

of

the

execution

time

on

the

By

contrast

the

two

transpose

operations

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

together

consume

only

about

of

the

execution

time

on

the

and

about

of

the

execution

time

on

the

The

difference

in

the

distribution

between

these

two

platforms

is

likely

due

in

part

to

the

fact

that

an

integer

is

bits

on

the

while

only

bits

on

the

By

contrast

doubles

are

bits

on

both

platforms

For

n

doubles

the

sequential

sorting

Figure

Scalability

of

sorting

integers

and

doubles

with

respect

to

machine

size

and

merging

performed

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

consume

approximately

of

the

execution

time

on

both

platforms

whereas

the

two

transpose

operations

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

together

consume

only

about

of

the

execution

time

Together

these

results

show

that

our

algorithm

is

extremely

efficient

in

its

communication

performance

Finally

Tables

VII

and

VIII

show

the

experimentally

derived

expected

value

-LRB-

E

-RRB-

and

sample

standard

deviation

-LRB-

STD

-RRB-

of

the

coefficients

and

used

to

describe

the

complexity

of

our

algorithm

in

Section

The

values

in

Table

VII

were

obtained

by

analyzing

data

collected

while

sorting

each

of

the

duplicate

benchmarks

-LSB-

DD

-RSB-

and

-LSB-

RD

-RSB-

times

on

a

Cray

For

each

trial

the

values

recorded

were

the

largest

occurrence

of

each

coefficient

at

any

of

the

processors

By

contrast

the

values

in

Table

VIII

were

obtained

by

analyzing

data

collected

while

sorting

each

of

the

unique

benchmarks

-LSB-

G

-RSB-

-LSB-

B

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

and

-LSB-

S

-RSB-

times

In

every

trial

a

different

seed

was

used

for

the

random

number

generator

both

to

generate

the

benchmark

where

appropriate

and

to

distribute

the

keys

into

bins

as

part

of

Step

-LRB-

-RRB-

The

experimentally

derived

expected

values

in

Table

VII

for

and

agree

strongly

with

the

theoretically

derived

bounds

for

duplicate

keys

of

and

for

n

ln

n

Similarly

the

experimentally

derived

expected

values

in

Table

VIII

for

and

agree

strongly

with

the

theoretically

derived

bounds

for

unique

keys

of

and

for

n

ln

n

Note

that

expected

values

for

and

are

actually

less

for

duplicate

values

than

for

unique

values

which

is

the

opposite

of

what

we

might

expect

from

the

computed

bounds

This

might

simply

reflect

our

limited

choice

of

Figure

Scalability

of

sorting

integers

and

doubles

with

respect

to

the

problem

size

for

differing

numbers

of

processors

Figure

Distribution

of

execution

time

amongst

the

eight

steps

of

sample

sort

Times

are

obtained

for

both

a

node

and

a

node

using

both

the

integer

and

the

double

versions

of

the

-LSB-

U

-RSB-

benchmark

keys/proc

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

Table

VII

Statistical

evaluation

of

the

experimentally

observed

values

of

the

algorithm

coefficients

on

a

node

using

the

duplicate

benchmarks

keys/proc

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

E

-LRB-

-RRB-

STD

-LRB-

-RRB-

Table

VIII

Statistical

evaluation

of

the

experimentally

observed

values

of

the

algorithm

coefficients

on

a

node

using

the

unique

benchmarks

benchmarks

or

it

may

suggest

that

the

bounds

computed

for

duplicate

are

looser

than

those

computed

for

unique

values

Comparison

with

Previous

Results

Despite

the

enormous

theoretical

interest

in

parallel

sorting

we

were

able

to

locate

relatively

few

empirical

studies

Of

these

only

a

few

were

done

on

machines

which

either

were

available

to

us

for

comparison

or

involved

code

which

could

be

ported

to

these

machines

for

comparison

In

Tables

IX

and

X

we

compare

the

performance

of

our

sample

sort

algorithm

with

two

other

sample

sort

algorithms

In

all

cases

the

code

was

written

in

Split-C

In

the

case

of

Alexandrov

et

al.

-LSB-

-RSB-

the

times

were

determined

by

us

directly

on

a

node

using

code

supplied

by

the

authors

which

had

been

optimized

for

a

Meiko

In

the

case

of

Dusseau

-LSB-

-RSB-

the

times

were

obtained

from

the

graphed

results

reported

for

a

node

Finally

there

are

the

results

for

the

NAS

Parallel

Benchmark

-LSB-

-RSB-

for

Integer

Sorting

-LRB-

IS

-RRB-

The

name

of

this

benchmark

is

somewhat

misleading

Instead

of

requiring

that

the

integers

be

placed

in

sorted

order

as

we

do

the

benchmark

only

requires

that

they

be

ranked

without

any

reordering

which

is

a

significantly

simpler

task

Specifically

the

Class

A

Benchmark

requires

ten

repeated

rankings

of

a

Gaussian

distributed

random

input

consisting

of

integers

ranging

in

value

from

to

The

Class

B

Benchmark

is

similar

except

that

the

input

consists

of

integers

ranging

in

value

from

to

Table

XI

compares

our

results

on

these

two

variations

of

the

NAS

Benchmark

with

the

best

reported

times

for

the

the

and

the

We

believe

that

our

results

which

were

-LSB-

U

-RSB-

-LSB-

G

-RSB-

-LSB-

-RSB-

-LSB-

B

-RSB-

-LSB-

S

-RSB-

int

proc

HBJ

AIS

HBJ

AIS

HBJ

AIS

HBJ

AIS

HBJ

AIS

y

y

y

y

y

y

y

y

Table

IX

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

a

variety

of

benchmarks

and

problem

sizes

comparing

our

version

of

sample

sort

-LRB-

HBJ

-RRB-

with

that

of

Alexandrov

et

al.

-LRB-

AIS

-RRB-

on

a

yWe

were

unable

to

run

the

-LRB-

AIS

-RRB-

code

on

this

input

-LSB-

U

-RSB-

-LSB-

B

-RSB-

-LSB-

Z

-RSB-

int

proc

HBJ

DUS

HBJ

DUS

HBJ

DUS

Table

X

Time

required

per

element

-LRB-

in

microseconds

-RRB-

to

sample

sort

integers

comparing

our

results

-LRB-

HBJ

-RRB-

with

those

obtained

from

the

graphed

results

reported

by

Dusseau

-LRB-

DUS

-RRB-

on

a

node

obtained

using

high-level

portable

code

compare

favorably

with

the

other

reported

times

which

were

obtained

by

the

vendors

using

machine-specific

implementations

and

perhaps

system

modifications

The

only

performance

studies

we

are

aware

of

on

similar

platforms

for

generalized

sorting

are

those

of

Tridgell

and

Brent

-LSB-

-RSB-

who

report

the

performance

of

their

algorithm

using

a

node

on

a

uniformly

distributed

random

input

of

signed

integers

as

described

in

Table

XII

Conclusion

In

this

paper

we

introduced

a

novel

variation

on

sample

sort

and

conducted

an

experimental

study

of

its

performance

on

a

number

of

platforms

using

widely

different

benchmarks

Our

results

illustrate

the

efficiency

and

scalability

of

our

algorithm

across

the

different

platforms

and

appear

to

improve

on

all

similar

results

known

to

the

authors

Our

results

also

compare

favorably

with

those

reported

for

the

simpler

ranking

problem

posed

by

the

NAS

Integer

Sorting

-LRB-

IS

-RRB-

Benchmark

We

have

also

studied

several

variations

on

our

algorithm

which

use

differing

strategies

to

ensure

that

every

bucket

in

Step

-LRB-

-RRB-

receives

an

equal

number

of

elements

The

results

obtained

for

these

variations

were

very

similar

to

those

reported

in

this

paper

On

no

platform

did

the

improvements

exceed

approximately

and

in

many

instances

they

actually

ran

more

slowly

We

believe

that

a

significant

improvement

of

our

algorithm

would

require

the

enhancement

of

the

sequential

sorting

and

merging

in

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

and

that

there

is

little

room

for

significant

improvement

in

either

the

Comparison

of

NAS

-LRB-

IS

-RRB-

Benchmark

Times

Class

A

Class

B

Number

Best

Our

Best

Our

Machine

of

Processors

Reported

Time

Time

Reported

Time

Time

Table

XI

Comparison

of

our

execution

time

-LRB-

in

seconds

-RRB-

with

the

best

reported

times

for

the

Class

A

and

Class

B

NAS

Parallel

Benchmark

for

integer

sorting

Note

that

while

we

actually

place

the

integers

in

sorted

order

the

benchmark

only

requires

that

they

be

ranked

without

actually

reordering

Problem

-LSB-

U

-RSB-

Size

HBJ

TB

Table

XII

Total

execution

time

-LRB-

in

seconds

-RRB-

required

to

sort

signed

integers

comparing

our

results

-LRB-

HBJ

-RRB-

with

those

of

Tridgell

and

Brent

-LRB-

TB

-RRB-

on

a

node

load

balance

or

the

communication

efficiency

Acknowledgements

We

would

like

to

thank

Ronald

Greenberg

of

UMCP

's

Electrical

Engineering

Department

for

his

valuable

comments

and

encouragement

We

would

also

like

to

thank

the

CASTLE/Split-C

group

at

The

University

of

California

Berkeley

especially

for

the

help

and

encouragement

from

David

Culler

Arvind

Krishnamurthy

and

Lok

Tin

Liu

We

acknowledge

the

use

of

the

UMIACS

IBM

which

was

provided

by

an

IBM

Shared

University

Research

award

and

an

NSF

Academic

Research

Infrastructure

Grant

No

Arvind

Krishnamurthy

provided

additional

help

with

his

port

of

Split-C

to

the

Cray

Research

-LSB-

-RSB-

The

Jet

Propulsion

Lab/Caltech

Cray

Supercomputer

used

in

this

investigation

was

provided

by

funding

from

the

NASA

Offices

of

Mission

to

Planet

Earth

Aeronautics

and

Space

Science

We

also

acknowledge

William

Carlson

and

Jesse

Draper

from

the

Center

for

Comput

ing

Science

-LRB-

formerly

Supercomputing

Research

Center

-RRB-

for

writing

the

parallel

compiler

AC

-LRB-

version

-RRB-

-LSB-

-RSB-

on

which

the

port

of

Split-C

has

been

based

We

also

thank

the

Numerical

Aerodynamic

Simulation

Systems

Division

of

NASA

's

Ames

Research

Center

for

use

of

their

IBM

This

work

also

utilized

the

at

National

Center

for

Supercomputing

Applications

University

of

Illinois

at

Urbana-Champaign

under

grant

number

Please

see

for

related

work

by

the

authors

All

the

code

used

in

this

paper

is

freely

available

to

interested

parties

from

our

anonymous

ftp

site

ftp

We

encourage

other

researchers

to

compare

with

our

results

on

similar

inputs

References

-LSB-

-RSB-

A.

Alexandrov

M.

Ionescu

K.

Schauser

and

C.

Scheiman

LogGP

Incorporating

Long

Messages

into

the

LogP

Model

One

step

closer

towards

a

realistic

model

for

parallel

computation

In

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

-LCB-

Santa

Barbara

CA

July

-LSB-

-RSB-

R.H.

Arpaci

D.E.

Culler

A.

Krishnamurthy

S.G.

Steinberg

and

K.

Yelick

Empirical

Evaluation

of

the

A

Compiler

Perspective

In

ACM

Press

editor

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

-LCB-

Santa

Margherita

Ligure

Italy

June

-LSB-

-RSB-

D.A.

Bader

D.R.

Helman

and

J.

J?aJ?a

Practical

Parallel

Algorithms

for

Personalized

Communication

and

Integer

Sorting

and

Technical

Report

UMIACS

and

Electrical

Engineering

University

of

Maryland

College

Park

MD

November

To

appear

in

ACM

Journal

of

Experimental

Algorithmics

-LSB-

-RSB-

D.A.

Bader

and

J.

J?aJ?a

Parallel

Algorithms

for

Image

Histogramming

and

Connected

Components

with

an

Experimental

Study

Technical

Report

and

UMIACS

and

Electrical

Engineering

University

of

Maryland

College

Park

MD

December

To

appear

in

Journal

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

D.A.

Bader

and

J.

J?aJ?a

Parallel

Algorithms

for

Image

Histogramming

and

Connected

Components

with

an

Experimental

Study

In

Fifth

ACM

SIGPLAN

Symposium

of

Principles

and

Practice

of

Parallel

Programming

pages

-LCB-

Santa

Barbara

CA

July

-LSB-

-RSB-

D.A.

Bader

and

J.

J?aJ?a

Practical

Parallel

Algorithms

for

Dynamic

Data

Redistribution

Median

Finding

and

Selection

Technical

Report

and

UMIACS

and

Electrical

Engineering

University

of

Maryland

College

Park

MD

July

Presented

at

the

International

Parallel

Processing

Symposium

pages

Honolulu

HI

April

-LSB-

-RSB-

D.

Bailey

E.

Barszcz

J.

Barton

D.

Browning

R.

Carter

L.

Dagum

R.

Fatoohi

S.

Fineberg

P.

Frederickson

T.

Lasinski

R.

Schreiber

H.

Simon

V.

Venkatakrishnan

and

S.

Weeratunga

The

NAS

Parallel

Benchmarks

Technical

Report

Numerical

Aerodynamic

Simulation

Facility

NASA

Ames

Research

Center

Moffett

Field

CA

March

-LSB-

-RSB-

V.

Bala

J.

Bruck

R.

Cypher

P.

Elustondo

A.

Ho

C.-T

Ho

S.

Kipnis

and

M.

Snir

CCL

A

Portable

and

Tunable

Collective

Communication

Library

for

Scalable

Parallel

Computers

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LCB-

-LSB-

-RSB-

K.

Batcher

Sorting

Networks

and

Their

Applications

In

Proceedings

of

the

AFIPS

Spring

Joint

Computer

Conference

pages

-LCB-

Reston

VA

-LSB-

-RSB-

G.E.

Blelloch

C.E.

Leiserson

B.M.

Maggs

C.G.

Plaxton

S.J.

Smith

and

M.

Zagha

A

Comparison

of

Sorting

Algorithms

for

the

Connection

Machine

In

Proceedings

of

the

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

-LCB-

July

-LSB-

-RSB-

W.W.

Carlson

and

J.M.

Draper

AC

for

the

Technical

Report

Supercomputing

Research

Center

Bowie

MD

February

-LSB-

-RSB-

H.

Chernoff

A

Measure

of

Asymptotic

Efficiency

for

Tests

of

a

Hypothesis

Based

on

the

Sum

of

Observations

Annals

of

Math

Stat.

-LCB-

-LSB-

-RSB-

Cray

Research

Inc.

SHMEM

Technical

Note

for

C

October

Revision

-LSB-

-RSB-

D.E.

Culler

A.

Dusseau

S.C.

Goldstein

A.

Krishnamurthy

S.

Lumetta

T.

von

Eicken

and

K.

Yelick

Parallel

Programming

in

Split-C

In

Proceedings

of

Supercomputing

pages

-LCB-

Portland

OR

November

-LSB-

-RSB-

D.E.

Culler

A.C.

Dusseau

R.P.

Martin

and

K.E.

Schauser

Fast

Parallel

Sorting

Under

LogP

From

Theory

to

Practice

In

Portability

and

Performance

for

Parallel

Processing

chapter

pages

-LCB-

John

Wiley

Sons

-LSB-

-RSB-

D.E.

Culler

R.M.

Karp

D.A.

Patterson

A.

Sahay

K.E.

Schauser

E.

Santos

R.

Subramonian

and

T.

von

Eicken

LogP

Towards

a

Realistic

Model

of

Parallel

Computation

In

Fourth

ACM

SIGPLAN

Symposium

on

Principles

and

Practice

of

Parallel

Programming

May

-LSB-

-RSB-

A.C.

Dusseau

Modeling

Parallel

Sorts

with

LogP

on

the

Technical

Report

UCB

CSD

Computer

Science

Division

University

of

California

Berkeley

-LSB-

-RSB-

T.

Hagerup

and

C.

R?ub

A

Guided

Tour

of

Chernoff

Bounds

Information

Processing

Letters

-LCB-

-LSB-

-RSB-

D.R.

Helman

D.A.

Bader

and

J.

J?aJ?a

Parallel

Algorithms

for

Personalized

Communication

and

Sorting

With

an

Experimental

Study

In

Proceedings

of

the

Eighth

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

-LCB-

Padua

Italy

June

-LSB-

-RSB-

W.L.

Hightower

J.F.

Prins

and

J.H.

Reif

Implementations

of

Randomized

Sorting

on

Large

Parallel

Machines

In

Proceedings

of

the

Symposium

on

Parallel

Architectures

and

Algorithms

pages

-LCB-

San

Diego

CA

July

-LSB-

-RSB-

J.S.

Huang

and

Y.C.

Chow

Parallel

Sorting

and

Data

Partitioning

by

Sampling

In

Proceedings

of

the

Computer

Software

and

Applications

Conference

pages

-LCB-

November

-LSB-

-RSB-

F.T.

Leighton

Tight

Bounds

on

the

Complexity

of

Parallel

Sorting

IEEE

Transactions

on

Computers

-LCB-

-LSB-

-RSB-

H.

Li

and

K.C.

Sevcik

Parallel

Sorting

by

Overpartitioning

Technical

Report

Computer

Systems

Research

Institute

University

of

Toronto

Canada

April

-LSB-

-RSB-

X.

Li

P.

Lu

J.

Schaeffer

J.

Shillington

P.S.

Wong

and

H.

Shi

On

the

Versatility

of

Parallel

Sorting

by

Regular

Sampling

Parallel

Computing

-LCB-

-LSB-

-RSB-

J.M.

Marberg

and

E.

Gafni

Sorting

in

Constant

Number

of

Row

and

Column

Phases

on

a

Mesh

Algorithmica

-LCB-

-LSB-

-RSB-

Message

Passing

Interface

Forum

MPI

A

Message-Passing

Interface

Standard

Technical

report

University

of

Tennessee

Knoxville

TN

June

Version

-LSB-

-RSB-

C.G.

Plaxton

Efficient

Computation

on

Sparse

Interconnection

Networks

Technical

Report

Department

of

Computer

Science

Stanford

University

Stanford

CA

September

-LSB-

-RSB-

M.J.

Quinn

Analysis

and

Benchmarking

of

Two

Parallel

Sorting

Algorithms

Hyperquicksort

and

Quickmerge

BIT

-LCB-

-LSB-

-RSB-

J.H.

Reif

and

L.G.

Valiant

A

Logarithmic

Time

Sort

for

Linear

Sized

Networks

Journal

of

the

ACM

-LCB-

-LSB-

-RSB-

S.

Saini

and

D.H.

Bailey

NAS

Parallel

Benchmarks

Results

Report

Numerical

Aerodynamic

Simulation

Facility

NASA

Ames

Research

Center

Moffett

Field

CA

December

-LSB-

-RSB-

H.

Shi

and

J.

Schaeffer

Parallel

Sorting

by

Regular

Sampling

Journal

of

Parallel

and

Distributed

Computing

-LCB-

-LSB-

-RSB-

A.

Tridgell

and

R.P.

Brent

An

Implementation

of

a

General-Purpose

Parallel

Sorting

Algorithm

Techical

Report

Computer

Sciences

Laboratory

Australian

National

University

Canberra

Australia

February

-LSB-

-RSB-

L.G.

Valiant

A

Bridging

Model

for

Parallel

Computation

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

S.C.

Woo

M.

Ohara

E.

Torrie

J.P.

Singh

and

A.

Gupta

The

Programs

Characterization

and

Methodological

Considerations

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

-LCB-

June

generalized sorting
integer sorting
parallel algorithms
sample sort