Executing

Multithreaded

Programs

Efficiently

by

Robert

D.

Blumofe

Sc.B.

Brown

University

-LRB-

-RRB-

S.M.

Massachusetts

Institute

of

Technology

-LRB-

-RRB-

Submitted

to

the

Department

of

Electrical

Engineering

and

Computer

Science

in

partial

fulfillment

of

the

requirements

for

the

degree

of

Doctor

of

Philosophy

at

the

MASSACHUSETTS

INSTITUTE

OF

TECHNOLOGY

September

c

Robert

D.

Blumofe

MCMXCV

All

rights

reserved

The

author

hereby

grants

to

MIT

permission

to

reproduce

and

distribute

publicly

paper

and

electronic

copies

of

this

thesis

document

in

whole

or

in

part

and

to

grant

others

the

right

to

do

so

Author

Department

of

Electrical

Engineering

and

Computer

Science

August

Certified

by

Charles

E.

Leiserson

Professor

Thesis

Supervisor

Accepted

by

Frederic

R.

Morgenthaler

Chairman

Departmental

Committee

on

Graduate

Students

Executing

Multithreaded

Programs

Efficiently

by

Robert

D.

Blumofe

Submitted

to

the

Department

of

Electrical

Engineering

and

Computer

Science

on

August

in

partial

fulfillment

of

the

requirements

for

the

degree

of

Doctor

of

Philosophy

Abstract

This

thesis

presents

the

theory

design

and

implementation

of

Cilk

-LRB-

pronounced

silk

-RRB-

and

Cilk-NOW

Cilk

is

a

C-based

language

and

portable

runtime

system

for

programming

and

executing

multithreaded

parallel

programs

Cilk-NOW

is

an

implementation

of

the

Cilk

runtime

system

that

transparently

manages

resources

for

parallel

programs

running

on

a

network

of

workstations

Cilk

is

built

around

a

provably

efficient

algorithm

for

scheduling

the

execution

of

fully

strict

multithreaded

computations

Based

on

the

technique

of

work

stealing

this

algorithm

achieves

time

space

and

communication

bounds

that

are

all

within

a

small

constant

factor

of

optimal

Using

these

performance

guarantees

Cilk

provides

a

simple

abstraction

of

performance

that

allows

programmers

to

predict

accurately

how

program

changes

will

affect

execution

time

Cilk-NOW

provides

high-level

resource

management

services

so

that

Cilk

programs

can

run

efficiently

on

a

network

of

workstations

These

services

include

adaptive

parallelism

and

fault

tolerance

With

adaptive

parallelism

programs

execute

on

a

pool

of

workstations

that

dynamically

grows

and

shrinks

in

response

to

both

the

availability

of

idle

machines

and

the

availability

of

parallelism

within

the

program

Cilk-NOW

also

provides

transparent

fault

tolerance

with

a

fully

distributed

checkpointing

mechanism

Thesis

Supervisor

Charles

E.

Leiserson

Title

Professor

Acknowledgments

I

remember

the

first

time

I

met

Charles

Leiserson

and

discussed

possible

research

projects

He

spoke

of

some

grand

ideas

and

he

waved

his

arms

a

lot

Something

about

trees

We

were

going

to

solve

some

important

problems

together

Only

one

difficulty

I

had

no

idea

what

he

was

talking

about

For

better

or

worse

I

now

seem

to

understand

him

and

our

collaboration

did

produce

some

results

that

I

am

very

proud

of

Charles

has

a

terrific

sense

of

research

direction

and

rather

than

push

me

in

any

direction

he

pulled

Rather

than

tell

me

where

to

go

he

showed

the

way

All

the

while

Charles

left

me

room

to

be

creative

and

pursue

paths

that

I

found

myself

He

supported

me

in

all

my

research

pursuits

with

his

own

enthusiasm

and

with

a

generous

allocation

of

resources

both

human

and

machine

Charles

also

has

a

great

sense

of

humor

and

we

had

fun

even

giggling

like

children

one

afternoon

as

we

imagined

how

much

more

fun

we

would

have

playing

the

recycling

game

-LRB-

see

Section

-RRB-

instead

of

hockey

Charles

tells

me

that

now

I

speak

in

grand

ideas

and

wave

my

arms

a

lot

Could

be

worse

Besides

serving

as

my

advisor

collaborator

and

friend

Charles

also

showed

me

how

to

clarify

my

writing

with

focused

work

and

patience

It

did

not

come

easy

After

one

par

ticularly

long

and

grueling

session

of

writing

-LRB-

Charles

has

infinite

patience

-RRB-

I

left

Charles

office

and

the

first

person

I

stumbled

upon

was

Tom

Cormen

With

a

sigh

I

asked

him

have

you

ever

written

a

paper

with

Charles

He

just

looked

at

me

like

that

is

the

stupi

dest

question

I

've

ever

asked

him

-LRB-

see

-LSB-

-RSB-

-RRB-

It

is

n't

Much

of

the

work

in

this

thesis

could

not

have

been

done

without

the

collaboration

of

the

entire

Cilk

team

Led

by

Charles

this

crackerjack

team

includes

or

has

included

Shail

Aditya

now

of

Hewlett

Packard

Matteo

Frigo

Michael

Halbherr

now

of

the

Boston

Consulting

Group

Chris

Joerg

Bradley

Kuszmaul

now

of

Yale

University

Phil

Lisiecki

The

research

reported

in

this

thesis

was

supported

in

part

by

the

Advanced

Research

Projects

Agency

-LRB-

ARPA

-RRB-

under

Grants

and

In

addition

I

was

generously

supported

throughout

my

graduate

career

by

a

National

Science

Foundation

-LRB-

NSF

-RRB-

Graduate

Student

Fellowship

and

an

ARPA

High-Performance

Computing

Graduate

Fellowship

administered

through

the

University

of

Maryland

Institute

for

Advanced

Computer

Study

iii

iv

Howard

Lu

Rob

Miller

now

of

Carnegie

Mellon

University

David

Park

Keith

Randall

Larry

Rudolph

Sivan

Toledo

now

of

IBM

and

Yuli

Zhou

now

of

AT&T

Bell

Laboratories

Many

people

at

MIT

have

made

significant

contributions

to

the

work

in

my

thesis

Much

of

the

combinatorial

technique

and

the

delay-sequence

argument

used

in

Chapter

I

learned

from

Tom

Leighton

Bonnie

Berger

helped

with

some

of

my

early

probabilistic

analysis

Esther

Jesurum

and

Ethan

Wolf

helped

with

the

lower

bound

Eric

Brewer

now

of

the

Uni

versity

of

California

at

Berkeley

was

my

collaborator

in

some

of

my

early

forays

into

sys

tem

building

Others

who

have

lent

inspiration

and

ideas

include

Arvind

Tom

Cormen

now

of

Dartmouth

College

Frans

Kaashoek

Arthur

Lent

Greg

Papadopoulos

now

of

Sun

and

Bill

Weihl

Scott

Blomquist

kept

the

and

our

network

of

workstations

up

and

running

David

Jones

provided

extensive

help

with

some

of

the

subtleties

of

LATEX

Others

who

have

helped

make

my

life

easy

-LRB-

and

fun

-RRB-

include

Be

Hubbard

Irena

Kveraga

and

Cheryl

Patton

Several

people

from

outside

MIT

also

made

significant

contributions

to

the

work

in

this

thesis

Bruce

Maggs

of

Carnegie

Mellon

University

outlined

the

delay

sequence

argument

and

Greg

Plaxton

of

the

University

of

Texas

at

Austin

gave

technical

comments

on

the

recy

cling

game

analysis

Others

who

contributed

ideas

include

Guy

Blelloch

of

Carnegie

Mel

lon

University

Warren

Burton

of

Simon

Fraser

University

and

Yanjun

Zhang

of

Southern

Methodist

University

I

would

not

have

made

it

to

MIT

from

Brown

-LRB-

where

I

was

an

undergraduate

-RRB-

had

it

not

been

for

some

of

the

inspiring

professors

in

Brown

's

math

and

computer

science

de

partments

Tom

Banchoff

first

turned

me

on

to

computer

graphics

Jeff

Vitter

now

of

Duke

University

first

turned

me

on

to

theory

and

algorithms

John

Savage

first

turned

me

on

to

parallel

computation

and

Andy

van

Dam

first

turned

me

on

to

my

future

in

computation

I

took

my

first

computer

science

class

from

Andy

and

I

took

my

first

adventure

into

research

in

Andy

's

graphics

group

From

this

beginning

Andy

infected

me

with

his

exuberant

sense

of

excitement

and

wonder

over

a

future

in

which

anything

can

happen

when

computation

meets

imagination

I

was

a

less

than

model

student

in

my

early

years

at

Brown

and

I

know

I

let

him

down

on

more

than

one

occasion

but

Andy

must

have

seen

something

that

nobody

else

could

because

he

continued

to

take

risks

with

me

presenting

me

with

one

opportunity

after

another

When

I

took

a

year

off

from

Brown

Andy

got

me

a

great

job

at

a

small

startup

company

called

Cadre

Technologies

and

later

it

was

Andy

who

first

suggested

that

I

apply

to

graduate

school

and

then

backed

up

the

suggestion

with

a

strong

letter

of

recommenda

v

tion

On

those

last

two

opportunities

I

think

I

finally

came

through

I

do

n't

know

why

Andy

took

such

a

personal

interest

in

me

or

why

he

was

so

unwaivering

in

his

confidence

but

I

think

about

him

often

and

with

gratitude

and

affection

My

years

to

date

have

been

filled

with

humor

adventure

satisfaction

and

fun

because

I

have

had

the

good

fortune

of

good

friends

some

of

whom

I

shall

now

mention

ordered

from

oldest

to

most

recent

Danny

Klein

Danny

was

my

summer

brother

and

best

friend

for

many

years

The

gang

out

at

Horton

Lakes

including

the

Horton

family?Jack

Gretchen

and

John?Greg

Cooper

and

many

others

who

have

come

and

gone

It

has

been

my

privi

lege

and

joy

to

waterski

with

these

folks

at

their

world-class

site

Max

Kennedy

I

've

never

met

anyone

with

more

energy

less

fear

or

a

bigger

heart

and

hey

man

Max

and

I

had

some

serious

fun

Jeff

Cutler

Jeff

and

I

drove

cross

country

four

times

discovered

the

precipice

watched

for

suicide

deer

went

to

Pete

Freehafer

's

wedding

as

the

Goldberg

brothers

and

had

countless

other

silly

adventures

I

ca

n't

even

write

this

without

laughing

Liz

Cohen

Liz

and

I

seem

to

have

an

unbreakable

bond

forged

through

some

strange

days

and

stranger

nights

Anna

Hedly

Everyone

who

knows

me

well

knows

how

important

Anna

was

and

is

to

me

We

rarely

talk

or

see

each

other

but

I

still

think

about

her

often

Also

Anna

got

me

started

as

a

pilot

Janet

Barber

Janet

is

probably

my

best

friend

and

the

person

I

most

en

joy

talking

to

and

hanging

around

with

Going

to

Dead

shows

and

living

large

with

Janet

in

Vegas

was

surely

the

best

possible

way

to

turn

thirty

The

gang

from

Motion

Dynamics

Slim

Dave

Mike

and

Leo

These

guys

fixed

my

little

race

car

whenever

I

wadded

it

up

in

a

ball

-LRB-

often

-RRB-

and

they

were

a

hell

of

a

lot

of

fun

to

hang

with

Former

and

present-day

Cadroids?Lou

Bershad

Reed

Flemming

Alan

Hecht

Lou

Mazzuchelli

Lou

Reynolds

and

others

They

were

and

still

are

a

talented

and

fun

group

of

people

to

work

and

play

with

The

entire

Bar-Yam

family?Miriam

Zvi

Yaneer

Aureet

and

Sageet

The

Bar-Yams

first

were

my

landlords

in

Cambridge

but

foremost

are

my

cherished

friends

Esther

Jesurum

Esther

got

me

started

playing

hockey

and

was

a

fun

waterskiing

partner

Helen

Greiner

Helen

was

always

up

for

some

fun

She

would

do

anything

except

wear

a

silly

hat

My

family

is

somewhat

distant

and

rather

peculiar

but

also

quite

wonderful

and

I

thank

them

for

their

support

and

tolerance

My

older

brother

Michael

was

my

role

model

growing

up

and

he

inspired

my

interest

in

math

and

science

Michael

is

now

an

emergency

room

doctor

and

saves

lives

every

day

-LRB-

On

a

good

day

I

can

improve

an

algorithm

-RRB-

My

older

sister

Maria

first

showed

me

the

adventurous

and

occasionally

dangerous

side

of

life

My

vi

younger

sister

Joanna

is

my

closest

relative

spatially

-LRB-

she

lives

nearby

in

the

Boston

area

-RRB-

temporally

-LRB-

we

are

one

year

and

six

days

apart

in

age

-RRB-

and

emotionally

Joanna

is

married

to

Brad

Meiseles

and

has

two

sons

Benjamin

and

Brandon

I

have

really

enjoyed

having

family

especially

Benjamin

and

Brandon

nearby

My

mother

gave

me

tremendous

con

fidence

and

high

expectations

and

these

qualities

have

brought

me

through

some

difficult

times

I

think

I

get

the

rational

side

of

my

character

from

her

My

father

came

to

parenthood

rather

late

in

life

but

I

do

n't

know

that

anyone

ever

worked

at

it

harder

or

enjoyed

it

more

I

think

I

get

the

emotional

side

of

my

character

from

him

I

hope

I

can

be

as

well

liked

and

well

respected

in

my

profession

as

he

seems

to

have

been

in

his

ROBERT

D.

BLUMOFE

Cambridge

Massachusetts

August

Contents

Executing

multithreaded

programs

efficiently

Scheduling

multithreaded

programs

Previous

results

and

related

work

Contributions

of

this

thesis

A

model

of

multithreaded

computation

Multithreaded

computation

Execution

time

Execution

space

Strict

multithreaded

computations

A

lower

bound

for

the

general

case

The

strictness

condition

The

busy-leaves

property

Work

stealing

A

randomized

work-stealing

algorithm

Atomic

messages

and

the

recycling

game

Analysis

of

the

work-stealing

algorithm

Parallel

programming

in

Cilk

The

Cilk

language

and

runtime

system

Spawns

and

syncs

Continuation-passing

threads

The

Cilk

work-stealing

scheduler

Performance

of

Cilk

applications

Cilk

applications

Application

performance

Performance

modeling

A

theoretical

analysis

of

the

Cilk

scheduler

Cilk

on

a

network

of

workstations

System

architecture

Adaptive

parallelism

Fault

tolerance

Conclusions

A

Proof

of

lower

bound

B

Cilk

project

status

Bibliography

List

of

Figures

Computing

Fibonacci

numbers

in

C

and

Cilk

Measured

Cilk

application

speedups

A

multithreaded

computation

A

execution

schedule

A

computation

dag

An

infinite-processor

execution

schedule

Doing

work

Making

progress

on

the

critical

path

An

activation

tree

An

activation

subtree

A

non-depth-first

computation

Linear

expansion

of

space

A

strict

multithreaded

computation

Passing

arguments

into

a

subcomputation

Strict

computations

are

depth-first

Depth-first

computations

can

be

strictified

A

busy-leaves

execution

schedule

The

structure

of

a

processor

's

ready

deque

The

ready

deque

before

and

after

a

spawn

The

ready

deque

before

and

after

a

death

The

Cilk

procedure

Fib

The

cilk

main

procedure

A

Cilk

spawn

tree

Cilk

threads

fib

and

sum

Successor

and

child

threads

Closures

and

continuations

A

closure

for

the

fib

thread

The

fib

thread

spawns

a

successor

The

fib

thread

spawns

two

children

A

Cilk

dag

The

scheduler

's

ready

pool

The

work

performed

in

ray

tracing

Performance

of

Cilk

applications

Measured

speedups

for

the

knary

benchmark

Measured

speedups

for

the

Socrates

chess

program

A

speedup

anomaly

A

Cilk

computation

Augmenting

the

Cilk

computation

dag

The

structure

of

a

processor

's

ready

pool

A

Cilk

job

starts

An

idle

machine

joins

a

Cilk

job

A

no-longer-idle

machine

leaves

a

Cilk

job

Assigning

a

closure

to

a

thief

subcomputation

Altering

the

continuations

in

a

stolen

closure

The

number

of

idle

machines

in

the

network

The

efficiency

of

adaptive

parallelism

A.

Constructing

a

computation

with

no

efficient

execution

schedule

A.

Scheduling

the

computation

's

chains

A.

Laying

out

the

computation

's

chains

into

threads

Chapter

Executing

multithreaded

programs

efficiently

In

writing

a

parallel

application

in

a

multithreaded

language

a

programmer

expresses

parallelism

by

coding

instructions

in

a

partial

execution

order

and

the

programmer

or

the

compiler

partitions

these

instructions

into

totally

ordered

sequences

called

threads

The

programmer

need

not

specify

which

processors

of

a

parallel

computer

execute

which

threads

nor

exactly

when

each

thread

should

be

executed

These

scheduling

decisions

are

made

automatically

by

the

runtime

system

's

scheduler

and

the

program

only

requires

that

each

instruction

of

each

thread

is

executed

by

some

processor

at

a

time

consistent

with

the

partial

order

Nevertheless

if

the

programmer

is

to

relinquish

control

over

these

scheduling

decisions

to

the

runtime

system

then

the

runtime

system

must

guarantee

that

it

will

make

good

scheduling

decisions

to

execute

the

program

efficiently

and

with

predictable

performance

In

this

thesis

we

develop

algorithmic

foundations

for

the

efficient

scheduling

of

multithreaded

programs

and

we

build

a

multithreaded

language

and

runtime

system

on

top

of

this

foundation

In

both

algorithmic

and

empirical

work

we

show

that

for

a

large

and

important

class

of

multithreaded

programs

a

runtime

system

can

deliver

efficient

and

predictable

performance

guaranteed

Scheduling

multithreaded

programs

A

multithreaded

language

provides

programmers

with

a

means

to

create

and

synchronize

multiple

computational

threads

and

the

runtime

system

for

such

a

language

automatically

schedules

the

execution

of

these

threads

on

the

processors

of

a

parallel

computer

To

execute

a

multithreaded

program

efficiently

the

runtime

system

's

scheduler

must

keep

the

processors

busy

doing

work

in

order

to

realize

parallel

speedup

and

simultaneously

it

must

maintain

memory

usage

to

within

reasonable

limits

and

avoid

interprocessor

communication

to

the

greatest

extent

possible

In

this

section

we

give

an

overview

of

the

Cilk

-LRB-

pronounced

silk

-RRB-

multithreaded

language

and

runtime

system

The

Cilk

runtime

system

's

scheduler

is

built

on

an

algorithmic

foundation

and

is

provably

efficient

with

respect

to

time

space

and

communication

We

shall

also

overview

Cilk-NOW

an

implementation

of

the

Cilk

runtime

system

for

networks

of

workstations

Cilk-NOW

leverages

properties

of

Cilk

's

scheduling

Chapter

Executing

multithreaded

programs

efficiently

algorithm

in

order

to

run

Cilk

programs

efficiently

in

the

dynamic

and

potentially

faulty

environment

of

a

network

of

workstations

Consider

a

program

that

uses

double

recursion

to

compute

the

Fibonacci

function

The

Fibonacci

function

fib

-LRB-

n

-RRB-

for

n

is

defined

as

fib

-LRB-

n

-RRB-

ae

n

if

n

fib

-LRB-

n

-RRB-

fib

-LRB-

n

-RRB-

otherwise

Figure

shows

how

this

function

is

written

in

C

and

in

Cilk

While

double

recursion

is

a

terrible

way

to

compute

Fibonacci

numbers

this

toy

example

does

illustrate

a

common

pattern

occurring

in

divide-and-conquer

applications

recursive

calls

solve

smaller

subcases

and

then

the

partial

results

are

merged

to

produce

the

final

result

Notice

that

other

than

the

int

Fib

-LRB-

int

n

-RRB-

f

if

-LRB-

n

-RRB-

return

n

else

f

int

x

y

x

Fib

-LRB-

-RRB-

y

Fib

-LRB-

-RRB-

return

-LRB-

x

y

-RRB-

g

g

-LRB-

a

-RRB-

C

function

cilk

int

Fib

-LRB-

int

n

-RRB-

f

if

-LRB-

n

-RRB-

return

n

else

f

int

x

y

x

spawn

Fib

-LRB-

-RRB-

y

spawn

Fib

-LRB-

-RRB-

sync

return

-LRB-

x

y

-RRB-

g

g

-LRB-

b

-RRB-

Cilk

procedure

Figure

A

C

function

and

a

Cilk

procedure

to

compute

the

nth

Fibonacci

number

cilk

keyword

which

identifies

a

Cilk

procedure

the

Cilk

version

differs

from

its

C

counterpart

only

in

its

use

of

spawn

and

sync

The

spawn

keywords

turn

the

recursive

calls

into

recursive

spawns

A

spawn

is

the

parallel

analogue

of

a

procedure

call

The

instructions

executed

by

Fib

form

a

thread

and

a

spawn

creates

a

new

child

thread

that

may

execute

concurrently

with

its

parent

Thus

when

Fib

performs

the

two

recursive

spawns

the

spawned

child

threads

may

execute

concurrently

After

performing

the

spawns

Fib

does

a

sync

to

synchronize

with

its

children

When

Fib

gets

to

the

sync

statement

it

must

suspend

and

wait

until

its

two

spawned

children

return

Only

then

may

Fib

proceed

to

add

the

return

values

Thus

the

spawns

and

synchronizations

specify

a

partial

execution

order

on

the

program

's

instructions

The

spawned

threads

must

be

executed

by

the

processors

of

a

parallel

machine

in

a

manner

consistent

with

the

program-specified

partial

order

and

in

general

the

scheduling

of

threads

to

processors

must

be

done

at

runtime

For

many

programs

the

spawning

of

threads

depends

on

the

input

data

and

for

these

programs

the

threads

can

not

even

be

identified

until

runtime

Furthermore

even

if

the

threads

can

be

statically

identified

determining

the

execution

time

of

any

given

thread

may

be

as

difficult

as

actually

running

the

thread

so

load-balancing

considerations

may

dictate

runtime

scheduling

For

these

types

of

applications

we

must

separate

the

static

expression

of

parallelism

in

the

program

from

the

dynamic

Scheduling

multithreaded

programs

scheduling

of

threads

at

runtime

A

multithreaded

language

permits

this

separation

by

incorporating

a

thread

scheduler

in

its

runtime

system

The

Cilk

language

supports

this

separation

with

a

set

of

extensions

to

the

C

language

for

expressing

parallelism

and

a

runtime

system

that

automatically

exploits

the

program

's

parallelism

on

a

parallel

computer

Cilk

applications

coded

to

date

include

protein

folding

graphic

rendering

backtrack

search

and

the

Socrates

chess

program

which

won

second

prize

in

the

ICCA

World

Computer

Chess

Championship

The

Cilk

runtime

system

is

particularly

easy

to

implement

and

easily

portable

In

fact

in

several

cases

individuals

unfamiliar

with

Cilk

have

ported

the

runtime

system

to

new

machines

in

under

week

Currently

the

Cilk

runtime

system

runs

on

the

Connection

Machine

MPP

the

Intel

Paragon

MPP

the

Sun

SparcStation

SMP

the

Silicon

Graphics

Power

Challenge

SMP

and

the

Cilk-NOW

network

of

workstations

discussed

below

The

Cilk

runtime

system

automatically

manages

the

low-level

details

of

thread

scheduling

and

it

does

so

with

a

work-stealing

scheduler

that

is

provably

efficient

Writing

a

high-performance

parallel

application

in

Cilk

the

programmer

can

focus

on

expressing

the

parallelism

in

the

algorithm

insulated

from

scheduling

details

and

secure

in

the

knowledge

that

the

runtime

system

delivers

guaranteed

efficient

performance

Figure

plots

measured

speedup

values

for

a

large

number

of

runs

of

the

Socrates

chess

program

on

the

Both

axes

are

normalized

as

we

shall

explain

when

we

revisit

this

speedup

plot

in

Section

For

now

we

think

of

the

normalized

machine

size

as

the

number

P

of

processors

increasing

from

left

to

right

and

we

think

of

the

normalized

speedup

simply

as

speedup

the

ratio

of

the

execution

time

to

the

P-processor

execution

time

The

line

and

the

horizontal

line

at

are

upper

bounds

on

the

achievable

speedup

This

plot

shows

that

every

run

achieved

near

optimal

speedup

This

performance

is

not

the

good

fortune

of

a

single

application

Rather

we

prove

that

for

a

large

and

important

class

of

programs

the

Cilk

runtime

system

achieves

near

optimal

performance

guaranteed

Moreover

Cilk

gives

the

user

an

algorithmic

model

of

application

performance

based

on

the

measures

of

work?the

total

number

of

instructions

executed?and

critical

path

length?the

length

of

a

longest

path

in

the

partial

order

of

instructions?which

can

be

used

to

predict

the

runtime

of

a

Cilk

program

accurately

Notice

that

the

measured

speedup

values

in

Figure

all

cluster

on

a

curve

As

we

shall

see

in

Section

this

curve

is

defined

as

a

function

of

only

three

parameters

work

critical

path

length

and

the

number

of

processors

Consequently

a

Cilk

programmer

can

tune

the

performance

of

his

or

her

application

by

focusing

on

the

work

and

critical

path

unworried

about

machine

specific

performance

details

In

order

to

execute

Cilk

programs

efficiently

on

a

network

of

workstations

the

CilkNOW

runtime

system

implements

adaptive

parallelism

and

transparent

fault

tolerance

Adaptive

parallelism

allows

a

Cilk

program

to

take

advantage

of

otherwise-idle

machines

whether

or

not

they

are

idle

when

the

program

starts

running

and

whether

or

not

they

will

remain

otherwise

idle

for

the

duration

of

the

run

When

a

given

workstation

is

left

idle

by

its

owner

it

automatically

joins

in

and

helps

out

with

the

execution

of

a

Cilk

program

When

the

owner

returns

to

work

the

machine

automatically

retreats

from

the

Cilk

program

The

criteria

to

determine

when

a

machine

is

idle

can

be

customized

for

each

machine

We

believe

that

maintaining

the

owner

's

sovereignty

is

essential

if

we

want

owners

to

allow

their

machines

to

be

used

for

parallel

computation

With

transparent

fault

tolerance

built

into

the

Cilk-NOW

runtime

system

Cilk

jobs

may

survive

machine

crashes

or

network

outages

Chapter

Executing

multithreaded

programs

efficiently

Measured

Value

Normalized

Speedup

Normalized

Machine

Size

Figure

Measured

speedups

for

the

Socrates

chess

program

despite

the

fact

that

Cilk

programs

are

fault

oblivious

having

been

coded

with

no

special

provision

for

handling

machine

or

network

failures

If

a

worker

crashes

then

other

workers

automatically

redo

any

work

that

was

lost

in

the

crash

In

the

case

of

a

more

catastrophic

failure

such

as

a

power

outage

or

a

total

network

failure

in

which

all

workers

may

crash

Cilk-NOW

provides

automatic

checkpointing

so

when

service

is

restored

the

Cilk

job

may

be

restarted

with

minimal

lost

work

Recently

we

ran

a

Cilk

protein-folding

application

using

Cilk-NOW

on

a

network

of

about

Sun

SparcStations

connected

by

Ethernet

to

solve

a

large-scale

problem

The

program

ran

for

days

survived

several

machine

crashes

and

reboots

utilized

processorhours

of

otherwise-idle

cycles

with

no

administrative

effort

on

our

part

while

other

users

of

the

network

went

about

their

business

unaware

of

the

program

's

presence

Adaptive

parallelism

and

fault

tolerance

were

invaluable

In

contrast

running

the

same

experiment

on

the

required

manually

breaking

the

job

into

pieces

small

enough

to

complete

in

the

interval

of

time

between

repartitionings

and

failures

of

the

machine

and

then

manually

combining

results

These

pieces

were

fed

to

the

machine

through

shell

scripts

to

ensure

that

the

machine

executed

these

pieces

even

when

we

were

not

around

to

perform

manual

feeding

The

efficient

and

predictable

performance

of

the

Cilk

runtime

system

and

the

adaptive

parallelism

and

fault

tolerant

features

of

Cilk-NOW

are

possible

because

Cilk

is

built

on

a

theoretically

sound

foundation

In

particular

the

Cilk

programming

model

is

highly

struc

Previous

results

and

related

work

tured

and

the

Cilk

runtime

system

's

scheduler

employs

a

provably

efficient

scheduling

algorithm

In

establishing

an

algorithmic

foundation

for

scheduling

multithreaded

computations

we

have

proven

a

lower

bound

showing

that

in

general

there

exist

multithreaded

computations

for

which

efficient

scheduling

is

impossible

and

we

have

proven

an

upper

bound

showing

that

for

an

important

class

of

computations

efficient

scheduling

is

possible

We

show

that

in

general

there

exist

multithreaded

computations

with

vast

amounts

of

provably

useless

parallelism

For

such

a

computation

any

execution

schedule

attempting

to

exploit

this

parallelism

must

use

an

amount

of

space

per

processor

vastly

in

excess

of

that

required

by

a

execution

On

the

other

hand

we

show

that

for

the

class

of

strict

computations

all

of

the

parallelism

can

be

exploited

Intuitively

a

strict

computation

is

one

in

which

threads

only

return

values

and

synchronize

with

their

ancestor

threads

Computations

such

as

those

derived

from

divide-and-conquer

backtracking

search

branch-and-bound

gametree

search

and

many

other

programs

are

naturally

strict

and

have

large

amounts

of

parallelism

We

show

that

for

any

strict

computation

and

any

number

of

processors

there

exists

an

execution

schedule

with

execution

time

that

is

within

a

factor

of

of

optimal

while

using

no

more

space

per

processor

than

that

required

by

a

execution

We

give

a

simple

though

impractical

and

nonscalable

algorithm

to

compute

such

efficient

schedules

For

practical

and

scalable

application

we

give

a

simple

and

fully

distributed

online

algorithm

for

scheduling

the

execution

of

multithreaded

computations

This

algorithm

is

based

on

the

technique

of

random

work

stealing

in

which

processors

needing

work

steal

threads

from

other

processors

chosen

at

random

We

show

that

for

fully

strict

-LRB-

well-structured

-RRB-

computations

this

algorithm

is

simultaneously

efficient

with

respect

to

time

space

and

communication

In

particular

this

work-stealing

algorithm

achieves

execution

time

that

is

universally

optimal

to

within

a

constant

factor

execution

space

that

is

existentially

optimal

to

within

a

constant

factor

and

communication

that

is

also

existentially

optimal

to

within

a

constant

factor

This

communication

bound

confirms

the

folk

wisdom

that

work-stealing

algorithms

require

much

less

communication

than

their

work-sharing

counterparts

In

our

analysis

we

conservatively

assume

that

when

multiple

processors

simultaneously

attempt

to

access

a

single

data

structure

then

their

accesses

are

serialized

in

an

order

determined

by

an

adversary

These

bounds

are

the

first

proven

bounds

for

the

case

of

computations

with

synchronization

The

Cilk

runtime

system

's

scheduler

implements

this

work-stealing

algorithm

Previous

results

and

related

work

The

Cilk

runtime

system

differs

from

other

systems

for

parallel

multithreaded

programming

primarily

in

its

algorithmic

foundation

and

its

ability

to

deliver

performance

guarantees

Nevertheless

the

algorithmic

work

builds

on

earlier

work

focused

on

more

restrictive

models

of

multithreaded

computation

Likewise

the

Cilk

programming

model

and

runtime

system?including

Cilk-NOW

build

on

ideas

found

in

earlier

systems

In

this

section

we

look

at

other

theoretical

results

and

systems

that

address

scheduling

issues

for

dynamic

parallel

computation

We

shall

not

look

at

data-parallel

systems

-LSB-

-RSB-

nor

at

systems

focused

Chapter

Executing

multithreaded

programs

efficiently

on

infrastructure

such

as

distributed

shared

memory

-LSB-

-RSB-

or

message

passing

-LSB-

-RSB-

Substantial

research

has

been

reported

in

the

theoretical

literature

concerning

the

scheduling

of

dynamic

computations

In

contrast

to

our

research

on

multithreaded

computations

however

other

theoretical

research

has

tended

to

ignore

space

requirements

and

communication

costs

Related

work

in

this

area

includes

a

randomized

work-stealing

algorithm

for

load

balancing

independent

jobs

-LSB-

-RSB-

algorithms

for

dynamically

embedding

trees

in

fixedconnection

networks

-LSB-

-RSB-

and

algorithms

for

backtrack

search

and

branch-and-bound

-LSB-

-RSB-

Backtrack

search

can

be

viewed

as

a

multithreaded

computation

with

no

synchronization

and

in

the

work

just

cited

the

only

algorithm

with

reasonable

space

bounds

is

the

random

work-stealing

algorithm

of

Karp

and

Zhang

-LSB-

-RSB-

though

they

did

not

make

this

observation

until

the

later

work

of

Zhang

and

Ortynski

-LSB-

-RSB-

Our

results

specialize

to

match

theirs

Likewise

most

of

the

systems-oriented

work

on

multithreading

has

ignored

the

space

issue

Notable

exceptions

include

the

k-bounded

loops

of

Culler

and

Arvind

-LSB-

-RSB-

and

the

throttling

mechanism

of

Ruggiero

and

Sargeant

-LSB-

-RSB-

These

techniques

and

others

-LSB-

-RSB-

have

met

with

some

success

though

have

any

algorithmic

foundation

In

algorithmic

work

that

considers

space

requirements

or

communication

costs

most

prior

work

has

focused

on

cases

like

backtrack

search

with

no

synchronization

or

has

focused

on

time

and

space

to

the

exclusion

of

communication

costs

Besides

the

work

of

Karp

Zhang

and

Ortynski

already

mentioned

Wu

and

Kung

-LSB-

-RSB-

proved

a

lower

bound

on

communication

requirements

in

parallel

divide-and-conquer

programs

and

Wu

-LSB-

-RSB-

gives

a

distributed

algorithm

for

scheduling

parallel

divide-and-conquer

programs

on

fixedconnection

networks

These

analyses

focus

on

the

tree-growing

stage

of

divide-and-conquer

programs

so

they

do

not

consider

the

case

when

synchronization

is

involved

For

programs

with

nested

fine-grained

parallelism

Blelloch

Gibbons

and

Matias

-LSB-

-RSB-

give

a

scheduling

algorithm

and

prove

that

it

is

efficient

with

respect

to

both

time

and

space

Burton

-LSB-

-RSB-

shows

how

to

limit

space

in

certain

parallel

computations

without

causing

deadlock

and

Burton

and

Simpson

-LSB-

-RSB-

give

an

offline

algorithm

that

is

efficient

with

respect

to

time

and

space

in

a

very

general

model

of

multithreaded

computation

In

contrast

our

work

presents

a

scheduling

algorithm

that

is

distributed

and

online

and

is

provably

efficient

with

respect

to

time

space

and

communication

for

a

large

class

of

multithreaded

computations

Like

the

algorithm

of

Karp

and

Zhang

our

algorithm

uses

the

technique

of

random

work

stealing

The

work-stealing

idea

is

not

a

new

one

but

until

our

results

studies

of

work

stealing

have

been

based

on

heuristic

notions

and

the

algorithmic

work

has

focused

on

particularly

simple

types

of

computations

such

as

the

backtrack

search

already

discussed

The

workstealing

idea

dates

back

at

least

as

far

as

Burton

and

Sleep

's

research

-LSB-

-RSB-

on

parallel

execution

of

functional

programs

and

Halstead

's

implementation

of

Multilisp

-LSB-

-RSB-

These

researchers

observed

that

heuristically

by

having

each

processor

work

as

if

it

is

the

only

one

-LRB-

i.e.

in

serial

depth-first

order

-RRB-

and

having

idle

processors

steal

threads

from

others

space

requirements

and

communication

requirements

should

be

curbed

Since

then

many

researchers

have

implemented

variants

on

this

strategy

-LSB-

-RSB-

Cilk

's

work-stealing

scheduler

is

very

similar

to

the

schedulers

in

some

of

these

other

systems

though

Cilk

's

algorithm

uses

randomness

and

is

provably

efficient

Many

multithreaded

programminglanguages

and

runtime

systems

are

based

on

heuristic

Contributions

of

this

thesis

scheduling

techniques

Though

systems

such

as

Charm

-LSB-

-RSB-

COOL

-LSB-

-RSB-

Id

-LSB-

-RSB-

Olden

-LSB-

-RSB-

and

others

-LSB-

-RSB-

are

based

on

sound

heuristics

that

seem

to

perform

well

in

practice

and

generally

have

wider

applicability

than

Cilk

are

able

to

provide

any

sort

of

performance

guarantee

or

accurate

machine-independent

performance

model

These

systems

require

that

performance-minded

programmers

become

intimate

with

a

collection

of

scheduling

heuristics

and

in

some

cases

machine-specific

details

In

contrast

Cilk

delivers

guaranteed

performance

and

provides

a

machine-independent

performance

abstraction

based

on

work

and

critical

path

length

The

use

of

work

and

critical

path

length

to

analyze

parallel

algorithms

and

model

application

performance

is

also

not

new

Work

and

critical

path

have

been

used

in

the

theory

community

for

years

to

analyze

parallel

algorithms

-LSB-

-RSB-

Blelloch

-LSB-

-RSB-

has

developed

a

performance

model

for

data-parallel

computations

based

on

these

same

two

abstract

measures

He

cites

many

advantages

to

such

a

model

over

machine-based

models

Cilk

provides

a

similar

performance

model

for

the

domain

of

multithreaded

computation

Adaptive

parallelism

as

implemented

in

Cilk-NOW

finds

earlier

incarnations

though

most

parallel

runtime

systems

employ

static

job-scheduling

strategies

Massively

parallel

supercomputers

such

as

the

Cray

Research

or

the

Thinking

Machines

for

example

either

dedicate

themselves

to

a

single

user

at

a

time

or

gang-timeshare

within

fixed

size

partitions

-LSB-

-RSB-

Systems

such

as

Charm

-LSB-

-RSB-

the

Parform

-LSB-

-RSB-

PVM/Hence

-LSB-

-RSB-

and

others

-LSB-

-RSB-

support

parallel

computing

on

a

network

of

workstations

In

these

systems

the

set

of

machines

on

which

the

program

runs

is

chosen

statically

by

the

user

Distributed

operating

systems

-LSB-

-RSB-

and

other

systems

-LSB-

-RSB-

provide

transparent

process

placement

and

-LRB-

in

some

cases

-RRB-

migration

but

these

systems

are

geared

towards

large

serial

programs

or

coarse-grain

distributed

programs

A

system

that

does

provide

adaptive

parallelism

is

Piranha

-LSB-

-RSB-

-LRB-

The

creators

of

Piranha

appear

to

have

coined

the

term

adaptive

parallelism

-RRB-

Based

on

Linda

-LSB-

-RSB-

Piranha

's

adaptive

parallelism

leverages

structure

in

the

Linda

programming

model

much

as

Cilk-NOW

leverages

structure

in

the

Cilk

programming

model

Adaptive

parallelism

is

also

present

in

the

Benevolent

Bandit

Laboratory

-LRB-

BBL

-RRB-

-LSB-

-RSB-

a

PC-based

system

The

BBL

system

architecture

is

closely

related

to

Cilk-NOW

's

The

Prospero

resource

manager

-LSB-

-RSB-

also

employs

a

similar

system

architecture

Runtime

systems

for

the

programming

language

COOL

-LSB-

-RSB-

running

on

symmetric

multiprocessors

-LSB-

-RSB-

and

cache-coherent

distributed

shared-memory

machines

-LSB-

-RSB-

use

process

control

to

support

adaptive

parallelism

These

systems

rely

on

special-purpose

operating

system

and

hardware

support

In

contrast

Cilk-NOW

supports

adaptive

parallelism

entirely

in

user-level

software

on

top

of

commercial

hardware

and

operating

systems

We

are

currently

aware

of

no

other

runtime

system

for

a

multithreaded

programming

language

that

provides

transparent

fault

tolerance

Contributions

of

this

thesis

This

thesis

shows

that

with

a

well-structured

programming

model

built

on

an

algorithmic

foundation

multithreaded

programs

can

be

executed

efficiently

with

guaranteed

and

pre

Chapter

Executing

multithreaded

programs

efficiently

dictable

performance

The

principal

contributions

of

this

thesis

are

the

following

ffl

A

lower

bound

that

shows

that

without

some

kind

of

structure

it

is

possible

to

write

multithreaded

programs

that

are

impossible

to

schedule

efficiently

for

parallel

execution

We

give

the

first

proof

that

parallelism

obtained

by

nonstrict

execution

order

may

be

chimerical

In

these

cases

any

attempt

to

exploit

this

parallelism

requires

vastly

more

space

per

processor

than

required

for

a

execution

ffl

An

upper

bound

that

shows

that

strictness

is

sufficient

structure

for

efficient

parallel

execution

of

multithreaded

programs

We

show

that

any

parallelism

obtainable

with

strict

execution

order

can

be

exploited

without

using

any

more

space

per

processor

than

required

for

a

execution

ffl

A

provably

efficient

scheduling

algorithm

for

multithreaded

programs

This

online

and

distributed

algorithm

is

based

on

the

popular

technique

of

work

stealing

We

prove

that

for

the

case

of

fully

strict

computations

this

algorithm

is

simultaneously

efficient

with

respect

to

space

time

and

communication

This

is

the

first

provably

efficient

algorithm

for

scheduling

multithreaded

computations

with

synchronization

ffl

Cilk

a

language

and

runtime

system

for

multithreaded

programming

The

Cilk

language

is

based

on

C

and

the

Cilk

runtime

system

uses

the

provably

efficient

workstealing

scheduler

Using

several

applications

we

show

empirically

that

Cilk

's

workstealing

scheduler

is

efficient

in

practice

We

further

show

that

the

performance

of

Cilk

applications

can

be

predicted

accurately

using

a

simple

model

based

on

two

abstract

measures

work

and

critical

path

length

ffl

Cilk-NOW

an

implementation

of

the

Cilk

runtime

system

on

a

network

of

workstations

Cilk-NOW

leverages

the

structure

in

Cilk

's

programming

model

in

order

to

implement

adaptive

parallelism

and

fault

tolerance

These

features

allow

Cilk

programs

to

run

efficiently

on

a

network

of

workstations

With

adaptive

parallelism

Cilk

programs

can

run

on

a

set

of

workstations

that

grows

and

shrinks

dynamically

With

fault

tolerance

Cilk

programs

continue

to

run

even

if

some

of

the

workstations

crash

The

remainder

of

this

thesis

is

organized

as

follows

In

Chapter

we

present

a

graphtheoretic

model

of

multithreaded

computation

This

model

forms

the

basis

for

our

analysis

In

Chapter

we

present

the

lower

bound

showing

that

in

general

there

exist

multithreaded

computations

for

which

efficient

scheduling

is

impossible

We

defer

the

proof

of

this

lower

bound

to

Appendix

A.

Additionally

in

Chapter

we

define

strictness

and

we

show

that

strictness

is

a

sufficient

structure

to

guarantee

the

existence

of

efficient

execution

schedules

In

Chapter

we

present

and

analyze

the

work-stealing

scheduling

algorithm

The

essential

ideas

of

this

algorithm

are

implemented

in

the

runtime

system

for

the

Cilk

multithreaded

language

In

Chapter

we

present

the

Cilk

language

and

the

implementation

of

the

Cilk

runtime

system

We

also

show

both

empirically

and

analytically

that

the

Cilk

runtime

system

delivers

efficient

and

predictable

performance

The

analysis

builds

on

the

work

of

Chapter

In

Chapter

we

present

the

implementation

of

adaptive

parallelism

and

fault

tolerance

in

Cilk-NOW

Finally

in

Chapter

we

conclude

and

discuss

current

and

planed

work

to

add

distributed

shared

memory

to

Cilk

using

dag

consistency

-LSB-

-RSB-

The

reader

Contributions

of

this

thesis

interested

only

in

the

system-building

contributions

of

this

thesis

may

safely

skip

ahead

to

Chapter

and

pass

over

Section

Information

about

the

current

and

forthcoming

Cilk

software

releases

can

be

found

in

Appendix

B.

Chapter

A

model

of

multithreaded

computation

The

execution

of

a

multithreaded

program

grows

a

directed

acyclic

graph

of

instructions

and

a

tree

of

threads

The

size

and

longest

path

length

of

the

graph

provide

bounds

on

the

achievable

execution

time

The

height

of

the

tree

provides

a

bound

on

the

achievable

execution

space

In

this

chapter

we

shall

introduce

our

graphical

model

of

multithreaded

computation

and

then

use

it

to

derive

simple

bounds

on

execution

time

and

space

Section

presents

the

model

We

then

use

this

model

in

Section

to

derive

time

bounds

and

in

Section

to

derive

space

bounds

This

model

and

associated

bounds

equip

us

with

an

algorithmic

foundation

for

analyzing

scheduling

algorithms

-LRB-

Chapters

and

-RRB-

and

predicting

Cilk

application

performance

-LRB-

Chapter

-RRB-

Multithreaded

computation

A

multithreaded

computation

models

the

time

and

space

resource

requirements

in

the

execution

of

a

multithreaded

program

This

model

contains

a

graph

of

instructions

and

a

tree

of

threads

that

unfold

dynamically

during

program

execution

In

this

section

we

present

this

model

and

define

what

it

means

for

a

parallel

computer

to

execute

a

multithreaded

computation

In

the

next

two

sections

we

shall

quantify

and

bound

the

time

and

space

requirements

A

multithreaded

computation

is

composed

of

a

set

of

threads

each

of

which

is

a

sequential

ordering

of

unit-size

instructions

A

processor

takes

one

unit

of

time

to

execute

one

instruction

In

the

example

computation

of

Figure

each

shaded

block

is

a

thread

with

circles

representing

instructions

and

the

horizontal

edges

called

continue

edges

representing

the

sequential

ordering

Thread

of

this

example

contains

instructions

and

The

instructions

of

a

thread

must

execute

in

this

sequential

order

from

the

first

-LRB-

leftmost

-RRB-

instruction

to

the

last

-LRB-

rightmost

-RRB-

instruction

In

order

to

execute

a

thread

we

allocate

for

it

a

block

of

memory

called

an

activation

frame

that

the

instructions

of

the

thread

can

use

to

store

the

values

on

which

they

compute

The

research

reported

in

this

chapter

is

joint

work

with

Charles

Leiserson

of

MIT

's

Laboratory

for

Computer

Science

and

was

first

published

in

-LSB-

-RSB-

Chapter

A

model

of

multithreaded

computation

Figure

A

multithreaded

computation

This

computation

contains

instructions

and

threads

An

execution

schedule

for

a

multithreaded

computation

determines

which

processors

of

a

parallel

computer

execute

which

instructions

at

each

step

In

any

given

step

of

an

execution

schedule

each

processor

either

executes

a

single

instruction

or

sits

idle

A

execution

schedule

for

our

example

computation

-LRB-

Figure

-RRB-

is

shown

in

Figure

At

step

of

this

example

processors

and

each

execute

an

instruction

while

processor

sits

idle

An

execution

schedule

depends

on

the

particular

multithreaded

computation

since

it

must

observe

the

sequential

ordering

of

the

instructions

in

each

thread

Specifically

if

an

instruction

has

a

predecessor?that

is

an

instruction

that

connects

to

it

via

a

continue

edge?in

its

thread

then

no

processor

may

execute

that

instruction

until

after

the

predecessor

has

been

executed

During

the

course

of

its

execution

a

thread

may

create

or

spawn

other

threads

Spawning

a

thread

is

like

a

subroutine

call

except

that

the

spawning

thread

can

operate

concurrently

with

the

spawned

thread

We

consider

spawned

threads

to

be

children

of

the

thread

that

did

the

spawning

and

a

thread

may

spawn

as

many

children

as

it

desires

In

this

way

the

threads

are

organized

into

a

spawn

tree

as

indicated

in

Figure

by

the

downwardpointing

shaded

edges

called

spawn

edges

that

connect

threads

to

their

spawned

children

The

spawn

tree

is

the

parallel

analog

of

a

call

tree

In

our

example

computation

the

spawn

tree

's

root

thread

has

two

children

and

and

thread

has

three

children

and

Threads

and

which

have

no

children

are

leaf

threads

Each

spawn

edge

goes

from

a

specific

instruction?the

instruction

that

actually

does

the

spawn

operation?in

the

parent

thread

to

the

first

instruction

of

the

child

thread

An

execution

schedule

must

obey

this

edge

in

that

no

processor

may

execute

an

instruction

in

a

spawned

child

thread

until

after

the

spawning

instruction

in

the

parent

thread

has

been

executed

In

our

example

computation

-LRB-

Figure

-RRB-

due

to

the

spawn

edge

-LRB-

-RRB-

instruction

can

not

be

executed

until

after

the

spawning

instruction

Consistent

with

our

unit-time

model

of

instructions

a

single

instruction

may

spawn

at

most

one

child

When

the

spawn

Multithreaded

computation

processor

activity

step

living

threads

Figure

A

execution

schedule

for

the

computation

of

Figure

This

schedule

lists

the

living

threads

at

the

start

of

each

step

and

the

instruction

-LRB-

if

any

-RRB-

executed

by

each

of

the

processors

and

at

each

step

Living

threads

that

are

ready

are

listed

in

bold

The

other

living

threads

are

stalled

ing

instruction

is

executed

we

allocate

an

activation

frame

for

the

new

child

thread

Once

a

thread

has

been

spawned

and

its

frame

has

been

allocated

we

say

the

thread

is

alive

or

living

When

the

last

instruction

of

a

thread

gets

executed

the

thread

dies

and

we

deallocate

its

frame

In

our

execution

schedule

-LRB-

Figure

-RRB-

thread

is

spawned

at

step

and

dies

at

step

Therefore

it

is

living

at

steps

and

Spawns

introduce

parallelism

but

a

given

execution

schedule

may

or

may

not

exploit

this

parallelism

For

example

when

thread

spawns

children

and

these

two

child

threads

may

be

executed

concurrently

on

different

processors

Alternatively

a

single

processor

may

execute

both

threads

possibly

interleaving

their

instructions

in

some

way

In

addition

to

the

continue

and

spawn

edges

a

multithreaded

computation

may

also

contain

dependency

edges

as

illustrated

in

Figure

by

the

curved

edges

Dependency

edges

model

the

data

and

control

dependencies

between

threads

As

an

example

of

a

data

dependency

consider

an

instruction

that

produces

a

data

value

consumed

by

another

instruction

Such

a

producer/consumer

relationship

precludes

the

consuming

instruction

from

executing

until

after

the

producing

instruction

A

dependency

edge

from

the

producing

instruction

to

the

consuming

instruction

enforces

this

ordering

An

execution

schedule

must

obey

this

edge

in

that

no

processor

may

execute

the

consuming

instruction

until

after

the

producing

instruction

has

been

executed

For

example

due

to

the

dependency

edge

-LRB-

-RRB-

instruction

can

not

be

executed

until

after

Dependency

edges

allow

threads

to

synchronize

Consider

again

the

case

of

a

dependency

edge

that

models

a

producer/consumer

data

dependency

If

the

execution

of

the

consuming

thread

arrives

at

the

consuming

instruction

before

the

producing

instruction

has

executed

then

execution

of

the

consuming

thread

can

not

continue?the

thread

stalls

and

must

suspend

its

execution

Once

the

producing

instruction

executes

the

dependency

is

resolved

Chapter

A

model

of

multithreaded

computation

thereby

enabling

the

consuming

thread

to

resume

with

its

execution?the

thread

becomes

ready

Thus

the

dependency

edge

allows

the

consuming

thread

to

synchronize

with

the

producing

thread

For

example

at

step

of

our

execution

schedule

-LRB-

Figure

-RRB-

thread

is

stalled

at

instruction

because

instruction

has

not

yet

been

executed

At

step

instruction

is

executed

by

processor

thereby

enabling

thread

At

step

thread

is

ready

at

instruction

A

multithreaded

computation

does

not

model

the

mechanism

by

which

dependencies

get

resolved

or

unresolved

dependencies

get

detected

In

Chapter

we

present

Cilk

's

implementation

of

such

a

mechanism

An

execution

schedule

must

obey

the

constraints

given

by

the

dependency

spawn

and

continue

edges

of

the

computation

These

edges

form

a

directed

graph

of

instructions

and

no

processor

may

execute

an

instruction

until

after

all

of

the

instruction

's

predecessors

in

this

graph

have

been

executed

So

that

execution

schedules

exist

this

graph

must

be

acyclic

That

is

it

must

be

a

directed

acyclic

graph

or

dag

At

any

given

step

of

an

execution

schedule

an

instruction

is

ready

if

all

of

its

predecessors

in

the

dag

have

been

executed

Only

ready

instructions

may

be

executed

The

notion

of

an

execution

schedule

is

independent

of

any

real

machine

characteristics

An

execution

schedule

simply

requires

that

no

processor

executes

more

than

one

instruction

per

time

step

and

every

instruction

is

executed

at

a

time

step

after

all

of

its

predecessor

instructions

-LRB-

which

connect

to

it

via

continue

spawn

or

dependency

edges

-RRB-

have

been

executed

A

given

execution

schedule

may

not

be

viable

for

a

real

machine

since

the

schedule

may

not

account

for

properties

such

as

communication

latency

For

example

in

our

processor

execution

schedule

-LRB-

Figure

-RRB-

instruction

is

executed

at

step

by

processor

exactly

one

step

after

is

executed

by

processor

even

though

there

is

a

dependency

between

them

that

surely

requires

some

latency

to

be

resolved

In

later

chapters

we

will

turn

attention

to

computing

execution

schedules

for

real

machines

To

summarize

a

multithreaded

computation

can

be

viewed

as

a

dag

of

instructions

connected

by

continue

spawn

and

dependency

edges

The

instructions

are

connected

by

continue

edges

into

threads

and

the

threads

form

a

tree

with

the

spawn

edges

When

a

thread

is

spawned

an

activation

frame

is

allocated

and

this

frame

remains

allocated

as

long

as

the

thread

remains

alive

A

living

thread

may

be

either

ready

or

stalled

due

to

an

unresolved

dependency

It

is

important

to

note

the

difference

between

what

we

are

calling

a

multithreaded

computation

and

a

program

A

multithreaded

computation

is

the

parallel

instruction

stream

resulting

from

the

execution

of

a

multithreaded

program

with

a

given

set

of

inputs

Unlike

a

serial

computation

in

which

the

instruction

stream

is

totally

ordered

a

multithreaded

computation

only

partially

orders

its

instructions

In

general

a

multithreaded

computation

is

not

a

statically

determined

object

rather

the

computation

unfolds

dynamically

during

execution

as

determined

by

the

program

and

the

input

data

For

example

a

program

may

have

conditionals

and

therefore

the

order

of

instructions

-LRB-

or

even

the

set

of

instructions

-RRB-

executed

in

a

thread

may

not

be

known

until

the

thread

is

actually

executed

We

can

think

of

a

multithreaded

computation

as

encapsulating

both

the

program

and

the

input

data

The

computation

then

reveals

itself

dynamically

during

execution

We

shall

characterize

the

time

and

space

of

an

execution

of

a

multithreaded

computation

in

terms

of

three

fundamental

parameters

work

critical

path

length

and

stack

depth

We

first

introduce

work

and

critical

path

length

which

relate

to

the

execution

time

and

then

we

Execution

time

pathlength

Figure

The

dag

corresponding

to

the

computation

of

Figure

The

critical

paths

each

instructions

long

are

shown

bold

focus

on

stack

depth

which

relates

to

the

storage

requirements

Execution

time

Execution

time

requirements

are

captured

in

two

fundamental

measures

of

the

multithreaded

computation

's

graph

of

instructions

work

and

critical

path

length

In

this

section

we

define

the

work

and

critical

path

length

of

a

multithreaded

computation

and

we

use

these

measures

to

derive

simple

bounds

on

execution

time

If

we

ignore

the

shading

in

Figure

that

organizes

instructions

into

threads

our

multithreaded

computation

is

just

a

dag

of

instructions

The

dag

corresponding

to

the

example

computation

of

Figure

is

shown

in

Figure

We

define

the

work

of

the

computation

to

be

the

total

number

of

instructions

and

the

critical

path

length

to

be

the

length

of

a

longest

directed

path

in

the

dag

In

the

case

of

our

example

the

work

is

instructions

and

the

critical

path

length

is

instructions

We

quantify

and

bound

the

execution

time

of

a

computation

in

terms

of

the

computation

's

work

and

critical

path

length

For

a

given

computation

let

T

-LRB-

X

-RRB-

denote

the

time

to

Chapter

A

model

of

multithreaded

computation

processor

activity

step

living

threads

Figure

An

infinite-processor

execution

schedule

for

the

dag

of

Figure

The

maximum

number

of

instructions

executed

at

any

time

step

is

-LRB-

steps

and

-RRB-

and

the

average

number

of

instructions

executed

per

time

step

is

-LRB-

total

instructions

divided

by

steps

-RRB-

execute

the

computation

using

a

given

P-processor

execution

schedule

X

and

let

TP

min

X

T

-LRB-

X

-RRB-

denote

the

minimum

execution

time

over

all

P-processor

execution

schedules

X

Then

is

the

work

of

the

computation

since

a

computer

can

only

execute

one

instruction

at

each

step

and

T

is

the

critical

path

length

since

even

with

arbitrarily

many

processors

each

instruction

on

a

path

must

execute

serially

In

our

example

dag

-LRB-

Figure

-RRB-

-LRB-

a

single

processor

can

execute

the

instructions

in

the

order

since

this

is

a

topological

sort

of

the

dag

-RRB-

and

T

-LRB-

an

infinite-processor

execution

schedule

that

achieves

this

time

is

shown

in

Figure

-RRB-

The

work

and

critical

path

length

T

are

not

intended

to

denote

the

execution

time

on

any

real

single-processor

or

infinite-processor

machine

These

quantities

are

abstractions

of

a

computation

and

are

independent

of

any

real

machine

characteristics

such

as

communication

latency

We

can

think

of

and

T

as

execution

times

on

an

ideal

machine

with

no

scheduling

overhead

Nevertheless

we

show

in

Chapter

that

despite

their

abstract

nature

with

a

good

scheduling

algorithm

work

and

critical

path

length

have

a

lot

to

say

about

actual

execution

time

of

actual

programs

on

actual

machines

Still

viewing

the

computation

as

a

dag

we

borrow

some

basic

results

on

dag

scheduling

to

bound

TP

A

computer

with

P

processors

can

execute

at

most

P

instructions

per

step

and

since

the

computation

has

instructions

we

have

the

lower

bound

TP

=P

And

of

course

we

also

have

the

lower

bound

TP

T

Early

independent

work

by

Brent

-LSB-

Lemma

-RSB-

and

Graham

-LSB-

-RSB-

yields

the

upper

bound

TP

=P

T

The

following

theorem

extends

these

results

minimally

to

show

that

this

upper

bound

on

TP

can

be

obtained

by

any

greedy

schedule

one

in

which

at

each

step

of

the

execution

if

at

least

P

instructions

are

ready

then

P

instructions

execute

and

if

fewer

than

P

instructions

are

ready

then

all

execute

Both

of

our

example

schedules

-LRB-

Figures

and

-RRB-

are

greedy

Execution

time

Theorem

-LRB-

The

greedy-scheduling

theorem

-RRB-

For

any

multithreaded

computation

with

work

and

critical

path

length

T

and

for

any

number

P

of

processors

any

greedy

P-processor

execution

schedule

X

achieves

T

-LRB-

X

-RRB-

=P

T

Proof

Let

G

-LRB-

V

E

-RRB-

denote

the

underlying

dag

of

the

computation

Thus

we

have

jV

j

and

a

longest

directed

path

in

G

has

length

T

Consider

a

greedy

P-processor

execution

schedule

X

where

the

set

of

instructions

executed

at

time

step

t

for

t

k

is

denoted

Vt

with

k

T

-LRB-

X

-RRB-

The

Vt

form

a

partition

of

V

We

shall

consider

the

progression

Gki

of

dags

where

G

and

for

t

k

we

have

Vt

Vt

Vt

and

Gt

is

the

subgraph

of

Gt

induced

by

Vt

In

other

words

Gt

is

obtained

from

Gt

by

removing

from

Gt

all

the

instructions

that

are

executed

by

X

at

step

t

and

all

edges

incident

on

these

instructions

We

shall

show

that

each

step

of

the

execution

either

decreases

the

size

of

the

dag

or

decreases

the

length

of

the

longest

path

in

the

dag

We

account

for

each

step

t

according

to

jVt

j.

Consider

a

step

t

with

jVt

j

P.

-LRB-

Such

a

step

of

our

example

execution

-LRB-

Figure

-RRB-

is

shown

in

Figure

-RRB-

In

this

case

jVtj

jVt

P

so

since

jV

j

there

can

be

at

most

Pc

such

steps

Now

consider

a

step

t

with

jVt

j

P.

-LRB-

Such

a

step

of

our

execution

is

shown

in

Figure

-RRB-

In

this

case

since

X

is

greedy

Vt

must

contain

every

vertex

of

Gt

with

indegree

Therefore

the

length

of

a

longest

path

in

Gt

is

one

less

than

the

length

of

a

longest

path

in

Gt

Since

the

length

of

a

longest

path

in

G

is

T

there

can

be

no

more

than

T

steps

t

with

jVt

j

P.

Consequently

the

time

it

takes

the

P-processor

schedule

X

to

execute

the

computation

is

T

-LRB-

X

-RRB-

Pc

T

=P

T

The

greedy-scheduling

theorem

-LRB-

Theorem

-RRB-

can

be

interpreted

in

two

important

ways

First

the

time

bound

given

by

the

theorem

says

that

any

greedy

schedule

yields

an

execution

time

that

is

within

a

factor

of

of

an

optimal

schedule

which

follows

because

=P

T

=P

T?g

and

TP

=P

T?g

This

observation

was

first

made

by

Graham

-LSB-

-RSB-

Second

the

greedy-scheduling

theorem

tells

us

when

we

can

obtain

linear

speedup

that

is

when

we

can

find

a

P-processor

execution

schedule

X

such

that

T

-LRB-

X

-RRB-

Q

-LRB-

=P

-RRB-

Specifically

when

the

number

P

of

processors

is

no

more

than

T

then

=P

T

which

implies

that

for

a

greedy

schedule

X

we

have

T

-LRB-

X

-RRB-

=P

The

ratio

T

is

called

the

average

parallelism

Looking

at

the

example

dag

of

Figure

if

we

think

of

the

work

as

the

area

of

the

dag

and

the

critical

path

length

T

as

the

height

of

the

dag

then

the

average

parallelism

T

is

the

average

width

of

the

dag

Our

example

has

average

parallelism

T

We

can

also

think

of

the

average

parallelism

as

the

average

number

of

instructions

executed

per

step

of

a

greedy

infinite-processor

execution

schedule

-LRB-

Figure

-RRB-

We

shall

be

especially

interested

in

the

regime

where

P

O

-LRB-

T

-RRB-

and

linear

speedup

is

possible

since

outside

this

regime

linear

speedup

is

impossible

to

achieve

because

TP

T

These

results

on

dag

scheduling

have

been

known

for

years

A

multithreaded

computation

however

adds

further

structure

to

the

dag

the

partitioning

of

instructions

into

a

tree

of

threads

This

additional

structure

allows

us

to

quantify

the

space

required

in

executing

a

multithreaded

computation

Once

we

have

quantified

space

requirements

we

will

look

Chapter

A

model

of

multithreaded

computation

Figure

Step

in

the

execution

of

our

example

computation

The

faint

instructions

have

already

been

executed

The

white-on-black

instructions

are

the

ones

actually

executed

at

this

step

All

processors

do

work

at

this

step

Figure

Step

in

the

execution

of

our

example

computation

At

this

step

a

processor

sits

idle

Nevertheless

every

instruction

with

in-degree

get

executed

and

consequently

the

length

of

the

critical

path

is

reduced

by

Execution

space

back

at

the

greedy-scheduling

theorem

and

consider

whether

there

exist

execution

schedules

that

achieve

similar

time

bounds

while

also

making

efficient

use

of

space

Of

course

we

will

have

to

quantify

a

space

bound

to

capture

what

we

mean

by

efficient

use

of

space

Execution

space

A

multithreaded

computation

models

execution

space

requirements

in

the

spawn

tree

of

threads

In

this

section

we

shall

focus

on

a

single

measure

of

this

tree

stack

depth

We

use

this

measure

to

derive

simple

bounds

on

execution

space

and

then

we

formalize

our

goal

to

achieve

efficient

use

of

space

In

our

analysis

of

space

requirements

we

shall

only

analyze

stack-like

memory

We

say

that

memory

is

stack-like

if

its

allocation

and

deallocation

obey

the

following

rules

No

instruction

allocates

memory

if

its

thread

could

possibly

have

a

living

child

when

the

instruction

is

executed

and

all

memory

allocated

for

a

thread

is

deallocated

before

or

when

the

thread

dies

For

now

we

shall

simplify

our

notion

of

stack-like

memory

by

assuming

that

all

memory

for

a

thread

is

allocated

in

an

activation

frame

when

the

thread

is

spawned

and

deallocated

when

the

thread

dies

With

this

assumption

the

space

being

used

at

any

time

step

t

is

equal

to

the

sum

of

the

sizes

of

the

activation

frames

of

the

threads

that

are

living

at

step

t

and

the

total

space

used

in

executing

a

computation

is

the

maximum

such

value

over

the

course

of

the

execution

For

now

we

shall

also

assume

that

a

parent

thread

remains

alive

until

all

its

children

die

Though

these

assumptions

are

not

absolutely

necessary

they

simplify

our

analysis

of

space

utilization

by

giving

the

execution

a

natural

structure

In

the

next

chapter

we

shall

see

how

these

assumptions

can

be

relaxed

to

account

for

arbitrary

stack-like

memory

The

multithreaded

computation

's

spawn

tree

of

threads

naturally

gives

rise

to

a

corresponding

tree

of

activation

frames

We

call

this

tree

the

activation

tree

The

activation

tree

corresponding

to

the

example

computation

of

Figure

is

shown

in

Figure

Each

node

of

the

activation

tree

is

an

activation

frame

drawn

as

a

block

with

height

equal

to

the

size

of

the

frame

We

define

the

stack

depth

of

a

thread

to

be

the

sum

of

the

sizes

of

the

activation

frames

of

all

its

ancestors

including

itself

The

stack

depth

of

a

multithreaded

computation

is

the

maximum

stack

depth

of

any

thread

In

Figure

each

child

frame

has

its

top

aligned

with

its

parent

's

bottom

Thus

we

can

view

the

computation

's

stack

depth

as

the

depth

of

the

activation

tree

We

define

the

spawn

subtree

at

any

time

step

t

to

be

the

portion

of

the

spawn

tree

consisting

of

just

those

threads

that

are

alive

at

step

t.

Analogously

we

define

the

activation

subtree

The

activation

subtree

at

step

of

our

example

execution

-LRB-

Figure

-RRB-

is

shown

in

Figure

The

space

used

at

time

step

t

equals

the

size

of

the

activation

subtree

at

step

t

and

the

total

space

used

in

executing

a

computation

is

the

maximum

such

value

over

the

course

of

the

execution

In

our

example

execution

this

maximum

space

usage

occurs

at

step

when

the

activation

subtree

contains

a

frame

for

every

thread

except

We

shall

denote

the

space

required

by

a

P-processor

execution

schedule

X

of

a

multithreaded

computation

by

S

-LRB-

X

-RRB-

Since

we

can

always

simulate

a

P-processor

execution

with

Chapter

A

model

of

multithreaded

computation

Figure

The

activation

tree

corresponding

to

the

computation

of

Figure

Figure

The

activation

subtree

at

step

in

the

execution

of

our

example

computation

Threads

not

currently

living

are

faint

a

execution

that

uses

no

more

space

we

have

S

-LRB-

X

-RRB-

where

min

X

S

-LRB-

X

-RRB-

denotes

the

minimum

space

used

over

all

execution

schedules

X

The

following

simple

theorem

shows

that

the

stack

depth

of

a

computation

is

a

lower

bound

on

the

space

required

to

execute

it

Theorem

Let

S

be

the

stack

depth

of

a

multithreaded

computation

and

let

X

be

a

P-processor

execution

schedule

of

the

computation

Then

we

have

S

-LRB-

X

-RRB-

S

and

more

specifically

we

have

S.

Proof

In

any

schedule

the

leaf

thread

with

greatest

stack

depth

must

be

alive

at

some

time

step

Since

we

assume

that

if

a

thread

is

alive

its

parent

is

alive

when

the

deepest

leaf

thread

is

alive

all

its

ancestors

are

alive

and

hence

all

its

ancestors

frames

are

allocated

But

the

sum

of

the

sizes

of

its

ancestors

activation

frames

is

just

the

stack

depth

Since

S

-LRB-

X

-RRB-

S

holds

for

any

P-processor

schedule

X

and

all

P

it

holds

for

the

minimum-space

execution

schedule

and

hence

we

have

S.

Given

the

lower

bound

of

stack

depth

on

the

space

used

by

a

P-processor

schedule

it

is

natural

to

ask

whether

the

stack

depth

can

be

achieved

as

an

upper

bound

In

general

the

Execution

space

answer

is

no

since

all

the

threads

in

a

computation

may

contain

a

cycle

of

dependencies

that

force

all

of

them

to

be

simultaneously

living

in

any

execution

schedule

Figure

illustrates

such

a

computation

For

the

class

of

depth-first

computations

however

space

equal

to

the

stack

depth

can

be

achieved

by

a

schedule

Figure

The

dependency

edges

in

this

computation

form

cycles

through

all

of

the

child

threads

though

they

do

not

create

any

cycle

of

instructions

Any

execution

schedule

must

have

all

of

this

computation

's

threads

simultaneously

living

at

some

step

This

computation

is

not

depth-first

since

the

dependency

edges

-LRB-

-RRB-

and

-LRB-

-RRB-

violate

the

left-to-right

depth-first

order

A

depth-first

multithreaded

computation

is

a

multithreaded

computation

in

which

the

left-to-right

depth-first

search

of

instructions

in

the

computation

always

visits

all

of

any

given

instruction

's

predecessors

in

the

dag

before

it

visits

the

given

instruction

Specifically

we

define

the

left-to-right

depth-first

order

of

instructions

as

follows

If

we

ignore

the

dependency

edges

in

the

dag

and

just

look

at

the

instructions

connected

by

continue

and

spawn

edges

we

have

a

binary

tree

rooted

at

the

first

instruction

of

the

root

thread

Any

instruction

with

children

in

this

tree

must

connect

to

one

child

by

a

spawn

edge

and

the

other

child

by

a

continue

edge

We

consider

a

child

connected

to

its

parent

by

a

spawn

edge

to

be

the

left

child

and

a

child

connected

to

its

parent

by

a

continue

edge

to

be

the

right

child

The

leftto-right

depth-first

order

is

then

the

order

of

instructions

visited

by

a

preorder

walk

of

this

tree

-LSB-

-RSB-

For

example

the

left-to-right

depth-first

order

for

the

computation

of

Figure

is

Now

considering

the

dependency

edges

again

a

multithreaded

computation

is

depth-first

if

this

left-to-right

depth-first

order

yields

a

execution

schedule

In

other

words

the

computation

is

depth-first

if

of

its

dependency

edges

violate

its

leftto-right

depth-first

order

The

computation

of

Figure

is

not

depth-first

since

the

dependency

edges

-LRB-

-RRB-

and

-LRB-

-RRB-

violate

the

left-to-right

depth-first

order

On

the

other

hand

our

example

computation

of

Figure

is

depth-first

processor

executing

instructions

in

the

order

is

a

valid

execution

schedule

In

fact

for

depth-first

computations

this

left-to-right

depth-first

order

produces

a

processor

execution

schedule

which

is

just

the

familiar

serial

stack-based

execution

This

execution

begins

with

the

root

thread

and

executes

its

instructions

until

it

either

spawns

a

child

thread

or

dies

If

the

thread

spawns

a

child

the

parent

thread

is

suspended

to

be

resumed

only

after

the

child

thread

dies

the

scheduler

then

begins

work

on

the

child

executing

the

child

until

it

either

spawns

or

dies

Theorem

For

any

depth-first

multithreaded

computation

with

stack

depth

S

we

have

S.

Chapter

A

model

of

multithreaded

computation

Proof

At

any

time

in

a

serial

depth-first

execution

of

the

computation

the

set

of

living

threads

always

forms

a

path

from

the

root

Therefore

the

space

required

is

just

the

stack

depth

of

the

computation

By

Theorem

we

have

S

and

thus

the

space

used

is

the

minimum

possible

For

the

remainder

of

this

thesis

we

shall

focus

on

depth-first

multithreaded

computations

and

therefore

we

shall

use

to

denote

stack

depth

We

now

turn

our

attention

to

determining

howmuch

space

S

-LRB-

X

-RRB-

a

P-processor

execution

schedule

X

can

use

and

still

be

considered

efficient

with

respect

to

space

usage

Our

strategy

is

to

compare

the

space

used

by

a

P-processor

schedule

with

the

space

required

by

an

optimal

schedule

Of

course

we

can

always

ignore

P

of

the

processors

to

match

the

single-processor

space

bounds

and

therefore

our

goal

is

to

use

small

space

while

obtaining

linear

speedup

We

argue

that

a

P-processor

execution

schedule

X

that

uses

space

S

-LRB-

X

-RRB-

O

-LRB-

-RRB-

is

efficient

There

exist

very

simple

multithreaded

computations

that

require

Q

-LRB-

-RRB-

space

in

order

to

achieve

linear

speedup

Consider

the

following

computation

The

root

thread

is

a

loop

that

spawns

a

child

thread

for

each

iteration

and

each

child

thread

is

a

leaf

The

computation

has

the

activation

tree

shown

in

Figure

The

root

activation

frame

is

negligible

in

size

compared

with

the

leaves

-LRB-

its

children

-RRB-

A

single

processor

executing

this

computation

uses

only

the

space

needed

for

a

single

iteration

-LRB-

plus

the

space

used

by

the

root

-RRB-

since

upon

completion

of

an

iteration

all

the

memory

can

be

freed

and

then

reused

for

the

next

iteration

Thus

is

slightly

larger

than

the

size

of

a

single

leaf

activation

frame

In

general

with

P

processors

obtaining

linear

speedup

requires

executing

Q

-LRB-

P

-RRB-

iterations

concurrently

Such

a

P-processor

execution

schedule

X

has

Q

-LRB-

P

-RRB-

leaf

threads

living

at

some

step

and

therefore

it

uses

space

S

-LRB-

X

-RRB-

Q

-LRB-

-RRB-

Figure

The

activation

tree

for

a

multithreaded

computation

that

requires

linear

expansion

of

space

in

order

to

achieve

linear

speedup

A

P-processor

execution

schedule

X

for

which

S

-LRB-

X

-RRB-

Q

-LRB-

-RRB-

is

said

to

exhibit

linear

expansion

of

space

Reasonable

example

computations

such

as

the

one

just

considered

show

that

for

some

computations

obtaining

linear

speedup

requires

a

linear

expansion

of

space

For

other

computations

we

would

like

to

do

better

Nevertheless

a

P-processor

schedule

X

for

which

S

-LRB-

X

-RRB-

O

-LRB-

-RRB-

is

arguably

efficient

since

on

average

each

of

the

P

processors

needs

no

more

memory

than

is

used

by

the

optimal

execution

Recalling

that

any

greedy

execution

schedule

achieves

linear

speedup

-LRB-

provided

that

the

number

of

processors

is

at

most

proportional

to

the

average

parallelism

-RRB-

searching

for

execution

schedules

that

simultaneously

achieve

linear

speedup

and

linear

expansion

of

space

appears

as

a

reasonable

and

maybe

even

modest

endeavor

In

the

next

chapter

however

Execution

space

we

shall

show

that

for

some

multithreaded

computations?even

depth-first

multithreaded

computations?this

search

must

fail

On

the

other

hand

we

also

show

in

the

next

chapter

that

for

the

class

of

strict

-LRB-

a

subclass

of

depth-first

-RRB-

multithreaded

computations

such

efficient

execution

schedules

do

exist

and

they

are

easy

to

find

Chapter

Strict

multithreaded

computations

Does

every

depth-first

multithreaded

computation

have

an

execution

schedule

that

is

simultaneously

efficient

with

respect

to

time

and

space

And

if

not

what

necessary

and

sufficient

conditions

can

we

place

on

computations

to

guarantee

the

existence

of

such

a

schedule

In

this

chapter

we

show

that

the

answer

to

the

first

of

these

questions

is

no

and

we

partially

answer

the

second

In

Section

we

present

the

lower

bound

that

answers

the

first

question

Then

in

Section

we

define

a

condition

called

strictness

and

in

Section

we

show

that

strictness

is

a

sufficient

condition

to

guarantee

the

existence

of

efficient

execution

schedules

We

leave

necessary

conditions

as

an

open

question

Strictness

provides

important

structure

to

computations

that

we

leverage

in

a

provably

efficient

scheduling

algorithm

-LRB-

Chapter

-RRB-

employed

by

the

Cilk

runtime

system

-LRB-

Chapter

-RRB-

and

in

the

implementation

of

adaptive

parallelism

and

fault

tolerance

on

a

network

of

workstations

-LRB-

Chapter

-RRB-

A

lower

bound

for

the

general

case

In

a

study

of

resource

requirements

for

dataflow

programs

published

in

Culler

and

Arvind

-LSB-

-RSB-

observed

applications

with

parallelism

that

they

conjectured

to

be

useless

Useless

parallelism

is

parallelism

that

requires

excessive

amounts

of

space

resource

to

exploit

While

Culler

and

Arvind

argued

convincingly

that

the

observed

useless

parallelism

is

in

fact

useless

they

came

short

of

a

proof

and

they

left

open

the

possibility

of

a

clever

scheduler

that

might

be

able

to

exploit

this

parallelism

without

using

excessive

amounts

of

space

With

loop-bounding

-LSB-

-RSB-

techniques

they

were

able

to

eliminate

the

useless

parallelism

with

only

a

small

decrease

in

the

average

parallelism

Their

applications

had

only

small

amounts

of

useless

parallelism

In

this

section

we

show

that

multithreaded

computations

may

contain

vast

quantities

of

provably

useless

parallelism

In

particular

we

show

that

there

exist

depth-first

multithreaded

computations

with

large

amounts

of

average

parallelism

such

that

any

execution

schedule

attempting

to

exploit

this

parallelism

must

use

excessive

amounts

of

space?that

is

much

more

space

per

processor

than

required

by

a

execution

This

lower

Some

of

the

research

reported

in

this

chapter

is

joint

work

with

Charles

Leiserson

of

MIT

's

Laboratory

for

Computer

Science

and

was

first

published

in

-LSB-

-RSB-

and

-LSB-

-RSB-

Chapter

Strict

multithreaded

computations

bound

motivates

our

consideration

in

later

sections

of

computations

with

more

structure

namely

strict

multithreaded

computations

Before

going

on

to

a

more

formal

statement

of

the

lower

bound

it

is

worth

noting

the

difference

between

useless

and

excess

parallelism

If

we

have

a

computation

with

average

parallelism

equal

to

one

million

and

we

are

executing

this

computation

on

a

parallel

computer

with

two

processors

then

the

computation

has

excess

parallelism

Excess

parallelism

is

not

necessarily

useless

If

our

computer

actually

had

one

million

processors

we

might

be

able

to

exploit

all

of

the

parallelism

while

using

only

modest

amounts

of

space

per

processor

For

any

amount

of

serial

space

and

any

-LRB-

reasonably

large

-RRB-

serial

execution

time

we

can

exhibit

a

depth-first

multithreaded

computation

with

work

and

stack

depth

but

with

provably

bad

time/space

tradeoff

characteristics

Being

depth-first

we

know

from

Theorem

-LRB-

page

-RRB-

that

our

computation

can

be

executed

using

space

Furthermore

we

know

from

the

greedy-scheduling

theorem

-LRB-

Theorem

page

-RRB-

that

for

any

number

P

of

processors

any

greedy

P-processor

execution

schedule

X

achieves

T

-LRB-

X

-RRB-

=P

T

Our

computation

has

critical

path

length

T

ss

and

consequently

for

P

=O

-LRB-

-RRB-

a

greedy

schedule

X

yields

T

-LRB-

X

-RRB-

O

-LRB-

=P

-RRB-

linear

speedup

We

show

however

that

any

P-processor

execution

schedule

X

achieving

T

-LRB-

X

-RRB-

O

-LRB-

=P

-RRB-

must

use

space

S

-LRB-

X

-RRB-

W

-LRB-

-LRB-

P

-RRB-

-RRB-

Of

course

may

be

much

larger

than

and

hence

this

space

bound

is

nowhere

near

linear

in

its

space

expansion

A

proof

of

the

following

theorem

is

presented

in

Appendix

A.

Theorem

For

any

and

any

there

exists

a

depth-first

multithreaded

computation

with

work

average

parallelism

T

and

stack

depth

such

that

the

following

holds

For

any

number

P

of

processors

and

any

value

r

in

the

range

r

T

if

X

is

a

P-processor

execution

schedule

that

achieves

speedup

r?that

is

T

-LRB-

X

-RRB-

r?then

S

-LRB-

X

-RRB-

-LRB-

r

-RRB-

A

word

about

units

Space

and

S

-LRB-

X

-RRB-

is

measured

in

bytes

Time

T

and

T

-LRB-

X

-RRB-

is

measured

in

microseconds

The

inequalities

relating

space

to

time

all

carry

constants

with

appropriate

conversion

units

To

see

what

this

theorem

is

saying

consider

two

possible

values

of

r.

Think

of

the

work

as

a

very

large

value

so

that

the

average

parallelism

T

-LRB-

-RRB-

W

-LRB-

-RRB-

is

also

reasonably

large

Achieving

a

speedup

of

should

be

easy

But

if

we

plug

r

into

our

bound

we

get

S

-LRB-

X

-RRB-

for

any

execution

schedule

X

Thus

with

processors

we

have

the

space-per-processor

growing

proportional

to

despite

the

fact

the

may

be

a

small

constant

And

suppose

we

want

to

realize

speedup

out

of

all

of

the

parallelism

Plugging

r

W

-LRB-

-RRB-

into

our

lower

bound

we

get

S

-LRB-

X

-RRB-

W

-LRB-

-RRB-

With

processors

we

again

have

the

space-per-processor

growing

proportional

to

In

more

recent

work

Frigo

using

a

somewhat

more

general

model

of

multithreaded

computation

coupled

with

the

same

technique

as

used

in

the

proof

of

Theorem

has

obtained

the

following

stronger

result

-LSB-

-RSB-

Theorem

-LRB-

Frigo

-RRB-

For

any

sufficiently

large

and

any

r

there

exists

a

multithreaded

computation

with

work

and

space

requirement

=O

-LRB-

-RRB-

such

that

any

execution

schedule

X

that

achieves

speedup

r

must

use

space

S

-LRB-

X

-RRB-

W

-LRB-

-RRB-

The

strictness

condition

With

Frigo

's

construction

just

realizing

a

speedup

of

requires

space-per-processor

that

grows

linearly

with

the

work

We

prove

this

lower

bound

by

constructing

a

depth-first

multithreaded

computation

with

the

desired

properties

and

then

proving

that

it

has

no

efficient

execution

schedule

Though

we

defer

the

proof

of

Theorem

to

Appendix

A

the

idea

embodied

in

the

constructed

computation

is

as

follows

Each

processor

working

on

a

thread

of

the

computation

spawns

several

child

threads

all

of

which

subsequently

stall

on

unresolved

dependencies

In

order

to

achieve

speedup

a

scheduler

can

not

allow

all

of

these

processors

to

sit

idle

so

some

of

them

must

find

other

ready

threads

to

work

on

Again

these

threads

spawn

several

children

and

then

they

all

stall

This

process

repeats

Thus

to

realize

any

speedup

the

scheduler

must

allow

processors

to

continually

spawn

new

threads

even

though

these

newly

spawned

threads

quickly

stall

still

holding

their

space

resources

A

good

scheduler

should

not

allocate

resources

until

it

is

ready

to

utilize

those

resources

and

a

multithreaded

computation

should

not

be

structured

in

such

a

way

as

to

force

a

scheduler

into

such

an

overcommitted

situation

in

order

to

achieve

speedup

The

strictness

condition

A

strict

multithreaded

computation

contains

dependency

edges

that

as

we

shall

see

forbid

the

allocation

of

resources

until

those

resources

can

be

utilized

Specifically

in

a

strict

multithreaded

computation

every

dependency

edge

goes

from

a

thread

to

one

of

its

ancestor

threads

For

example

the

computation

shown

in

Figure

-LRB-

a

-RRB-

is

nonstrict

since

the

bold

dependencies

which

we

refer

to

as

nonstrict

dependencies

violate

the

strictness

condition

But

by

replacing

these

nonstrict

dependencies

with

new

strict

ones

we

obtain

the

strict

computation

shown

in

Figure

-LRB-

b

-RRB-

This

strictness

condition

has

the

following

interpretation

in

terms

of

argument

passing

in

functional

programs

For

any

thread

G

we

define

the

subcomputation

rooted

at

G

as

G

and

all

of

its

descendant

threads

See

Figure

We

think

of

the

subcomputation

as

a

function

invocation

For

any

subcomputation

we

can

think

of

each

edge

from

a

thread

outside

the

subcomputation

to

a

thread

inside

the

subcomputation

as

passing

arguments

to

the

function

invocation

In

the

case

of

a

strict

computation

each

subcomputation

has

exactly

one

such

edge?the

spawn

edge

from

its

parent

Thus

in

a

strict

computation

no

function

can

be

invoked

until

all

of

its

arguments

have

been

evaluated

although

the

arguments

can

be

evaluated

in

parallel

In

other

words

a

strict

computation

requires

a

strict

evaluation

order

-LRB-

as

opposed

to

a

lenient

evaluation

order

-RRB-

-LSB-

-RSB-

In

later

chapters

we

will

also

consider

fully

strict

multithreaded

computations

In

a

fully

strict

computation

every

dependency

goes

from

a

thread

to

its

parent

The

strict

computation

in

Figure

-LRB-

b

-RRB-

is

also

fully

strict

Fully

strict

computations

are

well-structured

in

that

all

dependencies

from

a

subcomputation

emanate

from

the

subcomputation

's

root

thread

In

the

remainder

of

this

section

we

show

that

strict

computations

are

depth-first

and

that

any

depth-first

computation

can

be

made

strict

-LRB-

though

possibly

at

a

huge

cost

in

lost

parallelism

-RRB-

We

begin

by

showing

that

strict

computations

are

depth-first

Chapter

Strict

multithreaded

computations

-LRB-

a

-RRB-

Nonstrict

-LRB-

b

-RRB-

Strict

Figure

-LRB-

a

-RRB-

This

multithreaded

computation

-LRB-

the

same

as

Figure

on

page

-RRB-

is

nonstrict

since

it

has

nonstrict

dependencies

shown

bold

that

go

to

non-ancestor

threads

-LRB-

b

-RRB-

If

we

replace

the

nonstrict

dependencies

with

new

strict

ones

shown

bold

we

obtain

a

strict

computation

since

all

dependencies

go

from

a

child

thread

to

an

ancestor

thread

The

strictness

condition

AAA

AAA

AAA

AAA

AAA

AAA

AAAA

AAAA

AAAA

AAAA

AAAAAA

AAAAAA

AAAAA

AAAAA

AAAAA

AAAAA

AAAAA

AAAAA

AAAAAA

AAAAAA

AAAAAA

AAAAAA

AA

AA

Figure

The

spawn

edge

-LRB-

-RRB-

and

the

dependency

edges

-LRB-

-RRB-

and

-LRB-

-RRB-

pass

arguments

into

the

subcomputation

rooted

at

thread

This

computation

is

not

strict

Theorem

Every

strict

multithreaded

computation

is

a

depth-first

computation

Proof

Consider

any

strict

multithreaded

computation

and

let

vn

-LRB-

with

n

-RRB-

denote

the

left-to-right

depth-first

ordering

of

the

instructions

To

prove

that

the

computation

is

depth-first

we

must

show

that

every

dependency

edge

is

consistent

with

this

ordering

In

other

words

we

must

show

that

for

every

dependency

edge

-LRB-

v

j

vk

-RRB-

we

have

j

k.

Consider

such

an

edge

and

let

G

denote

the

thread

containing

instruction

vk

See

Figure

The

strictness

conditions

says

that

the

thread

containing

v

j

must

be

a

descendant

of

G

so

let

vi

be

the

instruction

of

G

that

spawns

the

subcomputation

containing

v

j

Observe

that

the

left-to-right

depth-first

order

numbers

every

instruction

in

this

subcomputation

less

than

every

instruction

to

the

right

of

vi

in

G.

Thus

since

v

j

is

in

this

subcomputation

we

only

need

to

show

that

vk

is

to

the

right

of

vi

in

G.

vivjvkG

Figure

Every

strict

computation

is

depth-first

since

a

strict

dependency

edge

such

as

-LRB-

v

j

vk

-RRB-

must

go

to

an

instruction

after

instruction

vi

in

G

to

avoid

introducing

a

cycle

Now

observe

that

there

is

a

path

from

every

instruction

in

G

to

the

left

of

-LRB-

and

including

-RRB-

vi

to

vk

the

path

follows

continue

edges

in

G

to

vi

then

it

follows

spawn

and

continue

edges

Chapter

Strict

multithreaded

computations

down

to

v

j

and

then

it

follows

our

dependency

edge

-LRB-

v

j

vk

-RRB-

Thus

to

prevent

a

cycle

vk

must

be

to

the

right

of

vi

The

transformation

illustrated

in

Figure

from

a

depth-first

multithreaded

computation

to

a

strict

multithreaded

computation

is

called

strictifying

and

we

say

that

the

resulting

computation

is

the

strictification

of

the

original

For

a

given

computation

G

the

computation

is

the

strictification

of

G

if

is

a

strict

computation

differing

from

G

only

in

its

dependency

edges

and

for

every

dependency

edge

-LRB-

vi

v

j

-RRB-

in

G

there

is

a

path

from

vi

to

v

j

in

the

dag

of

This

latter

condition

ensures

that

is

at

least

as

strong

as

G

in

the

sense

that

any

execution

schedule

for

is

also

an

execution

schedule

for

G

We

now

show

that

for

any

depth-first

computation

G

its

strictification

exists

Theorem

Every

depth-first

multithreaded

computation

can

be

strictified

Proof

We

strictify

a

depth-first

computation

by

replacing

each

nonstrict

edge

with

a

strict

one

Specifically

let

-LRB-

v

j

vl

-RRB-

denote

a

nonstrict

edge

See

Figure

Let

G

be

the

leastcommon

ancestor

thread

of

the

threads

containing

v

j

and

vl

Then

let

vi

denote

the

instruction

of

G

that

spawns

the

subcomputation

containing

v

j

and

let

vk

denote

the

instruction

of

G

that

spawns

the

subcomputation

containing

vl

Note

that

vi

and

vk

must

be

different

since

we

only

allow

a

single

instruction

to

perform

a

single

spawn

and

G

is

defined

as

the

least-common

ancestor

of

two

different

threads

To

strictify

the

computation

we

replace

dependency

edge

-LRB-

v

j

vl

-RRB-

with

a

new

dependency

edge

-LRB-

v

j

vk

-RRB-

and

repeat

for

every

such

nonstrict

edge

Such

a

replacement

strengthens

the

dag

since

there

is

a

path

of

spawn

and

continue

edges

from

vk

to

vl

vjvlvivkG

Figure

Every

depth-first

computationcan

be

strictified

by

replacing

every

nonstrict

edge

-LRB-

v

j

vl

-RRB-

with

a

strict

edge

-LRB-

v

j

vk

-RRB-

We

must

show

that

this

transformation

produces

a

strict

computation

After

the

transformation

every

dependency

edge

satisfies

the

strictness

condition

every

dependency

edge

goes

from

a

thread

to

one

of

its

ancestors

Thus

we

only

need

to

show

that

each

replacement

does

not

introduce

any

cycles

in

the

dag

In

fact

we

show

the

stronger

property

that

each

replacement

preserves

the

depth-first

property

Recall

that

the

left-to-right

depth-first

order

vn

is

defined

only

in

terms

of

the

continue

and

spawn

edges

so

it

does

not

change

when

we

replace

a

dependency

edge

When

we

replace

dependency

edge

-LRB-

v

j

vl

-RRB-

The

busy-leaves

property

with

-LRB-

v

j

vk

-RRB-

we

must

show

that

j

l

implies

j

k.

With

j

l

we

must

have

vk

-LRB-

the

instruction

that

spawns

the

subcomputation

containing

vl

-RRB-

to

the

right

of

vi

-LRB-

the

instruction

that

spawns

the

subcomputation

containing

v

j

-RRB-

in

G.

And

thus

we

have

j

k

In

some

cases

strictification

may

reduce

the

parallelism

by

increasing

the

critical

path

length

For

example

the

strictified

computation

of

Figure

-LRB-

b

-RRB-

has

a

critical

path

length

of

instructions

whereas

our

original

example

-LRB-

nonstrict

-RRB-

computation

has

a

critical

path

length

of

instructions

On

the

other

hand

many

computations

such

as

those

derived

from

divide-and-conquer

game-tree

search

and

many

other

programs

are

naturally

strict

and

still

have

very

short

critical

paths

In

Chapter

when

we

look

at

the

Cilk

language

and

runtime

system

we

shall

see

programs

that

give

rise

to

strict

computations

with

more

than

ten-thousand-fold

parallelism

For

now

we

focus

on

exploiting

the

beneficial

structure

of

strict

computations

to

compute

efficient

execution

schedules

The

busy-leaves

property

Once

a

thread

G

has

been

spawned

in

a

strict

computation

a

single

processor

can

complete

the

execution

of

the

entire

subcomputation

rooted

at

G

even

if

no

other

progress

is

made

on

other

parts

of

the

computation

In

other

words

from

the

time

the

thread

G

is

spawned

until

the

time

G

dies

there

is

always

at

least

one

thread

from

the

subcomputation

rooted

at

G

that

is

ready

In

particular

no

leaf

thread

in

a

strict

multithreaded

computation

can

stall

As

we

shall

see

this

property

allows

an

execution

schedule

to

keep

the

leaves

busy

By

combining

this

busy-leaves

property

with

the

greedy

property

we

derive

execution

schedules

that

simultaneously

exhibit

linear

speedup

and

linear

expansion

of

space

In

this

section

we

show

that

for

any

number

P

of

processors

and

any

strict

multithreaded

computation

with

work

critical

path

length

T

and

stack

depth

there

exists

a

P

processor

execution

schedule

X

that

achieves

time

T

-LRB-

X

-RRB-

=P

T

and

space

S

-LRB-

X

-RRB-

simultaneously

We

give

a

simple

online

P-processor

parallel

algorithm?the

Busy-Leaves

Algorithm?to

compute

such

a

schedule

The

Busy-Leaves

Algorithm

operates

online

in

the

following

sense

Before

the

tth

step

the

algorithm

has

computed

and

executed

the

first

t

steps

of

the

execution

schedule

At

the

tth

step

the

algorithm

uses

only

information

from

the

portion

of

the

computation

revealed

so

far

in

the

execution

to

compute

and

execute

the

tth

step

of

the

schedule

In

particular

it

does

not

use

any

information

from

instructions

not

yet

executed

or

threads

not

yet

spawned

The

Busy-Leaves

Algorithm

maintains

all

living

threads

in

a

single

thread

pool

which

is

uniformly

available

to

all

P

processors

When

spawns

occur

new

threads

are

added

to

this

global

pool

and

when

a

processor

needs

work

it

removes

a

ready

thread

from

the

pool

Though

we

describe

the

algorithm

as

a

P-processor

parallel

algorithm

we

shall

not

analyze

it

as

such

Specifically

in

computing

the

tth

step

of

the

schedule

we

allow

each

processor

to

add

threads

to

the

thread

pool

and

delete

threads

from

it

Thus

we

ignore

the

effects

of

processors

contending

for

access

to

the

pool

In

fact

we

shall

only

analyze

properties

of

Chapter

Strict

multithreaded

computations

the

schedule

itself

and

ignore

the

cost

incurred

by

the

algorithm

in

computing

the

schedule

Scheduling

overheads

come

into

play

in

the

next

chapter

The

Busy-Leaves

Algorithm

operates

as

follows

The

algorithm

begins

with

the

root

thread

in

the

global

thread

pool

and

all

processors

idle

At

the

beginning

of

each

step

each

processor

either

is

idle

or

has

a

thread

to

work

on

Those

processors

that

are

idle

begin

the

step

by

attempting

to

remove

any

ready

thread

from

the

pool

If

there

are

sufficiently

many

ready

threads

in

the

pool

to

satisfy

all

of

the

idle

processors

then

every

idle

processor

gets

a

ready

thread

to

work

on

Otherwise

some

processors

remain

idle

Then

each

processor

that

has

a

thread

to

work

on

executes

the

next

instruction

from

that

thread

In

general

once

a

processor

has

a

thread

call

it

Ga

to

work

on

it

executes

an

instruction

from

Ga

at

each

step

until

the

thread

either

spawns

stalls

or

dies

in

which

case

it

performs

according

to

the

following

rules

Spawns

If

the

thread

Ga

spawns

a

child

Gb

then

the

processor

finishes

the

current

step

by

returning

Ga

to

the

thread

pool

The

processor

begins

the

next

step

working

on

Gb

Stalls

If

the

thread

Ga

stalls

then

the

processor

finishes

the

current

step

by

returning

Ga

to

the

thread

pool

The

processor

begins

the

next

step

idle

Dies

If

the

thread

Ga

dies

then

the

processor

finishes

the

current

step

by

checking

to

see

if

Ga

's

parent

thread

Gb

currently

has

any

living

children

If

Gb

has

no

live

children

and

no

other

processor

is

working

on

Gb

then

the

processor

takes

Gb

from

the

pool

and

begins

the

next

step

working

on

Gb

Otherwise

the

processor

begins

the

next

step

idle

Figure

illustrates

these

three

rules

in

a

execution

schedule

computed

by

the

Busy-Leaves

Algorithm

on

the

strict

computation

of

Figure

-LRB-

b

-RRB-

Rule

At

step

processor

working

on

thread

executes

which

spawns

the

child

so

places

back

in

the

pool

-LRB-

to

be

picked

up

at

the

beginning

of

the

next

step

by

the

idle

-RRB-

and

begins

the

next

step

working

on

Rule

At

step

processor

working

on

thread

executes

and

stalls

so

returns

to

the

pool

and

begins

the

next

step

idle

-LRB-

and

remains

idle

since

the

thread

pool

contains

no

ready

threads

-RRB-

Rule

At

step

processor

working

on

executes

and

dies

so

retrieves

the

parent

from

the

pool

and

begins

the

next

step

working

on

Besides

being

greedy

for

any

strict

computation

the

schedule

computed

by

the

BusyLeaves

Algorithm

maintains

the

busy-leaves

property

at

every

time

step

during

the

execution

every

leaf

in

the

spawn

subtree

has

a

processor

working

on

it

In

other

words

at

every

time

step

every

living

thread

that

has

no

living

descendants

has

a

processor

working

on

it

We

shall

now

prove

this

fact

and

show

that

it

implies

linear

expansion

of

space

We

begin

by

showing

that

any

schedule

that

maintains

the

busy-leaves

property

exhibits

linear

expansion

of

space

The

space

bound

of

the

following

lemma

accounts

for

any

stack-like

memory

In

particular

we

allow

any

instruction

to

allocate

memory

for

its

thread

provided

that

the

instruction

can

not

be

executed

at

a

time

step

when

its

thread

has

a

living

child

We

allow

any

instruction

to

deallocate

memory

for

its

thread

Additionally

we

require

that

all

memory

allocated

for

a

The

busy-leaves

property

processor

activity

step

thread

pool

Figure

A

execution

schedule

computed

by

the

Busy-Leaves

Algorithm

for

the

strict

computation

of

Figure

-LRB-

b

-RRB-

This

schedule

lists

the

living

threads

in

the

global

thread

pool

at

each

step

just

after

each

idle

processor

has

removed

a

ready

thread

It

also

lists

the

ready

thread

being

worked

on

and

the

instruction

executed

by

each

of

the

processors

and

at

each

step

Living

threads

in

the

pool

that

are

ready

are

listed

in

bold

The

other

living

threads

are

stalled

thread

is

deallocated

before

the

thread

dies

At

any

given

time

step

during

the

execution

the

amount

of

memory

currently

allocated

for

a

given

living

thread

is

the

net

memory

allocated

for

the

thread

by

all

instructions

that

have

been

executed

The

memory

for

a

thread

need

not

be

allocated

in

a

single

contiguous

chunk

Nevertheless

we

may

continue

to

think

of

the

memory

allocated

for

a

thread

as

being

part

of

an

activation

frame

though

now

the

frame

may

grow

and

shrink

as

the

thread

executes

The

memory

is

still

stack-like

because

we

allow

the

frame

to

grow

only

when

the

thread

can

not

have

any

living

children

The

stack

depth

of

a

computation

is

the

amount

of

memory

used

by

the

left-to-right

depth-first

execution

Lemma

For

any

multithreaded

computation

with

stack

depth

any

P-processor

execution

schedule

X

that

maintains

the

busy-leaves

property

uses

space

bounded

by

S

-LRB-

X

-RRB-

Proof

The

busy-leaves

property

implies

that

at

all

time

steps

t

the

spawn

subtree

has

at

most

P

leaves

We

bound

the

space

in

use

at

time

step

t

by

assigning

every

living

thread

to

a

leaf

thread

and

then

showing

that

the

total

space

currently

allocated

for

all

the

threads

assigned

to

a

given

leaf

is

at

most

For

each

living

thread

we

assign

it

to

a

leaf

as

follows

If

the

thread

is

a

leaf

then

it

is

assigned

to

itself

Otherwise

the

thread

is

assigned

to

the

same

leaf

as

its

rightmost

living

child

-LRB-

though

actually

we

can

choose

any

living

child

-RRB-

Thus

the

threads

assigned

to

any

given

leaf

are

all

ancestors

of

that

leaf

Chapter

Strict

multithreaded

computations

Now

consider

any

leaf

thread

G

and

any

ancestor

thread

that

is

assigned

to

G.

Let

v

denote

the

instruction

of

G

executed

at

this

time

step

t

-LRB-

recall

that

G

is

busy

-RRB-

and

let

denote

the

instruction

of

that

spawned

the

subcomputation

containing

G

We

claim

that

the

amount

of

memory

currently

allocated

for

is

no

more

than

the

amount

of

memory

allocated

for

at

the

time

step

in

the

left-to-right

depth-first

execution

when

the

processor

executes

instruction

v

of

thread

G

We

verify

the

claim

by

observing

that

every

instruction

in

to

the

left

of

has

been

executed

and

though

there

may

also

be

some

instructions

to

the

right

of

that

have

been

executed

of

these

latter

instructions

may

allocate

memory

since

they

must

have

been

executed

while

had

a

living

child

As

for

thread

G

itself

the

amount

of

memory

currently

allocated

for

it

is

equal

to

the

amount

of

memory

allocated

for

it

at

the

time

step

in

the

execution

when

the

processor

executes

instruction

v.

Thus

the

total

memory

currently

allocated

for

all

threads

assigned

to

a

given

leaf

is

at

most

equal

to

the

amount

of

memory

in

use

by

a

execution

when

the

processor

is

executing

the

given

leaf

and

this

amount

of

memory

is

at

most

With

a

maximum

of

P

leaf

threads

the

total

memory

currently

in

use

is

at

most

and

this

bound

holds

for

every

time

step

The

bound

for

schedules

that

maintain

the

busy-leaves

property

is

conservative

By

charging

space

for

each

busy

leaf

we

may

be

overcharging

For

some

computations

by

knowing

that

the

schedule

preserves

the

busy-leaves

property

we

can

appeal

directly

to

the

fact

that

the

spawn

subtree

never

has

more

than

P

leaves

to

obtain

tight

bounds

on

space

usage

-LSB-

-RSB-

We

finish

this

chapter

by

showing

that

for

strict

computations

the

Busy-Leaves

Algorithm

computes

a

schedule

that

is

both

greedy

and

maintains

the

busy-leaves

property

Thus

we

show

that

every

strict

computation

has

execution

schedules

that

are

simultaneously

efficient

with

respect

to

time

and

space

Theorem

For

any

number

P

of

processors

and

any

strict

multithreaded

computation

with

work

critical

path

length

T

and

stack

depth

the

Busy-Leaves

Algorithm

computes

a

P-processor

execution

schedule

X

whose

execution

time

satisfies

T

-LRB-

X

-RRB-

=P

T

and

whose

space

satisfies

S

-LRB-

X

-RRB-

Proof

The

time

bound

follows

directly

from

the

greedy-scheduling

theorem

-LRB-

Theorem

page

-RRB-

since

the

Busy-Leaves

Algorithm

computes

a

greedy

schedule

The

space

bound

follows

from

Lemma

if

we

can

show

that

the

Busy-Leaves

Algorithm

maintains

the

busy-leaves

property

We

prove

this

fact

by

induction

on

the

number

of

steps

At

the

first

step

of

the

algorithm

the

spawn

subtree

contains

just

the

root

thread

which

is

a

leaf

and

some

processor

is

working

on

it

We

must

show

that

all

of

the

algorithm

rules

preserve

the

busy-leaves

property

When

a

processor

has

a

thread

Ga

to

work

on

it

executes

instructions

from

that

thread

until

it

either

spawns

stalls

or

dies

Rule

If

Ga

spawns

a

child

Gb

then

Ga

is

not

a

leaf

-LRB-

even

if

it

was

before

-RRB-

and

Gb

is

a

leaf

In

this

case

the

processor

works

on

Gb

so

the

new

leaf

is

busy

Rule

If

Ga

stalls

then

Ga

can

not

be

a

leaf

since

in

a

strict

computation

the

unresolved

dependency

must

come

from

a

descendant

Rule

If

Ga

dies

The

busy-leaves

property

then

its

parent

thread

Gb

may

turn

into

a

leaf

In

this

case

the

processor

works

on

Gb

unless

some

other

processor

already

is

so

the

new

leaf

is

guaranteed

to

be

busy

We

now

know

that

every

strict

multithreaded

computatation

has

an

efficient

execution

schedule

and

we

know

how

to

find

it

But

these

facts

take

us

only

so

far

Execution

schedules

must

be

computed

efficiently

online

and

though

the

Busy-Leaves

Algorithm

does

compute

efficient

execution

schedules

and

does

operate

online

it

surely

does

not

do

so

efficiently

except

possibly

in

the

case

of

small-scale

symmetric

multiprocessors

This

lack

of

scalability

is

a

consequence

of

employing

a

single

centralized

thread

pool

at

which

all

processors

must

contend

for

access

In

the

next

chapter

we

present

a

distributed

online

scheduling

algorithm

and

we

prove

that

it

is

both

efficient

and

scalable

Chapter

Work

stealing

To

execute

a

multithreaded

computation

on

a

parallel

computer

efficiently

a

scheduler

must

simultaneously

keep

the

processors

busy

doing

work

maintain

memory

usage

within

reasonable

limits

and

avoid

communication

to

the

greatest

extent

possible

In

this

chapter

we

give

the

first

provably

efficient

online

scheduling

algorithm

for

multithreaded

computations

with

dependencies

This

algorithm

is

based

on

the

technique

of

random

work

stealing

in

which

processors

needing

work

steal

computational

threads

from

other

processors

chosen

at

random

We

show

analytically

that

for

fully

strict

computations

this

algorithm

is

simultaneously

efficient

with

respect

to

time

space

and

communication

In

Section

we

present

the

randomized

work-stealing

algorithm

give

an

important

structural

lemma

and

bound

the

space

used

by

the

algorithm

In

Section

we

give

the

model

that

we

use

to

analyze

access

contention

in

the

algorithm

and

we

give

a

bound

on

the

delay

incurred

by

random

accesses

in

this

model

We

use

the

structural

lemma

of

Section

and

the

bound

of

Section

in

Section

to

prove

analytically

that

for

fully

strict

multithreaded

computations

this

work-stealing

algorithm

achieves

linear

speedup

with

little

communication

The

Cilk

runtime

system

-LRB-

Chapter

-RRB-

implements

this

work-stealing

algorithm

and

empirically

demonstrates

its

efficiency

A

randomized

work-stealing

algorithm

In

this

section

we

present

an

online

randomized

work-stealing

algorithm

for

scheduling

multithreaded

computations

on

a

parallel

computer

Also

we

present

an

important

structural

lemma

which

is

used

at

the

end

of

this

section

to

show

that

for

fully

strict

computations

this

algorithm

causes

at

most

a

linear

expansion

of

space

This

lemma

reappears

in

Section

to

show

that

for

fully

strict

computations

this

algorithm

achieves

linear

speedup

and

generates

existentially

optimal

amounts

of

communication

In

the

Work-Stealing

Algorithm

the

centralized

thread

pool

of

the

Busy-Leaves

Algorithm

is

distributed

across

the

processors

Specifically

each

processor

maintains

a

ready

deque

data

structure

of

threads

The

ready

deque

has

two

ends

a

top

and

a

bottom

Threads

can

be

inserted

on

the

bottom

and

removed

from

either

end

A

processor

treats

its

ready

The

research

reported

in

this

chapter

is

joint

work

with

Charles

Leiserson

of

MIT

's

Laboratory

for

Computer

Science

and

was

first

published

in

-LSB-

-RSB-

Chapter

Work

stealing

deque

like

a

call

stack

pushing

and

popping

from

the

bottom

Threads

that

are

migrated

to

other

processors

are

removed

from

the

top

In

general

a

processor

obtains

work

by

removing

the

thread

at

the

bottom

of

its

ready

deque

It

starts

working

on

the

thread

call

it

Ga

and

continues

executing

Ga

's

instructions

until

Ga

spawns

stalls

dies

or

enables

a

stalled

thread

in

which

case

it

performs

according

to

the

following

rules

Spawns

If

the

thread

Ga

spawns

a

child

Gb

then

Ga

is

placed

on

the

bottom

of

the

ready

deque

and

the

processor

commences

work

on

Gb

Stalls

If

the

thread

Ga

stalls

its

processor

checks

the

ready

deque

If

the

deque

contains

any

threads

then

the

processor

removes

and

begins

work

on

the

bottommost

thread

If

the

ready

deque

is

empty

however

the

processor

begins

work

stealing

it

steals

the

topmost

thread

from

the

ready

deque

of

a

randomly

chosen

processor

and

begins

work

on

it

-LRB-

This

work-stealing

strategy

is

elaborated

below

-RRB-

Dies

If

the

thread

Ga

dies

then

the

processor

follows

rule

as

in

the

case

of

Ga

stalling

Enables

If

the

thread

Ga

enables

a

stalled

thread

Gb

the

now-ready

thread

Gb

is

placed

on

the

bottom

of

the

ready

deque

of

Ga

's

processor

A

thread

can

simultaneously

enable

a

stalled

thread

and

die

in

which

case

we

first

perform

rule

for

enabling

and

then

rule

for

dying

Except

for

rule

for

the

case

when

a

thread

enables

a

stalled

thread

these

rules

are

analogous

to

the

rules

of

the

Busy-Leaves

Algorithm

and

as

we

shall

see

rule

is

needed

to

ensure

that

the

algorithm

maintains

important

structural

properties

including

the

busy-leaves

property

The

Work-Stealing

Algorithm

begins

with

all

ready

deques

empty

The

root

thread

of

the

multithreaded

computation

is

placed

in

the

ready

deque

of

one

processor

while

the

other

processors

start

work

stealing

When

a

processor

begins

work

stealing

it

operates

as

follows

The

processor

becomes

a

thief

and

attempts

to

steal

work

from

a

victim

processor

chosen

uniformly

at

random

The

thief

queries

the

ready

deque

of

the

victim

and

if

it

is

nonempty

the

thief

removes

and

begins

work

on

the

top

thread

If

the

victim

's

ready

deque

is

empty

however

the

thief

tries

again

picking

another

victim

at

random

We

now

state

and

prove

an

important

lemma

on

the

structure

of

threads

in

the

ready

deque

of

any

processor

during

the

execution

of

a

fully

strict

computation

This

lemma

is

used

later

in

this

section

to

analyze

execution

space

and

in

Section

to

analyze

execution

time

and

communication

Figure

illustrates

the

lemma

Lemma

In

the

execution

of

any

fully

strict

multithreaded

computation

by

the

WorkStealing

Algorithm

consider

any

processor

p

and

any

given

time

step

at

which

p

is

working

on

a

thread

Let

be

the

thread

that

p

is

working

on

let

k

be

the

number

of

threads

in

p

's

ready

deque

and

let

Gk

denote

the

threads

in

p

's

ready

deque

ordered

from

bottom

to

top

so

that

is

the

bottommost

and

Gk

is

the

topmost

If

we

have

k

then

the

threads

in

p

's

ready

deque

satisfy

the

following

properties

A

randomized

work-stealing

algorithm

currently

executing

thread

ready

deque

Figure

The

structure

of

a

processor

's

ready

deque

The

black

instruction

in

each

thread

indicates

the

thread

's

currently

ready

instruction

Only

thread

Gk

may

have

been

worked

on

since

it

last

spawned

a

child

The

dashed

edges

are

the

deque

edges

introduced

in

Section

For

i

k

thread

Gi

is

a

child

of

Gi

If

we

have

k

then

for

i

k

thread

Gi

has

not

been

worked

on

since

it

spawned

Gi

Proof

The

proof

is

a

straightforward

induction

on

execution

time

Execution

begins

with

the

root

thread

in

some

processor

's

ready

deque

and

all

other

ready

deques

empty

so

the

lemma

vacuously

holds

at

the

outset

Now

consider

any

step

of

the

algorithm

at

which

processor

p

executes

an

instruction

from

thread

Let

Gk

denote

the

k

threads

in

p

's

ready

deque

before

the

step

and

suppose

that

either

k

or

both

properties

hold

Let

denote

the

thread

-LRB-

if

any

-RRB-

being

worked

on

by

p

after

the

step

and

let

denote

the

threads

in

p

's

ready

deque

after

the

step

We

now

look

at

the

rules

of

the

algorithm

and

show

that

they

all

preserve

the

lemma

That

is

either

or

both

properties

hold

after

the

step

Rule

If

spawns

a

child

then

p

pushes

onto

the

bottom

of

the

ready

deque

and

commences

work

on

the

child

Thus

is

the

child

we

have

k

and

for

j

we

have

G

j

See

Figure

Now

we

can

check

both

properties

Property

If

then

for

j

thread

is

a

child

of

since

before

the

spawn

we

have

k

which

means

that

for

i

k

thread

Gi

is

a

child

of

Gi

Moreover

is

obviously

a

child

of

Property

If

then

for

j

thread

has

not

been

worked

on

since

it

spawned

because

before

the

spawn

we

have

k

which

means

that

for

i

k

thread

Gi

has

not

been

worked

on

since

it

spawned

Gi

Finally

thread

has

not

been

worked

on

since

it

spawned

because

the

spawn

only

just

occurred

Rules

and

If

stalls

or

dies

then

we

have

two

cases

to

consider

If

k

the

ready

deque

is

empty

so

the

processor

commences

work

stealing

and

when

the

processor

Chapter

Work

stealing

-LRB-

a

-RRB-

Before

spawn

G?k

G

G

G

G

-LRB-

b

-RRB-

After

spawn

Figure

The

ready

deque

of

a

processor

before

and

after

the

thread

that

it

is

working

on

spawns

a

child

steals

and

begins

work

on

a

thread

we

have

If

k

the

ready

deque

is

not

empty

so

the

processor

pops

the

bottommost

thread

off

the

deque

and

commences

work

on

it

Thus

we

have

-LRB-

the

popped

thread

-RRB-

and

k

and

for

j

we

have

G

j

See

Figure

Now

if

we

can

check

both

properties

Property

For

j

thread

is

a

child

of

since

for

i

k

thread

Gi

is

a

child

of

Gi

Property

If

then

for

j

thread

has

not

been

worked

on

since

it

spawned

because

before

the

death

we

have

k

which

means

that

for

i

k

thread

Gi

has

not

been

worked

on

since

it

spawned

Gi

-LRB-

a

-RRB-

Before

death

G?k

G

G

-LRB-

b

-RRB-

After

death

Figure

The

ready

deque

of

a

processor

before

and

after

the

thread

that

it

is

working

on

dies

Rule

If

enables

a

stalled

thread

then

due

to

the

fully

strict

condition

that

previously

stalled

thread

must

be

's

parent

There

are

two

cases

to

consider

If

k

then

the

processor

's

ready

deque

is

not

empty

and

this

parent

thread

must

be

bottommost

in

the

Atomic

messages

and

the

recycling

game

ready

deque

In

this

case

the

processor

does

nothing

If

k

then

the

ready

deque

is

empty

and

the

processor

places

the

parent

thread

on

the

bottom

of

the

ready

deque

In

this

case

we

have

and

k

with

denoting

the

newly

enabled

parent

We

only

have

to

check

the

first

property

Property

Thread

is

obviously

a

child

of

If

some

other

processor

steals

a

thread

from

processor

p

then

we

must

have

k

and

after

the

steal

we

have

k

If

holds

then

both

properties

are

clearly

preserved

All

other

actions

by

processor

p?such

as

work

stealing

or

executing

an

instruction

that

does

not

invoke

any

of

the

above

rules?clearly

preserve

the

lemma

Before

moving

on

it

is

worth

pointing

out

how

it

may

happen

that

thread

Gk

has

been

worked

on

since

it

spawned

Gk

since

Property

excludes

Gk

This

situation

arises

when

Gk

is

stolen

from

processor

p

and

then

stalls

on

its

new

processor

Later

Gk

is

reenabled

by

Gk

and

brought

back

to

processor

p

's

ready

deque

The

key

observation

is

that

when

Gk

is

reenabled

processor

p

's

ready

deque

is

empty

and

p

is

working

on

Gk

The

other

threads

Gk

Gk

shown

in

Figure

were

spawned

after

Gk

was

reenabled

We

conclude

this

section

by

bounding

the

space

used

by

the

Work-Stealing

Algorithm

executing

a

fully

strict

computation

This

bound

accounts

for

all

stack-like

memory

Theorem

For

any

fully

strict

multithreaded

computation

with

stack

depth

the

WorkStealing

Algorithm

run

on

a

computer

with

P

processors

uses

at

most

space

Proof

Like

the

Busy-Leaves

Algorithm

the

Work-StealingAlgorithm

maintains

the

busyleaves

property

at

every

time

step

of

the

execution

every

leaf

in

the

current

spawn

subtree

has

a

processor

working

on

it

If

we

can

establish

this

fact

then

Lemma

-LRB-

page

-RRB-

completes

the

proof

That

the

Work-Stealing

Algorithm

maintains

the

busy-leaves

property

is

a

simple

consequence

of

Lemma

At

every

time

step

every

leaf

in

the

current

spawn

subtree

must

be

ready

and

therefore

must

either

have

a

processor

working

on

it

or

be

in

the

ready

deque

of

some

processor

But

Lemma

guarantees

that

no

leaf

thread

sits

in

a

processor

's

ready

deque

while

the

processor

works

on

some

other

thread

With

the

space

bound

in

hand

we

now

turn

attention

to

analyzing

the

time

and

communication

bounds

for

the

Work-Stealing

Algorithm

Before

we

can

proceed

with

this

analysis

however

we

must

take

care

to

define

a

model

for

coping

with

the

contention

that

may

arise

when

multiple

thief

processors

simultaneously

attempt

to

steal

from

the

same

victim

Atomic

messages

and

the

recycling

game

This

section

presents

the

atomic-access

model

that

we

use

to

analyze

contention

during

the

execution

of

a

multithreaded

computation

by

the

Work-Stealing

Algorithm

We

introduce

a

combinatorial

balls

and

bins

game

which

we

use

to

bound

the

total

amount

of

delay

incurred

by

random

asynchronous

accesses

in

this

model

We

shall

use

the

results

of

this

section

in

Section

where

we

analyze

the

Work-Stealing

Algorithm

Chapter

Work

stealing

The

atomic-access

model

is

the

machine

model

we

use

to

analyze

the

Work-Stealing

Algorithm

We

assume

that

the

machine

is

an

asynchronous

parallel

computer

with

P

processors

and

its

memory

can

be

either

distributed

or

shared

Our

analysis

assumes

that

concurrent

accesses

to

the

same

data

structure

are

serially

queued

by

an

adversary

as

in

the

atomic

message-passing

model

of

-LSB-

-RSB-

This

assumption

is

more

stringent

than

that

in

the

model

of

Karp

and

Zhang

-LSB-

-RSB-

They

assume

that

if

concurrent

steal

requests

are

made

to

a

deque

in

one

time

step

one

request

is

satisfied

and

all

the

others

are

denied

In

the

atomicaccess

model

we

also

assume

that

one

request

is

satisfied

but

the

others

are

queued

by

an

adversary

rather

than

being

denied

Moreover

from

the

collection

of

waiting

requests

for

a

given

deque

the

adversary

gets

to

choose

which

is

serviced

and

which

continue

to

wait

The

only

constraint

on

the

adversary

is

that

if

there

is

at

least

one

request

for

a

deque

then

the

adversary

can

not

choose

that

be

serviced

The

main

result

of

this

section

is

to

show

that

if

requests

are

made

randomly

by

P

processors

to

P

deques

with

each

processor

allowed

at

most

one

outstanding

request

then

the

total

amount

of

time

that

the

processors

spend

waiting

for

their

requests

to

be

satisfied

is

likely

to

be

proportional

to

the

total

number

M

of

requests

no

matter

which

processors

make

the

requests

and

no

matter

how

the

requests

are

distributed

over

time

In

order

to

prove

this

result

we

introduce

a

balls

and

bins

game

that

models

the

effects

of

queueing

by

the

adversary

The

-LRB-

P

M

-RRB-

recycling

game

is

a

combinatorial

game

played

by

the

adversary

in

which

balls

are

tossed

at

random

into

bins

The

parameter

P

is

the

number

of

balls

in

the

game

which

is

equal

to

the

number

of

bins

The

parameter

M

is

the

total

number

of

ball

tosses

executed

by

the

adversary

Initially

all

P

balls

are

in

a

reservoir

separate

from

the

P

bins

At

each

step

of

the

game

the

adversary

executes

the

following

two

operations

in

sequence

The

adversary

chooses

some

of

the

balls

in

the

reservoir

-LRB-

possibly

all

and

possibly

-RRB-

and

then

for

each

of

these

balls

the

adversary

removes

it

from

the

reservoir

selects

one

of

the

P

bins

uniformly

and

independently

at

random

and

tosses

the

ball

into

it

The

adversary

inspects

each

of

the

P

bins

in

turn

and

for

each

bin

that

contains

at

least

one

ball

the

adversary

removes

any

one

of

the

balls

in

the

bin

and

returns

it

to

the

reservoir

The

adversary

is

permitted

to

make

a

total

of

M

ball

tosses

The

game

ends

when

M

ball

tosses

have

been

made

and

all

balls

have

been

removed

from

the

bins

and

placed

back

in

the

reservoir

The

recycling

game

models

the

servicing

of

steal

requests

by

the

Work-Stealing

Algorithm

We

can

view

each

ball

and

each

bin

as

being

owned

by

a

distinct

processor

If

a

ball

is

in

the

reservoir

it

means

that

the

ball

's

owner

is

not

making

a

steal

request

If

a

ball

is

in

a

bin

it

means

that

the

ball

's

owner

has

made

a

steal

request

to

the

deque

of

the

bin

's

owner

but

that

the

request

has

not

yet

been

satisfied

When

a

ball

is

removed

from

a

bin

and

returned

to

the

reservoir

it

means

that

the

request

has

been

serviced

After

each

step

t

of

the

game

there

are

some

number

nt

of

balls

left

in

the

bins

which

correspond

to

steal

requests

that

have

not

been

satisfied

We

shall

be

interested

in

the

total

delay

D

Tt

nt

where

T

is

the

total

number

of

steps

in

the

game

The

goal

of

the

adversary

is

to

make

the

total

delay

as

large

as

possible

The

next

lemma

shows

that

despite

the

Atomic

messages

and

the

recycling

game

choices

that

the

adversary

makes

about

which

balls

to

toss

into

bins

and

which

to

return

to

the

reservoir

the

total

delay

is

unlikely

to

be

large

Lemma

For

any

e

with

probability

at

least

e

the

total

delay

in

the

-LRB-

P

M

-RRB-

recycling

game

is

O

-LRB-

M+P

lg

P+P

lg

-LRB-

e

-RRB-

-RRB-

The

expected

total

delay

is

at

most

M

In

other

words

the

total

delay

incurred

by

M

random

requests

made

by

P

processors

in

the

atomicaccess

model

is

O

-LRB-

M+P

lgP

P

lg

-LRB-

e

-RRB-

-RRB-

with

probability

at

least

e

and

the

expected

total

delay

is

at

most

M.

Proof

We

first

make

the

observation

that

the

strategy

by

which

the

adversary

chooses

a

ball

from

each

bin

is

immaterial

and

thus

we

can

assume

that

balls

are

queued

in

their

bins

in

a

first-in-first-out

-LRB-

FIFO

-RRB-

order

The

adversary

removes

balls

from

the

front

of

the

queue

and

when

the

adversary

tosses

a

ball

it

is

placed

on

the

back

of

the

queue

If

several

balls

are

tossed

into

the

same

bin

at

the

same

step

they

can

be

placed

on

the

back

of

the

queue

in

any

order

The

reason

that

assuming

a

FIFO

discipline

for

queuing

balls

in

a

bin

does

not

affect

the

total

delay

is

that

the

number

of

balls

in

a

given

bin

at

a

given

step

is

the

same

no

matter

which

ball

is

removed

and

where

balls

are

tossed

has

nothing

to

do

with

which

ball

is

tossed

For

any

given

ball

and

any

given

step

the

step

either

finishes

with

the

the

ball

in

a

bin

or

in

the

reservoir

Define

the

delay

of

ball

r

to

be

the

random

variable

dr

denoting

the

total

number

of

steps

that

finish

with

ball

r

in

a

bin

Then

we

have

D

P

r

dr

-LRB-

-RRB-

Define

the

ith

cycle

of

a

ball

to

be

those

steps

in

which

the

ball

remains

in

a

bin

from

the

ith

time

it

is

tossed

until

it

is

returned

to

the

reservoir

Define

also

the

ith

delay

of

a

ball

to

be

the

number

of

steps

in

its

ith

cycle

We

shall

analyze

the

total

delay

by

focusing

without

loss

of

generality

on

the

delay

d

of

ball

If

we

let

m

denote

the

number

of

times

that

ball

is

tossed

by

the

adversary

and

for

i

m

let

di

be

the

random

variable

denoting

the

ith

delay

of

ball

then

we

have

d

mi

di

We

say

that

the

ith

cycle

of

ball

is

delayed

by

another

ball

r

if

the

ith

toss

of

ball

places

it

in

some

bin

k

and

ball

r

is

removed

from

bin

k

during

the

ith

cycle

of

ball

Since

the

adversary

follows

the

FIFO

rule

it

follows

that

the

ith

cycle

of

ball

can

be

delayed

by

another

ball

r

either

once

or

not

at

all

Consequently

we

can

decompose

each

random

variable

di

into

a

sum

di

xim

of

indicator

random

variables

where

xir

ae

if

the

ith

cycle

of

ball

is

delayed

by

ball

r

otherwise

Thus

we

have

d

m

i

P

r

xir

-LRB-

-RRB-

Plaxton

of

the

University

of

Texas

Austin

has

improved

this

bound

to

O

-LRB-

M

-RRB-

for

the

case

when

e

is

at

most

polynomial

in

M

and

P

-LSB-

-RSB-

Chapter

Work

stealing

We

now

prove

an

important

property

of

these

indicator

random

variables

Consider

any

set

S

of

pairs

-LRB-

i

r

-RRB-

each

of

which

corresponds

to

the

event

that

the

ith

cycle

of

ball

is

delayed

by

ball

r.

For

any

such

set

S

we

claim

that

Pr

-LRB-

i

r

-RRB-

-LRB-

xir

-RRB-

P?jSj

-LRB-

-RRB-

The

crux

of

proving

the

claim

is

to

show

that

Pr

xir

-LRB-

-RRB-

-LRB-

-RRB-

=P

-LRB-

-RRB-

where

S?f

-LRB-

i

r

-RRB-

g

whence

the

claim

-LRB-

-RRB-

follows

from

Bayes

's

Theorem

We

can

derive

Inequality

-LRB-

-RRB-

from

a

careful

analysis

of

dependencies

Because

the

adversary

follows

the

FIFO

rule

we

have

that

xir

only

if

when

the

adversary

executes

the

ith

toss

of

ball

it

falls

into

whatever

bin

contains

ball

r

if

any

A

priori

this

event

happens

with

probability

either

=P

or

and

hence

with

probability

at

most

=P

Conditioning

on

any

collection

of

events

relating

which

balls

delay

this

or

other

cycles

of

ball

can

not

increase

this

probability

as

we

now

argue

in

two

cases

In

the

first

case

the

indicator

random

variables

where

i

tell

whether

other

cycles

of

ball

are

delayed

This

information

tells

nothing

about

where

the

ith

toss

of

ball

goes

Therefore

these

random

variables

are

independent

of

xir

and

thus

the

probability

=P

upper

bound

is

not

affected

In

the

second

case

the

indicator

random

variables

tell

whether

the

ith

toss

of

ball

goes

to

the

bin

containing

ball

but

this

information

tells

us

nothing

about

whether

it

goes

to

the

bin

containing

ball

r

because

the

indicator

random

variables

tell

us

nothing

to

relate

where

ball

r

and

ball

are

located

Moreover

no

collusion

among

the

indicator

random

variables

provides

any

more

information

and

thus

Inequality

-LRB-

-RRB-

holds

Equation

-LRB-

-RRB-

shows

that

the

delay

d

encountered

by

ball

throughout

all

of

its

cycles

can

be

expresses

as

a

sum

of

m

-LRB-

P

-RRB-

indicator

random

variables

In

order

for

d

to

equal

or

exceed

a

given

value

D

there

must

be

some

set

containing

D

of

these

indicator

random

variables

each

of

which

must

be

For

any

specific

such

set

Inequality

-LRB-

-RRB-

says

that

the

probability

is

at

most

P?D

that

all

random

variables

in

the

set

are

Since

there

are

m

-LRB-

P

-RRB-

D

-LRB-

emP

=D

-RRB-

D

such

sets

we

have

Prfd

Dg

emP

D

D

P?D

em

D

D

e

=P

whenever

D

lg

P

lg

-LRB-

e

-RRB-

g.

Although

our

analysis

was

performed

for

ball

it

applies

to

any

other

ball

as

well

Consequently

for

any

given

ball

r

which

is

tossed

mr

times

the

probability

that

its

delay

dr

exceeds

lgP

lg

-LRB-

e

-RRB-

g

is

at

most

e

=P

By

Boole

's

inequality

and

Equation

-LRB-

-RRB-

Analysis

of

the

work-stealing

algorithm

it

follows

that

with

probability

at

least

e

the

total

delay

D

is

at

most

D

P

r

lgP

lg

-LRB-

e

-RRB-

g

Q

-LRB-

M+P

lgP

Plg

-LRB-

e

-RRB-

-RRB-

since

M

Pr

mr.

The

upper

bound

E

-LSB-

D

-RSB-

M

can

be

obtained

as

follows

Recall

that

each

dr

is

the

sum

of

-LRB-

P

-RRB-

mr

indicator

random

variables

each

of

which

has

expectation

at

most

=P

Therefore

by

linearity

of

expectation

E

-LSB-

dr

-RSB-

mr.

Using

Equation

-LRB-

-RRB-

and

again

using

linearity

of

expectation

we

obtain

E

-LSB-

D

-RSB-

M.

With

this

bound

on

the

total

delay

incurred

by

M

random

requests

now

in

hand

we

turn

back

to

the

Work-Stealing

Algorithm

Analysis

of

the

work-stealing

algorithm

In

this

section

we

analyze

the

time

and

communication

cost

of

executing

a

fully

strict

multithreaded

computation

with

the

Work-Stealing

Algorithm

For

any

fully

strict

computation

with

work

and

critical

path

length

T

we

show

that

the

expected

running

time

with

P

processors

including

scheduling

overhead

is

O

-LRB-

=P

T

-RRB-

Moreover

for

any

e

the

execution

time

on

P

processors

is

O

-LRB-

=P

T

lgP

lg

-LRB-

e

-RRB-

-RRB-

with

probability

at

least

e

We

also

show

that

the

expected

total

communication

during

the

execution

of

a

fully

strict

computation

is

O

-LRB-

PT

-LRB-

nd

-RRB-

Smax

-RRB-

where

nd

is

the

maximumnumber

of

dependency

edges

from

a

thread

to

its

parent

and

Smax

is

the

largest

size

of

any

activation

frame

Unlike

in

the

Busy-Leaves

Algorithm

the

ready

pool

in

the

Work-Stealing

Algorithm

is

distributed

and

so

there

is

no

contention

at

a

centralized

data

structure

Nevertheless

it

is

still

possible

for

contention

to

arise

when

several

thieves

happen

to

descend

on

the

same

victim

simultaneously

In

this

case

as

we

have

indicated

in

the

previous

section

we

make

the

conservative

assumption

that

an

adversary

serially

queues

the

work-stealing

requests

To

analyze

the

running

time

of

the

Work-Stealing

Algorithm

executing

a

fully

strict

multithreaded

computation

with

work

and

critical

path

length

T

on

a

computer

with

P

processors

we

use

an

accounting

argument

At

each

step

of

the

algorithm

we

collect

P

dollars

one

from

each

processor

At

each

step

each

processor

places

its

dollar

in

one

of

three

buckets

according

to

its

actions

at

that

step

If

the

processor

executes

an

instruction

at

the

step

then

it

places

its

dollar

into

the

WORK

bucket

If

the

processor

initiates

a

steal

attempt

at

the

step

then

it

places

its

dollar

into

the

STEAL

bucket

And

if

the

processor

merely

waits

for

a

queued

steal

request

at

the

step

then

it

places

its

dollar

into

the

WAIT

bucket

We

shall

derive

the

running

time

bound

by

bounding

the

number

of

dollars

in

each

bucket

at

the

end

of

the

execution

summing

these

three

bounds

and

then

dividing

by

P

We

first

bound

the

total

number

of

dollars

in

the

WORK

bucket

Chapter

Work

stealing

Lemma

The

execution

of

a

fully

strict

multithreaded

computation

with

work

by

the

Work-Stealing

Algorithm

on

a

computer

with

P

processors

terminates

with

exactly

dollars

in

the

WORK

bucket

Proof

A

processor

places

a

dollar

in

the

WORK

bucket

only

when

it

executes

an

instruction

Thus

since

there

are

instructions

in

the

computation

the

execution

ends

with

exactly

dollars

in

the

WORK

bucket

Bounding

the

total

dollars

in

the

STEAL

bucket

requires

a

significantly

more

involved

delay-sequence

argument

We

first

introduce

the

notion

of

a

round

of

work-steal

attempts

and

we

must

also

define

an

augmented

dag

that

we

then

use

to

define

critical

instructions

The

idea

is

as

follows

If

during

the

course

of

the

execution

a

large

number

of

steals

are

attempted

then

we

can

identify

a

sequence

of

instructions?the

delay

sequence

in

the

augmented

dag

such

that

each

of

these

steal

attempts

was

initiated

while

some

instruction

from

the

sequence

was

critical

We

then

show

that

a

critical

instruction

is

unlikely

to

remain

critical

across

a

modest

number

of

steal

attempts

We

can

then

conclude

that

such

a

delay

sequence

is

unlikely

to

occur

and

therefore

an

execution

is

unlikely

to

suffer

a

large

number

of

steal

attempts

A

round

of

work-steal

attempts

is

a

set

of

at

least

but

fewer

than

consecutive

steal

attempts

such

that

if

a

steal

attempt

that

is

initiated

at

time

step

t

occurs

in

a

particular

round

then

all

other

steal

attempts

initiated

at

time

step

t

are

also

in

the

same

round

We

can

partition

all

the

steal

attempts

that

occur

during

an

execution

into

rounds

as

follows

The

first

round

contains

all

steal

attempts

initiated

at

time

steps

where

is

the

earliest

time

such

that

at

least

steal

attempts

were

initiated

at

or

before

We

say

that

the

first

round

starts

at

time

step

and

ends

at

time

step

In

general

if

the

ith

round

ends

at

time

step

ti

then

the

-LRB-

i

-RRB-

st

round

begins

at

time

step

ti

and

ends

at

the

earliest

time

step

ti

ti

such

that

at

least

steal

attempts

were

initiated

at

time

steps

between

ti

and

ti

inclusive

These

steal

attempts

belong

to

round

i

By

definition

each

round

contains

at

least

consecutive

steal

attempts

and

since

at

most

P

steal

attempts

can

be

initiated

in

a

single

time

step

each

round

contains

fewer

than

steal

attempts

The

sequence

of

instructions

that

make

up

the

delay

sequence

is

defined

with

respect

to

an

augmented

dag

obtained

by

slightly

modifying

the

original

dag

Let

G

denote

the

original

dag

that

is

the

dag

consisting

of

the

computation

's

instructions

as

vertices

and

its

continue

spawn

and

dependency

edges

as

edges

The

augmented

dag

is

the

original

dag

G

together

with

some

new

edges

as

follows

For

every

set

of

instructions

u

v

and

w

such

that

-LRB-

u

v

-RRB-

is

a

spawn

edge

and

-LRB-

u

w

-RRB-

is

a

continue

edge

the

deque

edge

-LRB-

w

v

-RRB-

is

placed

in

These

deque

edges

are

shown

dashed

in

Figure

We

make

the

technical

assumption

that

instruction

w

has

no

incoming

dependency

edges

and

so

is

a

dag

-LRB-

If

a

cycle

is

created

a

new

instruction

between

u

and

w

can

be

created

which

does

not

affect

our

asymptotic

bounds

-RRB-

If

T

is

the

length

of

a

longest

path

in

G

then

the

longest

path

in

has

length

at

most

It

is

worth

pointing

out

that

is

only

an

analytical

tool

The

deque

edges

have

no

effect

on

the

scheduling

and

execution

of

the

computation

by

the

Work-Stealing

Algorithm

The

deque

edges

are

the

key

to

defining

critical

instructions

At

any

time

step

during

the

execution

we

say

that

an

instruction

v

is

critical

if

every

instruction

that

precedes

v

Analysis

of

the

work-stealing

algorithm

-LRB-

either

directly

or

indirectly

-RRB-

in

has

been

executed

that

is

if

for

every

instruction

w

such

that

there

is

a

directed

path

from

w

to

v

in

instruction

w

has

been

executed

A

critical

instruction

must

be

ready

since

contains

every

edge

of

G

but

a

ready

instruction

may

or

may

not

be

critical

Intuitively

the

structural

properties

of

a

ready

deque

enumerated

in

Lemma

guarantee

that

if

a

thread

is

deep

in

a

ready

deque

then

it

can

not

be

critical

because

the

predecessor

of

the

thread

's

current

instruction

across

the

deque

edge

has

not

yet

been

executed

We

say

that

a

given

round

of

steal

attempts

occurs

while

instruction

v

is

critical

if

each

of

the

steal

attempts

that

comprise

the

round

is

initiated

at

a

time

step

when

v

is

critical

but

is

not

executed

We

now

formalize

our

definition

of

a

delay

sequence

Definition

A

delay

sequence

is

a

-LRB-

U

R

;P

-RRB-

satisfying

the

following

conditions

ffl

U

-LRB-

uL

-RRB-

is

a

maximal

directed

path

in

Specifically

for

i

L

the

edge

-LRB-

ui

ui

-RRB-

belongs

to

instruction

has

no

incoming

edges

in

-LRB-

instruction

must

be

the

first

instruction

of

the

root

thread

-RRB-

and

instruction

uL

has

no

outgoing

edges

in

-LRB-

instruction

uL

must

be

the

last

instruction

of

the

root

thread

-RRB-

ffl

R

is

a

positive

integer

ffl

P

-LRB-

pL

-RRB-

is

a

partition

of

the

integer

R

The

delay

sequence

-LRB-

U

R

;P

-RRB-

is

said

to

occur

during

an

execution

if

for

each

i

L

at

least

pi

steal-attempt

rounds

occur

while

instruction

ui

is

critical

The

following

lemma

states

that

if

a

large

number

of

steal

attempts

take

place

during

an

execution

then

a

delay

sequence

with

large

R

must

occur

Lemma

Consider

the

execution

of

a

fully

strict

multithreaded

computation

with

critical

path

length

T

by

the

Work-Stealing

Algorithm

on

a

computer

with

P

processors

If

at

least

-LRB-

R

-RRB-

steal

attempts

occur

during

the

execution

then

some

-LRB-

U

R

;P

-RRB-

delay

sequence

must

occur

Proof

For

a

given

execution

in

which

at

least

-LRB-

R

-RRB-

steal

attempts

take

place

we

construct

a

delay

sequence

-LRB-

U

R

;P

-RRB-

and

show

that

it

occurs

With

at

least

-LRB-

R

-RRB-

steal

attempts

there

must

be

at

least

R

rounds

of

steal

attempts

We

construct

the

delay

sequence

by

identifying

a

set

of

instructions

on

a

directed

path

in

such

that

for

every

time

step

during

the

execution

one

of

these

instructions

is

critical

There

are

at

most

instructions

on

the

delay

sequence

so

at

most

steal-attempt

rounds

could

overlap

a

time

step

at

which

one

of

these

instructions

gets

executed

Therefore

there

must

be

at

least

R

stealattempt

rounds

that

occur

while

an

instruction

from

the

delay

sequence

is

critical

To

finish

the

proof

we

need

only

produce

the

directed

path

U

-LRB-

uL

-RRB-

such

that

for

every

time

step

during

the

execution

one

of

the

ui

is

critical

The

partition

P

-LRB-

pL

-RRB-

can

be

derived

by

simply

letting

pi

equal

the

number

of

steal-attempt

rounds

that

occur

while

ui

is

critical

We

work

backwards

from

the

last

instruction

of

the

root

thread

which

we

denote

by

Let

denote

the

-LRB-

not

necessarily

immediate

-RRB-

predecessor

instruction

of

in

with

the

Chapter

Work

stealing

latest

execution

time

Let

-LRB-

-RRB-

denote

a

directed

path

from

to

in

We

extend

this

path

back

to

the

first

instruction

of

the

root

thread

by

iterating

this

construction

as

follows

At

the

ith

iteration

we

have

an

instruction

vli

and

a

directed

path

in

from

vli

to

We

let

wi

denote

the

predecessor

of

vli

in

with

the

latest

execution

time

and

let

-LRB-

vli

vli

vli

-RRB-

where

vli

wi

denote

a

directed

path

from

wi

to

vli

in

We

finish

iterating

the

construction

when

we

get

to

an

iteration

k

in

which

vlk

is

the

first

instruction

of

the

root

thread

Our

desired

sequence

is

then

U

-LRB-

uL

-RRB-

where

L

lk

and

ui

vL?i

for

i

L

One

can

verify

that

at

every

time

step

of

the

execution

one

of

the

vli

is

critical

and

therefore

the

sequence

has

the

desired

property

We

now

establish

that

a

critical

instruction

is

unlikely

to

remain

critical

across

a

modest

number

of

steal-attempt

rounds

Specifically

we

first

show

that

a

critical

instruction

must

be

the

ready

instruction

of

a

thread

that

is

one

of

the

top

in

its

processor

's

ready

deque

We

then

use

this

fact

to

show

that

after

O

-LRB-

-RRB-

steal-attempt

rounds

a

critical

instruction

is

very

likely

to

be

executed

Lemma

At

every

time

step

during

the

execution

of

a

fully

strict

multithreaded

computation

by

the

Work-Stealing

Algorithm

each

critical

instruction

must

be

the

ready

instruction

of

a

thread

that

is

one

of

the

top

in

its

processor

's

ready

deque

Proof

Consider

any

time

step

and

let

be

the

critical

instruction

of

a

thread

Since

is

critical

must

be

ready

and

therefore

must

be

in

the

ready

deque

of

some

processor

p

If

is

not

one

of

the

top

threads

in

p

's

ready

deque

then

Lemma

guarantees

that

each

of

the

at

least

threads

above

in

p

's

ready

deque

is

an

ancestor

of

Let

Gk

denote

's

ancestor

threads

where

is

the

parent

of

and

Gk

is

the

root

thread

Further

for

i

k

let

ui

denote

the

instruction

of

thread

Gi

that

spawned

thread

Gi

and

let

wi

denote

ui

's

successor

instruction

in

thread

Gi

In

the

dag

we

have

deque

edges

-LRB-

wi

ui

-RRB-

for

i

k.

Consequently

since

is

critical

for

i

k

each

instruction

wi

must

have

been

executed

since

it

is

a

predecessor

of

in

Moreover

because

each

wi

is

the

successor

of

the

spawn

instruction

ui

in

thread

Gi

each

thread

Gi

for

i

k

must

have

been

worked

on

since

the

time

step

at

which

it

spawned

thread

Gi

But

Lemma

guarantees

that

only

the

topmost

thread

in

p

's

ready

deque

can

have

this

property

Thus

is

the

only

thread

that

can

possibly

be

above

in

p

's

ready

deque

Lemma

Consider

the

execution

of

any

fully

strict

multithreaded

computation

by

the

Work-Stealing

Algorithm

on

a

parallel

computer

with

P

processors

For

any

instruction

v

and

any

number

r

of

steal-attempt

rounds

the

probability

that

r

rounds

occur

while

the

instruction

is

critical

is

at

most

the

probability

that

only

or

of

the

steal

attempts

initiated

in

the

r

rounds

choose

v

's

processor

which

is

at

most

e

Proof

Let

ta

denote

the

first

time

step

at

which

instruction

v

is

critical

and

let

p

denote

the

processor

in

whose

ready

deque

v

's

thread

resides

at

time

step

ta

Suppose

r

steal-attempt

rounds

occur

while

instruction

v

is

critical

and

consider

the

steal

attempts

that

comprise

these

rounds

of

which

there

must

be

at

least

Let

tb

denote

the

time

step

at

which

the

last

of

these

steal

attempts

is

initiated

which

must

occur

before

the

time

step

at

which

instruction

Analysis

of

the

work-stealing

algorithm

v

is

executed

At

least

P

-LRB-

-RRB-

P

of

these

steal

attempts

must

be

initiated

at

a

time

step

before

tb

since

fewer

than

P

steal

attempts

can

be

initiated

at

time

step

tb

We

shall

first

show

that

of

these

-LRB-

-RRB-

P

steal

attempts

initiated

while

instruction

v

is

critical

and

at

least

time

steps

before

v

is

executed

at

most

of

them

can

choose

processor

p

as

its

target

for

otherwise

v

would

be

executed

at

or

before

tb

Recall

fromLemma

that

instruction

v

is

the

ready

instruction

of

a

thread

G

which

must

be

among

the

top

threads

in

p

's

ready

deque

as

long

as

v

is

critical

If

G

is

topmost

then

another

thread

can

not

become

topmost

until

after

instruction

v

is

executed

since

only

by

processor

p

executing

instructions

from

G

can

another

thread

be

placed

on

the

top

of

its

ready

deque

Consequently

if

a

steal

attempt

targeting

processor

p

is

initiated

at

some

time

step

t

ta

we

are

guaranteed

that

instruction

v

is

executed

at

a

time

step

no

later

than

t

either

by

thread

G

being

stolen

and

executed

or

by

p

executing

the

thread

itself

Now

suppose

G

is

second

from

the

top

in

p

's

ready

deque

with

thread

on

top

In

this

case

if

a

steal

attempt

targeting

processor

p

is

initiated

at

time

step

t

ta

then

thread

gets

stolen

from

p

's

ready

deque

no

later

than

time

step

t.

Suppose

further

that

another

steal

attempt

targeting

processor

p

is

initiated

at

time

step

where

ta

t

tb

Then

we

know

that

a

second

steal

will

be

serviced

by

p

at

or

before

time

step

If

this

second

steal

gets

thread

G

then

instruction

v

must

get

executed

at

or

before

time

step

tb

which

is

impossible

since

v

is

executed

after

time

step

tb

Consequently

this

second

steal

must

get

thread

same

thread

that

the

first

steal

got

But

this

scenario

can

only

occur

if

in

the

intervening

time

period

thread

stalls

and

is

subsequently

reenabled

by

the

execution

of

some

instruction

from

thread

G

in

which

case

instruction

v

must

be

executed

before

time

step

tb

which

is

once

again

impossible

Thus

we

must

have

-LRB-

-RRB-

P

steal

attempts

each

initiated

at

a

time

step

t

such

that

ta

t

tb

and

at

most

of

which

targets

processor

p

The

probability

that

either

or

of

-LRB-

-RRB-

P

steal

attempts

chooses

processor

p

is

P

-LRB-

-RRB-

P

-LRB-

-RRB-

P

P

P

-LRB-

-RRB-

P

P

r

P

-LRB-

-RRB-

P

e

for

r

We

now

complete

the

delay-sequence

argument

and

bound

the

total

dollars

in

the

STEAL

bucket

Lemma

Consider

the

execution

of

any

fully

strict

multithreaded

computation

with

critical

path

length

T

by

the

Work-Stealing

Algorithm

on

a

parallel

computer

with

P

processors

For

any

e

with

probability

at

least

e

at

most

O

-LRB-

P

-LRB-

T

lg

-LRB-

e

-RRB-

-RRB-

-RRB-

work-steal

attempts

occur

The

expected

number

of

steal

attempts

is

O

-LRB-

PT

-RRB-

In

other

words

with

Chapter

Work

stealing

probability

at

least

e

the

execution

terminates

with

at

most

O

-LRB-

P

-LRB-

T

lg

-LRB-

e

-RRB-

-RRB-

-RRB-

dollars

in

the

STEAL

bucket

and

the

expected

number

of

dollars

in

this

bucket

is

O

-LRB-

PT

-RRB-

Proof

From

Lemma

we

know

that

if

at

least

-LRB-

R

-RRB-

steal

attempts

occur

then

some

delay

sequence

-LRB-

U

R

;P

-RRB-

must

occur

Consider

a

particular

delay

sequence

-LRB-

U

R

;P

-RRB-

having

U

-LRB-

uL

-RRB-

and

P

-LRB-

pL

-RRB-

where

pL

R

and

L

We

shall

compute

the

probability

that

-LRB-

U

R

;P

-RRB-

occurs

Such

a

sequence

occurs

if

for

each

i

L

at

least

pi

steal-attempt

rounds

occur

while

instruction

ui

is

critical

From

Lemma

we

know

that

the

probability

of

at

least

pi

rounds

occurring

while

a

given

instruction

ui

is

critical

is

at

most

the

probability

that

only

or

steal

attempts

initiated

in

the

pi

rounds

choose

v

's

processor

which

is

at

most

e

provided

pi

-LRB-

For

those

values

of

i

with

pi

we

use

as

an

upper

bound

on

this

probability

-RRB-

Moreover

since

the

targets

of

the

work-steal

attempts

in

the

pi

rounds

are

chosen

independently

from

the

targets

chosen

in

other

rounds

we

can

bound

the

probability

of

the

particular

delay

sequence

-LRB-

U

R

;P

-RRB-

occurring

as

follows

Prf

-LRB-

U

R

;P

-RRB-

occursg

i

L

Prfpi

rounds

occur

while

ui

is

criticalg

i

L

pi

e

exp

i

L

pi

pi

exp

B

i

L

pi

i

L

pi

pi

C

A

e

-LRB-

R

-RRB-

To

bound

the

probability

of

some

-LRB-

U

R

;P

-RRB-

delay

sequence

occurring

we

need

to

count

the

number

of

such

delay

sequences

and

multiply

by

the

probability

that

a

particular

such

sequence

occurs

The

directed

path

U

in

the

modified

dag

starts

at

the

first

instruction

of

the

root

thread

and

ends

at

the

last

instruction

of

the

root

thread

If

the

original

dag

has

degree

d

then

has

degree

at

most

d

Consistent

with

our

unit-time

assumption

for

instructions

we

assume

that

the

degree

d

is

a

constant

Since

the

length

of

a

longest

path

in

is

at

most

there

are

at

most

-LRB-

d

-RRB-

ways

of

choosing

the

path

U

-LRB-

uL

-RRB-

There

are

at

most

L+R

R

R

R

ways

to

choose

P

since

P

partitions

R

into

L

pieces

As

we

have

just

shown

a

given

delay

sequence

has

at

most

an

e

-LRB-

R

-RRB-

e

-LRB-

R

-RRB-

chance

of

occurring

Multiplying

these

three

factors

together

bounds

the

probability

that

any

-LRB-

U

R

;P

-RRB-

delay

sequence

occurs

by

-LRB-

d

-RRB-

R

R

e

T

-LRB-

-RRB-

which

is

at

most

e

for

R

Q

-LRB-

T

lg

d

lg

-LRB-

e

-RRB-

-RRB-

Thus

the

probability

that

at

least

-LRB-

Analysis

of

the

work-stealing

algorithm

R

-RRB-

Q

-LRB-

P

-LRB-

T

lgd

lg

-LRB-

e

-RRB-

-RRB-

-RRB-

Q

-LRB-

P

-LRB-

T

lg

-LRB-

e

-RRB-

-RRB-

-RRB-

steal

attempts

occur

is

at

most

e

The

expectation

bound

follows

because

the

tail

of

the

distribution

decreases

exponentially

With

bounds

on

the

number

of

dollars

in

the

WORK

and

STEAL

buckets

we

now

state

the

theorem

that

bounds

the

total

execution

time

for

a

fully

strict

multithreaded

computation

by

the

Work-Stealing

Algorithm

and

we

complete

the

proof

by

bounding

the

number

of

dollars

in

the

WAIT

bucket

Theorem

Consider

the

execution

of

any

fully

strict

multithreaded

computation

with

work

and

critical

path

length

T

by

the

Work-Stealing

Algorithm

on

a

parallel

computer

with

P

processors

The

expected

running

time

including

scheduling

overhead

is

O

-LRB-

=P

T

-RRB-

Moreover

for

any

e

with

probability

at

least

e

the

execution

time

on

P

processors

is

O

-LRB-

=P

T

lgP

lg

-LRB-

e

-RRB-

-RRB-

Proof

Lemmas

and

bound

the

dollars

in

the

WORK

and

STEAL

buckets

so

we

now

must

bound

the

dollars

in

the

WAIT

bucket

This

bound

is

given

by

Lemma

which

bounds

the

total

delay?that

is

the

total

dollars

in

the

WAIT

bucket?as

a

function

of

the

number

M

of

steal

attempts?that

is

the

total

dollars

in

the

STEAL

bucket

This

lemma

says

that

for

any

e

with

probability

at

least

e

the

number

of

dollars

in

the

WAIT

bucket

is

at

most

a

constant

times

the

number

of

dollars

in

the

STEAL

bucket

plus

O

-LRB-

P

lgP

P

lg

-LRB-

e

-RRB-

-RRB-

and

the

expected

number

of

dollars

in

the

WAIT

bucket

is

at

most

the

number

in

the

STEAL

bucket

We

now

add

up

the

dollars

in

the

three

buckets

and

divide

by

P

to

complete

this

proof

The

next

theorem

bounds

the

total

amount

of

communication

that

a

multithreaded

computation

executed

by

the

Work-Stealing

Algorithm

performs

in

a

distributed

model

The

analysis

makes

the

assumption

that

at

most

a

constant

number

of

bytes

need

be

communicated

along

a

dependency

edge

to

resolve

the

dependency

Theorem

Consider

the

execution

of

any

fully

strict

multithreaded

computation

with

critical

path

length

T

by

the

Work-Stealing

Algorithm

on

a

parallel

computer

with

P

processors

Then

the

total

number

of

bytes

communicated

has

expectation

O

-LRB-

PT

-LRB-

nd

-RRB-

Smax

-RRB-

where

nd

is

the

maximum

number

of

dependency

edges

from

a

thread

to

its

parent

and

Smax

is

the

size

in

bytes

of

the

largest

activation

frame

in

the

computation

Moreover

for

any

e

the

probability

is

at

least

e

that

the

total

communication

incurred

is

O

-LRB-

P

-LRB-

T

lg

-LRB-

e

-RRB-

-RRB-

-LRB-

nd

-RRB-

Smax

-RRB-

Proof

We

prove

the

bound

for

the

expectation

The

high-probability

bound

is

analogous

By

our

bucketing

argument

the

expected

number

of

steal

attempts

is

at

most

O

-LRB-

PT

-RRB-

When

a

thread

is

stolen

the

communication

incurred

is

at

most

Smax

Communication

also

occurs

whenever

a

dependency

edge

enters

a

parent

thread

from

one

of

its

children

and

the

parent

has

been

stolen

but

since

each

dependency

edge

accounts

for

at

most

a

constant

number

of

Plaxton

's

bound

-LSB-

-RSB-

for

Lemma

this

bound

becomes

TP

O

-LRB-

=P

T

-RRB-

whenever

e

is

at

most

polynomial

in

M

and

P.

Chapter

Work

stealing

bytes

the

communication

incurred

is

at

most

O

-LRB-

nd

-RRB-

per

steal

Finally

we

can

have

communication

when

a

child

thread

enables

its

parent

and

puts

the

parent

into

the

child

's

processor

's

ready

deque

This

event

can

happen

at

most

nd

times

for

each

time

the

parent

is

stolen

so

the

communication

incurred

is

at

most

ndSmax

per

steal

Thus

the

expected

total

communication

cost

is

O

-LRB-

PT

-LRB-

nd

-RRB-

Smax

-RRB-

The

communication

bounds

in

this

theorem

are

existentially

tight

in

that

there

exist

fully

strict

computations

that

require

W

-LRB-

PT

-LRB-

nd

-RRB-

Smax

-RRB-

total

communication

for

any

execution

schedule

that

achieves

linear

speedup

This

result

follows

directly

from

a

theorem

of

Wu

and

Kung

-LSB-

-RSB-

who

showed

that

divide-and-conquer

computations?a

special

case

of

fully

strict

computations

with

nd

require

this

much

communication

In

the

case

when

we

have

nd

=O

-LRB-

-RRB-

and

the

algorithm

achieves

linear

expected

speedup

that

is

when

P

O

-LRB-

T

-RRB-

the

total

communication

is

at

most

O

-LRB-

-RRB-

Moreover

if

P

o

T

the

total

communication

is

much

less

than

which

confirms

the

folk

wisdom

that

work-stealing

algorithms

require

much

less

communication

than

the

possibly

Q

-LRB-

-RRB-

communication

of

work-sharing

algorithms

Chapter

Parallel

programming

in

Cilk

Writing

a

high-performance

parallel

application

in

Cilk

the

programmer

can

focus

on

expressing

the

parallelism

in

the

algorithm

insulated

from

communication

protocols

load

balancing

and

other

runtime

scheduling

issues

The

Cilk

language

is

an

explicitly

parallel

multithreaded

extension

of

the

C

language

The

Cilk

runtime

system

automatically

manages

the

low-level

details

involved

in

executing

a

Cilk

program

on

a

parallel

machine

and

it

does

so

with

a

work-stealing

scheduler

that

is

efficient

in

theory

as

well

as

in

practice

Moreover

Cilk

gives

the

user

an

algorithmic

model

of

application

performance

based

on

the

measures

of

work

and

critical

path

length

which

can

be

used

to

predict

the

runtime

of

a

Cilk

program

accurately

Consequently

a

Cilk

programmer

can

tune

the

performance

of

his

or

her

application

by

focusing

on

the

work

and

critical

path

unworried

about

machine

specific

performance

details

In

this

chapter

we

explain

the

Cilk

language

and

runtime

system

and

then

we

demonstrate

the

efficiency

of

Cilk

's

scheduler

both

empirically

and

analytically

The

language

and

runtime

system

are

covered

in

Section

In

Section

we

use

several

applications

to

demonstrate

empirically

the

efficiency

of

the

Cilk

runtime

system

These

applications

include

protein

folding

graphic

rendering

backtrack

search

and

the

Socrates

chess

program

which

won

second

prize

in

the

ICCA

World

Computer

Chess

Championship

We

also

show

in

Section

how

work

and

critical

path

length

can

be

used

to

model

accurately

the

parallel

runtime

of

Cilk

applications

Analytically

we

prove

in

Section

that

for

fully

strict

-LRB-

well-structured

-RRB-

programs

Cilk

's

work-stealing

scheduler

uses

space

time

and

communication

all

within

a

constant

factor

of

optimal

To

date

all

of

the

applications

that

we

have

coded

are

fully

strict

The

Cilk

language

and

runtime

system

The

Cilk

language

-LSB-

-RSB-

extends

C

with

primitives

to

express

parallelism

and

the

Cilk

runtime

system

maps

the

expressed

parallelism

into

parallel

execution

A

Cilk

program

is

pre

Some

of

the

research

reported

in

this

chapter

is

joint

work

with

members

of

the

Cilk

team

Matteo

Frigo

Michael

Halbherr

Chris

Joerg

Bradley

Kuszmaul

Charles

Leiserson

Rob

Miller

Keith

Randall

and

Yuli

Zhou

all

currently

or

formerly

of

MIT

's

Laboratory

for

Computer

Science

Some

of

the

material

in

this

chapter

was

previously

published

in

-LSB-

-RSB-

Chapter

Parallel

programming

in

Cilk

processed

to

C

using

the

-LSB-

-RSB-

and

then

compiled

and

linked

with

a

runtime

library

to

run

on

the

target

platform

Currently

supported

targets

include

the

Connection

Machine

MPP

the

Intel

Paragon

MPP

the

Sun

SparcStation

SMP

the

Silicon

Graphics

Power

Challenge

SMP

and

the

Cilk-NOW

network

of

workstations

-LRB-

Chapter

-RRB-

In

this

section

we

shall

discuss

the

Cilk

language

primitives

for

expressing

parallelism

as

well

as

the

runtime

system

mechanisms

that

implement

these

primitives

Beginning

with

two

simple

high-level

language

primitives

spawn

and

sync

we

shall

then

dive

into

the

lower-level

primitives

based

on

continuation-passing

threads

that

are

supported

directly

by

the

runtime

system

We

conclude

this

section

by

discussing

the

work-stealing

scheduler

employed

by

the

runtime

system

to

execute

a

Cilk

program

in

parallel

Spawns

and

syncs

A

Cilk

program

contains

one

or

more

Cilk

procedures

and

Cilk

procedures

can

spawn

children

for

parallel

execution

A

Cilk

procedure

is

the

parallel

equivalent

of

a

C

function

and

a

spawn

is

the

parallel

equivalent

of

a

function

call

A

spawn

differs

from

a

call

in

that

when

a

procedure

spawns

a

child

the

parent

and

child

may

execute

concurrently

The

Cilk

language

provides

a

mechanism

to

define

Cilk

procedures

a

primitive

that

procedures

may

use

to

spawn

children

and

a

primitive

that

procedures

may

use

to

synchronize

with

their

spawned

children

To

illustrate

these

Cilk

constructs

we

shall

use

the

double

recursive

implementation

of

the

Fibonacci

function

as

a

running

example

Recall

that

the

Fibonacci

function

fib

-LRB-

n

-RRB-

for

n

is

defined

as

fib

-LRB-

n

-RRB-

ae

n

if

n

fib

-LRB-

n

-RRB-

fib

-LRB-

n

-RRB-

otherwise

Figure

shows

how

this

function

is

written

as

a

Cilk

procedure

This

toy

example

illustrates

a

common

pattern

occurring

in

-LRB-

parallel

-RRB-

divide-and-conquer

applications

recursive

calls

-LRB-

spawns

-RRB-

solve

smaller

subcases

and

then

the

partial

results

are

merged

to

produce

the

final

result

The

keyword

cilk

identifies

a

Cilk

procedure

The

definition

of

a

procedure

P

includes

a

return

type

an

argument

list

and

a

body

just

like

a

C

function

cilk

type

P

-LRB-

arg-decls

-RRB-

f

stmts

g

The

actual

work

in

a

Cilk

procedure

is

performed

by

ordinary

C

code

in

the

body

which

is

executed

serially

To

express

parallelism

a

procedure

may

spawn

children

using

the

spawn

keyword

and

it

may

contain

synchronization

points

identified

by

the

sync

statement

Notice

that

besides

the

keywords

cilk

spawn

and

sync

the

Cilk

procedure

Fib

is

identical

to

its

C

counterpart

-LRB-

Figure

page

-RRB-

Cilk

programs

create

parallelism

at

runtime

when

a

procedure

spawns

children

A

Cilk

procedure

may

spawn

a

child

P

as

follows

translator

was

written

by

Rob

Miller

formerly

of

MIT

's

Laboratory

for

Computer

Science

and

now

of

the

School

of

Computer

Science

at

Carnegie

Mellon

University

Rob

's

implementation

builds

on

earlier

work

by

Yuli

Zhou

formerly

of

MIT

's

Laboratory

for

Computer

Science

and

now

of

AT&T

Bell

Laboratories

The

Cilk

language

and

runtime

system

cilk

int

Fib

-LRB-

int

n

-RRB-

f

if

-LRB-

n

-RRB-

return

n

else

f

int

x

y

x

spawn

Fib

-LRB-

-RRB-

y

spawn

Fib

-LRB-

-RRB-

sync

return

-LRB-

x

y

-RRB-

g

g

Figure

A

Cilk

procedure

to

compute

the

nth

Fibonacci

number

-LSB-

var

spawn

P

-LRB-

args

-RRB-

Besides

the

keyword

spawn

this

construct

is

identical

to

a

C

function

call

Semantically

the

difference

is

as

follows

When

a

C

function

calls

a

child

the

parent

immediately

suspends

executing

and

waits

for

the

child

to

return

before

resuming

A

C

function

call

is

synchronous

On

the

other

hand

when

a

Cilk

procedure

spawns

a

child

the

parent

need

not

immediately

wait

for

its

child

to

return

A

spawn

is

asynchronous

For

example

when

the

Fib

procedure

spawns

its

first

child

the

parent

may

continue

on

to

the

second

spawn

while

the

child

may

execute

concurrently

on

some

other

processor

The

programmer

specifies

neither

where

nor

exactly

when

the

spawned

child

will

execute

The

scheduler

makes

these

decisions

A

Cilk

procedure

can

not

use

the

return

values

of

the

children

it

has

spawned

until

those

children

return

For

example

the

Fib

procedure

can

not

add

the

return

values

x

and

y

until

its

spawned

children

have

completed

computing

and

returned

those

values

Therefore

Fib

uses

the

sync

statement

after

the

spawns

and

before

the

addition

The

sync

statement

forces

the

procedure

to

suspend

executing

and

wait

until

all

of

its

spawned

children

return

Once

all

of

its

children

return

the

procedure

may

resume

Thus

the

sync

statement

synchronizes

the

parent

with

all

of

its

children

Cilk

program

execution

begins

at

the

Cilk

procedure

cilk

main

Like

the

C

main

function

cilk

main

has

the

prototype

cilk

int

cilk

main

-LRB-

int

argc

char

argv

-LSB-

-RSB-

-RRB-

and

receives

the

command-line

arguments

as

its

parameters

For

example

Figure

gives

the

definition

of

cilk

main

for

the

Fibonacci

program

The

Cilk

procedures

cilk

main

and

Fib

together

constitute

a

complete

Cilk

Fibonacci

program

At

runtime

a

Cilk

program

grows

a

spawn

tree

of

procedures

that

unfolds

dynamically

as

procedures

spawn

children

The

spawn

tree

is

rooted

at

cilk

main

and

in

general

connects

procedures

to

the

children

they

spawn

For

example

Figure

shows

the

spawn

tree

grown

by

an

execution

of

the

Fibonacci

program

Notice

that

we

use

the

term

procedure

to

denote

both

the

static

Cilk

procedure

-LRB-

identified

by

the

keyword

cilk

-RRB-

that

is

part

of

a

Cilk

program

and

the

dynamic

procedure

that

-LRB-

as

a

consequence

of

a

spawn

occurring

at

runtime

-RRB-

is

a

node

in

the

spawn

tree

The

spawn

tree

is

analogous

to

a

conventional

call

Chapter

Parallel

programming

in

Cilk

cilk

int

cilk

main

-LRB-

int

argc

char

argv

-LSB-

-RSB-

-RRB-

f

int

n

result

n

atoi

-LRB-

argv

-LSB-

-RSB-

-RRB-

result

spawn

Fib

-LRB-

n

-RRB-

sync

printf

-LRB-

Fib

-LRB-

d

-RRB-

d.

n

n

result

-RRB-

return

g

Figure

The

cilk

main

procedure

for

the

Fibonacci

program

-LRB-

-RRB-

F

ib

-LRB-

-RRB-

F

ib

-LRB-

-RRB-

F

ib

-LRB-

-RRB-

F

ib

-LRB-

-RRB-

Figure

The

spawn

tree

grown

by

an

execution

of

the

Fibonacci

program

tree

and

it

is

equivalent

to

the

spawn

tree

described

in

Chapter

except

that

in

Cilk

we

use

the

term

procedure

instead

of

thread

In

Cilk

terminology

thread

has

a

different

meaning

Continuation-passing

threads

Rather

than

work

with

procedures

that

may

suspend

waiting

to

synchronize

with

their

children

the

Cilk

runtime

system

manipulates

and

schedules

nonsuspending

pieces

of

procedures

called

Cilk

threads

Also

Cilk

threads

do

not

interact

with

each

other

in

spawn/return

style

as

procedures

do

Instead

as

we

shall

explain

shortly

Cilk

threads

interact

in

the

continuation-passing

style

supported

by

the

runtime

system

To

be

executed

by

the

Cilk

runtime

system

each

procedure

must

be

partitioned

into

one

or

more

threads

The

first

thread

executed

as

part

of

a

procedure

is

called

the

initial

thread

and

subsequent

threads

are

called

successor

threads

For

a

Cilk

procedure

defined

with

the

keyword

cilk

the

translator

automatically

partitions

the

procedure

into

continuation-passing

threads

using

the

sync

statements

as

the

dividing

points

For

example

partitions

the

Fib

procedure

of

Figure

into

two

Cilk

threads

the

initial

thread

before

the

sync

statement

and

the

successor

thread

after

Currently

is

able

to

perform

this

automatic

partitioning

only

for

procedures

that

The

Cilk

language

and

runtime

system

synchronize

exclusively

via

the

sync

statement

which

waits

for

all

spawned

children

For

procedures

that

require

more

complex

synchronization

Cilk

provides

primitives

so

that

the

programmer

can

explicitly

define

Cilk

threads

in

continuation-passing

style

Figure

shows

the

Fibonacci

function

written

explicitly

as

two

continuation-passing

threads

the

initial

thread

fib

and

its

successor

sum

Essentially

translates

the

Fib

procedure

of

Figure

as

if

it

had

been

written

as

in

Figure

thread

fib

-LRB-

cont

int

k

int

n

-RRB-

f

if

-LRB-

n

-RRB-

send

argument

-LRB-

k

n

-RRB-

else

f

cont

int

x

y

spawn

next

sum

-LRB-

k

x

y

-RRB-

spawn

fib

-LRB-

x

-RRB-

spawn

fib

-LRB-

y

-RRB-

g

g

thread

sum

-LRB-

cont

int

k

int

x

int

y

-RRB-

f

send

argument

-LRB-

k

x

y

-RRB-

g

Figure

A

Cilk

procedure

written

in

explicit

continuation-passing

style

to

compute

the

nth

Fibonacci

number

This

procedure

contains

two

threads

fib

and

its

successor

sum

The

Cilk

keyword

thread

identifies

a

Cilk

thread

The

definition

of

a

Cilk

thread

T

includes

an

argument

list

and

a

body

similar

to

a

C

function

definition

thread

T

-LRB-

arg-decls

-RRB-

f

stmts

g

The

actual

work

in

a

Cilk

thread

is

performed

by

ordinary

C

code

in

the

body

which

is

executed

serially

and

without

suspending

Threads

are

not

allowed

to

contain

sync

statements

spawnspawn

next

or

accumulate

successorthreadchildthreadchildthread

Figure

A

thread

spawns

child

threads

to

create

parallelism

and

a

successor

thread

to

wait

for

the

values

returned

by

the

children

The

background

shading

denotes

procedures

The

child

threads

each

start

a

new

child

procedure

and

the

successor

thread

is

part

of

the

same

procedure

as

its

predecessor

Chapter

Parallel

programming

in

Cilk

A

Cilk

thread

generates

parallelism

at

runtime

by

spawning

a

child

thread

that

becomes

the

initial

thread

of

a

child

procedure

After

spawning

one

or

more

children

the

parent

thread

can

not

then

wait

for

its

children

to

return?in

Cilk

threads

never

suspend

Rather

as

illustrated

in

Figure

the

parent

thread

must

additionally

spawn

a

successor

thread

to

wait

for

the

values

returned

from

the

children

The

spawned

successor

is

part

of

the

same

procedure

as

its

predecessor

The

child

procedures

return

values

to

the

parent

procedure

by

sending

those

values

to

the

parent

's

waiting

successor

Thus

a

thread

may

wait

to

begin

executing

but

once

it

begins

executing

it

can

not

suspend

Notice

that

we

use

the

term

thread

to

denote

both

the

static

Cilk

thread

-LRB-

identified

by

the

keyword

thread

-RRB-

that

is

part

of

a

Cilk

program

and

the

dynamically

spawned

thread

that

occurs

at

runtime

Spawning

successor

and

child

threads

is

done

with

the

spawn

next

and

spawn

keywords

respectively

Sending

a

value

to

a

waiting

thread

is

done

with

the

send

argument

or

accumulate

statement

The

Cilk

runtime

system

implements

these

primitives

using

two

basic

data

structures

closures

and

continuations

as

illustrated

in

Figure

waiting

r

eady

b

codec

ontinuation

join

counters

arguments

Figure

Closures

and

continuations

Closures

are

data

structures

employed

by

the

runtime

system

to

keep

track

of

and

schedule

the

execution

of

spawned

threads

Whenever

a

thread

is

spawned

the

runtime

system

allocates

a

closure

for

it

from

a

simple

heap

A

closure

for

a

thread

T

consists

of

a

pointer

to

the

code

for

T

a

slot

for

each

of

T

's

specified

arguments

and

a

join

counter

indicating

the

number

of

missing

arguments

that

need

to

be

supplied

before

T

is

ready

to

run

The

closure

or

equivalently

the

spawned

thread

is

ready

if

it

has

obtained

all

of

its

arguments

and

it

is

waiting

if

some

arguments

are

missing

Figure

shows

a

closure

for

the

fib

thread

its

join

counter

is

zero

so

the

thread

is

ready

and

it

contains

two

arguments

a

continuation

k

-LRB-

as

explained

below

-RRB-

and

an

integer

n.

Notice

that

these

two

arguments

are

the

formal

arguments

declared

in

the

definition

of

the

fib

thread

To

run

a

ready

closure

the

Cilk

scheduler

invokes

the

thread

using

the

values

in

the

closure

as

arguments

When

the

thread

dies

the

closure

is

freed

A

continuation

is

a

global

reference

to

an

empty

argument

slot

of

a

closure

implemented

as

a

compound

data

structure

containing

a

pointer

to

a

closure

and

an

offset

that

designates

one

of

the

closure

's

argument

slots

-LRB-

see

Figure

-RRB-

Continuations

are

typed

with

the

C

data

The

Cilk

language

and

runtime

system

Figure

A

closure

for

the

fib

thread

type

of

the

slot

in

the

closure

In

the

Cilk

language

continuations

are

declared

by

the

type

modifier

keyword

cont

For

example

the

fib

thread

declares

two

integer

continuations

x

and

y

A

thread

spawns

a

successor

thread

by

creating

a

closure

for

the

successor

The

successor

thread

is

part

of

the

same

procedure

as

its

predecessor

Spawning

a

successor

thread

T

is

specified

in

the

Cilk

language

as

follows

spawn

next

T

-LRB-

args

-RRB-

This

statement

allocates

a

closure

for

thread

T

fills

in

all

available

arguments

and

initializes

the

join

counter

to

the

number

of

missing

arguments

Available

arguments

are

specified

as

in

C.

To

specify

a

missing

argument

the

user

specifies

a

continuation

variable

preceded

by

a

question

mark

For

example

in

the

fib

thread

the

statement

spawn

next

sum

-LRB-

k

x

y

-RRB-

allocates

a

closure

with

sum

as

the

thread

and

three

argument

slots

as

shown

in

Figure

The

first

slot

is

initialized

with

the

continuation

k

and

the

last

two

slots

are

empty

The

continuation

variables

x

and

y

are

initialized

to

refer

to

these

two

empty

slots

and

the

join

counter

is

set

to

This

closure

is

waiting

In

general

if

the

closure

is

ready

then

spawn

next

causes

the

closure

to

be

immediately

posted

to

the

scheduler

for

execution

In

typical

applications

successor

threads

are

spawned

waiting

for

some

missing

arguments

y

Figure

When

the

fib

thread

spawns

its

successor

sum

it

creates

a

new

closure

with

sum

as

the

thread

and

two

empty

argument

slots

referred

to

by

the

continuations

x

and

y

The

background

shading

denotes

that

both

threads

are

part

of

the

same

procedure

A

thread

spawns

a

child

thread

by

creating

a

closure

for

the

child

The

child

thread

is

the

initial

thread

of

a

newly

spawned

child

procedure

Spawning

a

child

thread

T

is

specified

in

the

Cilk

language

as

follows

spawn

T

-LRB-

args

-RRB-

Chapter

Parallel

programming

in

Cilk

This

statement

is

semantically

identical

to

spawn

next

but

it

informs

the

scheduler

that

the

new

closure

should

be

treated

as

a

child

as

opposed

to

a

successor

-LRB-

This

difference

will

be

explained

when

we

discuss

the

scheduler

-RRB-

For

example

the

fib

thread

spawns

two

children

as

shown

in

Figure

The

statement

spawn

fib

-LRB-

x

-RRB-

allocates

a

closure

with

fib

as

the

thread

and

two

argument

slots

The

first

slot

is

initialized

with

the

continuation

x

which

as

a

consequence

of

the

previous

statement

refers

to

a

slot

in

its

parent

's

successor

closure

The

second

slot

is

initialized

with

the

value

of

The

join

counter

is

set

to

zero

so

the

thread

is

ready

and

it

is

posted

to

the

scheduler

for

execution

In

typical

applications

child

threads

are

spawned

with

no

missing

arguments

Figure

When

the

fib

thread

spawns

children

it

creates

for

each

child

a

new

closure

with

fib

as

the

thread

and

no

empty

argument

slots

Each

child

closure

has

a

continuation

referring

to

an

empty

slot

in

the

parent

's

successor

closure

Each

child

thread

is

the

initial

thread

of

a

new

child

procedure

A

thread

sends

a

value

to

a

waiting

thread

by

placing

the

value

into

an

argument

slot

of

the

waiting

thread

's

closure

Cilk

provides

the

following

primitives

to

send

values

from

one

thread

to

another

send

argument

-LRB-

k

value

-RRB-

accumulate

-LRB-

k

op

value

-RRB-

The

send

argument

statement

sends

the

value

value

to

the

empty

argument

slot

of

a

waiting

closure

specified

by

the

continuation

k

The

types

of

the

continuation

and

the

value

must

be

compatible

The

join

counter

of

the

waiting

closure

is

decremented

and

if

it

becomes

zero

then

the

closure

is

ready

and

is

posted

to

the

scheduler

For

example

when

the

fib

thread

reaches

the

boundary

case

the

statement

send

argument

-LRB-

k

n

-RRB-

writes

the

value

of

n

into

an

empty

argument

slot

in

the

parent

procedure

's

waiting

sum

closure

and

decrements

its

join

counter

When

the

sum

closure

's

join

counter

reaches

zero

it

is

posted

to

the

scheduler

When

the

sum

thread

gets

executed

it

adds

its

two

arguments

x

and

y

The

Cilk

language

and

runtime

system

fibsumfibsumfibfibfib

Figure

The

dag

grown

by

an

execution

of

the

Fibonacci

program

Dag

edges

created

by

spawn

next

are

horizontal

dag

edges

created

by

spawn

are

straight

shaded

and

point

downward

and

dag

edges

created

by

send

argument

are

curved

and

point

upwards

The

background

shading

groups

the

threads

into

procedures

and

then

uses

send

argument

to

return

this

result

up

to

its

parent

procedure

's

waiting

sum

thread

Cilk

also

provides

a

special

type

of

continuation

called

a

signal

that

can

be

used

in

a

send

argument

without

any

value

The

accumulate

statement

is

the

same

as

send

argument

except

that

it

uses

the

function

op

to

accumulate

value

into

the

argument

slot

This

style

of

linking

Cilk

threads

is

called

continuation-passing

style

-LSB-

-RSB-

and

contrasts

with

the

spawn/return

style

that

links

procedures

In

spawn/return

style

a

spawned

child

always

returns

to

its

parent

and

the

parent

after

performing

the

spawn

can

suspend

waiting

for

the

child

to

return

That

the

child

should

return

values

and

control

to

its

parent

is

implicit

in

the

spawn/return

style

In

continuation-passing

style

a

spawned

child

never

returns

to

its

parent

Cilk

threads

never

return

and

they

never

sync

Instead

a

spawned

child

is

passed

a

continuation

as

an

explicit

argument

that

specifies

where

values

and

control

should

go

when

it

dies

At

runtime

a

Cilk

program

grows

a

dag

-LRB-

directed

acyclic

graph

-RRB-

of

threads

that

unfolds

dynamically

as

threads

spawn

successors

and

children

For

example

Figure

shows

the

dag

grown

by

an

execution

of

the

Fibonacci

program

The

dag

contains

an

edge

from

one

thread

to

another

if

either

the

first

thread

spawned

the

second

-LRB-

with

spawn

next

or

spawn

-RRB-

or

the

first

thread

sends

a

value

-LRB-

or

signal

-RRB-

to

the

second

-LRB-

with

send

argument

or

accumulate

-RRB-

We

can

think

of

closures

and

continuations

as

the

data

structures

employed

by

the

runtime

system

to

keep

track

of

the

dag

as

it

grows

This

dag

is

analogous

to

the

dag

described

in

Chapter

except

that

rather

than

having

unit-size

instructions

as

dag

nodes

this

dag

has

arbitrary-size

threads

We

shall

examine

the

consequences

of

this

difference

in

Section

Chapter

Parallel

programming

in

Cilk

To

summarize

the

Fibonacci

procedure

it

consists

of

two

threads

fib

and

its

successor

sum

Reflecting

the

explicit

continuation

passing

style

the

first

argument

to

each

thread

is

the

continuation

specifying

where

the

return

value

should

be

placed

When

the

fib

thread

is

invoked

it

first

checks

to

see

if

the

boundary

case

has

been

reached

in

which

case

it

uses

send

argument

to

send

the

value

of

n

as

an

argument

to

the

waiting

thread

specified

by

continuation

k.

Otherwise

it

spawns

the

successor

thread

sum

as

well

as

two

children

to

compute

the

two

subcases

Each

of

these

two

children

is

given

a

continuation

specifying

to

which

argument

in

the

sum

thread

it

should

send

its

result

The

sum

thread

simply

adds

the

two

arguments

when

they

arrive

and

sends

this

result

to

the

waiting

thread

designated

by

k.

Although

writing

in

explicit

continuation

passing

style

is

somewhat

onerous

for

the

programmer

the

decision

to

break

procedures

into

separate

nonsuspending

threads

with

heapallocated

closures

simplifies

the

Cilk

runtime

system

Each

Cilk

thread

runs

to

completion

without

suspending

and

leaves

the

C

runtime

stack

empty

when

it

dies

A

common

alternative

-LSB-

-RSB-

is

to

directly

support

spawn/return

threads

-LRB-

or

procedures

-RRB-

in

the

runtime

system

possibly

with

stack-allocated

activation

frames

In

such

a

system

threads

can

suspend

waiting

for

synchronization

and

leave

temporary

values

on

the

calling

stack

Consequently

this

alternative

strategy

requires

that

the

runtime

system

either

employs

multiple

stacks

or

a

mechanism

to

save

these

temporaries

in

heap-allocated

storage

Another

advantage

of

Cilk

's

strategy

is

that

it

allows

multiple

children

to

be

spawned

from

a

single

nonsuspending

thread

which

saves

on

context

switching

In

Cilk

r

children

can

be

spawned

and

executed

with

only

r

context

switches

whereas

the

alternative

of

suspending

whenever

a

thread

is

spawned

causes

context

switches

Since

our

primary

interest

is

in

understanding

how

to

build

runtime

systems

that

efficiently

schedule

multithreaded

programs

we

chose

the

alternative

of

burdening

the

programmer

with

a

requirement

which

is

perhaps

less

elegant

linguistically

but

which

yields

a

simple

and

portable

runtime

system

implementation

Cilk

supports

a

variety

of

additional

primitives

that

give

the

programmer

greater

control

over

runtime

performance

For

example

when

the

last

action

of

a

thread

is

to

spawn

a

child

the

programmer

can

use

the

keyword

call

instead

of

spawn

to

call

the

new

thread

immediately

without

invoking

the

scheduler

Additionally

if

the

called

thread

is

the

same

as

the

callee

then

the

programmer

can

use

the

tail

call

keyword

that

produces

a

tail

call

to

avoid

both

the

scheduler

overhead

and

the

C

function

call

overhead

Cilk

also

allows

arrays

and

subarrays

to

be

passed

as

arguments

to

threads

Other

features

include

various

abilities

to

override

the

scheduler

's

decisions

including

how

to

pack

and

unpack

data

when

a

closure

is

migrated

from

one

processor

to

another

The

Cilk

work-stealing

scheduler

Cilk

's

scheduler

uses

the

technique

of

work

stealing

in

which

a

processor

-LRB-

the

thief

-RRB-

who

runs

out

of

work

selects

another

processor

-LRB-

the

victim

-RRB-

from

whom

to

steal

work

and

then

steals

the

shallowest

ready

thread

in

the

victim

's

spawn

tree

Cilk

's

strategy

is

for

thieves

to

choose

victims

at

random

Essentially

Cilk

implements

the

scheduling

algorithm

described

and

analyzed

in

Chapter

We

shall

now

present

Cilk

's

implementation

of

this

algorithm

At

runtime

each

processor

maintains

a

local

ready

pool

to

hold

ready

closures

Each

closure

has

an

associated

level

which

corresponds

to

the

thread

's

depth

in

the

spawn

tree

The

Cilk

language

and

runtime

system

The

closures

for

the

threads

in

the

cilk

main

procedure

have

level

the

closures

for

the

threads

in

cilk

main

's

child

procedures

have

level

and

so

on

The

ready

pool

is

an

array

illustrated

in

Figure

in

which

the

Lth

element

contains

a

linked

list

of

all

ready

closures

having

level

L.

Cilk

begins

executing

the

user

program

by

initializing

all

ready

pools

to

be

empty

placing

the

initial

thread

of

cilk

main

into

the

list

of

Processor

's

pool

and

then

starting

a

scheduling

loop

on

each

processor

At

each

iteration

through

the

scheduling

loop

a

processor

first

checks

to

see

whether

its

ready

pool

is

empty

If

it

is

the

processor

commences

work

stealing

which

will

be

described

shortly

Otherwise

the

processor

performs

the

following

steps

Remove

the

closure

at

the

head

of

the

list

of

the

deepest

nonempty

level

in

the

ready

pool

Extract

the

thread

from

the

closure

and

invoke

it

As

a

thread

executes

it

may

spawn

or

send

arguments

to

other

threads

When

the

thread

dies

control

returns

to

the

scheduling

loop

which

advances

to

the

next

iteration

When

a

thread

at

level

L

performs

a

spawn

of

a

child

thread

T

the

processor

executes

the

following

operations

Allocate

and

initialize

a

closure

for

T

Copy

the

available

arguments

into

the

closure

initialize

any

continuations

to

point

to

missing

arguments

and

initialize

the

join

counter

to

the

number

of

missing

arguments

Label

the

closure

with

level

L

If

there

are

no

missing

arguments

post

the

closure

to

the

ready

pool

by

inserting

it

at

the

head

of

the

level

-LRB-

L

-RRB-

list

Execution

of

spawn

next

is

similar

except

that

the

closure

is

labeled

with

level

L

and

if

it

is

ready

posted

to

the

level-L

list

When

a

thread

performs

a

send

argument

-LRB-

k

value

-RRB-

the

processor

executes

the

following

operations

Find

the

closure

and

argument

slot

referenced

by

the

continuation

k

Place

value

in

the

argument

slot

and

decrement

the

join

counter

of

the

closure

If

the

join

counter

goes

to

zero

post

the

closure

to

the

ready

pool

at

the

appropriate

level

Execution

of

accumulate

is

similar

except

that

the

value

is

accumulated

into

the

argument

slot

with

a

combining

function

When

the

continuation

k

refers

to

a

closure

on

a

remote

processor

network

communication

ensues

In

this

case

the

processor

that

initiated

the

send

argument

function

sends

a

message

to

the

remote

processor

to

perform

the

operations

The

only

subtlety

occurs

in

step

If

the

closure

must

be

posted

it

is

posted

to

the

ready

pool

of

the

initiating

processor

rather

than

to

that

of

the

remote

processor

This

Chapter

Parallel

programming

in

Cilk

next

closure

to

steal

next

closure

to

execute

Figure

A

processor

's

ready

pool

At

each

iteration

through

the

scheduling

loop

the

processor

executes

the

closure

at

the

head

of

the

deepest

nonempty

level

in

the

ready

pool

If

the

ready

pool

is

empty

the

processor

becomes

a

thief

and

steals

the

closure

at

the

tail

of

the

shallowest

nonempty

level

in

the

ready

pool

of

a

victim

processor

chosen

uniformly

at

random

Performance

of

Cilk

applications

policy

is

necessary

for

the

scheduler

to

be

provably

efficient

but

as

a

practical

matter

we

have

also

had

success

with

posting

the

closure

to

the

remote

processor

's

pool

If

a

processor

begins

an

iteration

of

the

scheduling

loop

and

finds

that

its

ready

pool

is

empty

the

processor

becomes

a

thief

and

commences

work

stealing

as

follows

Select

a

victim

processor

uniformly

at

random

If

the

victim

's

ready

pool

is

empty

go

back

to

step

If

the

victim

's

ready

pool

is

nonempty

extract

the

closure

from

the

tail

of

the

list

in

the

shallowest

nonempty

level

of

the

ready

pool

and

execute

it

Work

stealing

is

implemented

with

a

simple

request-reply

communication

protocol

between

the

thief

and

victim

Why

steal

work

from

the

shallowest

level

of

the

ready

pool

The

reason

is

two-fold

one

heuristic

and

one

algorithmic

First

to

lower

communication

costs

we

would

like

to

steal

large

amounts

of

work

and

in

a

tree-structured

computation

shallow

threads

are

likely

to

spawn

more

work

than

deep

ones

This

heuristic

notion

is

the

justification

cited

by

earlier

researchers

-LSB-

-RSB-

who

proposed

stealing

work

that

is

shallow

in

the

spawn

tree

We

can

not

however

prove

that

shallow

threads

are

more

likely

to

spawn

work

than

deep

ones

What

we

prove

in

Section

is

the

following

algorithmic

property

The

threads

that

are

on

the

critical

path

in

the

dag

are

always

at

the

shallowest

level

of

a

processor

's

ready

pool

Consequently

if

processors

are

idle

the

work

they

steal

makes

progress

along

the

critical

path

Performance

of

Cilk

applications

The

Cilk

runtime

system

executes

Cilk

applications

efficiently

and

with

predictable

performance

Specifically

for

dynamic

asynchronous

tree-like

applications

Cilk

's

work-stealing

scheduler

produces

near

optimal

parallel

speedup

while

using

small

amounts

of

space

and

communication

Furthermore

Cilk

application

performance

can

be

modeled

accurately

as

a

simple

function

of

work

and

critical

path

length

In

this

section

we

empirically

demonstrate

these

facts

by

experimenting

with

several

applications

This

section

begins

with

a

look

at

these

applications

and

then

proceeds

with

a

look

at

the

performance

of

these

applications

We

close

this

section

with

a

look

at

application

performance

modeling

The

empirical

results

of

this

section

confirm

the

analytical

results

of

the

next

section

Cilk

applications

We

experimented

with

the

Cilk

runtime

system

using

several

applications

some

synthetic

and

some

real

The

applications

are

described

below

ffl

fib

is

the

same

as

was

presented

in

Section

except

that

the

second

recursive

spawn

is

replaced

by

a

tail

call

that

avoids

the

scheduler

This

program

is

a

good

measure

of

Cilk

overhead

because

the

thread

length

is

so

small

Chapter

Parallel

programming

in

Cilk

ffl

queens

is

a

backtrack-search

program

that

solves

the

problem

of

placing

N

queens

on

a

N?N

chessboard

so

that

no

two

queens

attack

each

other

The

Cilk

program

is

based

on

serial

code

by

R.

Sargent

of

MIT

's

Media

Laboratory

Thread

length

was

enhanced

by

serializing

the

bottom

levels

of

the

search

tree

ffl

pfold

is

a

protein-foldingprogram

that

finds

hamiltonian

paths

in

a

three-dimensional

grid

using

backtrack

search

-LSB-

-RSB-

Written

by

Chris

Joerg

of

MIT

's

Laboratory

for

Computer

Science

and

V.

Pande

of

MIT

's

Center

for

Material

Sciences

and

Engineering

pfoldwas

the

first

program

to

enumerate

all

hamiltonian

paths

in

a

grid

We

timed

the

enumeration

of

all

paths

starting

with

a

certain

sequence

ffl

ray

is

a

parallel

program

for

graphics

rendering

based

on

the

serial

POV-Ray

program

which

uses

a

ray-tracing

algorithm

The

entire

POV-Ray

system

contains

over

lines

of

C

code

but

the

core

of

POV-Ray

is

a

simple

doubly

nested

loop

that

iterates

over

each

pixel

in

a

two-dimensional

image

For

ray

we

converted

the

nested

loops

into

a

divide-and-conquer

control

structure

using

spawns

Our

measurements

do

not

include

the

approximately

seconds

of

startup

time

required

to

read

and

process

the

scene

description

file

ffl

knary

-LRB-

n

k

r

-RRB-

is

a

synthetic

benchmark

whose

parameters

can

be

set

to

produce

a

variety

of

values

for

work

and

critical

path

length

It

generates

a

tree

of

depth

n

and

branching

factor

k

in

which

the

first

r

children

at

every

level

are

executed

serially

and

the

remainder

are

executed

in

parallel

At

each

node

of

the

tree

the

program

runs

an

empty

for

loop

for

iterations

ffl

Socrates

is

a

parallel

chess

program

that

uses

the

Jamboree

search

algorithm

-LSB-

-RSB-

to

parallelize

a

minmax

tree

search

The

work

of

the

algorithm

varies

with

the

number

of

processors

because

it

does

speculative

work

that

may

be

aborted

during

runtime

Socrates

was

written

by

a

team

of

engineers

headed

by

Charles

Leiserson

of

MIT

's

Laboratory

for

Computer

Science

with

Don

Dailey

formerly

of

Heuristic

Software

and

Chris

Joerg

of

MIT

's

Laboratory

for

Computer

Science

as

the

lead

programmers

Socrates

won

second

prize

in

the

ICCA

World

Computer

Chess

Championship

running

on

the

Intel

Paragon

at

Sandia

National

Laboratories

Many

of

these

applications

place

heavy

demands

on

the

runtime

system

due

to

their

dynamic

and

irregular

nature

For

example

in

the

case

of

queens

and

pfold

the

size

and

shape

of

the

backtrack-search

tree

can

not

be

determined

without

actually

performing

the

the

Cilk

ray

program

was

about

percent

faster

than

the

serial

POV-Rayprogram

running

on

one

processor

The

reason

was

that

the

divide-and-conquer

decomposition

performed

by

the

Cilk

code

provides

better

locality

than

the

doubly

nested

loop

of

the

serial

code

Modifying

the

serial

code

to

imitate

the

Cilk

decomposition

improved

its

performance

Timings

for

the

improved

version

are

given

in

Figure

other

members

of

the

Socrates

team

are

I.M.

Larry

Kaufmann

formerly

of

Heuristic

Software

Robert

Blumofe

of

MIT

's

Laboratory

for

Computer

Science

Bradley

Kuszmaul

formerly

of

MIT

's

Laboratory

for

Computer

Science

and

now

of

the

Computer

Science

Department

at

Yale

University

Rolf

Riesen

of

Sandia

National

Laboratories

and

Yuli

Zhou

formerly

of

MIT

's

Laboratory

for

Computer

Science

and

now

of

AT&T

Bell

Laboratories

Performance

of

Cilk

applications

search

and

the

shape

of

the

tree

often

turns

out

to

be

highly

irregular

With

speculative

work

that

may

be

aborted

the

Socrates

minmax

tree

carries

this

dynamic

and

irregular

structure

to

the

extreme

In

the

case

of

ray

the

amount

of

time

it

takes

to

compute

the

color

of

a

pixel

in

an

image

is

hard

to

predict

and

may

vary

widely

from

pixel

to

pixel

as

illustrated

in

Figure

In

all

of

these

cases

high

performance

demands

efficient

dynamic

load

balancing

at

runtime

-LRB-

a

-RRB-

Ray-traced

image

-LRB-

b

-RRB-

Work

at

each

pixel

Figure

-LRB-

a

-RRB-

An

image

rendered

with

the

ray

program

-LRB-

b

-RRB-

This

image

shows

the

amount

of

time

ray

took

to

compute

each

pixel

value

The

whiter

the

pixel

the

longer

ray

worked

to

compute

the

corresponding

pixel

value

All

experiments

were

run

on

a

supercomputer

The

is

a

massively

parallel

computer

based

on

SPARC

processors

with

a

fat-tree

interconnection

network

-LSB-

-RSB-

The

Cilk

runtime

system

on

the

performs

communication

among

processors

using

the

Strata

-LSB-

-RSB-

active-message

library

Application

performance

By

running

our

applications

and

measuring

a

suite

of

performance

parameters

we

empirically

answer

a

number

of

questions

about

the

effectiveness

of

the

Cilk

runtime

system

We

focus

on

the

following

questions

How

efficiently

does

the

runtime

system

implement

the

language

primitives

As

we

add

processors

how

much

faster

will

the

program

run

How

much

more

space

will

it

require

And

how

much

more

communication

will

it

perform

We

show

that

for

dynamic

asynchronous

tree-like

programs

the

Cilk

runtime

system

efficiently

implements

the

language

primitives

and

is

simultaneously

efficient

with

respect

to

time

space

and

communication

In

Section

we

reach

the

same

conclusion

by

analytic

means

but

in

this

section

we

focus

on

empirical

data

from

the

execution

of

our

Cilk

programs

Chapter

Parallel

programming

in

Cilk

The

execution

of

a

Cilk

program

with

a

given

set

of

inputs

grows

a

Cilk

computation

that

consists

of

a

tree

of

procedures

and

a

dag

of

threads

These

structures

were

introduced

in

the

previous

section

We

benchmark

our

applications

based

on

two

fundamental

measures

of

the

computation

work

and

critical

path

length

The

work

denoted

by

is

the

time

to

execute

the

Cilk

computation

on

one

processor

which

corresponds

to

the

sum

of

the

execution

times

of

all

the

threads

in

the

dag

The

method

used

to

measure

depends

on

whether

the

program

is

deterministic

For

deterministic

programs

the

computation

only

depends

on

the

program

and

its

inputs

and

hence

it

is

independent

of

the

number

of

processors

and

runtime

scheduling

decisions

All

of

our

applications

except

Socrates

are

deterministic

For

these

deterministic

applications

the

work

performed

by

any

P-processor

run

of

the

program

is

equal

to

the

work

performed

by

a

run

-LRB-

with

the

same

input

values

-RRB-

so

we

measure

the

work

directly

by

timing

the

run

The

Socrates

program

on

the

other

hand

uses

speculative

execution

and

therefore

the

computation

depends

on

the

number

of

processors

and

scheduling

decisions

made

at

runtime

In

this

case

timing

a

run

is

not

a

reasonable

way

to

measure

the

work

performed

by

a

run

with

more

processors

We

must

realize

that

the

work

of

an

execution

with

P

processors

is

defined

as

the

time

it

takes

to

execute

the

same

computation

not

the

same

program

-LRB-

with

the

same

inputs

-RRB-

For

Socrates

we

estimate

the

work

of

a

P-processor

run

by

performing

the

P-processor

run

and

timing

the

execution

of

every

thread

and

summing

This

method

yields

an

underestimate

since

it

does

not

include

scheduling

costs

In

either

case

a

P-processor

execution

of

a

Cilk

computation

with

work

must

take

time

at

least

=P

A

P-processor

execution

that

takes

time

equal

to

this

=P

lower

bound

is

said

to

achieve

perfect

linear

speedup

The

critical

path

length

denoted

by

T

is

the

time

to

execute

the

Cilk

computation

with

infinitely

many

processors

which

corresponds

to

the

largest

sum

of

thread

execution

times

along

any

path

in

the

dag

We

measure

critical

path

length

by

timestamping

each

thread

in

the

dag

with

the

earliest

time

at

which

it

could

have

been

executed

Specifically

this

timestamp

is

the

maximum

of

the

earliest

time

that

the

thread

could

have

been

spawned

and

for

each

argument

the

earliest

time

that

the

argument

could

have

been

sent

These

values

in

turn

are

computed

from

the

timestamp

of

the

thread

that

performed

the

spawn

or

sent

the

argument

In

particular

if

a

thread

performs

a

spawn

then

the

earliest

time

that

the

spawn

could

occur

is

equal

to

the

earliest

time

at

which

the

thread

could

have

been

executed

-LRB-

its

timestamp

-RRB-

plus

the

amount

of

time

the

thread

ran

for

until

it

performed

the

spawn

The

same

property

holds

for

the

earliest

time

that

an

argument

could

be

sent

The

initial

thread

of

cilk

main

is

timestamped

zero

and

the

critical

path

length

is

then

computed

as

the

maximum

over

all

threads

of

its

timestamp

plus

the

amount

of

time

it

executes

for

The

measured

critical

path

length

does

not

include

scheduling

and

communication

costs

A

P-processor

execution

of

a

Cilk

computation

must

take

at

least

as

long

as

the

computation

's

critical

path

length

T

Thus

if

T

exceeds

=P

then

perfect

linear

speedup

can

not

be

achieved

Figure

is

a

table

showing

typical

performance

measures

for

our

Cilk

applications

Each

column

presents

data

from

a

single

run

of

a

benchmark

application

We

adopt

the

fol

programs

are

deterministic

if

we

consider

the

sequence

of

values

generated

by

the

source

of

randomness

to

be

inputs

to

the

program

practice

we

sometimes

beat

the

=P

lower

bound

Such

superlinear

speedup

is

a

consequence

of

the

fact

that

as

we

add

processors

we

also

add

other

physical

resources

such

as

registers

cache

and

main

memory

fib

queens

pfold

ray

knary

knary

Socrates

Socrates

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

depth

-RRB-

-LRB-

depth

-RRB-

-LRB-

proc

-RRB-

-LRB-

proc

-RRB-

-LRB-

computation

parameters

-RRB-

Tserial

Tserial

T

T

threads

thread

length

ms

ms

ms

ms

ms

ms

-LRB-

experiments

-RRB-

TP

=P

T

TP

-LRB-

P

TP

-RRB-

space/proc

requests/proc

steals/proc

-LRB-

experiments

-RRB-

TP

=P

T

TP

-LRB-

P

TP

-RRB-

space/proc

requests/proc

steals/proc

Figure

Performance

of

Cilk

on

various

applications

All

times

are

in

seconds

except

where

noted

Chapter

Parallel

programming

in

Cilk

lowing

notations

For

each

application

we

have

an

efficient

serial

C

implementation

compiled

using

gcc

whose

measured

runtime

is

denoted

Tserial

The

Cilk

computation

's

work

and

critical

path

length

T

are

measured

on

the

as

described

above

The

measured

execution

time

of

the

Cilk

program

running

on

P

processors

of

the

is

given

by

TP

The

row

labeled

threads

indicates

the

number

of

threads

executed

and

thread

length

is

the

average

thread

length

-LRB-

work

divided

by

the

number

of

threads

-RRB-

Certain

derived

parameters

are

also

displayed

in

the

table

The

ratio

Tserial

is

the

efficiency

of

the

Cilk

program

relative

to

the

C

program

The

ratio

T

is

the

average

parallelism

The

value

=P

T

is

a

simple

model

of

the

runtime

which

will

be

discussed

later

The

speedup

is

TP

and

the

parallel

efficiency

is

-LRB-

P

TP

-RRB-

The

row

labeled

space/proc

indicates

the

maximum

number

of

closures

allocated

at

any

time

on

any

processor

The

row

labeled

requests/proc

indicates

the

average

number

of

steal

requests

made

by

a

processor

during

the

execution

and

steals/proc

gives

the

average

number

of

closures

actually

stolen

The

data

in

Figure

shows

two

important

relationships

one

between

efficiency

and

thread

length

and

another

between

speedup

and

average

parallelism

Considering

the

relationship

between

efficiency

Tserial

and

thread

length

we

see

that

for

programs

with

moderately

long

threads

the

Cilk

runtime

system

induces

little

overhead

The

queens

pfold

ray

and

knary

programs

have

threads

with

average

length

greater

than

microseconds

and

have

efficiency

greater

than

percent

On

the

other

hand

the

fib

program

has

low

efficiency

because

the

threads

are

so

short

fib

does

almost

nothing

besides

spawn

and

send

argument

Despite

it

's

long

threads

the

Socratesprogram

has

low

efficiency

because

its

parallel

Jamboree

search

algorithm

is

based

on

speculatively

searching

subtrees

that

are

not

searched

by

a

serial

algorithm

Consequently

as

we

increase

the

number

of

processors

the

program

executes

more

threads

and

hence

does

more

work

For

example

the

execution

did

seconds

of

work

whereas

the

execution

did

only

seconds

of

work

Both

of

these

executions

did

considerably

more

work

than

the

serial

program

's

seconds

of

work

Thus

although

we

observe

low

efficiency

it

is

due

to

the

parallel

algorithm

and

not

to

Cilk

overhead

Looking

at

the

speedup

TP

measured

on

and

processors

we

see

that

when

the

average

parallelism

T

is

large

compared

with

the

number

P

of

processors

Cilk

programs

achieve

nearly

perfect

linear

speedup

but

when

the

average

parallelism

is

small

the

speedup

is

much

less

The

fib

queens

pfold

and

ray

programs

for

example

have

in

excess

of

parallelism

and

achieve

more

than

percent

of

perfect

linear

speedup

on

processors

and

more

than

percent

of

perfect

linear

speedup

on

processors

The

Socrates

program

exhibits

somewhat

less

parallelism

and

also

somewhat

less

speedup

On

processors

the

Socratesprogram

has

parallelism

yielding

percent

of

perfect

linear

speedup

while

on

processors

it

has

parallelism

yielding

percent

of

perfect

linear

speedup

With

even

less

parallelism

as

exhibited

in

the

knary

benchmarks

less

speedup

is

obtained

For

example

the

knary

-LRB-

-RRB-

benchmark

exhibits

only

parallelism

and

it

realizes

barely

more

than

speedup

on

processors

fact

the

ray

program

achieves

superlinear

speedup

even

when

comparing

to

the

efficient

serial

implementation

We

suspect

that

cache

effects

cause

this

phenomenon

Performance

of

Cilk

applications

-LRB-

less

than

percent

of

perfect

linear

speedup

-RRB-

With

parallelism

knary

-LRB-

-RRB-

achieves

speedup

on

processors

-LRB-

percent

of

perfect

linear

speedup

-RRB-

but

only

speedup

on

processors

-LRB-

percent

of

perfect

linear

speedup

-RRB-

Although

these

speedup

measures

reflect

the

Cilk

scheduler

's

ability

to

exploit

parallelism

to

obtain

application

speedup

we

must

factor

in

the

efficiency

of

the

Cilk

program

compared

with

the

serial

C

program

Specifically

the

application

speedup

Tserial

TP

is

the

product

of

efficiency

Tserial

and

speedup

TP

For

example

applications

such

as

fib

and

Socrateswith

low

efficiency

generate

correspondingly

low

application

speedup

The

Socrates

program

with

efficiency

and

speedup

on

processors

exhibits

application

speedup

of

For

the

purpose

of

understanding

scheduler

performance

we

prefer

to

decouple

the

efficiency

of

the

application

from

the

efficiency

of

the

scheduler

Looking

more

carefully

at

the

cost

of

a

spawn

in

Cilk

we

find

that

it

takes

a

fixed

overhead

of

about

cycles

to

allocate

and

initialize

a

closure

plus

about

cycles

for

each

word

argument

In

comparison

a

C

function

call

on

a

SPARC

processor

takes

cycles

of

fixed

overhead

-LRB-

assuming

no

register

window

overflow

-RRB-

plus

cycle

for

each

word

argument

-LRB-

assuming

all

arguments

are

transferred

in

registers

-RRB-

Thus

a

spawn

in

Cilk

is

roughly

an

order

of

magnitude

more

expensive

than

a

C

function

call

This

Cilk

overhead

is

quite

apparent

in

the

fib

program

which

does

almost

nothing

besides

spawn

and

send

argument

Based

on

fib

's

measured

efficiency

of

we

can

conclude

that

the

aggregate

average

cost

of

a

spawn/send

argument

in

Cilk

is

between

and

times

the

cost

of

a

function

call/return

in

C.

Efficient

execution

of

programs

with

short

threads

requires

a

low-overhead

spawn

operation

As

can

be

observed

from

Figure

the

vast

majority

of

threads

execute

on

the

same

processor

on

which

they

are

spawned

For

example

the

fib

program

executed

over

million

threads

but

migrated

only

-LRB-

per

processor

-RRB-

when

run

with

processors

Taking

advantage

of

this

property

other

researchers

-LSB-

-RSB-

have

developed

techniques

for

implementing

spawns

such

that

when

the

child

thread

executes

on

the

same

processor

as

its

parent

the

cost

of

the

spawn

operation

is

roughly

equal

the

cost

of

a

function

call

We

hope

to

incorporate

such

techniques

into

future

implementations

of

Cilk

Finally

we

make

two

observations

concerning

the

space

and

communication

measures

in

Figure

Looking

at

the

space/proc

rows

we

observe

that

the

space

per

processor

is

generally

quite

small

and

does

not

grow

with

the

number

of

processors

For

example

Socrates

on

processors

executes

over

million

threads

yet

no

processor

ever

has

more

than

allocated

closures

On

processors

the

number

of

executed

threads

nearly

doubles

to

over

million

but

the

space

per

processor

barely

changes

In

Section

we

show

formally

that

for

an

important

class

of

Cilk

programs

the

space

per

processor

does

not

grow

as

we

add

processors

Looking

at

the

requests/proc

and

steals/proc

rows

in

Figure

we

observe

that

the

amount

of

communication

grows

with

the

critical

path

length

but

does

not

grow

with

the

work

For

example

fib

queens

pfold

and

ray

all

have

critical

path

lengths

under

a

tenth

of

a

second

long

and

perform

fewer

than

requests

and

steals

per

processor

whereas

knary

-LRB-

-RRB-

and

Socrates

have

critical

path

lengths

more

than

seconds

long

and

perform

more

than

requests

and

steals

per

processor

The

table

does

Chapter

Parallel

programming

in

Cilk

not

show

any

clear

correlation

between

work

and

either

requests

or

steals

For

example

ray

does

more

than

twice

as

much

work

as

knary

-LRB-

-RRB-

yet

it

performs

two

orders

of

magnitude

fewer

requests

In

Section

we

show

that

for

a

class

of

Cilk

programs

the

communication

per

processor

grows

at

most

linearly

with

the

critical

path

length

and

does

not

grow

as

a

function

of

the

work

Performance

modeling

We

further

document

the

effectiveness

of

the

Cilk

scheduler

by

showing

empirically

that

Cilk

application

performance

can

be

modeled

accurately

with

a

simple

function

of

work

and

critical

path

length

T

Specifically

we

use

the

knary

synthetic

benchmark

to

show

that

the

runtime

of

an

application

on

P

processors

can

be

modeled

as

TP

ss

=P

c?T

where

c

is

a

small

constant

-LRB-

about

for

knary

-RRB-

determined

by

curve

fitting

This

result

shows

that

we

obtain

nearly

perfect

linear

speedup

when

the

critical

path

is

short

compared

with

the

average

amount

of

work

per

processor

We

also

show

that

a

model

of

this

kind

is

accurate

even

for

Socrates

which

is

our

most

complex

application

programmed

to

date

We

would

like

our

scheduler

to

execute

a

Cilk

computation

with

work

in

=P

time

on

P

processors

Such

perfect

linear

speedup

can

not

be

obtained

whenever

the

computation

's

critical

path

length

T

exceeds

=P

since

we

always

have

TP

T

or

more

generally

TP

=P

T?g

The

critical

path

length

T

is

the

stronger

lower

bound

on

TP

whenever

P

exceeds

the

average

parallelism

T

and

=P

is

the

stronger

bound

otherwise

A

good

scheduler

should

meet

each

of

these

bounds

as

closely

as

possible

In

order

to

investigate

how

well

the

Cilk

scheduler

meets

these

two

lower

bounds

we

used

our

synthetic

knary

benchmark

which

can

grow

computations

that

exhibit

a

range

of

values

for

work

and

critical

path

length

Figure

shows

the

outcome

from

many

experiments

of

running

knary

with

various

input

values

-LRB-

n

k

and

r

-RRB-

on

various

numbers

of

processors

The

figure

plots

the

measured

speedup

TP

for

each

run

against

the

machine

size

P

for

that

run

In

order

to

compare

the

outcomes

for

runs

with

different

input

values

we

have

normalized

the

plotted

value

for

each

run

as

follows

In

addition

to

the

speedup

we

measure

for

each

run

the

work

and

the

critical

path

length

T

as

previously

described

We

then

normalize

the

machine

size

and

the

speedup

by

dividing

these

values

by

the

average

parallelism

T

For

each

run

the

horizontal

position

of

the

plotted

datum

is

P

-LRB-

T

-RRB-

and

the

vertical

position

of

the

plotted

datum

is

-LRB-

TP

-RRB-

-LRB-

T

-RRB-

T

TP

Consequently

on

the

horizontal

axis

the

normalized

machine-size

is

when

the

average

parallelism

is

equal

to

the

number

of

processors

On

the

vertical

axis

the

normalized

speedup

is

when

the

runtime

equals

the

critical

path

length

We

can

draw

the

two

lower

bounds

on

time

as

upper

bounds

on

speedup

The

horizontal

line

at

is

the

upper

bound

on

speedup

obtained

from

the

critical

path

TP

T

and

the

line

is

the

linear

speedup

bound

TP

=P

As

can

be

seen

from

the

figure

on

the

knary

runs

for

which

the

average

parallelism

exceeds

the

number

of

processors

-LRB-

normalized

machine

size

less

than

-RRB-

the

Cilk

scheduler

obtains

nearly

perfect

linear

speedup

In

the

region

where

the

number

of

processors

is

large

compared

to

the

average

parallelism

-LRB-

normalized

machine

size

greater

than

-RRB-

the

data

is

more

scattered

but

the

speedup

is

always

within

a

factor

of

of

the

critical-path

upper

bound

The

theoretical

results

from

Section

show

that

the

expected

running

time

of

a

Cilk

Performance

of

Cilk

applications

Curve

Fit

=P

Model

=P

Model

=P

Measured

Value

Linear

Speedup

Bound

Critical

Path

Bound

Normalized

Speedup

Normalized

Machine

Size

Figure

Normalized

speedups

for

the

knary

synthetic

benchmark

using

from

to

processors

The

horizontal

axis

is

the

number

P

of

processors

and

the

vertical

axis

is

the

speedup

TP

but

each

data

point

has

been

normalized

by

dividing

by

T

computation

on

P

processors

is

TP

O

-LRB-

=P

T

-RRB-

Thus

it

makes

sense

to

try

to

fit

the

knary

data

to

a

curve

of

the

form

TP

-LRB-

=P

-RRB-

c

-LRB-

T

-RRB-

A

least-squares

fit

to

the

data

to

minimize

the

relative

error

yields

and

c

with

percent

confidence

The

correlation

coefficient

of

the

fit

is

and

the

mean

relative

error

is

percent

The

curve

fit

is

shown

in

Figure

which

also

plots

the

simpler

curves

TP

=P

T

and

TP

=P

T

for

comparison

As

can

be

seen

from

the

figure

little

is

lost

in

the

linear

speedup

range

of

the

curve

by

assuming

that

the

coefficient

on

the

=P

term

equals

Indeed

a

fit

to

TP

=P

c

-LRB-

T

-RRB-

yields

c

with

and

a

mean

relative

error

of

percent

which

is

in

some

ways

better

than

the

fit

that

includes

a

term

-LRB-

The

measure

is

a

little

worse

but

the

mean

relative

error

is

much

better

-RRB-

It

makes

sense

that

the

data

points

become

more

scattered

when

P

is

close

to

or

exceeds

the

average

parallelism

In

this

range

the

amount

of

time

spent

in

work

stealing

becomes

a

significant

fraction

of

the

overall

execution

time

The

real

measure

of

the

quality

of

a

scheduler

is

how

much

larger

than

P

the

average

parallelism

T

must

be

before

TP

shows

substantial

influence

from

the

critical

path

One

can

see

from

Figure

that

if

the

average

parallelism

exceeds

P

by

a

factor

of

the

critical

path

has

almost

no

impact

on

the

running

time

Chapter

Parallel

programming

in

Cilk

Curve

Fit

=P

Model

=P

Model

=P

Measured

Value

Linear

Speedup

Bound

Critical

Path

Bound

Normalized

Speedup

Normalized

Machine

Size

Figure

Normalized

speedups

for

the

Socrates

chess

program

To

confirm

our

simple

model

of

the

Cilk

scheduler

's

performance

on

a

real

application

we

ran

Socrates

on

a

variety

of

chess

positions

using

various

numbers

of

processors

Figure

shows

the

results

of

our

study

which

confirm

the

results

from

the

knary

synthetic

benchmark

The

best

fit

to

TP

-LRB-

=P

-RRB-

c

-LRB-

T

-RRB-

yields

and

c

with

percent

confidence

The

correlation

coefficient

of

the

fit

is

and

the

mean

relative

error

is

percent

By

using

work

and

critical

path

length

to

model

the

performance

of

an

application

under

development

we

can

avoid

being

trapped

by

the

following

interesting

anomaly

After

making

an

improvement

to

the

program

we

find

that

in

test

runs

on

a

small-scale

parallel

machine

the

program

runs

faster

Lacking

any

other

information

we

may

conclude

that

the

improved

program

is

indeed

superior

to

the

original

But

by

measuring

work

and

critical

path

length

we

model

our

program

's

performance

and

predict

that

on

a

large-scale

machine

the

improved

program

is

actually

slower

than

the

original

Of

course

we

then

confirm

this

prediction

by

performing

a

test

on

the

large

machine

This

speedup

anomaly

occurs

because

the

improved

program

does

less

work

at

the

cost

of

a

longer

critical

path

Figure

illustrates

this

phenomenon

with

synthetic

numbers

In

this

example

the

improved

program

runs

percent

faster

than

the

original

-LRB-

seconds

versus

seconds

-RRB-

on

processors

But

on

processors

the

improved

program

is

actually

twice

as

slow

as

the

original

A

theoretical

analysis

of

the

Cilk

scheduler

original

program

improved

program

seconds

T

second

seconds

T

seconds

T

seconds

seconds

T

seconds

seconds

Figure

A

speedup

anomaly

We

use

the

simple

model

TP

=P

T

On

processors

the

improved

program

runs

percent

faster

than

the

original

taking

seconds

compared

to

the

original

's

seconds

But

it

does

so

with

less

work

at

the

cost

of

a

longer

critical

path

and

on

processors

the

improved

program

is

actually

twice

as

slow

as

the

original

Indeed

as

some

of

us

were

developing

and

tuning

heuristics

to

increase

the

performance

of

Socrates

we

used

work

and

critical

path

length

as

our

measures

of

progress

At

that

time

our

platform

for

competition

was

the

entire

in

dedicated

mode

at

the

National

Center

for

Supercomputing

Applications

-LRB-

NCSA

-RRB-

at

the

University

of

Illinois

at

Urbana-Champaign

Unable

to

use

this

machine

in

our

day-to-day

work

we

did

development

and

testing

on

a

time-shared

partition

of

a

that

we

have

in

house

More

than

once

by

using

work

and

critical

path

measurements

taken

from

our

runs

to

predict

the

runtime

of

our

program

on

the

machine

we

avoided

falling

into

the

trap

of

the

speedup

anomaly

just

described

A

theoretical

analysis

of

the

Cilk

scheduler

Cilk

's

work-stealing

scheduler

executes

any

fully

strict

Cilk

program

using

space

time

and

communication

all

within

a

constant

factor

of

optimal

In

previous

chapters

we

proved

analogous

results

using

a

model

of

multithreaded

computation

that

is

somewhat

simpler

than

Cilk

's

model

Rather

than

use

the

more

complex

Cilk

model

in

these

previous

proofs

we

chose

to

use

the

simpler

model

in

order

to

make

the

proofs

more

tractable

and

keep

from

obscuring

the

essential

ideas

In

this

section

we

show

how

these

proofs

must

be

modified

Some

of

the

research

reported

in

this

section

is

joint

work

with

Charles

Leiserson

and

Keith

Randall

both

of

MIT

's

Laboratory

for

Computer

Science

The

material

in

this

section

generalizes

results

previously

published

in

-LSB-

-RSB-

Chapter

Parallel

programming

in

Cilk

to

account

for

the

Cilk

model

Recall

that

a

Cilk

computation

models

the

execution

of

a

Cilk

program

as

a

tree

of

procedures

and

a

dag

of

threads

that

unfold

dynamically

during

program

execution

For

analysis

we

refine

the

dag

of

threads

into

a

dag

of

unit-size

instructions

as

follows

Each

thread

is

broken

into

instructions

connected

by

continue

edges

into

a

linear

sequence

from

the

first

instruction

of

the

thread

to

the

last

In

the

example

Cilk

computation

illustrated

in

Figure

thread

contains

instructions

and

If

a

thread

t

spawns

a

child

with

the

spawn

primitive

then

the

dag

has

a

spawn

edge

from

the

appropriate

instruction

of

t

to

the

first

instruction

of

Recall

that

the

child

thread

is

the

initial

thread

of

a

child

procedure

In

our

example

computation

thread

spawns

the

child

thread

-LRB-

the

initial

thread

of

procedure

-RRB-

as

represented

by

the

spawn

edge

-LRB-

-RRB-

If

t

spawns

a

successor

with

the

spawn

next

primitive

then

the

dag

has

a

spawn-next

edge

from

the

appropriate

instruction

of

t

to

the

first

instruction

of

Recall

that

the

successor

thread

is

in

the

same

procedure

as

t

In

our

example

computation

thread

spawns

the

successor

thread

as

represented

by

the

spawn-next

edge

-LRB-

-RRB-

Procedure

contains

threads

and

In

general

the

instructions

are

formed

into

threads

by

continue

edges

the

threads

are

formed

into

procedures

by

spawn-next

edges

and

the

procedures

are

formed

into

a

spawn

tree

by

the

spawn

edges

In

our

example

computation

the

root

procedure

has

children

and

and

procedure

has

children

and

Procedures

and

which

have

no

children

are

leaf

procedures

If

a

thread

t

sends

a

value

-LRB-

or

signal

-RRB-

to

another

thread

with

the

send

argument

or

accumulate

statement

then

the

dag

has

a

dependency

edge

from

the

appropriate

instruction

of

t

to

the

first

instruction

of

In

our

example

computation

thread

sends

a

value

-LRB-

or

signal

-RRB-

to

as

represented

by

the

dependency

edge

-LRB-

-RRB-

Notice

that

only

an

instruction

that

is

the

first

instruction

of

its

thread

can

have

an

incoming

edge

that

is

not

a

continue

edge

Consistent

with

our

unit-time

model

of

instructions

we

assume

that

the

out-degree

of

each

instruction

is

at

most

some

constant

The

work

is

the

number

of

instructions

in

the

dag

and

the

critical

path

length

T

is

the

length

of

a

longest

path

in

the

dag

In

our

example

computation

we

have

and

T

Notice

the

difference

between

the

execution

order

of

our

example

Cilk

computation

-LRB-

Figure

-RRB-

and

the

left-to-right

depth-first

execution

order

of

the

similar

multithreaded

computation

shown

in

Figure

-LRB-

page

-RRB-

In

order

to

discuss

the

execution

of

Cilk

computations

we

borrow

and

adapt

some

of

the

terminology

developed

in

Chapter

When

a

thread

is

spawned

we

allocate

a

closure

for

it

and

we

say

the

thread

is

alive

or

living

At

any

given

time

step

during

the

execution

an

instruction

is

ready

if

all

of

its

predecessors

in

the

dag

have

been

executed

and

a

living

thread

is

ready

if

its

first

instruction

is

ready

Ready

threads

can

be

executed

by

the

scheduler

When

the

last

instruction

of

a

thread

is

executed

the

thread

dies

and

we

free

its

closure

When

the

initial

thread

of

a

procedure

is

spawned

we

say

the

procedure

is

alive

or

living

and

when

the

procedure

no

longer

has

any

living

threads

then

the

procedure

dies

A

Cilk

computation

is

fully

strict

if

every

dependency

edge

goes

from

a

procedure

to

either

itself

or

its

parent

procedure

Our

example

computation

-LRB-

Figure

-RRB-

is

fully

strict

Specifically

the

fully

strict

condition

requires

that

for

every

dependency

edge

-LRB-

v

-RRB-

where

v

is

an

instruction

in

thread

t

in

procedure

G

and

is

an

instruction

in

thread

in

procedure

we

have

either

G?so

is

a

successor

of

t?or

is

the

parent

of

G?so

is

a

successor

of

the

thread

that

spawned

t

as

a

child

In

other

words

in

a

fully

strict

Cilk

computation

A

theoretical

analysis

of

the

Cilk

scheduler

Figure

A

Cilk

computation

This

computation

contains

instructions

represented

by

the

circles

and

threads

represented

by

the

dark-shaded

rectangles

and

procedures

represented

by

the

light-shaded

rounded

rectangles

The

continue

edges

are

horizontal

within

a

thread

the

spawn-next

edges

are

dark

shaded

within

a

procedure

the

spawn

edges

are

light

shaded

between

procedures

and

the

dependency

edges

are

curved

between

procedures

The

spawn

edges

emerging

from

a

thread

cross

each

other

so

that

the

execution

order

proceeds

from

left

to

right

This

figure

does

not

show

the

ghosts

required

for

the

analysis

threads

only

send

values

-LRB-

or

signals

-RRB-

to

their

successors

or

their

parent

thread

's

successors

For

any

fully

strict

Cilk

computation

with

work

and

critical

path

length

T

and

for

any

number

P

of

processors

we

shall

prove

the

following

bounds

on

execution

space

time

and

communication

ffl

Space

The

space

used

to

execute

the

computation

is

at

most

where

is

the

space

used

by

a

execution

of

the

Cilk

computation

ffl

Time

The

expected

time

to

execute

the

computation

is

O

-LRB-

=P

nlT

-RRB-

where

nl

is

the

maximum

number

of

threads

that

any

procedure

can

have

simultaneously

living

in

the

computation

ffl

Communication

The

expected

number

of

bytes

communicated

during

the

execution

is

O

-LRB-

PnlT

-LRB-

nd

Smax

-RRB-

-RRB-

where

nd

is

the

maximum

number

of

dependency

edges

between

any

pair

of

threads

and

Smax

is

the

size

of

the

largest

closure

in

the

computation

The

expected

bounds

on

both

time

and

communication

can

be

converted

to

high-probability

bounds

with

the

addition

of

some

small

extra

terms

The

reader

interested

in

the

proofs

of

these

bounds

should

be

familiar

with

the

proofs

in

Chapter

before

proceeding

The

cornerstone

in

proving

these

results

is

a

structural

lemma

-LRB-

analogous

to

Lemma

page

-RRB-

that

characterizes

the

procedures

in

the

ready

pool

of

any

processor

at

any

time

Chapter

Parallel

programming

in

Cilk

tt

G

Figure

For

each

thread

t

that

spawns

children

we

augment

the

computation

as

follows

Thread

t

spawns

a

successor

ghost

thread

t

as

the

last

thing

it

does

Each

time

t

spawns

a

child

procedure

G

it

also

spawns

a

ghost

child

G

The

children

are

ordered

from

youngest

to

oldest

going

from

left

to

right

The

child

procedures

are

linked

by

pool

edges

shown

curved

and

gray

In

Cilk

the

ready

pool

actually

contains

threads

not

procedures

but

each

thread

belongs

to

a

procedure

and

we

use

the

procedures

to

characterize

the

structure

of

the

ready

pool

In

order

to

state

and

prove

this

structural

lemma

we

first

need

two

technical

assumptions

and

some

new

terminology

Our

first

technical

assumption

illustrated

in

Figure

is

that

any

thread

t

that

spawns

children

also

as

the

last

thing

it

does

before

dying

spawns

a

ready

successor

thread

t

We

call

t

a

ghost

thread

A

ghost

thread

contains

only

a

single

ghost

instruction

that

takes

no

time

to

execute

so

it

is

purely

a

technical

convenience

Nevertheless

in

our

analysis

we

shall

assume

that

t

is

handled

by

the

scheduler

just

like

any

other

thread

This

ready

successor

thread

simplifies

the

structural

lemma

-LRB-

The

example

computation

of

Figure

does

not

show

the

required

ghost

threads

-RRB-

Two

procedures

and

are

tuplets

if

in

addition

to

having

the

same

parent

procedure

they

also

have

the

same

parent

thread

Since

the

instructions

in

a

thread

are

totally

ordered

the

tuplet

procedures

can

be

ordered

by

age

the

first

child

spawned

is

considered

to

be

older

than

the

second

and

so

on

In

our

example

Cilk

computation

-LRB-

Figure

-RRB-

the

procedures

and

are

tuplets

with

being

the

older

of

the

two

Our

second

technical

assumption

illustrated

in

Figure

is

that

whenever

a

thread

t

spawns

a

child

procedure

G

it

also

spawns

a

second

child

procedure

G

called

a

ghost

procedure

A

ghost

procedure

contains

only

a

ghost

thread

and

therefore

takes

no

time

to

execute

We

think

of

the

parent

thread

t

as

spawning

children

in

pairs

-LRB-

G

G

-RRB-

with

each

pair

having

one

ghost

procedure

The

scheduler

treats

G

-LRB-

or

rather

its

ghost

thread

-RRB-

as

having

been

spawned

just

before

G

and

therefore

the

ghost

is

considered

to

be

the

older

of

the

pair

Ghost

procedures

are

purely

a

technical

mechanism

to

facilitate

our

analysis

-LRB-

The

example

computation

of

Figure

does

not

show

the

required

ghost

procedures

-RRB-

At

any

given

time

step

during

the

execution

and

for

any

processor

p

we

define

the

list

of

procedures

at

level

L

as

follows

We

start

with

the

list

of

threads

at

level

L.

Let

tni

be

the

list

of

threads

in

the

level

L

list

of

p

's

ready

pool

ordered

from

A

theoretical

analysis

of

the

Cilk

scheduler

level

L

level

L

l

evel

evel

evel

Figure

The

structure

of

procedures

in

a

processor

's

ready

pool

The

processor

p

is

executing

a

thread

of

procedure

at

level

L

In

this

example

we

have

l

so

for

each

i

the

procedures

hGi

j

i

are

tuplet

children

of

Gi

The

procedures

and

G

must

be

ghost

procedures

at

the

head

of

the

list

to

tn

at

the

tail

If

processor

p

is

not

executing

a

thread

at

level

L

then

the

list

of

threads

at

level

L

is

tni

If

processor

p

is

executing

a

thread

t

at

level

L

then

the

list

of

threads

at

level

L

is

ht

tni

The

list

of

procedures

at

level

L

is

the

list

Gki

of

procedures

derived

from

the

list

of

threads

by

replacing

each

thread

with

its

procedure

and

collapsing

adjacent

equal

entries

into

one

entry

We

say

that

there

are

k

procedures

at

level

L

As

shorthand

we

shall

use

the

notation

hGii

to

denote

the

list

Gki

and

when

we

use

double

subscripts

we

shall

use

the

notation

hGi

j

i

to

denote

the

list

hGi

Gi

Gi

kii

for

a

given

value

of

i

We

now

state

and

prove

the

lemma

characterizing

the

structure

of

procedures

in

the

ready

pool

of

any

processor

during

the

execution

of

a

fully

strict

Cilk

computation

This

lemma

is

the

Cilk

analog

of

the

structural

lemma

-LRB-

Lemma

page

-RRB-

in

Chapter

which

applied

to

our

simpler

call-return

model

of

multithreaded

computation

Figure

illustrates

the

lemma

Lemma

During

the

execution

of

any

fully

strict

Cilk

computation

consider

any

processor

p

and

any

given

time

step

at

which

p

executes

an

instruction

of

a

thread

t.

Let

be

t

's

procedure

and

let

L

be

t

's

level

For

any

i

let

hGi

Gi

Gi

kii

denote

the

list

of

procedures

at

level

L

i

where

ki

is

the

number

of

procedures

at

level

L

i.

Let

l

be

the

largest

integer

such

that

kl

holds

Then

these

lists

of

procedures

satisfy

the

following

four

properties

For

i

l

we

have

ki

for

i

we

have

ki

and

for

i

we

have

ki

Chapter

Parallel

programming

in

Cilk

For

every

i

the

procedures

hGi

j

i

are

tuplets

ordered

from

youngest

to

oldest

For

i

l

the

procedures

hGi

j

i

are

children

of

Gi

and

the

procedures

hG

j

i

are

children

of

thread

t.

For

every

i

if

we

have

ki

then

for

j

ki

procedure

Gi

j

has

never

been

worked

on

and

if

we

have

k

then

G

also

has

never

been

worked

on

If

l

holds

then

for

i

l

we

have

ki

and

if

ki

then

Gi

ki

is

a

ghost

Proof

The

proof

is

a

straightforward

induction

on

execution

time

Execution

begins

with

the

initial

thread

of

the

root

procedure

in

some

processor

's

ready

pool

and

all

other

ready

pools

empty

so

the

lemma

vacuously

holds

at

the

outset

Now

consider

any

step

of

the

execution

at

which

processor

p

executes

an

instruction

from

thread

t

in

procedure

at

level

L.

Let

hGi

Gi

Gi

kii

denote

the

list

of

procedures

at

level

L

i

let

l

be

the

largest

integer

such

that

kl

holds

and

assume

that

all

four

properties

hold

Let

denote

the

thread

-LRB-

if

any

-RRB-

being

worked

on

by

p

after

the

step

let

be

its

procedure

and

its

level

let

i

denote

the

list

of

procedures

at

level

i

after

the

step

and

let

be

the

largest

integer

such

that

after

the

step

holds

The

proof

consists

of

looking

at

the

actions

of

the

Cilk

scheduler

and

showing

that

all

four

properties

hold

after

the

step

Before

looking

at

the

Cilk

scheduler

's

actions

we

first

review

the

things

that

a

thread

in

a

fully

strict

computation

may

do

that

cause

scheduler

action

Thread

t

may

spawn

a

child

with

Cilk

's

spawn

primitive

-LRB-

Recall

that

when

a

thread

spawns

a

child

we

also

spawn

a

ghost

so

spawns

occur

in

pairs

-RRB-

Thread

t

may

enable

a

successor

thread

by

either

performing

a

spawn

next

with

no

missing

arguments

or

by

performing

a

send

argument

or

accumulate

to

a

previously

spawned

successor

and

having

the

successor

's

join

counter

decrement

to

zero

Thread

t

may

enable

a

thread

in

its

parent

procedure

by

performing

a

send

argument

or

accumulate

as

just

described

For

fully

strict

computations

a

thread

t

may

only

enable

threads

in

its

own

procedure

or

its

parent

procedure

These

enabling

activities

all

cause

the

scheduler

to

post

a

ready

thread

to

the

ready

pool

Thread

t

dies

when

it

executes

its

last

instruction

and

this

event

causes

the

scheduler

to

start

another

iteration

of

its

loop

We

now

examine

these

scheduler

actions

individually

and

show

that

they

each

preserve

the

four

properties

of

the

lemma

If

the

thread

t

spawns

a

pair

of

children

then

p

posts

the

children

at

the

head

of

the

ready

pool

's

level

L

list

In

this

case

only

the

list

of

procedures

at

level

L

changes

Specifically

we

have

k

with

and

being

the

new

child

procedures

Procedure

is

the

ghost

and

is

considered

to

be

older

than

Also

if

holds

then

we

have

j

G

j

for

j

Now

we

can

check

that

the

four

properties

still

hold

The

first

property

does

not

apply

to

the

level

L

list

so

we

only

check

the

other

three

Property

The

procedures

j

i

must

be

tuplet

children

ordered

from

youngest

to

oldest

of

thread

t

in

procedure

because

before

the

spawn

the

procedures

hG

j

i

are

tuplet

children

of

t

ordered

from

youngest

to

oldest

Property

of

the

procedures

j

i

have

ever

been

worked

on

Property

We

have

because

t

just

spawned

children

Moreover

if

holds

then

is

a

ghost

because

before

the

spawn

we

have

l

which

means

that

if

k

holds

then

G

k

is

a

ghost

In

this

case

A

theoretical

analysis

of

the

Cilk

scheduler

G

k

is

a

ghost

Otherwise

we

have

k

so

is

the

newly

spawned

ghost

If

the

thread

t

enables

a

successor

thread

then

p

posts

the

successor

at

the

head

of

the

ready

pool

's

level

L

list

In

this

case

the

list

of

procedures

at

level

L

does

not

change

because

the

successor

thread

is

part

of

the

same

procedure

as

t

and

this

procedure

is

already

at

the

head

of

the

list

With

no

change

in

any

of

the

lists

of

procedures

all

properties

continue

to

hold

If

the

thread

t

enables

a

thread

in

its

parent

procedure

then

p

posts

this

newly

enabled

thread

at

the

head

of

the

ready

pool

's

level

L

list

If

we

have

l

then

the

parent

procedure

is

already

at

the

head

of

the

level

L

list

and

therefore

of

the

lists

of

procedures

change

On

the

other

hand

if

we

have

l

then

we

update

l

with

and

the

list

of

procedures

at

level

L

is

the

list

containing

only

the

parent

procedure

of

In

this

case

the

properties

are

easily

checked

Property

We

have

Property

The

procedures

ji

are

children

of

because

is

a

child

of

and

the

procedures

j

i

ji

are

tuplets

Property

This

property

does

not

apply

to

the

level

L

list

because

we

have

Property

Again

this

property

does

not

apply

because

we

have

If

the

thread

t

dies

then

p

starts

another

iteration

of

the

scheduling

loop

If

the

ready

pool

is

empty

then

it

commences

work

stealing

and

the

properties

hold

vacuously

Otherwise

it

removes

and

executes

the

thread

at

the

bottommost

nonempty

level

of

the

ready

pool

We

consider

two

cases

depending

on

whether

the

level

L

list

is

nonempty

If

the

level

L

list

is

nonempty

-LRB-

k

-RRB-

then

p

removes

and

executes

the

thread

at

the

head

of

the

ready

pool

's

level

L

list

Recall

that

because

thread

t

of

procedure

spawned

children

it

also

spawns

a

ready

successor

thread

t

as

the

last

thing

it

does

before

dying

Therefore

the

ready

pool

's

list

at

level

L

must

have

a

thread

of

at

its

head

when

t

dies

We

have

L

and

l

We

also

have

and

for

i

we

have

ki

with

j

Gi

j

for

j

We

now

check

that

the

properties

still

hold

Properties

and

We

have

n't

actually

changed

any

of

the

lists

of

procedures

only

renamed

them

Property

No

procedure

j

with

j

has

ever

been

worked

on

Also

we

have

Property

This

property

only

applies

to

the

level

list

if

we

have

If

holds

then

we

have

and

is

a

ghost

because

before

this

step

l

holds

which

means

that

we

have

k

and

G

k

is

a

ghost

Now

suppose

that

the

level

L

list

is

empty

when

thread

t

dies

We

further

break

this

situation

down

into

two

cases

depending

on

whether

the

ready

pool

's

level

L

list

is

nonempty

If

the

level

L

list

is

nonempty

then

p

removes

and

executes

the

thread

at

the

head

of

this

list

If

this

thread

is

in

procedure

then

no

list

of

procedures

changes

and

the

properties

continue

to

hold

Otherwise

we

must

have

and

we

now

have

and

j

j

for

j

If

is

a

ghost

then

we

execute

it

immediately

and

advance

to

the

next

iteration

of

the

scheduling

loop

Therefore

we

only

need

to

check

that

the

properties

still

hold

in

the

case

that

is

not

a

ghost

The

first

three

properties

are

easily

checked

and

they

continue

to

hold

Property

This

property

only

applies

to

the

level

L

list

if

we

have

l

If

holds

then

we

have

and

is

a

ghost

because

before

this

step

we

have

l

and

is

a

ghost

which

means

that

we

must

have

or

else

would

be

a

ghost

Chapter

Parallel

programming

in

Cilk

If

the

level

L

list

is

empty

but

the

ready

pool

as

a

whole

is

nonempty

then

the

ready

pool

's

level

L

list

must

be

nonempty

because

otherwise

we

would

violate

Property

In

this

case

p

removes

and

executes

the

thread

at

the

head

of

the

level

L

list

We

now

have

L

and

l

We

also

have

and

for

i

we

have

ki

with

j

Gi

j

for

j

In

this

case

all

four

properties

are

easily

checked

and

they

continue

to

hold

Finally

if

some

other

processor

steals

a

thread

from

processor

p

then

it

removes

the

thread

from

the

tail

of

the

topmost

nonempty

list

of

p

's

ready

pool

Suppose

that

we

have

l

In

this

case

the

stolen

thread

is

in

procedure

Gl

kl

and

the

list

of

procedures

at

level

L

l

may

be

shortened

by

one

If

so

then

we

have

kl

and

if

we

also

have

then

we

update

l

In

these

cases

the

properties

are

easily

checked

and

they

continue

to

hold

Now

suppose

that

we

have

l

If

also

holds

then

the

stolen

thread

is

in

procedure

and

the

properties

continue

to

hold

as

before

On

the

other

hand

if

we

have

then

the

level

L

list

of

procedures

contains

only

and

there

are

two

possibilities

to

consider

If

the

ready

pool

's

level

L

list

is

nonempty

then

the

stolen

thread

is

a

thread

of

procedure

In

this

case

the

list

of

procedures

at

level

L

does

not

change

and

the

properties

continue

to

hold

On

the

other

hand

if

the

ready

pool

's

level

L

list

is

empty

then

the

stolen

thread

will

be

a

thread

from

procedure

G

k

In

this

case

the

list

of

procedures

at

level

L

shortens

but

again

the

properties

continue

to

hold

All

other

activity

by

processor

p?such

as

work

stealing

or

executing

an

instruction

that

does

not

invoke

any

of

the

above

actions?clearly

preserve

the

lemma

Our

bound

on

space

accounts

for

any

stack-like

memory

Specifically

we

allow

any

instruction

to

allocate

memory

for

its

procedure

provided

that

the

instruction

is

totally

ordered

with

respect

to

every

other

instruction

in

its

procedure

and

provided

that

the

instruction

can

not

be

executed

at

a

time

step

when

its

procedure

has

a

living

child

We

allow

any

instruction

to

deallocate

memory

for

its

procedure

provided

that

the

instruction

is

totally

ordered

with

respect

to

every

other

instruction

in

its

procedure

Additionally

we

require

that

all

memory

allocated

for

a

procedure

is

deallocated

before

the

procedure

dies

At

any

given

time

step

during

the

execution

the

amount

of

memory

currently

allocated

for

a

given

living

procedure

is

the

net

memory

allocated

for

the

procedure

by

all

instructions

that

have

been

executed

The

space

bound

follows

from

the

busy-leaves

property

which

characterizes

the

living

procedures

at

all

time

steps

during

the

execution

At

any

given

time

step

during

the

execution

we

say

that

a

procedure

is

a

leaf

if

it

has

no

living

children

and

we

say

that

a

leaf

procedure

is

a

primary

leaf

if

in

addition

either

it

has

no

younger

tuplets

living

or

it

has

been

worked

on

by

some

processor

The

busy-leaves

property

states

that

every

primary

leaf

procedure

has

a

processor

either

working

on

it

or

working

on

its

parent

thread

To

prove

the

space

bound

we

show

that

Cilk

's

scheduler

maintains

the

busy-leaves

property

and

then

we

show

that

the

busy-leaves

property

implies

the

space

bound

Theorem

For

any

fully

strict

Cilk

computation

if

is

the

space

used

to

execute

the

computation

on

processor

then

with

any

number

P

of

processors

Cilk

's

work-stealing

scheduler

uses

at

most

space

Proof

We

first

show

that

Cilk

's

work-stealing

scheduler

maintains

the

busy-leaves

property

and

then

we

show

that

the

busy-leaves

property

implies

the

space

bound

A

theoretical

analysis

of

the

Cilk

scheduler

To

see

that

Cilk

's

scheduler

maintains

the

busy-leaves

property

consider

any

time

step

during

the

execution

and

any

primary

leaf

procedure

G

Since

G

is

a

leaf

and

the

computation

is

strict

there

must

be

some

thread

in

G

that

is

ready

Therefore

G

is

in

the

list

of

procedures

at

some

level

of

some

processor

p

's

ready

pool

Suppose

processor

p

is

executing

a

thread

t

at

level

L

and

let

hGi

j

i

denote

the

list

of

procedures

at

level

L

i.

From

Lemma

we

know

that

among

all

of

these

procedures

the

only

one

that

can

be

a

primary

leaf

is

either

-LRB-

if

we

have

k

-RRB-

or

G

-LRB-

otherwise

-RRB-

In

the

former

case

we

have

G

and

p

is

executing

a

thread

of

G

In

the

latter

case

we

have

G

G

and

p

is

executing

the

thread

that

spawned

G

In

either

case

procedure

G

is

busy

The

P-processor

space

bound

is

obtained

by

showing

that

at

all

time

steps

t

during

the

execution

every

living

procedure

can

be

associated

with

a

primary

leaf

procedure

and

that

the

total

space

currently

allocated

for

all

procedures

assigned

to

a

given

primary

leaf

is

at

most

We

assign

a

procedure

to

a

primary

leaf

as

follows

If

the

procedure

is

a

primary

leaf

then

we

assign

it

to

itself

If

the

procedure

is

a

leaf

but

it

is

not

a

primary

leaf

then

we

assign

it

to

the

same

primary

leaf

as

its

youngest

tuplet

If

the

procedure

is

not

a

leaf

then

we

assign

it

to

the

same

primary

leaf

as

any

one

of

its

living

children

The

procedures

assigned

to

any

given

primary

leaf

G

are

G

's

ancestors

and

older

tuplets

of

G

's

ancestors

Now

consider

any

primary

leaf

procedure

G

Since

G

is

busy

there

must

be

a

processor

p

that

is

executing

either

a

thread

of

G

or

the

thread

that

spawned

G.

Let

t

denote

the

thread

that

p

is

executing

and

let

v

denote

the

instruction

that

p

executes

at

this

time

step

For

any

other

procedure

assigned

to

G

we

claim

that

the

amount

of

memory

currently

allocated

for

in

our

P-processor

execution

is

no

more

than

the

amount

of

memory

allocated

for

at

the

time

step

in

the

execution

when

the

processor

executes

instruction

v.

To

verify

this

claim

consider

the

two

possible

relationships

that

may

have

with

G.

Suppose

is

an

ancestor

of

G.

Consider

the

set

of

allocating

or

deallocating

instructions

in

that

have

been

executed

in

our

P-processor

execution

This

set

must

be

a

superset

of

the

instructions

in

that

have

been

executed

in

the

case

because

the

allocating

and

deallocating

instructions

are

totally

ordered

with

respect

to

every

other

instruction

in

the

procedure

Moreover

those

allocating

or

deallocating

instructions

-LRB-

if

any

-RRB-

that

have

been

executed

in

our

P-processor

case

that

have

n't

been

executed

in

the

case

must

in

fact

be

deallocating

instructions

since

they

must

have

been

executed

while

had

a

living

child

Thus

the

memory

allocated

for

in

our

P-processor

case

is

no

more

than

the

memory

allocated

for

in

the

case

Now

suppose

is

the

older

tuplet

of

one

of

G

's

ancestors

In

this

case

since

is

a

leaf

we

know

that

has

never

been

worked

on

Likewise

in

the

case

must

be

alive

but

never

worked

on

Finally

consider

G

itself

The

set

of

allocating

or

deallocating

instructions

in

G

that

have

been

executed

in

our

P-processor

execution

must

be

the

same

as

those

executed

in

the

case

Thus

for

every

procedure

assigned

to

G

the

amount

of

memory

currently

allocated

for

it

in

our

P-processor

execution

is

no

more

than

the

amount

of

memory

allocated

for

it

in

the

case

Thus

the

total

space

assigned

to

G

is

at

most

Since

Cilk

's

scheduler

keeps

all

primary

leaves

busy

with

P

processors

we

are

guaranteed

that

at

every

time

step

during

the

execution

at

most

P

primary-leaf

procedures

can

be

living

Every

living

procedure

is

assigned

to

one

of

these

P

primary

leaves

and

the

total

space

of

the

procedures

assigned

to

a

given

primary

leaf

is

at

most

Therefore

the

total

Chapter

Parallel

programming

in

Cilk

space

of

all

living

procedures

is

at

most

In

bounding

execution

time

we

assume

that

the

machine

is

an

asynchronous

parallel

computer

with

P

processors

and

its

memory

can

be

either

distributed

or

shared

We

further

assume

that

concurrent

accesses

to

the

same

data

structure

are

serially

queued

by

an

adversary

as

in

the

atomic-access

model

of

Section

Specifically

if

a

processor

attempts

to

steal

a

closure

from

a

victim

processor

and

no

other

thief

is

attempting

to

steal

from

the

same

victim

then

the

steal

attempt?successful

or

not?takes

one

unit

of

time

If

multiple

thieves

simultaneously

attempt

to

steal

from

the

same

victim

then

their

requests

are

handled

one

per

time

step

in

an

order

determined

by

an

adversary

In

our

analysis

of

execution

time

we

follow

the

same

accounting

argument

as

in

Section

At

each

time

step

we

collect

P

dollars

one

per

processor

At

each

time

step

each

processor

places

its

dollar

in

one

of

three

buckets

according

to

its

actions

at

that

step

If

the

processor

executes

an

instruction

at

the

step

then

it

places

its

dollar

into

the

WORK

bucket

If

the

processor

initiates

a

steal

attempt

at

the

step

then

it

places

its

dollar

into

the

STEAL

bucket

And

if

the

processor

merely

waits

for

a

queued

steal

request

at

the

step

then

it

places

its

dollar

into

the

WAIT

bucket

We

shall

derive

the

running

time

bound

by

bounding

the

number

of

dollars

in

each

bucket

at

the

end

of

the

execution

summing

these

three

bounds

and

then

dividing

by

P

The

bounds

on

the

dollars

in

the

WORK

and

WAIT

buckets

are

exactly

as

in

Section

Execution

ends

with

dollars

in

the

WORK

bucket

since

there

are

instructions

in

the

computation

-LRB-

Lemma

page

-RRB-

Lemma

-LRB-

page

-RRB-

bounds

the

number

of

dollars

in

the

WAIT

bucket

as

a

function

of

the

number

of

dollars

in

the

STEAL

bucket

With

high

probability

the

number

of

dollars

in

the

WAIT

bucket

is

at

most

a

constant

times

the

number

of

dollars

in

the

STEAL

bucket

To

bound

the

dollars

in

the

STEAL

bucket

we

use

a

delay-sequence

argument

very

similar

the

one

we

used

in

Section

but

we

must

modify

this

argument

slightly

to

account

for

the

Cilk

model

of

multithreaded

computation

As

in

Section

the

work-steal

attempts

are

partitioned

into

rounds

of

at

least

but

fewer

than

consecutive

steal

attempts

Also

as

in

Section

the

delay

sequence

is

defined

in

terms

of

an

augmented

dag

of

instructions

obtained

from

the

original

dag

G

by

adding

some

new

edges

For

a

Cilk

computation

these

new

edges

are

called

pool

edges

and

for

each

thread

t

that

spawns

children

we

add

pool

edges

as

follows

Let

Gn

denote

the

spawned

child

procedures

and

recall

that

we

must

have

n

since

we

require

the

spawns

to

occur

in

pairs

with

ghost

procedures

Then

as

illustrated

in

Figure

for

each

i

n

we

add

a

pool

edge

from

the

first

instruction

of

Gi

to

the

first

instruction

of

Gi

If

T

is

the

length

of

a

longest

path

in

G

then

the

longest

path

in

the

augmented

dag

has

length

at

most

At

any

given

time

step

during

the

execution

an

instruction

is

critical

if

all

of

its

predecessors

in

have

been

executed

If

during

the

course

of

the

execution

a

large

number

of

steals

are

attempted

then

we

can

identify

a

sequence

of

instructions?the

delay

sequence?in

this

augmented

dag

such

that

a

large

number

of

rounds

of

steal

attempts

were

initiated

while

an

instruction

from

the

sequence

was

critical

We

show

that

such

delay

sequences

are

unlikely

to

occur

because

a

critical

instruction

is

unlikely

to

remain

critical

across

a

modest

number

of

steal-attempt

rounds

A

theoretical

analysis

of

the

Cilk

scheduler

The

delay

sequence

is

defined

exactly

as

in

Definition

-LRB-

page

-RRB-

We

repeat

the

definition

here

Definition

A

delay

sequence

is

a

-LRB-

U

R

;P

-RRB-

satisfying

the

following

conditions

ffl

U

-LRB-

uL

-RRB-

is

a

maximal

directed

path

in

Specifically

for

i

L

the

edge

-LRB-

ui

ui

-RRB-

belongs

to

instruction

has

no

incoming

edges

in

-LRB-

instruction

must

be

the

first

instruction

of

the

initial

thread

of

the

root

procedure

-RRB-

and

instruction

uL

has

no

outgoing

edges

in

ffl

R

is

a

positive

integer

ffl

P

-LRB-

pL

-RRB-

is

a

partition

of

the

integer

R

The

delay

sequence

-LRB-

U

R

;P

-RRB-

is

said

to

occur

during

an

execution

if

for

each

i

L

at

least

pi

steal-attempt

rounds

occur

while

instruction

ui

is

critical

The

following

lemma

states

that

if

a

large

number

of

steal

attempts

take

place

during

an

execution

then

a

delay

sequence

with

large

R

must

occur

This

lemma

and

its

proof

are

identical

to

Lemma

-LRB-

page

-RRB-

Lemma

Consider

the

execution

of

a

fully

strict

Cilk

computation

with

critical

path

length

T

on

a

computer

with

P

processors

If

at

least

-LRB-

R

-RRB-

steal

attempts

occur

during

the

execution

then

some

-LRB-

U

R

;P

-RRB-

delay

sequence

must

occur

We

now

establish

that

a

critical

instruction

is

unlikely

to

remain

critical

across

a

modest

number

of

steal-attempt

rounds

Specifically

we

first

show

that

if

no

procedure

can

ever

have

more

than

nl

simultaneously

living

threads

then

after

O

-LRB-

nl

-RRB-

steal-attempt

rounds

a

critical

instruction

is

very

likely

to

be

executed

The

following

lemma

establishes

facts

analogous

to

those

established

in

Lemma

-LRB-

page

-RRB-

and

Lemma

-LRB-

page

-RRB-

Lemma

Consider

the

execution

of

any

fully

strict

Cilk

computation

on

a

parallel

computer

with

P

processors

If

no

procedure

of

the

computation

can

ever

have

more

than

nl

simultaneously

living

threads

then

for

any

instruction

v

and

any

number

r

of

stealattempt

rounds

the

probability

that

r

rounds

occur

while

v

is

critical

is

at

most

e?r

Proof

We

first

use

our

structural

lemma

Lemma

to

show

that

if

instruction

v

is

critical

at

some

time

step

then

it

must

be

the

ready

instruction

of

a

thread

that

either

is

currently

being

executed

or

is

at

the

tail

of

a

list

near

the

top

of

some

processor

's

ready

pool

We

then

use

this

fact

to

establish

the

probabilistic

bound

Consider

any

time

step

at

which

instruction

v

is

critical

If

some

processor

is

executing

v

's

thread

then

v

will

be

executed

at

this

time

step

and

the

lemma

holds

Therefore

suppose

that

v

is

the

first

instruction

of

a

thread

t

in

procedure

G

and

t

is

in

the

ready

pool

of

a

processor

p

Since

v

is

critical

every

one

of

G

's

older

tuplets

must

have

been

worked

on

at

some

earlier

time

step

so

by

Property

of

Lemma

procedure

G

must

be

at

the

tail

of

the

list

of

procedures

at

some

level

By

the

same

reasoning

every

nonempty

list

of

procedures

above

this

level

must

contain

only

one

procedure

Then

Property

of

Lemma

ensures

Chapter

Parallel

programming

in

Cilk

that

this

level

is

at

most

from

the

top

Thus

since

each

procedure

can

have

at

most

nl

simultaneously

living

threads

once

processor

p

has

satisfied

work-steal

requests

we

are

guaranteed

that

v

has

been

executed

Mimicking

the

proof

of

Lemma

we

observe

that

if

instruction

v

remains

critical

across

r

steal-attempt

rounds

then

of

the

at

least

-LRB-

-LRB-

-RRB-

-RRB-

P

steal

attempts

initiated

at

least

time

steps

before

v

is

executed

fewer

than

of

them

choose

a

particular

processor

p

as

the

victim

Letting

the

random

variable

X

denote

the

number

of

these

steal

attempts

that

do

choose

processor

p

we

bound

the

probability

that

X

is

less

than

by

using

a

Chernoff

bound

-LSB-

-RSB-

on

the

lower

tail

of

a

binomial

distribution

with

mean

m

PrfX

m?ag

for

any

a

In

our

case

we

have

m

and

to

bound

PrfX

we

have

a

m

Thus

the

probability

that

v

remains

critical

across

r

steal-attempt

rounds

is

at

most

PrfX

exp

h

-LRB-

-RRB-

-LRB-

-RRB-

i

e?r

for

r

We

now

complete

the

delay-sequence

argument

and

bound

the

total

dollars

in

the

STEAL

bucket

The

proof

of

the

following

lemma

is

nearly

identical

to

the

proof

of

Lemma

-LRB-

page

-RRB-

The

only

change

is

that

we

use

the

probabilistic

bound

of

Lemma

instead

of

the

bound

given

by

Lemma

Lemma

Consider

the

execution

of

any

fully

strict

Cilk

computation

with

critical

path

length

T

on

a

parallel

computer

with

P

processors

If

no

procedure

ever

has

more

than

nl

simultaneously

living

threads

then

for

any

e

with

probability

at

least

e

the

execution

terminates

with

at

most

O

-LRB-

P

-LRB-

nlT

lg

-LRB-

e

-RRB-

-RRB-

-RRB-

dollars

in

the

STEAL

bucket

and

the

expected

number

of

dollars

in

this

bucket

is

O

-LRB-

PnlT

-RRB-

With

bounds

on

all

three

buckets

we

now

state

and

prove

the

theorem

that

bounds

the

total

execution

time

for

a

fully

strict

Cilk

computation

Theorem

Consider

any

fully

strict

Cilk

computation

with

work

and

critical

path

length

T

such

that

no

procedure

can

ever

have

more

than

nl

simultaneously

living

threads

With

any

number

P

of

processors

Cilk

's

work-stealing

scheduler

runs

the

computation

in

expected

time

O

-LRB-

=P

nlT

-RRB-

Moreover

for

any

e

with

probability

at

least

e

the

execution

time

on

P

processors

is

O

-LRB-

=P

nlT

lgP

lg

-LRB-

e

-RRB-

-RRB-

Proof

Add

up

the

dollars

in

the

three

buckets

and

divide

by

P

The

next

theorem

bounds

the

total

amount

of

communication

that

a

Cilk

computation

performs

in

a

distributed

model

The

analysis

assumes

that

at

most

a

constant

number

of

bytes

need

to

be

communicated

to

perform

a

send

argument

or

accumulate

in

the

case

when

the

join

counter

does

not

go

to

zero

In

the

case

when

the

join

counter

does

go

to

zero

then

an

entire

closure

may

need

to

be

communicated

A

theoretical

analysis

of

the

Cilk

scheduler

Theorem

Consider

any

fully

strict

Cilk

computation

with

work

and

critical

path

length

T

such

that

no

procedure

ever

has

more

than

nl

simultaneously

living

threads

With

any

number

P

of

processors

the

total

number

of

bytes

communicated

by

Cilk

's

work-stealing

scheduler

has

expectation

O

-LRB-

PnlT

-LRB-

nd

Smax

-RRB-

-RRB-

where

nd

is

the

maximum

number

of

dependency

edges

between

any

pair

of

threads

and

Smax

is

the

size

in

bytes

of

the

largest

closure

in

the

computation

Moreover

for

any

e

the

probability

is

at

least

e

that

the

total

communication

incurred

is

O

-LRB-

P

-LRB-

nlT

lg

-LRB-

e

-RRB-

-RRB-

-LRB-

nd

Smax

-RRB-

-RRB-

Proof

We

prove

the

bound

for

the

expectation

The

high-probability

bound

is

analogous

By

our

bucketing

argument

the

expected

number

of

steal

attempts

is

at

most

O

-LRB-

PnlT

-RRB-

When

a

thread

is

stolen

the

communication

incurred

is

at

most

Smax

We

also

have

communication

when

a

processor

executing

a

thread

performs

a

send

argumentor

accumulate

to

a

thread

whose

closure

is

on

another

processor

If

the

join

counter

does

not

go

to

zero

then

the

amount

of

communication

is

O

-LRB-

-RRB-

and

this

event

can

occur

at

most

nd

times

per

steal

If

the

join

counter

does

go

to

zero

then

the

amount

of

communication

is

at

most

Smax

and

this

event

can

occur

at

most

once

per

steal

Thus

the

expected

total

communication

cost

is

O

-LRB-

PnlT

-LRB-

nd

Smax

-RRB-

-RRB-

Chapter

Cilk

on

a

network

of

workstations

In

order

to

execute

Cilk

programs

efficiently

on

a

network

of

workstations

the

Cilk-NOW

runtime

system

implements

adaptive

parallelism

and

transparent

fault

tolerance

Adaptive

parallelism

allows

a

Cilk

application

to

run

on

a

set

of

workstations

that

may

grow

and

shrink

dynamically

during

program

execution

When

a

given

workstation

is

not

being

used

by

its

owner

the

workstation

automatically

joins

in

and

helps

out

with

the

execution

of

a

Cilk

program

When

the

owner

returns

to

work

the

machine

automatically

retreats

from

the

Cilk

program

Fault

tolerance

allows

a

Cilk

application

to

continue

execution

even

in

the

face

of

individual

workstation

crashes

or

reboots

Applications

may

take

advantage

of

this

feature

despite

being

fault

oblivious

The

application

is

written

as

an

ordinary

Cilk

program

with

no

special

provision

for

handling

faults

Recently

we

ran

the

Cilk

protein-folding

application

pfold

-LRB-

see

page

-RRB-

using

Cilk-NOW

on

a

network

of

about

Sun

SparcStations

connected

by

Ethernet

to

solve

a

large-scale

protein-folding

problem

The

program

ran

for

days

surviving

several

machine

crashes

and

reboots

utilizing

processor-hours

of

otherwise-idle

cycles

with

no

administrative

effort

on

our

part

-LRB-

besides

typing

pfold

at

the

command-line

to

begin

execution

-RRB-

while

other

users

of

the

network

went

about

their

business

unaware

of

the

program

's

presence

In

this

chapter

we

show

how

the

Cilk-NOW

runtime

system

leverages

the

structure

in

Cilk

's

programming

model

to

implement

adaptive

parallelism

and

fault

tolerance

In

Section

we

present

the

architecture

of

Cilk-NOW

Then

in

Sections

and

we

present

the

implementation

of

adaptive

parallelism

and

fault

tolerance

System

architecture

The

Cilk-NOW

runtime

system

consists

of

several

component

programs

that

work

together

recruiting

idle

machines

in

the

network

to

work

on

the

execution

of

Cilk

programs

scheduling

these

idle

machines

among

all

the

Cilk

programs

that

are

running

and

managing

the

execution

of

each

individual

Cilk

program

In

this

section

we

shall

cover

the

architecture

of

Some

of

the

research

reported

in

this

chapter

is

joint

work

with

David

Park

formerly

of

MIT

's

Laboratory

for

Computer

Science

and

now

of

McKinsey

Company

and

Phil

Lisiecki

of

MIT

's

Laboratory

for

Computer

Science

Some

of

the

material

in

this

chapter

was

previously

published

in

-LSB-

-RSB-

At

that

time

Cilk-NOW

was

in

the

prototype

stage

and

was

called

Phish

Chapter

Cilk

on

a

network

of

workstations

the

Cilk-NOW

runtime

system

explaining

the

operation

of

each

component

program

and

their

interactions

In

Cilk-NOW

terminology

we

refer

to

an

executing

Cilk

program

as

a

Cilk

job

Since

Cilk

programs

are

parallel

programs

a

Cilk

job

consists

of

several

processes

running

on

several

machines

One

process

called

the

clearinghouse

in

each

Cilk

job

runs

a

program

called

CilkChouse

CilkChouse

is

a

system-supplied

program

that

is

responsible

for

keeping

track

of

all

the

other

processes

that

comprise

a

given

job

These

other

processes

are

called

workers

A

worker

is

a

process

running

the

actual

executable

of

a

Cilk

program

such

as

ray

or

pfold

Since

Cilk

jobs

are

adaptively

parallel

the

set

of

workers

is

dynamic

At

any

given

time

during

the

execution

of

a

job

a

new

worker

may

join

the

job

or

an

existing

worker

may

leave

Thus

each

Cilk

job

consists

of

one

or

more

workers

and

a

clearinghouse

to

keep

track

of

them

The

Cilk-NOW

runtime

system

contains

two

additional

components

to

keep

track

of

the

Cilk

jobs

and

the

individual

workstations

in

the

network

The

job

broker

is

a

processes

running

a

system-supplied

program

called

CilkJobBroker

The

job

broker

runs

on

a

single

machine

in

the

network

and

keeps

track

of

the

set

of

Cilk

jobs

running

in

the

network

A

node

manager

is

a

process

running

a

system-supplied

program

called

CilkNodeManager

A

node

manager

runs

as

a

background

daemon

on

every

machine

in

the

network

It

continually

monitors

its

machine

to

determine

when

the

machine

is

idle

and

when

it

is

busy

When

the

node

manager

finds

that

its

machine

is

idle

it

contacts

the

job

broker

to

find

a

running

Cilk

job

that

the

machine

can

work

on

To

see

how

all

of

these

components

work

together

in

managing

the

execution

of

Cilk

jobs

we

shall

run

though

an

example

Suppose

that

the

job

broker

is

running

on

a

machine

called

Vulture

and

a

user

sits

down

at

a

machine

called

Penguin

to

run

the

pfold

program

In

our

example

the

user

types

pfold

at

the

shell

thereby

launching

a

Cilk

job

to

enumerate

all

protein

foldings

using

initial

folding

sequences

and

starting

with

the

one

The

new

Cilk

job

begins

execution

as

illustrated

in

Figure

The

new

process

running

the

pfold

executable

is

the

first

worker

and

begins

execution

by

forking

a

clearinghouse

with

the

command

line

CilkChouse

pfold

Thus

the

clearinghouse

knows

that

it

is

in

charge

of

a

job

whose

workers

are

running

pfold

The

clearinghouse

begins

execution

by

sending

a

job

description

to

the

job

broker

The

job

description

is

a

record

containing

several

fields

Among

these

fields

is

the

name

of

the

Cilk

program

executable?in

this

case

pfold?and

the

clearinghouse

's

network

address

The

clearinghouse

also

sends

its

network

address

through

a

pipe

back

to

the

first

worker

the

worker

that

forked

the

clearinghouse

The

clearinghouse

then

goes

into

a

service

loop

waiting

for

messages

from

its

workers

After

receiving

the

clearinghouse

's

address

from

the

pipe

the

first

worker

registers

with

the

clearinghouse

by

sending

the

clearinghouse

a

message

containing

its

own

network

address

Now

the

clearinghouse

knows

about

one

worker

and

it

responds

to

that

worker

by

assigning

it

a

unique

name

Workers

are

named

System

architecture

WorkerPenguinJob

brokerV

ultureC

learinghouse

First

worker

forks

a

clearinghouse

and

then

registers

Clearinghouse

submits

job

to

job

broker

Figure

A

Cilk

job

starts

The

first

worker

forks

a

clearinghouse

and

then

the

clearinghouse

submits

the

job

to

the

job

broker

with

numbers

starting

with

number

The

first

worker

to

register

is

named

with

number

the

second

worker

to

register

is

named

with

number

and

so

on

Having

registered

worker

begins

executing

the

Cilk

program

as

described

in

the

previous

chapter

We

now

have

a

running

Cilk

job

with

one

worker

A

second

worker

joins

the

Cilk

job

when

some

other

workstation

in

the

network

becomes

idle

as

illustrated

in

Figure

Suppose

the

node

manager

on

a

machine

named

Sparrow

detects

that

the

machine

is

idle

The

node

manager

sends

a

message

to

the

job

broker

informing

the

job

broker

of

the

idle

machine

The

job

broker

responds

with

the

job

description

of

a

Cilk

job

for

the

machine

to

work

on

In

this

case

the

job

description

specifies

our

pfold

job

by

giving

the

name

of

the

executable?pfold?and

the

network

address

of

the

clearinghouse

The

node

manager

then

uses

this

information

to

fork

a

new

worker

as

a

child

with

the

command

line

pfold

NoChouse

Address

clearinghouse-address

The

NoChouseflag

on

the

command

line

tells

the

worker

that

it

is

to

be

an

additional

worker

in

an

already

existing

Cilk

job

-LRB-

Without

this

flag

the

worker

would

fork

a

new

clearinghouse

and

start

a

new

Cilk

job

-RRB-

The

Address

field

on

the

command

line

tells

the

worker

where

in

the

network

to

find

the

clearinghouse

The

worker

uses

this

address

to

send

a

registration

message

containing

its

own

network

address

to

the

clearinghouse

The

clearinghouse

responds

with

the

worker

's

assigned

name?in

this

case

number

and

the

job

's

command-line

arguments?in

this

case

pfold

Additionally

the

clearinghouse

responds

with

a

list

of

the

network

addresses

of

all

other

registered

workers

Now

the

new

Chapter

Cilk

on

a

network

of

workstations

WorkerPenguinJob

brokerV

ultureC

learinghouseWorkerSparrowNode

Node

manager

obtains

job

from

job

broker

Worker

registers

with

clearinghouse

Worker

steals

work

from

another

Node

manager

detects

idle

Node

manager

forks

worker

Figure

An

idle

machine

joins

a

Cilk

job

When

the

node

manager

detects

that

its

machine

is

idle

it

obtains

a

job

from

the

job

broker

and

then

forks

a

worker

The

worker

registers

with

the

clearinghouse

and

then

begins

work

stealing

worker

knows

the

addresses

of

the

other

workers

so

it

can

commence

execution

of

the

Cilk

program

and

steal

work

as

described

in

the

previous

chapter

We

now

have

a

running

Cilk

job

with

two

workers

Now

suppose

that

someone

touches

the

keyboard

on

Sparrow

In

this

case

the

node

manager

detects

that

the

machine

is

busy

and

the

machine

leaves

the

Cilk

job

as

illustrated

in

Figure

After

detecting

that

the

machine

is

busy

the

node

manager

sends

a

kill

signal

to

its

child

worker

The

worker

catches

this

signal

and

prepares

to

leave

the

job

First

the

worker

offloads

all

of

its

closures

to

other

workers

as

explained

in

more

detail

in

the

next

section

Next

the

worker

sends

a

message

to

the

clearinghouse

to

unregister

Finally

the

worker

terminates

When

the

node

manager

detects

that

its

child

worker

has

terminated

it

notifies

the

job

broker

so

that

the

job

broker

can

keep

track

of

the

number

of

workers

in

each

Cilk

job

When

a

Cilk

job

is

running

each

worker

checks

in

with

the

clearinghouse

once

every

seconds

Specifically

each

worker

every

seconds

sends

a

message

to

the

clearinghouse

The

clearinghouse

responds

with

an

update

message

informing

the

worker

of

any

other

workers

that

have

left

the

job

and

any

new

workers

that

have

joined

the

job

For

each

new

worker

that

has

joined

the

clearinghouse

also

provides

the

network

address

If

the

clearinghouse

does

not

receive

any

messages

from

a

given

worker

for

over

seconds

then

the

clearinghouse

determines

that

the

worker

has

crashed

In

later

update

messages

the

clearinghouse

informs

the

other

workers

of

the

crash

and

the

other

workers

take

appropriate

System

architecture

WorkerPenguinJob

brokerV

ultureC

learinghouseSparrowNode

Node

manager

detects

busy

Node

manager

sends

kill

signal

to

worker

Worker

offloads

work

to

others

orker

unregisters

Node

manager

notifies

job

broker

Figure

A

no-longer-idle

machine

leaves

a

Cilk

job

When

the

node

manager

detects

that

its

machine

is

no-longer

idle

it

sends

a

kill

signal

to

the

worker

The

worker

catches

this

signal

offloads

its

work

to

other

workers

unregisters

with

the

clearinghouse

and

then

terminates

remedial

action

as

described

in

Section

If

the

clearinghouse

incorrectly

determines

that

a

worker

has

crashed

and

then

receives

a

message

from

that

worker

then

the

clearinghouse

refuses

to

reply

with

update

messages

A

worker

that

receives

no

update

messages

from

the

clearinghouse

for

over

seconds

commits

suicide

Thus

a

worker

incorrectly

determined

to

be

crashed

will

eventually

crash

and

the

communication

protocols

ensure

correct

operation

in

this

event

With

each

worker

communicating

with

the

clearinghouse

once

to

register

once

to

unregister

and

once

every

seconds

for

an

update

we

expect

that

a

clearinghouse

can

service

up

to

workers

Beyond

this

level

and

in

a

wide-area

network

we

may

require

multiple

clearinghouses

configured

in

a

hierarchy

All

of

the

communication

between

workers

and

between

workers

and

the

clearinghouse

is

implemented

with

UDP/IP

-LSB-

-RSB-

UDP/IP

is

an

unreliable

datagram

protocol

built

on

top

of

the

internet

protocol

-LSB-

-RSB-

The

protocols

implemented

in

the

Cilk-NOW

runtime

system

all

use

UDP/IP

to

perform

split-phase

communication

so

except

in

the

case

of

work

stealing

a

worker

never

sits

idle

waiting

for

a

reply

or

an

acknowledgment

Knowing

that

UDP

datagrams

are

unreliable

the

Cilk-NOW

protocols

incorporate

appropriate

mechanisms

such

as

acknowledgments

retries

and

timeouts

to

ensure

correct

operation

when

messages

get

lost

We

shall

not

discuss

these

mechanisms

in

any

detail

and

in

order

to

simplify

our

exposition

of

Cilk-NOW

we

shall

often

speak?and

indeed

we

already

have

spoken?of

messages

being

sent

and

received

as

if

they

are

reliable

What

we

will

say

about

these

mechanisms

is

that

they

are

not

built

on

top

of

UDP

in

any

effort

to

create

a

reliable

message-passing

layer

Rather

these

mechanisms

are

built

directly

into

the

runtime

system

's

split-phase

protocols

Chapter

Cilk

on

a

network

of

workstations

so

in

the

common

case

when

a

message

does

get

through

Cilk-NOW

pays

no

overhead

to

make

the

message

reliable

We

chose

to

build

Cilk-NOW

's

communication

protocols

using

an

unreliable

messagepassing

layer

instead

of

a

reliable

one

for

two

reasons

First

reliable

layers

such

as

TCP/IP

-LSB-

-RSB-

PVM

-LSB-

-RSB-

and

MPI

-LSB-

-RSB-

all

perform

implicit

acknowledgments

and

retries

to

achieve

reliability

Therefore

such

layers

either

preclude

the

use

of

split-phase

communication

or

require

extra

buffering

and

copying

A

layer

such

as

UDP

which

provides

minimal

service

guarantees

can

be

implemented

with

considerably

less

software

overhead

than

a

layer

with

more

service

features

In

the

common

case

when

the

additional

service

is

not

needed

the

minimal

layer

can

easily

outperform

its

fully-featured

counterpart

Second

in

an

environment

where

machines

can

crash

and

networks

can

break

the

notion

of

a

reliable

messagepassing

layer

is

somewhat

suspect

A

runtime

system

operating

in

an

inherently

unreliable

environment

can

not

expect

the

message-passing

layer

to

make

the

environment

reliable

Rather

the

runtime

system

must

incorporate

appropriate

mechanisms

into

its

protocols

to

take

action

when

a

communication

endpoint

or

link

fails

For

these

reasons

we

chose

to

build

the

Cilk-NOW

runtime

system

on

top

of

a

minimal

layer

of

message-passing

service

and

incorporate

mechanisms

directly

into

the

runtime

system

's

protocols

in

order

to

handle

issues

of

reliability

The

downside

to

this

approach

is

complexity

The

protocols

implemented

in

the

Cilk-NOW

runtime

system

are

complex

the

code

for

these

protocols

takes

almost

percent

of

the

total

runtime

system

code

and

the

programming

effort

was

probably

near

half

of

the

total

Nevertheless

this

was

a

one-time

effort

that

we

expect

will

reap

performance

rewards

for

a

long

time

to

come

The

job

broker

CilkJobBroker

and

node

manager

CilkNodeManager

are

implemented

using

remote

procedure

calls

-LRB-

RPC

-RRB-

-LSB-

-RSB-

in

the

standard

client/server

configuration

with

the

job

broker

as

the

server

When

the

node

manager

on

a

machine

finds

that

its

machine

is

idle

it

makes

a

remote

procedure

call

to

the

job

broker

to

obtain

a

Cilk

job

description

We

shall

finish

this

section

by

explaining

the

operation

of

the

node

manager

and

the

job

broker

in

their

current

incarnations

Each

machine

in

the

network

runs

a

node

manager

in

the

background

In

general

when

the

machine

is

being

used

the

node

manager

wakes

up

every

seconds

to

determine

if

the

machine

has

gone

idle

It

looks

at

how

much

time

has

elapsed

since

the

keyboard

and

mouse

have

been

touched

the

number

of

users

logged

in

and

the

processor

load

averages

The

node

manager

then

passes

these

values

through

a

predicate

to

decide

if

the

machine

is

idle

This

predicate

can

be

customized

for

each

machine

A

typical

predicate

might

require

that

the

keyboard

and

mouse

have

not

been

touched

for

at

least

minutes

and

the

processor

load

average

is

below

Alternatively

the

owner

of

a

machine

might

set

the

predicate

to

require

that

no

users

are

logged

in

We

believe

that

maintaining

the

owner

's

sovereignty

is

essential

if

we

want

owners

to

allow

their

machines

to

be

used

for

parallel

computation

When

the

predicate

is

satisfied

the

machine

is

idle

and

the

node

manager

obtains

a

Cilk

job

description

from

the

job

broker

and

forks

a

worker

The

node

manager

then

monitors

the

worker

and

continues

to

monitor

the

machine

With

a

worker

running

the

node

manager

wakes

up

once

every

second

to

determine

if

the

machine

is

still

idle

-LRB-

adding

to

any

processor

load-average

threshold

-RRB-

If

the

machine

is

no

longer

idle

then

the

node

manager

sends

a

kill

signal

to

the

worker

as

previously

described

When

the

worker

process

dies

for

any

reason

the

node

manager

takes

one

of

two

possible

actions

If

the

machine

is

Adaptive

parallelism

still

idle

then

it

goes

back

to

the

job

broker

for

another

job

to

work

on

If

the

machine

is

no

longer

idle

then

it

returns

to

monitoring

the

machine

once

every

seconds

The

job

broker

determines

which

idle

machines

work

on

which

Cilk

jobs

In

its

current

implementation

the

job

broker

uses

a

simple

nonpreemptive

round-robin

scheduling

policy

This

policy

is

extremely

unfair

in

that

it

allows

a

single

job

to

hold

all

the

idle

machines

to

the

exclusion

of

all

other

jobs

For

example

if

one

job

is

running

and

using

all

the

idle

machines

then

when

a

second

job

starts

it

will

get

Not

until

some

machines

leave

the

first

job

and

then

later

get

reassigned

by

the

job

broker

will

machines

join

the

second

job

We

are

currently

analyzing

and

experimenting

with

a

simple

probabilistic

scheme

based

on

random

preemptive

reassignments

in

order

to

obtain

fair

scheduling

Our

proposed

scheme

uses

the

work-steal

rates

of

individual

workers

to

govern

the

reassignment

probabilities

Each

job

should

get

its

fair

share

of

the

idle

machines

but

no

job

should

get

more

machines

than

it

can

efficiently

utilize

In

this

proposed

scheme

the

functionality

of

the

job

broker

is

implemented

in

a

totally

distributed

manner

instead

of

in

the

current

client/server

manner

Adaptive

parallelism

Adaptive

parallelism

allows

a

Cilk

job

to

take

advantage

of

idle

machines

whether

or

not

they

are

idle

when

the

job

starts

and

whether

or

not

they

will

remain

idle

for

the

duration

of

the

job

In

order

to

efficiently

utilize

machines

that

may

join

and

leave

a

running

job

the

overhead

of

supporting

this

feature

must

not

excessively

slow

down

the

work

of

any

worker

at

a

time

when

it

is

not

joining

or

leaving

As

we

saw

in

the

previous

section

a

new

worker

joins

a

job

easily

enough

by

stealing

a

closure

A

worker

leaves

a

job

by

migrating

all

of

its

closures

to

other

workers

and

here

the

danger

lies

When

we

migrate

a

waiting

closure

other

closures

with

continuations

that

refer

to

this

closure

must

somehow

update

these

continuations

so

they

can

find

the

waiting

closure

at

its

new

location

-LRB-

Without

adaptive

parallelism

waiting

closures

never

move

-RRB-

Naively

each

migrated

waiting

closure

would

have

to

inform

every

other

closure

of

its

new

location

In

this

section

we

show

how

we

can

take

advantage

of

strictness

and

the

work-stealing

scheduler

to

make

this

migration

extremely

simple

and

efficient

Our

approach

is

to

impose

additional

structure

on

the

organization

of

closures

and

continuations

such

that

the

structure

is

cheap

to

maintain

while

simplifying

the

migration

of

closures

Specifically

we

maintain

closures

in

subcomputations

that

migrate

en

masse

and

every

continuation

in

a

closure

refers

to

a

closure

in

the

same

subcomputation

In

order

to

send

a

value

from

a

closure

in

one

subcomputation

to

a

closure

in

another

we

forward

the

value

through

intermediate

result

closures

and

give

each

result

closure

the

ability

to

send

the

value

to

precisely

one

other

closure

in

one

other

subcomputation

With

this

structure

and

these

mechanisms

all

of

the

overhead

associated

with

adaptive

parallelism

-LRB-

other

than

the

actual

migration

of

closures

-RRB-

occurs

only

when

closures

are

stolen

and

as

we

saw

in

Chapter

the

number

of

steals

grows

at

most

linearly

with

the

critical

path

of

the

computation

but

is

not

a

function

of

the

work

The

bulk

of

this

section

's

exposition

concerns

the

organization

of

closures

in

subcomputations

and

the

implementation

of

continuations

After

Chapter

Cilk

on

a

network

of

workstations

covering

these

topics

the

mechanism

by

which

closures

are

migrated

to

facilitate

adaptive

parallelism

is

quite

straightforward

In

Cilk-NOW

every

closure

is

maintained

in

one

of

three

pools

associated

with

a

data

structure

called

a

subcomputation

A

subcomputation

is

a

record

containing

-LRB-

among

other

things

-RRB-

three

pools

of

closures

The

ready

pool

is

the

leveled

lists

of

ready

closures

described

in

Section

The

waiting

pool

is

a

list

of

waiting

closures

The

assigned

pool

is

a

list

of

ready

closures

that

have

been

stolen

away

Program

execution

begins

with

one

subcomputation?the

root

subcomputation?allocated

by

worker

and

containing

a

single

closure?the

initial

thread

of

cilk

main?in

the

ready

pool

In

general

a

subcomputation

with

any

closures

in

its

ready

pool

is

said

to

be

ready

and

ready

subcomputations

can

be

executed

by

the

scheduler

as

described

in

Section

with

the

additional

provision

that

each

waiting

closure

is

kept

in

the

waiting

pool

and

then

moved

to

the

ready

pool

when

its

join

counter

decrements

to

zero

The

assigned

pool

is

used

in

work

stealing

as

we

shall

now

see

When

a

ready

closure

is

stolen

from

the

ready

pool

of

a

victim

worker

's

subcomputation

the

closure

is

moved

to

the

assigned

pool

and

assigned

to

a

thief

subcomputation

newly

allocated

by

the

thief

worker

The

assignment

is

recorded

by

giving

the

thief

subcomputation

a

unique

name

and

storing

that

name

in

a

record

of

assignment

information

attached

to

the

assigned

closure

as

illustrated

in

Figure

The

subcomputation

's

name

is

formed

by

concatenating

the

worker

's

name

and

a

number

unique

to

that

worker

The

first

subcomputation

allocated

by

a

worker

r

is

named

r

the

second

is

named

r

and

so

on

The

root

subcomputation

is

named

The

thief

subcomputation

stores

its

own

name

and

the

name

of

its

victim

worker

The

victim

's

assigned

closure

stores

the

name

of

the

thief

worker

and

the

name

of

the

thief

subcomputation

in

its

assignment

information

We

refer

to

the

assigned

closure

as

the

thief

subcomputation

's

victim

closure

Thus

the

victim

closure

and

thief

subcomputation

can

refer

to

each

other

via

the

thief

subcomputation

's

name

which

is

stored

both

in

the

victim

closure

's

assignment

information

and

in

the

thief

subcomputation

This

link

between

a

victim

closure

and

a

thief

subcomputation

is

created

during

work

stealing

as

follows

If

a

worker

needs

to

steal

work

then

before

sending

a

steal

request

to

a

victim

it

allocates

a

new

thief

subcomputation

from

a

simple

runtime

heap

The

thief

subcomputation

's

name

is

contained

in

the

steal

request

message

When

the

victim

worker

gets

the

request

message

if

it

has

any

ready

subcomputations

then

it

chooses

a

ready

subcomputation

in

round-robin

fashion

removes

the

closure

at

the

tail

of

the

topmost

nonempty

level

in

the

subcomputation

's

ready

pool

and

places

this

victim

closure

in

the

assigned

pool

The

victim

worker

then

assigns

the

closure

to

the

thief

subcomputation

by

adding

to

the

closure

an

assignment

information

record

allocated

from

a

simple

runtime

heap

and

then

storing

the

name

of

the

thief

worker

and

the

name

of

the

thief

subcomputation

-LRB-

as

contained

in

the

steal

request

message

-RRB-

in

the

assignment

information

Finally

the

victim

worker

sends

a

copy

of

the

closure

to

the

thief

When

the

thief

receives

the

stolen

closure

it

records

the

name

of

the

victim

worker

in

its

thief

subcomputation

and

it

places

the

closure

in

the

subcomputation

's

ready

pool

Now

the

thief

subcomputation

is

ready

and

the

thief

worker

may

commence

executing

it

When

a

worker

finishes

executing

a

thief

subcomputation

the

link

between

the

thief

subcomputation

and

its

victim

closure

is

destroyed

Specifically

when

a

subcomputation

has

no

closures

in

any

of

its

three

pools

then

the

subcomputation

is

finished

A

worker

with

a

finished

thief

subcomputation

sends

a

message

containing

the

subcomputation

's

name

to

the

Adaptive

parallelism

Assigned

pool

SubcomputationClosureAssignment

informationW

orker

sW

orker

rsr

js

i

Subcomputation

name

Victim

worker

namerr

j

Thief

worker

nameThief

subcomputation

name

Figure

A

victim

closure

stolen

from

the

subcomputation

s

i

of

victim

worker

s

is

assigned

to

the

thief

subcomputation

r

j

The

victim

closure

is

placed

in

the

assigned

pool

and

augmented

with

assignment

information

that

records

the

name

of

the

thief

worker

and

the

name

of

the

thief

subcomputation

The

thief

subcomputation

records

its

own

name

and

the

name

of

the

victim

worker

Thus

the

victim

closure

and

thief

subcomputation

can

refer

to

each

other

via

the

thief

subcomputation

's

name

subcomputation

's

victim

worker

Using

this

name

the

victim

worker

finds

the

victim

closure

This

closure

is

removed

from

its

subcomputation

's

assigned

pool

and

then

the

closure

and

its

assignment

information

are

freed

The

victim

worker

then

acknowledges

the

message

and

when

the

thief

worker

receives

the

acknowledgment

it

frees

the

subcomputation

When

the

root

subcomputation

is

finished

the

entire

Cilk

job

is

finished

In

addition

to

allocating

a

new

subcomputation

whenever

a

worker

steals

a

closure

it

alters

any

continuations

in

the

closure

so

that

they

all

refer

to

closures

within

the

same

subcomputation

Consider

a

thief

stealing

a

closure

and

suppose

the

victim

closure

contains

a

continuation

referring

to

a

closure

that

we

call

the

target

The

victim

and

target

closures

must

be

in

the

same

subcomputation

in

the

victim

worker

Continuations

are

implemented

as

the

address

of

the

target

closure

concatenated

with

the

index

of

an

argument

slot

in

the

target

closure

Therefore

the

continuation

in

the

victim

closure

contains

the

address

of

the

target

closure

and

this

address

is

only

meaningful

to

the

victim

worker

When

the

thief

worker

receives

the

stolen

closure

it

replaces

the

continuation

with

a

new

continuation

referring

to

a

new

result

closure

as

follows

First

the

thief

must

locate

the

continuation

with

the

aid

of

a

thread

signature

For

each

thread

in

the

program

the

translator

creates

a

signature

which

specifies

the

thread

address

-LRB-

a

pointer

to

the

thread

's

code

-RRB-

and

the

type

of

each

argument

to

the

thread

All

of

the

thread

signatures

are

stored

in

a

table

To

find

the

Chapter

Cilk

on

a

network

of

workstations

continuations

in

a

closure

the

worker

uses

the

closure

's

thread

to

lookup

the

signature

in

the

table

Then

the

worker

uses

the

signature

to

locate

which

arguments

are

continuations

Having

located

the

continuation

in

the

stolen

closure

the

thief

allocates

a

new

result

closure

and

replaces

the

continuation

with

a

new

continuation

referring

to

a

slot

in

the

result

closure

as

illustrated

in

Figure

The

result

closure

's

thread

is

a

special

system

thread

whose

operation

we

shall

explain

shortly

This

thread

takes

two

arguments

a

continuation

index

and

a

result

value

The

continuation

index

is

supplied

as

the

continuation

's

argument

slot

number

in

the

stolen

closure

The

result

value

is

missing

and

the

continuation

in

the

stolen

closure

is

set

to

refer

to

this

argument

slot

The

result

closure

is

waiting

and

its

join

counter

is

In

general

the

thief

allocates

a

result

closure

and

performs

this

alteration

for

each

continuation

in

the

stolen

closure

The

stolen

and

result

closures

are

part

of

the

same

subcomputation

Worker

rsr

jR

eadypoolResult

closureb

b

a

closure

Argument

slot

Continuation

index

Empty

slot

for

result

value

Waiting

pool

Figure

When

the

thief

worker

r

steals

a

closure

a

which

contains

a

continuation

in

its

argument

slot

the

thief

replaces

this

continuation

with

a

new

continuation

referring

to

an

empty

slot

in

a

newly

allocated

result

closure

b.

Additionally

the

thief

stores

the

continuation

's

slot

number

in

an

argument

slot

of

the

result

closure

Using

continuations

to

send

values

from

one

thread

to

another

operates

as

described

in

Section

but

when

a

value

is

sent

to

a

result

closure

communication

between

different

subcomputations

occurs

When

a

result

closure

receives

its

result

value

it

becomes

ready

and

when

its

thread

executes

it

forwards

the

result

value

to

another

closure

in

another

subcomputation

as

follows

When

a

worker

executing

a

subcomputation

executes

a

result

closure

's

thread

it

sends

a

message

to

the

subcomputation

's

victim

worker

This

message

contains

the

subcomputation

's

name

as

well

as

the

continuation

index

and

result

value

that

are

the

thread

's

arguments

When

the

victim

worker

receives

this

message

it

uses

the

subcomputation

name

to

find

the

victim

closure

Then

it

uses

the

continuation

index

to

find

a

continuation

in

the

victim

closure

Finally

it

uses

this

continuation

to

send

the

result

value

to

the

target

To

summarize

each

subcomputation

contains

a

collection

of

closures

and

every

continuation

in

a

closure

refers

to

another

closure

in

the

same

subcomputation

To

send

a

value

from

a

closure

in

one

subcomputation

to

a

closure

in

another

the

value

must

be

forwarded

through

Adaptive

parallelism

intermediate

result

closures

passing

from

subcomputation

to

subcomputation

by

way

of

the

links

between

thief

subcomputations

and

victim

closures

All

relations

between

different

subcomputations

are

confined

to

these

links

With

this

structure

migrating

a

subcomputation

from

one

worker

x

to

another

worker

y

is

fairly

straightforward

First

all

of

the

subcomputation

's

closures

are

swizzled

in

order

to

replace

addresses

with

values

that

have

meaning

independent

of

any

particular

worker

's

address

space

as

follows

The

thread

field

in

each

closure

is

translated

from

an

address

to

an

index

in

the

table

of

thread

signatures

and

each

closure

is

assigned

a

unique

index

Then

using

the

thread

signatures

again

for

each

continuation

in

each

closure

the

pointer

portion

of

the

continuation

is

replaced

with

the

target

closure

's

index

Having

swizzled

all

of

its

closures

the

subcomputation

and

the

closures

are

sent

in

messages

from

worker

x

to

worker

y

The

subcomputation

keeps

its

name

When

the

entire

subcomputation

and

all

of

its

closures

have

been

migrated

to

worker

y

the

closures

are

unswizzled

to

replace

the

thread

and

closure

indices

with

actual

addresses

Finally

worker

y

sends

a

message

to

the

subcomputation

's

victim

worker

to

inform

the

victim

closure

of

its

thief

subcomputation

's

new

thief

worker

Additionally

for

each

of

the

subcomputation

's

assigned

closures

it

sends

a

message

to

the

thief

worker

to

inform

the

thief

subcomputation

of

its

victim

closure

's

new

victim

worker

Thus

all

of

the

links

between

victim

closures

and

thief

subcomputations

are

preserved

Adaptive

parallelism

proved

extremely

valuable

in

the

protein-folding

experiment

described

at

the

top

of

this

chapter

Figure

plots

the

number

of

machines

that

were

idle

at

each

point

in

time

over

the

course

of

a

typical

week

for

our

network

of

SparcStations

As

can

be

seen

from

this

plot

though

many

more

machines

are

idle

at

night

a

significant

number

of

machines

are

idle

at

various

times

throughout

the

day

Therefore

by

adaptively

using

idle

machines

both

day

and

night

we

can

take

advantage

of

significantly

more

machine

resources

than

if

we

run

our

parallel

jobs

as

batch

jobs

during

the

night

During

the

day

however

a

given

machine

is

less

likely

to

remain

idle

for

an

extended

period

of

time

Thus

in

order

to

efficiently

use

machines

adaptively

during

the

day

the

runtime

system

must

be

able

to

utilize

potentially

short

intervals

of

idle

time

To

document

the

efficiency

with

which

Cilk-NOW

can

utilize

short

periods

of

idle

time

we

ran

the

following

experiment

We

used

SparcStation

workstations

connected

by

Ethernet

and

we

ran

the

knary

-LRB-

-RRB-

program

-LRB-

see

page

-RRB-

several

times

with

varying

degrees

of

adaptiveness

We

controlled

the

amount

of

adaptiveness

by

killing

each

worker

and

then

starting

a

new

worker

at

fixed

intervals

on

each

machine

Figure

shows

the

results

of

this

experiment

The

horizontal

position

of

each

plotted

datum

is

the

amount

of

adaptiveness

in

the

run

as

measured

by

the

average

execution

time

for

each

worker

For

example

a

datum

plotted

at

a

horizontal

position

of

seconds

represents

a

run

in

which

the

average

worker

running

on

each

of

the

machines

worked

for

seconds

before

leaving

the

job

The

knary

-LRB-

-RRB-

program

performs

approximately

seconds

of

work

so

such

a

run

involves

at

least

total

workers

The

vertical

position

of

each

plotted

datum

is

the

efficiency

of

the

run

as

measured

by

taking

the

ratio

of

the

work

in

the

computation

-LRB-

or

execution

time

with

one

worker

and

no

adaptiveness

-RRB-

to

the

sum

of

the

execution

node

manager

's

idleness

predicate

on

all

machines

was

conservatively

set

to

require

that

the

keyboard

and

mouse

have

not

been

touched

for

minutes

and

the

and

minute

processor

load

averages

are

below

and

respectively

Chapter

Cilk

on

a

network

of

workstations

Time

-LRB-

days

-RRB-

Number

of

Workers

SatFriThuWedTueMonSun

Figure

This

plot

shows

the

number

of

machines

out

of

the

machines

in

our

network

that

are

idle

over

the

course

of

one

typical

week

in

March

times

of

each

worker

in

the

run

If

the

efficiency

is

then

the

total

worker

execution

time

equals

the

work

of

the

computation

and

every

worker

is

utilized

with

perfect

efficiency

This

plot

shows

that

even

when

the

average

worker

stays

with

the

job

for

only

seconds

the

efficiency

is

still

over

percent

and

if

the

average

worker

stays

with

the

job

for

minute

then

the

efficiency

is

generally

over

percent

Fault

tolerance

With

transparent

fault

tolerance

built

into

the

Cilk-NOW

runtime

system

Cilk

jobs

may

survive

machine

crashes

or

network

outages

despite

the

fact

that

Cilk

programs

are

fault

oblivious

having

been

coded

with

no

special

provision

for

handling

machine

or

network

failures

If

a

worker

crashes

then

other

workers

automatically

redo

any

work

that

was

lost

in

the

crash

In

the

case

of

a

more

catastrophic

failure

such

as

a

power

outage

a

total

network

failure

or

a

crash

of

the

file

server

then

all

workers

may

crash

For

this

case

Cilk-NOW

provides

automatic

checkpointing

so

when

service

is

restored

the

Cilk

job

may

be

restarted

with

minimal

lost

work

In

this

section

we

show

how

the

structure

used

to

support

adaptive

parallelism?which

leverages

strictness

and

the

work-stealing

scheduler?may

be

further

leveraged

to

build

these

fault

tolerant

capabilities

in

Cilk-NOW

Given

adaptive

parallelism

fault

tolerance

is

only

a

short

step

away

With

adaptive

parallelism

a

worker

may

leave

a

Cilk

job

but

before

doing

so

it

first

migrates

all

of

its

subcomputations

to

other

workers

In

contrast

when

a

worker

crashes

all

of

its

subcomputa

Fault

tolerance

50 100 150

Average

participation

time

-LRB-

in

seconds

-RRB-

Efficiency

Figure

This

plot

shows

the

efficiency

of

executions

of

the

knary

-LRB-

-RRB-

program

as

a

function

of

the

amount

of

adaptiveness

The

adaptiveness

is

measured

as

the

average

amount

of

time

each

worker

participated

in

the

job

The

efficiency

is

measured

as

the

ratio

of

the

work

in

the

computation

to

the

total

worker

execution

time

tions

are

lost

To

support

fault

tolerance

we

add

a

mechanism

that

allows

surviving

workers

to

redo

any

work

that

was

done

by

the

lost

subcomputations

Such

a

mechanism

must

address

two

fundamental

issues

First

not

all

work

is

necessarily

idempotent

so

redoing

work

may

present

problems

We

address

this

issue

by

ensuring

that

the

work

done

by

any

given

subcomputation

does

not

affect

the

state

of

any

other

subcomputations

until

the

given

subcomputation

finishes

Thus

from

the

point-of-view

of

any

other

subcomputation

the

work

of

a

subcomputation

appears

as

a

transaction

either

the

subcomputation

finishes

and

commits

its

work

by

making

it

visible

to

other

subcomputations

or

the

subcomputation

never

happened

Second

the

lost

subcomputations

may

have

done

a

large

amount

of

work

and

we

would

like

to

minimize

the

amount

of

work

that

needs

to

be

redone

We

address

this

issue

by

incorporating

a

transparent

and

fully

distributed

checkpointing

facility

This

checkpointing

facility

also

allows

a

Cilk

job

to

be

restarted

in

the

case

of

a

total

system

failure

in

which

every

worker

crashes

To

make

the

work

of

a

subcomputation

appear

from

the

outside

as

a

transaction

we

modify

the

behavior

of

the

subcomputation

's

result

closures

by

delaying

their

execution

until

the

subcomputation

finishes

Rather

than

set

the

join

counter

in

each

result

closure

to

we

set

the

join

counter

to

so

the

result

closure

will

never

be

ready

and

always

wait

Additionally

rather

than

keep

the

result

closures

in

the

subcomputation

's

waiting

pool

we

keep

them

in

a

special

result

pool

When

the

subcomputation

's

ready

waiting

and

assigned

pools

are

all

empty

then

the

subcomputation

is

finished

and

the

result

closures

may

be

executed

The

Chapter

Cilk

on

a

network

of

workstations

thread

executed

by

each

of

these

closures

sends

a

message

to

the

subcomputation

's

victim

worker

Also

the

fact

that

the

subcomputation

is

finished

warrants

a

message

to

the

victim

worker

We

bundle

all

of

these

messages

into

a

single

larger

message

sent

to

the

victim

worker

When

the

victim

worker

receives

this

message

it

commits

all

of

the

thief

subcomputation

's

work

by

sending

the

appropriate

result

values

from

the

victim

closure

freeing

the

victim

closure

and

sending

an

acknowledgment

back

to

the

thief

worker

By

delaying

the

execution

of

the

result

closures

we

have

a

very

simple

technique

for

making

the

work

of

a

subcomputation

appear

as

a

transaction

but

on

the

negative

side

this

delaying

technique

may

result

in

a

longer

critical

path

So

far

of

our

applications

suffer

from

this

effect

because

all

of

our

applications

are

fully

strict

and

each

procedure

sends

a

value

to

its

parent

only

as

the

last

thing

it

does

Should

the

need

arise

we

may

allow

subcomputations

to

perform

commits

before

they

finish

by

tying

these

commits

to

checkpoints

With

subcomputations

having

this

transactional

nature

a

Cilk

job

can

tolerate

individual

worker

crashes

as

follows

Suppose

a

worker

crashes

Eventually

the

clearinghouse

will

detect

the

crash

and

the

other

living

workers

will

learn

of

the

crash

at

the

next

update

from

the

clearinghouse

When

a

worker

learns

of

a

crash

it

goes

through

all

of

its

subcomputations

checking

each

assigned

closure

to

see

if

it

is

assigned

to

the

crashed

worker

Each

such

closure

is

moved

from

the

assigned

pool

back

to

the

ready

pool

-LRB-

and

its

assignment

information

is

freed

-RRB-

Thus

all

of

the

work

done

by

the

closure

's

thief

subcomputation

which

has

been

lost

in

the

crash

will

eventually

be

redone

Additionally

when

a

worker

learns

of

a

crash

it

goes

through

all

of

its

subcomputations

to

see

if

it

has

any

that

record

the

crashed

worker

as

the

subcomputation

's

victim

For

each

such

subcomputation

the

worker

aborts

it

as

follows

The

worker

goes

through

all

of

the

subcomputation

's

assigned

closures

sending

to

each

thief

worker

an

abort

message

specifying

the

name

of

the

thief

subcomputation

Then

the

worker

frees

the

subcomputation

and

all

of

its

closures

When

a

worker

receives

an

abort

message

it

finds

the

thief

subcomputation

named

in

the

message

and

recursively

aborts

it

All

of

the

work

done

by

these

aborted

subcomputations

must

eventually

be

redone

In

order

to

avoid

aborting

all

of

these

subcomputations

-LRB-

which

may

comprise

the

entire

job

in

the

case

when

the

root

subcomputation

is

lost

-RRB-

and

redoing

potentially

vast

amounts

of

work

and

in

order

to

allow

restarting

when

the

entire

job

is

lost

we

need

checkpointing

Cilk-NOW

performs

automatic

checkpointing

without

any

synchronization

among

different

workers

and

without

any

notion

of

global

state

Specifically

each

subcomputation

is

periodically

checkpointed

to

a

file

named

with

the

subcomputation

's

name

For

example

a

subcomputation

named

r

i

would

be

checkpointed

to

a

file

named

scomp

r

i

We

assume

that

all

workers

in

the

job

have

access

to

a

common

file

system

-LRB-

through

NFS

or

AFS

for

example

-RRB-

and

all

checkpoint

files

are

written

to

a

common

checkpoint

directory

To

write

a

checkpoint

file

for

a

subcomputation

r

i

the

worker

first

opens

a

file

named

scomp

r

i.temp

Then

it

swizzles

all

of

the

subcomputation

's

closures

as

described

in

the

previous

section

writes

the

subcomputation

record

and

all

of

the

closures?including

the

assignment

information

for

the

assigned

closures?into

the

file

and

unswizzles

the

closures

Finally

it

atomically

renames

the

file

scomp

r

i.temp

to

scomp

r

i

overwriting

any

previous

checkpoint

file

A

checkpoint

file

can

be

read

to

recover

the

subcomputa

have

not

yet

implemented

any

sort

of

distributed

file

system

In

the

current

implementation

workers

implicitly

synchronize

when

they

write

checkpoint

files

since

they

all

access

a

common

file

system

Fault

tolerance

tion

by

simply

unswizzling

the

closures

as

described

in

the

previous

section

On

writing

a

checkpoint

file

the

worker

additionally

prunes

any

no-longer-needed

checkpoint

files

as

follows

Suppose

that

when

the

previous

checkpoint

file

was

written

the

subcomputation

had

a

closure

assigned

to

a

thief

subcomputation

s

j

but

since

then

the

thief

subcomputation

has

finished

and

the

assigned

closure

has

been

freed

In

this

case

as

soon

as

the

new

checkpoint

file

scomp

r

i

is

written

the

checkpoint

file

scomp

s

j

is

no

longer

needed

The

worker

deletes

the

checkpoint

file

scomp

s

j

as

follows

It

first

reads

the

file

and

for

each

of

the

subcomputation

's

assigned

closures

it

recursively

deletes

the

thief

subcomputation

's

checkpoint

file

as

this

checkpoint

file

is

also

no

longer

needed

Finally

the

worker

actually

removes

the

file

If

workers

crash

the

lost

subcomputations

can

be

recovered

from

checkpoint

files

In

the

case

of

a

single

worker

crash

the

lost

subcomputations

can

be

recovered

automatically

When

a

surviving

worker

finds

that

it

has

a

subcomputation

with

a

closure

assigned

to

the

crashed

worker

then

it

can

recover

the

thief

subcomputation

by

reading

the

checkpoint

file

It

is

possible

that

the

checkpoint

file

may

not

exist

but

in

this

case

the

thief

subcomputation

having

not

written

a

checkpoint

file

can

not

have

done

very

much

work

so

little

is

lost

by

redoing

the

work

as

previously

described

In

the

case

of

a

large-scale

failure

in

which

every

worker

crashes

the

Cilk

job

can

be

restarted

from

checkpoint

files

by

setting

the

Recover

flag

on

the

command

line

Recovery

begins

with

the

root

subcomputation

whose

checkpoint

file

is

scomp

After

recovering

the

root

subcomputation

then

every

other

subcomputation

can

be

recovered

by

recursively

recovering

the

thief

subcomputation

for

each

of

the

root

subcomputation

's

assigned

closures

Without

checkpointing

we

could

not

have

completed

the

protein-folding

experiment

described

at

the

top

of

this

chapter

In

addition

to

several

single-machine

crashes

-LRB-

mostly

due

to

power

cycling

-RRB-

we

experienced

a

total

system

failure

when

our

network

's

NFS

server

was

down

for

maintenance

When

the

server

came

back

on

line

we

were

able

to

restart

the

program

from

where

it

left

off

Chapter

Conclusions

Though

this

thesis

has

established

an

algorithmic

foundation

on

which

to

build

parallel

multithreaded

systems

much

work

remains

to

be

done

We

have

shown

that

strictness

is

a

sufficient

condition

for

efficient

scheduling

though

surely

it

is

not

a

necessary

condition

If

we

can

identify

useful

forms

of

nonstrictness

and

develop

provably

efficient

scheduling

algorithms

for

these

cases

then

we

can

generalize

the

applicability

of

Cilk

Some

forms

of

nonstrictness

for

example

might

allow

Cilk

to

efficiently

execute

more

synchronous

types

of

applications

Our

current

work

is

focused

on

incorporating

in

Cilk

a

distributed

shared

memory

using

dag

consistency

-LSB-

-RSB-

A

distributed

shared

memory

provides

a

global

virtual

address

space

so

an

instruction

may

load

from

or

store

to

a

virtual

address

that

has

meaning

independent

of

which

processor

executes

the

instruction

With

deterministic

dag

consistency

we

specify

the

value

returned

when

an

instruction

performs

a

load

as

follows

All

of

the

stores

to

any

given

virtual

address

must

be

totally

ordered

with

respect

to

each

other

in

the

computation

's

dag

of

instructions

Additionally

each

individual

load

from

a

virtual

address

must

be

totally

ordered

with

respect

to

all

of

the

stores

to

that

address

Then

the

value

returned

by

a

load

instruction

is

the

value

stored

by

the

immediately

preceding

store

instruction

in

this

total

order

Dag

consistency

can

be

generalized

to

the

case

when

no

such

total

order

exists

by

introducing

nondeterministic

execution

Dag

consistency

builds

upon

our

algorithmic

foundations

and

for

the

case

of

fully

strict

computations

we

can

prove

bounds

on

the

number

of

page

-LRB-

or

object

-RRB-

faults

We

have

recently

implemented

dag

consistency

and

a

simple

cactus-stack

memory

allocator

on

the

and

we

have

coded

several

divideand-conquer

applications

including

blocked

matrix

multiply

Strassen

's

matrix

multiply

and

a

Barnes-Hut

N-body

simulation

Cilk

together

with

dag

consistency

makes

coding

these

applications

particularly

simple

Strassen

's

algorithm

was

coded

in

one

evening

Preliminary

results

with

these

applications

is

extremely

encouraging

though

we

do

not

have

numbers

to

report

here

We

further

plan

to

add

dag

consistent

shared

memory

to

Cilk-NOW

We

have

proven

that

dag

consistency

can

be

maintained

using

a

simple

and

efficient

checkout/commit

algorithm

and

the

nature

of

this

algorithm

dovetails

perfectly

into

Cilk-NOW

's

implementation

of

adaptive

parallelism

and

fault

tolerance

This

thesis

has

shown

that

by

building

a

multithreaded

language

and

runtime

system

on

top

of

an

algorithmic

foundation

programmers

can

focus

on

expressing

the

parallelism

in

their

algorithm

and

leave

the

runtime

system

's

scheduler

to

manage

the

low-level

details

of

scheduling

secure

and

confident

in

the

scheduler

's

ability

to

deliver

performance

Cilk

Chapter

Conclusions

is

just

such

a

language

and

runtime

system

In

Cilk

we

can

delineate

a

specific

class

of

programs?fully

strict

programs?and

for

this

class

we

can

guarantee

efficient

execution

and

performance

that

is

predictable

with

a

simple

model

based

on

work

and

critical

path

length

Thus

besides

the

programming

abstractions

provided

by

the

language

Cilk

provides

a

performance

abstraction

We

argue

that

rather

than

think

about

machine-specific

quantities

such

as

execution

time

and

communication

costs

the

performance-minded

programmer

should

instead

think

about

program

abstractions

such

as

work

and

critical

path

length

and

then

rely

on

the

runtime

system

's

performance

model

to

translate

these

abstract

quantities

into

real

quantities

The

value

of

Cilk

's

performance

abstraction

is

intimately

linked

to

its

provable

efficiency

Abstraction

requires

guarantees

Appendix

A

Proof

of

lower

bound

Theorem

For

any

and

any

there

exists

a

depth-first

multithreaded

computation

with

work

average

parallelism

T

and

stack

depth

such

that

the

following

holds

For

any

number

P

of

processors

and

any

value

r

in

the

range

r

T

if

X

is

a

P-processor

execution

schedule

that

achieves

speedup

r?that

is

T

-LRB-

X

-RRB-

r?then

S

-LRB-

X

-RRB-

-LRB-

r

-RRB-

Proof

To

exhibit

the

desired

depth-first

multithreaded

computation

with

work

critical

path

length

T

and

stack

depth

we

first

ignore

the

partitioning

of

instructions

into

threads

and

consider

just

the

dag

structure

of

the

computation

Minus

a

few

instructions

and

dependencies

the

dag

appears

as

in

Figure

A.

-LRB-

a

-RRB-

The

instructions

are

organized

into

m

p

separate

components

Cm

that

we

call

chains

Each

chain

begins

with

l

p

instructions

that

we

call

headers

-LRB-

vertical

hashed

in

Figure

A.

-LRB-

a

-RRB-

-RRB-

After

the

headers

each

chain

contains

n

p

instructions

-LRB-

plain

white

in

Figure

A.

-LRB-

a

-RRB-

-RRB-

that

form

the

trunk

At

the

end

of

each

chain

there

are

l

blockers

-LRB-

horizontal

hashed

in

Figure

A.

-LRB-

a

-RRB-

-RRB-

Each

chain

therefore

consists

of

n

-LRB-

-RRB-

instructions

Since

there

are

m

chains

the

total

number

of

instructions

accounted

for

by

the

m

chains

is

-LRB-

-RRB-

and

this

number

is

no

more

than

since

The

remaining

-LRB-

at

least

-RRB-

instructions

form

the

parts

of

the

computation

not

shown

in

Figure

A.

-LRB-

a

-RRB-

There

are

no

dependencies

between

different

chains

so

the

average

parallelism

T

is

at

least

m

and

the

critical

path

length

T

is

no

more

than

as

promised

Now

consider

the

partitioning

of

the

instructions

from

each

chain

into

the

actual

threads

As

alluded

to

in

Figure

A.

-LRB-

b

-RRB-

the

root

thread

has

m

child

threads

each

of

which

is

what

follows

we

refer

to

a

number

x

of

objects

-LRB-

such

as

instructions

-RRB-

when

x

may

not

be

integral

Rounding

these

quantities

to

integers

does

not

affect

the

correctness

of

the

proof

For

ease

of

exposition

we

shall

not

consider

the

issue

Appendix

A.

Proof

of

lower

bound

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

AA

-LRB-

a

-RRB-

Chains

of

instructions

iteration

-LRB-

b

-RRB-

Outer

iterations

Figure

A.

Constructing

a

computation

with

no

efficient

execution

schedule

The

header

instructions

of

chain

Ci

and

the

blocker

instructions

of

chain

Ci

are

both

placed

in

the

threads

of

the

ith

outer

iteration

-LRB-

f

-RRB-

-LRB-

s

-RRB-

-LRB-

s

-RRB-

-LRB-

f

-RRB-

-LRB-

f

-RRB-

-LRB-

s

-RRB-

-LRB-

s

-RRB-

-LRB-

f

-RRB-

-LRB-

f

-RRB-

-LRB-

s

-RRB-

tm

-LRB-

s

-RRB-

tm

-LRB-

f

-RRB-

Figure

A.

Scheduling

the

execution

of

the

chains

A

solid

vertical

interval

from

t

-LRB-

s

-RRB-

i

to

t

-LRB-

f

-RRB-

i

indicates

the

time

during

which

the

trunk

of

chain

Ci

is

being

executed

When

t

-LRB-

s

-RRB-

i

t

-LRB-

f

-RRB-

i

we

can

define

an

interval

shown

dashed

of

length

ti

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

during

which

chain

Ci

is

exposed

the

root

of

a

subcomputation

that

we

call

an

outer

iteration

-LRB-

The

outer

iterations

contain

inner

iterations

that

will

be

discussed

later

-RRB-

Each

of

these

outer

iterations

contains

threads

As

indicated

by

the

shading

in

Figure

A.

the

ith

outer

iteration

for

i

m

contains

both

the

header

instructions

of

chain

Ci

and

the

blocker

instructions

of

chain

Ci

These

instructions

are

organized

into

the

threads

of

the

outer

iteration

so

as

to

ensure

that

chain

Ci

can

not

begin

executing

its

trunk

instructions

until

all

of

the

outer

iteration

's

threads

have

been

spawned

and

of

these

threads

can

die

until

chain

Ci

begins

executing

its

blocker

instructions

-LRB-

We

will

exhibit

this

organization

later

-RRB-

Thus

if

chain

Ci

begins

executing

its

trunk

instructions

before

chain

Ci

finishes

its

then

the

execution

will

require

at

least

space

For

any

number

P

of

processors

consider

any

valid

P-processor

execution

schedule

X

For

each

chain

Ci

let

t

-LRB-

s

-RRB-

i

denote

the

time

step

at

which

X

executes

the

first

trunk

instruction

of

Ci

and

let

t

-LRB-

f

-RRB-

i

denote

the

first

time

step

at

which

X

executes

a

blocker

instruction

of

Ci

Since

the

trunk

has

length

n

and

no

blocker

instruction

of

Ci

can

execute

until

after

the

last

trunk

instruction

of

Ci

we

have

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

n

Now

consider

two

chains

Ci

and

Ci

and

suppose

t

-LRB-

s

-RRB-

i

t

-LRB-

f

-RRB-

i

this

is

the

scenario

we

described

as

using

at

least

space

In

this

case

we

consider

the

time

interval

from

t

-LRB-

s

-RRB-

i

-LRB-

inclusive

-RRB-

to

t

-LRB-

f

-RRB-

i

-LRB-

exclusive

-RRB-

during

which

we

say

that

chain

Ci

is

exposed

and

we

let

Appendix

A.

Proof

of

lower

bound

ti

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

denote

the

amount

of

time

chain

Ci

is

exposed

See

Figure

A.

If

t

-LRB-

s

-RRB-

i

t

-LRB-

f

-RRB-

i

then

chain

Ci

is

never

exposed

and

we

let

ti

As

we

have

seen

over

the

time

interval

during

which

a

chain

is

exposed

it

uses

at

least

space

We

will

show

that

in

order

for

an

execution

schedule

X

to

achieve

speedup

r?that

is

T

-LRB-

X

-RRB-

r?there

must

be

some

time

step

during

the

execution

at

which

at

least

r

chains

are

exposed

If

schedule

X

is

such

that

T

-LRB-

X

-RRB-

r

then

we

must

have

t

-LRB-

f

-RRB-

m

t

-LRB-

s

-RRB-

r

We

can

expand

this

inequality

to

yield

r

t

-LRB-

f

-RRB-

m

t

-LRB-

s

-RRB-

m

i

-LRB-

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

-RRB-

m

i

-LRB-

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

-RRB-

-LRB-

A.

-RRB-

Considering

the

first

sum

we

recall

that

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

n

hence

m

i

-LRB-

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

-RRB-

mn

-LRB-

A.

-RRB-

Considering

the

second

sum

of

Inequality

-LRB-

A.

-RRB-

when

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

-LRB-

so

Ci

is

exposed

-RRB-

we

have

ti

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

and

otherwise

ti

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

Therefore

m

i

-LRB-

t

-LRB-

f

-RRB-

i

t

-LRB-

s

-RRB-

i

-RRB-

m

i

ti

-LRB-

A.

-RRB-

Substituting

Inequality

-LRB-

A.

-RRB-

and

Inequality

-LRB-

A.

-RRB-

back

into

Inequality

-LRB-

A.

-RRB-

we

obtain

m

i

ti

r

Let

exposed

-LRB-

t

-RRB-

denote

the

number

of

chains

exposed

at

time

step

t

and

observe

that

r

t

exposed

-LRB-

t

-RRB-

m

i

i

ti

Then

the

average

number

of

exposed

chains

per

time

step

is

r

r

t

exposed

-LRB-

t

-RRB-

r

m

i

ti

r

-LRB-

r

-RRB-

since

m

and

n

There

must

be

some

time

step

t

for

which

exposed

-LRB-

t

-RRB-

is

at

least

the

average

and

consequently

exposed

-LRB-

t

-RRB-

ss

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

AAA

Figure

A.

Laying

out

the

chains

into

the

threads

of

a

multithreaded

computation

As

before

the

header

instructions

are

vertical

hashed

and

the

blocker

instructions

are

horizontal

hashed

In

this

example

each

activation

frame

has

unit

size

so

S

Also

in

this

example

l

n

and

only

the

first

out

of

the

m

instructions

in

the

root

thread

are

shown

Each

instruction

of

the

root

thread

spawns

a

child

-LRB-

an

outer

iteration

-RRB-

and

each

child

thread

contains

l

instructions

the

first

l

of

these

spawn

a

child

thread

which

is

the

root

of

an

inner

iteration

with

stack

depth

S

and

the

last

one

spawns

a

leaf

thread

with

the

n

trunk

instructions

of

a

single

chain

Appendix

A.

Proof

of

lower

bound

Now

recalling

that

each

exposed

chain

uses

space

we

have

S

-LRB-

X

-RRB-

ss

p

-LRB-

r

-RRB-

p

for

-LRB-

which

is

true

since

-RRB-

All

that

remains

is

exhibiting

the

organization

of

the

instructions

of

each

chain

into

a

depth-first

multithreaded

computation

with

work

critical

path

length

T

and

stack

depth

S

in

such

a

way

that

each

exposed

chain

uses

space

There

are

actually

many

ways

of

creating

such

a

computation

One

such

way

that

uses

unit

size

activation

frames

for

each

thread

is

shown

in

Figure

A.

For

the

multithreaded

computation

of

Figure

A.

the

root

thread

contains

m

instructions

each

of

which

spawns

a

child

thread

-LRB-

an

outer

iteration

-RRB-

Each

child

thread

contains

l

instructions

the

first

l

of

these

spawn

a

child

thread

which

is

the

root

of

a

subcomputation

that

we

call

an

inner

iteration

Each

inner

iteration

has

stack

depth

-LRB-

since

-RRB-

and

the

last

one

spawns

a

leaf

thread

with

the

n

trunk

instructions

of

a

single

chain

Each

of

these

inner

iterations

contains

a

single

header

from

one

chain

and

a

single

blocker

from

the

previous

chain

-LRB-

except

in

the

case

of

the

first

group

of

l

-RRB-

as

shown

in

Figure

A.

The

header

and

blocker

in

an

inner

iteration

are

organized

such

that

in

order

to

execute

the

header

all

of

the

threads

in

the

inner

iteration

must

be

spawned

and

of

them

can

die

until

the

blocker

executes

Thus

when

a

chain

is

exposed

all

l

of

these

inner

iterations

have

all

of

their

threads

living

thereby

using

space

l

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

We

can

verify

from

Figure

A.

and

from

the

given

values

of

m

l

and

n

that

this

construction

actually

has

work

slightly

less

than

in

order

to

make

the

work

equal

to

we

can

just

add

the

extra

instructions

evenly

among

the

threads

that

contain

the

trunk

of

each

chain

-LRB-

thereby

increasing

n

by

a

bit

-RRB-

Also

we

can

verify

that

T

Finally

looking

at

Figure

A.

we

can

see

that

this

computation

is

indeed

depth-first

The

construction

of

a

multithreaded

computation

with

provably

bad

time/space

characteristics

as

just

described

can

be

modified

in

various

ways

to

accommodate

various

restrictions

to

the

model

while

still

obtaining

the

same

result

For

example

some

real

multithreaded

systems

require

limits

on

the

number

of

instructions

in

a

thread

dependencies

that

only

go

to

the

first

instruction

of

a

thread

limited

fan-in

for

dependencies

or

a

limit

on

the

number

of

children

a

thread

can

have

Simple

changes

to

the

construction

just

described

can

produce

multithreaded

computations

that

accommodate

any

or

all

of

these

restrictions

and

still

have

the

same

provably

bad

time/space

tradeoff

Thus

the

lower

bound

of

Theorem

holds

even

for

multithreaded

computations

with

any

or

all

of

these

restrictions

Appendix

B

Cilk

project

status

Up

to

date

Cilk

information

papers

and

software

releases

are

available

on

the

World

Wide

Web

-LRB-

WWW

-RRB-

at

the

following

two

locations

The

Cilk

team

can

be

contacted

by

electronic

mail

at

the

following

location

-RRB-

cilk-developers@theory.lcs.mit.edu

Contact

the

team

to

get

on

the

cilk-users

mailing

list

The

current

software

release

is

version

This

release

is

essentially

as

described

in

Chapter

The

Cilk

release

includes

the

Cilk-to-C

translator

a

collection

of

example

programs

a

reference

manual

and

runtime

system

support

for

two

multiprocessor

platforms?the

Thinking

Machines

and

the

Sun

SparcStation

SMP?and

uniprocessor

platforms

running

SunOS

or

Linux

Cilk

has

also

been

ported

to

PVM

the

Intel

Paragon

MPP

and

the

Silicon

Graphics

Power

Challenge

SMP

but

these

ports

are

not

included

in

the

current

release

For

ports

to

other

machines

the

Cilk

Reference

Manual

includes

an

Implementor

's

Guide

Contact

the

Cilk

team

for

help

porting

Cilk

to

a

new

platform

The

next

major

release

Cilk

will

include

distributed

shared

memory

A

release

of

the

Cilk-NOW

runtime

system

is

planned

for

the

near

future

This

release

will

support

Cilk

A

release

supporting

Cilk

will

come

sometime

later

Watch

the

web

pages

or

get

on

the

cilk-users

mailing

list

to

keep

abreast

of

forthcoming

software

releases

August

Bibliography

-LSB-

-RSB-

Noga

Alon

and

Joel

H.

Spencer

The

Probabilistic

Method

John

Wiley

Sons

-LSB-

-RSB-

Andrew

W.

Appel

Compiling

with

Continuations

Cambridge

University

Press

New

York

-LSB-

-RSB-

Arvind

Rishiyur

S.

Nikhil

and

Keshav

K.

Pingali

I-structures

Data

structures

for

parallel

computing

ACM

Transactions

on

Programming

Languages

and

Systems

-LRB-

-RRB-

October

-LSB-

-RSB-

Brian

N.

Bershad

Matthew

J.

Zekauskas

and

Wayne

A.

Sawdon

The

Midway

distributed

shared

memory

system

In

Digest

of

Papers

from

the

Thirty-Eighth

IEEE

Computer

Society

International

Conference

-LRB-

Spring

COMPCON

-RRB-

pages

San

Francisco

California

February

-LSB-

-RSB-

Sandeep

Bhatt

David

Greenberg

Tom

Leighton

and

Pangfeng

Liu

Tight

bounds

for

on-line

tree

embeddings

In

Proceedings

of

the

Second

Annual

ACM-SIAM

Symposium

on

Discrete

Algorithms

pages

San

Francisco

California

January

-LSB-

-RSB-

Andrew

Birrell

Greg

Nelson

Susan

Owicki

and

Edward

Wobber

Network

objects

In

Proceedings

of

the

Fourteenth

ACM

Symposium

on

Operating

Systems

Principles

-LRB-

SOSP

-RRB-

pages

Asheville

North

Carolina

December

-LSB-

-RSB-

Andrew

D.

Birrell

An

introduction

to

programming

with

threads

Technical

Report

Digital

Equipment

Corporation

Systems

Research

Center

January

-LSB-

-RSB-

Guy

E.

Blelloch

Programming

parallel

algorithms

In

Proceedings

of

the

Dartmouth

Institute

for

Advanced

Graduate

Studies

-LRB-

DAGS

-RRB-

Symposium

on

Parallel

Computation

pages

Hanover

New

Hampshire

June

-LSB-

-RSB-

Guy

E.

Blelloch

Phillip

B.

Gibbons

and

Yossi

Matias

Provably

efficient

scheduling

for

languages

with

fine-grained

parallelism

In

Proceedings

of

the

Seventh

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

Santa

Barbara

California

July

-LSB-

-RSB-

Robert

D.

Blumofe

Matteo

Frigo

Christopher

F.

Joerg

Bradley

C.

Kuszmaul

Charles

E.

Leiserson

Rob

Miller

Keith

H.

Randall

and

Yuli

Zhou

Cilk

Reference

Manual

MIT

Laboratory

for

Computer

Science

Technology

Square

Cambridge

Massachusetts

June

Bibliography

-LSB-

-RSB-

Robert

D.

Blumofe

Matteo

Frigo

Christopher

F.

Joerg

Charles

E.

Leiserson

and

Keith

H.

Randall

Dag-consistent

distributed

shared

memory

In

Proceedings

of

the

International

Parallel

Processing

Symposium

-LRB-

IPPS

-RRB-

April

to

appear

-LSB-

-RSB-

Robert

D.

Blumofe

Christopher

F.

Joerg

Bradley

C.

Kuszmaul

Charles

E.

Leiserson

Keith

H.

Randall

and

Yuli

Zhou

Cilk

An

efficient

multithreaded

runtime

system

In

Proceedings

of

the

Fifth

ACM

SIGPLAN

Symposium

on

Principles

and

Practice

of

Parallel

Programming

-LRB-

PPoPP

-RRB-

pages

Santa

Barbara

California

July

-LSB-

-RSB-

Robert

D.

Blumofe

and

Charles

E.

Leiserson

Space-efficient

scheduling

of

multithreaded

computations

In

Proceedings

of

the

Twenty

Fifth

Annual

ACM

Symposium

on

Theory

of

Computing

-LRB-

STOC

-RRB-

pages

San

Diego

California

May

-LSB-

-RSB-

Robert

D.

Blumofe

and

Charles

E.

Leiserson

Scheduling

multithreaded

computations

by

work

stealing

In

Proceedings

of

the

Annual

Symposium

on

Foundations

of

Computer

Science

-LRB-

FOCS

-RRB-

pages

Santa

Fe

New

Mexico

November

-LSB-

-RSB-

Robert

D.

Blumofe

and

David

S.

Park

Scheduling

large-scale

parallel

computations

on

networks

of

workstations

In

Proceedings

of

the

Third

International

Symposium

on

High

Performance

Distributed

Computing

-LRB-

HPDC

-RRB-

pages

San

Francisco

California

August

-LSB-

-RSB-

Richard

P.

Brent

The

parallel

evaluation

of

general

arithmetic

expressions

Journal

of

the

ACM

-LRB-

-RRB-

April

-LSB-

-RSB-

Eric

A.

Brewer

and

Robert

Blumofe

Strata

A

multi-layer

communications

library

Technical

Report

to

appear

MIT

Laboratory

for

Computer

Science

Available

as

ftp

-LSB-

-RSB-

F.

Warren

Burton

Storage

management

in

virtual

tree

machines

IEEE

Transactions

on

Computers

-LRB-

-RRB-

March

-LSB-

-RSB-

F.

Warren

Burton

and

David

J.

Simpson

Space

efficient

execution

of

deterministic

parallel

programs

Unpublished

manuscript

-LSB-

-RSB-

F.

Warren

Burton

and

M.

Ronan

Sleep

Executing

functional

programs

on

a

virtual

tree

of

processors

In

Proceedings

of

the

Conference

on

Functional

Programming

Languages

and

Computer

Architecture

pages

Portsmouth

New

Hampshire

October

-LSB-

-RSB-

Clemens

H.

Cap

and

Volker

Strumpen

Efficient

parallel

computing

in

distributed

workstation

environments

Parallel

Computing

-LSB-

-RSB-

Martin

C.

Carlisle

Anne

Rogers

John

H.

Reppy

and

Laurie

J.

Hendren

Early

experiences

with

Olden

In

Proceedings

of

the

Sixth

Annual

Workshop

on

Languages

and

Compilers

for

Parallel

Computing

Portland

Oregon

August

Bibliography

-LSB-

-RSB-

Nicholas

Carriero

Eric

Freeman

and

David

Gelernter

Adaptive

parallelism

on

multiprocessors

Preliminary

experience

with

Piranha

on

the

In

Proceedings

of

the

Sixth

Annual

Workshop

on

Languages

and

Compilers

for

Parallel

Computing

Portland

Oregon

August

-LSB-

-RSB-

Nicholas

Carriero

and

David

Gelernter

Linda

in

context

Communications

of

the

ACM

-LRB-

-RRB-

April

-LSB-

-RSB-

Vint

Cerf

and

Robert

Kahn

A

protocol

for

packet

network

intercommunication

IEEE

Transactions

on

Computers

-LRB-

-RRB-

May

-LSB-

-RSB-

Rohit

Chandra

Scott

Devine

Ben

Verghese

Anoop

Gupta

and

Mendel

Rosenblum

Scheduling

and

page

migration

for

multiprocessor

compute

servers

In

Proceedings

of

the

Sixth

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pages

San

Jose

California

October

-LSB-

-RSB-

Rohit

Chandra

Anoop

Gupta

and

John

L.

Hennessy

Data

locality

and

load

balancing

in

COOL

In

Proceedings

of

the

Fourth

ACM

SIGPLAN

Symposium

on

Principles

and

Practice

of

Parallel

Programming

-LRB-

PPoPP

-RRB-

pages

San

Diego

California

May

-LSB-

-RSB-

Rohit

Chandra

Anoop

Gupta

and

John

L.

Hennessy

COOL

An

object-based

language

for

parallel

programming

IEEE

Computer

-LRB-

-RRB-

August

-LSB-

-RSB-

Jeffrey

S.

Chase

Franz

G.

Amador

Edward

D.

Lazowska

Henry

M.

Levy

and

Richard

J.

Littlefield

The

Amber

system

Parallel

programming

on

a

network

of

multiprocessors

In

Proceedings

of

the

Twelfth

ACM

Symposium

on

Operating

Systems

Principles

-LRB-

SOSP

-RRB-

pages

Litchfield

Park

Arizona

December

-LSB-

-RSB-

David

R.

Cheriton

The

V

distributed

system

Communications

of

the

ACM

-LRB-

-RRB-

March

-LSB-

-RSB-

Andrew

A.

Chien

and

William

J.

Dally

Concurrent

Aggregates

-LRB-

CA

-RRB-

In

Proceedings

of

the

Second

ACM

SIGPLAN

Symposium

on

Principles

and

Practice

of

Parallel

Programming

-LRB-

PPoPP

-RRB-

pages

Seattle

Washington

March

Also

MIT

Artificial

Intelligence

Laboratory

Technical

Report

MIT/AI/TR

-LSB-

-RSB-

Henry

Clark

and

Bruce

McMillin

DAWGS?a

distributed

compute

server

utilizing

idle

workstations

Journal

of

Parallel

and

Distributed

Computing

-LRB-

-RRB-

February

-LSB-

-RSB-

Thomas

H.

Cormen

Charles

E.

Leiserson

and

Ronald

L.

Rivest

Introduction

to

Algorithms

The

MIT

Press

Cambridge

Massachusetts

-LSB-

-RSB-

David

E.

Culler

Resource

management

for

the

tagged

token

dataflow

architecture

Master

's

thesis

Department

of

Electrical

Engineering

and

Computer

Science

Massachusetts

Institute

of

Technology

January

Also

MIT

Laboratory

for

Computer

Science

Technical

Report

MIT/LCS/TR

Bibliography

-LSB-

-RSB-

David

E.

Culler

Managing

Parallelism

and

Resources

in

Scientific

Dataflow

Programs

PhD

thesis

Department

of

Electrical

Engineering

and

Computer

Science

Massachusetts

Institute

of

Technology

March

Also

MIT

Laboratory

for

Computer

Science

Technical

Report

MIT/LCS/TR

-LSB-

-RSB-

David

E.

Culler

and

Arvind

Resource

requirements

of

dataflow

programs

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

Honolulu

Hawaii

May

Also

MIT

Laboratory

for

Computer

Science

Computation

Structures

Group

Memo

-LSB-

-RSB-

David

E.

Culler

Anurag

Sah

Klaus

Erik

Schauser

Thorsten

von

Eicken

and

John

Wawrzynek

Fine-grain

parallelism

with

minimal

hardware

support

A

compilercontrolled

threaded

abstract

machine

In

Proceedings

of

the

Fourth

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pages

Santa

Clara

California

April

-LSB-

-RSB-

William

J.

Dally

Linda

Chao

Andrew

Chien

Soha

Hassoun

Waldemar

Horwat

Jon

Kaplan

Paul

Song

Brian

Totty

and

Scott

Wills

Architecture

of

a

message-driven

processor

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

Pittsburgh

Pennsylvania

June

-LSB-

-RSB-

Babak

Falsafi

Alvin

R.

Lebeck

Steven

K.

Reinhardt

Ioannis

Schoinas

Mark

D.

Hill

James

R.

Larus

Anne

Rogers

and

David

A.

Wood

Application-specific

protocols

for

user-level

shared

memory

In

Supercomputing

pages

Washington

D.C.

November

-LSB-

-RSB-

Robert

E.

Felderman

Eve

M.

Schooler

and

Leonard

Kleinrock

The

Benevolent

Bandit

Laboratory

A

testbed

for

distributed

algorithms

IEEE

Journal

on

Selected

Areas

in

Communications

-LRB-

-RRB-

February

-LSB-

-RSB-

Rainer

Feldmann

Peter

Mysliwietz

and

Burkhard

Monien

Studying

overheads

in

massively

parallel

min/max-tree

evaluation

In

Proceedings

of

the

Sixth

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

Cape

May

New

Jersey

June

-LSB-

-RSB-

Raphael

Finkel

and

Udi

Manber

DIB?a

distributed

implementation

of

backtracking

ACM

Transactions

on

Programming

Languages

and

Systems

-LRB-

-RRB-

April

-LSB-

-RSB-

The

MPI

Forum

MPI

A

message

passing

interface

In

Supercomputing

pages

Portland

Oregon

November

-LSB-

-RSB-

Vincent

W.

Freeh

David

K.

Lowenthal

and

Gregory

R.

Andrews

Distributed

Filaments

Efficient

fine-grain

parallelism

on

a

cluster

of

workstations

In

Proceedings

of

the

First

Symposium

on

Operating

Systems

Design

and

Implementation

pages

Monterey

California

November

-LSB-

-RSB-

Matteo

Frigo

June

Private

communication

Bibliography

-LSB-

-RSB-

David

Gelernter

and

David

Kaminsky

Supercomputing

out

of

recycled

garbage

Preliminary

experience

with

Piranha

In

Proceedings

of

the

ACM

International

Conference

on

Supercomputing

pages

Washington

D.C.

July

-LSB-

-RSB-

Seth

Copen

Goldstein

Klaus

Erik

Schauser

and

David

Culler

Enabling

primitives

for

compiling

parallel

languages

In

Third

Workshop

on

Languages

Compilers

and

Run-Time

Systems

for

Scalable

Computers

Troy

New

York

May

-LSB-

-RSB-

R.

L.

Graham

Bounds

for

certain

multiprocessing

anomalies

The

Bell

System

Technical

Journal

November

-LSB-

-RSB-

R.

L.

Graham

Bounds

on

multiprocessing

timing

anomalies

SIAM

Journal

on

Applied

Mathematics

-LRB-

-RRB-

March

-LSB-

-RSB-

Michael

Halbherr

Yuli

Zhou

and

Chris

F.

Joerg

MIMD-style

parallel

programming

with

continuation-passing

threads

In

Proceedings

of

the

International

Workshop

on

Massive

Parallelism

Hardware

Software

and

Applications

Capri

Italy

September

-LSB-

-RSB-

Robert

H.

Halstead

Jr.

Implementation

of

Multilisp

Lisp

on

a

multiprocessor

In

Conference

Record

of

the

ACM

Symposium

on

Lisp

and

Functional

Programming

pages

Austin

Texas

August

-LSB-

-RSB-

Robert

H.

Halstead

Jr.

Multilisp

A

language

for

concurrent

symbolic

computation

ACM

Transactions

on

Programming

Languages

and

Systems

-LRB-

-RRB-

October

-LSB-

-RSB-

W.

Hillis

and

G.

Steele

Data

parallel

algorithms

Communications

of

the

ACM

-LRB-

-RRB-

December

-LSB-

-RSB-

Waldemar

Horwat

Andrew

A.

Chien

and

William

J.

Dally

Experience

with

CST

Programming

and

implementation

In

Proceedings

of

the

ACM

SIGPLAN

Conference

on

Programming

Language

Design

and

Implementation

pages

Portland

Oregon

June

-LSB-

-RSB-

Wilson

C.

Hsieh

Paul

Wang

and

William

E.

Weihl

Computation

migration

Enhancing

locality

for

distributed-memory

parallel

systems

In

Proceedings

of

the

Fourth

ACM

SIGPLAN

Symposium

on

Principles

and

Practice

of

Parallel

Programming

-LRB-

PPoPP

-RRB-

pages

San

Diego

California

May

-LSB-

-RSB-

Suresh

Jagannathan

and

Jim

Philbin

A

customizable

substrate

for

concurrent

languages

In

Proceedings

of

the

ACM

SIGPLAN

Conference

on

Programming

Language

Design

and

Implementation

pages

San

Francisco

California

June

-LSB-

-RSB-

Suresh

Jagannathan

and

Jim

Philbin

A

foundation

for

an

efficient

multi-threaded

Scheme

system

In

Proceedings

of

the

ACM

Conference

on

Lisp

and

Functional

Programming

pages

San

Francisco

California

June

Bibliography

-LSB-

-RSB-

Chris

Joerg

and

Bradley

C.

Kuszmaul

Massively

parallel

chess

In

Proceedings

of

the

Third

DIMACS

Parallel

Implementation

Challenge

Rutgers

University

New

Jersey

October

Available

as

ftp

theory.lcs.mit.edu

Z.

-LSB-

-RSB-

Kirk

L.

Johnson

M.

Frans

Kaashoek

and

Deborah

A.

Wallach

CRL

Highperformance

all-software

distributed

shared

memory

In

Proceedings

of

the

Fifteenth

ACM

Symposium

on

Operating

Systems

Principles

pages

Copper

Mountain

Resort

Colorado

December

-LSB-

-RSB-

Eric

Jul

Henry

Levy

Norman

Hutchinson

and

Andrew

Black

Fine-grained

mobility

in

the

Emerald

system

ACM

Transactions

on

Computer

Systems

February

-LSB-

-RSB-

Christos

Kaklamanis

and

Giuseppe

Persiano

Branch-and-bound

and

backtrack

search

on

mesh-connected

arrays

of

processors

In

Proceedings

of

the

Fourth

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

San

Diego

California

June

-LSB-

-RSB-

David

Louis

Kaminsky

Adaptive

Parallelism

with

Piranha

PhD

thesis

Yale

University

May

-LSB-

-RSB-

Vijay

Karamcheti

and

Andrew

Chien

Concert?efficient

runtime

support

for

concurrent

object-oriented

programming

languages

on

stock

hardware

In

Supercomputing

pages

Portland

Oregon

November

-LSB-

-RSB-

Richard

M.

Karp

and

Vijaya

Ramachandran

Parallel

algorithms

for

shared-memory

machines

In

J.

van

Leeuwen

editor

Handbook

of

Theoretical

Computer

Science

Volume

A

Algorithms

and

Complexity

chapter

pages

MIT

Press

Cambridge

Massachusetts

-LSB-

-RSB-

Richard

M.

Karp

and

Yanjun

Zhang

Randomized

parallel

algorithms

for

backtrack

search

and

branch-and-bound

computation

Journal

of

the

ACM

-LRB-

-RRB-

July

-LSB-

-RSB-

Pete

Keleher

Alan

L.

Cox

Sandhya

Dwarkadas

and

Willy

Zwaenepoel

TreadMarks

Distributed

shared

memory

on

standard

workstations

and

operating

systems

In

USENIX

Winter

Conference

Proceedings

pages

San

Francisco

California

January

-LSB-

-RSB-

David

A.

Kranz

Robert

H.

Halstead

Jr.

and

Eric

Mohr

Mul-T

A

high-performance

parallel

Lisp

In

Proceedings

of

the

SIGPLAN

Conference

on

Programming

Language

Design

and

Implementation

pages

Portland

Oregon

June

-LSB-

-RSB-

Phillip

Krueger

and

Rohit

Chawla

The

Stealth

distributed

scheduler

In

Proceedings

of

the

International

Conference

on

Distributed

Computing

Systems

pages

Arlington

Texas

May

Bibliography

-LSB-

-RSB-

Jeffrey

Kuskin

David

Ofelt

Mark

Heinrich

John

Heinlein

Richard

Simoni

Kourosh

Gharachorloo

John

Chapin

David

Nakahira

Joel

Baxter

Mark

Horowitz

Anoop

Gupta

Mendel

Rosenblum

and

John

Hennessy

The

Stanford

Flash

multiprocessor

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

Chicago

Illinois

April

-LSB-

-RSB-

Bradley

C.

Kuszmaul

Synchronized

MIMD

Computing

PhD

thesis

Department

of

Electrical

Engineering

and

Computer

Science

Massachusetts

Institute

of

Technology

May

Available

as

MIT

Laboratory

for

Computer

Science

Technical

Report

MIT/LCS/TR

or

ftp

phd.ps.Z

-LSB-

-RSB-

Tom

Leighton

Mark

Newman

Abhiram

G.

Ranade

and

Eric

Schwabe

Dynamic

tree

embeddings

in

butterflies

and

hypercubes

In

Proceedings

of

the

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

Santa

Fe

New

Mexico

June

-LSB-

-RSB-

Charles

E.

Leiserson

Zahi

S.

Abuhamdeh

David

C.

Douglas

Carl

R.

Feynman

Mahesh

N.

Ganmukhi

Jeffrey

V.

Hill

W.

Daniel

Hillis

Bradley

C.

Kuszmaul

Margaret

A.

St.

Pierre

David

S.

Wells

Monica

C.

Wong

Shaw-Wen

Yang

and

Robert

Zak

The

network

architecture

of

the

Connection

Machine

In

Proceedings

of

the

Fourth

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

San

Diego

California

June

-LSB-

-RSB-

Kai

Li

and

Paul

Hudak

Memory

coherence

in

shared

virtual

memory

systems

ACM

Transactions

on

Computer

Systems

-LRB-

-RRB-

November

-LSB-

-RSB-

Michael

J.

Litzkow

Miron

Livny

and

Matt

W.

Mutka

Condor?a

hunter

of

idle

workstations

In

Proceedings

of

the

International

Conference

on

Distributed

Computing

Systems

pages

San

Jose

California

June

-LSB-

-RSB-

Pangfeng

Liu

William

Aiello

and

Sandeep

Bhatt

An

atomic

model

for

messagepassing

In

Proceedings

of

the

Fifth

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

Velen

Germany

June

-LSB-

-RSB-

Robert

C.

Miller

A

type-checking

preprocessor

for

Cilk

a

multithreaded

C

language

Master

's

thesis

Department

of

Electrical

Engineering

and

Computer

Science

Massachusetts

Institute

of

Technology

May

-LSB-

-RSB-

Eric

Mohr

David

A.

Kranz

and

Robert

H.

Halstead

Jr.

Lazy

task

creation

A

technique

for

increasing

the

granularity

of

parallel

programs

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

July

-LSB-

-RSB-

B.

Clifford

Neuman

and

Santosh

Rao

The

Prospero

resource

manager

A

scalable

framework

for

processor

allocation

in

distributed

systems

Concurrency

Practice

and

Experience

-LRB-

-RRB-

June

Bibliography

-LSB-

-RSB-

David

A.

Nichols

Using

idle

workstations

in

a

shared

computing

environment

In

Proceedings

of

the

Eleventh

ACM

Symposium

on

Operating

Systems

Principles

-LRB-

SOSP

-RRB-

pages

Austin

Texas

November

-LSB-

-RSB-

Rishiyur

S.

Nikhil

A

multithreaded

implementation

of

Id

using

P-RISC

graphs

In

Proceedings

of

the

Sixth

Annual

Workshop

on

Languages

and

Compilers

for

Parallel

Computing

number

in

Lecture

Notes

in

Computer

Science

pages

Portland

Oregon

August

Springer-Verlag

-LSB-

-RSB-

Rishiyur

S.

Nikhil

Cid

A

parallel

shared-memory

C

for

distributed-memory

machines

In

Proceedings

of

the

Seventh

Annual

Workshop

on

Languages

and

Compilers

for

Parallel

Computing

August

-LSB-

-RSB-

John

K.

Ousterhout

Andrew

R.

Cherenson

Frederick

Douglis

Michael

N.

Nelson

and

Brent

B.

Welch

The

Sprite

network

operating

system

IEEE

Computer

-LRB-

-RRB-

February

-LSB-

-RSB-

Vijay

S.

Pande

Christopher

F.

Joerg

Alexander

Yu

Grosberg

and

Toyoichi

Tanaka

Enumerations

of

the

hamiltonian

walks

on

a

cubic

sublattice

Journal

of

Physics

A

-LSB-

-RSB-

Joseph

D.

Pehoushek

and

Joseph

S.

Weening

Low-cost

process

creation

and

dynamic

partitioning

in

Qlisp

In

Parallel

Lisp

Languages

and

Systems

Proceedings

of

the

US/Japan

Workshop

number

in

Lecture

Notes

in

Computer

Science

pages

Sendai

Japan

June

Springer-Verlag

-LSB-

-RSB-

C.

Gregory

Plaxton

August

Private

communication

-LSB-

-RSB-

Abhiram

Ranade

Optimal

speedup

for

backtrack

search

on

a

butterfly

network

In

Proceedings

of

the

Third

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

Hilton

Head

South

Carolina

July

-LSB-

-RSB-

Steven

K.

Reinhardt

James

R.

Larus

and

David

A.

Wood

Tempest

and

Typhoon

User-level

shared

memory

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

Chicago

Illinois

April

-LSB-

-RSB-

Martin

C.

Rinard

Daniel

J.

Scales

and

Monica

S.

Lam

Jade

A

high-level

machineindependent

language

for

parallel

programming

Computer

-LRB-

-RRB-

June

-LSB-

-RSB-

Larry

Rudolph

Miriam

Slivkin-Allalouf

and

Eli

Upfal

A

simple

load

balancing

scheme

for

task

allocation

in

parallel

machines

In

Proceedings

of

the

Third

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

Hilton

Head

South

Carolina

July

-LSB-

-RSB-

Carlos

A.

Ruggiero

and

John

Sargeant

Control

of

parallelism

in

the

Manchester

dataflow

machine

In

Functional

Programming

Languages

and

Computer

Architecture

number

in

Lecture

Notes

in

Computer

Science

pages

SpringerVerlag

Bibliography

-LSB-

-RSB-

Vikram

A.

Saletore

J.

Jacob

and

M.

Padala

Parallel

computations

on

the

CHARM

heterogeneous

workstation

cluster

In

Proceedings

of

the

Third

International

Symposium

on

High

Performance

Distributed

Computing

pages

San

Francisco

California

August

-LSB-

-RSB-

Daniel

J.

Scales

and

Monica

S.

Lam

The

design

and

evaluation

of

a

shared

object

system

for

distributed

memory

machines

In

Proceedings

of

the

First

Symposium

on

Operating

Systems

Design

and

Implementation

pages

Monterey

California

November

-LSB-

-RSB-

Daniel

J.

Scales

and

Monica

S.

Lam

An

efficient

shared

memory

system

for

distributed

memory

machines

Technical

Report

Computer

Systems

Laboratory

Stanford

University

July

-LSB-

-RSB-

Jaswinder

Pal

Singh

Anoop

Gupta

and

Marc

Levoy

Parallel

visualization

algorithms

Performance

and

architectural

implications

IEEE

Computer

-LRB-

-RRB-

July

-LSB-

-RSB-

W.

Richard

Stevens

UNIX

Network

Programming

Prentice-Hall

Englewood

Cliffs

New

Jersey

-LSB-

-RSB-

V.

S.

Sunderam

PVM

A

framework

for

parallel

distributed

computing

Concurrency

Practice

and

Experience

-LRB-

-RRB-

December

-LSB-

-RSB-

V.

S.

Sunderam

and

Vernon

J.

Rego

EcliPSe

A

system

for

high

performance

concurrent

simulation

Software?Practice

and

Experience

-LRB-

-RRB-

November

-LSB-

-RSB-

Andrew

S.

Tanenbaum

Henri

E.

Bal

and

M.

Frans

Kaashoek

Programming

a

distributed

system

using

shared

objects

In

Proceedings

of

the

Second

International

Symposium

on

High

Performance

Distributed

Computing

pages

Spokane

Washington

July

-LSB-

-RSB-

Andrew

S.

Tanenbaum

Robbert

van

Renesse

Hans

van

Staveren

Gregory

J.

Sharp

Sape

J.

Mullender

Jack

Jansen

and

Guido

van

Rossum

Experiences

with

the

Amoeba

distributed

operating

system

Communications

of

the

ACM

-LRB-

-RRB-

December

-LSB-

-RSB-

Kenneth

R.

Traub

Implementation

of

Non-Strict

Functional

Programming

Languages

The

MIT

Press

Cambridge

Massachusetts

-LSB-

-RSB-

Andrew

Tucker

Efficient

Scheduling

on

Multiprogrammed

Shared-Memory

Multiprocessors

PhD

thesis

Stanford

University

December

-LSB-

-RSB-

Andrew

Tucker

and

Anoop

Gupta

Process

control

and

scheduling

issues

for

multiprogrammed

shared-memory

multiprocessors

In

Proceedings

of

the

Twelfth

ACM

Symposium

on

Operating

Systems

Principles

-LRB-

SOSP

-RRB-

pages

Litchfield

Park

Arizona

December

Bibliography

-LSB-

-RSB-

Mark

T.

Vandevoorde

and

Eric

S.

Roberts

WorkCrews

An

abstraction

for

controlling

parallelism

International

Journal

of

Parallel

Programming

-LRB-

-RRB-

August

-LSB-

-RSB-

Thorsten

von

Eicken

Anindya

Basu

and

Vineet

Buch

Low-latency

communication

over

ATM

networks

using

active

messages

IEEE

Micro

-LRB-

-RRB-

February

-LSB-

-RSB-

Thorsten

von

Eicken

David

E.

Culler

Seth

Copen

Goldstein

and

Klaus

Erik

Schauser

Active

messages

a

mechanism

for

integrated

communication

and

computation

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

Gold

Coast

Australia

May

-LSB-

-RSB-

I-Chen

Wu

Efficient

parallel

divide-and-conquer

for

a

class

of

interconnection

topologies

In

Proceedings

of

the

International

Symposium

on

Algorithms

number

in

Lecture

Notes

in

Computer

Science

pages

Taipei

Republic

of

China

December

Springer-Verlag

-LSB-

-RSB-

I-Chen

Wu

and

H.

T.

Kung

Communication

complexity

for

parallel

divide-andconquer

In

Proceedings

of

the

Annual

Symposium

on

Foundations

of

Computer

Science

pages

San

Juan

Puerto

Rico

October

-LSB-

-RSB-

Y.

Zhang

and

A.

Ortynski

The

efficiency

of

randomized

parallel

backtrack

search

In

Proceedings

of

the

IEEE

Symposium

on

Parallel

and

Distributed

Processing

Dallas

Texas

October

To

appear

-LSB-

-RSB-

Yanjun

Zhang

Parallel

Algorithms

for

Combinatorial

Search

Problems

PhD

thesis

Department

of

Electrical

Engineering

and

Computer

Science

University

of

California

at

Berkeley

November

Also

University

of

California

at

Berkeley

Computer

Science

Division

Technical

Report

UCB/CSD

-LSB-

-RSB-

Songnian

Zhou

Jingwen

Wang

Xiaohu

Zheng

and

Pierre

Delisle

Utopia

A

load

sharing

facility

for

large

heterogeneous

distributed

computer

systems

Software

Practice

and

Experience

-LRB-

-RRB-

December

adaptive parallelism
cilk
computer chess
dataflow
delay sequence
distributed computing
fault tolerance
multithreaded programs
network of workstations
parallel computing
parallel languages
parallel runtime systems
scheduling algorithms
strict execution
thread scheduling
work stealing

massively parallel supercomputer
randomized algorithms
stack memory