Laboratoire

de

l'Informatique

du

Parall?lisme

Ecole

Normale

Sup?rieure

de

Lyon

Unit

de

recherche

associ?e

au

CNRS

n

The

Data-Parallel

Programming

Model

a

Semantic

Perspective

-LRB-

Final

Version

-RRB-

Luc

Boug?e

September

Research

Report

No

Ecole

Normale

Sup?rieure

de

Lyon

Adresse

lectronique

lip@lip.ens-lyon.fr

T?l?phone

-LRB-

-RRB-

T?l?copieur

-LRB-

-RRB-

All?e

d'Italie

Lyon

Cedex

France

The

Data-Parallel

Programming

Model

a

Semantic

Perspective

-LRB-

Final

Version

-RRB-

Luc

Boug?e

September

Abstract

We

provide

a

short

introduction

to

the

data-parallel

programming

model

We

argue

that

parallel

computing

often

makes

little

distinction

between

the

execution

model

and

the

programming

model

This

results

in

poor

programming

and

low

portability

Using

the

GOTO

considered

harmful

analogy

we

show

that

data

parallelism

can

be

seen

as

a

way

out

of

this

difficulty

We

show

that

important

aspects

of

the

data-parallel

model

were

already

present

in

earlier

approaches

to

parallel

programming

and

demonstrate

that

the

data-parallel

programming

model

can

be

characterized

by

a

small

number

of

concepts

with

simple

semantics

R?esum?e

Nous

pr?esentons

une

courte

introduction

au

mod?ele

de

programmation

a

parall?elisme

de

donn?ees

Nous

montrons

que

la

distinction

entre

mod?ele

d'ex

ecution

et

mod?ele

de

programmation

a

souvent

et?e

mal

comprise

dans

le

calcul

parall?ele

Ceci

conduit

a

une

programmation

de

faible

qualit?e

et

peu

portable

A

partir

de

l'analogie

Les

GOTO

sont-ils

nuisibles

nous

montrons

que

le

parall?elisme

de

donn?ees

peut

etre

une

solution

a

cette

difficult?e

Il

appara

t

que

des

aspects

importants

de

ce

mod?ele

etaient

d?ej?a

pr?esents

dans

la

programmation

parall?ele

classique

Nous

montrons

que

le

mod?ele

de

programmation

a

parall?elisme

de

donn?ees

peut

etre

caract?eris?e

par

un

petit

nombre

de

concepts

s?emantiques

simples

Mots-cl

es

Programmation

parall?ele

sp?ecification

et

validation

de

programmes

s?emantique

des

langages

de

programmation

parall?elisme

de

donn?ees

parall?elisme

de

t

aches

The

Data-Parallel

Programming

Model

a

Semantic

Perspective

-LRB-

final

version

-RRB-

Luc

Boug?e

September

Abstract

We

provide

a

short

introduction

to

the

data-parallel

programming

model

We

argue

that

parallel

computing

often

makes

little

distinction

between

the

execution

model

and

the

programming

model

This

results

in

poor

programming

and

low

portability

Using

the

GOTO

considered

harmful

analogy

we

show

that

data

parallelism

can

be

seen

as

a

way

out

of

this

difficulty

We

show

that

important

aspects

of

the

data-parallel

model

were

already

present

in

earlier

approaches

to

parallel

programming

and

demonstrate

that

the

data-parallel

programming

model

can

be

characterized

by

a

small

number

of

concepts

with

simple

semantics

Citation

An

early

version

of

this

paper

has

been

published

in

French

in

Technique

et

science

informatiques

-LRB-

TSI

-RRB-

Special

issue

on

Data-Parallel

Languages

Vol

No.

p.

-LCB-

This

paper

has

been

published

in

The

Data

Parallel

Programming

Model

G.-R

Perrin

A.

Darte

-LRB-

Eds

-RRB-

Lecture

Notes

in

Computer

Science

Tutorial

Series

p.

-LCB-

Please

mention

this

reference

in

citations

Introduction

Execution

Model

vs.

programming

Model

Any

abstract

approach

towards

computing

has

to

deal

with

two

different

kinds

of

objects

computers

which

compute

and

programs

which

are

the

medium

used

to

explain

to

the

computer

what

to

compute

Computers

differ

according

to

their

architectures

which

can

be

classified

according

to

the

execution

model

they

implement

Similarly

programs

differ

according

to

the

language

in

which

they

are

expressed

Languages

can

be

classified

according

to

the

programming

model

they

reflect

In

terms

of

equations

Computing

Computer

Program

and

Abstract

Computing

Model

Execution

Model

Programming

Model

Author

's

address

LIP

URA

CNRS

ENS

Lyon

all?ee

d'Italie

Lyon

Cedex

France

E-mail

Luc.Bouge@lip.ens-lyon.fr

This

work

has

been

partly

supported

by

the

French

CNRS

Coordinated

Research

Program

on

Parallelism

and

Department

of

Defense

DRET

Contract

Luc

Boug?e

is

a

member

of

the

INRIA

Project

ReMaP

Sequential

Programming

In

the

world

of

sequential

computing

much

work

has

been

done

to

understand

and

take

advantage

of

this

dichotomy

The

usual

execution

model

is

the

Von

Neumann

model

contents

of

memory

locations

are

repeatedly

loaded

into

registers

combined

and

stored

back

by

the

processor

On

the

other

hand

many

programming

models

have

been

proposed

Early

programming

languages

such

as

Fortran

derive

from

a

model

very

close

to

the

Von

Neumann

execution

model

The

Pascal

language

introduced

a

more

abstract

structured

programming

model

Programming

models

derived

from

non-operational

approaches

have

definitely

proven

fruitful

logic

programming

-LRB-

e.g.

Prolog

-RRB-

functional

programming

-LRB-

e.g.

Lisp

FP

Scheme

ML

Haskell

-RRB-

object-oriented

programming

-LRB-

e.g.

C++

and

Eiffel

-RRB-

In

all

cases

a

compiler

-LRB-

sometimes

an

interpreter

-RRB-

has

to

map

the

statements

of

the

programming

model

into

the

execution

model

while

preserving

the

semantics

and

maximizing

efficiency

Observe

by

the

way

that

the

distinction

between

programming

and

execution

models

is

relative

to

a

design

level

for

instance

the

designers

of

RISC

processor

microcode

would

probably

consider

the

Von

Neumann

model

as

the

programming

model

-LRB-

load/store

assembly

language

-RRB-

and

the

internal

superscalar

pipelined

machinery

as

the

execution

model

Looking

back

over

the

last

years

one

can

observe

a

continuous

trend

towards

more

and

more

abstract

programming

models

As

the

gap

between

programming

and

execution

models

becomes

larger

compiler

technology

becomes

a

more

and

more

central

concern

for

designers

Parallel

Programming

The

situation

in

the

world

of

parallel

computing

is

less

advanced

Much

effort

has

been

expended

over

more

than

years

to

build

more

and

more

powerful

computers

and

many

different

execution

models

have

been

explored

and

compared

Flynn

-LSB-

-RSB-

proposed

to

classify

these

models

into

four

categories

They

differ

in

the

number

Single

or

M

ultiple

of

I

nstruction

streams

and

Data

streams

which

are

-LRB-

conceptually

-RRB-

active

in

each

point

of

the

computation

SISD

-LRB-

PCs

workstations

etc.

-RRB-

MISD

-LRB-

vector

processors

pipeline

architecture

systolic

arrays

to

some

extent

etc.

-RRB-

SIMD

-LRB-

array

processors

so-called

massively

parallel

computers

etc.

-RRB-

and

MIMD

-LRB-

multicomputers

multiprocessors

networks

of

processors

Transputer-based

architectures

etc.

-RRB-

In

comparison

little

effort

has

been

devoted

to

developing

the

right

part

of

the

above

equation

at

least

in

the

area

of

numerically

intensive

scientific

parallel

computing

In

fact

looking

at

commercial

announcements

of

recent

years

it

seems

that

-LRB-

so-called

-RRB-

massively

parallel

machines

have

developed

so

fast

that

little

time

was

left

to

design

suitable

languages

The

general

tendency

has

thus

been

to

design

languages

by

expanding

the

architecture

that

is

to

define

one

type

of

statement

to

control

each

hardware

feature

This

tendency

has

resulted

in

programming

models

that

form

a

subset

of

the

the

execution

model

They

are

isomorphically

derived

from

the

execution

model

or

more

usually

a

part

of

it

Programming

Model

Execution

Model

A

striking

example

of

this

process

is

the

Transputer

architecture

and

the

Occam

programming

language

Running

two

Transputers

in

parallel

is

expressed

by

the

PAR

Occam

construct

The

synchronous

communication

protocol

over

hardware

Transputer

links

is

expressed

by

the

chan

and

chan

Occam

communication

constructs

Language

restrictions

on

communication

guards

-LRB-

no

output

guards

-RRB-

are

precisely

tuned

to

reflect

the

behavior

of

the

communication

protocol

Another

example

is

the

Connection

Machine

architecture

-LSB-

-RSB-

and

the

-LRB-

new

-RRB-

C

programming

language

-LSB-

-RSB-

The

SIMD

control

of

the

execution

model

and

the

virtual

processor

facility

of

the

sequencer

are

reflected

respectively

in

the

semantics

of

the

where

construct

and

the

shape

definition

These

concepts

were

even

more

crucial

for

the

C/ParIS

and

the

Lisp

languages

which

were

so

closely

connected

to

the

architecture

that

they

could

not

survive

the

advent

of

the

-LSB-

-RSB-

Towards

Abstract

Programming

Models

Many

researchers

have

already

noted

this

imbalance

and

its

bad

effects

How

to

design

more

abstract

models

for

parallel

programming

One

possible

solution

is

to

import

abstract

programming

models

from

the

sequential

world

One

can

extend

them

with

additional

parallel

features

as

needed

and

map

these

models

onto

as

many

parallel

execution

models

as

possible

This

approach

has

been

explored

for

high-level

sequential

programming

models

such

as

logic

programming

-LRB-

e.g.

Parallel

Prolog

etc.

-RRB-

functional

programming

-LRB-

e.g.

the

many

varieties

of

parallel

and/or

concurrent

Lisp

ML

Haskell

and

others

-RRB-

object-oriented

programming

-LRB-

e.g.

parallel

and

distributed

extensions

of

object-oriented

languages

such

as

CC

pC

etc.

-RRB-

Due

to

the

high

level

of

abstraction

designing

efficient

implementations

for

these

models

is

a

very

difficult

challenge

especially

when

performance

is

the

main

goal

as

in

numerically

intensive

parallel

computing

A

notable

exception

is

object-oriented

languages

-LRB-

see

-LSB-

-RSB-

for

instance

-RRB-

where

the

concept

of

class

provides

a

natural

way

of

smoothly

introducing

and

manipulating

parallel

objects

Yet

it

remains

a

promising

research

direction

for

the

future

especially

because

only

high-level

languages

can

support

powerful

and

reliable

software

engineering

design

methods

A

similar

approach

can

also

be

applied

for

less

abstract

sequential

programming

models

such

as

Pascal

or

Dijkstra

's

Guarded

Command

language

In

his

Communicating

Sequential

Processes

-LRB-

CSP

-RRB-

programming

model

-LSB-

-RSB-

Hoare

simply

composes

ordinary

Guarded

Commands

programs

with

an

explicit

parallelism

control

operator

This

programming

model

is

thus

a

direct

abstraction

of

the

MIMD

execution

model

This

approach

has

proven

fruitful

as

it

has

been

the

basis

of

many

parallel

languages

such

as

Occam

the

task

model

of

Ada

and

communication

routine

libraries

used

on

commercially

available

MIMD

parallel

machines

-LRB-

PVM

MPI

etc.

-RRB-

Yet

it

turns

out

that

programming

with

such

an

explicit

parallel

control

is

difficult

and

error-prone

especially

when

a

large

number

of

interacting

processes

come

into

play

Furthermore

this

programming

model

is

poorly

portable

to

non-MIMD

parallel

architectures

-LRB-

SIMD

and

vector

processing

architectures

for

instance

-RRB-

With

the

recent

definition

of

Fortran

which

extends

the

sequential

Fortran

language

with

parallel

arrays

and

the

successful

effort

towards

a

unified

High

Performance

Fortran

-LSB-

-RSB-

a

promising

intermediate

solution

is

now

emerging

It

is

often

referred

to

as

the

data-parallel

programming

model

to

emphasize

that

control

remains

sequential

whereas

data

access

is

done

in

parallel

The

goal

of

this

presentation

is

to

analyze

the

semantic

nature

of

this

model

and

to

discuss

its

relationship

with

the

CSP

model

We

show

that

the

data-parallel

model

of

HPF

or

C

can

be

seen

as

a

structured

version

of

the

control-parallel

model

of

CSP

Moreover

we

show

that

the

data-parallel

model

can

be

characterized

by

a

small

number

of

semantic

concepts

From

Control

Parallelism

to

Data

Parallelism

It

is

useful

at

this

point

to

look

back

at

what

happened

in

the

's

in

sequential

computing

At

that

time

machine

architectures

were

designed

according

to

the

Von

Neumann

execution

model

a

processor

repeatedly

fetches

an

instruction

at

the

memory

address

stored

in

the

program

counter

-LRB-

PC

-RRB-

register

executes

it

and

finally

adjusts

the

contents

of

the

PC

register

for

the

next

cycle

In

this

execution

model

the

crucial

concept

is

the

ability

for

the

processor

to

store

an

arbitrary

value

into

the

PC

register

that

is

to

JUMP

The

programming

model

used

to

control

such

architectures

was

directly

derived

from

this

execution

model

the

JUMP

hardware

facility

is

simply

reflected

by

the

GOTO

instruction

in

the

Fortran

language

The

famous

question

of

Dijkstra

GOTO

considered

harmful

stated

in

-LSB-

-RSB-

was

the

starting

point

of

a

long

and

fruitful

effort

to

make

the

distinction

between

the

programming

model

and

the

execution

model

clearer

The

design

of

the

Pascal

language

originated

in

this

movement

It

proposed

an

alternative

programming

model

based

on

the

structured

while

construct

instead

of

the

low-level

unstructured

GOTO

Observe

that

this

new

programming

construct

does

not

change

the

execution

model

in

any

way

It

is

interesting

to

assess

this

important

step

with

respect

to

control

management

In

the

Fortran

programming

model

the

flow

of

control

is

arbitrary

and

unbounded

You

can

GOTO

from

anywhere

to

anywhere

Only

semantic

analysis

can

allow

one

to

discover

structures

in

the

control

flow

or

determine

the

bounds

of

a

GOTO

In

contrast

the

control

flow

of

the

Pascal

programming

model

is

structured

and

bounded

-LRB-

more

precisely

nested

-RRB-

Semantic

information

is

made

explicit

in

the

syntax

at

the

price

of

some

restriction

in

expressiveness

Let

us

illustrate

these

issues

with

the

following

example

It

is

easy

to

identify

an

iteration

controlled

by

a

test

in

Pascal

as

it

appears

explicitly

in

the

program

text

while

x

do

x

x

In

contrast

a

more

careful

analysis

is

required

to

determine

that

the

following

Fortran

program

IF

-LRB-

x

GE

-RRB-

GOTO

x

x

GOTO

CONTINUE

is

such

an

iteration

whereas

IF

-LRB-

x

GE

-RRB-

GOTO

x

x

GOTO

CONTINUE

boils

down

to

a

mere

alternative

if

x

then

x

x

and

IF

-LRB-

x

GE

-RRB-

GOTO

x

x

GOTO

CONTINUE

is

an

infinite

loop

In

this

respect

Pascal

can

be

qualified

as

more

informative

than

Fortran

On

the

other

hand

it

is

easy

to

translate

a

structured

construct

of

Pascal

such

as

a

while

repeat

or

for

loop

into

a

complex

assembly

of

GOTO

's

Besides

the

marginal

problem

of

label

generation

this

translation

involves

local

manipulations

only

-LRB-

it

is

called

compositional

-RRB-

In

contrast

it

is

much

more

difficult

to

translate

a

program

written

with

GOTO

's

into

a

program

written

with

while

loops

only

-LRB-

It

is

a

classical

and

pleasant

exercise

to

prove

that

any

program

can

be

transformed

into

a

program

made

of

only

one

while

loop

indeed

and

we

leave

it

to

the

interested

reader

-RRB-

A

global

manipulation

of

the

program

text

is

needed

In

this

respect

Fortran

can

be

qualified

as

more

expressive

than

Pascal

This

discussion

is

summed

up

in

Figure

Programming

model

GOTO

while

Language

Fortran

Pascal

Control

flow

arbitrary

unbounded

structured

bounded

-LRB-

nested

-RRB-

Structure

information

semantic

syntactic

Expressiveness

Informativeness

Figure

GOTO

considered

harmful

Our

thesis

is

that

a

similar

phenomenon

is

now

occurring

in

the

area

of

parallel

computing

In

the

CSP

programming

model

as

seen

in

the

Occam

language

parallel

composition

applies

to

arbitrary

programs

The

asynchrony

between

them

is

arbitrary

and

unbounded

The

resulting

behavior

is

non-deterministic

in

general

Any

information

concerning

synchronization

boundedness

of

scopes

and

determinism

has

to

be

derived

from

a

semantic

study

of

the

program

This

analysis

is

in

general

very

complex

Following

the

Fortran/Pascal

seminal

analogy

the

challenge

is

thus

to

design

a

programming

model

in

which

parallel

composition

is

structured

bounded

in

scope

and

deterministic

and

such

that

all

this

information

is

made

available

in

the

syntax

Paraphrasing

Dijkstra

we

dare

to

ask

the

question

PAR

considered

harmful

Consider

an

Occam

program

of

the

form

-LRB-

-RRB-

k

-LRB-

-RRB-

where

processes

Pi

and

Qj

may

be

very

complex

Because

of

the

unbounded

asynchrony

it

may

well

happen

that

the

left-hand

process

already

executes

while

the

right-hand

one

still

completes

In

some

situation

this

may

be

a

bad

thing

for

instance

if

produces

a

value

required

by

and

modifies

this

value

or

if

some

message

sent

by

is

caught

by

instead

of

being

delivered

to

as

intended

Of

course

some

semantic

information

based

on

the

necessary

synchronizations

between

and

may

help

us

to

rule

out

this

case

We

may

for

instance

prove

that

the

left-hand

process

can

not

enter

before

the

right-hand

one

has

left

The

challenge

is

here

to

make

this

information

explicit

in

the

syntax

Ultimately

this

amounts

to

writing

the

program

-LRB-

k

-RRB-

-LRB-

k

-RRB-

Such

a

syntax

is

more

restrictive

than

the

original

one

but

it

is

more

informative

It

is

the

compiler

's

job

to

detect

whether

a

program

written

in

this

restricted

but

informative

syntax

can

be

optimized

into

a

program

written

in

the

unrestricted

syntax

We

sum

up

this

discussion

in

Figure

Programming

model

PAR

Language

Occam

Control

flow

arbitrary

unbounded

non-deterministic

structured

bounded

deterministic

Paradigm

-LRB-

-RRB-

k

-LRB-

-RRB-

PAR

of

SEQ

-LRB-

k

-RRB-

-LRB-

k

-RRB-

SEQ

of

PAR

Structure

information

semantic

syntactic

Expressiveness

Informativeness

Figure

PAR

considered

harmful

In

the

left-hand

model

a

program

is

described

as

a

PARallel

composition

of

SEQuential

actions

Note

that

the

name

of

Hoare

's

CSP

language

-LSB-

-RSB-

stands

precisely

for

Communicating

Sequential

Processes

We

can

therefore

call

this

model

control-parallel

To

illustrate

the

application

of

this

model

consider

three

arrays

A

-LSB-

N

-RSB-

B

-LSB-

N

-RSB-

C

-LSB-

N

-RSB-

A

typical

control-parallel

program

to

compute

the

component-wise

sum

would

be

-LRB-

A

-LSB-

-RSB-

B

-LSB-

-RSB-

C

-LSB-

-RSB-

-RRB-

k

k

-LRB-

A

-LSB-

N

-RSB-

B

-LSB-

N

-RSB-

C

-LSB-

N

-RSB-

-RRB-

In

the

right-hand

model

a

program

is

viewed

as

a

SEQuential

composition

of

actions

on

PARallel

data

structures

We

can

therefore

call

this

model

data-parallel

A

typical

program

for

the

same

problem

would

be

the

following

using

the

parallel

extent

notation

of

Perrot

's

Actus

language

-LSB-

-RSB-

A

-LSB-

N

-RSB-

B

-LSB-

N

-RSB-

C

-LSB-

N

-RSB-

or

more

concisely

A

B

C

The

pros

and

cons

of

the

control-parallel

model

versus

the

data-parallel

model

can

be

discussed

along

the

same

lines

as

for

the

GOTO

versus

the

while

programming

models

Aspects

of

program

design

validation

transformation

optimization

and

maintenance

can

be

considered

in

turn

The

common

baseline

of

the

discussion

eventually

reduces

to

whether

to

choose

to

emphasize

flexibility

or

structure

expressivity

or

safety

short

term

decisions

or

long

term

decisions

etc.

Moreover

as

the

control-parallel

model

is

a

straightforward

abstraction

of

the

execution

model

it

is

useful

to

consider

the

relationships

between

the

two

models

in

term

of

compilation

and

optimization

It

is

much

easier

and

safer

for

the

programmer

to

work

in

the

structured

bounded

and

deterministic

data-parallel

programming

model

On

the

other

hand

it

is

much

easier

and

safer

for

the

compiler

to

work

with

a

program

expressed

in

the

control-parallel

model

The

discussion

is

thus

closely

dependent

on

the

balance

between

the

work

handled

by

the

compiler

and

that

left

to

the

programmer

These

issues

are

illustrated

in

Figure

reprinted

from

-LSB-

p.

-RSB-

Structured

Programming

Data

Parallel

Programming

Primitive

Computation

Assignment

statements

local

computations

Basic

Patterns

BEGIN

END

IF

THEN

ELSE

WHILE

DO

replication

reduction

permutation

parallel

prefix

Common

Compound

Patterns

DO

loops

CASE

statements

REPEAT

UNTIL

regular

grids

stencils

fetch-with-add

sorting

fast

transforms

Low-Level

Mechanisms

GOTO

message

passing

Figure

Structuring

programs

reprinted

from

the

CM

Technical

Summary

-LSB-

-RSB-

Data

Parallelism

as

a

Programming

Model

Data

parallelism

can

thus

be

thought

as

a

Paradigm

Revolution

with

respect

to

control

parallelism

The

essential

aspect

of

this

revolution

is

that

the

PAR

of

SEQ

execution

model

is

expressed

by

a

SEQ

of

PAR

programming

model

This

clear

distinction

between

programming

and

execution

models

is

the

source

of

an

enriched

vision

of

programs

In

fact

this

radical

breakthrough

can

already

be

seen

in

several

areas

of

parallel

programming

Below

we

shortly

describe

two

interesting

examples

where

data

parallelism

arises

as

the

natural

programming

model

associated

with

a

control-parallel

execution

model

Macroscopic

and

Microscopic

Viewpoints

The

data-parallel

programming

model

provides

two

alternative

viewpoints

for

a

single

program

-LSB-

-RSB-

The

existence

of

these

two

complementary

viewpoints

is

probably

one

of

the

reasons

for

the

growing

interest

in

the

data-parallel

model

the

intuitions

and

the

techniques

rooted

in

each

viewpoint

can

be

combined

to

design

manipulate

transform

validate

and

optimize

data-parallel

programs

providing

the

programmer

with

a

variety

of

powerful

tools

The

Macroscopic

Viewpoint

The

macroscopic

viewpoint

corresponds

to

the

high-level

vision

of

the

user

A

program

is

here

viewed

as

a

sequence

of

complex

operations

applied

on

data

structures

with

parallel

access

-LRB-

SEQ

of

PAR

-RRB-

Basically

this

is

thus

a

centralized

sequential

model

very

close

to

the

Von

Neumann

model

in

which

the

usual

scalar

data

are

replaced

with

vectors

that

is

arrays

with

parallel

access

or

collections

Seen

from

the

processor

a

memory

location

does

not

store

a

scalar

value

any

more

but

rather

a

parallel

array

of

values

This

viewpoint

can

thus

be

called

Processor

of

arrays

In

such

an

approach

the

notion

of

a

general

communication

amounts

to

a

global

rearrangement

of

the

elements

of

an

array

This

rearrangement

is

specified

by

an

auxiliary

array

of

coordinates

In

the

case

of

a

rearrangement

controlled

by

the

emitters

-LRB-

a

scatter

usually

called

a

send

communication

in

data-parallel

languages

-RRB-

each

component

of

this

auxiliary

array

stores

the

coordinates

of

the

destination

component

Observe

that

it

is

then

necessary

to

specify

the

intended

behavior

in

the

case

of

conflicts

on

the

destination

components

In

the

case

of

a

rearrangement

controlled

MemoryPro

cessor

Figure

The

macroscopic

viewpoint

a

processor

of

arrays

by

the

receivers

-LRB-

a

gather

usually

called

a

get

communication

in

data-parallel

languages

-RRB-

each

component

of

this

auxiliary

array

stores

the

coordinates

of

the

component

whose

value

is

requested

Observe

that

no

conflict

is

possible

here

A

subset

of

the

many

possible

rearrangements

can

be

specified

in

a

more

concise

way

These

regular

communication

schemes

often

support

optimized

implementations

depending

on

the

specific

features

of

the

target

architecture

-LRB-

especially

of

its

communication

network

-RRB-

Regular

communications

include

shifts

along

a

direction

toroidal

rotations

and

prefix

operations

usually

called

scans

Two

special

cases

of

scans

are

of

dramatic

importance

reduction

which

combines

several

values

into

one

variable

-LRB-

a

scatter

send-like

operation

-RRB-

and

its

dual

operation

broadcast

which

replicates

one

value

into

many

variables

-LRB-

a

gather

get-like

operation

-RRB-

Many

classical

algorithms

can

in

fact

be

expressed

as

a

combination

of

these

regular

communications

even

in

apparently

irregular

areas

Striking

examples

of

this

approach

applied

to

linked

lists

can

be

found

in

a

famous

paper

of

Steele

and

Hillis

-LSB-

-RSB-

In

fact

a

complete

methodology

of

programming

can

be

developed

with

these

only

communication

primitives

-LSB-

-RSB-

The

macroscopic

viewpoint

is

the

basis

for

data-parallel

array

manipulation

languages

We

must

of

course

cite

their

-LRB-

glorious

-RRB-

ancestors

APL

-LSB-

-RSB-

and

Actus

-LSB-

-RSB-

The

macroscopic

intuition

is

strongly

emphasized

in

the

definition

of

the

Fortran

language

and

of

HPF

In

particular

these

languages

define

specific

communication

instructions

for

each

of

the

regular

communication

schemes

above

The

next

step

forward

in

the

macroscopic

direction

is

to

view

an

array

with

parallel

access

as

an

abstract

function

from

some

domain

of

indices

to

some

domain

of

values

This

was

already

implicit

in

the

work

Steele

and

Hillis

about

CMLisp

-LSB-

-RSB-

The

reader

is

referred

to

the

contribution

of

Bjorn

Lisper

-LSB-

-RSB-

in

this

volume

for

an

extensive

presentation

of

data-parallel

programming

seen

as

a

special

case

of

functional

programming

The

Microscopic

Viewpoint

The

microscopic

viewpoint

corresponds

to

the

low-level

vision

of

the

compiler

The

program

is

seen

as

the

parallel

composition

of

actions

on

local

data

-LRB-

PAR

of

SEQ

-RRB-

with

an

additional

synchronization

mechanism

to

keep

these

independent

threads

of

actions

synchronous

This

approach

can

thus

be

called

Array

of

processors

On

SIMD

architectures

this

synchronization

mechanism

is

implemented

at

the

hardware

level

It

consists

of

a

global

sequencer

distinct

from

the

processing

elements

-LRB-

PE

-RRB-

Its

function

is

to

broadcast

the

current

common

instruction

to

be

executed

by

the

PE

on

their

local

data

at

each

cycle

On

MIMD

architectures

this

synchronization

mechanism

is

implemented

at

the

software

level

It

turns

out

that

data-parallel

programs

are

often

self-synchronizing

nothing

needs

to

be

done

in

this

case

and

the

synchronization

mechanism

is

void

-LRB-

see

-LSB-

-RSB-

for

a

detailed

study

of

such

a

property

-RRB-

If

the

program

does

not

satisfy

this

property

or

if

the

compiler

is

not

clever

enough

to

detect

it

then

it

has

to

insert

between

each

computation

phase

a

call

to

a

synchronization

routine

Such

a

routine

implements

a

global

synchronization

protocol

based

on

a

general

exchange

of

messages

and

it

may

be

very

inefficient

for

example

this

requires

of

the

order

of

one

millisecond

on

an

iPSC

with

processors

Therefore

several

recent

MIMD

architectures

explicitly

designed

to

support

dataparallel

languages

-LRB-

HPF

and

C

usually

-RRB-

have

been

equipped

with

specific

hardware

mechanism

to

optimize

this

function

It

is

in

particular

the

case

with

the

-LSB-

-RSB-

of

Thinking

Machine

and

the

Paragon

-LSB-

-RSB-

of

Intel

MIMD

architectures

with

an

optimized

global

synchronization

hardware

facility

are

sometimes

-LRB-

somewhat

improperly

see

below

-RRB-

called

SPMD

-LRB-

Single

Program

Multiple

Data

-RRB-

as

they

are

specifically

designed

to

execute

data-parallel

languages

efficiently

A

better

nickname

would

probably

be

C

machines

or

HPF-machines

very

much

like

the

Lispmachines

of

the

's

In

contrast

to

the

general

MIMD

model

all

communication

phases

are

synchronous

in

dataparallel

programs

On

entering

a

communication

phase

all

processors

are

ready

to

send

and

then

ready

to

receive

all

available

messages

The

communication

phase

ends

only

when

all

sent

messages

have

been

received

There

is

thus

no

deadlock

to

threaten

as

in

the

CSP

point-to-point

communication

model

In

the

microscopic

viewpoint

of

data-parallel

communication

the

basic

mode

is

the

all-to-all

message

exchange

In

the

emitter-controlled

communication

flavor

-LRB-

scatter

send

-RRB-

each

processor

specifies

the

destination

address

for

its

message

In

the

receiver-controlled

communication

flavor

-LRB-

gather

get

-RRB-

each

processor

specifies

the

source

address

for

its

message

These

addresses

are

specified

with

an

auxiliary

parallel

variable

As

was

the

case

for

the

macroscopic

viewpoint

a

subset

of

regular

communication

schemes

can

be

distinguished

and

given

an

optimized

implementation

The

microscopic

viewpoint

is

specially

promoted

by

languages

such

as

C

MPL

and

HyperC

Usually

these

languages

do

not

provide

any

specific

syntactic

construct

to

identify

regular

communication

schemes

An

exception

is

the

MPL

xnet

construct

which

is

very

closely

connected

to

the

hardware

features

of

the

underlying

interconnection

network

Yet

this

information

can

in

certain

cases

be

recognized

by

the

compiler

from

the

program

text

and

then

compiled

to

calls

to

specific

optimized

procedures

Compilers

that

can

perform

such

recognition

are

often

called

communication

compilers

Regular

communication

operations

can

also

be

specified

by

the

programmer

through

directives

or

explicit

calls

to

specialized

communication

libraries

The

lack

of

any

explicit

syntactic

constructs

to

express

regular

communications

is

probably

a

major

weakness

of

these

languages

Sequencer

Network

of

inteconnected

processing

elements

Figure

The

microscopic

viewpoint

an

array

of

processors

Automatic

Parallelization

Several

methods

have

been

proposed

for

automatic

parallelization

of

Fortran

scientific

programs

on

MIMD

parallel

machines

with

distributed

memory

A

number

of

authors

have

already

noticed

that

data

parallelism

provides

an

adequate

framework

for

this

purpose

as

data

parallelism

is

in

some

sense

hidden

behind

control

parallelism

in

the

methodology

We

follow

here

the

general

lines

of

the

presentation

of

Feautrier

-LSB-

-RSB-

The

reader

is

referred

to

Feautrier

's

contribution

in

this

volume

for

further

details

-LSB-

-RSB-

Many

studies

on

Fortran

scientific

programs

have

shown

that

the

core

of

such

programs

consists

of

sequences

and

nestings

of

DO-loops

with

simple

-LRB-

e.g.

affine

-RRB-

index

dependencies

For

example

DO

i

M

DO

j

N

-LRB-

i

-RRB-

A

-LSB-

i

j

-RSB-

A

-LSB-

f

-LRB-

i

j

-RRB-

g

-LRB-

i

j

-RRB-

-RSB-

where

N

-LRB-

i

-RRB-

f

-LRB-

i

j

-RRB-

and

g

-LRB-

i

j

-RRB-

are

simple

functions

When

parallelizing

these

programs

the

first

step

is

to

analyze

dependencies

that

is

to

compute

-LRB-

symbolically

-RRB-

a

graph

made

of

edges

-LRB-

i

j

-RRB-

-LRB-

f

-LRB-

i

j

-RRB-

g

-LRB-

i

j

-RRB-

-RRB-

Then

each

occurrence

-LRB-

i

j

-RRB-

is

allocated

on

a

processor

Pu

-LRB-

i

j

-RRB-

Finally

the

parallelized

program

is

generated

as

a

PARallel

composition

of

SEQuential

processes

according

to

the

intended

execution

model

The

parallelized

program

is

usually

of

the

following

form

P

-LSB-

k

k

where

each

process

Pu

is

in

charge

of

computing

variables

A

-LSB-

i

j

-RSB-

allocated

to

it

This

strategy

is

called

the

Owner

Computes

Rule

-LRB-

OCR

-RRB-

Each

process

Pu

has

the

following

form

Pu

tu

while

oku

do

Communicationu

Local-computationu

tu

tu

Synchronizationu

end

The

local

variable

tu

stores

the

iteration

number

The

local

variable

oku

is

set

to

false

when

the

computation

is

locally

terminated

the

parallelization

method

guarantees

that

this

occurs

at

the

same

iteration

number

for

all

processes

In

the

program

part

Communicationu

processes

exchange

the

data

required

to

the

current

iteration

New

data

are

computed

and

assigned

locally

in

Local-computationu

The

Synchronizationu

part

implements

a

global

synchronization

protocol

in

order

to

keep

everybody

in

phase

As

we

explain

below

this

synchronization

can

often

be

eliminated

in

an

optimization

step

Looking

once

again

at

the

general

shape

of

the

program

it

is

clear

that

the

Synchronization

part

can

be

more

simply

expressed

by

commuting

the

PAR

and

the

SEQ.

Also

local

variables

tu

and

oku

can

be

merged

respectively

as

single

parallel

variables

t

and

ok

These

remarks

lead

to

the

following

form

for

the

whole

program

-LRB-

We

write

Cu

for

Communicationu

and

Lu

for

Local-computationu

-RRB-

t

while

ok

do

-LRB-

k

-LRB-

Cu

Lu

-RRB-

k

k

-LRB-

Cv

Lv

-RRB-

k

:-RRB-

t

t

end

Observe

now

that

Lu

contains

local

computations

only

Therefore

we

can

safely

introduce

an

additional

global

synchronization

between

the

C

's

and

L

's

The

inner

part

can

thus

be

written

-LRB-

k

Cu

k

k

Cv

k

:-RRB-

-LRB-

k

Lu

k

k

Lv

k

:-RRB-

Now

processes

can

test

their

own

identity

Say

it

is

stored

in

some

special

parallel

variable

-LRB-

it

was

called

this

in

the

old

C

now

called

Data-Parallel

C

-LSB-

-RSB-

-RRB-

There

is

thus

no

need

to

assume

that

programs

for

Pu

and

Pv

have

distinct

syntactic

forms

and

we

can

write

t

while

ok

do

-LRB-

k

C

k

k

C

k

:-RRB-

-LRB-

k

L

k

k

L

k

:-RRB-

t

t

end

or

more

concisely

t

while

ok

do

-LRB-

C

L

t

t

-RRB-

end

that

is

as

a

typical

data-parallel

program

The

data-parallel

SEQ

of

PAR

model

appears

thus

here

as

the

natural

framework

in

which

this

systematic

parallelization

can

be

carried

out

and

studied

In

fact

using

this

level

of

abstraction

allows

one

to

concentrate

on

extracting

data

dependencies

in

the

programming

model

without

any

reference

to

process

synchronization

problems

introduced

by

the

underlying

execution

model

The

problem

of

how

much

synchronization

is

needed

between

processes

to

respect

the

semantic

correctness

of

the

program

is

then

seen

as

an

optimization

question

which

is

raised

at

a

level

lower

than

that

of

the

intrinsic

correctness

of

the

method

CSP

Communication-Closed

Layers

A

dozen

years

ago

Elrad

and

Francez

-LSB-

-RSB-

performed

an

extensive

semantic

study

on

the

correctness

of

certain

design

methodologies

for

programs

written

in

Hoare

's

CSP

language

-LSB-

-RSB-

This

work

applies

to

Occam

programs

as

well

In

essence

their

work

addressed

the

following

problem

Consider

a

CSP

PARallel

program

of

the

form

-LRB-

-RRB-

k

-LRB-

-RRB-

Under

which

condition

is

it

semantically

equivalent

to

the

following

SEQuential

program

-LRB-

k

-RRB-

-LRB-

k

-RRB-

In

the

terminology

of

this

paper

under

which

condition

can

a

control-parallel

program

be

recast

into

a

data-parallel

form

To

get

a

feeling

for

the

subtlety

of

the

question

consider

the

following

CSP

programs

P

-LSB-

c

x

skip

true!skip

-RSB-

Q

-LSB-

c

skip

true!skip

-RSB-

and

take

P

Q.

Intuitively

P

says

I

am

ready

to

receive

a

message

from

Q

or

to

terminate

and

Q

says

I

am

ready

to

send

a

message

to

P

or

to

terminate

and

the

underlying

communication

protocol

ensures

that

Q

can

send

the

message

if

and

only

if

P

receives

it

-LRB-

synchronous

handshake

-RRB-

It

is

clear

that

P

k

Q

always

terminates

But

-LRB-

-RRB-

k

-LRB-

-RRB-

is

not

semantically

equivalent

to

-LRB-

k

-RRB-

-LRB-

k

-RRB-

as

a

communication

between

and

may

occur

in

the

former

and

not

in

the

latter

according

to

the

the

semantics

of

CSP

A

program

-LRB-

k

-RRB-

such

that

-LRB-

-RRB-

k

-LRB-

-RRB-

is

semantically

equivalent

to

-LRB-

k

-RRB-

-LRB-

k

-RRB-

for

any

program

-LRB-

k

-RRB-

is

called

by

Elrad

and

Francez

a

communication-closed

layer

As

observed

by

the

authors

designing

a

CSP

parallel

program

as

a

sequence

of

communication-closed

layers

allows

one

to

validate

each

layer

in

isolation

and

yet

to

compile

the

entire

program

without

any

additional

global

synchronization

This

is

therefore

a

highly

desirable

property

often

referred

as

modularity

or

compositionality

Observe

also

that

adding

a

synchronization

barrier

at

the

end

of

a

CSP

program

makes

it

communication-closed

at

the

cost

of

an

additional

global

communication

Basic

Semantic

Concepts

for

Data

Parallelism

As

shown

above

the

data-parallel

programming

model

can

be

characterized

by

the

SEQ

of

PAR

paradigm

that

is

a

sequential

composition

of

actions

on

objects

with

parallel

access

In

this

section

we

shortly

describe

five

fundamental

semantic

concepts

related

to

this

paradigm

All

of

them

can

be

found

more

or

less

explicitly

in

programming

languages

such

as

C

-LSB-

-RSB-

for

the

-LSB-

-RSB-

and

-LSB-

-RSB-

MPL

-LSB-

-RSB-

for

the

MasPar

-LSB-

-RSB-

HyperC

-LSB-

-RSB-

a

language

originally

designed

for

the

POMP

massively

parallel

SIMD

machine

-LSB-

-RSB-

Fortran

and

its

derived

companions

CM

Fortran

-LSB-

-RSB-

for

the

and

MP-Fortran

for

the

MasPar

-LSB-

-RSB-

and

the

recent

High

Performance

Fortran

HPF

-LSB-

-RSB-

The

reader

interested

in

a

more

detailed

presentation

together

with

a

formal

semantics

is

referred

to

-LSB-

-RSB-

and

the

other

contribution

in

this

volume

-LSB-

-RSB-

Conceptually

a

parallel

variable

X

is

made

of

a

fixed

number

of

components

A

parallel

access

to

a

parallel

variable

yields

a

parallel

value

We

are

not

concerned

here

with

the

spatial

structure

of

these

variables

-LRB-

layout

shape

geometry

etc.

-RRB-

or

with

their

physical

implementation

-LRB-

virtual

processors

etc.

-RRB-

We

rather

concentrate

on

the

various

ways

in

which

parallel

variables

are

handled

Data-Parallel

Assignment

A

data-parallel

assignment

X

E

synchronously

assigns

the

components

of

a

parallel

value

to

the

respective

components

of

parallel

variable

X

Values

are

computed

from

a

parallel

expression

E

where

the

semantics

of

usual

operators

is

extended

component-wise

No

inter-component

effect

occurs

Also

observe

that

expression

E

can

be

arbitrarily

complex

and

involve

tests

on

a

parallel

variable

say

this

whose

value

at

each

component

is

precisely

the

index

of

this

component

One

can

thus

define

a

single

expression

whose

behavior

is

arbitrarily

different

at

each

component

switch

-LRB-

this

-RRB-

f

case

do

something

case

do

something

else

g

The

data-parallel

assignment

concept

is

thus

sufficiently

general

to

account

for

what

is

often

called

SPMD

programming

-LRB-

Single

Program

M

ultiple

Data

-RRB-

Observe

this

terminology

is

a

somewhat

improper

generalization

of

the

SIMD

term

In

the

SPMD

term

the

word

Program

refers

to

syntax

this

simply

expresses

that

the

program

text

is

identical

for

all

processes

without

any

restriction

for

their

respective

dynamic

behaviors

This

is

thus

more

related

to

software

engineering

than

with

semantics

In

contrast

in

the

SIMD

term

the

word

Instruction

refers

primarily

to

an

identical

dynamic

behavior

of

all

processes

This

is

thus

much

stronger

General

Data-Parallel

Communication

At

the

conceptual

level

of

this

presentation

a

general

communication

is

deeply

different

from

a

component-wise

computation

It

consists

in

rearranging

the

components

of

a

parallel

variable

according

to

a

given

addressing

It

is

thus

related

to

the

distribution

of

the

components

among

the

nodes

of

the

network

without

regard

to

their

values

In

contrast

a

local

computation

is

related

to

the

respective

values

of

the

components

without

regard

to

their

location

One

can

therefore

say

that

these

two

concepts

are

dual

Communication

or

more

precisely

rearrangement

should

thus

be

seen

as

a

first-class

concept

in

the

model

It

should

be

considered

as

an

instruction

distinct

from

the

synchronous

parallel

assignment

though

it

is

presented

in

most

data-parallel

languages

as

a

special

case

of

expression

-LRB-

the

router

and

xnet

notation

of

MPL

the

left-hand

bracketing

of

parallel

variables

of

C

etc.

-RRB-

This

aspect

deserves

careful

discussion

as

it

has

dramatic

consequences

for

the

design

of

dataparallel

languages

The

structure

of

current

parallel

machines

is

an

assembly

of

powerful

generalpurpose

processors

-LRB-

Sparc

Alpha

etc.

-RRB-

interconnected

by

a

network

This

network

is

often

comparatively

slow

This

description

includes

both

real

parallel

machines

and

virtual

ones

-LRB-

more

or

less

integrated

networks

of

workstations

-RRB-

A

communication

operation

can

then

be

more

than

times

more

expensive

than

a

local

arithmetic

operation

It

is

thus

desirable

that

programmers

specify

communications

as

explicitely

as

possible

so

that

they

are

encouraged

to

avoid

them

-LRB-

or

at

least

become

aware

of

the

reasons

for

poor

performance

-RRB-

The

recent

history

of

parallel

programming

languages

provides

a

striking

illustration

of

the

imprortance

of

making

communication

operations

explicit

In

the

framework

of

the

project

Thinking

Machines

designed

a

language

called

C

a

data-parallel

extension

of

C.

-LRB-

This

language

is

now

called

Data-Parallel

C

DPC

-LSB-

-RSB-

-RRB-

In

this

innovative

design

the

concepts

of

rearrangement

and

local

computations

were

unified

by

using

-LRB-

pseudo

-RRB-

pointers

in

the

C++

spirit

A

component

of

a

variable

value

is

accessed

through

an

indirection

Pseudo-pointer

this

gives

access

to

the

local

component

this

to

the

component

of

the

neighbor

etc.

Thus

the

statement

this

x

-LRB-

this

x

-RRB-

performs

a

local

incrementation

of

each

component

of

parallel

variable

x

whereas

the

statement

this

x

-LRB-

this

-RRB-

x

stores

into

the

local

component

of

parallel

variable

x

the

value

of

the

neighbor

one

shift

distant

A

difficulty

with

this

notation

is

that

the

latter

operation

may

be

or

times

more

expensive

than

the

former

It

is

therefore

very

difficult

to

compile

such

a

language

as

it

is

impossible

to

obtain

statically

any

reasonable

prediction

concerning

the

value

of

the

pseudo-pointers

and

thus

to

decide

whether

an

access

is

local

or

remote

It

is

also

very

difficult

to

estimate

program

performance

by

inspecting

its

text

In

order

to

get

good

performances

the

programmer

is

led

to

enrich

the

program

text

with

comments

and/or

macro

definitions

so

as

to

tag

the

potentially

expensive

parts

which

adds

an

extra

level

of

difficulty

to

parallel

programming

The

language

designers

have

therefore

produced

a

new

version

of

the

language

which

is

the

one

currently

known

under

the

C

name

In

this

new

version

communications

are

made

explicit

in

the

syntax

Accessing

a

non-local

component

of

a

parallel

variable

is

denoted

by

left-hand

bracketing

-LSB-

-RSB-

x

The

C

language

allows

-LRB-

and

even

encourages

-RRB-

assignment

expressions

like

y

-LRB-

x

-RRB-

A

natural

extension

of

this

feature

is

to

introduce

rearrangement

as

expressions

too

y

-LSB-

-RSB-

x

Here

again

we

can

witness

a

trade-off

between

expressiveness

and

informativeness

in

language

design

The

first

version

of

C

was

quite

expressive

but

not

informative

enough

the

language

was

very

difficult

to

use

in

practice

for

large

software

development

The

new

version

of

C

enhances

informativeness

at

the

price

of

a

slightly

more

complex

syntax

Another

important

aspect

of

data-parallel

language

design

is

the

conceptual

nature

of

the

rearrangement

specification

At

the

level

of

the

execution

models

one

often

finds

two

distinct

communication

primitives

send

and

get

or

in

the

Fortran

terminology

for

vector

architectures

scatter

and

gather

Their

precise

relationship

is

strongly

dependent

on

the

specific

features

of

the

underlying

architecture

ffl

On

a

a

get

operation

is

twice

as

expensive

as

a

send

operation

A

get

is

in

fact

implemented

as

two

waves

of

send

operations

the

first

conveys

the

requests

of

the

receiving

processors

to

the

processors

owning

the

values

the

second

brings

the

values

back

to

the

requesting

processors

ffl

In

contrast

on

the

architecture

these

two

operations

have

similar

costs

In

both

cases

a

circuit

has

to

be

set

up

between

the

requester

and

the

owner

Observe

that

several

successive

circuits

may

have

to

be

set

up

in

case

of

access

conflicts

the

cost

of

the

communication

strongly

depends

on

the

communication

pattern

Today

many

parallel

architectures

use

interconnection

networks

in

which

each

message

is

independently

routed

This

is

in

particular

the

case

for

virtual

parallel

architectures

built

on

packetswitched

interconnection

networks

-LRB-

TCP/IP

-RRB-

In

this

case

send

operations

are

much

cheaper

than

get

operations

and

very

complex

methods

have

been

proposed

to

compile

sequences

of

high-level

get

instructions

into

low-level

send

operations

-LRB-

see

for

instance

-LSB-

-RSB-

-RRB-

The

situation

may

nevertheless

evolve

dramatically

within

the

coming

years

with

the

advent

of

new

network

technologies

it

is

very

likely

that

the

circuit-switched

ATM

networks

will

require

specific

optimizations

different

from

the

current

packet-switched

TCP/IP

networks

In

contrast

at

the

level

of

programming

models

the

balance

seems

to

be

reversed

In

effect

the

get

primitive

enjoys

a

much

simpler

behavior

than

send

ffl

The

get

primitive

does

not

generate

any

write

conflict

each

component

receives

at

most

one

value

In

the

case

of

a

send

a

single

component

may

be

selected

as

the

target

by

many

components

It

is

thus

necessary

for

the

programmer

to

specify

the

merging

protocol

explicitly

combining

the

value

through

some

associative

commutative

operator

with

a

neutral

element

-LRB-

min

max

logical

conjunction

and

disjunction

etc.

-RRB-

ffl

The

get

primitive

allows

each

component

to

be

aware

of

its

own

state

If

a

component

does

not

require

any

value

then

its

internal

state

is

not

modified

by

the

instruction

In

contrast

the

send

operation

does

not

respect

this

basic

intuition

a

component

may

be

selected

as

the

target

of

a

rearrangement

independently

of

its

own

behavior

In

a

formal

semantic

framework

these

issues

can

be

expressed

by

stating

that

the

get

primitive

respects

a

fundamental

invariant

a

component

is

modified

only

if

it

so

requests

This

invariant

is

not

respected

by

the

send

primitive

In

this

respect

it

make

sense

to

consider

get-based

dataparallel

programming

methodologies

as

safer

than

send-based

ones

even

though

the

former

turns

out

to

yield

less

efficient

performances

on

most

currently

available

architectures

Synchronous

Parallel

Control

As

far

as

the

programming

model

is

concerned

the

flow

of

control

P

Q

is

completely

synchronous

Each

assignment

is

processed

globally

in

turn

Of

course

it

is

likely

that

this

is

not

the

case

in

the

execution

model

-LRB-

think

of

a

network

of

asynchronous

MIMD

processors

-RRB-

It

is

up

to

the

compiler

to

guarantee

that

the

asynchronous

implementation

of

a

data-parallel

program

is

correct

with

respect

to

the

synchronous

semantics

The

parallel

iteration

while

B

do

P

end

raises

a

problem

It

is

clear

that

it

exits

as

soon

as

all

relevant

components

of

the

boolean

parallel

expression

B

evaluate

to

false

But

does

the

iteration

exit

locally

at

a

component

where

B

evaluates

to

false

Because

of

the

send

semantics

it

can

happen

that

the

value

of

B

changes

at

one

component

even

though

no

assignment

has

been

done

on

this

component

The

answer

to

this

subtle

question

varies

from

language

to

language

For

instance

the

answer

is

yes

for

MPL

and

it

is

no

for

HyperC

The

interested

reader

will

find

in

-LSB-

-RSB-

a

detailed

study

of

the

semantic

consequences

of

such

design

choices

Bounded

Asynchrony

All

of

the

concepts

detailed

above

induce

a

fully

synchronous

behavior

on

parallel

components

It

is

thus

important

to

add

on

top

of

them

an

orthogonal

concept

to

cater

for

a

bounded

asynchrony

This

concept

is

usually

called

where

In

a

where

B

do

P

end

program

all

components

for

which

the

parallel

boolean

expression

B

evaluates

to

true

execute

program

P

synchronously

whereas

the

remaining

ones

skip

it

Again

observe

that

this

construct

does

not

say

anything

about

how

the

execution

model

will

implement

such

a

semantic

specification

In

a

SIMD

execution

model

the

components

where

B

evaluates

to

false

will

remain

idle

waiting

for

the

other

components

to

proceed

through

P

In

contrast

in

a

MIMD

execution

model

the

compiler

will

seek

to

transform

the

code

so

that

the

idle

components

can

safely

initiate

the

rest

of

the

program

as

soon

as

possible

This

amounts

to

checking

that

the

implicit

synchronization

barrier

at

the

end

of

the

where

block

can

be

removed

without

modifying

the

overall

semantics

of

the

program

It

is

interesting

to

note

that

the

where

construct

respects

the

fundamental

invariant

mentioned

above

once

a

component

has

turned

idle

on

entering

a

where

block

its

internal

state

is

not

modified

until

the

where

block

is

exited

Indeed

nesting

where

blocks

may

only

restrict

activity

Yet

this

is

not

the

case

in

presence

of

some

unconditional

activation

constructs

such

as

all

in

MPL

and

everywhere

in

C

In

this

case

an

idle

component

may

be

temporarily

re-activated

This

makes

reasoning

about

programs

more

difficult

very

much

as

the

escape

constructs

break

and

continue

found

in

the

C

language

make

it

more

difficult

to

reason

about

loops

than

in

Pascal

In

this

respect

it

makes

sense

to

state

that

using

unconditional

all

or

everywhere

constructs

within

where

blocks

is

not

good

programming

practice

and

should

be

avoided

as

much

as

possible

Conclusion

This

paper

describes

a

preliminary

attempt

towards

an

abstract

presentation

of

data

parallelism

We

have

mostly

discussed

the

aspects

related

to

control

that

is

logical

correctness

of

programs

But

many

other

aspects

are

of

dramatic

importance

for

the

efficient

use

of

such

languages

A

similar

study

should

be

performed

concerning

data

distribution

and

communication

compiling

The

interested

reader

will

find

in

-LSB-

-RSB-

an

early

presentation

of

these

problems

A

recent

synthesis

can

be

found

in

-LSB-

-RSB-

We

could

dare

to

say

that

parallel

computing

is

still

in

its

infancy

This

judgment

has

to

be

understood

in

the

sense

that

there

is

not

yet

a

clear

distinction

between

the

execution

model

and

the

programming

model

This

distinction

took

many

years

to

clarify

in

the

area

of

scalar

sequential

computing

and

we

can

see

now

that

there

is

no

fixed

connection

between

architectures

and

languages

in

the

sequential

context

It

is

considered

as

a

desirable

feature

that

all

languages

be

available

on

all

-LRB-

most

-RRB-

architectures

This

effort

has

paved

the

way

for

the

emergence

of

programming

models

with

no

immediate

operational

counterparts

functional

programming

logic

programming

some

aspects

of

object-oriented

programming

etc.

A

similar

awareness

is

now

slowly

emerging

in

parallel

computing

-LSB-

-RSB-

In

some

sense

the

emergence

of

PVM

and

now

MPI

is

a

first

step

in

this

direction

at

least

a

standard

low-level

programming

model

for

MIMD

Distributed

Memory

-LRB-

MIMD-DM

-RRB-

architectures

is

now

widely

accepted

But

this

programming

model

is

still

very

close

to

the

execution

model

very

much

as

the

Fortran

programming

model

is

close

to

the

Von

Neumann

execution

model

There

is

still

a

long

way

to

go

before

we

achieve

a

complete

and

clear

separation

between

parallel

architectures

and

parallel

languages

so

that

each

can

freely

develop

according

to

its

own

philosophy

We

have

tried

to

demonstrate

that

data

parallelism

is

a

first

step

in

this

direction

The

emergence

of

abstract

parallel

languages

may

probably

be

the

next

step

the

Linda

language

among

many

others

suggests

an

interesting

direction

-LSB-

-RSB-

Another

remark

is

that

such

a

separation

is

desirable

only

if

compilers

are

intelligent

enough

to

fill

the

gap

between

the

programming

model

and

the

execution

model

We

would

even

say

intelligent

enough

to

use

this

gap

to

provide

users

with

an

implementation

better

than

what

they

themselves

could

have

coded

by

hand

A

crucial

milestone

on

the

way

is

thus

parallel

compiler

technology

This

is

already

a

noticeable

trend

It

is

not

so

difficult

today

to

build

a

Teraflops

machine

within

a

couple

of

years

Many

companies

and

academic

research

centers

have

already

announced

such

architectures

The

real

Grand

Challenge

is

to

build

compilers

to

use

their

power

-LSB-

-RSB-

Today

nobody

would

accept

to

program

a

workstation

in

assembly

language

because

it

is

too

complex

and

wasteful

programming

in

C

is

much

easier

and

often

leads

to

better

overall

performance

The

real

power

comes

from

the

compiler

The

data-parallel

programming

model

appears

as

a

fundamental

model

for

massively

parallel

programming

mainly

oriented

towards

regular

applications

It

allows

programmers

to

reason

about

their

program

according

to

the

usual

sequential

model

without

struggling

with

the

combinatorial

explosion

inherent

in

the

potential

interactions

between

thousands

of

asynchronous

processes

Moreover

the

sequential

nature

of

the

programming

model

enables

one

to

import

many

software

engineering

methods

originally

designed

for

scalar

languages

such

as

Fortran

Pascal

C

etc.

programming

environments

for

program

design

validation

compilation

optimization

monitoring

debugging

etc.

These

methods

simply

have

to

be

extended

to

handle

arrays

as

first-class

citizen

One

of

the

main

reasons

for

the

relative

lack

of

success

of

control-parallel

languages

such

as

Occam

was

the

lack

of

any

programming

environment

after

a

first

phase

of

enthusiasm

and

low-level

hacking

a

language

can

not

survive

on

a

large

scale

without

such

tools

In

contrast

Thinking

Machines

and

MasPar

could

announce

within

one

year

the

availability

of

powerful

programming

environments

-LRB-

Prism

for

TMC

MPPE

for

MasPar

-RRB-

They

are

in

fact

based

mainly

on

similar

tools

for

the

C

language

-LRB-

dbxtool

C

profiler

etc.

-RRB-

Of

course

the

counterpart

is

the

difficulty

of

designing

an

efficient

compiler

for

these

languages

on

MIMD

architectures

Finally

it

is

important

to

stress

that

data

parallelism

and

control

parallelism

are

not

mutually

exclusive

The

two

models

can

be

usefully

combined

They

are

rather

two

complementary

programming

models

The

former

is

a

high-level

model

and

the

latter

is

a

low-level

model

Just

as

it

can

be

useful

to

mix

C

and

an

assembly

language

to

get

really

high

performances

in

a

sequential

language

one

can

use

a

data-parallel

procedure

within

a

control-parallel

harness

or

a

control-parallel

program

within

a

data-parallel

harness

The

C

programming

environment

explicitly

supports

the

latter

approach

the

programmer

can

make

holes

in

a

data-parallel

program

where

the

lowlevel

message-passing

library

can

be

directly

activated

The

HyperC

and

the

HPF

programming

models

are

also

specifically

extended

in

this

direction

The

interested

reader

is

referred

to

the

contribution

of

Ian

Foster

in

this

volume

-LSB-

-RSB-

for

a

detailed

discussion

on

mixing

the

two

models

in

high-performance

languages

At

this

time

very

little

verification

can

be

done

by

the

compiler

about

the

actual

correctness

of

the

interaction

between

the

two

models

This

is

at

the

programmer

's

own

risk

just

as

a

C

compiler

does

not

verify

the

low-level

manipulations

specified

by

an

asm

instruction

Also

little

guideline

exists

to

assist

the

programmer

in

mixing

the

two

models

the

only

criterion

is

still

to

use

the

right

model

to

do

the

right

thing

Mastering

such

heterogeneous

parallel

programming

models

will

probably

become

a

major

research

challenge

in

the

coming

years

Acknowledgments

This

work

is

the

result

of

many

discussions

with

the

members

of

the

ParaDigme

project

It

has

also

been

substantially

improved

by

the

remarks

of

Ian

Foster

and

Sanjay

Rajopadhye

Their

contribution

is

gratefully

acknowledged

References

-LSB-

-RSB-

C.

Bareau

B.

Caillaud

C.

Jard

and

R.

Thoraval

Correctness

of

automated

distribution

of

sequential

programs

In

Parallel

Architectures

and

Languages

Europe

-LRB-

PARLE

-RRB-

volume

of

Lecture

Notes

in

Computer

Science

pages

-LCB-

-LSB-

-RSB-

T.

Blank

The

MasPar

architecture

In

Proc

of

the

IEEE

Computer

Society

Int

Conf.

Spring

COMPCON

pages

-LCB-

San

Francisco

-LSB-

-RSB-

G.E.

Blelloch

Vector

models

for

data-parallel

computing

MIT

Press

Cambridge

-LSB-

-RSB-

L.

Boug?e

D.

Cachera

Y.

Le

Guyadec

G.

Utard

and

B.

Virot

The

Data

Parallel

Programming

Model

chapter

Formal

validation

of

data-parallel

programs

introducing

the

assertional

approach

pages

-LCB-

Lecture

Notes

in

Computer

Science

Tutorial

Springer

Verlag

-LSB-

-RSB-

L.

Boug?e

On

the

semantics

of

languages

for

massively

parallel

SIMD

architectures

-LRB-

extended

abstract

-RRB-

In

Proc

Parallel

Arch

and

Lang

Europe

Conf.

PARLE

vol

number

in

Lect

Notes

Comp

Science

pages

-LCB-

Eindhoven

Springer

-LSB-

-RSB-

L.

Boug?e

and

J.-L

Levaire

Control

structures

for

data-parallel

SIMD

languages

semantics

and

implementation

Future

Generation

Computer

Systems

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

D.

Callahan

and

K.

Kennedy

Compiling

programs

for

distributed-memory

multiprocessors

Journal

of

Supercomputing

-LCB-

-LSB-

-RSB-

N.

Carreiro

and

D.

Gelertner

Linda

in

context

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

E.W.G.

Dijkstra

GO

TO

statement

considered

harmful

Communications

of

the

ACM

-LCB-

-LSB-

-RSB-

Tz

Elrad

and

N.

Francez

Decomposition

of

distributed

programs

into

communication-closed

layers

Science

of

Computer

Programming

-LCB-

-LSB-

-RSB-

R.

Esser

and

R.

Knecht

Intel

Paragon

XP/S

-LCB-

architecture

and

software

environment

Technical

Report

Central

Institute

for

Mathemantics

Research

Center

Juelich

-LSB-

-RSB-

P.

Feautrier

Asymptotically

efficient

algorithms

for

parallel

architectures

In

Decentralized

Systems

pages

-LCB-

IFIP

Working

Group

North-Holland

-LSB-

-RSB-

Paul

Feautrier

The

Data

Parallel

Programming

Model

chapter

Automatic

Parallelization

in

the

Polytope

Model

pages

-LCB-

Lecture

Notes

in

Computer

Science

Tutorial

Springer

Verlag

-LSB-

-RSB-

M.J.

Flynn

Some

computer

organizations

and

their

effectiveness

IEEE

Trans

on

Computers

C

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

High

Performance

Fortran

Forum

High

Performance

Fortran

language

specification

-LRB-

draft

version

-RRB-

CITI/CRPC

Rice

Univ.

Houston

January

Version

Draft

-LSB-

-RSB-

Ian

Foster

Designing

and

Building

Parallel

Programs

Addison-Wesley

-LSB-

-RSB-

Ian

Foster

The

Data

Paralle

Programming

Model

chapter

Task

Parallelism

and

High-Performance

Languages

pages

-LCB-

Lecture

Notes

in

Computer

Science

Toturial

Springer

Verlag

-LSB-

-RSB-

Ph.J

Hatcher

and

M.

Quinn

DataParallel

C

Compiling

DataParallel

Languages

on

MIMD

architectures

MIT

Press

-LSB-

-RSB-

W.D.

Hillis

and

G.L.

Steele

Jr.

Data-parallel

algorithms

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

C.A.R.

Hoare

Communicating

sequential

processes

Communications

of

the

ACM

-LCB-

-LSB-

-RSB-

Intel

Corporation

Beaverton

OR

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

K.E.

Iverson

A

programming

langage

Wiley

New-York

-LSB-

-RSB-

R.

Keryell

Ph.

Matherat

and

N.

Paris

POMP

or

how

to

design

a

massively

parallel

machine

with

small

developments

In

Proc

Conf

on

Parallel

Arch

and

Lang

Europe

PARLE

vol

number

in

Lect

Notes

Comp

Science

Eindhoven

Springer

-LSB-

-RSB-

Bj?orn

Lisper

The

Data

Parallel

Programming

Model

chapter

Data

Parellelism

and

Functional

Programming

pages

-LCB-

Lecture

Notes

in

Computer

Science

Tutorial

Springer

Verlag

-LSB-

-RSB-

MasPar

Computer

Corporation

Sunnyvale

CA

Maspar

Parallel

Application

Language

Reference

Manual

-LSB-

-RSB-

MasPar

Computer

Corporation

Sunnyvale

CA

MP-Fortran

Reference

Manual

-LSB-

-RSB-

N.

Paris

D?efinition

de

POMPC

-LRB-

version

-RRB-

Technical

Report

LIENS

Paris

English

version

in

preparation

-LSB-

-RSB-

R.H.

Perrot

A

language

for

array

and

vector

processors

ACM

Trans

on

Progr

Lang

and

Syst.

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

G.L.

Steele

Jr.

and

W.D.

Hillis

Connection

machine

lisp

Fine-grain

parallel

symbolic

processing

In

Proc

ACM

Conf

on

Lisp

and

Funct

Progr.

pages

-LCB-

Cambridge

Mass.

-LSB-

-RSB-

Thinking

Machine

Corporation

Cambridge

MA

C

Programming

Guide

-LSB-

-RSB-

Thinking

Machine

Corporation

Cambridge

MA

CM-FORTRAN

Programming

Guide

-LSB-

-RSB-

Thinking

Machine

Corporation

Cambridge

MA

Technical

Summary

-LSB-

-RSB-

Thinking

Machine

Corporation

Cambridge

MA

Technical

Summary

specifying and verifying and reasoning about programs