Reliable

Hardware

Barrier

Synchronization

Schemes

Rajeev

Sivaram

Craig

B.

Stunkel

and

Dhabaleswar

K.

Panda

Technical

Report

Reliable

Hardware

Barrier

Synchronization

Schemes

Rajeev

Sivaram

Craig

B.

Stunkely

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

I.B.M.

T.

J.

Watson

Research

Centery

The

Ohio

State

University

P.

O.

Box

Columbus

OH

Yorktown

Heights

NY

Email

fsivaram,pandag@cis.ohio-state.edu

Email

stunkel@watson.ibm.com

Contact

Author

Craig

B.

Stunkel

Abstract

Barrier

synchronization

is

a

crucial

operation

for

parallel

systems

Many

schemes

have

been

proposed

in

the

literature

to

achieve

fast

barrier

through

software

hardware

or

a

combination

of

these

mechanisms

However

of

these

schemes

emphasize

fault-tolerant

barrier

operations

In

this

paper

we

describe

inexpensive

support

that

can

be

added

to

network

switches

for

achieving

reliable

hardware-based

barrier

synchronization

while

recovering

from

lost

or

corrupted

messages

Necessary

modifications

to

the

switch

architecture

and

the

associated

fault-tolerant

message-passing

protocols

are

presented

The

protocols

are

optimized

for

the

no-fault

case

while

providing

means

to

detect

the

failure

of

any

step

of

the

operation

and

to

recover

from

it

The

proposed

scheme

is

evaluated

with

and

without

specialized

support

at

the

network

interface

and

compared

with

similar

approaches

using

software-based

schemes

It

promises

significant

potential

to

be

applied

to

switch-based

parallel

systems

especially

the

emerging

networks

of

workstations

This

research

is

supported

in

part

by

NSF

Grant

NSF

Career

Award

and

an

IBM

Cooperative

Fellowship

Contents

Introduction

Architectural

requirements

Packlet

and

switch

architecture

description

The

barrier

unit

Setup

of

barrier

state

Fault

model

Protocols

to

Overcome

Message

Loss

Normal

Operation

Handling

Faults

Analytical

Evaluation

of

Alternative

Barrier

Schemes

Completely

Software

Based

Scheme

-LRB-

UMin

-RRB-

Software

Scheme

with

Network

Interface

Support

-LRB-

UMin-NI

-RRB-

Hardware

Combining

and

Reporting

-LRB-

and

-RRB-

Analytical

Evaluation

of

Performance

Performance

Evaluation

Experiments

Effect

of

Startup

Overhead

Effect

of

Switch

Size

Effect

of

System

Size

Conclusions

and

Future

Research

Introduction

Barrier

synchronization

or

barrier-sync

is

a

crucial

collective

communication

operation

for

message-passing

systems

and

particularly

for

shared-memory

parallel

systems

In

this

paper

we

describe

methods

for

embedding

fault-tolerant

yet

inexpensive

barrier

support

into

the

interconnection

network

of

a

parallel

system

Barriers

provide

a

means

of

ensuring

that

no

process

in

a

group

of

processes

advances

beyond

a

particular

point

in

a

computation

until

all

have

arrived

at

that

point

-LSB-

-RSB-

The

barrier

is

useful

for

bringing

the

group

of

processes

to

a

known

global

state

before

proceeding

to

a

new

phase

of

computation

The

efficiency

of

barrier-sync

affects

how

small

the

grain-size

of

tasks

in

an

application

can

become

without

significantly

degrading

performance

Parallel

systems

have

offered

a

range

of

barrier

solutions

from

totally

software-based

approaches

to

dedicated

hardware

The

IBM

the

Intel

Paragon

and

most

networks

of

workstations

implement

a

totally

software-based

messaging

approach

a

total

data-less

exchange

is

performed

by

sending

several

phases

of

messages

via

the

system

message-transmission

routines

-LSB-

-RSB-

Each

of

these

messages

incurs

a

significant

software

startup

overhead

and

the

number

of

messages

sent

is

proportional

to

N

where

N

is

the

number

of

processes

in

the

group

This

software

methodology

is

easily

portable

to

new

systems

but

results

in

a

high

latency

for

barrier

operations

To

address

high

latency

several

machines

-LSB-

-RSB-

have

implemented

dedicated

hardware

networks

to

perform

barrier-sync

and

some

other

important

collective

communication

operations

This

approach

results

in

much

lower

latency

for

barrier

synchronization

but

for

a

large

parallel

processor

it

may

be

expensive

to

maintain

multiple

networks

for

specialized

operations

Many

other

dedicated

hardware

schemes

have

been

proposed

-LSB-

-RSB-

A

compromise

is

to

provide

hardware

support

for

barrier-sync

within

the

existing

data

network

-LSB-

-RSB-

The

-LSB-

-RSB-

is

particularly

noteworthy

for

embedding

barriersync

trees

within

the

data

network

because

its

predecessor

the

-LSB-

-RSB-

maintained

a

separate

network

for

this

purpose

A

related

strategy

is

to

provide

extra

support

at

the

network

interface

boundary

to

avoid

the

large

software

overheads

for

software

multicast

trees

-LSB-

-RSB-

To

our

knowledge

the

fault-tolerant

operation

of

barrier-sync

has

not

been

specifically

addressed

in

the

literature

and

this

is

the

prime

focus

of

our

paper

We

describe

inexpensive

support

that

can

be

added

to

network

switches

for

improving

barrier-sync

while

recovering

from

lost

or

corrupted

messages

It

is

our

assumption

that

faults

and

errors

happen

infrequently

in

typical

systems

and

thus

we

optimize

for

the

no-fault

case

while

providing

means

for

the

appropriate

participating

nodes

to

detect

the

failure

of

any

step

of

the

barrier-sync

and

to

initiate

the

recovery

of

its

operation

Each

switch

in

the

network

is

augmented

with

a

barrier

unit

that

is

attached

to

the

input

and

output

of

the

internal

switch

crossbar

The

barrier

unit

operates

on

tiny

barrier

packets

in

several

phases

during

a

barrier-sync

and

contains

sufficient

state

to

keep

track

of

many

simultaneous

barrier

operations

within

the

network

In

the

initial

phase

processor

nodes

send

Combine

packets

which

are

successively

combined

by

barrier

units

until

one

resulting

packet

reaches

a

node

that

serves

as

the

root

of

a

multicast

tree

In

the

second

phase

this

root

node

sends

a

Multicast

packet

to

all

other

nodes

in

the

tree

with

the

replication

controlled

by

the

barrier

units

In

the

final

phase

which

can

proceed

concurrently

with

post-barrier

execution

each

non-root

node

sends

an

Ack

packet

which

is

again

combined

into

one

packet

before

it

reaches

the

root

node

We

introduce

protocols

that

enable

all

fault

recovery

to

be

handled

at

the

nodes

by

retries

of

Combine

or

Multicast

packets

The

hardware

modifications

and

protocols

are

equally

applicable

to

both

direct

and

indirect

-LRB-

switch-based

-RRB-

networks

We

do

not

focus

on

the

parallel

application

performance

improvement

possible

via

network

hardware

barrier

operations

Several

papers

have

adequately

addressed

this

topic

-LSB-

-RSB-

and

our

hardware

scheme

should

be

similar

in

performance

to

other

tree-based

hardware

combining

schemes

However

we

do

provide

a

potential

implementation

and

we

compare

the

barrier-sync

latency

of

this

implementation

to

software-based

schemes

In

addition

we

assess

the

effect

of

moving

software

function

from

a

node

processor

to

a

communication

co-processor

as

is

possible

in

both

software-based

and

network

hardware-based

barrier

approaches

Section

describes

our

architectural

requirements

for

supporting

barrier

synchronization

and

Section

explains

the

fault-tolerant

protocol

In

Section

we

analytically

compare

performance

for

several

combinations

of

barrier

hardware

and

software

In

Section

we

compare

latencies

for

several

different

barrier

implementations

Architectural

requirements

Barrier-sync

is

equivalent

to

a

data-less

total

exchange

every

process

in

the

group

must

be

able

to

observe

when

every

other

process

has

arrived

at

the

barrier

There

are

several

ways

of

implementing

this

total

exchange

-LSB-

-RSB-

Number

each

processing

node

or

node

from

to

N

where

N

is

the

number

of

nodes

in

the

barrier-sync

group

We

will

refer

to

data-less

message

packets

as

packlets

When

N

is

a

power

of

packlet

exchanges

can

be

performed

along

successive

dimensions

in

the

group

If

p

is

the

local

node

id

this

algorithm

can

formulated

at

each

node

in

C-like

syntax

as

for

-LRB-

d

d

N

d

-RRB-

packlet-exchange

-LRB-

p

p

XOR

-RRB-

In

another

method

one

node

is

arbitrarily

defined

as

the

root

and

each

non-root

node

sends

a

packlet

to

the

root

When

the

root

node

receives

packlets

from

all

non-root

nodes

the

root

sends

packlets

to

all

of

the

non-root

nodes

to

complete

the

barrier

This

method

is

easily

applicable

to

any

number

of

processes

This

latter

method

is

advantageous

for

a

network

that

-LRB-

a

-RRB-

can

combine

packlets

traveling

toward

the

root

node

and

-LRB-

b

-RRB-

can

replicate

a

packlet

-LRB-

i.e.

multicast

a

packlet

-RRB-

traveling

from

the

root

to

all

non-root

nodes

In

this

section

we

describe

an

architecture

which

can

perform

these

combining

and

multicast

functions

only

for

packlets

Note

this

is

a

much

simpler

requirement

than

supporting

general

packet

multicast

-LSB-

-RSB-

because

of

the

small

fixedsize

of

the

packlet

and

it

is

simpler

than

supporting

general

combining

functions

-LSB-

-RSB-

The

Cray

-LSB-

-RSB-

network

already

performs

packlet

multicast

-LRB-

called

eureka

-RRB-

and

applies

a

combination

of

eureka

and

packlet

combining

hardware

to

perform

barrier-sync

Packlet

and

switch

architecture

description

The

packlet

though

carrying

no

data

to

be

replicated

during

multicast

and

no

data

to

be

combined

during

combining

nevertheless

contains

several

fields

-LRB-

a

-RRB-

a

packlet

type

-LRB-

b

-RRB-

a

barrier

tag

and

-LRB-

c

-RRB-

a

barrier

sequence

number

Figure

shows

a

possible

packlet

implementation

A

packlet

type

-LRB-

bits

in

Figure

-RRB-

indicates

that

this

packet

is

a

packlet

and

Figure

A

sample

packlet

definition

for

a

flit

distinguishes

how

the

packlet

is

being

used

within

the

barrier-sync

such

as

in

a

combining

phase

or

in

a

multicast

phase

The

exact

types

will

be

discussed

in

Section

The

barrier

tag

identifies

the

packlet

's

barrier

group

If

B

is

the

total

number

of

barriers

supported

by

the

network

hardware

then

B

can

not

be

greater

than

for

the

packlet

definition

in

Figure

The

barrier

sequence

number

-LRB-

bit

-RRB-

is

required

to

distinguish

between

consecutive

barrier-syncs

executed

by

the

same

barrier

group

Assuming

packet

flits

a

packlet

could

fit

within

one

packet

flit

while

allowing

hundreds

of

barrier

groups

and

thus

for

the

remainder

of

the

paper

we

shall

assume

packlets

to

simplify

the

discussion

of

both

architecture

and

performance

The

interconnection

network

comprises

switches

and

processing

nodes

connected

by

links

Each

network

link

is

bidirectional

connecting

an

input

port

i

and

an

output

port

i

from

one

device

to

an

input

port

j

and

an

output

port

j

of

another

device

We

assume

that

each

switch

in

an

interconnection

network

connects

k

flit-wide

input

ports

to

k

flit-wide

output

ports

via

a

k

k

crossbar

For

assessing

performance

we

assume

wormhole

routing

and

flit-wide

links

where

a

flit

is

the

sub-packet

unit

upon

which

network

flow-control

is

performed

-LSB-

-RSB-

Each

input

port

contains

a

FIFO

that

can

buffer

flits

A

switch

modified

for

packlet

multicast

and

combining

is

shown

in

Figure

One

additional

input

and

output

have

been

added

to

the

internal

crossbar

providing

paths

to

and

from

a

new

function

the

barrier

unit

The

barrier

unit

stores

and

processes

barrier

information

for

up

to

B

concurrent

barriers

in

the

system

To

simplify

the

diagram

we

have

not

shown

any

flow-control

signals

our

simulations

in

Section

assume

a

token-based

flow-control

similar

to

that

used

in

the

IBM

-LSB-

-RSB-

The

barrier

unit

The

barrier

unit

detailed

in

Figure

contains

a

B-entry

SRAM

Entry

i

in

the

SRAM

stores

the

current

barrier

state

for

the

barrier

group

i

and

is

addressed

by

packlets

with

barrier

tag

i

When

a

packlet

arrives

at

the

barrier

unit

it

is

placed

into

the

control

register

during

the

first

cycle

of

a

barrier

operation

During

the

second

cycle

the

barrier

tag

field

of

the

packlet

is

used

to

fetch

the

correct

barrier

state

into

the

state

register

In

parallel

the

packlet

's

error

detection

code

is

used

to

detect

corrupted

packlets

such

packlets

are

-LRB-

k

-RRB-

x

-LRB-

k

-RRB-

Crossbar

Barrier

Unit

Output

Figure

A

switch

incorporating

packlet

barrier

logic

Figure

The

barrier

unit

immediately

discarded

Dependent

upon

the

packlet

type

in

the

third

cycle

the

state

is

modified

During

the

fourth

cycle

the

state

is

written

back

into

the

SRAM

and

the

barrier

unit

operation

is

complete

In

many

instances

the

protocol

will

also

dictate

the

generation

of

a

new

packlet

which

is

immediately

enqueued

in

the

FIFO

at

the

output

of

the

barrier

unit

during

the

fourth

cycle

A

full

FIFO

stalls

the

barrier

unit

Similarly

the

barrier

unit

does

not

accept

input

from

the

crossbar

while

the

control

register

is

occupied

This

barrier

unit

operation

could

easily

be

pipelined

to

achieve

processing

of

up

to

one

packlet

per

cycle

We

have

chosen

to

describe

minimal

hardware

functionality

to

simplify

the

discussion

Each

entry

in

the

SRAM

records

the

current

local

state

of

the

barrier-sync

as

displayed

in

Figure

The

barrier

protocols

traverse

a

number

of

phases

as

will

be

discussed

in

detail

in

Section

To

distinguish

between

packlets

of

the

current

barrier-sync

and

packlets

of

old

or

new

barrier-syncs

a

sequence

number

is

sufficient

and

is

compared

to

the

sequence

number

field

of

the

packlet

in

the

control

register

Figure

An

entry

in

the

barrier

state

SRAM

The

TreeInfo

field

records

the

local

information

about

the

multicast-combine

tree

that

connects

all

nodes

of

the

associated

barrier-sync

group

All

multicast-combine

trees

are

rooted

at

a

single

node

and

branch

to

all

other

nodes

in

the

barrier-group

from

there

thus

forming

a

directed

tree

The

use

of

a

single

root

is

a

crucial

enabler

of

the

endto-end

fault

recovery

to

be

described

in

Section

TreeInfo

consists

of

k

bits

An

active

bit

i

indicates

that

output

port

i

is

an

outgoing

port

of

the

packlet

multicast

tree

-LRB-

and

an

incoming

port

of

the

combine

tree

-RRB-

The

Parent

Port

field

indicates

to

which

output

port

a

combined

packlet

will

be

forwarded

This

port

is

also

the

port

entered

by

Multicast

packlets

for

the

associated

barrier

group

Two

other

fields

Combine

State

-LRB-

CS

-RRB-

and

Ack

State

-LRB-

AS

-RRB-

are

used

to

monitor

the

progress

of

packlet

combining

during

different

phases

of

the

protocol

Combine

State

represent

the

ports

are

still

expected

to

receive

a

Combine

packlet

during

the

current

barrier

sync

Ack

State

records

similar

information

for

Ack

Packlets

When

TreeInfo

is

first

initialized

Combine

State

must

also

be

initialized

to

the

same

value

Packlets

of

type

Multicast

require

additional

handling

While

these

packlets

are

forwarded

through

the

crossbar

to

enter

the

barrier

unit

they

also

remain

at

the

head

of

their

current

input

FIFO

After

the

current

barrier

state

is

read

into

the

state

register

the

Ack

State

field

is

sent

to

the

input

FIFO

and

determines

the

output

ports

to

which

to

forward

the

Multicast

packlet

The

input

port

then

requests

use

of

the

crossbar

for

each

of

these

output

ports

After

packlets

are

sent

to

all

outputs

the

packlet

is

removed

from

the

head

of

the

input

port

FIFO

Setup

of

barrier

state

We

assume

in

Section

that

the

Parent

Port

TreeInfo

and

Combine

fields

of

the

barrier

state

are

correctly

initialized

before

any

barriers

of

that

group

are

initiated

Figure

illustrates

a

bidirectional

MIN

-LSB-

-RSB-

network

in

which

these

fields

have

been

initialized

for

a

barrier

group

This

setup

can

be

accomplished

in

various

ways

Special

circuit

setup

packets

may

be

sent

to

every

switch

involved

in

the

associated

multicast/combine

tree

in

a

manner

similar

to

ATM

virtual

path/channel

setup

-LSB-

-RSB-

Alternately

if

the

switches

already

support

a

means

of

general

multicast

-LSB-

-RSB-

a

multicast

packet

could

be

used

to

carry

the

barrier

tag

and

the

SRAM

barrier

state

pointed

to

by

the

barrier

tag

would

then

store

the

identity

of

the

calculated

local

output

ports

of

the

multicast

packet

The

particular

choice

and

implementation

of

TreeInfo

setup

is

not

the

focus

of

this

paper

We

are

primarily

concerned

with

the

reliable

operation

of

repeated

barrier-syncs

with

a

barrier-group

101 123 4567 123

101 00001111

001 00000001

Non-Root

Participant

Root

Participant

Figure

Initialized

barrier

state

within

a

bidirectional

MIN

network

Fault

model

In

this

paper

we

assume

a

lost

packet

fault

model

Messages

sent

from

a

source

to

a

destination

may

be

lost

because

of

transient

errors

occurring

anywhere

along

the

message

path

Messages

that

are

corrupted

in

transit

yet

reach

the

destination

are

detected

by

error

code

checking

and

are

discarded

which

is

equivalent

to

losing

the

message

For

normal

message-passing

transient

error

recovery

can

be

entirely

handled

at

the

nodes

-LRB-

the

source

and

destination

-RRB-

However

for

combining

and

multicasting

operations

within

the

network

the

sources

and

destinations

of

messages

are

often

switches

We

desire

to

keep

these

switches

as

simple

as

possible

continuing

to

push

fault

recovery

to

the

nodes

The

protocols

described

in

Section

accomplish

this

in

cooperation

with

switch

barrier

units

We

do

not

consider

permanent

faults

because

these

faults

are

typically

handled

by

system

reconfiguration

software

in

commercial

systems

-LSB-

-RSB-

Depending

on

the

permanent

fault

reconfiguration

software

may

need

to

reinitialize

the

barrier

state

for

affected

barriers

Protocols

to

Overcome

Message

Loss

In

this

section

we

present

the

protocols

followed

by

the

switches

root

and

non-root

processes

so

as

to

ensure

that

a

barrier

completes

successfully

and

that

all

processes

can

continue

computation

We

first

present

the

protocols

followed

under

normal

operation

when

no

messages

are

lost

We

then

handle

the

case

where

messages

get

lost

or

corrupted

in

transit

Finite

state

diagrams

are

presented

to

illustrate

the

working

of

the

protocols

Barrier

synchronization

proceeds

in

three

main

phases

the

last

of

which

is

necessary

to

ensure

that

the

barrier

completes

despite

message

loss

The

first

phase

is

called

the

Combine

phase

In

this

phase

the

non-root

participants

send

a

Combine

packlet

towards

the

root

processor

when

they

reach

the

barrier

The

second

phase

begins

when

the

root

-LRB-

i

-RRB-

reaches

the

corresponding

barrier

and

-LRB-

ii

-RRB-

receives

a

Combine

message

that

represents

collected

knowledge

that

all

non-root

participants

have

reached

the

barrier

In

this

second

phase

-LRB-

called

the

Multicast

phase

-RRB-

the

root

processor

informs

the

non-root

processors

that

the

barrier

has

been

reached

via

a

Multicast

packlet

To

ensure

that

all

non-root

processors

have

successfully

received

the

Multicast

packlet

-LRB-

so

that

they

can

resume

computation

-RRB-

the

non-root

processors

are

required

to

acknowledge

the

receipt

of

the

Multicast

packlet

This

constitutes

the

third

phase

-LRB-

called

the

Ack

phase

-RRB-

The

acknowledgment

packlet

sent

by

the

non-root

processors

in

this

phase

is

called

an

Ack

packlet

Normal

Operation

Under

normal

operation

the

events

that

occur

are

as

follows

If

a

root

process

-LRB-

the

protocol

followed

by

a

root

process

during

a

barrier

is

shown

in

Fig.

-RRB-

reaches

a

barrier

it

waits

for

a

Combine

packlet

to

arrive

from

the

non-root

processes

If

a

Combine

packlet

has

already

arrived

at

the

root

when

it

reaches

the

barrier

the

root

responds

immediately

with

a

Multicast

packlet

to

inform

the

non-root

processes

that

everyone

has

reached

the

barrier

IDLE

Wait

for

Combine

Wait

for

Ack

Wait

forBarrierReached

BarrierReached

Ack

discard

Combine

send

Multicast

Combine

-LSB-

new

-RSB-

Toggle

seq.

bit

send

Multicast

Combine

discard

Timeout

send

Multicast

Combine

Combine

-LSB-

new

-RSB-

send

Multicast

Ack-discard

Ack

Toggle

seq.

bit

Toggle

seq.

bit

Combine-discard

Figure

Finite

State

Machine

diagram

for

the

algorithm

executed

by

the

root

of

a

barrier

The

thick

arcs

represent

the

transitions

that

are

made

under

normal

system

operation

The

thinner

arcs

represent

transitions

made

under

extraordinary

circumstances

including

cases

where

packlets

get

lost

or

corrupted

When

a

non-root

process

-LRB-

the

protocol

followed

by

a

non-root

process

is

shown

in

Fig.

-RRB-

reaches

a

barrier

it

sends

a

Combine

packlet

towards

the

root

process

-LRB-

we

assume

that

a

processor

is

connected

to

a

switch

by

a

single

bidirectional

link

thus

there

is

no

choice

to

be

made

before

getting

to

the

first

switch

-RRB-

Upon

receiving

a

Combine

message

a

switch

-LRB-

the

protocol

followed

by

the

switch

is

shown

in

Fig

-RRB-

turns

off

the

bit

in

CS

corresponding

to

the

bidirectional

link

on

which

the

message

was

received

to

denote

that

an

acknowledgment

has

been

received

from

a

subtree

of

the

network

If

the

value

of

CS

becomes

because

of

this

turning

off

of

a

bit

the

switch

forwards

a

single

Combine

message

to

the

port

that

leads

it

towards

its

root

IDLE

Wait

for

Multicast

BarrierReached

Toggle

seq.

bit

send

Combine

Multicast

sendAck

Timeout

send

Combine

Multicast

-LSB-

old

-RSB-

discard

Multicasr

send

Ack

Multicast

-LSB-

old

-RSB-

discard

Figure

Finite

State

Machine

diagram

for

the

algorithm

executed

by

a

non-root

participant

in

a

barrier

Using

the

method

described

above

a

single

Combine

packlet

reaches

the

root

process

The

root

process

then

sends

out

a

Multicast

packlet

to

all

non-root

participants

to

inform

them

that

all

the

processes

in

the

group

have

reached

the

barrier

and

that

they

can

now

proceed

The

root

processor

can

proceed

with

computation

as

soon

as

it

sends

out

the

Multicast

packlet

We

assume

however

that

the

process

that

handles

barrier

related

computations

remains

active

to

process

the

acknowledgments

to

the

Multicast

packlet

An

intermediate

switch

receiving

the

Multicast

packlet

copies

TreeInfo

to

CS

and

AS

and

forwards

the

message

to

appropriate

output

ports

Note

that

it

is

not

necessary

for

the

multicast

to

carry

a

routing

header

the

intermediate

switches

can

use

the

value

of

TreeInfo

stored

to

route

the

multicast

packlets

The

CS

values

are

also

set

up

by

the

multicast

-LRB-

in

addition

to

setting

up

the

AS

values

-RRB-

for

the

combine

phase

of

the

next

barrier

Upon

receiving

a

Multicast

packlet

a

non-root

processor

acknowledges

the

receipt

of

this

message

with

an

Ack

packlet

The

intermediate

switches

turn

off

the

appropriate

bit

in

AS

and

forward

an

Ack

packlet

if

AS

becomes

due

to

the

turning

off

of

this

bit

The

root

process

knows

upon

receiving

Ack

that

the

barrier

has

completed

successfully

Thus

in

the

no

fault

case

the

only

overhead

incurred

by

the

nodes

is

the

sending

or

receiving

of

an

Ack

packlet

This

overhead

can

potentially

be

overlapped

with

computation

especially

with

the

use

of

a

smart

network

interface

as

described

in

Section

Handling

Faults

The

above

described

method

is

only

for

the

case

where

nothing

goes

wrong

In

practice

a

number

of

things

may

go

wrong

First

the

Combine

Multicast

or

Ack

packlet

may

get

lost

or

corrupted

in

transit

Thus

we

may

need

to

timeout

and

re-send

at

least

some

of

these

messages

As

described

before

corrupted

messages

-LRB-

which

fail

the

error

detection

test

-RRB-

are

discarded

We

therefore

do

not

handle

corrupted

messages

any

differently

from

the

way

we

handle

lost

messages

Secondly

processes

in

the

system

may

get

skewed

so

that

processes

arrive

at

barriers

at

times

that

are

farther

apart

than

the

timeout

value

set

In

such

cases

multiple

messages

of

a

type

may

have

to

be

dealt

with

These

scenarios

may

even

lead

to

a

situation

where

one

Multicast

discard

it

forward

Ack

Reset

CS

bit

Reset

AS

bit

IDLE

Combine

Recvd

Multicast

Recvd

AllCombines

Recvd.Multicast

-LSB-

old

-RSB-

discard

it

Combineif

CS

Reset

CS

bitif

CS

send

Combine

Packlet

Multicast

copy

TreeInfo

to

CS

AS

forward

Multicast

Combine

discard

itAll

Combines

-LSB-

new

-RSB-

forward

Combine

Multicast

forward

to

ports

in

AS

Intermediate

Combine

-LSB-

new

-RSB-

Reset

CS

bit

Int

Ack

Reset

AS

bit

Last

Ack

CS

TreeInfo

forward

Ack

Intermediate

Combine

Packlet

Combine

-LSB-

new

-RSB-

Multicast

discard

it

forward

Ack

-LRB-

backwards

-RRB-

discard

it

Ack-discard

it

set

CS

TreeInfo

Reset

CS

bit

Last

Combine

Packlet

forward

Combine

Packlet

Combine

Reset

CS

bit

Ack

-LRB-

backwards

-RRB-

to

ports

in

TreeInfo

causing

AS

forward

Ack

Lost

Ack

-LRB-

TreeInfo

CS

-RRB-

forward

Ack

Figure

Finite

State

Machine

diagram

for

the

functionality

of

a

switch

during

a

barrier

of

the

processes

in

the

group

moves

from

one

barrier

to

the

next

and

sends

out

a

Combine

message

for

the

new

barrier

while

other

processes

are

still

waiting

for

the

Multicast

packlet

to

arrive

-LRB-

because

an

initial

Multicast

packlet

reached

some

participants

who

sent

out

an

Ack

packlet

and

proceeded

from

the

barrier

while

the

Multicast

did

n't

reach

other

participants

who

have

to

wait

for

a

re-sent

Multicast

-RRB-

The

protocols/algorithms

outlined

in

Figures

and

take

into

account

all

these

anomalous

cases

We

present

some

of

these

anomalous

cases

and

how

they

are

handled

below

Combine

Lost

To

deal

with

the

case

where

the

Combine

packlet

sent

from

one

or

a

group

of

non-root

participants

gets

lost

we

propose

that

every

non-root

participant

time-out

on

the

Combine

packlet

i.e.

if

the

participant

does

not

receive

a

Multicast

packlet

within

a

fixed

time

of

its

sending

of

a

Combine

packlet

it

resends

the

Combine

packlet

An

intermediate

switch

that

is

waiting

for

Combine

packlets

may

have

seen

the

original

Combine

packlet

In

the

case

where

this

switch

is

still

waiting

for

acknowledgments

from

other

it

treats

the

re-sent

Combine

just

as

if

it

were

the

first

Combine

packlet

from

that

port

and

just

resets

the

-LRB-

already

-RRB-

appropriate

CS

bit

If

it

has

already

received

all

other

Combine

packlets

and

forwarded

a

Combine

packlet

the

switch

-LRB-

in

the

state

All

Combines

Received

-RRB-

copies

TreeInfo

to

CS

sets

off

the

appropriate

CS

bit

and

awaits

-LRB-

re-sent

-RRB-

Combines

from

other

may

be

due

to

a

variety

of

reasons

The

processes

may

be

skewed

so

that

some

of

them

do

n't

reach

their

barrier

long

after

others

do

Alternatively

the

expected

acknowledgments

may

be

lost

If

a

Combine

packlet

did

reach

the

root

the

root

would

have

responded

with

a

Multicast

packlet

This

Multicast

overwrites

CS

setting

it

to

TreeInfo

again

If

any

more

re-sent

Combines

are

received

by

this

switch

-LRB-

which

is

now

in

the

state

Multicast

Received

-RRB-

they

are

discarded

Note

that

the

crossing

of

re-sent

Combines

and

re-sent

Multicasts

causes

no

problems

-LCB-

the

Combine

packlet

will

simply

be

dropped

at

the

next

switch

it

visits

-LRB-

the

last

switch

that

Multicast

visited

-RRB-

and

the

Multicast

packlet

will

simply

overwrite

the

CS

value

at

the

next

switch

it

visits

-LRB-

the

last

switch

visited

by

the

Combine

-RRB-

Multicast

Lost

After

a

Multicast

packlet

is

sent

out

if

an

Ack

packlet

is

not

received

within

a

certain

timeout

period

a

Multicast

packlet

is

re-sent

to

the

non-root

participants

The

root

can

not

distinguish

between

the

loss

of

an

Multicast

packlet

and

the

loss

of

the

corresponding

acknowledgment

Ack

Similarly

the

non-root

participant

can

not

distinguish

between

a

lost

Combine

and

a

lost

Multicast

We

now

explain

how

our

solution

deals

with

the

case

where

a

re-sent

Multicast

and

an

Ack

-LRB-

sent

in

response

to

a

previous

Multicast

-RRB-

cross

each

other

on

a

link

Let

us

assume

that

the

Multicast

and

Ack

packlets

cross

while

the

Multicast

packlet

is

traveling

from

a

node/switch

A

to

a

node/switch

B

and

while

Ack

is

traveling

in

the

opposite

direction

Let

us

first

examine

the

case

where

both

A

and

B

are

switches

When

Multicast

crosses

switch

A

switch

A

is

in

the

state

Multicast

Received

Since

switch

B

has

forwarded

an

Ack

packlet

towards

switch

A

it

must

be

the

case

that

switch

B

must

be

in

the

state

IDLE

Combine

Recvd

or

All

Combines

Recvd

The

Multicast

packlet

is

discarded

upon

receipt

at

switch

B

and

an

Ack

packlet

is

sent

out

behind

the

previous

Ack

If

the

previous

Ack

packlet

does

not

cause

AS

to

become

all

then

A

remains

in

the

state

Multicast

Received

the

Ack

packlet

sent

by

B

in

response

to

Multicast

is

treated

just

like

the

previous

Ack

packlet

-LRB-

the

bit

in

AS

that

has

already

been

reset

is

reset

again

-RRB-

On

the

other

hand

if

the

previous

Ack

packlet

causes

AS

to

become

all

then

the

state

corresponding

to

the

barrier

at

A

is

IDLE

or

Combine

Recvd

the

Ack

packlet

sent

by

B

is

sunk

at

A

If

A

is

the

root

node

and

B

is

a

switch

the

Ack

sent

by

B

in

response

to

the

Multicast

packlet

will

get

discarded

by

the

root

if

it

received

the

previous

Ack

packlet

-LRB-

the

previous

Ack

would

have

left

the

root

in

the

IDLE

state

-RRB-

If

A

is

a

switch

and

B

is

a

non-root

node

B

sends

an

Ack

packlet

in

response

to

the

Multicast

received

provided

it

has

not

already

reached

the

next

In

the

latter

case

the

Multicast

packlet

is

simply

discarded

The

events

that

occur

at

switch

A

are

as

described

reason

we

have

to

do

this

is

that

the

previous

Combine

message

forwarded

from

this

switch

may

have

been

lost

If

we

simply

discard

the

arriving

Combine

message

we

may

never

be

able

to

inform

the

root

that

the

barrier

has

been

reached

with

the

root

waiting

for

a

Combine

packlet

that

has

been

lost

-LRB-

and

which

will

never

arrive

if

we

discard

the

arriving

Combine

packlet

-RRB-

We

may

simply

forward

a

Combine

message

that

arrives

after

CS

becomes

but

this

will

result

in

unnecessary

additional

traffic

in

the

case

where

processes

are

skewed

and

the

Combine

message

arrives

at

the

switch

just

after

it

has

forwarded

the

previous

Combine

message

The

adopted

approach

is

a

compromise

that

reduces

traffic

that

a

non-root

participant

can

continue

with

computation

as

soon

as

it

sends

the

Ack

packlet

for

the

first

time

for

the

case

where

A

and

B

are

both

switches

Ack

Lost

When

an

Ack

packlet

is

lost

the

root

times

out

and

sends

a

Multicast

packlet

Note

that

an

Ack

packlet

is

only

sent

out

in

response

to

a

Multicast

packlet

received

by

a

non-root

participant

Almost

all

cases

dealing

with

lost/resent

Combine

packlets

have

been

dealt

with

above

However

since

we

assume

that

a

non-root

participant

can

proceed

from

a

barrier

as

soon

as

it

sends

its

first

Ack

packlet

for

the

barrier

we

may

face

the

situation

that

the

non-root

participant

reaches

the

next

barrier

coupled

with

the

loss

of

the

Ack

packlet

sent

In

such

a

case

some

of

the

switches

are

still

in

the

state

Multicast

Received

awaiting

the

arrival

of

more

Ack

packlets

However

if

a

Combine

with

a

different

sequence

number

arrives

at

the

switch

on

a

port

on

which

it

is

awaiting

an

Ack

packlet

the

switch

can

use

this

Combine

packlet

as

an

implicit

Ack

packlet

the

switch

resets

the

appropriate

AS

bit

in

addition

to

resetting

the

appropriate

CS

bit

-LRB-

note

that

the

Multicast

packlet

already

initialized

CS

for

the

next

barrier

-RRB-

We

have

presented

protocols

for

reliable

hardware

barrier

synchronization

above

In

the

next

section

we

present

alternative

barrier

synchronization

schemes

and

analytical

estimates

of

their

performance

Section

experimental

results

comparing

these

alternative

barrier

synchronization

schemes

Analytical

Evaluation

of

Alternative

Barrier

Schemes

In

this

section

we

provide

an

analytical

comparison

of

the

performance

of

two

flavors

of

the

hardware

scheme

proposed

in

this

paper

and

two

flavors

of

a

software

scheme

We

first

begin

by

describing

the

two

flavors

of

the

hardware

and

software

schemes

For

each

of

the

schemes

we

describe

the

situations

under

which

the

best

and

worst

case

barrier

latency

is

obtained

for

a

bidi-MIN

network

-LRB-

Fig.

-RRB-

The

barrier

synchronization

latency

is

the

amount

of

time

that

elapses

between

from

the

point

where

the

last

process

in

the

group

of

participants

reaches

the

barrier

and

the

point

where

the

last

of

the

processors

in

the

group

has

been

informed

that

everyone

has

reached

the

barrier

The

barrier

latency

depends

on

the

times

at

which

the

participating

processors

reach

their

respective

barriers

Finally

we

present

an

analytical

estimate

of

the

best

and

worst

case

performance

of

each

of

the

schemes

for

a

bidi-MIN

system

using

turnaround

routing

-LSB-

-RSB-

Completely

Software

Based

Scheme

-LRB-

UMin

-RRB-

Various

methods

for

software

multicast

on

bidi-MINs

have

been

proposed

The

U-Min

algorithm

-LSB-

-RSB-

uses

a

contention

free

binomial

tree

for

performing

multicast

with

the

source

node

as

root

The

multicast

proceeds

by

having

intermediate

destinations

act

as

secondary

sources

in

succeeding

phases

of

the

described

in

Sec.

efficient

synchronization

can

be

performed

using

packlet

exchange

However

this

method

is

only

applicable

for

barriers

where

the

participant

set

size

is

a

power

of

two

We

therefore

do

not

consider

this

scheme

A

binomial

tree

constructed

for

multicast

can

also

be

used

for

barrier

synchronization

-LSB-

-RSB-

The

basic

idea

is

to

use

the

reverse

binomial

tree

to

collect

information

about

the

participants

that

have

reached

the

barrier

Once

the

root

of

the

binomial

tree

learns

that

all

participants

have

reached

the

barrier

it

informs

everyone

that

they

can

proceed

from

the

barrier

by

sending

a

multicast

-LRB-

using

the

binomial

tree

-RRB-

Under

the

first

scheme

we

use

the

binomial

tree

constructed

using

the

U-Min

algorithm

-LSB-

-RSB-

to

perform

software

barrier

synchronization

Under

this

approach

the

worst

case

barrier

performance

occurs

when

the

process

that

arrives

at

the

barrier

last

is

a

leaf

node

in

the

binomial

tree

that

is

farthest

from

the

root

The

best

case

occurs

when

the

last

arriving

processor

is

a

leaf

node

in

the

binomial

tree

that

is

a

child

of

the

root

and

this

processor

arrives

at

its

barrier

after

the

root

has

received

intimation

from

the

other

processors

that

they

have

reached

the

barrier

Software

Scheme

with

Network

Interface

Support

-LRB-

UMin-NI

-RRB-

It

has

been

noted

that

a

completely

software

based

scheme

results

in

very

high

latency

because

every

message

reception

and

transmission

has

to

go

through

a

number

of

software

layers

before

being

processed

by

the

root

-LSB-

-RSB-

This

situation

can

be

improved

if

a

processing

unit

at

the

network

interface

can

be

programmed

to

handle

barrier

related

messages

Under

this

scheme

we

consider

the

binomial

tree-based

barrier

synchronization

scheme

described

in

the

previous

section

but

with

the

addition

of

a

smart

network

interface

This

network

interface

efficiently

handles

the

reception

and

retransmission

of

arriving

messages

to

other

nodes

in

the

binomial

tree

-LRB-

reverse

or

forward

depending

on

the

phase

of

the

barrier

-RRB-

The

effect

of

this

smart

network

interface

is

to

reduce

the

overhead

at

non-leaf

processors

of

the

binomial

tree

as

these

nodes

have

to

absorb

and

retransmit

a

number

of

messages

The

associated

processor

is

informed

that

the

barrier

can

be

crossed

when

the

network

interface

receives

the

second

phase

multicast

message

of

the

barrier

the

associated

processor

can

continue

computation

while

the

network

interface

handles

the

forwarding

of

the

multicast

message

to

other

nodes

of

the

binomial

tree

The

best

and

worst

cases

under

this

scheme

occur

in

circumstances

identical

to

those

for

the

scheme

described

in

the

previous

section

Hardware

Combining

and

Reporting

-LRB-

and

-RRB-

This

is

the

scheme

proposed

in

this

paper

For

performance

we

consider

two

flavors

of

this

scheme

-LRB-

i

-RRB-

a

scheme

where

the

messages

-LRB-

for

both

combining

phases

and

the

reporting

multicast

phase

-RRB-

are

received

and

transmitted

at

the

processors

by

software

and

-LRB-

ii

-RRB-

a

scheme

where

a

smart

network

interface

reduces

this

overhead

of

software

processing

These

network

interfaces

execute

the

reliability

protocol

at

the

root

and

non-root

participants

in

addition

to

reporting

to

the

associated

processors

that

they

may

proceed

from

the

barrier

The

worst

case

barrier

latency

under

this

scheme

-LRB-

with

or

without

network

interface

support

-RRB-

is

obtained

when

all

participants

reach

the

barrier

simultaneously

The

best

case

barrier

latency

is

obtained

when

one

of

the

processors

reaches

the

barrier

after

all

others

have

reported

that

they

have

reached

the

barrier

Analytical

Evaluation

of

Performance

Table

lists

the

symbols

that

we

use

in

the

following

discussion

We

analyze

separately

the

best

and

worst

case

performance

under

each

of

the

four

schemes

described

above

We

assume

that

the

barriers

are

being

performed

on

a

bidirectional

MIN

network

consisting

of

N

kn

nodes

interconnected

by

n

stages

of

bidirectional

k

k

switches

The

unicast

messages

use

turnaround

routing

-LSB-

-RSB-

i.e.

to

travel

between

a

source

and

a

given

destination

they

travel

up

to

the

least

common

ancestor

stage

of

the

MIN

and

then

turn

around

to

travel

to

the

destination

For

the

multidestination

messages

we

assume

a

similar

form

of

routing

the

multidestination

messages

travel

up

to

the

least

common

ancestor

stage

between

a

source

and

a

given

set

of

destinations

and

then

turn

around

getting

replicated

and

routed

to

the

destinations

on

the

way

back

We

assume

that

all

barrier

related

messages

are

packlets

of

size

one

flit

Table

Symbols

used

for

analytical

estimates

of

performance

their

meaning

and

values

used

for

experiments

in

Section

Symbol

Meaning

Values

N

Number

of

processors

in

the

system

n

Number

of

stages

of

bidi-switches

k

Size

of

bidi-switch

-LRB-

unidirectional

switch

-RRB-

lca

Least

common

ancestor

stage

for

hardware

scheme

b

Avg

number

of

ports

for

forwarding

and

receiving

messages

h

Avg

number

of

switches

crossed

by

a

unicast

message

d

Number

of

participants

in

the

barrier

ts

Software

overhead

at

processor

-LRB-

send

receive

includes

tnif

s

-RRB-

cycles

tnif

s

Start-up

overhead

at

network

interface

-LRB-

send

receive

-RRB-

cycles

tp

Time

required

for

one

flit

to

cross

a

physical

link

cycles

trt

Time

required

to

route

a

unicast

packet

at

a

switch

cycles

tsw

Time

required

for

a

flit

to

cross

the

switch

crossbar

cycle

tcp

Time

required

for

a

flit

to

be

read

from

the

input

buffer

cycle

th

Time

required

for

handling

a

packlet

cycles

tarb

Time

required

for

arbitration

between

input

and

output

cycles

tf

Time

required

for

forming

a

packlet

at

the

switch

cycles

Completely

Software

Based

Scheme

As

described

above

in

the

worst

case

a

leaf

node

in

the

binomial

tree

reaches

the

barrier

last

For

the

root

of

the

binomial

tree

to

know

that

the

barrier

is

complete

we

need

to

go

through

demessage

passing

steps

from

the

point

where

this

leaf

processor

reaches

the

barrier

Every

message

passing

step

incurs

the

latency

-LSB-

ts

-LRB-

h

-RRB-

tp

h

-LRB-

trt

tarb

tcp

tsw

-RRB-

-RSB-

The

amount

of

time

taken

for

the

root

to

get

the

information

that

all

have

reached

the

barrier

is

therefore

de

-LSB-

ts

-LRB-

h

-RRB-

tp

h

-LRB-

trt

tarb

tcp

tsw

-RRB-

-RSB-

It

takes

the

same

desteps

to

multicast

the

message

back

to

the

destinations

The

total

time

in

the

worst

case

is

therefore

TMaxUMin

de

-LSB-

ts

-LRB-

h

-RRB-

tp

h

-LRB-

trt

tarb

tcp

tsw

-RRB-

-RSB-

-LRB-

-RRB-

In

the

best

case

the

time

required

for

the

combine

phase

is

equal

to

the

time

required

for

a

single

message

to

travel

from

a

last-arriving

participant

to

the

root

This

participant

is

a

child

of

the

root

in

the

binomial

tree

The

best

case

barrier

time

is

therefore

TMinUMin

-LRB-

de

-RRB-

-LSB-

ts

-LRB-

h

-RRB-

tp

h

-LRB-

trt

tarb

tcp

tsw

-RRB-

-RSB-

-LRB-

-RRB-

Software

Based

Scheme

with

Network

Interface

Support

The

worst

case

and

best

case

barrier

performance

under

this

scheme

occurs

in

circumstances

identical

to

the

ones

presented

above

In

the

worst

case

the

leaf

processor

that

arrives

at

the

barrier

last

has

to

go

through

software

overhead

before

it

enables

to

the

network

interface

to

take

over

operations

This

software

overhead

is

the

send

portion

of

ts

-LRB-

tsend

s

-RRB-

Every

intermediate

node

in

the

binomial

tree

takes

tnif

s

to

absorb

and

retransmit

the

arriving

packlet

-LRB-

to

its

parent

or

child

in

the

binomial

tree

depending

on

the

phase

-RRB-

compared

to

ts

under

the

previous

scheme

Similarly

the

processor

that

receives

the

multicast

message

last

can

not

begin

till

the

network

interface

informs

it

that

it

can

continue

To

do

this

software

overhead

corresponding

to

the

receive

portion

of

ts

-LRB-

trecv

s

-RRB-

has

to

be

incurred

However

tsend

s

and

trecv

s

include

the

send

and

received

portions

of

tnif

s

respectively

The

worst

case

performance

under

this

scheme

can

therefore

be

obtained

by

substitution

tnif

s

for

ts

in

the

expression

for

worst

case

performance

obtained

above

and

adding

a

ts

tnif

s

factor

The

worst

case

performance

is

therefore

TMaxUMinNI

ts

tnif

s

de

-LSB-

tnif

s

-LRB-

h

-RRB-

tp

h

-LRB-

trt

tarb

tcp

tsw

-RRB-

-RSB-

-LRB-

-RRB-

A

similar

argument

can

also

be

made

for

the

best

case

barrier

performance

This

best

case

performance

is

therefore

TMinUMinNI

ts

tnif

s

-LRB-

de

-RRB-

-LSB-

tnif

s

-LRB-

h

-RRB-

tp

h

-LRB-

trt

tarb

tcp

tsw

-RRB-

-RSB-

-LRB-

-RRB-

Hardware

Combining

and

Reporting

-LRB-

without

Network

Interface

Support

-RRB-

We

break

our

analysis

of

best

and

worst

case

performance

into

an

analysis

of

the

combine

phase

and

an

analysis

of

the

time

required

for

the

multicast

phase

The

time

taken

for

the

multicast

phase

does

not

differ

in

the

best

and

worst

case

this

time

is

only

dependent

on

the

distribution

of

the

non-root

participants

and

not

on

their

respective

arrival

times

at

the

barrier

The

time

required

for

the

combine

phase

differs

in

the

best

and

worst

cases

In

the

worst

case

all

participants

reach

the

barrier

simultaneously

The

switches

therefore

have

to

deal

with

Combine

packlets

that

arrive

simultaneously

At

a

given

switch

the

time

required

for

handling

p

simultaneous

Combine

arrivals

sequentially

and

for

forwarding

a

Combine

message

is

p

th

tf

The

time

required

for

the

combine

as

well

as

multicast

phases

depends

on

the

distribution

of

the

destinations

and

the

amount

of

replication

that

has

to

be

done

to

reach

them

The

worst

case

for

the

combining

phase

is

given

by

the

following

expression

TMaxCombine

ts

-LRB-

lca

-RRB-

-LSB-

b

th

tf

tsw

tp

-RSB-

lca

-LSB-

th

tf

tsw

tp

-RSB-

tp

-LRB-

-RRB-

The

time

required

for

the

combine

phase

to

reach

the

least

common

ancestor

switch

is

given

by

tsend

s

-LRB-

lca

-RRB-

-LSB-

b?th

tf

tsw

tp

-RSB-

The

expression

-LRB-

lca

-LSB-

th

tf

tsw

tp

-RSB-

tp

trecv

s

-RRB-

gives

the

time

required

for

the

forwarded

packlet

from

the

least

common

ancestor

switch

to

reach

the

root

processor

Note

that

ts

tsend

s

trecv

s

The

time

required

for

the

multicast

phase

is

given

by

Tmcast

ts

tp

lca

-LSB-

th

tf

tsw

tp

-RSB-

-LRB-

lca

-RRB-

-LSB-

th

b

tcp

tsw

tp

-RSB-

-LRB-

-RRB-

The

time

required

for

the

multicast

to

reach

the

least

common

ancestor

switch

is

given

by

tsend

s

lca

-LSB-

th

tf

tsw

tp

-RSB-

tp

ts

and

the

time

requires

for

the

message

to

reach

the

destinations

from

this

switch

is

given

by

-LRB-

lca

-RRB-

-LSB-

th

b

tcp

tsw

tp

-RSB-

trecv

s

If

we

assume

that

the

last

process

receiving

the

multicast

crosses

the

barrier

only

after

sending

out

an

Ack

message

a

further

latency

of

tsend

s

has

to

be

incurred

The

worst

case

barrier

latency

is

obtained

by

summing

the

time

required

for

the

worst

case

combine

phase

the

time

required

for

the

multicast

phase

and

the

overhead

tsend

s

of

sending

out

the

last

Ack

TMaxCombine

Tmcast

tsend

s

-LRB-

-RRB-

Using

an

average

value

for

the

number

of

ports

used

at

a

switch

-LRB-

b

-RRB-

on

the

downward

path

from

the

least

common

ancestor

is

however

less

accurate

than

desirable

A

combination

of

experiment

and

analysis

can

give

us

a

better

estimate

Given

an

arbitrary

participant

set

the

basic

idea

is

to

set

up

a

tree

from

a

switch

in

the

least

common

ancestor

stage

of

this

set

which

has

the

non-root

participants

as

its

leaves

Starting

with

the

leaves

the

worst

case

time

taken

to

reach

the

least

common

ancestor

stage

can

be

recursively

calculated

The

worst

case

time

at

the

leaf

switches

is

first

calculated

assuming

that

all

Combine

packlets

arrive

simultaneously

from

the

participants

attached

to

the

respective

switches

The

time

at

each

switch

at

a

higher

level

in

the

tree

is

obtained

by

finding

the

worst

case

time

required

at

the

switch

assuming

that

Combine

packlets

arrive

simultaneously

along

on

the

output

ports

that

lead

to

its

children

in

the

tree

and

adding

to

this

the

maximum

of

the

latencies

at

the

child

switches

At

the

end

of

this

recursive

procedure

the

switch

at

the

least

common

ancestor

stage

has

an

estimate

of

the

worst

case

time

required

for

the

combined

message

to

reach

it

and

to

be

forwarded

towards

the

root

processor

an

actual

implementation

the

simultaneous

arrivals

can

be

handled

in

a

more

efficient

pipelined

fashion

However

we

will

assume

a

simple

non-pipelined

implementation

here

as

in

the

rest

of

the

paper

that

the

stages

of

the

MIN

are

numbered

starting

with

stage

The

nodes

are

attached

to

the

switches

of

stage

The

expression

lca

-LSB-

th

tf

tsw

tp

-RSB-

tp

accurately

depicts

the

time

taken

to

reach

the

root

processor

from

this

least

common

ancestor

switch

Since

the

tree

constructed

from

the

least

common

ancestor

switch

to

the

non-root

participants

is

dependent

on

the

distribution

of

the

destinations

the

worst

case

latencies

generated

by

the

above

method

varies

with

differing

destination

sets

For

different

participant

sets

of

a

given

size

such

worst

case

delays

may

be

estimated

and

the

maximum

among

these

chosen

for

the

worst

case

time

We

use

such

a

method

for

performance

evaluation

in

Sec.

In

the

best

case

a

non-root

participant

arrives

at

the

barrier

much

after

all

the

other

participants

The

time

required

for

the

combine

message

is

therefore

the

time

required

for

a

packlet

to

travel

from

this

last

arriving

destination

to

the

root

processor

This

time

is

given

by

substituting

b

in

the

expression

for

worst

case

combine

time

TMinCombine

ts

-LRB-

lca

-RRB-

-LSB-

th

tf

tsw

tp

-RSB-

lca

-LSB-

th

tf

tsw

tp

-RSB-

tp

-LRB-

-RRB-

The

best

case

barrier

latency

is

obtained

by

adding

the

values

for

the

best

case

combining

latency

the

multicast

phase

latency

and

the

time

to

send

an

Ack

packlet

TMinCombine

Tmcast

tsend

s

-LRB-

-RRB-

Hardware

Combining

and

Reporting

-LRB-

with

Network

Interface

Support

-RRB-

The

best

and

worst

case

performances

for

the

hardware

scheme

with

network

interface

support

-LRB-

-RRB-

is

obtained

in

a

fashion

similar

to

the

best

and

worst

case

performances

obtained

for

the

UMin-NI

scheme

ts

is

replaced

by

tnif

s

in

the

expressions

for

the

best

and

worst

case

performances

under

the

scheme

and

a

factor

of

ts

tnif

s

is

added

to

the

expressions

The

best

and

worst

case

performances

are

therefore

obtained

as

follows

-LRB-

tnif

s

-RRB-

ts

tnif

s

-LRB-

-RRB-

-LRB-

tnif

s

-RRB-

ts

tnif

s

-LRB-

-RRB-

where

-LRB-

tnif

s

-RRB-

is

the

value

of

with

tnif

s

substituted

for

ts

and

-LRB-

tnif

s

-RRB-

is

the

value

of

with

tnif

s

substituted

for

ts

Performance

Evaluation

In

this

section

we

experimentally

compare

the

time

taken

for

barrier

synchronization

under

the

four

different

schemes

described

in

the

previous

section

on

a

bidirectional

MIN

network

The

barrier

synchronization

latency

is

the

amount

of

time

that

elapses

between

from

the

point

where

the

last

process

in

the

group

of

participants

reaches

the

barrier

and

the

point

where

the

last

of

the

processors

in

the

group

has

been

informed

that

everyone

has

reached

the

barrier

The

time

taken

for

a

barrier

depends

on

the

times

at

which

the

participating

processors

reach

their

respective

barriers

In

our

performance

evaluation

we

consider

the

best

and

worst

case

performance

of

barriers

under

each

of

the

four

schemes

Experiments

The

performance

benefits

obtained

from

a

hardware

barrier

synchronization

scheme

have

already

been

studied

quite

extensively

The

purpose

of

the

experiments

in

this

section

is

to

verify

that

similar

performance

gains

are

obtained

under

the

proposed

scheme

and

to

measure

the

extent

of

these

gains

Experiments

were

performed

to

determine

the

barrier

latency

under

the

best

and

worst

cases

-LRB-

depending

on

the

arrival

of

the

participants

at

the

barrier

-RRB-

for

each

of

the

four

schemes

These

experiments

were

performed

for

a

range

of

a

system

sizes

switch

sizes

and

software

startup

overheads

For

the

software

based

schemes

-LRB-

with

and

without

network

interface

support

-RRB-

we

performed

experiments

using

a

C++

CSIM

based

simulator

testbed

For

the

hardware

schemes

we

used

the

method

described

in

the

previous

section

to

arrive

at

estimates

of

the

maximum

and

minimum

barrier

latencies

As

described

in

the

previous

section

these

estimates

are

conservative

the

actual

maxima

and

minima

are

likely

to

be

lower

For

each

set

of

system

size

switch

size

and

startup

parameters

experiments

were

performed

for

different

participant

sets

that

were

generated

using

a

random

number

generator

Every

point

in

the

curves

showing

the

maximum

and

minimum

barrier

latencies

under

each

of

the

four

schemes

were

obtained

by

taking

the

maximum

and

minimum

values

obtained

over

a

number

of

runs

-LRB-

runs

for

the

unicast

based

schemes

and

runs

for

the

hardware

based

schemes

-RRB-

The

values

of

the

various

parameters

used

in

the

experiments

are

shown

in

Table

The

cycle

time

was

assumed

to

be

nsecs

Effect

of

Startup

Overhead

We

performed

experiments

assuming

two

different

values

for

the

startup

overhead

and

secs

respectively

As

shown

in

Fig.

for

a

startup

overhead

of

ts

secs

the

performs

up

to

times

better

than

the

minimum

barrier

synchronization

time

under

UMin

for

barriers

in

a

node

system

with

bidi-switches

The

gain

is

upto

a

factor

of

when

compared

with

the

maximum

barrier

synchronization

time

under

the

UMin

scheme

The

maximum

and

minimum

values

under

-LRB-

and

-RRB-

do

not

differ

by

much

and

the

participant

set

size

does

not

greatly

affect

the

barrier

latency

The

gain

is

by

factors

of

and

when

compared

to

the

maximum

and

minimum

software

barrier

times

for

a

low

startup

overhead

of

ts

secs

for

a

similar

node

system

-LRB-

Fig.

-RRB-

To

study

the

effect

of

the

network

interface

more

closely

for

a

startup

value

of

secs

we

compared

the

performance

of

UMin-NI

and

as

shown

in

Fig.

Comparing

the

maximum

time

required

for

a

barrier

under

with

the

minimum

time

required

for

a

barrier

in

the

UMin-NI

scheme

we

notice

that

there

can

be

up

to

a

factor

of

improvement

in

performance

for

ts

secs

For

ts

secs

the

gain

is

up

to

a

factor

of

Comparing

the

maxima

of

and

UMin-NI

we

see

that

the

gains

can

be

up

to

a

factor

of

for

ts

and

up

to

a

factor

of

for

ts

secs

For

ts

performs

better

than

the

UMin-NI

for

a

node

system

with

switches

the

gain

is

a

factor

of

if

the

maximum

barrier

times

of

the

schemes

are

compared

and

a

factor

of

about

if

the

maximum

barrier

latencies

under

is

compared

with

the

minimum

barrier

latencies

under

UMin-NI

A

lower

startup

therefore

helps

the

hardware

based

scheme

with

network

interface

support

more

than

the

software

based

scheme

This

is

because

with

a

low

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

100 200 300 400

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

Figure

Effect

of

increasing

system

and

switch

size

on

barrier

latency

for

a

startup

of

ts

sec

value

for

startup

ts

becomes

approximately

equal

to

tnif

s

and

under

these

circumstances

the

hardware

based

scheme

performs

relatively

better

Effect

of

Switch

Size

The

primary

effect

of

switch

size

is

to

enlarge

the

difference

between

the

maximum

and

minimum

times

for

the

hardware

based

schemes

This

is

because

with

a

larger

switch

more

simultaneous

Combine

packlets

can

arrive

at

a

switch

during

the

combine

phase

similarly

more

copies

of

the

packlets

have

to

be

made

during

the

multicast

phase

increasing

the

time

required

for

the

last

of

the

participants

to

receive

the

packlet

However

these

increases

are

to

some

extent

offset

by

the

decrease

in

path

length

For

the

unicast

based

schemes

a

decrease

in

the

maximum

barrier

times

can

be

observed

due

to

this

decrease

in

path

length

Effect

of

System

Size

Our

experiments

show

that

in

general

the

gain

in

performance

of

a

hardware

based

scheme

increases

with

system

size

However

the

performance

gains

remain

considerable

even

for

a

small

system

size

of

nodes

Gains

by

factors

of

and

can

be

achieved

for

ts

and

ts

respectively

by

the

scheme

over

UMin

For

a

node

system

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

50 100 150 200

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

100 200 300 400

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin

Min-UMin

Max-UMin-NI

Min-UMin-NI

Figure

Effect

of

increasing

system

and

switch

size

on

barrier

latency

for

a

startup

of

ts

sec

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin-NI

Min-UMin-NI

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin-NI

Min-UMin-NI

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin-NI

Min-UMin-NI

50 100 150 200

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin-NI

Min-UMin-NI

100 200 300 400

Barrier

Latency

-LRB-

microsec

-RRB-

Number

of

participants

Barrier

Synchronization

-LRB-

ts

microsec

n

k

-RRB-

Max-UMin-NI

Min-UMin-NI

Figure

Effect

of

increasing

system

and

switch

size

on

barrier

latency

for

a

startup

of

ts

sec

considering

only

the

schemes

with

network

interface

support

the

corresponding

gains

are

-LRB-

for

ts

-RRB-

and

-LRB-

for

ts

-RRB-

while

the

gains

of

and

can

be

achieved

for

a

node

system

Conclusions

and

Future

Research

In

this

paper

we

have

proposed

schemes

to

achieve

reliable

hardware-based

barrier

synchronization

in

switch-based

networks

Reliability

is

achieved

by

incorporating

inexpensive

support

to

the

network

switches

Details

of

such

support

including

necessary

modifications

to

the

switch

logic

and

the

barrier

unit

are

presented

Messaging

protocols

to

accompany

the

modified

switch

architecture

are

also

presented

The

protocols

are

analyzed

for

their

completeness

to

handle

different

types

of

message

loss

It

is

demonstrated

that

the

complete

scheme

together

with

the

protocols

optimize

for

the

no-fault

case

while

allowing

flexibility

to

recover

from

faulty

situations

with

message

loss

The

benefits

of

the

proposed

schemes

are

compared

with

software-based

schemes

and

it

is

shown

that

an

improvement

up

to

a

factor

of

can

be

achieved

The

proposed

schemes

demonstrate

significant

potential

to

be

applied

to

switch-based

parallel

systems

especially

the

emerging

networks

of

workstations

In

such

systems

the

probability

of

messages

getting

lost

or

corrupted

is

little

higher

compared

to

the

tightlycoupled

MPPs

Thus

our

schemes

are

directly

applicable

to

such

systems

In

this

paper

we

have

only

focused

on

barrier

synchronization

It

will

be

interesting

to

see

how

reliable

schemes

for

other

collective

communication

operations

can

also

be

designed

in

a

cost-effective

manner

References

-LSB-

-RSB-

J.

B.

Andrews

C.

J.

Beckmann

and

D.

K.Poulsen

Notification

and

multicast

networks

for

synchronization

and

coherence

J.

of

Parallel

and

Distributed

Computing

-LCB-

Aug.

-LSB-

-RSB-

ATM

Forum

ATM

User-Network

Interface

Specification

Version

September

-LSB-

-RSB-

T.

S.

Axelrod

Effects

of

Synchronization

Barriers

on

Multiprocessor

Performance

Parallel

Computing

-LCB-

-LSB-

-RSB-

V.

Bala

J.

Bruck

R.

Cypher

P.

Elustondo

A.

Ho

C.-T

Ho

S.

Kipnis

and

M.

Snir

CCL

A

portable

and

tunable

collective

communication

library

for

scalable

parallel

computers

IEEE

Trans

on

Parallel

and

Distributed

Computing

-LRB-

-RRB-

-LCB-

Feb.

-LSB-

-RSB-

C.

J.

Beckmann

and

C.

D.

Polychronopoulos

Broadcast

Networks

for

Fast

Synchronization

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

W.

E.

Cohen

H.

G.

Dietz

and

J.

B.

Sponaugle

Dynamic

barrier

architecture

for

multimode

fine-grain

parallelism

using

conventional

processors

In

Int

Conf

on

Parallel

Processing

pages

I

-LCB-

-LCB-

Aug.

-LSB-

-RSB-

W.

J.

Dally

Performance

analysis

of

k-ary

n-cube

interconnection

networks

IEEE

Trans

on

Computers

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

A.

Gottlieb

R.

Grishman

C.

P.

Kruskal

K.

P.

McAuliffe

L.

Rudolph

and

M.

Snir

The

NYU

Ultracomputer

Designing

an

MIMD

Shared

Memory

Parallel

Computer

IEEE

Transaction

on

Computers

-LRB-

-RRB-

-LCB-

Feb

-LSB-

-RSB-

C.

Huang

and

P.

K.

McKinley

Communication

issues

in

parallel

computing

across

ATM

networks

IEEE

Parallel

and

Distributed

Technology

-LRB-

-RRB-

-LCB-

Winter

-LSB-

-RSB-

Y.

Huang

and

P.

K.

McKinley

Efficient

collection

operations

with

ATM

network

interface

support

In

Proc

Int

Conf

on

Parallel

Processing

pages

I

-LCB-

-LCB-

Aug.

-LSB-

-RSB-

H.

Ishihata

T.

Horie

and

T.

Shimizu

Architecture

for

the

highly

parallel

computer

Fujitsu

Scientific

and

Technical

J.

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

R.

E.

Kessler

and

J.

L.

Schwarzmeier

Cray

A

new

dimension

for

Cray

Research

In

Proc

IEEE

Int

Computer

Conf.

pages

-LCB-

Spring

-LSB-

-RSB-

C.

E.

Leiserson

et

al.

The

network

architecture

of

the

Connection

Machine

In

Proc

Symp

Parallel

Algorithms

and

Architectures

pages

-LCB-

ACM

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

J.

M.

Mellor-Crummey

and

M.

L.

Scott

Algorithms

for

scalable

synchronization

on

shared-memory

multiprocessors

ACM

Trans

on

Computer

Systems

-LRB-

-RRB-

-LCB-

Feb.

-LSB-

-RSB-

D.

K.

Panda

Fast

Barrier

Synchronization

in

Wormhole

k-ary

n-cube

Networks

with

Multidestination

Worms

In

International

Symposium

on

High

Performance

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

and

R.

Sivaram

Fast

Broadcast

and

Multicast

in

Wormhole

Multistage

Networks

with

Multidestination

Worms

Technical

Report

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

April

-LSB-

-RSB-

I.

D.

Scherson

and

C.-H

Chien

Least

common

ancestor

networks

In

Proc

Int

Parallel

Processing

Symp.

pages

-LCB-

-LSB-

-RSB-

S.

L.

Scott

Synchronization

and

communication

in

the

multiprocessor

In

ASPLOS-VII

Sept.

-LSB-

-RSB-

S.

Shang

and

K.

Hwang

Distributed

hardwired

barrier

synchronization

for

scalable

multiprocessor

clusters

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

R.

Sivaram

D.

K.

Panda

and

C.

B.

Stunkel

Efficient

Broadcast

and

Multicast

on

Multistage

Interconnection

Networks

using

Multiport

Encoding

In

Proceedings

of

the

Symposium

on

Parallel

and

Distributed

Processing

Oct

to

be

presented

-LSB-

-RSB-

C.

B.

Stunkel

D.

G.

Shea

B.

Abali

et

al.

The

high-performance

switch

IBM

System

Journal

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

K.

Verstoep

K.

Langendoen

and

H.

Bal

Efficient

Reliable

Multicast

on

Myrinet

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

-LSB-

-RSB-

H.

Xu

Y.-D

Gui

and

L.

M.

Ni

Optimal

Software

Multicast

in

Wormhole-Routed

Multistage

Networks

In

Proceedings

of

the

Supercomputing

Conference

pages

-LCB-

-LSB-

-RSB-

H.

Xu

P.

K.

McKinley

and

L.

Ni

Efficient

Implementation

of

Barrier

Synchronization

in

Wormhole-routed

Hypercube

Multicomputers

Journal

of

Parallel

and

Distributed

Computing

-LCB-

barrier synchronization
collective communication
networks of workstations

fault-tolerance
interprocessor communication
parallel computer architecture