A

Necessary

and

Sufficient

Condition

for

Deadlock-Free

Wormhole

Routing

Loren

Schwiebert

and

D.

N.

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

floren,jayasimg@cis.ohio-state.edu

April

Revised

December

of

this

work

was

done

when

this

author

was

on

leave

at

NASA

Lewis

Research

Center

Cleveland

OH

Abstract

An

important

open

problem

in

wormhole

routing

has

been

to

find

a

necessary

and

sufficient

condition

for

deadlock-free

adaptive

routing

Recently

Duato

has

solved

this

problem

for

a

restricted

class

of

adaptive

routing

algorithms

In

this

paper

a

necessary

and

sufficient

condition

is

proposed

that

can

be

used

for

any

adaptive

or

nonadaptive

routing

algorithm

for

wormhole

routing

as

long

as

only

local

information

is

required

for

routing

The

underlying

proof

technique

introduces

a

new

type

of

dependency

graph

the

channel

waiting

graph

which

omits

most

channel

dependencies

that

can

not

be

used

to

create

a

deadlock

configuration

The

necessary

and

sufficient

condition

can

be

applied

in

a

straightforward

manner

to

most

routing

algorithms

This

is

illustrated

by

proving

deadlock

freedom

for

a

partially

adaptive

nonminimal

mesh

routing

algorithm

that

does

not

require

virtual

channels

and

a

fully

adaptive

minimal

hypercube

routing

algorithm

with

two

virtual

channels

per

physical

channel

Both

routing

algorithms

are

more

adaptive

than

any

previously

proposed

routing

algorithm

with

similar

virtual

channel

requirements

Introduction

Wormhole

routing

-LSB-

-RSB-

has

become

the

switching

technique

of

choice

in

modern

distributed-memory

multiprocessors

such

as

the

Intel

Paragon

the

Cray

the

MIT

J-machine

the

Caltech

MO

SAIC

and

the

Implementations

of

wormhole

routing

typically

divide

each

message

into

packets

which

are

then

divided

into

flits

The

header

flit

of

a

packet

contains

the

routing

information

and

the

data

flits

of

the

packet

follow

the

header

flit

through

the

network

Since

the

network

treats

each

packet

as

a

separate

message

the

term

message

is

used

for

a

single

packet

When

the

header

arrives

at

an

intermediate

router

the

router

immediately

forwards

the

message

header

to

a

neighboring

router

if

an

output

channel

the

message

can

use

is

available

The

data

flits

then

follow

the

header

flit

in

a

pipelined

fashion

If

the

header

is

unable

to

proceed

because

no

appropriate

output

channel

is

free

the

router

buffers

only

a

few

flits

rather

than

the

entire

message

Since

the

data

flits

contain

no

routing

information

messages

can

not

share

channels

Hence

each

channel

in

the

path

is

reserved

from

the

time

the

header

flit

acquires

the

channel

until

the

last

flit

of

the

message

has

traversed

the

channel

Since

the

flits

of

a

message

are

forwarded

as

soon

as

possible

the

message

latency

is

largely

insensitive

to

the

distance

between

the

source

and

destination

This

is

especially

important

for

multiprocessors

with

a

high

diameter

such

as

meshes

On

the

other

hand

packet

switching

buffers

the

entire

message

at

every

intermediate

node

before

forwarding

any

part

of

the

message

Hence

wormhole

routing

has

lower

message

latency

when

there

is

little

or

no

channel

contention

In

addition

wormhole

routing

requires

only

enough

storage

at

a

router

to

buffer

a

few

flits

rather

than

the

entire

packet

These

two

properties

account

for

the

popularity

of

wormhole

routing

in

distributed-memory

multiprocessors

See

Ni

and

McKinley

-LSB-

-RSB-

for

an

in-depth

discussion

of

wormhole

routing

The

primary

drawbacks

to

wormhole

routing

are

the

message

latency

and

contention

that

can

occur

with

even

moderate

network

traffic

Whenever

a

message

is

unable

to

proceed

due

to

contention

the

header

and

data

flits

are

not

removed

from

the

network

Instead

the

message

holds

all

the

channels

it

currently

occupies

Since

all

the

channels

in

the

path

from

the

source

to

the

destination

are

held

from

the

time

they

are

acquired

until

the

entire

message

has

traversed

the

channel

-LRB-

which

is

after

the

entire

path

has

been

established

except

for

relatively

short

messages

that

fit

in

the

intermediate

channel

buffers

-RRB-

performance

degradation

due

to

contention

can

be

severe

and

message

latency

can

be

unacceptably

high

A

message

that

requires

several

channels

can

block

many

messages

while

being

transmitted

These

blocked

messages

can

in

turn

block

other

messages

which

further

increases

the

message

latency

Providing

additional

physical

channels

between

nodes

in

the

network

can

reduce

both

latency

and

contention

This

is

an

expensive

solution

however

and

can

actually

increase

message

latency

if

the

routers

are

pin-out

constrained

A

more

cost-effective

method

of

reducing

message

latency

proposed

by

Dally

-LSB-

-RSB-

is

to

allow

multiple

virtual

channels

to

share

the

same

physical

channel

Each

virtual

channel

has

a

separate

buffer

with

multiple

messages

multiplexed

over

the

same

physical

channel

Both

latency

and

contention

can

be

further

reduced

by

using

the

multiple

paths

that

exist

in

the

network

between

the

source

and

destination

nodes

However

Dally

and

Seitz

-LSB-

-RSB-

have

shown

that

since

a

message

holds

channels

until

the

entire

message

has

been

transmitted

a

routing

algorithm

with

no

restrictions

on

the

use

of

virtual

or

physical

channels

can

result

in

deadlock

The

simplest

routing

algorithms

are

nonadaptive

and

define

a

single

path

between

the

source

and

destination

Adaptive

routing

algorithms

on

the

other

hand

support

multiple

paths

between

the

source

and

destination

Adaptive

routing

algorithms

are

either

minimal

or

nonminimal

Minimal

routing

algorithms

allow

only

shortest

paths

to

be

chosen

while

nonminimal

routing

algorithms

do

not

require

messages

to

use

only

shortest

paths

Minimal

routing

algorithms

provide

higher

throughput

for

high

message

traffic

and

are

generally

simpler

to

implement

Nonminimal

routing

algorithms

are

useful

for

fault

tolerance

Gaughan

and

Yalamanchili

-LSB-

-RSB-

present

a

good

overview

of

adaptive

routing

protocols

Whether

minimal

or

nonminimal

adaptive

routing

algorithms

can

be

further

differentiated

by

the

fraction

of

shortest

paths

they

allow

Partially

adaptive

routing

algorithms

do

not

allow

all

messages

to

use

any

shortest

path

Fully

adaptive

routing

algorithms

do

allow

all

messages

to

use

any

shortest

path

Although

all

fully

adaptive

routing

algorithms

allow

a

message

to

use

any

physical

channel

that

is

part

of

a

shortest

path

different

restrictions

may

be

placed

on

the

choice

of

virtual

channels

on

that

physical

channel

Hence

not

all

fully

adaptive

routing

algorithms

are

equivalent

Some

fully

adaptive

routing

algorithms

allow

more

adaptiveness

than

others

by

placing

fewer

restrictions

on

the

choice

of

virtual

channels

Since

each

virtual

channel

needs

a

separate

buffer

and

the

virtual

channels

are

multiplexed

over

the

physical

channel

the

number

of

virtual

channels

required

by

an

adaptive

routing

algorithm

gives

a

good

approximation

of

the

hardware

cost

of

the

router

Routing

algorithms

that

require

more

virtual

channels

need

additional

router

control

logic

and

are

usually

more

complex

Multiplexing

and

scheduling

virtual

channels

on

a

physical

channel

is

more

complicated

with

additional

virtual

channels

Router

latency

and

cycle

time

also

increase

with

the

number

of

virtual

channels

-LSB-

-RSB-

so

fewer

virtual

channels

are

generally

better

Reducing

the

number

of

virtual

channels

needed

for

a

given

degree

of

adaptiveness

is

accomplished

by

using

a

less

restrictive

routing

algorithm

Conversely

when

the

same

number

of

virtual

channels

is

used

a

less

restrictive

routing

algorithm

has

better

performance

than

a

more

restrictive

routing

algorithm

-LSB-

-RSB-

Recent

research

on

adaptive

routing

algorithms

has

partially

addressed

both

of

these

issues

by

reducing

the

virtual

channel

requirements

and

imposing

fewer

restrictions

on

the

virtual

channels

A

natural

question

that

arises

is

Exactly

how

restrictive

must

the

routing

algorithm

be

to

guarantee

deadlock

freedom

In

other

words

what

is

a

necessary

and

sufficient

condition

for

deadlock-free

routing

In

this

paper

we

present

a

theoretical

result

for

minimizing

the

restrictions

imposed

for

deadlock-free

wormhole

routing

Besides

providing

a

necessary

and

sufficient

condition

for

deadlock

freedom

the

routing

algorithms

developed

using

this

proof

technique

are

substantially

less

restrictive

than

previous

routing

algorithms

The

only

restriction

we

impose

on

the

routing

algorithms

is

that

only

local

information

available

at

the

router

is

used

to

make

the

routing

decision

In

general

routing

is

done

based

solely

on

local

information

because

of

the

overhead

of

transmitting

non-local

information

and

the

additional

router

complexity

that

is

required

to

utilize

this

information

Previous

Work

Designing

deadlock-free

routing

algorithms

for

wormhole

routing

was

simplified

by

Dally

and

Seitz

with

a

proof

that

an

acyclic

channel

dependency

graph

guarantees

deadlock

freedom

-LSB-

-RSB-

Each

ver

tex

of

the

channel

dependency

graph

is

a

virtual

channel

There

is

a

directed

edge

from

one

virtual

channel

to

another

if

a

message

is

permitted

to

use

the

second

virtual

channel

immediately

after

the

first

Since

the

graph

is

acyclic

deadlock

freedom

can

be

shown

by

assigning

a

numbering

to

the

edges

of

the

graph

ensuring

that

all

channels

are

used

in

strictly

increasing

or

strictly

decreasing

order

Dally

and

Seitz

proposed

their

proof

technique

for

nonadaptive

routing

algorithms

Nonadaptive

routing

algorithms

can

be

characterized

by

functions

of

the

form

R

C

N

N

C

where

the

input

channel

belonging

to

the

set

of

channels

C

and

the

current

and

destination

node

belonging

to

the

set

of

nodes

N

define

an

output

channel

on

which

to

route

the

message

An

acyclic

channel

dependency

graph

has

also

been

used

as

a

basis

for

developing

adaptive

routing

algorithms

defined

by

relations

of

the

form

R

C

N

N

Cp

where

a

set

of

output

channels

rather

than

a

single

channel

is

defined

on

which

to

route

the

message

-LSB-

-RSB-

Glass

and

Ni

-LSB-

-RSB-

and

Boura

and

Das

-LSB-

-RSB-

have

proposed

methodologies

for

generating

deadlockfree

algorithms

but

both

proof

techniques

require

an

acyclic

channel

dependency

graph

Glass

and

Ni

propose

a

method

of

analyzing

routing

algorithms

based

on

the

permitted

and

prohibited

dependencies

from

one

channel

to

another

These

dependencies

are

characterized

as

turns

with

the

set

of

possible

turns

defined

by

the

topology

For

meshes

the

turns

can

be

turns

-LRB-

when

switching

from

a

channel

in

one

dimension

to

a

channel

in

a

different

dimension

-RRB-

ffi

turns

-LRB-

when

switching

from

one

channel

to

another

channel

in

the

same

direction

-RRB-

and

turns

-LRB-

when

switching

from

one

channel

to

a

channel

in

the

opposite

direction

of

the

same

dimension

-RRB-

The

turn

model

groups

the

turns

into

cycles

and

breaks

all

cycles

by

prohibiting

some

turns

This

is

equivalent

to

removing

edges

from

the

channel

dependency

graph

It

is

then

necessary

to

show

that

cycles

can

not

be

created

from

the

remaining

turns

based

on

the

assumption

that

an

acyclic

channel

dependency

graph

is

required

for

deadlock

freedom

Boura

and

Das

propose

a

method

of

proving

deadlock

freedom

by

partitioning

the

channels

into

two

sets

and

requiring

messages

to

route

completely

in

the

first

set

before

using

channels

in

the

second

set

Duato

-LSB-

-RSB-

proved

that

requiring

an

acyclic

channel

dependency

graph

is

too

restrictive

for

routing

algorithms

defined

by

relations

of

the

form

R

N

N

Cp

where

the

current

node

and

the

destination

node

independent

of

the

input

channel

define

a

set

of

output

channels

on

which

to

route

the

message

Cycles

are

permitted

in

the

channel

dependency

graph

if

some

subset

of

channels

defines

a

connected

routing

subfunction

with

an

acyclic

extended

channel

dependency

graph

An

extended

channel

dependency

graph

contains

both

the

direct

and

the

indirect

dependencies

Each

edge

in

the

channel

dependency

subgraph

defines

a

direct

dependency

An

indirect

dependency

is

a

dependency

between

two

channels

in

the

subgraph

that

exists

only

because

of

the

intermediate

use

of

one

or

more

channels

not

in

the

subgraph

This

methodology

has

been

used

by

Schwiebert

and

Jayasimha

-LSB-

-RSB-

to

propose

a

fully

adaptive

minimal

routing

algorithm

for

meshes

Berman

et

al.

-LSB-

-RSB-

propose

a

torus

routing

algorithm

with

a

routing

relation

of

the

form

R

C

N

N

Cp

that

allows

cyclic

dependencies

between

the

channels

Dally

and

Aoki

-LSB-

-RSB-

prove

deadlock

freedom

for

a

routing

algorithm

with

cyclic

dependencies

by

guaranteeing

an

acyclic

packet

wait-for

graph

A

packet

wait-for

graph

is

defined

dynamically

by

the

packets

in

the

network

and

contains

an

edge

from

packet

pi

to

packet

pj

if

packet

pi

is

waiting

for

a

channel

held

by

packet

pj

All

these

proof

techniques

provide

only

a

sufficient

condition

for

deadlock-free

adaptive

routing

Although

Dally

and

Seitz

proved

that

an

acyclic

channel

dependency

graph

is

a

necessary

and

sufficient

condition

for

nonadaptive

routing

algorithms

-LSB-

-RSB-

determining

what

constitutes

a

necessary

and

sufficient

condition

for

adaptive

routing

algorithms

has

remained

an

open

problem

Lin

McKinley

and

Ni

-LSB-

-RSB-

propose

a

proof

technique

based

on

the

fact

that

a

routing

algorithm

is

deadlock-free

if

of

the

channels

in

the

network

can

be

held

forever

If

every

message

that

uses

a

given

channel

is

guaranteed

to

reach

its

destination

no

matter

which

path

-LRB-

of

those

allowed

by

the

routing

algorithm

-RRB-

the

message

takes

then

a

deadlock

configuration

can

not

arise

from

the

use

of

this

channel

Since

sink

channels

can

not

be

part

of

a

deadlock

configuration

the

proof

starts

with

the

sink

channels

and

works

backward

through

the

network

If

it

is

possible

to

show

that

no

channel

can

be

held

forever

by

a

message

regardless

of

the

destination

and

path

taken

then

the

routing

algorithm

is

deadlock-free

This

proof

technique

was

proposed

as

a

necessary

and

sufficient

condition

although

Duato

points

out

that

only

sufficiency

is

proved

-LSB-

-RSB-

Duato

-LSB-

-RSB-

has

recently

proposed

a

necessary

and

sufficient

condition

for

proving

deadlock

free

dom

for

a

restricted

class

of

adaptive

routing

algorithms

This

proof

technique

applies

only

to

routing

relations

of

the

form

R

N

N

Cp

The

proof

requires

the

identification

of

a

subset

of

channels

C

which

has

an

acyclic

extended

channel

dependency

graph

Unlike

the

sufficient

condition

the

set

of

channels

in

can

differ

for

different

source-destination

pairs

Hence

Duato

introduces

the

notion

of

cross

dependencies

A

cross

dependency

is

a

dependency

from

ci

to

cj

where

ci

and

cj

are

both

in

but

for

different

source-destination

pairs

As

with

regular

dependencies

there

are

direct

cross

dependencies

and

indirect

cross

dependencies

A

routing

algorithm

is

deadlock-free

if

and

only

if

some

connected

exists

with

no

cycles

arising

from

direct

indirect

direct

cross

and

indirect

cross

dependencies

The

technique

proposed

in

this

paper

applies

to

routing

relations

of

the

form

R

C?N?N

Cp

while

Duato

's

proof

technique

is

more

restrictive

and

applies

only

to

routing

relations

of

the

form

R

N

N

Cp

The

latter

routing

relations

can

always

be

converted

to

routing

relations

of

the

former

type

by

providing

the

same

set

of

output

channels

for

every

input

channel

that

the

message

could

have

used

to

reach

that

node

-LRB-

including

input

from

the

source

when

the

source

is

the

current

node

-RRB-

However

in

general

routing

relations

of

the

form

R

C

N

N

Cp

can

not

be

converted

to

routing

relations

of

the

form

R

N

N

Cp

since

the

set

of

output

channels

can

differ

for

the

same

destination

when

the

message

arrives

on

different

input

channels

In

addition

to

requiring

the

routing

relation

to

be

of

the

form

R

N

N

Cp

Duato

also

imposes

two

further

restrictions

on

the

routing

algorithms

for

which

the

proof

technique

is

a

necessary

and

sufficient

condition

neither

of

which

is

imposed

by

our

proof

technique

First

the

routing

algorithm

must

provide

a

minimal

path

between

every

pair

of

nodes

even

for

nonminimal

routing

algorithms

Second

the

routing

algorithm

must

be

coherent

A

routing

algorithm

is

coherent

if

it

permits

every

partial

path

from

any

source

to

any

destination

to

be

used

by

the

same

source

to

reach

an

intermediate

node

on

the

path

or

by

an

intermediate

node

on

the

path

to

reach

the

same

destination

In

other

words

a

coherent

routing

algorithm

that

allows

a

message

from

processor

ni

to

nj

to

route

through

processor

nk

must

allow

a

message

to

use

the

partial

path

between

ni

and

nj

when

routing

from

processor

ni

to

processor

nk

or

from

processor

nk

to

processor

nj

Although

requiring

coherence

appears

to

be

a

modest

restriction

in

section

we

propose

two

simple

routing

algorithms

that

are

not

coherent

Assumptions

and

Definitions

Several

assumptions

and

definitions

are

introduced

to

facilitate

the

presentation

of

the

necessary

and

sufficient

condition

These

are

standard

assumptions

made

when

proving

deadlock

freedom

for

wormhole-routed

routing

algorithms

and

have

also

appeared

in

-LSB-

-RSB-

A

node

can

generate

messages

of

arbitrary

length

destined

for

any

other

node

at

any

rate

A

message

arriving

at

its

destination

is

eventually

consumed

Since

wormhole

routing

is

used

once

a

channel

queue

accepts

the

header

flit

of

a

message

it

must

accept

all

the

flits

of

the

message

before

accepting

any

flits

from

another

message

A

channel

queue

can

not

contain

flits

belonging

to

more

than

one

message

at

a

time

The

channel

must

transmit

the

tail

flit

of

the

current

message

before

the

channel

queue

accepts

the

header

flit

of

the

next

message

An

available

channel

queue

arbitrates

among

messages

simultaneously

requesting

the

channel

However

messages

already

waiting

on

a

channel

queue

are

chosen

in

an

order

that

prevents

starvation

Definition

An

interconnection

network

I

is

a

strongly

connected

directed

multigraph

I

G

-LRB-

N

C

-RRB-

where

the

vertices

ni

N

are

the

processors

and

the

arcs

ci

C

are

channels

that

connect

neighboring

processors

Each

arc

ci

can

forward

messages

from

one

processor

denoted

si

to

a

neighboring

processor

denoted

di

Definition

A

routing

algorithm

RA

on

interconnection

network

I

is

represented

by

RA

-LRB-

ni

nj

-RRB-

and

for

each

source-destination

pair

determines

the

set

of

possible

paths

taken

by

the

message

The

routing

algorithm

is

generated

by

repeated

application

of

a

routing

relation

and

a

selection

function

at

each

router

between

the

source

and

destination

of

the

message

The

routing

algorithm

may

be

adaptive

or

nonadaptive

minimal

or

nonminimal

fault-tolerant

or

not

fault-tolerant

Definition

Routing

algorithm

RA

is

prefix-closed

if

a

path

that

RA

permits

from

node

ni

to

node

nj

that

routes

through

node

nk

implies

that

RA

also

permits

the

partial

path

from

ni

to

nk

when

nk

is

the

destination

Definition

Routing

algorithm

RA

is

suffix-closed

if

a

path

that

RA

permits

from

node

ni

to

node

nj

that

routes

through

node

nk

implies

that

RA

also

permits

the

partial

path

from

nk

to

nj

when

nk

is

the

source

Every

routing

algorithm

with

a

routing

relation

of

the

form

R

N

N

Cp

is

suffix-closed

Definition

Routing

algorithm

RA

is

coherent

if

RA

is

prefix-closed

and

suffix-closed

Definition

A

routing

relation

has

the

form

R

C

N

N

Cp

and

specifies

a

set

of

output

channels

based

on

the

input

channel

the

current

node

and

the

destination

of

the

message

Definition

A

selection

function

has

the

form

S

C

Cp

F

p

C

and

chooses

a

single

output

channel

from

a

set

of

output

channels

based

on

the

input

channel

and

the

status

of

the

output

channels

F

represents

the

possible

states

of

an

output

channel

For

fault-tolerant

routing

F

ffree

busy

faultyg

otherwise

F

ffree

busyg

Definition

A

waiting

channel

is

a

channel

at

the

source

or

an

intermediate

node

for

which

the

message

waits

when

the

message

is

unable

to

proceed

because

every

channel

the

message

can

use

is

unavailable

A

message

may

have

multiple

waiting

channels

at

the

source

or

an

intermediate

node

Definition

The

channel

waiting

graph

-LRB-

CWG

-RRB-

for

a

given

routing

algorithm

RA

and

interconnection

network

I

is

a

directed

graph

CWG

G

-LRB-

C

E

-RRB-

The

vertices

of

CWG

are

the

channels

of

I

The

arcs

of

CWG

are

pairs

of

channels

that

satisfy

the

following

relationship

E

f

-LRB-

ci

cj

-RRB-

jcj

is

a

waiting

channel

for

some

message

that

has

used

cig

Note

There

is

no

requirement

that

the

message

waits

for

cj

immediately

after

using

ci

only

that

the

message

is

long

enough

to

fill

the

channel

queues

from

ci

to

cj

Otherwise

the

message

can

not

hold

channel

ci

while

waiting

for

cj

Hence

when

analyzing

edges

in

the

channel

waiting

graph

we

restrict

the

examination

to

messages

of

sufficient

length

Definition

Consider

a

path

from

a

source

to

a

destination

Routing

algorithm

RA

is

wait-connected

if

for

every

input

channel

on

a

path

there

exists

a

waiting

channel

through

which

the

message

can

be

routed

In

other

words

regardless

of

which

input

channel

the

message

uses

-LRB-

including

the

injection

channel

when

the

message

is

at

the

source

-RRB-

there

is

always

an

output

channel

for

which

the

message

can

wait

More

formally

nj

N

C

such

that

ca

RA

-LRB-

ni

nj

-RRB-

and

da

nj

and

for

ni

's

injection

channel

-LRB-

ca

-RRB-

C

such

that

the

message

from

ni

to

nj

can

wait

for

cb

immediately

after

using

ca

Note

By

definition

wormhole

routing

does

not

remove

a

message

from

the

network

when

the

message

is

unable

to

proceed

Whenever

a

message

is

blocked

it

must

be

possible

for

the

message

to

wait

for

at

least

one

output

channel

Otherwise

a

message

is

never

delivered

if

it

reaches

an

intermediate

node

where

all

the

output

channels

are

busy

Therefore

wormhole

routing

requires

a

wait-connected

routing

algorithm

Definition

A

configuration

is

an

assignment

of

messages

to

channels

The

header

and

data

flits

of

each

message

are

stored

in

the

channel

queues

and

each

channel

queue

holds

flits

from

at

most

one

message

The

leading

channel

is

the

channel

the

message

has

most

recently

acquired

and

contains

the

message

header

Any

other

channels

occupied

by

this

message

contain

only

data

flits

A

configuration

is

legal

if

each

message

in

the

configuration

occupies

one

or

more

consecutive

channels

and

the

message

header

is

stored

at

the

head

of

the

leading

channel

queue

that

the

message

occupies

each

message

occupies

only

channels

the

routing

algorithm

permits

the

message

to

use

and

the

storage

capacity

of

each

occupied

channel

has

not

been

exceeded

Definition

A

deadlock

configuration

for

routing

algorithm

RA

on

interconnection

network

I

is

a

non-empty

legal

configuration

consisting

of

a

set

of

messages

mn

n

where

each

message

mi

in

the

set

has

acquired

at

least

one

channel

The

header

flit

of

mi

has

not

reached

its

destination

and

is

unable

to

proceed

because

every

output

channel

for

mi

is

unavailable

Moreover

every

waiting

channel

for

mi

is

occupied

by

either

data

flits

of

mi

or

the

header

or

data

flits

of

another

message

in

the

set

The

data

flits

at

the

head

of

any

other

channel

queue

held

by

mi

are

unable

to

proceed

because

the

queue

of

the

next

channel

occupied

by

mi

is

full

Thus

each

message

is

blocked

and

must

wait

for

an

unavailable

waiting

channel

held

by

another

message

in

the

set

Alternatively

when

n

waits

for

a

channel

already

held

by

itself

The

set

of

messages

can

be

ordered

such

that

mi

waits

for

a

channel

used

by

mi

i

n

and

mn

waits

for

a

channel

used

by

Livelock

Freedom

Livelock

freedom

is

another

issue

that

arises

in

nonminimal

routing

algorithms

Livelock

occurs

when

a

message

is

continually

routed

from

one

channel

to

another

yet

never

reaches

its

destination

Since

minimal

routing

always

moves

the

message

closer

to

the

destination

livelock

is

only

possible

with

nonminimal

routing

Livelock

could

occur

with

a

long

message

that

is

always

routed

away

from

the

destination

however

this

eventually

results

in

deadlock

If

the

message

length

exceeds

the

storage

capacity

of

all

the

channel

queues

in

the

network

the

message

finally

arrives

at

an

intermediate

node

where

the

message

already

occupies

every

output

channel

the

message

is

permitted

to

use

Livelock

could

also

occur

when

a

short

message

is

routed

in

a

cycle

of

n

channels

and

the

message

can

fit

in

n

channel

queues

The

message

could

continually

route

to

the

same

channel

used

a

previous

time

through

the

cycle

However

by

assumption

this

message

can

not

prevent

other

messages

from

using

these

channels

when

the

channels

become

free

so

this

message

can

not

create

a

deadlock

configuration

due

solely

to

the

livelock

problem

Hence

deadlock

freedom

can

be

proved

even

for

routing

algorithms

that

are

not

livelock-free

Necessary

and

Sufficient

Condition

Most

techniques

for

proving

that

routing

algorithms

for

wormhole

routing

are

deadlock-free

require

that

the

channel

dependency

graph

be

acyclic

in

some

manner

The

idea

of

waiting

channels

was

introduced

independently

by

Lin

McKinley

and

Ni

-LSB-

-RSB-

however

the

methodology

used

in

this

paper

is

novel

The

channel

dependency

graph

describes

the

order

in

which

channels

can

be

used

However

from

definition

it

is

clear

that

any

deadlock

configuration

is

based

on

the

waiting

channels

rather

than

the

channels

a

message

could

use

The

routing

algorithm

may

allow

a

message

to

use

a

channel

when

the

channel

is

free

even

if

the

message

is

not

permitted

to

wait

for

this

channel

when

the

channel

is

busy

This

is

our

motivation

for

using

the

CWG

since

it

ignores

dependencies

that

can

not

result

in

deadlock

Since

the

channel

waiting

graph

is

a

subset

of

the

channel

dependency

graph

requiring

an

acyclic

channel

waiting

graph

is

less

restrictive

than

requiring

an

acyclic

channel

dependency

graph

Theorem

If

routing

algorithm

RA

is

wait-connected

and

the

CWG

for

RA

is

acyclic

then

RA

is

deadlock-free

Proof

RA

is

wait-connected

so

every

message

always

has

a

waiting

channel

when

all

output

channels

are

busy

Assume

there

is

a

deadlock

configuration

involving

n

messages

If

n

then

there

is

a

cycle

in

the

CWG

from

a

channel

to

itself

which

is

not

possible

since

the

CWG

is

acyclic

Otherwise

-LRB-

n

-RRB-

there

is

an

edge

in

the

CWG

from

every

channel

used

by

mi

to

the

channel

used

by

mi

for

which

mi

is

waiting

-LRB-

call

this

channel

-RRB-

There

is

also

an

edge

in

the

CWG

from

every

channel

used

by

mn

to

the

channel

used

by

for

which

mn

is

waiting

-LRB-

call

this

channel

-RRB-

Hence

there

is

an

edge

in

the

CWG

from

to

-LRB-

n

-RRB-

and

from

to

However

the

CWG

for

RA

is

acyclic

so

no

such

set

of

edges

is

possible

Therefore

no

deadlock

configuration

exists

and

RA

is

deadlock-free

An

edge

in

the

CWG

requires

only

the

existence

of

some

path

from

a

channel

to

a

waiting

channel

The

specific

intermediate

channels

used

between

a

channel

and

a

waiting

channel

are

not

considered

when

creating

the

CWG

It

is

possible

that

a

cycle

arises

from

the

simultaneous

use

of

the

same

channel

by

more

than

one

message

For

this

reason

we

divide

cycles

in

the

CWG

into

two

classes

False

Resource

Cycles

and

True

Cycles

A

False

Resource

Cycle

is

a

cycle

in

the

CWG

that

requires

at

least

one

channel

to

be

used

simultaneously

by

more

than

one

message

in

order

to

create

the

cycle

Obviously

a

False

Resource

Cycle

can

not

occur

since

this

is

physically

impossible

Therefore

a

False

Resource

Cycle

can

not

be

used

to

create

a

deadlock

configuration

A

True

Cycle

is

a

cycle

in

the

CWG

that

permits

every

message

in

the

cycle

to

use

different

channels

In

section

we

provide

a

more

complete

description

of

False

Resource

Cycles

and

present

a

technique

for

distinguishing

between

False

Resource

Cycles

and

True

Cycles

c

n

n

n

n

Figure

Duato

's

Example

of

an

Incoherent

Routing

Algorithm

To

illustrate

the

difference

between

False

Resource

Cycles

and

True

Cycles

Duato

's

example

-LSB-

-RSB-

of

an

incoherent

routing

algorithm

is

presented

The

processors

and

channels

are

shown

in

figure

The

routing

algorithm

permits

only

minimal

routing

with

the

exception

of

channel

Channel

can

be

used

by

a

message

destined

for

only

node

Clearly

this

routing

algorithm

is

not

coherent

since

a

message

from

to

can

be

routed

through

using

channel

However

a

message

from

to

can

not

use

channel

The

CWG

for

this

routing

algorithm

has

a

False

Resource

Cycle

and

a

True

Cycle

A

message

whose

input

channel

is

can

wait

for

or

If

the

message

waits

for

there

is

a

True

Cycle

from

to

that

uses

Otherwise

the

True

Cycle

is

a

cycle

from

to

that

uses

There

is

a

False

Resource

Cycle

that

involves

two

messages

A

message

that

uses

and

and

waits

for

and

a

message

that

uses

and

and

waits

for

Obviously

this

False

Resource

Cycle

exists

only

because

both

messages

simultaneously

use

which

is

impossible

A

message

is

unable

to

proceed

when

all

output

channels

the

message

is

permitted

to

use

are

busy

This

situation

can

be

resolved

in

one

of

two

different

ways

-LRB-

-RRB-

The

message

could

wait

for

a

specific

output

channel

to

become

free

or

-LRB-

-RRB-

The

message

could

wait

until

any

permitted

output

channel

becomes

free

For

case

-LRB-

-RRB-

the

routing

algorithm

can

not

choose

an

arbitrary

channel

for

which

to

wait

but

must

choose

a

channel

for

which

waiting

is

permitted

Although

it

is

possible

that

the

routing

algorithm

has

a

choice

of

more

than

one

waiting

channel

once

a

waiting

channel

is

chosen

the

message

must

wait

for

that

specific

channel

to

become

free

For

case

-LRB-

-RRB-

the

routing

algorithm

also

has

the

possibility

of

waiting

on

just

a

subset

of

more

than

one

output

channel

In

fact

case

-LRB-

-RRB-

includes

any

routing

algorithm

that

does

not

conform

to

case

-LRB-

-RRB-

That

is

any

routing

algorithm

that

does

not

select

a

specific

waiting

channel

and

wait

for

that

channel

until

it

becomes

free

We

first

prove

a

necessary

and

sufficient

condition

for

routing

algorithms

that

belong

to

case

-LRB-

-RRB-

followed

by

a

necessary

and

sufficient

condition

for

routing

algorithms

that

belong

to

case

-LRB-

-RRB-

Theorem

A

routing

algorithm

RA

that

requires

a

message

to

wait

for

a

specific

output

channel

is

deadlock-free

iff

RA

is

wait-connected

and

the

CWG

for

RA

has

no

True

Cycles

Proof

First

notice

that

RA

is

wait-connected

by

definition

From

theorem

an

acyclic

CWG

is

a

sufficient

condition

for

deadlock

freedom

A

False

Resource

Cycle

can

not

result

in

deadlock

so

any

False

Resource

Cycles

can

be

ignored

Since

there

are

no

True

Cycles

the

routing

algorithm

is

deadlock-free

To

prove

necessity

assume

that

a

True

Cycle

with

n

messages

exists

A

deadlock

configuration

can

be

created

from

this

True

Cycle

For

each

i

n

allow

message

mi

to

use

channel

some

additional

channels

if

necessary

and

then

wait

to

acquire

channel

in

use

by

message

mi

-LRB-

Assume

that

mi

and

are

defined

as

before

-RRB-

Similarly

message

mn

uses

channel

and

waits

for

channel

Since

each

message

mi

in

the

cycle

uses

different

channels

it

is

possible

to

generate

a

set

of

messages

that

are

able

to

use

the

appropriate

channel

-LRB-

s

-RRB-

and

then

wait

for

the

appropriate

channel

To

force

mi

to

wait

for

the

appropriate

channel

it

is

necessary

to

guarantee

that

any

output

channel

mi

could

use

is

busy

For

any

output

channel

available

to

mi

that

is

also

available

to

the

source

assume

the

source

has

injected

a

message

that

is

using

this

channel

However

since

RA

is

of

the

form

R

C

N

N

Cp

it

is

possible

that

some

of

the

output

channels

available

to

mi

can

be

used

only

by

messages

arriving

on

the

input

channel

used

by

mi

For

these

output

channels

assume

that

a

previous

message

mj

used

this

input

channel

and

was

forwarded

on

one

of

the

output

channels

In

addition

the

length

of

mj

is

assumed

to

be

short

enough

that

it

releases

the

input

channel

that

mi

uses

however

mj

is

long

enough

that

it

occupies

the

output

channel

at

this

node

By

assumption

mj

can

remain

in

the

network

-LRB-

and

thus

hold

this

output

channel

-RRB-

for

an

indefinite

amount

of

time

Hence

it

is

always

possible

to

force

mi

to

wait

for

Clearly

each

message

in

the

set

is

waiting

for

a

channel

used

by

another

message

in

the

set

and

of

the

messages

can

make

progress

Therefore

a

deadlock

configuration

can

always

be

constructed

from

a

True

Cycle

For

routing

algorithms

that

permit

a

message

to

wait

for

any

of

the

output

channels

to

become

free

an

acyclic

CWG

is

not

a

necessary

condition

Since

each

blocked

message

can

select

any

output

channel

messages

may

be

able

to

avoid

channels

that

form

cycles

in

the

CWG

by

using

an

alternative

channel

that

is

not

part

of

a

cycle

However

deadlock

can

be

avoided

only

if

at

least

one

of

the

output

channels

is

guaranteed

to

become

free

For

this

reason

we

selectively

remove

edges

from

the

CWG

to

resolve

all

True

Cycles

as

long

as

the

routing

algorithm

for

the

resulting

graph

called

remains

wait-connected

We

next

prove

that

if

no

such

exists

then

the

routing

algorithm

is

not

deadlock-free

However

if

such

a

does

exist

then

the

following

theorem

can

be

used

to

prove

deadlock

freedom

Theorem

A

routing

algorithm

RA

that

permits

a

message

to

wait

for

any

output

channel

is

deadlock-free

iff

RA

is

wait-connected

for

some

and

this

has

no

True

Cycles

Proof

If

RA

is

wait-connected

for

the

CWG

and

the

CWG

has

no

True

Cycles

then

the

result

follows

immediately

from

theorem

with

CWG

Assume

the

CWG

contains

True

Cycles

In

this

case

RA

must

be

wait-connected

for

some

without

True

Cycles

We

first

prove

sufficiency

Consider

a

potential

deadlock

configuration

for

RA

involving

a

cycle

of

n

messages

-LRB-

n

-RRB-

This

requires

that

every

message

in

the

configuration

is

waiting

for

a

channel

held

by

itself

or

another

message

in

the

cycle

However

the

message

actually

waits

for

all

the

output

channels

Since

RA

is

wait-connected

for

at

least

one

of

the

waiting

channels

for

each

message

is

in

An

output

channel

in

eventually

becomes

free

and

some

message

in

the

set

is

forwarded

However

there

is

no

guarantee

that

the

output

channel

the

message

mi

actually

acquires

is

a

channel

in

-LRB-

It

is

possible

that

mi

is

forwarded

along

a

different

channel

before

the

output

channel

in

becomes

free

-RRB-

If

mi

has

reached

its

destination

then

the

cycle

has

been

resolved

Otherwise

whether

or

not

mi

acquires

a

channel

in

mi

can

acquire

an

output

channel

in

at

the

next

router

because

RA

is

wait-connected

for

Hence

one

of

the

messages

can

always

be

routed

and

a

deadlock

configuration

can

not

occur

We

now

prove

necessity

by

showing

that

the

routing

algorithm

is

not

deadlock-free

if

there

is

no

wait-connected

without

True

Cycles

No

wait-connected

without

True

Cycles

exists

so

it

is

possible

to

generate

a

set

of

messages

that

have

no

waiting

channel

that

is

guaranteed

to

become

available

Furthermore

these

messages

are

all

blocking

each

other

since

otherwise

it

would

be

possible

to

guarantee

that

a

waiting

channel

does

become

free

Hence

this

set

of

messages

form

a

deadlock

configuration

However

since

the

routing

algorithm

is

deadlock-free

no

deadlock

configurations

are

possible

Therefore

a

wait-connected

without

True

Cycles

must

exist

It

is

possible

to

prove

that

the

incoherent

routing

algorithm

previously

discussed

is

deadlockfree

by

using

the

necessary

and

sufficient

condition

just

proposed

The

routing

algorithm

is

not

deadlock-free

if

a

message

waits

for

a

specific

channel

to

become

free

Consider

two

messages

one

from

to

and

one

from

to

Assume

the

message

from

to

uses

and

and

the

message

from

to

uses

and

If

this

second

message

waits

for

then

a

deadlock

configuration

occurs

If

this

message

instead

waits

for

then

a

deadlock

configuration

occurs

when

the

first

message

uses

instead

of

and

the

second

message

uses

instead

of

Since

the

routing

algorithm

does

not

know

which

channels

have

already

been

used

by

a

message

when

selecting

a

waiting

channel

the

routing

algorithm

is

not

deadlock-free

if

the

message

waits

for

a

specific

channel

On

the

other

hand

if

the

message

waits

for

both

channels

and

then

by

theorem

the

routing

algorithm

is

deadlock-free

Figure

depicts

the

CWG

for

this

routing

algorithm

As

discussed

previously

there

are

True

Cycles

and

a

False

Resource

Cycle

in

this

channel

waiting

graph

cc

cc

c

Figure

The

CWG

for

the

Incoherent

Routing

Algorithm

The

CWG

has

True

Cycles

however

it

is

possible

to

create

a

with

no

True

Cycles

since

the

routing

algorithm

is

deadlock-free

Edges

are

removed

from

the

CWG

to

create

using

the

following

observations

ffl

Since

any

message

waiting

for

is

guaranteed

to

reach

and

both

and

can

be

used

only

for

messages

whose

destination

is

no

message

is

required

to

wait

for

Therefore

the

routing

algorithm

remains

wait-connected

for

even

if

all

edges

to

are

removed

from

the

CWG

ffl

A

message

that

arrives

on

after

using

eventually

acquires

since

the

message

using

is

destined

for

either

or

If

the

destination

is

then

the

message

has

reached

its

destination

and

will

become

free

If

the

destination

is

then

the

message

using

is

either

using

or

waiting

for

Since

any

message

waiting

for

eventually

reaches

eventually

becomes

free

Similarly

a

message

that

arrives

on

after

using

eventually

acquires

Therefore

the

edge

from

to

and

the

edge

from

to

can

be

removed

from

to

create

a

new

and

the

routing

algorithm

is

still

waitconnected

for

this

new

Figure

depicts

after

these

two

modifications

have

been

made

There

are

no

True

Cycles

in

The

only

cycle

in

is

a

False

Resource

Cycle

from

to

and

back

to

which

requires

both

messages

in

the

cycle

to

use

Since

RA

is

wait-connected

for

deadlock

freedom

follows

immediately

from

theorem

c

cc

cc

Figure

The

final

for

the

Incoherent

Routing

Algorithm

False

Resource

Cycles

The

CWG

is

a

static

graph

however

the

dependencies

that

arise

among

the

channels

are

dynamic

False

Resource

Cycles

capture

this

notion

of

dynamic

dependencies

among

the

channels

When

two

edges

in

the

CWG

both

require

the

use

of

a

common

channel

then

these

two

dependencies

can

not

occur

simultaneously

Cycles

in

the

CWG

that

are

formed

from

such

dependencies

can

not

occur

in

reality

and

hence

can

not

lead

to

a

deadlock

configuration

False

Resource

Cycles

could

arise

with

minimal

or

nonminimal

routing

algorithms

Duato

's

incoherent

routing

algorithm

is

an

example

of

the

latter

We

now

present

an

example

of

a

False

n

n

n

n

nnnnn

n

Figure

A

False

Resource

Cycle

for

Minimal

Routing

Resource

Cycle

using

minimal

routing

The

routing

algorithm

is

presented

for

the

ring

-LRB-

torus

-RRB-

in

figure

The

routing

algorithm

permits

a

message

to

use

V

if

the

destination

of

the

message

is

an

even

numbered

node

and

V

if

the

destination

is

an

odd

numbered

node

A

message

on

V

or

V

stays

on

V

or

V

respectively

with

two

exceptions

-LRB-

-RRB-

After

a

message

uses

a

wraparound

channel

the

message

switches

from

V

Ci

to

V

Ci

-LRB-

mod

-RRB-

and

-LRB-

-RRB-

A

message

that

uses

V

at

node

uses

V

if

the

destination

is

an

odd

numbered

node

and

V

if

the

destination

is

an

even

numbered

node

It

is

possible

to

create

a

cycle

in

the

CWG

only

if

two

messages

arrive

at

node

and

both

messages

leave

on

V

Clearly

both

messages

can

acquire

V

at

the

same

time

so

this

is

a

False

Resource

Cycle

There

are

no

True

Cycles

in

the

CWG

so

the

routing

algorithm

is

deadlock-free

In

order

to

create

a

cycle

each

message

in

the

set

mi

must

acquire

before

mi

arrives

at

This

is

always

possible

with

a

True

Cycle

since

each

message

uses

different

channels

A

False

Resource

Cycle

however

requires

that

at

least

two

messages

in

the

cycle

share

a

channel

A

False

Resource

Cycle

can

arise

in

one

of

two

ways

Either

a

channel

in

the

cycle

is

shared

or

a

channel

outside

the

cycle

is

shared

by

at

least

two

messages

in

the

cycle

prior

to

entering

the

cycle

If

the

shared

channel

is

part

of

the

cycle

then

each

message

mi

that

uses

the

shared

channel

has

already

acquired

Hence

the

False

Resource

Cycle

would

be

a

True

Cycle

if

mi

could

reach

without

using

a

shared

channel

On

the

other

hand

if

the

shared

channel

is

used

prior

to

the

channels

in

the

cycle

then

mi

can

not

acquire

before

mi

However

the

False

Resource

Cycle

would

be

a

True

Cycle

if

mi

could

reach

without

using

a

shared

channel

The

first

possibility

is

a

False

Resource

Cycle

involving

only

the

channels

that

form

the

cycle

We

need

to

consider

channels

at

only

those

nodes

that

are

used

by

more

than

one

message

in

the

cycle

A

message

that

can

form

its

part

of

the

cycle

without

using

any

node

used

by

another

message

in

the

cycle

can

not

be

using

a

shared

channel

Hence

ignore

any

message

that

can

route

through

nodes

that

no

other

message

in

the

cycle

can

use

Also

ignore

any

message

that

can

route

using

channels

that

no

other

message

in

the

cycle

can

use

even

if

other

messages

share

a

node

with

this

message

For

the

remaining

messages

see

if

a

channel-disjoint

path

can

be

found

for

each

message

This

is

done

by

arbitrarily

choosing

a

path

for

one

of

the

remaining

messages

Continue

selecting

channel-disjoint

paths

for

the

other

remaining

messages

If

a

channel-disjoint

path

can

be

found

for

each

message

then

there

is

no

False

Resource

Cycle

formed

with

the

channels

used

in

the

cycle

Otherwise

backtrack

selecting

a

different

channel-disjoint

path

for

the

previous

message

If

all

possible

paths

have

been

considered

and

there

is

still

no

channel-disjoint

path

for

all

messages

then

this

is

a

False

Resource

Cycle

The

second

possibility

is

a

False

Resource

Cycle

created

only

when

at

least

two

messages

share

a

channel

that

is

used

prior

to

the

cycle

This

is

possible

only

with

routing

algorithms

that

are

not

suffix-closed

because

if

a

routing

algorithm

is

suffix-closed

the

cycle

can

be

formed

without

using

channels

outside

the

cycle

For

these

routing

algorithms

first

ignore

any

message

mi

that

can

route

from

to

without

using

any

channels

before

Next

ignore

any

message

that

could

route

to

by

using

nodes

that

no

other

message

in

the

cycle

can

use

Also

ignore

any

message

that

can

route

using

channels

that

no

other

message

in

the

cycle

can

use

For

the

remaining

messages

consider

the

paths

that

a

message

could

take

to

reach

Proceed

as

in

the

previous

case

selecting

channel-disjoint

paths

and

backtracking

as

necessary

If

a

channel-disjoint

path

can

be

found

for

each

message

then

this

is

a

True

Cycle

Otherwise

this

cycle

may

be

a

False

Resource

Cycle

We

do

not

have

an

algorithmic

method

of

determining

this

A

message

that

uses

a

shared

channel

outside

the

cycle

can

be

short

enough

to

release

this

channel

after

it

has

been

used

and

hold

only

the

channels

that

form

this

message

's

part

of

the

cycle

Hence

even

though

the

channel

is

shared

by

more

than

one

message

in

the

cycle

the

channel

could

be

shared

consecutively

rather

than

only

simultaneously

The

cycle

is

a

True

Cycle

if

the

cycle

can

be

formed

even

when

all

the

shared

channels

are

used

consecutively

rather

than

simultaneously

However

if

even

one

shared

channel

must

be

used

simultaneously

by

more

than

one

message

in

the

cycle

then

this

is

a

False

Resource

Cycle

Identifying

In

this

section

a

formal

design

methodology

for

reducing

the

CWG

to

is

described

This

methodology

is

needed

only

for

routing

algorithms

that

do

not

require

a

message

to

wait

for

a

specific

output

channel

The

design

methodology

requires

the

identification

of

all

cycles

in

the

CWG

Since

the

problem

of

counting

the

number

of

cycles

in

a

directed

graph

is

NP-hard

-LSB-

-RSB-

an

exponential

time

algorithm

is

used

for

finding

Other

general

techniques

for

proving

deadlock

freedom

also

require

exponential

time

in

the

worst

case

-LSB-

-RSB-

The

algorithm

for

reducing

the

CWG

to

has

the

following

steps

Create

a

list

L

of

all

cycles

in

the

CWG

Each

entry

in

L

contains

three

sets

the

set

of

edges

that

form

the

cycle

the

set

of

edges

in

the

cycle

the

algorithm

has

attempted

to

remove

and

the

set

of

edges

that

are

currently

removed

from

the

cycle

Initially

for

all

cycles

in

L.

Remove

all

False

Resource

Cycles

from

L

This

is

done

by

examining

each

cycle

in

L

using

the

method

described

in

section

or

by

informally

examining

CWG

Then

number

the

True

Cycles

TCi

-LRB-

i

:-RRB-

and

let

i

Remove

an

edge

from

TCi

as

long

as

the

routing

algorithm

remains

wait-connected

Any

edge

that

can

not

be

removed

with

the

routing

algorithm

still

wait-connected

is

added

to

If

no

edge

in

TCi

can

be

removed

with

the

routing

algorithm

still

wait-connected

then

backtrack

to

TCi

Before

backtracking

reset

for

TCi

Otherwise

TCi

can

be

resolved

so

add

this

edge

to

and

for

every

cycle

in

L

that

contains

this

edge

Increment

i

and

select

a

True

Cycle

TCi

from

L

with

Repeat

steps

until

all

True

Cycles

in

L

have

been

resolved

or

the

routing

algorithm

has

backtracked

to

and

the

algorithm

has

already

attempted

to

remove

every

edge

If

all

True

Cycles

in

L

have

been

resolved

then

the

routing

algorithm

is

deadlock-free

and

consists

of

the

edges

that

have

not

been

removed

Otherwise

no

acyclic

exists

for

this

routing

algorithm

so

the

routing

algorithm

is

not

deadlock-free

Routing

Examples

In

order

to

demonstrate

the

usefulness

of

the

necessary

and

sufficient

condition

a

partially

adaptive

nonminimal

routing

algorithm

for

n-dimensional

meshes

is

proposed

The

routing

algorithm

does

not

require

any

virtual

channels

and

does

not

have

an

acyclic

channel

dependency

graph

Our

proof

technique

is

also

used

to

show

deadlock

freedom

for

a

fully

adaptive

minimal

hypercube

routing

algorithm

The

necessary

and

sufficient

condition

is

then

used

to

prove

that

any

relaxation

of

the

restrictions

imposed

by

the

routing

algorithm

introduces

the

possibility

of

deadlock

Deadlock

freedom

can

not

be

proved

for

either

routing

algorithm

using

Duato

's

necessary

and

sufficient

condition

Previous

Routing

Algorithms

Torus

and

hypercube

topologies

can

be

characterized

as

k-ary

n-cubes

where

k

is

the

radix

and

n

is

the

dimension

For

example

an

hypercube

is

a

and

a

torus

is

a

An

n-dimensional

mesh

is

similar

to

a

torus

except

a

mesh

does

not

have

wrap-around

channels

Partially

adaptive

routing

algorithms

for

wormhole-routed

hypercubes

and

meshes

have

been

proposed

by

many

authors

-LSB-

-RSB-

Boura

and

Das

-LSB-

-RSB-

as

well

as

Glass

and

Ni

-LSB-

-RSB-

have

proposed

methods

for

producing

partially

adaptive

routing

algorithms

for

hypercubes

and

meshes

The

partially

adaptive

routing

algorithm

for

hypercubes

proposed

by

Li

-LSB-

-RSB-

has

the

addi

tional

advantage

of

ensuring

that

the

multiple

paths

are

edge-disjoint

for

many

source-destination

pairs

All

three

algorithms

require

only

a

single

virtual

channel

per

physical

channel

In

section

we

show

how

to

increase

the

adaptiveness

in

a

mesh

without

virtual

channels

by

using

a

routing

algorithm

that

permits

cycles

in

the

channel

dependency

graph

The

hypercube

routing

algorithm

proposed

by

Draper

and

Ghosh

-LSB-

-RSB-

uses

two

virtual

channels

for

each

physical

channel

Each

message

routes

in

dimension

order

along

the

first

set

of

channels

but

may

skip

some

dimensions

in

which

the

message

needs

to

route

The

message

then

routes

in

dimension

order

along

the

second

set

of

channels

The

message

can

no

longer

skip

dimensions

and

must

wait

for

the

channels

to

become

free

The

hypercube

routing

algorithm

proposed

by

Yang

and

Tsai

-LSB-

-RSB-

also

requires

two

virtual

channels

per

physical

channel

A

message

first

uses

any

dimension

in

which

it

needs

to

route

in

a

positive

direction

When

the

message

finishes

with

all

such

dimensions

or

finds

them

all

busy

the

message

repeats

this

process

for

all

negative

directions

The

message

then

switches

to

the

second

set

of

virtual

channels

and

routes

first

in

all

remaining

positive

directions

and

then

all

negative

directions

waiting

for

busy

channels

when

necessary

A

fully

adaptive

hypercube

routing

algorithm

has

been

proposed

by

several

authors

-LSB-

-RSB-

This

routing

algorithm

requires

two

virtual

channels

per

physical

channel

A

message

routes

in

dimension

order

along

the

first

set

of

virtual

channels

Each

message

also

has

the

possibility

of

routing

in

any

dimension

that

moves

the

message

closer

to

the

destination

along

the

second

set

of

virtual

channels

Besides

routing

algorithms

proposed

specifically

for

hypercubes

any

routing

algorithm

for

arbitrary

dimension

meshes

or

tori

can

be

extended

to

hypercubes

since

an

n-dimensional

hypercube

is

a

special

case

of

an

n-dimensional

mesh

or

torus

Only

a

few

fully

adaptive

routing

algorithms

have

been

designed

for

n-dimensional

mesh

and

torus

topologies

-LSB-

-RSB-

When

restricted

to

hypercubes

the

routing

algorithms

proposed

by

Jesshope

Miller

and

Yantchev

-LSB-

-RSB-

and

by

Linder

and

Harden

-LSB-

-RSB-

require

more

virtual

channels

than

the

routing

algorithm

proposed

by

Duato

-LSB-

-RSB-

The

routing

algorithm

proposed

by

Dally

and

Aoki

-LSB-

-RSB-

is

more

restrictive

than

the

routing

algorithm

proposed

by

Duato

The

routing

algorithm

proposed

by

Berman

et

al.

-LSB-

-RSB-

when

restricted

to

hypercubes

is

equivalent

to

the

routing

algorithm

proposed

by

Duato

The

routing

algorithm

proposed

by

Schwiebert

and

Jayasimha

-LSB-

-RSB-

when

modified

for

hypercubes

is

more

adaptive

than

the

routing

algorithm

proposed

by

Duato

The

routing

algorithm

proposed

by

Duato

-LSB-

-RSB-

uses

the

first

set

of

virtual

channels

for

nonadaptive

routing

and

the

second

set

of

virtual

channels

for

fully

adaptive

routing

In

section

we

show

how

to

increase

adaptiveness

by

using

the

first

set

of

virtual

channels

for

partially

adaptive

routing

Mesh

Routing

Algorithm

We

now

propose

a

partially

adaptive

nonminimal

routing

algorithm

for

meshes

which

does

not

require

an

acyclic

channel

dependency

graph

and

does

not

require

any

virtual

channels

-LRB-

For

a

mesh

this

routing

algorithm

is

similar

to

north-last

proposed

by

Glass

and

Ni

-LSB-

-RSB-

although

our

routing

algorithm

permits

messages

to

make

more

use

of

turns

-RRB-

This

new

routing

algorithm

called

the

Highest

Positive

Last

routing

algorithm

is

defined

below

Let

h

be

the

highest

dimension

in

which

the

message

still

needs

to

route

in

the

negative

direction

to

reach

the

destination

The

following

set

of

restrictions

is

applied

to

the

channels

ffl

A

message

can

use

any

channel

in

a

dimension

lower

than

h

even

if

the

message

does

not

need

to

route

in

that

direction

so

nonminimal

routing

is

permitted

A

message

can

also

route

in

the

negative

direction

of

dimension

h.

ffl

A

message

that

needs

to

route

in

only

positive

directions

must

use

the

positive

channels

in

increasing

dimension

order

although

the

message

is

permitted

to

misroute

in

the

negative

direction

of

a

higher

dimension

if

desired

ffl

A

message

is

allowed

to

make

a

turn

from

the

negative

direction

to

the

positive

direction

of

the

same

dimension

when

the

message

needs

to

route

in

the

positive

direction

A

message

is

permitted

to

make

a

turn

from

the

positive

direction

to

the

negative

direction

of

the

same

dimension

only

when

the

message

needs

to

route

in

the

negative

direction

of

this

dimension

and

some

higher

dimension

ffl

If

all

output

channels

a

message

can

use

are

busy

the

message

waits

for

the

channel

in

the

negative

direction

of

dimension

h

If

the

message

needs

to

route

only

in

positive

directions

then

the

message

waits

for

the

channel

in

the

positive

direction

of

the

lowest

dimension

in

which

the

message

needs

to

route

The

Highest

Positive

Last

routing

algorithm

has

a

routing

relation

of

the

form

R

C?N?N

Cp

For

example

consider

a

message

that

needs

to

route

only

East

and

North

When

the

message

is

due

South

of

its

destination

the

message

is

permitted

to

route

South

if

the

input

channel

was

East

However

the

message

can

not

route

South

if

the

input

channel

was

North

-LRB-

If

this

restriction

is

not

imposed

the

routing

algorithm

is

not

deadlock-free

-RRB-

Also

routing

algorithms

of

this

type

are

not

coherent

-LSB-

-RSB-

because

a

message

can

route

West

and

immediately

route

East

The

message

has

now

routed

through

the

same

processor

twice

However

a

routing

algorithm

is

not

coherent

if

a

message

can

route

through

any

processor

more

than

once

Furthermore

the

routing

algorithm

is

not

coherent

even

for

minimal

paths

For

example

on

an

mesh

a

message

from

-LRB-

-RRB-

to

-LRB-

-RRB-

can

route

through

-LRB-

-RRB-

However

a

message

from

-LRB-

-RRB-

to

-LRB-

-RRB-

can

not

route

through

-LRB-

-RRB-

Finally

the

routing

algorithm

does

not

have

an

acyclic

channel

dependency

graph

but

does

have

an

acyclic

channel

waiting

graph

For

example

a

message

that

needs

to

route

in

the

negative

direction

of

the

Z

dimension

can

use

the

channels

in

the

X

and

Y

dimensions

in

any

order

but

always

waits

for

the

channel

in

the

negative

direction

of

the

Z

dimension

Duato

's

proof

technique

requires

the

existence

of

an

acyclic

extended

channel

dependency

graph

for

some

connected

subset

of

the

channels

-LSB-

-RSB-

Since

there

is

just

one

channel

between

neighboring

nodes

and

the

channel

dependency

graph

has

cycles

Duato

's

proof

technique

can

not

be

applied

to

the

Highest

Positive

Last

routing

algorithm

The

routing

algorithm

is

substantially

more

adaptive

than

any

previously

proposed

mesh

routing

algorithm

that

requires

only

one

channel

Previous

partially

adaptive

routing

algorithms

have

required

an

acyclic

channel

dependency

graph

For

example

Glass

and

Ni

-LSB-

-RSB-

propose

the

negativefirst

routing

algorithm

for

n-dimensional

meshes

The

negative-first

routing

algorithm

requires

messages

to

route

in

all

negative

directions

and

then

all

positive

directions

The

negative-first

routing

algorithm

also

permits

nonminimal

routing

on

the

negative

channels

before

using

any

positive

channels

Glass

and

Ni

prove

that

an

acyclic

channel

dependency

graph

prohibits

at

least

n

-LRB-

n

-RRB-

turns

for

an

n-dimensional

mesh

With

negative-first

routing

a

message

can

not

use

a

negative

channel

after

using

a

positive

channel

so

each

of

the

n

positive

channels

is

prohibited

from

using

the

negative

channel

in

the

other

n

dimensions

The

Highest

Positive

Last

routing

algorithm

prevents

a

message

from

using

a

channel

in

a

lower

dimension

after

using

the

positive

channel

in

a

higher

dimension

which

restricts

n

-LRB-

n

-RRB-

turns

However

the

restrictions

imposed

by

the

Highest

Positive

Last

routing

algorithm

are

not

absolute

A

message

can

use

a

positive

channel

before

a

channel

in

a

lower

dimension

whenever

the

message

needs

to

route

in

the

negative

direction

of

some

still

higher

dimension

This

is

a

significant

relaxation

of

the

routing

restrictions

especially

on

the

positive

channel

in

the

lower

dimensions

Although

the

positive

channel

in

dimension

i

can

not

be

used

before

the

positive

or

negative

channel

in

a

lower

dimension

this

restriction

is

removed

when

the

message

also

needs

to

route

in

the

negative

direction

of

any

dimension

higher

than

i

The

lower

the

dimension

the

more

likely

this

restriction

can

be

avoided

Theorem

The

Highest

Positive

Last

routing

algorithm

is

deadlock-free

Proof

Since

the

routing

algorithm

requires

a

message

to

wait

for

a

specific

output

channel

the

necessary

and

sufficient

condition

proved

in

theorem

is

used

to

prove

deadlock

freedom

Any

cycle

in

the

CWG

requires

at

least

two

dimensions

since

a

message

routing

in

the

positive

direction

can

not

wait

for

a

channel

in

the

negative

direction

without

first

routing

in

the

negative

direction

of

another

dimension

Consider

any

potential

cycle

in

the

CWG

involving

the

channels

used

by

the

messages

that

form

the

cycle

It

is

possible

to

partition

this

cycle

into

two

hyperplanes

by

dividing

the

cycle

along

the

highest

dimension

used

in

the

cycle

For

the

cycle

to

exist

some

message

must

use

a

channel

in

the

positive

direction

of

the

highest

dimension

in

the

cycle

and

then

wait

for

a

channel

in

the

positive

half

of

the

cycle

In

other

words

some

message

must

cross

the

partition

in

the

positive

direction

and

then

wait

for

a

channel

in

this

half

of

the

partition

for

the

cycle

to

exist

However

a

message

is

not

permitted

to

use

a

channel

in

the

positive

direction

and

then

wait

for

a

channel

in

a

lower

dimension

without

first

using

a

channel

in

the

negative

direction

of

a

higher

dimension

Since

the

message

has

already

used

the

positive

direction

of

the

highest

dimension

in

the

cycle

the

message

has

not

used

the

negative

channel

in

a

still

higher

dimension

and

this

channel

is

not

held

by

any

other

message

in

the

set

Thus

no

message

in

the

cycle

can

satisfy

this

requirement

and

the

CWG

is

acyclic

Deadlock

freedom

follows

immediately

from

theorem

Hypercube

Routing

Algorithm

The

following

conventions

are

used

in

this

section

Channels

are

assumed

to

be

bidirectional

virtual

channels

V

Cin

is

used

to

denote

virtual

channel

n

in

the

direction

of

dimension

i.

For

example

V

is

virtual

channel

one

in

the

negative

direction

of

the

third

dimension

An

asterisk

in

the

superscript

denotes

all

dimensions

Thus

V

C

denotes

the

second

virtual

channel

in

the

negative

direction

of

all

dimensions

Each

processor

of

an

n-dimensional

hypercube

can

be

uniquely

labeled

using

an

n

bit

string

The

source

of

a

message

is

denoted

S

-LRB-

sn

sn

-RRB-

and

the

destination

is

denoted

D

-LRB-

dn

dn

-RRB-

A

message

routes

from

the

source

to

the

destination

by

routing

in

dimensions

in

which

the

corresponding

bit

in

the

source

differs

from

that

of

the

destination

The

message

routes

in

the

positive

direction

of

dimension

i

if

si

and

di

Similarly

the

message

routes

in

the

negative

direction

of

dimension

i

if

si

and

di

With

minimal

routing

a

message

routes

in

each

dimension

at

most

once

and

a

message

does

not

route

in

dimension

i

if

si

di

We

now

propose

a

fully

adaptive

minimal

routing

algorithm

for

hypercubes

All

previous

fully

adaptive

routing

algorithms

specifically

for

hypercubes

require

that

the

first

set

of

virtual

channels

be

used

only

for

nonadaptive

routing

This

new

routing

algorithm

permits

the

first

set

of

virtual

channels

to

be

used

for

partially

adaptive

routing

The

routing

algorithm

called

the

Enhanced

Fully

Adaptive

routing

algorithm

is

defined

by

the

following

steps

ffl

Assign

two

bidirectional

virtual

channels

to

each

dimension

ffl

Allow

a

message

to

route

along

the

second

virtual

channel

at

any

time

Let

l

be

the

lowest

dimension

in

which

the

message

still

needs

to

route

The

following

set

of

restrictions

is

applied

to

the

first

set

of

virtual

channels

ffl

A

message

that

needs

to

route

in

the

negative

direction

of

dimension

l

can

use

any

of

the

first

set

of

virtual

channels

ffl

A

message

that

needs

to

route

in

the

positive

direction

of

dimension

l

must

use

V

C

ffl

If

all

output

channels

a

message

can

use

are

busy

the

message

waits

for

V

C

The

routing

algorithm

is

substantially

more

adaptive

than

any

previously

proposed

fully

adaptive

hypercube

routing

algorithm

The

Enhanced

Fully

Adaptive

routing

algorithm

restricts

a

message

from

using

the

first

virtual

channel

in

the

positive

direction

after

using

the

first

virtual

channel

in

a

higher

dimension

However

a

message

can

use

the

first

virtual

channel

in

a

higher

dimension

first

whenever

the

message

needs

to

route

in

the

negative

direction

of

the

lowest

dimension

in

which

the

message

still

needs

to

route

This

is

a

significant

relaxation

of

the

routing

restrictions

especially

compared

with

using

dimension

order

routing

for

the

first

set

of

virtual

channels

Degree

of

Adaptiveness

Hypercube

Dimensions

Comparison

of

Enhanced

e-cube

and

Duato

's

Algorithms

Enhanced

Duato

e-cube

Figure

Degree

of

Adaptiveness

for

Hypercube

Routing

Algorithms

The

degree

of

adaptiveness

is

the

proportion

of

virtual

paths

permitted

by

the

routing

algorithm

compared

to

a

routing

algorithm

with

no

restrictions

averaged

over

all

source-destination

pairs

-LSB-

-RSB-

The

degree

of

adaptiveness

for

both

routing

algorithms

is

shown

in

figure

For

comparison

the

degree

of

adaptiveness

for

e-cube

-LRB-

nonadaptive

dimension

order

-RRB-

routing

is

also

shown

Paradoxically

the

degree

of

adaptiveness

is

not

zero

for

nonadaptive

routing

This

occurs

because

nonadaptive

routing

always

allows

one

of

paths

and

the

degree

of

adaptiveness

would

be

zero

only

if

there

were

no

permitted

paths

There

are

relatively

few

paths

when

the

source

is

near

the

destination

so

even

a

single

path

can

represent

a

relative

large

proportion

of

the

paths

For

example

when

the

distance

between

the

source

and

destination

is

two

nonadaptive

routing

can

use

half

the

paths

c

n

n

n

n

c

cc

c

c

c

c

n

n

n

n

c

c

cc

c

c

c

c

c

c

c

cc

cc

Figure

Hypercube

with

Virtual

Channels

per

Physical

Channel

Duato

's

necessary

and

sufficient

condition

-LSB-

-RSB-

can

not

be

used

to

prove

that

the

Enhanced

Fully

Adaptive

routing

algorithm

is

deadlock-free

Duato

's

proof

technique

requires

the

existence

of

an

acyclic

extended

channel

dependency

graph

for

some

connected

subset

of

the

channels

Regardless

of

which

channels

are

chosen

there

are

cycles

formed

by

direct

dependencies

among

the

channels

Consider

the

hypercube

in

figure

The

routing

algorithm

places

no

restrictions

on

channel

usage

for

messages

routing

from

to

from

to

and

from

to

Also

a

message

from

to

can

use

any

channel

between

and

before

it

routes

from

to

Of

course

this

is

not

a

counter-example

to

Duato

's

necessary

and

sufficient

condition

since

the

Enhanced

Fully

Adaptive

routing

algorithm

is

not

coherent

For

example

a

message

from

to

can

route

through

using

the

first

virtual

channel

However

a

message

from

to

can

not

route

through

using

the

first

virtual

channel

Theorem

The

Enhanced

Fully

Adaptive

routing

algorithm

is

deadlock-free

Proof

This

proof

uses

the

necessary

and

sufficient

condition

proved

in

theorem

RA

is

waitconnected

since

a

message

is

always

permitted

to

wait

for

virtual

channel

one

in

the

lowest

dimension

in

which

the

message

needs

to

route

Since

a

message

can

wait

for

only

V

C

any

cycle

in

the

CWG

must

be

created

from

waiting

dependencies

among

the

V

C

channels

-LRB-

although

V

C

channels

could

be

used

to

create

these

dependencies

-RRB-

Since

minimal

routing

is

used

any

cycle

in

the

CWG

requires

at

least

two

dimensions

and

must

use

both

directions

of

each

dimension

in

the

cycle

Let

l

be

the

lowest

dimension

of

the

potential

cycle

A

cycle

in

the

CWG

requires

one

of

two

situations

either

a

message

mi

waits

for

V

C

after

using

V

C

in

a

higher

dimension

of

the

cycle

or

mi

uses

V

C

or

V

C

after

using

V

C

in

a

higher

dimension

and

then

waits

for

V

C

in

a

different

higher

dimension

Since

l

is

the

lowest

dimension

in

the

cycle

mi

does

not

need

to

route

in

any

dimension

lower

than

l

after

using

V

C

in

a

higher

dimension

Otherwise

mi

would

be

waiting

for

a

channel

in

this

lower

dimension

and

l

would

not

be

the

lowest

dimension

in

the

cycle

Clearly

neither

situation

can

occur

since

mi

can

not

use

V

C

in

a

higher

dimension

when

mi

needs

to

route

in

V

C

l

and

l

is

the

lowest

dimension

in

which

mi

needs

to

route

Therefore

the

routing

algorithm

is

deadlock-free

Theorem

No

restrictions

imposed

on

the

Enhanced

Fully

Adaptive

routing

algorithm

can

be

relaxed

without

permitting

a

deadlock

configuration

Proof

There

are

no

restrictions

on

the

use

of

the

second

set

of

virtual

channels

so

the

restrictions

on

only

the

first

set

of

virtual

channels

must

be

considered

The

only

restriction

on

the

first

set

of

virtual

channels

is

that

a

channel

in

a

higher

dimension

can

not

be

used

by

a

message

that

needs

to

route

in

the

positive

direction

of

the

lowest

dimension

in

which

the

message

needs

to

route

Without

loss

of

generality

let

l

be

the

lowest

dimension

and

assume

that

the

channel

used

in

a

higher

dimension

is

the

first

virtual

channel

in

the

positive

direction

of

dimension

i

Then

there

is

an

edge

in

the

CWG

from

V

to

V

C

The

CWG

already

has

edges

from

V

C

to

V

from

V

to

V

C

and

from

V

C

to

V

These

edges

form

a

True

Cycle

A

message

waits

for

only

V

C

and

the

routing

algorithm

is

not

deadlock-free

if

V

C

is

not

a

waiting

channel

Thus

from

theorem

the

routing

algorithm

is

not

longer

deadlock-free

Conclusion

A

necessary

and

sufficient

condition

for

proving

deadlock

freedom

for

both

nonadaptive

and

adaptive

routing

algorithms

using

wormhole

routing

has

been

proposed

The

usefulness

of

this

proof

technique

has

been

demonstrated

with

a

partially

adaptive

nonminimal

routing

algorithm

for

ndimensional

meshes

This

routing

algorithm

is

more

adaptive

than

any

other

routing

algorithm

proposed

for

meshes

without

additional

channels

The

proof

technique

described

in

this

paper

has

also

been

used

to

prove

deadlock

freedom

for

a

fully

adaptive

minimal

routing

algorithm

for

hypercubes

This

new

hypercube

routing

algorithm

is

substantially

more

adaptive

than

any

previous

fully

adaptive

routing

algorithm

for

hypercubes

The

necessary

and

sufficient

condition

can

also

be

used

to

develop

adaptive

routing

algorithms

for

the

torus

and

other

topologies

We

have

shown

that

the

restrictions

on

the

Enhanced

Fully

Adaptive

routing

algorithm

can

not

be

relaxed

without

creating

a

deadlock

configuration

It

is

possible

although

unlikely

that

a

routing

algorithm

with

a

different

set

of

restrictions

could

be

deadlock-free

while

being

less

restrictive

The

number

of

restrictions

can

not

be

reduced

however

since

there

is

only

one

restriction

for

each

pair

of

dimensions

A

partial

result

has

been

given

for

distinguishing

between

False

Resource

Cycles

and

True

Cycles

This

technique

can

be

applied

to

any

suffix-closed

routing

algorithm

which

represents

the

all

but

a

very

restricted

class

of

routing

algorithms

In

practice

the

identification

of

False

Resource

Cycles

should

not

require

resorting

to

this

formal

technique

A

formal

approach

is

provided

however

for

cases

where

the

irregularity

of

the

interconnection

network

or

the

routing

algorithm

prevent

a

straightforward

analysis

The

proof

of

theorem

requires

the

identification

of

This

is

needed

only

for

routing

algorithms

that

do

not

require

a

message

to

wait

for

a

specific

output

channel

In

general

it

should

be

straightforward

to

determine

for

regular

topologies

such

as

k-ary

n-cubes

and

meshes

Most

existing

topologies

are

regular

and

this

regularity

is

usually

incorporated

into

the

routing

algorithm

However

a

formal

design

methodology

has

been

provided

for

those

cases

where

it

is

difficult

to

reduce

the

CWG

to

This

automates

the

task

of

proving

deadlock

freedom

and

should

be

of

use

for

routing

algorithm

designers

Acknowledgments

The

authors

thank

Jos?e

Duato

for

many

helpful

suggestions

The

authors

also

thank

Yu-Chee

Tseng

for

insightful

comments

on

the

idea

of

channel

waiting

Jeff

May

for

valuable

discussions

concerning

the

necessary

and

sufficient

condition

and

Kant

Patel

Dave

Lutz

and

Anish

Arora

for

many

thoughtful

comments

which

have

improved

the

quality

of

the

paper

References

-LSB-

-RSB-

P.

Berman

L.

Gravano

G.

Pifarr?e

and

J.

Sanz

Adaptive

Deadlock

and

Livelock-Free

Routing

With

All

Minimal

Paths

in

Torus

Networks

In

Proceedings

of

the

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

-LSB-

-RSB-

Y.

M.

Boura

and

C.

R.

Das

A

Class

of

Partially

Adaptive

Routing

Algorithms

for

n

dimensional

Meshes

In

International

Conference

on

Parallel

Processing

volume

III

pages

-LSB-

-RSB-

A.

A.

Chien

A

Cost

and

Speed

Model

for

k-ary

n-cube

Wormhole

Routers

In

Hot

Interconnects

August

-LSB-

-RSB-

A.

A.

Chien

and

J.

H.

Kim

Planar-Adaptive

Routing

Low-cost

Adaptive

Networks

for

Multiprocessors

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

W.

J.

Dally

Fine-Grain

Message-Passing

Concurrent

Computers

In

Proceedings

of

the

Third

Conference

on

Hypercube

Concurrent

Computers

volume

pages

-LSB-

-RSB-

W.

J.

Dally

Virtual-Channel

Flow

Control

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

March

-LSB-

-RSB-

W.

J.

Dally

and

H.

Aoki

Deadlock-Free

Adaptive

Routing

in

Multicomputer

Networks

Using

Virtual

Channels

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

April

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

Deadlock-Free

Message

Routing

in

Multiprocessor

Interconnection

Networks

IEEE

Transactions

on

Computers

-LRB-

-RRB-

May

-LSB-

-RSB-

J.

T.

Draper

and

J.

Ghosh

Multipath

E-Cube

Algorithms

-LRB-

MECA

-RRB-

for

Adaptive

Wormhole

Routing

and

Broadcasting

in

k-ary

n-cubes

In

International

Parallel

Processing

Symposium

pages

-LSB-

-RSB-

J.

Duato

On

the

Design

of

Deadlock-Free

Adaptive

Routing

Algorithms

for

Multicomputers

Design

Methodologies

In

Parallel

Architectures

and

Languages

Europe

volume

I

pages

-LSB-

-RSB-

J.

Duato

A

New

Theory

of

Deadlock-Free

Adaptive

Routing

in

Wormhole

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

December

-LSB-

-RSB-

J.

Duato

A

Necessary

and

Sufficient

Condition

for

Deadlock-Free

Adaptive

Routing

in

Wormhole

Networks

In

International

Conference

on

Parallel

Processing

volume

I

pages

-LSB-

-RSB-

J.

Duato

Private

Communication

May

-LSB-

-RSB-

M.

R.

Garey

and

D.

S.

Johnson

Computers

and

Intractability

A

Guide

to

the

Theory

of

NP

completeness

W.

H.

Freeman

and

Company

New

York

NY

-LSB-

-RSB-

P.

T.

Gaughan

and

S.

Yalamanchili

Adaptive

Routing

Protocols

for

Hypercube

Interconnection

Networks

IEEE

Computer

-LRB-

-RRB-

May

-LSB-

-RSB-

C.

Glass

and

L.

M.

Ni

Adaptive

Routing

in

Mesh-Connected

Networks

In

International

Conference

on

Distributed

Computing

Systems

pages

-LSB-

-RSB-

C.

Glass

and

L.

M.

Ni

Maximally

Fully

Adaptive

Routing

in

Meshes

In

International

Conference

on

Parallel

Processing

volume

I

pages

-LSB-

-RSB-

C.

Glass

and

L.

M.

Ni

The

Turn

Model

for

Adaptive

Routing

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

L.

Gravano

G.

Pifarr?e

G.

Denicolay

and

J.

Sanz

Adaptive

Deadlock-free

Worm-hole

Routing

in

Hypercubes

In

International

Parallel

Processing

Symposium

pages

-LSB-

-RSB-

C.

R.

Jesshope

P.

R.

Miller

and

J.

T.

Yantchev

High

Performance

Communications

in

Processor

Networks

In

Proceedings

of

the

Annual

International

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

Q.

Li

Minimum

Deadlock-Free

Message

Routing

Restrictions

in

Binary

Hypercubes

Journal

of

Parallel

and

Distributed

Computing

-LRB-

-RRB-

June

-LSB-

-RSB-

X.

Lin

P.

K.

McKinley

and

L.

M.

Ni

The

Message

Flow

Model

for

Routing

in

WormholeRouted

Networks

In

International

Conference

on

Parallel

Processing

volume

I

pages

-LSB-

-RSB-

D.

H.

Linder

and

J.

C.

Harden

An

Adaptive

and

Fault

Tolerant

Wormhole

Routing

Strategy

for

k-ary

n-cubes

IEEE

Transactions

on

Computers

-LRB-

-RRB-

January

-LSB-

-RSB-

J.

May

D.

N.

Jayasimha

and

K.

Patel

Comparison

of

Multiplexing

Schemes

for

WormholeRouted

Distributed

Memory

Multiprocessors

In

International

Workshop

on

Parallel

Processing

December

To

appear

-LSB-

-RSB-

L.

M.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

-LRB-

-RRB-

February

-LSB-

-RSB-

L.

Schwiebert

and

D.

N.

Jayasimha

Optimal

Fully

Adaptive

Wormhole

Routing

for

Meshes

In

Supercomputing

pages

An

extended

version

of

this

result

to

appear

in

the

Journal

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

C.-C

Su

and

K.

G.

Shin

Adaptive

Deadlock-Free

Routing

in

Multicomputers

Using

Only

One

Extra

Virtual

Channel

In

International

Conference

on

Parallel

Processing

volume

I

pages

-LSB-

-RSB-

C.

S.

Yang

and

Y.

M.

Tsai

Fault-Tolerant

and

Deadlock-Free

Wormhole

Routing

in

the

Hypercube

Network

In

International

Conference

on

Parallel

and

Distributed

Systems

pages

channel waiting graph
deadlock freedom
necessary and sufficient condition
routing algorithms
wormhole routing

hypercube architectures
mesh architectures