Parallelizing

Tabled

Evaluations

Extended

Abstract

Juliana

Freire

Rui

Hu

y

Terrance

Swift

David

S.

Warren

z

Department

of

Computer

Science

State

University

of

New

York

at

Stony

Brook

Stony

Brook

NY

Email

fjuliana,ruihu,tswift,warreng@cs.sunysb.edu

Phone

-LRB-516-RRB- 632-8470

Abstract

SLG

is

a

table-oriented

resolution

method

that

extends

SLD

evaluation

in

two

ways

It

computes

the

well-founded

model

for

logic

programs

with

negation

with

polynomial

data

complexity

and

it

terminates

for

programs

with

the

bounded-term-size

property

Furthermore

SLG

has

an

efficient

sequential

implementation

for

modularly

stratified

programs

in

the

SLG-WAM

of

XSB

This

paper

addresses

general

issues

involved

in

parallelizing

tabled

evaluations

by

introducing

a

model

of

shared-memory

parallelism

which

we

call

table

parallelism

and

by

comparing

it

to

traditional

models

of

parallelizing

SLD

A

basic

architecture

for

supporting

table

parallelism

in

the

framework

of

the

SLG-WAM

is

also

presented

along

with

an

algorithm

for

detecting

termination

of

subcomputations

Introduction

The

deficiencies

of

SLD

resolution

are

well

known

and

extended

efforts

have

been

made

to

remedy

these

deficiencies

For

instance

while

SLD

can

be

combined

efficiently

with

negation-by-failure

in

SLDNF

the

semantics

of

SLDNF

have

proven

unacceptable

for

many

purposes

in

particular

for

non-monotonic

reasoning

Even

without

negation

SLD

is

susceptible

to

infinite

loops

and

redundant

subcomputations

making

it

unacceptable

for

deductive

databases

The

latter

deficiency

that

of

repeating

subcomputations

has

given

rise

to

many

systems

which

table

subcomputations

OLDT

-LSB-

-RSB-

and

SLG-AL

-LSB-

-RSB-

and

SLG

Partially

supported

by

CAPES-Brazil

ySupported

by

NSF

grant

zPartially

supported

by

grants

NSF

and

NSF

CCR

-LSB-

-RSB-

are

three

tabling

methods

which

have

been

implemented

At

an

abstract

level

systems

which

use

magic

evaluation

can

be

thought

of

tabling

systems

as

well

Substantiation

for

this

claim

stems

both

from

the

asymptotic

results

of

-LSB-

-RSB-

and

the

experimental

results

of

-LSB-

-RSB-

Tabling

also

appears

to

be

relevant

for

computing

the

well-founded

semantics

besides

SLG

well-founded

ordered

search

-LSB-

-RSB-

and

the

tabulated

resolution

of

-LSB-

-RSB-

are

two

recent

proposals

which

also

use

tabling

Due

to

the

power

of

tabling

many

approaches

have

been

formulated

for

its

implementation

indeed

-LSB-

-RSB-

cites

dozens

Nearly

all

of

these

approaches

are

sequential

however

We

propose

an

abstract

model

called

table-parallelism

for

parallelizing

tabled

evaluations

In

a

sequential

framework

SLG

and

SLD

can

be

combined

both

theoretically

and

practically

the

SLG-WAM

evaluates

SLD

resolution

with

minimal

overhead

and

allows

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

free

intermixture

of

SLD

and

SLG

predicates

There

is

every

reason

to

believe

that

when

SLG

is

parallelized

it

will

be

possible

to

mix

table-parallelism

with

and

and

or

SLD

parallelism

While

table-parallelism

can

exploit

or-parallelism

and

some

and-parallelism

present

in

programs

it

is

orthogonal

to

both

as

will

become

clear

In

parallelizing

practical

programs

it

is

envisioned

that

a

combination

of

all

three

approaches

will

prove

beneficial

just

as

a

combination

of

SLG

and

SLD

works

best

for

practical

sequential

programs

The

idea

behind

table-parallelism

is

simple

the

table

can

be

thought

of

as

a

large

structured

buffer

through

which

cooperating

threads

communicate

Example

table

p

-LRB-

a

b

-RRB-

p

-LRB-

a

d

-RRB-

p

-LRB-

b

c

-RRB-

q

-LRB-

b

-RRB-

q

-LRB-

c

-RRB-

a

-LRB-

X

Z

-RRB-

a

-LRB-

X

Y

-RRB-

a

-LRB-

Y

Z

-RRB-

a

-LRB-

X

Z

-RRB-

p

-LRB-

X

Z

-RRB-

q

-LRB-

Z

-RRB-

a

-LRB-

a

Z

-RRB-

As

a

motivation

for

the

parallel

SLG

model

consider

the

SLG

forest

in

Figure

for

the

program

of

Example

Each

SLG-subgoal

can

be

thought

of

starting

a

new

thread

Thus

the

call

to

a

-LRB-

b

Z

-RRB-

in

node

begins

a

new

thread

as

does

the

call

to

a

-LRB-

c

Z

-RRB-

in

node

Nodes

like

or

which

use

program

clause

resolution

to

produce

answers

are

called

generator

nodes

while

nodes

like

or

which

perform

SLD

resolution

are

called

interior

nodes

Since

each

thread

is

itself

an

independent

SLG-tree

each

will

have

its

own

stacks

as

well

as

its

own

heap

and

trail

The

table

space

will

be

kept

in

shared-memory

as

will

the

completion

stack

kept

for

determining

when

to

complete

an

SLG-subgoal

At

a

broad

level

the

parallel

SLG-WAM

differs

from

its

sequential

counterpart

-LSB-

-RSB-

in

that

it

decouples

the

return

of

answers

from

the

scheduling

of

their

resolution

and

in

its

use

of

a

more

complicated

completion

For

in-memory

queries

the

SLG-WAM

appears

to

be

the

fastest

sequential

tabling

implementation

currently

available

-LRB-

-LSB-

-RSB-

-RRB-

and

we

believe

its

speed

is

due

to

its

use

of

WAM-style

Prolog

compilation

technology

This

paper

sketches

a

basic

architecture

for

a

shared-memory

SLG-WAM

as

described

in

-LSB-

-RSB-

uses

a

tuple-at-a-time

strategy

for

resolving

answers

with

subgoals

While

this

strategy

is

efficient

for

logic

programming

an

alternate

semi-naive

strategy

has

also

been

implemented

for

executing

calls

to

disk

parallel

SLG-WAM

for

definite

programs

an

architecture

whose

implementation

is

currently

under

development

We

also

discuss

in

detail

how

to

detect

when

a

tabled

subcomputation

has

been

completed

In

a

tabled

evaluation

a

given

predicate

may

be

part

of

a

mutually

dependent

set

of

subgoals

called

a

strongly

connected

component

or

The

ability

to

dynamically

determine

both

the

membership

of

subgoals

in

an

SCC

and

to

detect

when

an

SCC

has

been

completed

is

necessary

both

to

evaluate

programs

with

negation

and

to

efficiently

evaluate

definite

programs

In

a

parallel

framework

the

detection

of

completion

has

much

in

common

with

termination

detection

for

concurrent

systems

and

we

make

use

of

results

from

concurrency

theory

in

the

proof

of

our

algorithms

correctness

The

rest

of

this

paper

is

organized

as

follows

In

Section

we

describe

the

parallel

execution

model

for

SLG

The

relations

between

table-parallelism

and

other

forms

of

parallelism

found

in

Prolog

is

discussed

in

Section

The

issue

of

completion

detection

is

addressed

in

Section

and

in

Section

the

implementation

framework

of

the

parallel

SLG-WAM

is

fully

presented

Table-Parallelism

Tabling

systems

in

general

be

they

SLG

OLDT

Magic

or

anything

else

have

four

types

of

actions

not

found

in

SLD

evaluation

Creation

of

a

new

tabled

subgoal

Addition

of

an

answer

to

a

table

Consumption

of

an

answer

from

a

table

Determining

when

a

set

of

subgoals

is

completely

evaluated

While

the

details

of

parallelizing

these

operations

in

a

WAM

framework

are

presented

in

Sections

and

the

fundamental

idea

behind

table-parallelism

is

to

parallelize

at

creation

of

new

tabled

subgoals

-LRB-

Or

as

a

practical

matter

at

creation

of

certain

tabled

subgoals

-RRB-

New

threads

for

tabled

subgoals

are

rooted

in

generator

nodes

and

called

generator

threads

they

generate

answers

and

copy

them

into

the

table

The

originating

term

recursive

component

is

also

sometimes

used

for

SCCs

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

a

-LRB-

a

Z

-RRB-

answer

a

-LRB-

a

b

-RRB-

a

-LRB-

a

Z

-RRB-

active

a

-LRB-

a

Z

-RRB-

a

-LRB-

b

Z

-RRB-

a

-LRB-

a

Z

-RRB-

active

a

-LRB-

a

Z

-RRB-

a

-LRB-

a

Y

-RRB-

a

-LRB-

Y

Z

-RRB-

a

-LRB-

a

Z

-RRB-

answer

a

-LRB-

a

c

-RRB-

a

-LRB-

a

Z

-RRB-

interior

a

-LRB-

a

Z

-RRB-

p

-LRB-

a

Z

-RRB-

q

-LRB-

Z

-RRB-

-RRB-

a

-LRB-

b

Z

-RRB-

answer

a

-LRB-

b

c

-RRB-

a

-LRB-

c

Z

-RRB-

active

a

-LRB-

c

Z

-RRB-

a

-LRB-

c

Y

-RRB-

a

-LRB-

Y

Z

-RRB-

a

-LRB-

c

Z

-RRB-

active

a

-LRB-

c

Z

-RRB-

p

-LRB-

c

Y

-RRB-

q

-LRB-

Y

-RRB-

a

-LRB-

a

Z

-RRB-

interior

a

-LRB-

a

b

-RRB-

q

-LRB-

b

-RRB-

a

-LRB-

a

d

-RRB-

interior

a

-LRB-

a

d

-RRB-

q

-LRB-

d

-RRB-

a

-LRB-

a

Z

-RRB-

generator

a

-LRB-

a

Z

-RRB-

a

-LRB-

b

Z

-RRB-

generator

a

-LRB-

b

Z

-RRB-

a

-LRB-

b

Z

-RRB-

active

a

-LRB-

b

Z

-RRB-

a

-LRB-

b

Y

-RRB-

a

-LRB-

Y

Z

-RRB-

a

-LRB-

b

Z

-RRB-

interior

a

-LRB-

b

Z

-RRB-

p

-LRB-

b

Z

-RRB-

q

-LRB-

Z

-RRB-

a

-LRB-

b

Z

-RRB-

interior

a

-LRB-

b

c

-RRB-

q

-LRB-

c

-RRB-

a

-LRB-

b

Z

-RRB-

active

a

-LRB-

b

Z

-RRB-

a

-LRB-

c

Z

-RRB-

a

-LRB-

c

Z

-RRB-

generator

a

-LRB-

c

Z

-RRB-

Subgoal

Answer

a

-LRB-

a

X

-RRB-

a

-LRB-

a

b

-RRB-

a

-LRB-

a

c

-RRB-

a

-LRB-

b

X

-RRB-

a

-LRB-

c

X

-RRB-

a

-LRB-

b

c

-RRB-

Figure

SLG

forest

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

Table

Space

Completion

Stack

Subgoal

Dependency

Lists

Glob.Local

Trail

Choice

Glob.Local

Trail

Choice

Point

Point

Figure

Shared

Memory

Layout

active

node

will

asynchronously

consume

answers

as

they

are

added

to

the

table

as

noted

in

Section

Since

communication

is

through

the

table

there

are

clear

benefits

to

sharing

the

table

Likewise

the

engine

must

dynamically

detect

when

a

set

of

subgoals

is

mutually

dependent

as

well

as

when

such

a

set

has

been

completely

evaluated

Thus

the

completion

stack

which

supports

this

decision

is

also

kept

in

shared

memory

In

order

to

support

efficient

completion

detection

a

further

structure

the

subgoal

dependency

list

will

be

kept

in

memory

as

well

its

function

will

be

explained

below

Memory

layout

for

the

parallel

SLG-WAM

is

shown

in

Figure

Unlike

many

parallel

Prologs

neither

the

local

stack

nor

the

heap

is

shared

rather

variable

bindings

are

explicitly

copied

on

subgoal

call

and

answer

return

Indeed

for

tabling

to

work

properly

a

subgoal

must

be

traversed

in

its

entirety

to

determine

whether

it

is

new

or

not

Also

bindings

of

answers

must

in

general

be

explicitly

unified

with

the

variables

of

subgoals

As

a

result

there

is

much

less

incentive

to

share

WAM

stacks

in

a

tabling

system

than

in

a

Prolog

Evaluations

of

the

sequential

SLG-WAM

are

deterministic

a

property

which

the

engine

uses

heavily

For

instance

every

time

a

new

answer

for

a

tabled

subgoal

is

generated

it

is

scheduled

on

the

choice

point

stack

to

be

returned

to

the

active

subgoals

that

are

variants

of

the

tabled

predicate

Properties

of

a

depth-first

evaluation

are

also

used

to

detect

of

completion

of

an

SCC

The

first

subgoal

of

an

SCC

visited

by

a

sequential

evaluation

is

called

the

root

of

the

SCC

Whenever

the

system

fails

SLG-WAM

described

in

-LSB-

-RSB-

avoids

the

reunification

of

answer

clauses

in

certain

but

not

all

cases

over

the

root

of

an

SCC

it

is

provable

that

each

subgoal

inside

the

component

is

completed

and

all

answers

have

been

generated

-LRB-

Details

can

be

found

in

-LSB-

-RSB-

-RRB-

The

strategy

outlined

above

is

unsuitable

for

parallel

SLG

since

active

nodes

for

a

particular

tabled

predicated

might

be

in

different

threads

As

a

reflection

of

this

difference

each

thread

will

need

to

maintain

a

subgoal

dependency

list

which

keeps

a

pointer

to

the

answers

last

resolved

by

each

active

node

that

corresponds

to

an

uncompleted

While

the

subgoal

dependency

list

keeps

thread-specific

data

it

is

kept

in

a

global

area

for

use

in

the

completion

algorithm

as

will

be

discussed

in

Section

The

SLG-forest

grows

dynamically

as

predicates

are

called

and

thus

the

number

of

active

threads

depends

on

the

particular

program

For

programs

with

high

degree

of

table-parallelism

the

number

of

active

threads

can

be

much

larger

than

the

number

of

actual

processors

This

would

probably

lead

to

a

slow

down

if

compared

to

the

sequential

execution

as

most

thread

schedulers

are

slow

at

making

general

re-scheduling

decisions

-LSB-

-RSB-

In

addition

most

systems

impose

a

limit

on

the

number

of

threads

that

can

be

executing

at

a

time

As

a

first

approach

we

will

require

the

user

to

decide

which

predicates

to

execute

in

parallel

Future

research

will

investigate

scheduling

strategies

to

distribute

work

among

threads

For

instance

a

global

queue

may

be

kept

so

that

whenever

a

tabled

predicate

is

called

for

the

first

time

instead

of

spawning

a

new

thread

the

task

would

be

added

to

the

global

queue

Available

threads

would

then

steal

work

from

this

queue

Trailing

and

scheduling

mechanisms

of

the

SLG-WAM

already

support

the

context

switching

required

for

this

method

One

of

the

main

goals

of

the

parallel

logic

programming

community

is

to

exploit

parallelism

implicitly

including

full

Prolog

with

standard

semantics

-LSB-

-RSB-

Prolog

's

implementation

of

SLD

generates

answer

substitutions

in

textual

order

but

this

notion

does

not

exist

in

SLG

where

the

order

of

answer

substitutions

produced

through

clause

resolution

for

generator

nodes

may

not

resemble

the

order

produced

by

answer

clause

resolution

with

active

-LRB-

consuming

-RRB-

nodes

While

SLD

with

cuts

and

SLG

can

be

intermixed

in

the

sequential

framework

certain

semantic

changes

to

the

cut

are

necessary

to

preserve

the

correctness

of

SLG

Parallelizing

programs

with

the

SLG

cut

remains

an

open

issue

Aside

from

that

optimization

is

available

for

completed

subgoals

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

r

-LRB-

X

Y

-RRB-

p

-LRB-

-RRB-

q

-LRB-

Y

-RRB-

p

-LRB-

-RRB-

q

-LRB-

Y

-RRB-

p

-LRB-

-RRB-

q

-LRB-

Y

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

q

-LRB-

-RRB-

p

-LRB-

-RRB-

p

-LRB-

-RRB-

p

-LRB-

-RRB-

a

-LRB-

-RRB-

a

-LRB-

-RRB-

a

-LRB-

-RRB-

q

-LRB-

X

Y

-RRB-

a

-LRB-

X

Y

-RRB-

q

-LRB-

X

Y

-RRB-

a

-LRB-

X

Z

-RRB-

q

-LRB-

Z

Y

-RRB-

r

-LRB-

X

Y

-RRB-

p

-LRB-

X

Z

-RRB-

q

-LRB-

Z

Y

-RRB-

Figure

Join

in

Parallel

Prolog

support

for

full

Prolog

can

be

provided

by

using

standard

techniques

adopted

by

current

or-parallel

systems

such

as

Muse

-LSB-

-RSB-

and

Aurora

-LSB-

-RSB-

A

major

advantage

of

parallel

SLG

over

other

parallel

Prolog

systems

is

the

avoidance

of

redundant

computation

In

Figure

for

instance

any

or-parallel

Prolog

would

have

to

compute

the

relation

q

-LRB-

Y

-RRB-

twice

In

SLG

if

the

predicate

q

were

declared

as

tabled

it

would

be

computed

only

once

and

any

other

subsequent

call

would

simply

consult

the

table

This

avoidance

of

redundant

computation

has

long

been

recognized

as

necessary

for

data-oriented

queries

No

less

important

is

that

SLG

terminates

for

programs

of

bounded

term

size

The

copying

of

a

subgoal

each

time

a

thread

is

created

was

a

drawback

of

the

Abstract

Model

-LSB-

-RSB-

Tabling

systems

however

require

copying

of

subgoals

whether

they

are

sequential

or

When

used

judiciously

tabling

may

greatly

improve

performance

over

SLD

-LRB-

-LSB-

-RSB-

-RRB-

and

copying

does

not

add

a

particular

overhead

for

the

parallel

version

An

advantage

of

using

a

copy-based

method

is

that

each

thread

can

work

independently

on

physically

separated

data

and

issues

related

to

environment

sharing

are

avoided

-LRB-

This

property

brings

up

the

possibility

of

implementing

parallel

SLG

on

distributed

memory

machines

-RRB-

Table

vs.

And/Or

Parallelism

Because

SLG

is

a

different

resolution

method

than

SLD

the

idea

of

parallelism

in

the

context

of

SLG

departs

from

traditional

and

and

or-parallelism

Parallelism

will

occur

whenever

a

-LRB-

parallel

-RRB-

tabled

predicate

is

executed

not

in

the

parallel

execution

of

subgoals

in

the

body

of

a

clause

nor

in

the

parallel

execution

of

multiple

clauses

copying

has

been

investigated

for

tabling

systems

in

many

contexts

Like

most

tabling

systems

the

SLG-WAM

uses

an

array

of

static

and

dynamic

techniques

to

address

these

issues

-LRB-

-LSB-

-RSB-

-RRB-

table

-LRB-

X

Y

-RRB-

p

-LRB-

X

Y

-RRB-

fail

-LRB-

X

Z

-RRB-

r

-LRB-

X

Z

-RRB-

fail

-LRB-

X

Y

Z

-RRB-

p

-LRB-

X

Y

-RRB-

r

-LRB-

X

Z

-RRB-

q

-LRB-

X

-RRB-

-LRB-

X

Y

Z

-RRB-

p

-LRB-

X

Y

-RRB-

-LRB-

X

Y

Z

-RRB-

r

-LRB-

X

Z

-RRB-

-LRB-

X

Y

Z

-RRB-

p

-LRB-

X

Y

-RRB-

r

-LRB-

X

Z

-RRB-

-LRB-

X

Y

Z

-RRB-

Table

r

-LRB-

X

Z

-RRB-

p

-LRB-

X

Y

-RRB-

Figure

And-parallelism

in

SLG

defining

a

predicate

The

use

of

a

copying

strategy

and

the

lack

of

stack

sharing

between

different

threads

allows

each

thread

to

work

independently

of

the

others

on

its

own

SLG

tree

in

the

SLG

forest

Therefore

there

is

a

great

flexibility

on

the

choice

of

evaluation

strategy

used

to

compute

answers

for

the

tabled

predicate

within

each

tree

The

strategy

could

consist

of

sequential

SLG

sequential

SLD

or

any

parallel

model

In

this

sense

tableparallelism

can

be

considered

orthogonal

to

both

andand

or-parallelism

An

interesting

characteristic

of

table-parallelism

is

that

by

adopting

an

appropriate

scheduling

method

it

may

emulate

or-parallelism

Given

a

predicate

consisting

of

N

clauses

one

needs

only

fold

the

body

of

each

clause

into

a

unique

tabled

predicate

If

the

subgoals

are

not

present

in

the

table

the

caller

spawns

threads

for

each

predicate

in

turn

failing

as

a

means

of

looking

for

new

work

between

each

-LRB-

If

the

subgoals

are

present

in

the

table

the

consuming

process

reads

them

out

sequentially

-RRB-

We

do

not

yet

have

any

information

about

when

such

a

strategy

may

be

efficient

It

is

believed

that

table-parallelism

may

be

most

useful

at

coarser

levels

or

when

redundant

subcomputations

are

a

problem

Table-parallelism

can

also

exploit

some

of

the

andparallelism

present

in

logic

programs

Consider

the

goal

q

-LRB-

X

-RRB-

p

-LRB-

X

Y

-RRB-

r

-LRB-

X

Z

-RRB-

In

parallel

SLG

we

could

simulate

and-parallelism

by

rewriting

that

goal

as

in

Figure

and

declaring

and

and

as

tabled

In

this

case

all

answers

for

and

would

be

com

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

puted

in

parallel

assuming

that

they

are

not

already

in

the

table

As

they

are

being

computed

they

will

be

fed

to

the

third

clause

of

and

where

and

will

be

active

-LRB-

consumer

-RRB-

nodes

A

preprocessor

could

be

easily

written

to

generate

code

that

emulates

either

behavior

If

it

were

desired

for

table

parallelism

to

emulate

and

or

or

parallelism

analysis

could

be

performed

to

determine

where

a

program

contains

and

and

or

parallelism

using

traditional

techniques

The

output

of

the

analysis

phase

would

then

be

adapted

as

input

to

such

a

preprocessor

In

practice

table-parallelism

has

its

own

strengths

which

are

mainly

complimentary

to

traditional

methods

Parallel

Completion

Detection

of

completion

is

a

non-trivial

problem

in

a

sequential

framework

and

the

difficulties

are

compounded

for

parallel

evaluations

While

various

approaches

to

this

problem

are

possible

implementation

of

a

workable

engine

requires

a

minimum

of

synchronization

accordingly

we

base

our

approach

on

distributed

termination

detection

algorithms

as

in

-LSB-

-RSB-

Section

introduced

the

shared

completion

stack

used

to

detect

completion

of

an

SCC

The

completion

stack

consists

of

frames

for

each

incomplete

tabled

subgoal

The

frames

are

pushed

onto

the

stack

when

the

subgoals

are

called

and

consist

of

the

following

elements

DFN

Unique

Depth-first

number

PosLink

Earliest

direct

dependency

PosMin

Earliest

indirect

dependency

StateFlag

Thread

is

working

AnswerFlag

Thread

has

consumed

all

answers

ColorFlag

Thread

may

affect

other

subgoals

DFN

PosMin

and

PosLink

-LRB-

-LSB-

-RSB-

-RRB-

are

used

in

the

sequential

model

as

well

as

the

parallel

and

together

determine

the

extent

of

the

SCC

For

a

given

tabled

subgoal

A

PosLink

is

originally

set

to

the

unique

DFN

of

A

If

A

calls

another

incomplete

tabled

predicate

B

whose

DFN

is

less

than

that

of

PosLink

-LRB-

A

-RRB-

P

oslink

-LRB-

A

-RRB-

is

set

to

P

oslink

-LRB-

B

-RRB-

so

that

PosLink

reflects

the

earliest

direct

dependency

of

A.

Conversely

subgoals

on

the

completion

stack

are

assumed

to

depend

on

other

subgoals

of

higher

DFN

Consider

the

following

situation

Subgoal

A

calls

a

new

subgoal

B

and

a

new

frame

is

placed

on

the

completion

stack

B

later

calls

a

subgoal

C

which

is

an

ancestor

of

both

In

this

situation

A

depends

on

C

through

B

and

of

these

subgoals

can

be

completed

until

they

all

are

We

thus

can

not

complete

A

before

C

because

A

depends

on

an

incomplete

subgoal

B

younger

than

A

and

B

depends

on

a

subgoal

C

older

than

A

In

other

words

A

B

and

C

are

in

the

same

SCC

rooted

at

C.

To

prevent

improper

completion

information

about

the

PosLink

values

for

these

subgoals

must

be

propagated

to

A

before

it

can

be

completed

The

characteristics

of

this

propagation

vary

from

the

sequential

to

the

parallel

model

-LRB-

See

-LSB-

-RSB-

for

the

sequential

case

-RRB-

We

let

PosMin

-LRB-

A

-RRB-

stand

for

the

minimum

PosLink

value

of

all

subgoals

younger

than

A

information

which

is

available

from

the

completion

stack

The

StateF

lag

indicates

whether

the

thread

is

doing

some

computation

while

the

AnswerF

lag

indicates

whether

there

are

still

answers

for

the

thread

to

process

The

StateF

lag

can

have

values

done

undone

or

unconditionally

done

Conceptually

the

AnswerF

lag

can

have

values

done

or

undone

however

it

is

implemented

as

a

pointer

into

the

subgoal

dependency

list

The

AnswerF

lag

for

a

thread

T

will

be

a

disjunction

of

the

next

answer

pointer

for

all

Active

nodes

in

T

The

AnswerF

lag

will

be

done

when

all

answers

have

been

used

In

order

to

complete

correctly

a

computation

needs

to

maintain

the

following

invariants

Invariant

-LRB-

StateFlag

-RRB-

The

StateF

lag

is

done

for

a

given

thread

iff

it

is

performing

no

answer

or

program

clause

resolution

-LRB-

i.e.

doing

no

work

-RRB-

Invariant

-LRB-

AnswerFlag

-RRB-

The

AnswerF

lag

is

done

for

a

given

thread

iff

all

active

nodes

in

the

SLG

tree

for

the

thread

has

resolved

all

applicable

answers

in

the

table

Whenever

a

thread

exhausts

all

of

its

program

and

answer

clauses

it

sets

both

StateF

lag

and

AnswerF

lag

to

done

and

goes

to

sleep

Whenever

a

new

answer

is

added

to

a

table

for

any

of

the

Active

nodes

in

the

thread

the

thread

's

AnswerF

lag

is

effectively

set

to

undone

since

the

AnswerF

lag

is

a

list

of

pointers

into

the

table

After

a

thread

wakes

up

it

executes

the

following

steps

The

thread

set

its

StateF

lag

to

undone

The

thread

traverses

its

subgoal

dependency

list

to

see

if

new

answers

have

been

returned

for

its

active

nodes

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

If

new

answers

have

been

returned

the

thread

lays

down

an

answer

return

choice

point

-LRB-

presented

in

Section

-RRB-

for

each

uncompleted

subgoal

The

engine

will

backtrack

through

the

choice

points

and

continue

its

normal

execution

The

engine

iterates

steps

and

while

there

is

work

to

do

When

there

is

no

more

work

left

the

thread

sets

StateF

lag

to

done

-LRB-

AnswerF

lag

has

been

effectively

set

to

done

-RRB-

and

sleeps

Under

the

formalism

of

-LSB-

-RSB-

or

-LSB-

-RSB-

the

following

proposition

can

easily

be

proven

Proposition

-LRB-

Parallel

Completion

-RRB-

Let

E

be

a

computation

in

which

Invariants

and

hold

Let

DFN

-LRB-

S

-RRB-

PosLink

-LRB-

S

-RRB-

min

-LRB-

PosLink

-LRB-

A

-RRB-

-RRB-

for

all

A

younger

than

S

on

the

completion

stack

Also

let

StateF

lag

-LRB-

-RRB-

AnswerF

lag

-LRB-

-RRB-

done

where

denotes

S

and

every

A

younger

than

S

on

the

completion

stack

Then

the

subgoals

from

S

to

the

top

of

the

stack

constitute

a

maximal

SCC

and

have

been

completely

evaluated

The

intuition

behind

Proposition

is

that

if

the

PosLink

values

are

as

specified

the

subgoal

forms

the

root

-LRB-

oldest

subgoal

-RRB-

of

an

SCC

If

the

flag

values

are

as

specified

and

if

the

invariants

hold

for

the

computation

then

all

mutually

dependent

subgoals

have

performed

all

answer

and

program

clause

resolution

Proving

correctness

of

completion

then

devolves

upon

ensuring

that

the

invariants

hold

As

an

example

of

how

the

invariants

can

fail

to

hold

consider

the

following

situation

We

make

use

here

and

throughout

the

paper

of

a

convention

that

the

root

thread

-LRB-

the

thread

for

the

root

subgoal

of

an

SCC

-RRB-

leads

the

check

for

completion

of

the

SCC

Let

root

be

the

root

of

SCCroot

which

is

currently

being

checked

for

completion

And

suppose

SCCroot

frn

rn

rootg

root

will

check

the

status

of

the

flags

of

each

other

subgoal

from

rn

to

in

this

order

Suppose

while

root

is

checking

the

flags

for

ri

after

finding

that

all

flags

for

rl

l

i

are

done

another

rk

r

ir

j

checkflags

done

whiter

ColorFlag

done

white

done

white

done

black

undone

done

whiter

StateFlag

AnswerFlag

done

done

done

done

Figure

Snapshot

of

Completion

Stack

thread

rj

where

j

i

generates

a

new

answer

and

activates

a

thread

rk

that

has

been

already

checked

by

root

The

root

will

wrongly

assume

that

rk

is

done

and

it

will

incorrectly

complete

Following

the

model

of

-LSB-

-RSB-

we

address

this

problem

by

introducing

another

flag

the

ColorF

lag

for

each

tabled

subgoal

Definition

-LRB-

ColorFlag

-RRB-

The

ColorF

lag

indicates

that

a

node

in

the

SCC

may

have

generated

new

answers

after

another

node

was

checked

as

completed

by

the

root

It

can

be

either

black

if

new

answers

were

added

or

white

otherwise

Initially

the

ColorF

lags

for

all

tabled

subgoals

are

set

to

white

When

after

waking

up

a

thread

r

generates

some

new

answers

it

will

set

its

own

color

to

black

During

the

completion

check

if

the

root

finds

a

black

node

it

will

know

that

some

node

that

it

had

already

checked

might

have

been

awakened

-LRB-

see

Figure

-RRB-

and

therefore

completion

has

to

be

restarted

Implementation

Framework

This

section

presents

the

instruction

level

changes

needed

to

parallelize

the

SLG-WAM

While

SLG-WAM

instructions

for

definite

programs

have

been

presented

in

-LSB-

-RSB-

a

detailed

knowledge

of

these

instructions

is

not

needed

to

for

understanding

this

section

since

the

changes

mainly

involve

adding

concurrency

features

to

tabling

operations

The

changes

are

as

follows

ffl

When

calling

a

tabled

subgoal

in

the

new

active

instruction

If

the

selected

subgoal

is

not

al

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

ready

in

the

table

add

an

entry

for

it

create

a

thread

and

copy

the

subgoal

into

the

thread

Update

the

calling

thread

's

subgoal

dependency

list

with

a

pointer

to

the

active

node

Then

fail

in

order

to

find

other

work

The

copy

of

the

subgoal

is

required

so

that

variables

are

not

shared

between

threads

On

the

other

hand

if

the

selected

subgoal

is

already

in

the

table

but

not

complete

the

PosLink

value

for

the

root

subgoal

should

be

updated

along

with

the

subgoal

dependency

list

Resolution

can

then

begin

against

answers

in

the

table

Finally

if

the

subgoal

is

in

the

table

and

completed

the

answers

are

treated

as

if

they

were

asserted

code

Neither

the

completion

stack

nor

the

subgoal

dependency

list

needs

to

be

updated

ffl

When

adding

an

answer

for

a

tabled

subgoal

in

the

new

answer

instruction

Besides

adding

an

answer

to

the

table

-LRB-

if

the

answer

is

not

already

there

-RRB-

the

instruction

will

set

its

ColorFlag

to

black

The

AnswerF

lag

which

is

a

-LRB-

pointer

to

a

-RRB-

list

of

pointers

into

the

table

will

be

effectively

updated

as

discussed

in

Section

ffl

When

checking

for

completion

in

the

completion

instruction

The

thread

must

first

check

that

no

answers

have

been

added

that

might

unify

with

any

of

its

active

subgoals

If

answers

have

been

added

the

process

schedules

their

return

on

the

choice

point

stack

Otherwise

the

predicate

marks

its

StateF

lag

as

done

and

if

the

thread

is

the

root

of

its

SCC

it

begins

the

completion

check

algorithm

to

be

described

below

This

model

requires

synchronization

between

threads

when

subgoals

are

created

and

when

they

are

completed

but

not

otherwise

Adding

a

new

SLG

subgoal

to

the

table

will

require

a

lock

to

prevent

multiple

threads

from

simultaneously

executing

the

new

active

instruction

for

the

subgoal

In

the

SLG-WAM

the

table

for

each

predicate

is

structured

as

a

tree

so

that

only

a

subtree

need

be

locked

allowing

a

great

deal

of

concurrency

Contention

will

be

more

likely

to

arise

in

the

completion

stack

For

adding

answers

to

the

table

however

no

locking

will

be

necessary

as

there

is

a

single

generating

thread

for

each

subgoal

-LRB-

although

possibly

multiple

consuming

threads

-RRB-

The

issue

of

returning

new

answers

to

existing

subgoals

differs

from

the

tuple-at-a-time

method

described

in

-LSB-

-RSB-

but

resembles

that

of

a

sequential

semi-naive

implementation

-LRB-

to

be

described

fully

in

a

forthcoming

paper

-RRB-

In

the

semi-naive

implementation

an

iteration

consists

of

all

work

that

occurs

between

failures

back

to

the

completion

instruction

for

the

root

of

an

SCC

At

iteration

i

the

engine

uses

answers

from

iteration

i

to

determine

a

new

set

of

answers

At

the

end

of

each

iteration

the

completion

instruction

checks

whether

new

answers

have

been

added

in

iteration

i

or

whether

the

SCC

has

instead

reached

The

semi-naive

strategy

can

be

incorporated

into

parallel

execution

at

each

completion

the

engine

returns

any

new

answers

added

by

other

threads

Only

in

the

case

where

no

answers

have

been

added

and

there

is

no

other

work

to

do

does

the

parallel

algorithm

differ

from

sequential

seminaive

Rather

than

being

able

to

determine

fixpoint

the

consuming

thread

must

coordinate

with

other

threads

to

determine

if

all

answers

have

been

derived

for

its

active

subgoals

If

the

thread

is

a

root

thread

it

checks

the

flags

of

the

completion

stack

using

an

algorithm

described

below

Otherwise

the

thread

must

poll

at

intervals

to

see

whether

new

answers

have

been

added

Returning

answers

is

summarized

in

the

procedure

Iterate

in

Figure

The

subgoal

dependency

list

maintains

information

for

subgoals

which

are

active

for

each

thread

In

order

to

make

use

of

new

answers

at

the

end

of

an

iteration

the

thread

will

exploit

two

properties

for

an

answer

trie

A

path

from

the

root

of

the

trie

to

a

leaf

node

constitutes

an

answer

for

the

trie

The

first

property

exploited

is

that

each

leaf

of

the

trie

is

linked

together

in

a

list

for

sequential

access

When

answers

are

added

to

a

trie

the

trie

is

extended

and

the

answers

are

also

added

to

the

end

of

the

list

using

a

global

listend

pointer

to

the

end

of

the

trie

For

each

subgoal

in

the

dependency

list

the

thread

creates

answer

return

choice

points

which

contain

among

other

information

a

local

listend

value

for

iteration

i

and

a

local

listend

value

for

iteration

i

The

answer

return

choice

point

then

backtracks

through

each

active

node

for

its

subgoal

and

returns

each

answer

in

the

listend

interval

to

the

node

Of

course

there

may

be

discrepancies

between

global

and

local

listend

values

a

generator

thread

may

add

an

answer

as

a

consumer

thread

reads

the

listend

While

the

discrepancy

is

probably

not

important

while

a

back

to

the

root

of

an

SCC

does

not

always

reclaim

space

since

environments

for

SLG

predicates

are

maintained

between

iterations

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

Iterate

If

subgoal

dependency

list

is

empty

AnswerF

lag

StateF

lag

uncond

done

update

subgoal

stack

exit

Else

For

each

subgoal

in

list

If

-LRB-

subgoal

is

unconditionally

complete

and

there

are

no

unexamined

answers

-RRB-

Remove

subgoal

from

list

Else

If

there

are

unexamined

answers

last

listend

local

listend

local

listend

global

listend

create

answer

return

choice

point

Figure

Algorithm

for

iterating

computation

thread

is

in

the

process

of

iteration

it

must

be

rectified

by

the

time

the

subgoals

are

completed

When

a

thread

wakes

up

to

poll

for

answers

it

will

set

its

StateF

lag

undone

It

will

then

run

through

its

subgoal

dependency

list

and

return

the

new

answers

from

the

global

table

to

the

suspended

nodes

in

its

subtree

and

continue

its

normal

sequential

SLG

execution

When

it

runs

out

of

work

-LRB-

or

if

there

was

no

work

to

begin

with

-RRB-

it

resets

its

StateF

lag

to

done

If

in

the

course

of

its

execution

this

thread

generates

new

answers

it

will

set

its

ColorF

lag

to

black

indicating

that

other

nodes

might

have

been

activated

Finally

if

the

current

thread

is

not

the

leader

of

its

SCC

it

goes

to

sleep

Otherwise

it

will

start

the

completion

check

The

completion

instruction

uses

the

algorithm

in

Figure

In

the

case

where

a

thread

depends

only

on

subgoals

which

are

unconditionally

done

and

where

all

answers

have

been

examined

the

thread

marks

its

table

as

unconditionally

complete

and

exits

Otherwise

if

a

thread

traverses

the

subgoal

dependency

list

and

finds

no

new

answers

its

AnswerF

lag

is

effectively

done

so

it

sets

its

StateFlag

to

done

in

the

completion

stack

-LRB-

Since

we

assume

only

one

thread

per

subgoal

there

will

be

no

confusion

-RRB-

If

a

new

answer

is

added

for

a

subgoal

upon

which

other

subgoals

depend

the

subgoal

dependency

lists

and

AnswerF

lags

will

reflect

this

change

The

thread

generating

a

new

answer

also

resets

its

ColorF

lag

CheckCompletion

-LRB-

S

-RRB-

If

Iterate

exits

exit

if

S

depends

on

a

subgoal

which

is

not

complete

if

any

choice

point

has

been

added

StateF

lag

AnswerF

lag

undone

fail

fbacktrack

through

the

choicepointsg

Else

if

S

belongs

to

the

youngest

SCC

StateF

lag

AnswerF

lag

done

if

S

is

the

leader

for

each

ri

S

's

SCC

such

that

ri

is

not

uncond

done

wait

-LRB-

-LRB-

ri

StateF

lag

done

-RRB-

and

-LRB-

ri

StateF

lag

done

-RRB-

-RRB-

if

-LRB-

ri

ColorF

lag

black

-RRB-

set

the

ColorF

lags

for

all

subgoals

in

the

SCC

to

white

suspend

and

restart

completion

later

complete

tables

for

all

subgoals

in

SCC

Else

sleep

Else

fail

fnot

youngest

SCCg

Figure

Pseudo-code

for

completion

instruction

to

black

During

completion

check

the

leader

will

check

all

subgoals

in

its

SCC

that

are

not

unconditionally

done

If

any

subgoal

has

either

AnswerF

lag

or

StateF

lag

undone

the

leader

will

wait

until

both

are

set

to

done

If

both

flags

are

done

the

leader

will

check

the

ColorF

lag

If

the

ColorF

lag

is

white

the

leader

will

proceed

to

the

next

subgoal

Whenever

the

leader

reaches

its

own

frame

in

the

completion

stack

by

Theorem

A.

all

subgoals

in

the

SCC

are

completely

evaluated

and

its

is

safe

to

mark

the

tables

complete

pop

the

completion

stack

and

terminate

After

the

root

subgoal

or

leader

completes

all

tables

it

notifies

the

threads

that

it

is

safe

to

exit

and

exits

itself

Otherwise

if

the

leader

finds

some

black

subgoal

a

subgoal

may

have

generated

new

answers

that

have

not

been

consumed

The

leader

then

fails

and

completion

check

will

be

restarted

later

The

root

thread

thus

enters

a

loop

where

it

traverses

the

subgoal

dependency

list

and

either

returns

answers

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

or

checks

for

completion

while

the

non-root

threads

either

return

answers

or

mark

themselves

as

conditionally

complete

by

setting

its

flags

to

done

Since

a

given

SCC

may

grow

as

new

dependencies

are

added

different

threads

may

become

root

at

different

times

If

invariants

and

hold

the

algorithm

handles

this

situation

Suppose

that

thread

is

root

and

thread

later

becomes

root

would

only

complete

after

it

was

determined

that

there

were

no

unresolved

answer

or

program

clauses

in

the

SCC

On

the

other

hand

would

have

been

made

root

only

if

some

subgoal

in

the

SCC

rooted

at

discovered

a

dependency

on

But

this

can

not

occur

after

all

answer

and

program

clauses

have

been

resolved

By

formalizing

the

algorithm

outlined

in

this

section

it

is

provable

that

ffl

Threads

terminate

iff

their

subgoals

have

been

completely

evaluated

ffl

There

will

be

no

deadlock

or

starvation

The

formal

statement

of

these

results

and

their

proof

are

contained

in

the

Appendix

Conclusion

Tabling

adds

the

ability

to

evaluate

programs

according

to

the

well-founded

semantics

it

adds

important

termination

properties

to

evaluations

whether

or

not

the

program

contains

negation

and

it

computes

queries

with

polynomial

data

complexity

again

whether

or

not

negation

is

present

in

the

program

that

evaluates

the

query

Depending

on

one

's

point

of

view

tabling

is

thus

a

desirable

addition

or

a

necessary

addition

to

an

evaluation

This

paper

constitutes

a

first

step

in

the

design

of

a

parallel

SLG-WAM

for

definite

programs

Its

implementation

intended

to

be

a

continuation

of

the

semi-naive

sequential

model

as

indicated

in

Section

While

implementation

results

are

not

yet

available

the

foregoing

discussion

indicates

that

only

instructions

for

tabled

predicates

are

to

be

changed

by

the

design

As

a

result

for

sequential

SLD

the

times

for

the

parallel

engine

should

equal

the

sequential

SLD

times

for

the

sequential

engine

about

a

overhead

added

to

the

WAM

While

we

have

introduced

the

model

of

table-parallelism

due

to

space

limitations

we

have

only

discussed

its

most

basic

model

As

a

first

instance

the

model

parallelizes

all

tabled

evaluation

clearly

this

is

not

always

a

wise

strategy

A

clear

first

extension

would

allow

the

user

to

parallelize

only

certain

tabled

predicates

As

a

second

extension

SCCs

could

be

analyzed

statically

and

threads

matched

to

SCCs

rather

than

to

subgoals

A

third

extension

might

be

to

make

the

completion

stack

into

a

tree

so

that

SCCs

could

complete

early

While

this

last

extension

might

cause

some

overhead

in

determining

the

root

of

the

SCC

it

would

be

expected

to

be

reduce

the

synchronization

needed

for

completion

These

three

extensions

are

straightforward

but

their

worth

can

only

be

ascertained

in

the

context

of

an

implemented

system

for

which

work

is

now

underway

Acknowledgements

We

are

indebted

to

Phil

Lewis

for

useful

discussions

on

the

concurrency

issues

of

the

completion

algorithm

References

-LSB-

-RSB-

K.M.

Ali

and

R.

Karlsson

The

Muse

or-parallel

Prolog

model

and

its

performance

In

Proceedings

of

the

North

American

Conference

on

Logic

Programming

pages

-LCB-

MIT

Press

-LSB-

-RSB-

Krzysztof

R.

Apt

Correctness

proofs

of

distributed

termination

algorithms

ACM

Transactions

on

Programming

Languages

and

Systems

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

F.

Banchilhon

and

R.

Ramakrishnan

An

amateur

's

introdution

to

recursive

query

processing

strategies

In

Proc

of

SIGMOD

Conf.

pages

-LCB-

ACM

-LSB-

-RSB-

A.D.

Birrel

An

introduction

to

programming

with

threads

Technical

report

Digital

Systems

Research

Center

Palo

Alto

California

-LSB-

-RSB-

R.

Bol

and

L.

Degerstedt

Tabulated

resolution

for

the

well-founded

semantics

In

Proc

ILPS

Workshop

on

Programming

with

Logic

Databases

MIT

Press

-LSB-

-RSB-

W.

Chen

T.

Swift

and

D.S.

Warren

Efficient

implementation

of

general

logical

queries

J.

Logic

Programming

To

Appear

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

-LSB-

-RSB-

W.

Chen

and

D.

S.

Warren

Query

evaluation

under

the

well-founded

semantics

In

Proc

of

PODS

-LSB-

-RSB-

E.W.

Dijks

tra

W.H.J

Feijen

and

A.J.M.

van

Gasteren

Derivation

of

a

termination

detection

algorithm

for

distributed

computations

Information

Processing

Letters

pages

-LCB-

June

-LSB-

-RSB-

Nissim

Francez

Program

Verification

AddisonWesley

-LSB-

-RSB-

Gupta

G.

Ali

K.M.

Carlsson

M.

and

M.

Hermenegildo

Parallel

Logic

Programming

A

Survey

Unpublished

manuscript

pages

Available

by

anonymous

ftp

at

ftp.cs.nmsu.edu

-LSB-

-RSB-

E.

Lusk

R.

Butler

T.

Disz

R.

Olson

R.

Overbeek

D.H.D.

Warren

A.

Calderwood

P.

Szeredi

S.

Haridi

P.

Brand

M.

Carlsson

A.

Ciepielewski

and

B.

Hausman

The

Aurora

or-parallel

Prolog

system

New

Generation

Computing

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

R.

Ramakrishnan

and

J.

Ullman

A

survey

of

research

on

deductive

database

systems

Journal

of

Logic

Programming

To

appear

-LSB-

-RSB-

P.

Rao

I.V.

Ramakrishnan

T.

Swift

and

D.S.

Warren

Dynamic

argument

reduction

for

in-memory

queries

In

ICLP

Workshop

on

Logic

Programming

and

Deductive

Databases

-LSB-

-RSB-

H.

Seki

On

the

power

of

alexandrer

templates

In

Proc

of

PODS

pages

-LCB-

ACM

-LSB-

-RSB-

P.

Stuckey

and

S.

Sudarshan

Well-founded

ordered

search

In

conference

on

Foundations

of

Software

Technology

and

Theoretical

Computer

Science

pages

-LCB-

-LSB-

-RSB-

T.

Swift

and

D.

S.

Warren

An

abstract

machine

for

SLG

resolution

definite

programs

In

Proceedings

of

the

Symposium

on

Logic

Programming

To

Appear

-LSB-

-RSB-

T.

Swift

and

D.

S.

Warren

Analysis

of

sequential

SLG

evaluation

In

Proceedings

of

the

Symposium

on

Logic

Programming

To

Appear

-LSB-

-RSB-

H.

Tamaki

and

T.

Sato

OLDT

resolution

with

tabulation

In

Third

Int

l

Conf

on

Logic

Programming

pages

-LCB-

-LSB-

-RSB-

L.

Vieille

Recursive

query

processing

The

power

of

logic

Theoretical

Computer

Science

-LCB-

-LSB-

-RSB-

D.H.D.

Warren

Or-parallel

models

of

Prolog

In

Proceedings

of

the

International

Conference

on

Theory

and

Practice

of

Software

Development

Springer-Verlag

A

Appendix

Correctness

of

Com

pletion

Algorithm

Theorem

A.

-LRB-

Completion

Algorithm

-RRB-

Given

a

set

S

of

mutually

dependent

tabled

subgoals

the

completion

algorithm

of

Figure

will

mark

as

complete

the

subgoals

in

S

iff

S

A

is

completely

evaluated

Also

it

is

guaranteed

that

there

will

be

no

deadlocks

or

starvation

Proof

In

order

to

prove

Theorem

A.

we

need

to

prove

the

following

The

completion

algorithm

will

mark

as

complete

the

subgoals

in

S

iff

S

A

is

completely

evaluated

Deadlock

is

prevented

In

other

words

it

will

always

be

the

case

that

either

completion

detection

or

subgoal

computation

can

occur

There

will

no

be

starvation

In

other

words

during

completion

detection

if

the

subgoals

in

the

SCC

are

not

completely

evaluated

subgoal

computation

will

resume

Program

verification

techniques

such

as

the

ones

described

in

-LSB-

-RSB-

are

needed

for

formal

proof

of

the

above

four

properties

The

proof

here

is

based

on

the

distributed

completion

proof

presented

in

-LSB-

-RSB-

When

synchronizing

threads

for

completion

we

may

abstractly

assume

that

the

threads

of

the

SCCs

enter

one

of

three

procedures

wakeup

to

look

for

new

answers

completion

to

check

for

completion

of

the

nodes

in

the

SCC

or

terminate

Using

the

structure

of

Section

the

wakeup

choice

will

be

executed

each

time

a

tabled

predicate

starts

to

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

wakeup

r

StateF

lag

undone

reads

the

tables

of

all

active

subgoals

in

its

tree

r

AnswerF

lag

done

if

-LRB-

r

generates

new

answers

-RRB-

writes

the

answers

into

r

's

table

r

color

black

r

StateF

lag

done

sleep

terminate

exit

completion

if

subgoal

is

not

the

root

exit

for

-LRB-

i

i

i

-RRB-

while

-LRB-

-LRB-

ri

StateF

lag

undone

-RRB-

or

-LRB-

ri

AnswerF

lag

undone

-RRB-

-RRB-

if

-LRB-

ri

color

black

-RRB-

for

-LRB-

i

i

i

-RRB-

ri

color

white

suspend

and

restart

completion

later

broadcast

to

rn

to

terminate

exit

Figure

Non-deterministic

choices

for

tabled

predicates

execute

the

check

complete

instruction

At

this

point

it

will

check

whether

there

are

new

answers

to

the

active

subgoals

in

its

SLG

tree

and

if

that

is

the

case

return

the

answers

to

these

subgoals

If

while

executing

this

tabled

predicate

itself

generates

new

answers

it

will

set

its

ColorF

lag

to

black

since

it

might

have

awakened

some

other

tabled

subgoals

in

its

SCC

Only

the

root

-LRB-

leader

-RRB-

of

an

SCC

will

execute

the

completion

choice

During

completion

the

root

will

check

whether

all

subgoals

in

the

SCC

have

completed

and

if

that

is

the

case

it

will

complete

the

tables

signal

the

other

threads

to

exit

and

terminate

If

some

subgoal

in

the

SCC

is

either

executing

program

or

answer

clauses

the

root

will

wait

until

this

subgoal

is

done

to

resume

the

checking

If

while

scanning

the

completion

stack

the

root

finds

any

subgoal

whose

ColorF

lag

is

black

it

whitens

the

ColorF

lags

for

all

subgoals

in

its

SCC

and

suspends

the

completion

check

While

executing

the

completion

alternative

the

root

of

the

SCC

root

-LRB-

in

Figure

-RRB-

checks

the

status

of

threads

rn

root

in

this

order

The

necessary

condition

for

the

completion

of

root

's

SCC

is

that

root

succeeds

the

checking

for

all

ri

Let

t

denote

the

number

of

the

thread

whose

flags

are

currently

being

checked

by

the

root

then

completion

checking

ends

with

t

The

system

's

state

will

be

captured

by

an

invariant

which

we

call

P

In

the

sequel

P

will

be

constructed

in

a

number

of

steps

each

step

consisting

of

an

extension

of

the

state

space

considered

and

of

an

appropriate

adjustment

of

P

As

a

first

step

we

solve

the

problem

in

the

absence

of

wakeups

in

that

case

an

undone

-LRB-

active

-RRB-

thread

can

become

done

-LRB-

passive

-RRB-

but

a

done

thread

can

not

become

undone

The

conclusion

that

all

threads

are

done

has

to

follow

from

-LRB-

i

-RRB-

the

invariant

-LRB-

as

shown

below

-RRB-

-LRB-

ii

-RRB-

t

and

-LRB-

iii

-RRB-

the

fact

that

the

flags

of

the

root

itself

are

done

Furthermore

the

invariant

-LRB-

-RRB-

has

to

hold

independently

of

the

value

of

t

when

thread

root

has

initiated

the

completion

checking

i.e.

when

t

n

The

above

requirements

are

met

by

-LRB-

t

i

N

thread

ri

is

done

-RRB-

We

can

now

design

the

checking

of

AnswerF

lag

and

StateF

lag

of

each

thread

so

as

to

keep

invariant

Initiation

of

completion

checking

establishes

Sequentially

checking

the

remaining

threads

however

decreases

t

by

and

may

falsify

Invariance

of

is

achieved

by

adopting

Rule

A.

The

root

waits

until

the

flags

for

thread

ri

are

done

before

its

checks

ri

's

StateF

lag

and

AnswerF

lag

In

the

completion

alternative

of

Figure

Rule

A.

corresponds

to

the

statements

while

-LRB-

-LRB-

ri

StateF

lag

undone

-RRB-

or

-LRB-

ri

AnswerF

lag

undone

-RRB-

-RRB-

When

the

flags

of

the

other

threads

are

done

the

checking

of

flags

will

in

due

time

be

reached

by

root

when

in

addition

thread

root

is

done

termination

can

be

concluded

The

above

discussion

comprises

the

rules

for

checking

flags

Next

to

be

taken

into

account

is

the

possibility

of

wakeup

is

falsified

when

thread

ri

with

t

i

becomes

undone

on

account

of

being

waken

up

Since

only

undone

threads

wake

up

other

threads

we

deduce

that

ILPS

Design

and

Implementation

of

Parallel

Logic

Programming

Systems

the

waking

up

that

falsified

has

been

initiated

by

a

thread

rj

with

j

t

In

order

to

solve

the

situation

we

adopt

the

weaker

invariant

such

that

any

wakeup

that

may

falsify

establishes

To

this

end

each

thread

is

postulated

to

be

either

black

or

white

-LRB-

represented

by

the

ColorF

lag

of

each

subgoal

-RRB-

For

we

choose

-LRB-

j

t

such

that

thread

rj

is

black

-RRB-

Wakeup

is

prevented

from

falsifying

by

adopting

the

following

rule

Rule

A.

A

thread

wakes

up

another

thread

with

a

number

higher

than

its

own

makes

itself

black

We

have

to

verify

however

that

the

information

available

at

thread

root

combined

with

the

weaker

can

still

suffice

to

conclude

termination

Since

-LRB-

t

thread

root

is

white

-RRB-

=-RRB-

detection

of

termination

has

not

been

disabled

With

the

possibility

of

a

thread

's

having

the

color

black

a

new

phenomenon

has

been

introduced

viz

that

of

the

unsuccessful

completion

checking

when

a

black

thread

is

detected

the

conclusion

of

termination

can

not

be

drawn

In

a

first

instance

this

problem

can

be

tackled

by

adopting

Rule

A.

After

failing

the

completion

checking

thread

root

reinitiates

checking

for

completion

This

corresponds

to

the

statement

suspend

and

restart

completion

later

in

Figure

Without

the

possibility

of

transitions

from

black

to

white

such

a

rechecking

may

not

be

successful

Therefore

our

next

task

is

to

assure

that

the

whitenings

do

not

falsify

the

invariant

In

view

of

the

fact

that

initiating

a

check

for

completion

establishes

we

can

safely

adopt

Rule

A.

Thread

root

initiates

a

completion

checking

by

making

itself

white

and

starting

checking

the

status

of

thread

rn

consistency

with

Section

a

relaxed

Rule

is

used

in

Figure

This

relaxed

rule

effectively

states

that

a

thread

waking

up

another

thread

will

blacken

itself

The

thread

root

can

safely

be

whitened

and

we

turn

to

the

other

threads

Since

whitening

a

thread

can

falsify

only

but

does

do

so

when

that

thread

's

number

exceeds

t

we

can

safely

adopt

Rule

A.

When

it

restarts

completion

checking

the

root

whitens

all

threads

with

number

greater

than

t

Since

the

restart

of

completion

checking

changes

the

value

of

t

to

n

which

establishes

we

can

further

adopt

Rule

A.

When

it

restarts

the

completion

checking

the

root

whitens

all

threads

from

to

N

Rule

A.

assures

the

correctness

of

the

entire

for

loop

in

the

completion

choice

of

Figure

The

above

whitening

protocols

suffice

to

prove

that

when

the

algorithm

terminates

then

all

subgoals

of

the

SCC

are

completed

An

iteration

of

completion

checking

initiated

after

all

subgoals

are

completely

evaluated

will

end

with

all

threads

white

and

hence

a

reiteration

of

the

completion

check

is

guaranteed

to

succeed

The

property

that

when

all

subgoals

of

the

SCC

are

completed

then

the

completion

algorithm

will

terminate

derives

from

the

fact

that

the

root

will

eventually

check

for

completion

and

it

will

restart

completion

later

if

it

fails

this

time

The

absence

of

deadlock

can

be

easily

seen

from

the

fact

that

the

computation

of

subgoals

-LRB-

wakeup

alternative

of

Figure

-RRB-

and

the

completion

checking

of

the

root

-LRB-

completion

alternative

of

Figure

-RRB-

need

not

wait

for

each

other

In

order

to

prove

that

there

is

no

starvation

we

note

that

if

during

completion

detection

there

are

subgoals

in

the

SCC

which

are

not

completely

evaluated

these

subgoals

will

be

awakened

later

After

a

thread

wakes

up

to

poll

for

answers

it

will

run

through

its

subgoal

dependency

list

and

return

the

new

answers

from

the

global

table

to

the

suspended

nodes

in

its

subtree

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

parallel logic programming
slg
tabling
xsb