Scalable

Parallel

Data

Mining

for

Association

Rules

Eui-Hong

-LRB-

Sam

-RRB-

Han

George

Karypis

Vipin

Kumar

Department

of

Computer

Science

University

of

Minnesota

EECS

Bldg.

Union

St.

SE

Minneapolis

MN

USA

fhan,karypis,kumarg@cs.umn.edu

Last

updated

on

July

at

am

Abstract

One

of

the

important

problems

in

data

mining

is

discovering

association

rules

from

databases

of

transactions

where

each

transaction

consists

of

a

set

of

items

The

most

time

consuming

operation

in

this

discovery

process

is

the

computation

of

the

frequency

of

the

occurrences

of

interesting

subset

of

items

-LRB-

called

candidates

-RRB-

in

the

database

of

transactions

To

prune

the

exponentially

large

space

of

candidates

most

existing

algorithms

consider

only

those

candidates

that

have

a

user

defined

minimum

support

Even

with

the

pruning

the

task

of

finding

all

association

rules

requires

a

lot

of

computation

power

and

memory

Parallel

computers

offer

a

potential

solution

to

the

computation

requirement

of

this

task

provided

efficient

and

scalable

parallel

algorithms

can

be

designed

In

this

paper

we

present

two

new

parallel

algorithms

for

mining

association

rules

The

Intelligent

Data

Distribution

algorithm

efficiently

uses

aggregate

memory

of

the

parallel

computer

by

employing

intelligent

candidate

partitioning

scheme

and

uses

efficient

communication

mechanism

to

move

data

among

the

processors

The

Hybrid

Distribution

algorithm

further

improves

upon

the

Intelligent

Data

Distribution

algorithm

by

dynamically

partitioning

the

candidate

set

to

maintain

good

load

balance

The

experimental

results

on

a

Cray

parallel

computer

show

that

the

Hybrid

Distribution

algorithm

scales

linearly

exploits

the

aggregate

memory

better

and

can

generate

more

association

rules

with

a

single

scan

of

database

per

pass

This

work

was

supported

by

NSF

grant

Army

Research

Office

contract

Cray

Research

Inc.

Fellowship

and

IBM

partnership

award

the

content

of

which

does

not

necessarily

reflect

the

policy

of

the

government

and

no

official

endorsement

should

be

inferred

Access

to

computing

facilities

was

provided

by

AHPCRC

Minnesota

Supercomputer

Institute

Cray

Research

Inc.

and

NSF

grant

See

for

other

related

papers

Introduction

One

of

the

important

problems

in

data

mining

-LSB-

-RSB-

is

discovering

association

rules

from

databases

of

transactions

where

each

transaction

contains

a

set

of

items

The

most

time

consuming

operation

in

this

discovery

process

is

the

computation

of

the

frequencies

of

the

occurrence

of

subsets

of

items

also

called

candidates

in

the

database

of

transactions

Since

usually

such

transaction-based

databases

contain

a

large

number

of

distinct

items

the

total

number

of

candidates

is

prohibitively

large

Hence

current

association

rule

discovery

techniques

-LSB-

-RSB-

try

to

prune

the

search

space

by

requiring

a

minimum

level

of

support

for

candidates

under

consideration

Support

is

a

measure

of

the

number

of

occurrences

of

the

candidates

in

database

transactions

Apriori

-LSB-

-RSB-

is

a

recent

state-ofthe-art

algorithm

that

aggressively

prunes

the

set

of

potential

candidates

of

size

k

by

using

the

following

observation

a

candidate

of

size

k

can

meet

the

minimum

level

of

support

only

if

all

of

its

subsets

also

meet

the

minimum

level

of

support

In

the

k

t

h

iteration

this

algorithm

computes

the

occurrences

of

potential

candidates

of

size

k

in

each

of

the

transactions

To

do

this

task

efficiently

the

algorithm

maintains

all

potential

candidates

of

size

k

in

a

hash

tree

This

algorithm

does

not

require

the

transactions

to

stay

in

main

memory

but

requires

the

hash

trees

to

stay

in

main

memory

Even

with

the

highly

effective

pruning

method

of

Apriori

the

task

of

finding

all

association

rules

can

require

a

lot

of

computation

power

that

is

available

only

in

parallel

computers

Furthermore

the

size

of

the

main

memory

in

the

serial

computer

puts

an

upper

limit

on

the

number

of

the

candidates

that

can

be

considered

in

any

iteration

without

requiring

multiple

scans

of

the

data

This

effectively

puts

a

lower

bound

on

the

minimum

level

of

support

imposed

on

candidates

under

consideration

Parallel

computers

also

offer

increased

memory

to

solve

this

problem

Two

parallel

algorithms

Count

Distribution

and

Data

Distribution

were

proposed

in

-LSB-

-RSB-

The

Count

Distribution

algorithm

scales

linearly

and

has

excellent

speedup

and

sizeup

behavior

with

respect

to

the

number

of

transactions

-LSB-

-RSB-

However

this

algorithm

works

only

when

the

entire

hash

tree

in

each

pass

of

the

algorithm

fits

into

the

main

memory

of

single

processor

of

the

parallel

computers

Hence

the

Count

Distribution

algorithm

like

its

sequential

counterpart

Apriori

is

unscalable

with

respect

to

the

increasing

size

of

candidate

set

The

Data

Distribution

algorithm

addresses

the

memory

problem

of

the

Count

Distribution

algorithm

by

partitioning

the

candidate

set

and

assigning

a

partition

to

each

processor

in

the

system

However

this

algorithm

suffers

from

three

types

of

inefficiency

First

the

algorithm

results

in

high

communication

overhead

due

to

data

movement

Second

the

schedule

for

interactions

among

processors

is

such

that

it

can

cause

processors

to

idle

Third

each

transaction

has

to

visit

multiple

hash

trees

causing

redundant

computation

In

this

paper

we

present

two

parallel

algorithms

for

mining

association

rules

We

first

present

Intelligent

Data

Distribution

algorithm

that

improves

upon

the

Data

Distribution

algorithm

by

minimizing

communicationand

idling

overhead

and

by

eliminating

redundant

computation

The

Hybrid

Distribution

algorithm

further

improves

upon

the

Intelligent

Data

Distribution

algorithm

by

dynamically

grouping

processors

and

partitioning

the

candidate

set

accordingly

to

maintain

good

load

balance

The

experimental

results

on

a

Cray

parallel

computer

show

that

the

Hybrid

Distribution

algorithm

scales

linearly

exploits

the

aggregate

memory

efficiently

and

can

generate

more

association

rules

with

a

single

scan

of

database

per

pass

The

rest

of

this

paper

is

organized

as

follows

Section

provides

an

overview

of

the

serial

algorithm

for

mining

association

rules

Section

describes

existing

and

proposed

parallel

algorithms

Section

presents

the

performance

analysis

of

the

algorithms

Experimental

results

are

shown

in

Section

Section

contains

conclusions

A

preliminary

version

of

this

paper

appeared

in

-LSB-

-RSB-

TID

Items

Bread

Coke

Milk

Beer

Bread

Beer

Coke

Diaper

Milk

Beer

Bread

Diaper

Milk

Coke

Diaper

Milk

Table

Transactions

from

supermarket

Basic

Concepts

Let

T

be

the

set

of

transactions

where

each

transaction

is

a

subset

of

the

item-set

I

Let

C

be

a

subset

of

I

then

we

define

the

support

count

of

C

with

respect

to

T

to

be

oe

C

D

jftjt

T

C

t

gj

Thus

oe

C

is

the

number

of

transactions

that

contain

C

For

example

consider

a

set

of

transactions

from

supermarket

as

shown

in

Table

The

items

set

I

for

these

transactions

is

fBread

Beer

Coke

Diaper

Milkg

The

support

count

of

fDiaper

Milkg

is

oe

Diaper

Mi

lk

D

whereas

oe

Diaper

Mi

lk

Beer

D

An

association

rule

is

an

expression

of

the

form

X

s

ff

H

-RRB-

Y

where

X

I

and

Y

I

The

support

s

of

the

rule

X

s

ff

H

-RRB-

Y

is

defined

as

oe

X

-LSB-

Y

jT

j

and

the

confidence

ff

is

defined

as

oe

X

-LSB-

Y

oe

X

For

example

consider

a

rule

fDiaper

Milkg

H

-RRB-

fBeerg

i.e.

presence

of

diaper

and

milk

in

a

transaction

tends

to

indicate

the

presence

of

beer

in

the

transaction

The

support

of

this

rule

is

oe

Diaper

Mi

lk

Beer

D

The

confidence

of

this

rule

is

oe

Diaper

Mi

lk

Beer

oe

Diaper

Mi

lk

D

A

rule

that

has

a

very

high

confidence

-LRB-

i.e.

close

to

-RRB-

is

often

very

important

because

it

provides

an

accurate

prediction

on

the

association

of

the

items

in

the

rule

The

support

of

a

rule

is

also

important

since

it

indicates

how

frequent

the

rule

is

in

the

transactions

Rules

that

have

very

small

support

are

often

uninteresting

since

they

do

not

describe

significantly

large

populations

This

is

one

of

the

reasons

why

most

algorithms

-LSB-

-RSB-

disregard

any

rules

that

do

not

satisfy

the

minimum

support

condition

specified

by

the

user

This

filtering

due

to

the

minimum

required

support

is

also

critical

in

reducing

the

number

of

derived

association

rules

to

a

manageable

size

Note

that

the

total

number

of

possible

rules

is

proportional

to

the

number

of

subsets

of

the

item-set

I

which

is

j.

Hence

the

filtering

is

absolutely

necessary

in

most

practical

settings

The

task

of

discovering

an

association

rule

is

to

find

all

rules

X

s

ff

H

-RRB-

Y

such

that

s

is

greater

than

a

given

minimum

support

threshold

and

ff

is

greater

than

a

given

minimum

confidence

threshold

The

association

rule

discovery

is

composed

of

two

steps

The

first

step

is

to

discover

all

the

frequent

item-sets

-LRB-

candidate

sets

that

have

more

support

than

the

minimum

support

threshold

specified

-RRB-

The

second

step

is

to

generate

association

rules

from

these

frequent

item-sets

The

computation

of

finding

the

frequent

item-sets

is

much

more

expensive

than

finding

the

rules

from

these

frequent

item-sets

Hence

in

this

paper

we

only

focus

on

the

first

step

The

parallel

implementation

of

the

second

step

is

straightforward

and

is

discussed

in

-LSB-

-RSB-

A

number

of

algorithms

have

been

developed

for

discovering

frequent

item-sets

-LSB-

-RSB-

Our

parallel

algorithms

are

based

on

the

Apriori

algorithm

-LSB-

-RSB-

that

has

smaller

computational

complexity

compared

to

other

algorithms

In

the

rest

of

this

section

we

briefly

describe

the

Apriori

algorithm

The

reader

should

refer

to

-LSB-

-RSB-

for

further

details

The

high

level

structure

of

the

Apriori

algorithm

is

given

in

Figure

The

Apriori

algorithm

consists

of

a

number

of

f

frequent

for

-LRB-

k

Fk

OE

k

-RRB-

f

Ck

apriori

gen

-LRB-

Fk

-RRB-

for

all

transactions

t

T

f

subset

-LRB-

Ck

t

-RRB-

g

Fk

fc

Ck

j

c.count

minsupg

g

Answer

S

Fk

Figure

Apriori

Algorithm

passes

Initially

contains

all

the

items

-LRB-

i.e.

item

set

of

size

one

-RRB-

that

satisfy

the

minimum

support

requirement

During

pass

k

the

algorithm

finds

the

set

of

frequent

item-sets

Fk

of

size

k

that

satisfy

the

minimum

support

requirement

The

algorithm

terminates

when

Fk

is

empty

In

each

pass

the

algorithm

first

generates

Ck

the

candidate

item-sets

of

size

k.

Function

apriori

gen

-LRB-

Fk

-RRB-

constructs

Ck

by

extending

frequent

item-sets

of

size

k

This

ensures

that

all

the

subsets

of

size

k

of

a

new

candidate

item-set

are

in

Fk

Once

the

candidate

item-sets

are

found

their

frequencies

are

computed

by

counting

how

many

transactions

contain

these

candidate

item-sets

Finally

Fk

is

generated

by

pruning

Ck

to

eliminate

item-sets

with

frequencies

smaller

than

the

minimum

support

The

union

of

the

frequent

item-sets

S

Fk

is

the

frequent

item-sets

from

which

we

generate

association

rules

Computing

the

counts

of

the

candidate

item-sets

is

the

most

computationally

expensive

step

of

the

algorithm

One

naive

way

to

compute

these

counts

is

to

perform

string-matching

of

each

transaction

against

each

candidate

itemset

A

faster

way

of

performing

this

operation

is

to

use

a

candidate

hash

tree

in

which

the

candidate

item-sets

are

hashed

-LSB-

-RSB-

Here

we

explain

this

via

an

example

to

facilitate

the

discussions

of

parallel

algorithms

and

their

analysis

Figure

shows

one

example

of

the

candidate

hash

tree

with

candidates

of

size

The

internal

nodes

of

the

hash

tree

have

hash

tables

that

contain

links

to

child

nodes

The

leaf

nodes

contain

the

candidate

item-sets

A

hash

tree

of

candidate

item-sets

is

constructed

as

follows

Initially

the

hash

tree

contains

only

a

root

node

which

is

a

leaf

node

containing

no

candidate

item-set

When

each

candidate

item-set

is

generated

the

items

in

the

set

are

stored

in

sorted

order

Each

candidate

item-set

is

inserted

into

the

hash

tree

by

hashing

each

successive

item

at

the

internal

nodes

and

then

following

the

links

in

the

hash

table

Once

a

leaf

is

reached

the

candidate

item-set

is

inserted

at

the

leaf

if

the

total

number

of

candidate

item-sets

are

less

than

the

maximum

allowed

If

the

total

number

of

candidate

item-sets

at

the

leaf

exceeds

the

maximum

allowed

and

the

depth

of

the

leaf

is

less

than

k

the

leaf

node

is

converted

into

an

internal

node

and

child

nodes

are

created

for

the

new

internal

node

The

candidate

item-sets

are

distributed

to

the

child

nodes

according

to

the

hash

values

of

the

items

For

example

the

candidate

item

set

is

inserted

by

hashing

item

at

the

root

to

reach

the

left

child

node

of

the

root

hashing

item

at

that

node

to

reach

the

middle

child

node

hashing

item

to

reach

the

left

child

node

which

is

a

leaf

node

The

subset

function

traverses

the

hash

tree

from

the

root

with

every

item

in

a

transaction

as

a

possible

starting

item

of

a

candidate

In

the

next

level

of

the

tree

all

the

items

of

the

transaction

following

the

starting

item

are

hashed

This

is

done

recursively

until

a

leaf

is

reached

At

this

time

all

the

candidates

at

the

leaf

are

checked

against

the

transaction

and

their

counts

are

updated

accordingly

Figure

shows

the

subset

operation

at

the

first

level

of

the

tree

with

transaction

The

item

is

hashed

to

the

left

child

node

of

the

root

and

the

following

transaction

is

applied

Hash

Function

Transaction

Candidate

Hash

Tree

Figure

Subset

operation

on

the

root

of

a

candidate

hash

tree

Transaction

Candidate

Hash

Tree

Figure

Subset

operation

on

the

left

most

subtree

of

the

root

of

a

candidate

hash

tree

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Proc

Proc

Proc

Proc

N/P

Data

Count

N/P

Data

Count

N/P

Data

Count

N/P

Data

Count

N

number

of

data

items

M

size

of

candidate

set

P

number

of

processors

Global

Reduction

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

M

M

M

M

Figure

Count

Distribution

-LRB-

CD

-RRB-

Algorithm

recursively

to

the

left

child

node

The

item

is

hashed

to

the

middle

child

node

of

the

root

and

the

whole

transaction

is

checked

against

two

candidate

item-sets

in

the

middle

child

node

Then

item

is

hashed

to

the

right

child

node

of

the

root

and

the

following

transaction

is

applied

recursively

to

the

right

child

node

Figure

shows

the

subset

operation

on

the

left

child

node

of

the

root

Here

the

items

and

are

hashed

to

the

middle

child

node

and

the

following

transactions

and

respectively

are

applied

recursively

to

the

middle

child

node

The

item

is

hashed

to

the

right

child

node

and

the

remaining

transaction

is

applied

recursively

to

the

right

child

node

Parallel

Algorithms

In

this

section

we

will

focus

on

the

parallelization

of

the

task

that

finds

all

frequent

item-sets

We

first

discuss

two

parallel

algorithms

proposed

in

-LSB-

-RSB-

to

help

motivate

our

parallel

formulations

In

all

our

discussions

we

assume

that

the

transactions

are

evenly

distributed

among

the

processors

Count

Distribution

Algorithm

In

the

Count

Distribution

-LRB-

CD

-RRB-

algorithm

proposed

in

-LSB-

-RSB-

each

processor

computes

how

many

times

all

the

candidates

appear

in

the

locally

stored

transactions

This

is

done

by

building

the

entire

hash

tree

that

corresponds

to

all

the

candidates

and

then

performinga

single

pass

over

the

locally

stored

transactions

to

collect

the

counts

The

global

counts

of

the

candidates

are

computedby

summing

these

individual

counts

using

a

global

reduction

operation

-LSB-

-RSB-

This

algorithm

is

illustrated

in

Figure

Note

that

since

each

processor

needs

to

build

a

hash

tree

for

all

the

candidates

these

hash

trees

are

identical

at

each

processor

Thus

excluding

the

global

reduction

each

processor

in

the

CD

algorithm

executes

the

serial

Apriori

algorithm

on

the

locally

stored

transactions

This

algorithm

has

been

shown

to

scale

linearly

with

the

number

of

transactions

-LSB-

-RSB-

This

is

because

each

processor

can

compute

the

counts

independently

of

the

other

processors

and

needs

to

communicate

with

the

other

processors

only

once

at

the

end

of

the

computation

step

However

this

algorithm

works

well

only

when

the

hash

trees

can

fit

into

the

main

memory

of

each

processor

If

the

number

of

candidates

is

large

then

the

hash

tree

does

not

fit

into

the

main

memory

In

this

case

this

algorithm

has

to

partition

the

hash

tree

and

compute

the

counts

by

scanning

the

database

multiple

times

once

for

each

partition

of

the

hash

tree

Note

that

the

number

of

candidates

increases

if

either

the

number

of

distinct

items

in

the

database

increases

or

if

the

minimum

support

level

of

the

association

rules

decreases

Thus

the

CD

algorithm

is

effective

for

small

number

of

distinct

items

and

a

high

minimum

support

level

Data

Distribution

Algorithm

The

Data

Distribution

-LRB-

DD

-RRB-

algorithm

-LSB-

-RSB-

addresses

the

memory

problem

of

the

CD

algorithm

by

partitioning

the

candidate

item-sets

among

the

processors

This

partitioning

is

done

in

a

round

robin

fashion

Each

processor

is

responsible

for

computing

the

counts

of

its

locally

stored

subset

of

the

candidate

item-sets

for

all

the

transactions

in

the

database

In

order

to

do

that

each

processor

needs

to

scan

the

portions

of

the

transactions

assigned

to

the

other

processors

as

well

as

its

locally

stored

portion

of

the

transactions

In

the

DD

algorithm

this

is

done

by

having

each

processor

receive

the

portions

of

the

transactions

stored

in

the

other

processors

as

follows

Each

processor

allocates

P

buffers

-LRB-

each

one

page

long

and

one

for

each

processor

-RRB-

At

processor

Pi

the

i

t

h

buffer

is

used

to

store

transactions

from

the

locally

stored

database

and

the

remaining

buffers

are

used

to

store

transactions

from

the

other

processors

such

that

buffer

j

stores

transactions

from

processor

Pj

Now

each

processor

Pi

checks

the

P

buffers

to

see

which

one

contains

data

Let

k

be

this

buffer

-LRB-

ties

are

broken

in

favor

of

buffers

of

other

processors

and

ties

among

buffers

of

other

processors

are

broken

arbitrarily

-RRB-

The

processor

processes

the

transactions

in

this

buffer

and

updates

the

counts

of

its

own

candidate

subset

If

this

buffer

corresponds

to

the

buffer

that

stores

local

transactions

-LRB-

i.e.

k

D

i

-RRB-

then

it

is

sent

to

all

the

other

processors

-LRB-

via

asynchronous

sends

-RRB-

and

a

new

page

is

read

from

the

local

database

If

this

buffer

corresponds

to

a

buffer

that

stores

transactions

from

another

processor

-LRB-

i.e.

k

i

-RRB-

then

it

is

cleared

and

an

asynchronous

receive

request

is

issued

to

processor

Pk

This

continues

until

every

processor

has

processed

all

the

transactions

Having

computed

the

counts

of

its

candidate

item-sets

each

processor

finds

the

frequent

item-sets

from

its

candidate

item-set

and

these

frequent

item-sets

are

sent

to

every

other

processor

using

an

all-to-all

broadcast

operation

-LSB-

-RSB-

Figure

shows

the

high

level

operations

of

the

algorithm

Note

that

each

processor

has

a

different

set

of

candidates

in

the

candidate

hash

tree

This

algorithm

exploits

the

total

available

memory

better

than

CD

as

it

partitions

the

candidate

set

among

processors

As

the

number

of

processors

increases

the

number

of

candidates

that

the

algorithm

can

handle

also

increases

However

as

reported

in

-LSB-

-RSB-

the

performance

of

this

algorithm

is

significantly

worse

than

the

CD

algorithm

The

run

time

of

this

algorithm

is

to

times

more

than

that

of

the

CD

algorithm

on

processors

-LSB-

-RSB-

The

problem

lies

with

the

communication

pattern

of

the

algorithm

and

the

redundant

work

that

is

performed

in

processing

all

the

transactions

The

communication

pattern

of

this

algorithm

causes

three

problems

First

during

each

pass

of

the

algorithm

each

processor

sends

to

all

the

other

processors

the

portion

of

the

database

that

resides

locally

In

particular

each

processor

reads

the

locally

stored

portion

of

the

database

one

page

at

a

time

and

sends

it

to

all

the

other

processors

by

issuing

P

send

operations

Similarly

each

processor

issues

a

receive

operation

from

each

other

processor

in

order

to

receive

these

pages

If

the

interconnection

network

of

the

underlying

parallel

computer

is

fully

connected

-LRB-

i.e.

there

is

a

direct

link

between

all

pairs

of

processors

-RRB-

and

each

processor

can

receive

data

on

all

incoming

links

simultane

Proc

Proc

Proc

Proc

N/P

N/P

N/P

M/P

M/P

M/P

M/P

Local

Data

Remote

Data

CountCount

Local

Data

Remote

Data

CountCount

Local

Data

Remote

Data

CountCount

Local

Data

Remote

Data

CountCount

N/PData

N

number

of

data

items

M

size

of

candidate

set

P

number

of

processors

Data

Data

Broadcast

Broadcast

Broadcast

All-to-all

Broadcast

Data

Broadcast

Broadcast

Data

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Figure

Data

Distribution

-LRB-

DD

-RRB-

Algorithm

ously

then

this

communication

pattern

will

lead

to

a

very

good

performance

In

particular

if

O.N

=P

is

the

size

of

the

database

assigned

locally

to

each

processor

the

amount

of

time

spent

in

the

communication

will

be

O.N

However

on

all

realistic

parallel

computers

the

processors

are

connected

via

a

sparser

networks

-LRB-

such

as

or

hypercube

-RRB-

and

a

processor

can

receive

data

from

-LRB-

or

send

data

to

-RRB-

only

one

other

processor

at

a

time

On

such

machines

this

communication

pattern

will

take

significantly

more

than

O.N

time

because

of

contention

Second

with

finite

number

of

communication

buffers

in

each

processor

the

proposed

all-to-all

communication

scheme

causes

processors

to

idle

For

instance

consider

the

case

when

one

processor

finishes

its

operation

on

local

data

and

sends

the

buffer

to

all

other

processors

Now

if

the

communication

buffer

of

any

receiving

processors

is

full

the

send

operation

is

blocked

Third

if

we

look

at

the

size

of

the

candidate

sets

as

a

function

of

the

number

of

passes

of

the

algorithm

we

see

that

in

the

first

few

passes

the

size

of

the

candidate

sets

increases

and

after

that

it

decreases

In

particular

during

the

last

several

passes

of

the

algorithm

there

are

only

a

small

number

of

items

in

the

candidate

sets

However

each

processor

in

the

DD

algorithm

still

sends

the

locally

stored

portions

of

the

database

to

all

the

other

processors

Thus

even

though

the

computation

decreases

the

amount

of

communication

remains

the

same

The

redundant

work

is

introduced

due

to

the

fact

that

every

processor

has

to

process

every

single

transaction

in

the

database

In

CD

-LRB-

see

Figure

-RRB-

only

N

=P

transactions

go

through

each

hash

tree

of

M

candidates

whereas

in

DD

-LRB-

see

Figure

-RRB-

all

N

transactions

have

to

go

through

each

hash

tree

of

M

=P

candidates

Although

the

number

of

candidates

stored

at

each

processor

has

been

reduced

by

a

factor

of

P

the

amount

of

computation

performed

for

each

transaction

has

not

been

proportionally

reduced

If

the

amount

of

work

required

for

each

transaction

to

be

checked

against

the

hash

tree

of

M

=P

candidates

is

=P

of

that

of

the

hash

tree

of

M

candidates

then

there

is

no

extra

work

As

discussed

in

Section

in

general

the

amount

of

work

per

transaction

will

go

down

by

a

factor

much

smaller

than

P.

while

-LRB-

done

-RRB-

f

FillBuffer

-LRB-

fd

SBuf

-RRB-

for

-LRB-

k

k

k

-RRB-

f

send/receive

data

in

non-blocking

pipeline

MPI

Irecv

-LRB-

RBuf

left

-RRB-

MPI

Isend

-LRB-

SBuf

right

-RRB-

process

transactions

in

SBuf

and

update

hash

tree

Subset

-LRB-

HTree

SBuf

-RRB-

MPI

Waitall

-LRB-

-RRB-

swap

two

buffers

tmp

SBuf

SBuf

RBuf

RBuf

tmp

g

process

transactions

in

SBuf

and

update

hash

tree

Subset

-LRB-

HTree

SBuf

-RRB-

g

Figure

Pseudo

Code

for

Data

Movements

Intelligent

Data

Distribution

Algorithm

We

developed

the

Intelligent

Data

Distribution

-LRB-

IDD

-RRB-

algorithm

that

solves

the

problems

of

the

DD

algorithm

discussed

in

Section

In

IDD

the

locally

stored

portions

of

the

database

are

sent

to

all

the

other

processors

by

using

a

ringbased

all-to-all

broadcast

described

in

-LSB-

-RSB-

This

operation

does

not

suffer

from

the

contention

problems

of

the

DD

algorithm

and

it

takes

O.N

time

on

any

parallel

architecture

that

can

be

embedded

in

a

ring

Figure

shows

the

pseudo

code

for

this

data

movement

operation

In

our

algorithm

the

processors

form

a

logical

ring

and

each

processor

determines

its

right

and

left

neighboring

processors

Each

processor

has

one

send

buffer

-LRB-

SBuf

-RRB-

and

one

receive

buffer

-LRB-

RBuf

-RRB-

Initially

the

SBuf

is

filled

with

one

block

of

local

data

Then

each

processor

initiates

an

asynchronous

send

operation

to

the

right

neighboring

processor

with

SBuf

and

an

asynchronous

receive

operation

to

the

left

neighboring

processor

with

RBuf

While

these

asynchronous

operations

are

proceeding

each

processor

processes

the

transactions

in

SBuf

and

collects

the

counts

of

the

candidates

assigned

to

the

processor

After

this

operation

each

processor

waits

until

these

asynchronous

operations

complete

Then

the

roles

of

SBuf

and

RBuf

are

switched

and

the

above

operations

continue

for

P

times

Compared

to

DD

where

all

the

processors

send

data

to

all

other

processors

we

perform

only

a

point-to-point

communication

between

neighbors

thus

eliminating

any

communication

contention

Furthermore

if

the

time

to

process

a

buffer

does

not

vary

much

then

there

is

little

time

lost

in

idling

In

order

to

eliminate

the

redundant

work

due

to

the

partitioning

of

the

candidate

item-sets

we

must

find

a

fast

way

to

check

whether

a

given

transaction

can

potentially

contain

any

of

the

candidates

stored

at

each

processor

This

can

not

be

done

by

partitioning

Ck

in

a

round-robin

fashion

However

if

we

partition

Ck

among

processors

in

such

a

way

that

each

processor

gets

item-sets

that

begin

only

with

a

subset

of

all

possible

items

then

we

can

check

the

items

of

a

transaction

against

this

subset

to

determine

if

the

hash

tree

contains

candidates

starting

with

these

items

We

traverse

the

hash

tree

with

only

the

items

in

the

transaction

that

belong

to

this

subset

Thus

we

solve

the

redundant

work

problem

of

DD

by

Proc

Proc

Proc

Proc

N/P

N/P

N/P

Local

Data

Remote

Data

Local

Data

Remote

Data

Local

Data

Remote

Data

Local

Data

Remote

Data

N/PData

DataData

N

number

of

data

items

M

size

of

candidate

set

P

number

of

processors

Shift

Shift

Shift

Data

Shift

Data

Shift

M/P

M/P

M/P

M/P

All-to-all

Broadcast

Count

Count

Count

Count

Count

Count

Count

Count

Candidate

Hash

Tree

Candidate

Hash

TreeCandidate

Hash

Tree

Candidate

Hash

Tree

Bit

Map

Bit

Map

Bit

Map

Bit

Map

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Figure

Intelligent

Data

Distribution

-LRB-

IDD

-RRB-

Algorithm

the

intelligent

partitioning

of

Ck

Figure

shows

the

high

level

picture

of

the

algorithm

In

this

example

Processor

has

all

the

candidates

starting

with

items

and

Processor

has

all

the

candidates

starting

with

and

and

so

on

Each

processor

keeps

the

first

items

of

the

candidates

it

has

in

a

bit-map

In

the

Apriori

algorithm

at

the

root

level

of

hash

tree

every

item

in

a

transaction

is

hashed

and

checked

against

the

hash

tree

However

in

our

algorithm

at

the

root

level

each

processor

filters

every

item

of

the

transaction

by

checking

against

the

bit-map

to

see

if

the

processor

contains

candidates

starting

with

that

item

of

the

transaction

If

the

processor

does

not

contain

the

candidates

starting

with

that

item

the

processing

steps

involved

with

that

item

as

the

first

item

in

the

candidate

can

be

skipped

This

reduces

the

amount

of

transaction

data

that

has

to

go

through

the

hash

tree

thus

reducing

the

computation

For

example

let

be

a

transaction

that

processor

is

processing

in

the

subset

function

discussed

in

Section

At

the

top

level

of

the

hash

tree

processor

will

only

proceed

with

items

and

-LRB-

i.e.

and

-RRB-

When

the

page

containing

this

transaction

is

shifted

to

processor

this

processor

will

only

process

items

starting

with

and

-LRB-

i.e.

and

-RRB-

Figure

shows

how

this

scheme

works

when

a

processor

contains

only

those

candidate

item-sets

that

start

with

and

Thus

for

each

transaction

in

the

database

our

approach

partitions

the

amount

of

work

to

be

performed

among

processors

thus

eliminating

any

redundant

work

Note

that

both

the

judicious

partitioning

of

the

hash

tree

-LRB-

indirectly

caused

by

the

partitioning

of

candidate

item-set

-RRB-

and

the

filtering

step

are

required

to

eliminate

this

redundant

work

The

intelligent

partitioning

of

the

candidate

set

used

in

IDD

requires

our

algorithm

to

have

a

good

load

balancing

One

of

the

criteria

of

a

good

partitioning

involved

here

is

to

have

an

equal

number

of

candidates

in

all

the

processors

This

gives

about

the

same

size

hash

tree

in

all

the

processors

and

thus

provides

good

load

balancing

among

processors

Note

that

in

the

DD

algorithm

this

was

accomplished

by

distributing

candidates

in

a

round

robin

fashion

A

naive

Candidate

Hash

Tree

Transaction

Skipped

bitmap

Figure

Subset

operation

on

the

root

of

a

candidate

hash

tree

in

IDD

method

for

assigning

candidates

to

processors

can

lead

to

a

significant

load

imbalance

For

instance

consider

a

database

with

distinct

items

numbered

from

to

and

that

the

database

transactions

have

more

data

items

numbered

with

to

If

we

partition

the

candidates

between

two

processors

and

assign

all

the

candidates

starting

with

items

to

to

processor

and

candidates

starting

with

items

to

to

processor

then

there

would

be

more

work

for

processor

To

achieve

a

load-balanced

distribution

of

the

candidate

item-sets

we

use

a

partitioning

algorithm

that

is

based

on

bin-packing

-LSB-

-RSB-

For

each

item

we

first

compute

the

number

of

candidate

item-sets

starting

with

this

particular

item

Note

that

at

this

time

we

do

not

actually

store

the

candidate

item-sets

but

just

store

the

number

of

candidate

item-sets

starting

with

each

item

We

then

use

a

bin-packing

algorithm

to

partition

these

items

in

P

buckets

such

that

the

numbers

of

the

candidate

item-sets

starting

with

these

items

in

each

bucket

are

equal

To

remove

any

data

skew

our

bin-packing

algorithm

randomly

selects

the

item

to

be

assigned

next

in

a

bin

Once

the

location

of

each

candidate

item-set

is

determined

then

each

processor

locally

regenerates

and

stores

candidate

item-sets

that

are

assigned

to

this

processor

Figure

shows

the

partitioned

candidate

hash

tree

and

its

corresponding

bitmaps

in

each

processor

We

were

able

to

achieve

less

than

of

load

imbalance

with

the

bin

packing

method

described

here

Hybrid

Algorithm

The

IDD

algorithm

exploits

the

total

system

memory

by

partitioning

the

candidate

set

among

all

processors

The

average

number

of

candidates

assigned

to

each

processor

is

M

=P

where

M

is

the

number

of

total

candidates

As

more

processors

are

used

the

number

of

candidates

assigned

to

each

processor

decreases

This

has

two

implications

First

with

fewer

number

of

candidates

per

processor

it

is

much

more

difficult

to

balance

the

work

Second

the

smaller

number

of

candidates

gives

a

smaller

hash

tree

and

less

computation

work

per

transaction

Eventually

the

amount

of

computation

may

become

less

than

the

communication

involved

This

would

be

more

evident

in

the

later

passes

of

the

algorithm

as

the

hash

tree

size

further

decreases

dramatically

This

reduces

overall

efficiency

of

the

parallel

algorithm

This

will

be

an

even

more

serious

problem

in

a

system

that

can

not

perform

asynchronous

communication

The

Hybrid

Distribution

-LRB-

HD

-RRB-

algorithm

addresses

the

above

problem

by

combining

the

CD

and

the

IDD

algorithms

in

the

following

way

Consider

a

P-processor

system

in

which

the

processors

are

split

into

G

equal

size

groups

each

containing

P

G

processors

In

the

HD

algorithm

we

execute

the

CD

algorithm

as

if

there

were

only

P

G

processors

That

is

we

partition

the

transactions

of

the

database

into

P

G

parts

each

of

size

N

P

G

and

assign

the

task

of

computing

the

counts

of

the

candidate

set

Ck

for

each

subset

of

the

transactions

to

each

one

of

these

groups

of

processors

Within

each

group

these

counts

are

computed

using

the

IDD

algorithm

That

is

the

transactions

and

the

candidate

set

Ck

are

partitioned

among

the

processors

of

each

group

so

that

each

processor

gets

roughly

jCk

j

G

candidate

item-sets

and

N

=P

transactions

Now

each

group

of

processors

computes

the

counts

using

the

IDD

algorithm

and

the

overall

counts

are

computing

by

performing

a

reduction

operation

among

the

P

G

groups

of

processors

The

HD

algorithm

can

be

better

visualized

if

we

think

of

the

processors

as

being

arranged

in

a

two

dimensional

grid

of

G

rows

and

P

G

columns

The

transactions

are

partitioned

equally

among

the

P

processors

The

candidate

set

Ck

is

partitioned

among

the

processors

of

each

column

of

this

grid

This

partitioning

of

Ck

is

identical

for

each

column

of

processors

i.e.

the

processors

along

each

row

of

the

grid

get

the

same

subset

of

Ck

Figure

illustrates

the

HD

algorithm

for

a

grid

of

processors

In

this

example

the

HD

algorithm

executes

the

CD

algorithm

as

if

there

were

only

processors

where

the

processors

correspond

to

the

processor

columns

That

is

the

database

transactions

are

partitioned

in

parts

and

each

one

of

these

hypothetical

processors

computes

the

local

counts

of

all

the

candidate

item-sets

Then

the

global

counts

can

be

computed

by

performing

the

global

reduction

operation

discussed

in

Section

However

since

each

one

of

these

hypothetical

processors

is

made

up

of

processors

the

computation

of

local

counts

of

the

candidate

item-sets

in

a

hypothetical

processor

requires

the

computation

of

the

counts

of

the

candidate

item-sets

on

the

database

transactions

sitting

on

the

processors

This

operation

is

performed

by

executing

the

IDD

algorithm

within

each

of

hypothetical

processors

This

is

shown

in

the

step

of

Figure

Note

that

processors

in

the

same

row

have

exactly

the

same

candidates

and

candidate

sets

along

the

each

column

partition

the

total

candidate

set

At

the

end

of

this

operation

each

processor

has

complete

count

of

its

local

candidates

for

all

the

transactions

located

in

the

processors

of

the

same

column

-LRB-

i.e.

of

a

hypothetical

processor

-RRB-

Now

a

reduction

operation

is

performed

along

the

rows

such

that

all

processors

in

each

row

have

the

sum

of

the

counts

for

the

candidates

in

the

same

row

At

this

point

the

count

associated

with

each

candidate

item-set

corresponds

to

the

entire

database

of

transactions

Now

each

processor

finds

frequent

item-sets

by

dropping

all

those

candidate

item-sets

whose

frequency

is

less

than

the

threshold

for

minimum

support

These

candidate

item-sets

are

shown

as

shaded

in

Figure

-LRB-

b

-RRB-

In

the

next

step

each

processor

performs

all-to-all

broadcast

operation

along

the

columns

of

the

processor

mesh

At

this

point

all

the

processors

have

the

frequent

sets

and

are

ready

to

proceed

to

the

next

pass

The

HD

algorithm

determines

the

configuration

of

the

processor

grid

dynamically

In

particular

the

HD

algorithm

partitions

the

candidate

set

into

a

big

enough

section

and

assign

a

group

of

processors

to

each

partition

Let

m

be

the

number

of

candidate

item-sets

such

that

each

processor

can

hold

the

hash

tree

constructed

from

m

candidate

item-sets

in

memory

If

the

total

number

of

candidates

M

is

less

than

m

then

the

HD

algorithm

makes

G

equal

to

which

means

that

the

CD

algorithm

is

run

on

all

the

processors

Otherwise

G

is

set

to

dM

me

The

HD

algorithm

inherits

all

the

good

features

of

the

IDD

algorithm

It

also

provides

good

load

balance

and

enough

computation

work

by

maintaining

minimum

number

of

candidates

per

processor

At

the

same

time

the

amount

of

data

movement

in

this

algorithm

has

been

cut

down

to

G

of

the

IDD

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

Frequent

Item

Set

All-to-all

Broadcast

All-to-all

Broadcast

All-to-all

Broadcast

All-to-all

Broadcast

Step

Reduction

Operation

Along

the

Rows

Step

All-to-all

Broadcast

Operation

Along

the

Columns

Step

Partitioning

of

Candidate

Sets

and

Data

Movement

Along

the

Columns

Data

Shift

Data

Shift

Data

Shift

Data

Shift

Data

Shift

Data

Shift

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Data

Shift

Data

Shift

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

Tree

Data

Shift

Data

Shift

Data

Shift

Data

Shift

Candidate

Hash

Tree

Candidate

Hash

TreeCandidate

Hash

TreeCandidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

TreeCandidate

Hash

TreeCandidate

Hash

Tree

Candidate

Hash

Tree

Candidate

Hash

TreeCandidate

Hash

TreeCandidate

Hash

Tree

Figure

Hybrid

Distribution

-LRB-

HD

-RRB-

Algorithm

in

Processor

Mesh

-LRB-

G

D

P

D

-RRB-

symbol

definition

N

Total

number

of

transactions

P

Number

of

processors

M

Total

number

of

candidates

G

Number

of

partitions

of

candidates

in

the

HD

algorithm

k

Pass

number

in

Apriori

algorithm

I

Average

number

of

items

in

a

transaction

C

Average

number

of

potential

candidates

in

a

transaction

S

Average

number

of

candidates

at

the

leaf

node

L

Average

number

of

leaves

in

the

hash

tree

for

the

serial

Apriori

algorithm

ttravers

Cost

of

hash

tree

traversal

per

potential

candidate

tcheck

Cost

of

checking

at

the

leaf

with

S

candidates

Vi

j

Expected

number

of

leaves

visited

with

i

potential

candidates

and

j

leaves

Table

Symbols

used

in

the

analysis

Performance

Analysis

In

this

section

we

analyze

the

amount

of

work

done

by

each

algorithm

Throughout

the

analysis

we

ignore

the

cost

of

communication

Table

describes

the

symbols

used

in

this

section

As

discussed

in

Section

the

bulk

of

the

computation

is

performed

by

the

subset

function

Consider

a

transaction

that

has

I

items

During

the

k

t

h

pass

of

the

algorithm

this

transaction

has

C

D

I

k

potential

candidates

that

need

to

be

checked

against

the

candidate

hash

tree

Note

that

for

a

given

transaction

if

one

potential

candidate

visits

a

leaf

node

then

all

the

candidates

of

this

transaction

are

checked

against

the

leaf

node

As

a

result

if

this

node

is

revisited

due

to

a

different

candidate

from

the

same

transaction

no

checking

is

performed

Clearly

the

total

cost

of

checking

at

the

leaf

nodes

is

directly

proportional

to

the

number

of

distinct

leaf

nodes

visited

with

the

transaction

We

assume

that

the

average

number

of

candidate

item-sets

at

the

leaf

nodes

is

S.

Hence

the

average

number

of

leaf

nodes

in

a

hash

tree

is

L

D

M

S

In

the

implementation

of

the

algorithm

the

desired

value

of

S

can

be

obtained

by

adjusting

the

branching

factor

of

the

hash

tree

In

general

the

cost

of

traversal

for

each

potential

candidate

will

depend

on

the

depth

of

the

leaf

node

in

the

hash

tree

reached

by

the

traversal

To

simplify

the

analysis

we

assume

that

the

cost

of

each

traversal

is

the

same

Hence

the

total

traversal

cost

if

directly

proportional

to

C

For

each

potential

candidate

we

define

ttravers

to

be

the

cost

associated

with

the

traversal

of

the

hash

tree

and

tcheck

to

be

the

cost

associated

with

checking

the

candidate

item-sets

of

the

reached

leaf

node

Note

that

the

number

of

distinct

leaves

visited

by

a

transaction

is

in

general

smaller

than

the

number

of

potential

candidates

C

This

is

because

different

potential

candidates

may

lead

to

the

same

leaf

node

In

general

if

C

is

relatively

large

with

respect

to

the

number

of

leaf

nodes

in

the

hash

tree

then

the

number

of

distinct

leaf

nodes

visited

will

be

smaller

than

C

We

can

compute

the

expected

number

of

distinct

leaf

nodes

visited

as

follows

To

simplify

the

analysis

we

assume

that

each

traversal

of

the

hash

tree

due

to

a

different

potential

candidate

is

equally

likely

to

lead

to

any

leaf

node

of

the

hash

tree

Let

Vi

j

be

the

expected

number

of

distinct

leaf

nodes

visited

when

the

transaction

has

i

potential

candidates

and

the

hash

tree

has

j

leaf

nodes

j

D

Vi

j

D

Vi

j

Prob

of

reaching

a

previously

visited

node

C

Vi

j

C

Prob

of

reaching

a

new

node

D

Vi

j

Vi

j

j

C

Vi

j

C

j

Vi

j

j

D

C

j

j

Vi

j

D

j

i

j

D

j

i

j

j

i

-LRB-

-RRB-

Note

that

for

large

j

Vi

j

i

This

can

be

shown

by

taking

limit

on

Equation

lim

j

Vi

j

D

lim

j

j

i

j

j

i

D

-LSB-

i

i

-RSB-

j

-LSB-

i

i

-RSB-

j

i

i

D

i

j

i

j

D

i

-LRB-

-RRB-

This

shows

that

if

the

hash

tree

size

is

much

larger

than

the

number

of

potential

candidates

in

a

transaction

then

each

potential

candidate

is

likely

to

visit

a

distinct

leaf

node

in

the

hash

tree

Serial

Apriori

algorithm

Recall

that

in

the

serial

Apriori

algorithm

the

average

number

of

leaf

nodes

in

the

hash

tree

is

L

D

M

S.

Hence

the

number

of

distinct

leaf

visited

per

transaction

is

VC

L

and

the

computation

time

per

transaction

for

visiting

the

hash

tree

is

Ttrans

D

C

ttravers

C

VC

L

tcheck

So

the

run

time

of

the

serial

algorithm

for

processing

N

transactions

is

T

ser

i

al

comp

D

N

Ttrans

D

N

C

ttravers

C

N

VC

L

tcheck

-LRB-

-RRB-

The

CD

algorithm

In

the

CD

algorithm

the

entire

set

of

candidates

is

replicated

at

each

processor

Hence

the

average

number

of

leaf

nodes

in

the

local

hash

tree

at

each

processor

is

L

D

M

S

which

is

the

same

as

in

the

serial

Apriori

algorithm

Thus

the

CD

algorithm

performs

the

same

computation

per

transactions

as

the

serial

algorithm

but

each

processor

handles

only

N

=P

number

of

transactions

Hence

the

run

time

of

the

CD

algorithm

is

T

CD

comp

D

N

P

Ttrans

D

N

P

C

ttravers

C

N

P

VC

L

tcheck

-LRB-

-RRB-

Comparing

Equation

to

Equation

we

see

that

CD

performs

no

redundant

computation

In

particular

both

the

time

for

traversal

and

for

checking

scales

down

by

a

factor

of

P

The

DD

algorithm

In

the

DD

algorithm

the

number

of

candidates

per

processor

is

M

=P

as

the

candidate

set

is

partitioned

Hence

the

average

number

of

leaf

nodes

in

the

local

hash

tree

of

each

processor

is

L

=P

Therefor

the

number

of

distinct

leaf

nodes

visited

per

transaction

is

VC

LP

and

the

computation

time

per

transaction

is

T

DD

trans

D

C

ttravers

C

VC

LP

tcheck

The

number

of

transactions

processed

by

each

processor

is

N

as

the

transactions

are

shifted

around

the

processors

Hence

the

computation

per

processor

of

the

DD

algorithm

is

T

DD

comp

D

N

T

DD

trans

D

N

C

ttravers

C

N

VC

LP

tcheck

-LRB-

-RRB-

Comparing

Equation

with

the

serial

complexity

-LRB-

Equation

-RRB-

we

see

that

the

DD

algorithm

does

not

reduce

the

computation

associated

with

the

hash

tree

traversal

For

both

the

serial

Apriori

and

the

DD

algorithm

this

cost

is

N?C

ttravers

However

the

DD

algorithm

is

able

to

reduce

the

cost

associated

with

the

checking

at

the

leaf

nodes

In

particular

it

reduces

the

serial

cost

of

N

VC

L

tcheck

down

to

N

VC

LP

tcheck

However

because

VC

LP

VC

L

=P

the

reduction

achieved

in

this

part

is

less

than

a

factor

of

P

We

can

easily

see

this

if

we

consider

the

case

when

L

is

very

large

In

this

case

VC

LP

C

and

VC

L

=P

C

=P

by

Equation

Thus

the

number

of

leaf

nodes

checked

over

all

the

processors

by

the

DD

algorithm

is

higher

than

that

of

the

serial

algorithm

This

is

why

the

DD

algorithm

performs

redundant

computation

The

IDD

algorithm

In

the

IDD

algorithm

just

like

the

DD

algorithm

the

average

number

of

leaf

nodes

in

the

local

hash

tree

of

each

processor

is

L

=P

However

the

average

number

of

potential

candidates

that

need

to

be

checked

for

each

transaction

at

each

processor

is

much

less

than

DD

because

of

the

intelligent

partitioning

of

candidates

set

and

the

use

of

bitmap

to

prune

at

the

root

of

the

hash

tree

More

precisely

the

number

of

potential

candidates

that

need

to

be

checked

for

a

transaction

is

roughly

C

=P

assuming

that

we

have

a

good

balanced

partition

So

the

computation

per

transaction

is

T

I

DD

trans

D

C

P

ttravers

C

V

CP

LP

tcheck

Thus

the

computation

per

processor

is

T

I

DD

comp

D

N

T

I

DD

trans

D

N

C

P

ttravers

C

N

V

CP

LP

tcheck

-LRB-

-RRB-

Comparing

Equation

to

Equation

we

see

that

the

IDD

algorithm

is

successful

in

reducing

the

cost

associated

with

the

hash

tree

traversal

linearly

It

also

reduces

the

checking

cost

from

N

VC

L

tcheck

down

to

N

V

CP

LP

tcheck

Note

that

for

sufficiently

large

L

VC

L

C

and

V

CP

LP

C

=P

This

shows

that

IDD

is

also

able

to

linearly

reduce

the

cost

of

checking

at

the

leaf

nodes

and

thus

unlike

DD

it

performs

no

redundant

work

We

instrumented

our

algorithms

to

verify

that

the

average

number

of

distinct

leaf

node

visited

by

IDD

is

indeed

much

less

than

DD

Figure

shows

that

V

CP

LP

of

IDD

goes

down

by

factor

of

P

but

VC

LP

of

DD

does

not

go

down

by

factor

of

P.

However

P

must

be

relatively

small

for

IDD

to

have

a

good

load

balance

If

P

becomes

large

with

fixed

M

the

problem

of

load

imbalance

discussed

in

Section

makes

some

processors

work

on

more

than

=P

of

items

in

a

transaction

at

the

root

of

the

hash

Average

Number

of

Distinct

Leaf

Node

Visited

Per

Transaction

Number

of

processors

DD

IDD

Figure

Comparison

of

DD

and

IDD

in

terms

of

the

average

number

of

distinct

leaf

node

visited

per

transaction

with

transactions

per

processor

and

minimum

support

tree

The

HD

algorithm

In

the

HD

algorithm

the

number

of

potential

candidates

per

transactions

is

C

G

and

the

number

of

candidates

per

processor

is

M

G

So

the

computation

time

per

transaction

is

T

HD

trans

D

C

G

ttravers

C

V

CG

LG

tcheck

The

total

number

of

transactions

each

processor

has

to

process

is

GN

=P

Thus

the

computation

per

processor

is

T

HD

comp

D

G

N

P

T

HD

trans

D

G

N

P

C

G

ttravers

C

G

N

P

V

CG

LG

tcheck

-LRB-

-RRB-

Compared

to

the

serial

algorithm

Equation

shows

that

the

HD

algorithm

reduces

the

computation

linearly

with

respect

to

the

hash

tree

traversal

cost

The

traversal

cost

is

reduced

from

N

C

ttravers

down

to

N

C

ttravers

P

The

cost

of

checking

at

the

leaf

nodes

is

reduced

from

N

VC

L

tcheck

down

to

-LRB-

G

N

V

CG

LG

tcheck

=P

Note

that

for

sufficiently

large

L

N

VC

L

NC

and

G

N

V

CG

LG

=P

N

C

=P

Thus

the

HD

algorithm

has

a

linear

speedup

with

respect

to

the

cost

of

checking

at

the

leaf

nodes

Experimental

Results

We

implemented

our

parallel

algorithms

on

a

Cray

parallel

computer

Each

processor

on

the

is

a

Dec

Alpha

-LRB-

-RRB-

and

has

of

memory

The

processors

are

interconnected

via

a

three

dimensional

torus

network

that

has

a

peak

unidirectional

bandwidth

of

per

second

and

a

small

latency

For

communication

we

used

the

message

passing

interface

-LRB-

MPI

-RRB-

Our

experiments

have

shown

that

for

messages

we

obtain

a

bandwidth

of

and

an

effective

startup

time

of

microseconds

60 80 100 120

Response

time

-LRB-

sec

-RRB-

Number

of

processors

CD

IDD

HD

DD

DD

comm

Figure

Scaleup

result

with

transactions

and

minimum

support

We

generated

a

synthetic

dataset

using

a

tool

provided

by

-LSB-

-RSB-

and

described

in

-LSB-

-RSB-

The

parameters

for

the

data

set

chosen

are

average

transaction

length

of

and

average

size

of

frequent

item

sets

of

Data

sets

with

transactions

-LRB-

-RRB-

were

generated

for

different

processors

Due

to

the

absence

of

a

true

parallel

I/O

system

on

the

system

we

kept

a

set

of

transactions

in

a

main

memory

buffer

and

read

the

transactions

from

the

buffer

instead

of

the

actual

disks

For

the

experiments

involving

larger

data

sets

we

read

the

same

data

set

multiple

times

We

also

performed

similar

experiments

on

an

IBM

in

which

the

entire

database

resided

on

disks

Our

experiments

-LRB-

not

reported

here

-RRB-

show

that

the

I/O

requirements

do

not

change

the

relative

performance

of

the

various

schemes

We

do

present

the

results

of

one

experiment

on

for

comparing

CD

to

IDD

and

HD

when

CD

scans

database

multiple

times

due

to

the

partitioned

hash

tree

To

compare

the

scalability

of

the

four

schemes

-LRB-

CD

DD

IDD

and

HD

-RRB-

we

performed

scaleup

tests

with

transactions

per

processor

and

minimum

support

of

We

could

not

use

minimum

support

smaller

than

because

the

CD

algorithm

ran

out

of

main

memory

for

storing

the

hash

tree

For

this

experiment

in

the

HD

algorithm

we

have

set

the

threshold

on

the

number

of

candidates

for

switching

to

the

CD

algorithm

very

low

to

show

the

validity

of

our

approach

With

support

the

HD

algorithm

switched

to

CD

algorithm

in

pass

of

total

passes

and

of

the

overall

response

time

of

the

serial

code

was

spent

in

the

first

passes

These

scaleup

results

are

shown

in

Figure

As

noted

in

-LSB-

-RSB-

the

DD

algorithm

scales

very

poorly

However

the

performance

achieved

by

IDD

is

much

better

than

that

of

the

DD

algorithm

In

particular

on

processors

IDD

is

faster

than

DD

by

a

factor

of

It

can

be

seen

that

the

performance

gap

between

IDD

and

DD

widens

as

the

number

of

processors

increases

IDD

performs

better

than

DD

because

of

the

better

communication

mechanism

for

data

movements

and

the

intelligent

partitioning

of

the

candidate

set

To

show

the

effects

of

these

two

improvements

we

replaced

the

communication

mechanism

of

the

DD

algorithm

with

that

of

the

IDD

The

scaleup

result

of

this

improvement

is

shown

as

DD

comm

in

Figure

Hence

the

response

time

reduction

from

DD

to

DD

comm

is

due

to

the

the

better

communication

mechanism

for

data

movements

and

the

reduction

from

DD

comm

to

IDD

is

due

to

the

intelligent

partitioning

of

the

candidate

set

Note

that

the

response

time

of

IDD

increases

as

we

increase

the

number

of

processors

This

is

due

to

the

load

balanc

200 400 600 800

Response

time

-LRB-

sec

-RRB-

Number

of

transactions

per

processor

-LRB-

K

-RRB-

count

intelligent

data

hybrid

Figure

Sizeup

result

with

processors

and

minimum

support

ing

problem

discussed

in

Section

where

the

number

of

candidates

per

processor

decreases

as

the

number

of

processors

increases

Looking

at

the

performance

of

the

HD

algorithm

we

see

that

response

time

remains

almost

constant

as

we

increase

the

number

of

processors

while

keeping

the

number

of

transactions

per

processor

and

the

minimum

support

fixed

Comparing

against

CD

we

see

that

HD

actually

performs

better

as

the

number

of

processors

increases

Its

performance

on

processors

is

better

than

CD

This

performance

advantage

of

HD

over

CD

is

due

to

the

smaller

cost

of

reduction

in

HD

because

the

number

of

processors

involved

in

global

reduction

operation

of

counts

is

much

less

in

HD

than

in

CD

We

measuredhow

our

algorithms

perform

as

we

increase

the

number

of

transactions

per

processor

-LRB-

MB

-RRB-

to

-LRB-

MB

-RRB-

For

these

experiments

we

fixed

the

number

of

processors

at

and

the

minimum

support

at

These

results

are

shown

in

Figure

From

this

figure

we

can

see

that

CD

and

HD

perform

almost

identically

For

both

algorithms

the

response

time

increases

linearly

with

the

number

of

transactions

IDD

also

scales

linearly

but

because

of

its

load

imbalance

problem

its

performance

is

somewhat

worse

Our

experiments

so

far

have

shown

that

the

performance

of

HD

and

CD

are

quite

comparable

if

the

entire

hash

tree

can

reside

on

each

processor

However

the

real

advantage

of

HD

-LRB-

and

IDD

-RRB-

over

CD

is

that

they

do

not

require

the

whole

hash

tree

to

reside

on

each

processor

and

thus

better

exploit

the

available

memory

This

allows

us

to

use

a

smaller

minimum

support

in

the

Apriori

algorithm

To

verify

this

we

performed

experiments

in

which

we

fixed

the

number

of

transactions

per

processor

to

and

successively

decreased

the

minimum

support

level

These

experiments

for

processors

are

shown

in

Figures

and

respectively

A

couple

of

interesting

observations

can

be

made

from

these

results

First

both

IDD

and

HD

successfully

ran

using

lower

support

levels

that

CD

could

not

run

with

In

particular

IDD

and

HD

ran

down

to

a

support

level

of

on

processors

and

on

processors

In

contrast

CD

could

only

run

down

to

a

support

level

of

and

ran

out

of

memory

for

the

lower

supports

The

support

level

for

IDD

and

HD

becomes

smaller

with

the

increasing

number

of

processors

because

the

IDD

and

HD

algorithms

can

exploit

the

aggregate

memory

of

the

larger

number

of

processors

The

second

thing

to

notice

is

that

HD

performs

better

than

IDD

both

on

and

processors

and

the

relative

perfor

Response

time

-LRB-

sec

-RRB-

Minimum

support

-LRB-

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

CD

IDD

HD

simple

hybrid

Figure

Response

time

on

processors

with

transactions

as

the

minimum

support

varies

At

each

support

level

the

total

number

of

candidate

item-sets

is

shown

in

parenthesis

mance

of

HD

compared

to

IDD

get

better

as

the

number

of

processors

increases

As

discussed

earlier

this

performance

difference

is

due

to

the

load

imbalance

As

the

number

of

processors

increases

this

load

imbalance

gets

worse

However

on

processors

IDD

is

worse

than

HD

for

support

level

but

only

worse

for

support

of

This

is

because

as

the

support

level

decreases

the

number

of

candidates

-LRB-

shown

in

parenthesis

in

Figures

and

-RRB-

increases

which

improves

the

load

balance

Figures

and

also

show

the

performance

of

a

simple

hybrid

algorithm

obtained

by

combining

CD

and

IDD

In

this

scheme

in

each

pass

of

the

Apriori

algorithm

we

perform

CD

if

the

hash

table

can

fit

in

the

memory

of

each

processors

or

IDD

if

it

can

not

As

we

can

see

from

these

results

this

simple

hybrid

algorithm

performs

worse

than

HD

In

particular

the

relative

performance

of

this

scheme

compared

to

HD

gets

worse

as

the

number

of

processors

increases

For

example

for

a

support

level

of

it

is

worse

on

processors

and

worse

on

processors

Thus

the

HD

algorithm

achieves

better

performance

by

gradually

adjusting

the

subsets

of

processors

that

perform

IDD

and

CD

This

is

because

of

the

following

two

reasons

First

the

candidate

set

is

split

among

fewer

number

of

processors

which

minimizes

load

imbalance

and

second

the

reduction

operation

to

obtain

the

counts

in

CD

is

performed

among

fewer

processors

which

decreases

the

communication

overhead

Note

that

an

alternative

method

to

handle

large

candidate

set

in

CD

is

to

partition

them

such

that

each

partition

fits

in

the

main

memory

Now

the

entire

set

of

local

transactions

have

to

be

read

at

each

processor

as

many

time

as

the

number

of

partitions

This

method

increases

the

I/O

cost

On

the

system

in

which

I/O

is

scalable

and

fast

-LRB-

e.g.

IBM

-RRB-

this

cost

may

be

acceptable

We

implemented

the

CD

algorithm

to

partition

the

hash

tree

and

read

database

multiple

times

in

case

the

hash

tree

does

not

fit

into

main

memory

Figure

shows

the

performance

comparison

of

CD

IDD

and

HD

on

IBM

machine

as

the

number

of

candidates

increases

by

lowering

minimum

support

Unlike

the

earlier

experiments

on

Cray

machine

the

whole

transactions

were

read

in

from

the

file

Figure

shows

that

as

the

number

of

candidates

increases

both

IDD

and

HD

outperform

CD

This

is

due

to

the

increased

I/O

time

required

for

multiple

scan

of

the

database

and

increased

communication

time

required

for

global

reduction

operation

of

multiple

partitions

of

the

candidate

frequencies

Note

that

even

on

IBM

the

penalty

due

to

higher

I/O

cost

is

about

for

million

candidates

for

million

candidates

and

for

million

candidates

The

I/O

penalty

increases

as

the

Response

time

-LRB-

sec

-RRB-

Minimum

support

-LRB-

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

-LRB-

K

-RRB-

CD

IDD

HD

simple

hybrid

Figure

Response

time

on

processors

with

transactions

as

the

minimum

support

varies

At

each

support

level

the

total

number

of

candidate

item-sets

is

shown

in

parenthesis

Number

of

processors

Successful

down

to

Ran

out

of

memory

at

Table

Minimum

support

-LRB-

-RRB-

reachable

with

different

number

of

processors

in

our

algorithms

number

of

candidates

increases

On

systems

with

slower

I/O

the

penalty

can

be

much

larger

In

another

experiment

we

varied

the

number

of

processors

from

to

and

measured

how

low

we

can

go

with

minimum

support

for

the

IDD

and

HD

algorithms

Table

shows

the

result

for

these

algorithms

The

result

shows

that

as

we

have

more

processors

these

algorithms

can

handle

lower

minimum

support

Table

shows

how

the

HD

algorithm

chose

the

processor

configuration

based

on

the

number

of

candidates

at

each

pass

with

processors

and

minimum

support

Conclusion

In

this

paper

we

proposed

two

new

parallel

algorithms

for

mining

association

rules

The

IDD

algorithm

utilizes

total

main

memory

available

more

effectively

than

the

CD

algorithm

This

algorithms

improves

over

the

DD

algorithm

which

has

high

communication

overhead

and

redundant

work

As

shown

in

Section

for

each

transaction

the

DD

Pass

Configuration

No

of

Cand

Table

Processor

configuration

and

number

of

candidates

of

the

HD

algorithm

with

processors

and

minimum

support

for

each

pass

Note

that

configuration

is

the

same

as

the

DD

algorithm

and

is

the

same

as

the

CD

algorithm

The

total

number

of

pass

was

and

all

passes

after

had

configuration

Response

time

-LRB-

sec

-RRB-

Number

of

Candidates

-LRB-

in

milliions

-RRB-

HD

CD

IDD

Figure

Response

time

on

processor

IBM

with

transactions

as

the

minimum

support

varies

from

to

algorithm

performs

substantially

more

work

overall

than

the

serial

Apriori

algorithm

The

communication

and

idling

overheads

were

reduced

using

a

better

data

movement

communicationmechanism

and

redundant

work

was

reduced

by

partitioning

the

candidate

set

intelligently

and

using

bit

maps

to

prune

away

unnecessary

computation

Another

useful

feature

of

IDD

is

that

it

is

well

suited

for

the

system

environment

with

single

source

of

data

base

For

instance

when

all

the

data

is

coming

from

a

database

server

or

a

single

file

system

one

processor

can

read

data

from

the

single

source

and

pass

the

data

along

the

communication

pipe

line

defined

in

the

algorithm

However

as

the

number

of

available

processors

increases

the

efficiency

of

this

algorithm

decreases

due

to

load

imbalance

unless

the

amount

of

work

is

increased

by

having

more

number

of

candidates

The

HD

combines

the

advantages

of

CD

and

IDD

This

algorithm

partitions

candidate

sets

just

like

the

IDD

to

exploit

the

aggregate

main

memory

but

dynamically

determines

the

number

of

partitions

such

that

the

partitioned

candidate

set

fits

into

the

main

memory

of

each

processor

and

each

processor

has

enough

number

of

candidates

for

computation

It

also

exploits

the

advantage

of

CD

by

just

exchanging

counts

information

and

moving

around

the

minimum

number

of

transactions

among

the

smaller

subset

of

processors

The

experimental

results

on

a

Cray

parallel

machine

show

that

the

HD

algorithm

scales

just

as

well

as

the

CD

algorithm

with

respect

to

the

number

of

transactions

However

it

exploits

the

aggregate

main

memory

more

efficiently

and

thus

is

able

to

discover

more

association

rules

with

much

smaller

minimum

support

with

a

single

scan

of

database

per

pass

This

parallel

formulation

is

particularly

useful

in

domains

where

the

size

of

transaction

is

very

large

-LRB-

e.g.

similarity

tables

in

molecular

biology

-LSB-

-RSB-

generalized

association

rules

-LSB-

-RSB-

or

sequential

patterns

-LSB-

-RSB-

-RRB-

References

-LSB-

-RSB-

R.

Agrawal

T.

Imielinski

and

A.

Swami

Mining

association

rules

between

sets

of

items

in

large

databases

In

Proc

of

ACM-SIGMOD

Int

Conf

on

Management

of

Data

Washington

D.C.

-LSB-

-RSB-

R.

Agrawal

and

R.

Srikant

Fast

algorithms

for

mining

association

rules

In

Proc

of

the

VLDB

Conference

pages

Santiago

Chile

-LSB-

-RSB-

R.

Agrawal

and

J.C.

Shafer

Parallel

mining

of

association

rules

IEEE

Transactions

on

Knowledge

and

Data

Eng.

-LRB-

-RRB-

December

-LSB-

-RSB-

J.

Han

and

Y.

Fu

Discovery

of

multiple?level

association

rules

from

large

databases

In

Proc

of

the

VLDB

Conference

Zurich

Switzerland

-LSB-

-RSB-

E.H.

Han

G.

Karypis

and

V.

Kumar

Scalable

parallel

data

mining

for

association

rules

In

Proc

of

ACM-SIGMOD

Int

Conf

on

Management

of

Data

Tucson

Arizona

-LSB-

-RSB-

E.H.

Han

G.

Karypis

V.

Kumar

and

B.

Mobasher

Clustering

based

on

association

rule

hypergraphs

Technical

Report

Department

of

Computer

Science

University

of

Minnesota

Minneapolis

-LSB-

-RSB-

M.

A.

W.

Houtsma

and

A.

N.

Swami

Set-oriented

mining

for

association

rules

in

relational

databases

In

Proc

of

the

Int

l

Conf

on

Data

Eng.

pages

Taipei

Taiwan

-LSB-

-RSB-

Vipin

Kumar

Ananth

Grama

Anshul

Gupta

and

George

Karypis

Introduction

to

Parallel

Computing

Algorithm

Design

and

Analysis

Benjamin

Cummings

Addison

Wesley

Redwod

City

-LSB-

-RSB-

H.

Mannila

H.

Toivonen

and

A.

I.

Verkamo

Discovering

frequent

episodes

in

sequences

In

Proc

of

the

First

Int

l

Conference

on

Knowledge

Discovery

and

Data

Mining

pages

Montreal

Quebec

-LSB-

-RSB-

IBM

Quest

Data

Mining

Project

Quest

synthetic

data

generation

code

-LSB-

-RSB-

C.

H.

Papadimitriou

and

K.

Steiglitz

Combinatorial

Optimization

Algorithms

and

Complexity

PrenticeHall

Englewood

Cliffs

NJ

-LSB-

-RSB-

R.

Srikant

and

R.

Agrawal

Mining

generalized

association

rules

In

Proc

of

the

VLDB

Conference

pages

Zurich

Switzerland

-LSB-

-RSB-

R.

Srikant

and

R.

Agrawal

Mining

sequential

patterns

Generalizations

and

performance

improvements

In

Proc

of

the

Fifth

Int

l

Conference

on

Extending

Database

Technology

Avignon

France

-LSB-

-RSB-

M.

Stonebraker

R.

Agrawal

U.

Dayal

E.

J.

Neuhold

and

A.

Reuter

DBMS

research

at

a

crossroads

The

vienna

update

In

Proc

of

the

VLDB

Conference

pages

Dublin

Ireland

-LSB-

-RSB-

A.

Savasere

E.

Omiecinski

and

S.

Navathe

An

efficient

algorithm

for

mining

association

rules

in

large

databases

In

Proc

of

the

VLDB

Conference

pages

Zurich

Switzerland

Contents

Introduction

Basic

Concepts

Parallel

Algorithms

Count

Distribution

Algorithm

Data

Distribution

Algorithm

Intelligent

Data

Distribution

Algorithm

Hybrid

Algorithm

Performance

Analysis

Experimental

Results

Conclusion

List

of

Figures

Apriori

Algorithm

Subset

operation

on

the

root

of

a

candidate

hash

tree

Subset

operation

on

the

left

most

subtree

of

the

root

of

a

candidate

hash

tree

Count

Distribution

-LRB-

CD

-RRB-

Algorithm

Data

Distribution

-LRB-

DD

-RRB-

Algorithm

Pseudo

Code

for

Data

Movements

Intelligent

Data

Distribution

-LRB-

IDD

-RRB-

Algorithm

Subset

operation

on

the

root

of

a

candidate

hash

tree

in

IDD

Hybrid

Distribution

-LRB-

HD

-RRB-

Algorithm

in

Processor

Mesh

-LRB-

G

D

P

D

-RRB-

Comparison

of

DD

and

IDD

in

terms

of

the

average

number

of

distinct

leaf

node

visited

per

transaction

with

transactions

per

processor

and

minimum

support

Scaleup

result

with

transactions

and

minimum

support

Sizeup

result

with

processors

and

minimum

support

Response

time

on

processors

with

transactions

as

the

minimum

support

varies

At

each

support

level

the

total

number

of

candidate

item-sets

is

shown

in

parenthesis

Response

time

on

processors

with

transactions

as

the

minimum

support

varies

At

each

support

level

the

total

number

of

candidate

item-sets

is

shown

in

parenthesis

Response

time

on

processor

IBM

with

transactions

as

the

minimum

support

varies

from

to

List

of

Tables

Transactions

from

supermarket

Symbols

used

in

the

analysis

Minimum

support

-LRB-

-RRB-

reachable

with

different

number

of

processors

in

our

algorithms

Processor

configuration

and

number

of

candidates

of

the

HD

algorithm

with

processors

and

minimum

support

for

each

pass

Note

that

configuration

is

the

same

as

the

DD

algorithm

and

is

the

same

as

the

CD

algorithm

The

total

number

of

pass

was

and

all

passes

after

had

configuration

association rules
data mining
load balance
scalability