Query

Driven

Data

Allocation

Algorithms

for

Distributed

Database

Kamalakar

Karlaplem

Ng

Moon

Pun

University

of

Science

and

Technology

Department

of

Computer

Science

Clear

Water

Bay

Kowloon

Hong

Kong

e-mail

kamal@cs.ust.hk

Phone

+852 2358 6991

Fax

+852 2358 1477

Abstract

A

major

cost

in

executing

queries

in

a

distributed

database

system

is

the

data

transfer

cost

incurred

in

transferring

multiple

database

objects

-LRB-

fragments

-RRB-

accessed

by

a

query

from

different

sites

to

the

site

where

the

query

is

initiated

The

objective

of

a

data

allocation

algorithm

is

to

locate

the

fragments

at

different

sites

so

as

to

minimize

the

total

data

transfer

cost

incurred

in

executing

a

set

of

queries

We

develop

a

site-independent

fragment

dependency

graph

representation

to

model

the

dependencies

among

the

fragments

accessed

by

a

query

and

use

it

to

formulate

and

solve

data

allocation

problems

for

distributed

database

systems

based

on

-LRB-

query-site

and

move-small

-RRB-

query

execution

strategies

We

show

that

an

optimal

solution

can

be

achieved

when

the

query-site

query

execution

strategy

is

employed

and

for

the

move-small

query

execution

strategy

we

performed

experimental

evaluation

about

the

effectiveness

of

a

hillclimbing

heuristic

algorithm

in

achieving

a

near-optimal

solution

research

has

been

funded

by

RGC

CERG

grant

HKUST

Introduction

A

distributed

database

is

a

collection

of

data

which

belongs

logically

to

the

same

system

but

is

physically

spread

over

the

sites

of

a

computer

network

A

typical

user

can

access

the

complete

database

from

any

site

a

distributed

database

system

processes

and

executes

an

user

's

query

by

accessing

the

data

from

multiple

sites

A

major

component

of

query

execution

cost

is

the

data

transfer

cost

That

is

the

total

cost

involved

in

moving

the

data

from

the

sites

where

it

is

located

to

the

site

where

the

query

is

issued

It

is

desirable

to

reduce

the

amount

of

data

transfer

This

can

be

achieved

by

optimally

allocating

the

data

-LRB-

i.e.

database

objects

or

fragments

-RRB-

to

the

sites

of

the

distributed

database

system

Optimal

allocation

of

fragments

is

a

complex

problem

because

of

mutual

interdependency

between

allocation

scheme

-LRB-

which

gives

the

location

of

each

of

the

fragments

at

various

sites

of

a

distributed

database

system

-RRB-

and

query

optimization

strategy

-LRB-

which

decides

how

a

query

can

be

optimally

executed

given

an

allocation

scheme

-RRB-

We

model

the

dependencies

between

the

fragments

and

the

amount

of

data

transfer

incurred

to

execute

a

query

based

on

a

query

execution

strategy

as

site

independent

fragment

dependency

graphs

We

use

these

fragment

dependency

graphs

to

aid

in

coming

up

with

an

optimal

or

near

optimal

allocation

of

fragments

We

call

this

approach

to

data

allocation

problem

as

query-driven

data

allocation

in

distributed

database

systems

Figure

shows

the

steps

in

query-driven

approach

developed

in

this

paper

for

the

data

allocation

problem

There

are

two

aspects

to

this

problem

The

first

aspect

is

to

represent

and

evaluate

the

set

of

queries

accessing

the

distributed

database

The

second

aspect

is

to

use

this

information

to

come

up

with

a

formulation

and

solution

for

the

data

allocation

problem

The

distributed

query

processor

is

used

to

process

a

given

set

of

queries

This

query

processing

-LSB-

-RSB-

consists

of

decomposing

the

queries

and

data

localization

Data

localization

involves

identifying

the

fragments

accessed

by

the

query

and

generating

the

fragment

query

operator

trees

These

fragment

query

operator

trees

are

further

processed

by

taking

into

consideration

a

given

distributed

query

execution

strategy

and

the

information

about

the

fragment

sizes

to

generate

fragment

dependency

graphs

The

fragment

dependency

graph

models

the

dependencies

between

the

fragments

accessed

by

a

query

and

the

amount

of

data

transfer

incurred

to

execute

a

query

A

query

driven

data

allocation

algorithm

takes

as

inputs

i

-RRB-

the

fragment

dependency

graphs

ii

-RRB-

unit

data

transfer

costs

between

sites

iii

-RRB-

the

allocation

limit

on

the

number

of

fragments

that

can

be

allocated

at

a

site

and

iv

-RRB-

the

query

execution

frequencies

from

the

sites

as

inputs

to

output

an

allocation

scheme

Fragment

Allocation

Algorithm

Query

Evaluation

Inputs

Fragmentation

Scheme

Queries

Fragment

Sizes

Query

Decomposer

Data

Localization

FDG

Generator

Queries

Query

Operator

Trees

on

Relations

Query

Operator

Trees

on

Fragments

Fragment

Dependency

Graphs

Query

Execution

Strategy

Minimum

Total

Data

Transfer

Cost

Allocation

Scheme

Allocation

Algorithm

Inputs

C

unit

data

transfer

cost

A

query

frequency

l

allocation

limit/site

Figure

Steps

in

Query

Driven

Data

Allocation

The

objective

of

this

work

is

to

design

algorithms

to

generate

minimum

total

data

transfer

cost

allocation

scheme

In

this

paper

we

shall

develop

and

test

algorithms

by

taking

into

consideration

two

popularly

used

query

execution

strategies

-LRB-

i.e.

querysite

and

move-small

-RRB-

We

shall

consider

only

non-redundant

data

allocation

-LRB-

i.e.

each

fragment

is

allocated

to

exactly

one

site

-RRB-

However

it

should

be

noted

that

straight

forward

techniques

are

available

for

replicating

the

fragments

in

distributed

database

system

-LSB-

-RSB-

once

an

non-redundant

allocation

scheme

is

known

The

rest

of

the

paper

is

organized

as

follows

section

describes

the

inputs

to

the

data

allocation

problem

section

describes

a

cost

model

used

to

calculate

the

total

data

transfer

cost

incurred

to

execute

a

set

of

queries

section

describes

two

algorithms

developed

in

this

paper

section

describes

experiments

conducted

to

evaluate

the

effectiveness

of

a

hill-climbing

heuristic

algorithm

section

describes

the

related

work

and

section

presents

some

conclusions

Inputs

for

Data

Allocation

Problem

Consider

a

distributed

database

system

with

m

sites

with

each

site

having

its

own

processing

power

memory

and

a

database

system

Let

S

be

the

name

of

site

where

m

The

m

sites

of

the

distributed

database

system

are

connected

by

a

communication

network

A

link

between

two

sites

Si

and

Sj

-LRB-

if

it

exists

-RRB-

has

a

positive

integer

Cij

associated

with

it

giving

the

cost

for

a

unit

data

transferred

from

site

Si

to

site

Sj

If

two

sites

are

not

directly

connected

by

a

communication

link

then

the

cost

for

unit

data

transferred

is

given

by

the

sum

of

the

cost

of

links

of

a

chosen

path

from

site

Si

to

site

Sj

Let

Q

qn

be

the

most

important

queries

accounting

for

more

than

of

the

processing

in

the

distributed

database

system

Each

query

q

can

be

executed

from

any

site

with

a

certain

frequency

Let

aff

fi

be

the

frequency

with

which

query

q

is

executed

from

site

S

These

frequencies

of

executions

of

queries

at

all

sites

can

be

represented

by

a

m

n

matrix

A.

Let

there

be

n

fragments

-LRB-

or

database

objects

or

relations

-RRB-

named

Fk

A

query

accesses

one

or

more

fragments

Query

Representation

The

data

allocation

problem

is

difficult

because

of

the

mutual

interdependency

between

the

query

execution

strategy

-LRB-

decided

by

the

query

optimizer

-RRB-

and

the

allocation

of

the

fragments

The

optimal

allocation

of

the

fragments

depends

on

a

given

query

execution

strategy

and

the

optimal

query

execution

strategy

depends

on

a

fixed

materialization

of

the

fragments

-LRB-

i.e.

fixing

the

location

of

the

fragments

accessed

by

the

query

-RRB-

The

main

problem

in

deciding

on

the

optimal

allocation

is

the

lack

of

a

representational

model

of

the

dependencies

among

the

fragments

accessed

by

the

query

These

dependencies

arise

because

of

the

partitioning

of

the

relations

into

fragments

and/or

access

to

multiple

relations

by

a

query

We

use

the

distributed

query

decomposer

and

data

localization

algorithms

-LSB-

-RSB-

to

decompose

a

distributed

query

into

a

set

of

fragment

queries

This

decomposed

query

incorporates

the

dependencies

between

fragments

These

dependencies

model

binary

operations

-LRB-

like

join

union

-RRB-

between

the

fragments

that

need

to

be

processed

in

order

to

execute

the

distributed

query

We

estimate

the

sizes

of

the

intermediate

relations

-LSB-

-RSB-

generated

after

executing

the

unary

and

binary

operations

by

making

use

of

the

database

statistics

-LRB-

like

cardinality

and

lengths

of

tuples

of

the

fragments/relations

-RRB-

available

from

the

system

catalog

Since

we

are

incorporating

distributed

query

processing

the

query

decomposition

and

data

localization

phases

eliminate

access

to

irrelevant

fragments

and

generate

a

concise

decomposed

query

operator

trees

on

fragments

In

the

distributed

query

optimization

and

execution

phase

optimal

binary

operation

orderings

are

based

on

a

query

execution

strategy

A

query

execution

strategy

can

be

-LSB-

Move-Small

-RSB-

If

a

binary

operation

involves

two

fragments

located

at

two

different

sites

then

ship

the

smaller

fragment

to

the

site

of

the

large

fragment

-LSB-

Query-Site

-RSB-

Ship

all

the

fragments

to

the

site

of

query

origin

and

execute

the

query

Note

that

the

objective

of

data

allocation

is

to

minimize

the

total

data

transfer

cost

to

process

all

the

queries

by

using

one

of

the

above

query

execution

strategies

The

first

aim

of

data

allocation

is

to

maximize

the

locality

of

the

fragments

for

executing

the

queries

The

second

aim

is

to

incorporate

the

query

execution

strategy

when

a

query

needs

to

access

fragments

from

multiple

sites

and

reduce

the

total

data

transfer

cost

to

process

all

the

queries

Query

Evaluation

Each

of

the

k

queries

on

the

distributed

database

are

restructured

and

decomposed

to

generate

an

query

operator

tree

Example

Consider

the

following

query

taken

form

-LSB-

-RSB-

accessing

relations

E

-LRB-

Eno

Ename

Title

-RRB-

G

-LRB-

Eno

Jno

Resp

Dur

-RRB-

and

J

-LRB-

Jno

Jname

Budget

-RRB-

SELECT

Ename

FROM

J

G

E

WHERE

G.Eno

E.Eno

AND

G.Jno

J.

Jno

AND

Ename

J.Doe

AND

J.Name

CAD

CAM

AND

-LRB-

Dur

OR

Dur

-RRB-

The

corresponding

relational

algebra

tree

after

query

decomposition

and

data

localization

is

shown

in

figure

P

P

P

Ename

Jno

Jno

Ename

Jno

J

Jname

CAD/CAM

s

P

Jno

Eno

P

Eno

Ename

Eno

Ename

J.Doe

s

E

s

Dur

OR

Dur

G

-LRB-

G

-RRB-

-LRB-

G

-RRB-

-LRB-

J'

-RRB-

-LRB-

E

-RRB-

-LRB-

J'

-RRB-

Figure

An

Example

of

Distributed

Query

Operator

Tree

The

intermediate

relations

generated

after

the

query

restructuring

phase

are

J

and

J

Assume

that

the

size

of

is

greater

than

the

size

of

and

the

size

of

J

is

greater

than

the

size

of

By

using

the

move-small

query

execution

strategy

it

is

preferable

to

transfer

to

the

site

where

is

located

and

also

to

transfer

to

the

site

where

J

is

located

in

order

to

minimize

the

total

size

of

data

required

to

be

transferred

to

execute

the

query

In

case

of

query-site

query

execution

strategy

the

intermediate

relations

J

and

are

all

shipped

to

the

site

where

the

query

is

initiated

Based

on

these

two

query

execution

strategies

the

corresponding

fragment

dependency

graphs

that

are

generated

are

illustrated

in

figure

In

a

fragment

dependency

graph

the

fragment-nodes

-LRB-

like

Site

-LRB-

J

-RRB-

Site

-LRB-

G

-RRB-

etc.

-RRB-

represent

the

potential

sites

where

the

fragments

are

located

The

query-node

Site

-LRB-

Q

-RRB-

represents

the

site

where

the

query

is

initiated

-LRB-

i.e.

query

site

-RRB-

There

is

a

cost

value

attached

to

each

edge

of

the

graph

corresponding

to

the

amount

of

data

that

may

be

transferred

if

the

fragments

corresponding

to

the

two

nodes

of

the

edge

are

located

at

different

sites

or

the

location

of

the

fragment

and

the

query

site

are

different

sites

The

fragment

dependency

graph

models

the

execution

strategy

used

by

the

query

optimizer

without

exactly

fixing

the

locations

of

the

fragments

This

site-independent

representation

of

fragment

dependencies

in

query

facilitate

the

development

of

the

query-driven

approach

to

data

allocation

For

example

referring

to

figure

relations

E

and

G

are

located

at

different

sites

then

it

will

incur

Size

-LRB-

-RRB-

data

transfer

cost

to

Site

-LRB-

E

-RRB-

Site

-LRB-

G

-RRB-

Site

-LRB-

J

-RRB-

Site

-LRB-

Q

-RRB-

Size

-LRB-

J'

-RRB-

Size

-LRB-

E

-RRB-

Size

-LRB-

G

-RRB-

Move-Small

Query

Execution

Strategy

Query-Site

Query

Execution

Strategy

Site

-LRB-

E

-RRB-

Size

-LRB-

E

-RRB-

Site

-LRB-

G

-RRB-

Size

-LRB-

G

-RRB-

Site

-LRB-

J

-RRB-

Size

-LRB-

J'

-RRB-

Site

-LRB-

Q

-RRB-

Figure

Fragment

Dependency

Graphs

using

different

Query

Execution

Strategies

process

the

join

between

relations

and

when

using

move-small

query

execution

strategy

Similarly

if

Site

-LRB-

Q

-RRB-

is

different

from

the

site

where

relation

E

is

located

then

it

will

incur

Size

-LRB-

-RRB-

data

transfer

cost

to

transfer

relation

to

query

site

in

the

query-site

query

execution

strategy

Note

that

these

data

transfer

costs

can

be

augmented

by

the

system

dependent

costs

like

join

processing

cost

This

join

processing

cost

is

based

on

a

join

algorithm

that

is

used

For

example

for

the

join

of

relations

and

if

a

nested

loop

join

algorithm

is

used

and

relation

E

is

the

inner

relation

then

cost

incurred

for

storing

relation

E

as

a

temporary

relation

must

be

included

For

the

sake

of

simplicity

we

do

not

explicitly

model

this

cost

in

the

cost

model

developed

in

the

next

section

but

we

can

always

augment

the

cost

model

with

such

system

dependent

costs

for

a

distributed

database

system

In

summary

the

inputs

to

the

data

allocation

problem

are

A

set

of

k

fragments

F

Fk

m

sites

of

the

distributed

database

system

S

Sm

and

a

matrix

C

-LSB-

ci

j

-RSB-

the

cost

of

transporting

a

unit

of

data

from

Si

to

Sj

A

set

of

n

queries

Q

qn

and

a

matrix

A

-LSB-

ai

j

-RSB-

the

frequency

of

qj

initiated

in

Si

The

fragment

dependency

graphs

for

each

of

the

queries

along

with

the

estimates

of

the

intermediate

relation

sizes

A

vector

l

-LSB-

lj

-RSB-

the

limit

on

maximum

number

of

fragments

that

can

be

allocated

at

site

Sj

This

models

the

storage

constraint

in

data

allocation

problem

Based

on

the

above

inputs

we

develop

a

cost

model

for

total

data

transfer

incurred

to

process

all

the

queries

and

then

develop

a

solution

to

the

non-redundant

data

allocation

problem

A

Model

for

Total

Data

Transfer

Cost

There

are

two

aspects

of

the

data

transfer

cost

incurred

to

process

a

query

that

need

to

be

modeled

The

first

aspect

is

the

unit

data

transfer

cost

from

one

site

to

another

This

is

modeled

as

minimum

cost

path

and

the

corresponding

path

from

one

site

to

another

We

use

an

all-pairs

shortest

path

algorithm

to

generate

the

cost

matrix

C

where

Ci

j

is

cost

of

transporting

an

unit

of

data

from

site

Si

to

site

Sj

Note

that

even

if

the

network

is

fully

connected

the

shortest

path

between

two

sites

may

have

more

than

one

edge

In

order

to

find

the

best

allocation

of

a

fragment

-LRB-

i.e.

the

allocation

which

minimizes

the

total

data

transfer

cost

-RRB-

it

is

enough

to

know

the

size

of

data

from

every

fragment

that

is

required

from

every

site

This

is

the

second

aspect

of

data

transfer

cost

that

needs

to

be

calculated

However

the

problem

is

that

as

represented

in

the

fragments

dependence

graphs

the

size

of

data

from

a

fragment

that

is

required

from

a

site

varies

with

the

locations

of

other

fragments

The

fragment

dependence

graph

of

every

query

models

two

types

of

data

transfer

cost

The

first

type

is

from

the

sites

where

the

fragments

are

located

to

the

site

where

the

query

is

initiated

The

second

type

is

from

the

site

where

one

fragment

is

located

to

the

site

where

another

fragment

is

located

In

the

first

type

the

size

of

data

of

a

fragment

required

by

every

site

does

not

vary

with

the

location

of

other

fragments

as

there

is

no

dependency

between

the

fragments

accessed

by

the

query

This

is

true

in

case

of

query-site

query

processing

strategy

and

the

top

level

of

the

fragment

dependency

graphs

For

e.g

in

case

of

move-small

query

processing

strategy

-LRB-

see

figure

-RRB-

the

edge

Site

-LRB-

J

-RRB-

Site

-LRB-

Q

-RRB-

models

this

type

of

data

transfer

cost

Let

r

fl

be

defined

as

size

of

data

of

F

needed

to

be

transported

to

the

site

where

q

is

initiated

The

corresponding

matrix

n

k

is

R.

Let

there

be

a

query

q

initiate

from

site

S

aff

fi

times

in

an

unit

time

interval

And

let

q

request

fragment

F

and

each

request

require

r

fl

amount

of

data

transfer

from

the

site

where

F

is

located

Let

U

be

a

m?k

matrix

where

uff

fl

gives

the

amount

of

data

needed

to

be

transferred

from

the

site

where

fragment

F

is

allocated

to

the

site

S

where

the

queries

are

initiated

That

is

uff

fl

k

X

aff

fi

r

fl

-LRB-

-RRB-

And

in

matrix

representation

U

A

R

The

second

type

of

data

transfer

cost

corresponds

to

the

deeper

levels

of

the

fragments

dependence

graph

The

data

is

transported

from

the

site

where

one

fragment

is

located

to

the

site

where

the

other

fragment

is

located

in

order

to

perform

binary

operation

involving

two

-LRB-

or

more

different

-RRB-

fragments

In

this

case

the

amount

of

data

of

a

fragment

required

by

a

site

varies

with

the

allocation

of

other

fragments

Let

d

define

the

size

of

data

from

fragment

F

that

needs

to

be

transported

to

the

site

where

F

is

located

so

as

to

execute

some

binary

operation

Let

the

corresponding

matrix

k

k

be

D

But

this

is

dependent

on

the

query

that

is

to

be

processed

Therefore

for

each

query

we

need

to

extract

the

information

about

how

much

data

needs

to

be

transferred

from

site

where

one

fragment

is

located

to

the

site

where

another

fragment

is

located

given

that

both

the

fragments

are

accessed

by

the

query

This

information

is

extracted

by

the

fragments

dependency

graph

generator

which

processes

the

query

operator

trees

on

fragments

by

applying

a

query

execution

strategy

and

is

represented

in

the

fragment

dependency

graph

Let

d

be

the

size

of

data

of

F

needed

to

be

transported

to

the

site

where

F

is

located

to

process

q

And

let

the

corresponding

matrix

be

D

Then

the

amount

of

data

that

needs

to

be

transported

from

the

site

where

F

is

located

to

the

site

where

F

is

given

by

d

k

X

-LRB-

n

X

aff

fi

-RRB-

d

fl

-LRB-

-RRB-

And

in

matrix

representation

D

k

X

-LRB-

n

X

aff

fi

-RRB-

D

Let

site

-LRB-

F

-RRB-

denote

the

site

where

fragment

F

is

located

Then

the

total

transportation

cost

t

is

given

by

t

k

X

k

X

csite

-LRB-

F

-RRB-

site

-LRB-

F

-RRB-

d

m

X

k

X

csite

-LRB-

F

-RRB-

ff

uff

fl

-LRB-

-RRB-

where

the

first

term

gives

the

data

transfer

cost

incurred

to

process

the

binary

operations

between

the

fragments

located

at

different

sites

and

the

second

term

gives

the

data

transfer

cost

incurred

to

transfer

the

results

of

the

binary

operations

of

fragments

to

the

site

where

the

query

is

initiated

The

objective

in

data

allocation

problem

is

to

minimize

t

by

altering

the

function

site

-LRB-

F

-RRB-

-LRB-

which

maps

a

fragment

to

a

site

-RRB-

Algorithms

for

Query

Driven

Data

Allocation

The

solution

for

allocation

problem

depends

on

the

query

execution

strategy

employed

by

the

distributed

database

system

to

execute

the

query

In

this

section

we

shall

develop

solutions

for

the

allocation

problem

when

query-site

and

move-small

query

execution

strategies

are

used

Optimal

Data

Allocation

for

Query-Site

Query

Execution

Strategy

In

this

case

all

the

fragments

are

transferred

to

the

query

site

and

the

query

is

executed

at

the

site

of

its

origin

Let

fragment

F

be

allocated

at

S

then

let

W

be

a

m?k

matrix

where

w

fi

represents

the

cost

of

data

transferred

from

F

to

site

S

-LRB-

i.e.

the

second

term

in

equation

-RRB-

That

is

w

fi

m

X

c

ff

uff

fi

-LRB-

-RRB-

And

in

matrix

representation

W

C

U

Let

x

fi

be

if

F

is

allocated

at

site

S

and

it

is

zero

otherwise

The

allocation

problem

is

formulated

as

assign

zero-one

values

to

xi

j

under

the

constraints

Pm

x

fi

for

all

k

Pk

x

fi

l

for

all

m

so

as

to

minimize

Pm

Pk

x

fi

w

fi

The

first

constraint

ensures

that

each

fragment

is

allocated

to

at

least

one

site

and

the

second

constraint

ensures

that

no

site

is

allocated

more

fragments

than

the

maximum

number

that

can

be

allocated

at

that

site

Based

on

the

above

formalism

the

data

allocation

problem

in

this

case

can

be

viewed

as

a

mapping

problem

Given

the

costs

of

data

transfer

incurred

-LRB-

i.e.

w

fi

-RRB-

when

a

fragment

F

is

allocated

at

site

S

the

problem

is

to

map

the

fragments

to

sites

so

as

to

minimize

the

total

data

transfer

cost

This

problem

can

be

mapped

to

a

maximum

flow

minimum

cost

problem

and

an

optimal

mapping

can

be

achieved

Maximum

Flow

Minimum

Cost

Formulation

The

first

step

in

translating

allocation

problem

to

a

maximum

flow

minimum

cut

problem

is

calculate

the

cost

of

allocating

a

fragment

to

a

particular

site

This

is

done

by

calculating

the

matrix

W

as

given

by

the

equation

In

order

to

formulate

the

data

allocation

problem

as

a

maximum

flow

minimum

cut

problem

-LRB-

see

figure

-RRB-

we

need

to

perform

following

steps

Two

nodes

a

source

and

a

sink

are

created

named

S

and

T

respectively

m

nodes

are

created

each

one

corresponding

to

a

site

named

Sm

k

nodes

are

created

each

one

corresponding

to

a

fragment

named

Fk

For

each

node

Si

a

edge

from

S

to

it

is

created

and

the

capacity

and

cost

of

the

link

are

assigned

as

li

and

respectively

For

each

node

Fi

a

edge

from

it

to

T

is

created

and

the

capacity

and

cost

of

the

link

are

assigned

to

and

respectively

For

every

pair

of

nodes

Si

and

Fj

a

edge

from

Si

to

Fj

is

created

and

the

capacity

and

cost

of

the

link

are

assigned

to

and

wi

j

respectively

S

T

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Capacity

Cost

Figure

Mapping

Query-Site

Data

Allocation

Problem

as

Max-flow

Min-cut

Problem

The

problem

is

to

find

the

maximum

flow

with

minimum

cost

from

S

to

T

It

can

be

observed

that

the

maximum

flow

in

this

case

is

k

-LRB-

i.e.

the

number

of

fragments

in

the

distributed

database

system

-RRB-

since

it

is

bounded

by

the

sum

of

capacity

of

incoming

edges

to

T

The

sum

of

capacity

of

out

coming

edges

from

S

must

be

bigger

than

k

-LRB-

the

number

of

fragments

-RRB-

for

all

fragments

to

be

allocated

This

will

be

true

if

P

m

lalpha

k

At

the

point

of

maximum

flow

there

must

be

exactly

one

incoming

edge

-LRB-

say

from

node

Sj

-RRB-

to

Fi

with

flow

equal

to

and

all

other

incoming

edges

-LRB-

from

nodes

Sp

-RRB-

have

flow

equal

to

zero

for

p

k

and

p

j

This

is

equivalent

to

allocating

fragment

Fi

to

site

Sj

Since

the

capacity

of

flow

from

S

to

site

Si

is

assigned

li

i

k

no

site

will

be

allocated

more

than

the

maximum

limit

on

the

number

of

fragments

allowed

The

problem

then

is

to

achieve

maximum

flow

from

node

S

to

node

T

with

minimum

cost

This

is

equivalent

to

assigning

each

fragment

to

a

site

while

minimizing

the

total

data

transfer

cost

Claim

The

maximum-flow

and

minimum

cost

formulation

of

data

allocation

problem

with

query-site

query

execution

strategy

generates

the

optimal

solution

Justification

The

data

allocation

problem

can

be

restated

as

a

mapping

problem

-LRB-

i.e.

mapping

a

fragment

to

a

site

-RRB-

When

the

query-site

query

execution

strategy

is

used

the

fragments

dependency

graph

of

queries

have

only

one

level

This

implies

that

the

data

transfer

cost

incurred

to

execute

all

the

queries

when

a

fragment

is

located

at

a

site

is

fixed

-LRB-

i.e.

independent

of

the

allocation

of

other

fragments

-RRB-

That

is

all

the

values

for

di

j

is

zero

in

the

equations

-LRB-

-RRB-

for

i

j

k

The

data

transfer

cost

for

the

fragment

Fj

to

be

allocated

on

site

Si

can

be

found

by

summing

up

Pm

p

Pk

q

-LRB-

ci

;p

ap

q

rq

j

-RRB-

which

is

exactly

the

values

elements

of

matrix

W

given

by

equation

These

are

the

cost

values

on

the

edge

from

site

Si

to

fragment

Fj

in

figure

Therefore

by

finding

the

maximum

flows

of

this

graph

a

feasible

allocation

is

found

And

by

finding

the

minimum-cost

maximum

flow

a

minimum

total

data

transfer

cost

allocation

is

found

which

implies

that

the

optimal

solution

is

achieved

The

value

of

the

minimum

total

data

transfer

cost

is

the

sum

of

the

costs

wij

on

the

edges

from

Si

to

Fj

which

form

the

maximum

flow

and

minimum

cost

solution

S

E

T

J

G

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Capacity

Cost

Figure

Max-flow

min-cut

formulation

for

the

Example

Allocation

Problem

Example

-LRB-

Cont

-RRB-

Consider

a

distributed

database

system

with

fully

connected

sites

and

and

three

relations

E

G

and

J

Let

there

be

only

one

query

as

shown

in

figure

Let

this

query

be

initiated

from

site

with

frequency

from

site

with

frequency

and

from

site

with

frequency

The

fragment

dependency

graph

is

illustrated

in

figure

Let

the

sizes

of

intermediate

fragments

be

size

-LRB-

E

-RRB-

size

-LRB-

J

-RRB-

size

-LRB-

-RRB-

Since

there

is

only

one

query

the

matrix

R

-LSB-

-RSB-

corresponding

to

relations

E

and

J

respectively

and

the

matrix

A

-LSB-

-RSB-

t

-LRB-

i.e.

a

column

vector

-RRB-

corresponding

to

sites

and

respectively

Let

the

limit

vector

l

constraining

the

number

of

fragments

that

can

be

allocated

to

a

site

be

l

-LSB-

-RSB-

Then

the

matrix

U

A

R

is

as

follows

-LRB-

the

rows

correspond

to

sites

and

respectively

and

the

columns

correspond

to

fragments

E

G

and

J

respectively

-RRB-

U

Let

the

cost

matrix

for

unit

data

transfer

cost

from

one

site

to

another

be

-LRB-

the

rows

and

columns

correspond

to

sites

and

respectively

-RRB-

C

Then

the

total

data

transfer

from

a

fragment

to

a

site

is

given

by

the

matrix

W

C

U

as

follows

-LRB-

the

rows

correspond

to

sites

and

respectively

and

the

columns

correspond

to

fragments

E

G

and

J

respectively

-RRB-

W

45 180 270

45 180 270

105 420 630

Figure

illustrates

the

max-flow

min-cut

formulation

for

this

example

After

applying

the

Ford

and

Fulkerson

method

-LSB-

-RSB-

to

solve

for

maximum

flow

with

minimum

cost

we

get

the

solution

as

allocate

E

to

site

allocate

G

to

site

and

J

to

site

The

total

data

transfer

cost

is

Near-optimal

Data

Allocation

for

Move-Small

Query

Execution

Strategy

The

general

data

allocation

problem

has

been

proved

to

be

NP

complete

-LSB-

-RSB-

Finding

the

optimal

solution

by

exhaustive

search

would

require

O

-LRB-

km

-RRB-

in

the

worst

case

where

k

is

the

number

of

fragments

and

m

is

the

number

of

sites

Therefore

a

heuristic

algorithm

based

on

high

climbing

technique

is

developed

in

order

to

find

a

near

optimal

solution

The

general

data

allocation

problem

solution

consists

of

following

two

steps

As

a

first

step

neglect

the

dependencies

among

the

fragments

and

come

up

with

a

initial

data

allocation

by

using

the

max-flow

min-cut

formulation

of

the

problem

-LRB-

as

described

in

section

-RRB-

Iteratively

improve

the

initial

data

allocation

by

using

the

hill

climbing

heuristic

until

no

further

reduction

in

total

data

transfer

cost

can

be

achieved

This

is

done

by

applying

some

operations

on

the

initial

allocation

scheme

Hill-Climbing

Algorithm

for

Data

Allocation

The

initial

solution

generated

by

the

max-flow

min-cut

formulation

is

refined

by

applying

some

operations

on

it

The

objective

for

applying

these

operations

is

to

reduce

the

total

data

transfer

cost

There

are

two

types

of

operations

that

are

defined

namely

migrate

migrate

fragments

from

its

currently

allocated

site

to

newly

allocated

sites

and

swap

swap

the

locations

of

one

set

of

fragments

with

the

locations

of

new

set

of

fragments

These

operations

take

into

consideration

the

dependency

between

the

fragments

as

modeled

by

fragment

dependency

graphs

These

operations

are

iteratively

applied

till

no

more

reduction

is

observed

in

the

total

data

transfer

cost

Figure

shows

the

major

steps

in

the

hill-climbing

heuristic

algorithm

that

is

developed

in

this

section

The

set

of

migrate

and

swap

operations

are

as

follows

ffl

migrate

-LRB-

Fi

Sj

-RRB-

move

fragment

Fi

to

Sj

This

operation

can

be

applied

to

each

fragment

and

the

fragment

can

be

moved

to

any

one

of

m

sites

-LRB-

it

is

not

located

at

-RRB-

Therefore

there

can

be

maximum

of

k

-LRB-

m

-RRB-

migrate

operations

that

can

be

applied

during

each

iteration

The

migrate

operation

can

potentially

reduce

the

data

transfer

cost

involved

in

processing

the

binary

operation

over

two

fragments

If

one

fragment

is

located

at

one

site

and

another

fragment

is

located

at

a

different

site

then

migrating

one

fragment

to

the

site

of

another

can

potentially

reduce

data

transfer

cost

ffl

-LRB-

Fi

Sj

Fp

Sq

-RRB-

move

fragment

Fi

to

site

Sj

and

fragment

Fp

to

site

Sq.

This

operation

can

be

applied

to

each

pair

of

fragments

and

they

can

each

be

moved

to

any

one

of

m

sites

where

they

are

not

located

Therefore

there

can

be

maximum

of

-LRB-

k

-LRB-

m

-RRB-

-RRB-

operations

that

can

be

applied

during

each

iteration

As

is

case

of

migrate

operation

the

operation

can

potentially

reduce

data

transfer

costs

in

processing

binary

operations

involving

three

fragments

ffl

-LRB-

Fi

Sj

Fp

Sq

Fr

St

-RRB-

move

fragment

Fi

to

site

Sj

fragment

Fp

to

site

Sq

and

fragment

Fr

to

site

St.

This

operation

can

be

applied

to

each

triplet

Max-flow

Min-cut

Algorithm

Apply

Migrate

Operations

Apply

Swap

Operations

Any

Reduction

in

Total

Data

Transfer

Cost

Any

Reduction

in

Total

Data

Transfer

Cost

Initial

Allocation

Incrementally

Improved

Allocation

No

Yes

Yes

No

Final

Allocation

Incrementally

Improved

Allocation

Figure

Steps

in

Hill-Climbing

Algorithm

for

Data

Allocation

of

fragments

and

they

can

each

be

moved

to

any

one

of

m

sites

where

they

are

not

located

Therefore

there

can

be

a

maximum

of

-LRB-

k

-LRB-

m

-RRB-

-RRB-

operations

that

can

be

applied

during

each

iteration

The

operations

can

potentially

reduce

data

transfer

costs

in

processing

binary

operations

over

four

fragments

ffl

swap

-LRB-

Fi

Fj

-RRB-

swap

the

location

of

fragment

Fi

with

location

of

fragment

Fj

This

operation

can

be

applied

to

each

distinct

pair

of

fragments

Therefore

there

can

be

a

maximum

of

k

-LRB-

k

-RRB-

swap

operations

that

can

be

applied

during

each

iteration

Since

there

is

a

limit

on

number

of

fragments

that

can

be

allocated

at

a

site

by

swapping

fragments

we

maintain

this

constraint

and

simultaneously

explore

if

we

can

improve

the

allocation

Note

that

a

migrate

operation

can

not

be

applied

if

the

limit

on

the

number

of

fragments

that

can

be

located

at

a

site

is

exceeded

ffl

-LRB-

Fi

Fj

Fp

Fq

-RRB-

swap

the

location

of

fragment

Fi

with

location

of

fragment

Fj

and

location

of

Fp

with

location

of

Fq

This

operation

can

be

applied

to

each

of

the

distinct

pairs

of

fragments

fFi

Fjg

and

fFp

Fqg

The

operation

is

also

useful

in

reducing

the

total

data

transfer

cost

while

maintain-g

the

constraint

on

number

of

fragments

that

can

be

allocated

to

a

site

ffl

-LRB-

Fi

Fj

Fp

Fq

Fr

Ft

-RRB-

swap

the

location

of

fragment

Fi

with

location

of

fragment

Fj

location

of

Fp

with

location

of

Fq

and

location

of

Ft

with

location

of

Ft.

This

operation

can

be

applied

to

each

of

the

distinct

pairs

of

fragments

fFi

Fjg

fFp

Fqg

and

fFr

Ftg

The

operation

is

also

useful

in

reducing

the

total

data

transfer

cost

while

maintaing

the

constraint

on

number

of

fragments

that

can

be

allocated

to

a

site

In

the

experiments

that

we

have

conducted

we

noticed

that

migrate

and

swap

operations

are

applied

most

often

followed

by

and

operations

and

the

least

frequent

occurrences

were

that

of

and

operations

In

our

extensive

experimental

studies

we

found

that

many

a

times

just

applying

migrate

and

swap

operations

would

not

give

the

global

optimal

solution

but

would

give

a

global

solution

when

and/or

operations

or

and/or

operations

were

applied

This

was

the

reason

for

us

to

define

three

different

types

of

migrate

and

swap

operations

Example

-LRB-

Cont

-RRB-

We

shall

use

the

example

presented

in

section

with

the

distributed

database

system

using

the

move-small

query

execution

strategy

The

fragment

dependency

graph

is

as

illustrated

in

figure

Let

the

fragment

sizes

are

size

-LRB-

E

-RRB-

size

-LRB-

J

-RRB-

and

size

-LRB-

-RRB-

First

we

shall

use

the

top-level

of

the

fragment

dependency

graph

i.e

the

edge

from

Site

-LRB-

J

-RRB-

Site

-LRB-

Q

-RRB-

to

solve

for

initial

allocation

scheme

by

using

maximum-flow

minimum

cut

formulation

As

there

is

only

one

query

we

have

the

matrix

R

-LSB-

-RSB-

with

each

element

corresponding

to

relations

E

and

J

respectively

and

the

matrix

A

-LSB-

-RSB-

t

-LRB-

i.e.

a

column

vector

-RRB-

with

each

element

corresponding

to

sites

and

respectively

Let

the

cost

matrix

for

unit

data

transfer

cost

from

one

site

to

another

be

-LRB-

the

rows

and

columns

correspond

to

sites

and

respectively

-RRB-

C

After

formulating

the

problem

as

maximum-flow

and

minimum

cut

problem

and

solving

it

we

get

the

initial

solution

as

allocate

fragment

E

at

site

allocate

fragment

G

at

site

and

allocate

fragment

J

at

site

This

allocation

schema

is

represented

as

corresponding

to

the

sites

where

the

relations

E

G

and

J

are

to

be

located

respectively

The

matrix

giving

the

amount

of

data

needed

from

a

fragment

to

be

transported

to

site

of

another

fragment

derived

from

the

fragment

dependency

graph

-LRB-

shown

in

figure

-RRB-

is

The

row

and

columns

of

matrix

correspond

to

fragments

E

and

J

respectively

By

applying

equation

we

get

the

total

amount

of

data

that

must

be

transported

from

the

site

where

one

fragment

is

located

to

the

site

where

another

fragment

is

located

as

-LRB-

the

rows

and

columns

correspond

to

fragments

E

and

J

respectively

-RRB-

D

-LRB-

-RRB-

The

total

data

transfer

cost

given

by

equation

for

the

initial

allocation

is

-LRB-

-RRB-

-LRB-

-RRB-

The

cost

value

in

first

parenthesis

corresponds

to

the

data

transfer

cost

incurred

in

transferring

E

to

site

where

G

is

located

and

transferring

to

the

site

where

J

is

located

The

second

cost

value

corresponds

to

the

data

transfer

incurred

in

transferring

the

result

J

to

the

query

site

We

now

apply

the

hill-climbing

heuristic

algorithm

to

improve

upon

the

initial

solution

so

as

to

further

reduce

the

total

data

transfer

cost

if

possible

The

table

illustrates

the

working

of

hill-climbing

algorithm

Table

shows

the

migrate

and

swap

operations

applied

to

improve

the

initial

solution

provided

by

the

max-flow

min-cut

algorithm

Two

operations

were

applied

i

-RRB-

migrate

-LRB-

J

-RRB-

which

reduced

the

total

data

transfer

cost

from

to

and

ii

-RRB-

migrate

-LRB-

E

G

J

-RRB-

which

reduced

the

data

transfer

cost

from

to

The

solution

to

data

allocation

problem

generated

by

the

hill-climbing

algorithm

is

-LRB-

i.e.

allocate

relation

E

at

site

relation

G

at

site

and

relation

J

at

site

-RRB-

with

the

total

data

transfer

cost

incurred

to

execute

the

query

as

Table

Table

Output

of

the

Hill-Climbing

Algorithm

for

Data

Allocation

Current

Operation

Operation

New

Total

Data

Allocation

Type

Applied

Allocation

Transfer

Cost

migrate

migrate

-LRB-

J

-RRB-

migrate

applied

applied

migrate

-LRB-

E

G

J

-RRB-

migrate

Applied

Applied

Applied

swap

Applied

Applied

Applied

shows

all

feasible

allocation

schemes

and

the

total

data

transfer

cost

incurred

for

each

of

them

The

table

also

shows

the

itemized

costs

for

the

data

transfer

The

cost

involved

in

transferring

E

to

site

where

relation

relation

G

is

located

is

given

by

the

column

Size

-LRB-

E

-RRB-

Site

-LRB-

G

-RRB-

the

column

Size

-LRB-

-RRB-

Site

-LRB-

J

-RRB-

gives

the

cost

of

transferring

to

the

site

where

relation

J

is

located

the

column

Size

-LRB-

J

-RRB-

S

-LRB-

Q

-RRB-

gives

the

cost

of

transferring

the

J

to

the

query

site

Following

points

can

be

noted

after

comparing

hill-climbing

algorithm

with

the

results

of

exhaustive

solution

The

heuristic

algorithm

generates

the

optimal

solution

-LRB-

which

is

allocation

number

-RRB-

in

table

The

heuristic

algorithm

would

not

have

got

the

optimal

solution

if

only

migrate

operation

had

been

used

Note

that

only

after

applying

operation

optimal

solution

was

achieved

The

heuristic

algorithm

may

not

always

generate

an

optimal

solution

Experimental

Evaluation

of

the

Hill

Climbing

Algorithm

The

example

that

was

considered

in

the

previous

section

was

for

illustrating

how

the

hill-climbing

algorithm

works

But

it

had

only

one

query

three

fragments

and

three

Table

Enumeration

of

Feasible

Allocation

Schemes

and

their

Data

Transfer

Costs

Allocation

Allocation

Data

Transfer

Cost

-LRB-

Itemized

-RRB-

Total

Data

Number

Scheme

Size

-LRB-

-RRB-

Site

-LRB-

G

-RRB-

Size

-LRB-

-RRB-

Site

-LRB-

J

-RRB-

Size

-LRB-

J

-RRB-

S

-LRB-

Q

-RRB-

Transfer

Cost

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

sites

The

number

of

feasible

allocations

were

with

data

transfer

cost

ranging

from

to

Therefore

it

is

imperative

that

some

experiments

be

conducted

to

evaluate

how

the

hill-climbing

algorithm

performs

in

general

The

allocation

scheme

generated

by

the

hill-climbing

algorithm

needs

to

be

compared

with

the

exhaustive

solution

Since

the

exhaustive

solution

takes

large

amount

of

time

to

experiment

for

a

distributed

database

system

with

large

number

of

sites

and

fragments

-LRB-

for

k

fragments

and

m

sites

there

are

km

allocation

schemes

and

for

each

allocation

scheme

the

data

transfer

cost

-LRB-

given

by

equation

-RRB-

needs

to

be

calculated

-RRB-

We

conducted

sixteen

experiments

with

number

of

fragments

ranging

from

to

and

number

of

sites

ranging

from

to

Each

experiment

consisted

of

allocation

problems

with

number

of

sites

and

number

of

fragments

fixed

Each

allocation

problem

had

between

and

queries

and

each

query

had

a

fragment

dependency

graph

The

communication

network

the

relation

sizes

the

link

costs

and

the

structure

of

fragment

dependency

graph

were

randomly

generated

Each

allocation

problem

was

solved

by

using

hill-climbing

algorithm

and

its

solution

was

compared

with

the

optimal

solution

generated

by

exhaustive

search

Table

lists

the

for

each

of

the

experiments

conducted

column

wise

i

-RRB-

the

number

of

fragments

ii

-RRB-

number

of

sites

iii

-RRB-

number

of

problems

iv

-RRB-

number

of

problems

for

Table

Experimental

Evaluation

of

Near

Optimality

of

Hill-Climbing

Algorithm

Number

of

Number

of

Number

of

Number

of

Average

Number

of

Solutions

with

Deviation

in

Range

Fragments

Sites

Problems

Optimal

Solutions

Deviation

-LRB-

-RRB-

-LRB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

which

heuristic

algorithm

generated

the

optimal

solution

v

-RRB-

the

average

deviation

in

percentage

of

near

optimal

solutions

from

optimal

solution

-LRB-

for

those

allocation

problems

for

which

the

hill

climbing

algorithm

did

not

generate

optimal

solution

-RRB-

vi

-RRB-

number

of

near

optimal

solutions

with

deviation

less

than

vii

-RRB-

number

of

near

optimal

solutions

with

deviation

of

or

more

but

less

than

viii

-RRB-

number

of

near

optimal

solutions

with

deviation

of

or

more

but

less

than

and

ix

-RRB-

number

of

near

optimal

solutions

with

deviation

more

than

Based

on

this

table

following

points

can

be

noted

The

hill-climbing

algorithm

generated

optimal

solution

for

large

number

of

problems

-LRB-

on

an

average

for

greater

than

of

problems

-RRB-

The

average

deviation

of

a

near

optimal

solution

from

the

optimal

solution

was

on

an

average

less

than

in

total

data

transfer

cost

For

a

small

percentage

of

cases

about

the

deviation

of

near

optimal

solution

from

optimal

solution

was

greater

than

This

is

to

be

expected

of

a

hillclimbing

algorithm

in

that

it

wanders-off

and

finds

a

local

optimal

solution

We

conducted

further

experiments

by

considering

different

network

topologies

-LRB-

like

ring

star

fully

connected

tree

-RRB-

and

different

query

access

patterns

-LRB-

like

fewer

access

frequencies/site

-LRB-

unit

time

interval

-RRB-

to

large

access

frequencies

-LRB-

like

time

interval

-RRB-

and

the

results

were

similar

to

those

shown

in

table

We

have

not

included

these

results

for

the

lack

of

space

The

experimental

results

do

imply

that

the

hill-climbing

algorithm

can

be

used

in

practice

to

allocate

fragments

for

distributed

database

systems

Related

Work

The

data

allocation

problem

has

been

first

studied

in

terms

of

file

allocation

problem

in

a

multi

computer

system

and

later

on

as

a

data

allocation

problem

in

distributed

database

system

The

file

allocation

problem

does

not

take

into

consideration

the

semantics

of

the

processing

being

done

of

files

whereas

data

allocation

problem

must

take

into

consideration

the

interdependencies

among

accesses

to

multiple

fragments

by

a

query

Chu

-LSB-

-RSB-

was

the

first

to

study

the

problem

of

file

allocation

with

respect

to

multiple

files

on

a

multiprocessor

system

He

presented

a

global

optimization

model

to

minimize

overall

processing

costs

under

the

constraints

of

response

time

and

storage

capacity

with

a

fixed

number

of

copies

of

each

file

Casey

-LSB-

-RSB-

distinguished

between

updates

and

queries

on

files

Eswaran

-LSB-

-RSB-

proved

that

Casey

's

formulation

is

NP

complete

He

suggested

that

a

heuristic

rather

than

exhaustive

search

approach

is

more

suitable

Ramamoorthy

and

Wah

-LSB-

-RSB-

analyzed

a

file

allocation

problem

in

the

environment

of

a

distributed

database

Wah

developed

a

heuristic

approximation

algorithm

for

a

simple

file

allocation

problem

and

the

generalized

file

allocation

problem

He

had

also

proposed

a

model

for

file

migration

or

reallocation

that

is

identical

in

formulation

to

the

file

allocation

problem

Ceri

Martella

and

Pellagatti

-LSB-

-RSB-

consider

the

problem

of

file

allocation

for

typical

distributed

database

applications

with

a

simple

model

transaction

execution

with

considering

the

dependencies

between

accesses

to

multiple

fragments

Apers

-LSB-

-RSB-

considers

the

allocation

of

the

distributed

database

to

the

sites

so

as

to

minimize

total

data

transfer

cost

The

author

came

up

with

a

very

complicated

approach

to

allocate

relations

by

first

partitioning

them

into

innumerable

number

of

fragments

and

then

allocating

them

The

author

did

integrate

the

system

dependent

query

processing

strategy

with

the

logical

model

for

allocating

the

fragments

But

the

author

solves

the

complete

fragmentation

and

allocation

problem

That

is

the

fragmentation

schema

is

one

of

the

outputs

of

the

allocation

algorithm

This

curtails

the

applicability

of

this

methodology

when

fragmentation

schema

is

already

defined

and

allocation

scheme

must

be

generated

Cornell

and

Yu

-LSB-

Corn

-RSB-

propose

a

strategy

to

integrate

the

treatment

of

relation

assignment

and

query

strategy

to

optimize

performance

of

a

distributed

database

system

Though

they

do

take

into

consideration

the

query

execution

strategy

the

solution

they

came

up

with

is

a

complicated

linear

programming

solution

The

main

problem

in

their

approach

is

the

lack

of

simplicity

in

both

incorporation

of

the

query

execution

strategy

and

the

solution

procedure

There

have

been

many

linear

programming

formulations

proposed

for

data

allocation

problem

-LSB-

and

others

-RSB-

The

main

problem

with

these

approaches

is

the

lack

of

modeling

of

the

query

execution

strategy

Lin

Orlawska

and

Zhang

-LSB-

-RSB-

also

develop

a

heuristic

algorithm

for

minimum

overall

data

transfer

cost

by

considering

replicated

allocation

of

fragments

and

both

read

and

update

transactions

We

do

not

consider

replicated

allocation

of

fragments

and

leave

it

as

a

separate

problem

because

there

are

efficient

solutions

-LSB-

-RSB-

that

can

be

applied

to

replicate

fragments

once

an

initial

non-redundant

allocation

scheme

is

generated

Conclusion

The

problem

of

non-redundant

data

allocation

of

fragments

in

distributed

database

system

is

addressed

in

this

paper

A

query

driven

data

allocation

approach

is

developed

by

integrating

the

query

execution

strategy

with

the

formulation

of

the

data

allocation

problem

We

studied

the

data

allocation

problem

for

two

popularly

used

query

execution

strategies

namely

query-site

and

move-small

We

mapped

the

data

allocation

problem

with

query-site

execution

strategy

as

a

max-flow

min-cut

problem

and

showed

that

it

can

be

solved

to

generate

an

optimal

solution

A

hill-climbing

heuristic

algorithm

was

developed

for

the

data

allocation

problem

with

move-small

query

execution

strategy

We

studied

the

effectiveness

of

this

algorithm

by

conducting

a

set

of

experiments

which

show

that

the

hill-climbing

heuristic

algorithm

generates

the

optimal

solution

for

a

large

percentage

of

allocation

problems

Moreover

the

near

optimal

solution

generated

by

the

algorithm

has

on

an

average

small

percentage

of

deviation

from

the

optimal

solution

in

terms

of

the

total

data

transfer

costs

incurred

to

process

all

the

queries

Therefore

the

query

driven

data

allocation

algorithms

can

be

used

in

practice

allocating

fragments

in

distributed

database

systems

A

major

contribution

of

this

work

is

the

development

of

a

site-independent

fragment

dependency

graph

representation

of

data

transfer

costs

incurred

in

executing

a

query

by

using

a

particular

query

execution

strategy

This

enables

the

applica

tion

of

query

driven

allocation

approach

to

allocate

database

objects

under

different

database

systems

-LRB-

like

relational

object-oriented

network

etc.

-RRB-

and

facilitates

the

development

of

an

uniform

framework

within

which

to

define

and

solve

data

allocation

problems

A

complicated

query

execution

strategy

employed

by

a

distributed

database

system

will

generate

complex

fragment

dependency

graphs

which

will

involve

development

of

novel

techniques

to

solve

for

optimal

allocation

We

have

implemented

the

algorithms

presented

in

this

paper

as

part

of

a

distributed

relational

database

design

tool

We

are

currently

applying

this

framework

to

place

complex

objects

in

object-oriented

databases

and

as

a

first

step

developed

partitioning

schemes

for

object-oriented

databases

-LSB-

-RSB-

An

advantage

of

the

algorithms

developed

for

query-site

and

move-small

query

execution

strategies

is

that

they

can

execute

in

polynomial

time

Therefore

these

algorithms

can

be

applied

to

generate

allocation

schemes

for

limited

redesign

of

distributed

databases

-LSB-

-RSB-

Limited

redesign

implies

change

in

allocation

scheme

but

not

in

fragmentation

scheme

for

a

distributed

database

The

algorithms

developed

in

this

paper

can

be

integrated

with

algorithms

to

affect

the

limited

redesign

of

distributed

databases

-LSB-

-RSB-

thus

facilitating

on-line

tuning

of

a

distributed

relational

database

system

References

-LSB-

-RSB-

P.

M.

G.

Apers

Data

allocation

in

distributed

database

systems

ACM

Transactions

on

Database

Systems

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

R.

G.

Casey

Allocation

of

copies

of

a

file

in

an

information

network

In

Proceedings

of

Spring

Joint

Computer

Conference

IFIPS

pages

-LCB-

-LSB-

-RSB-

S.

Ceri

G.

Martella

and

G.

Pelagatti

Optimal

file

allocation

for

a

distributed

on

a

network

of

minicomputers

In

Proceedings

of

International

Conference

on

Databases

Aberdeen

pages

-LCB-

July

-LSB-

-RSB-

S.

Ceri

and

G.

Pelagatti

Distributed

Databases

Principles

and

Systems

McGraw

Hill

New

York

-LSB-

-RSB-

W.

W.

Chu

Optimal

file

allocation

in

a

multiple

computer

system

IEEE

Transactions

on

Computers

-LRB-

-RRB-

-LSB-

-RSB-

K.

P.

Eswaran

Placement

of

records

in

a

file

and

file

allocation

in

a

computer

network

Information

Processing

pages

-LCB-

-LSB-

-RSB-

B.

Gavish

and

H.

Pirkul

Computer

and

database

location

in

distributed

computer

systems

IEEE

Transactions

on

Computers

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

K.

Karlapalem

and

Q.

Li

Partitioning

schemes

for

object

oriented

databases

In

Proceedings

of

Fifth

Int

Workshop

on

Research

Issues

on

Data

Engineering

Distributed

Object

Management

RIDE-DOM

-LSB-

-RSB-

X.-M

Lin

M.E.

Orlowska

and

Y.-C

Zhang

Database

placement

in

communication

networks

for

minimizing

the

overall

transmission

cost

Mathematical

and

Computer

Modelling

-LRB-

-RRB-

-LCB-

Jan

-LSB-

-RSB-

T.

Ozsu

and

P.

Valduriez

Principles

of

Distributed

Database

Systems

PrinticeHall

Inc.

-LSB-

-RSB-

S.

Ram

and

R.E.

Marsten

A

model

for

database

allocation

incorporating

a

concurrency

control

mechanism

IEEE

Transactions

on

Knowledge

and

Data

Engineering

-LRB-

-RRB-

-LCB-

Sept

-LSB-

-RSB-

C.

V.

Ramamoorthy

and

B.

Wah

The

placement

of

relations

on

a

distributed

relational

database

In

Proceedings

of

Int

Conf

Distributed

Computing

Systems

Huntsville

Alabama

September

Oct

pp

-LSB-

-RSB-

P.

I.

Rivera-Vega

R.

Varadarajan

and

S.

B.

Navathe

Scheduling

data

redistribution

in

distributed

databases

In

Proceedings

of

International

Conference

on

Data

Engineering

IEEE

February

-LSB-

-RSB-

Robet

Endre

Tarjan

Data

Structures

and

Network

Algorithms

volume

CBMS

NSF

SIAM

-LSB-

-RSB-

B.

Wilson

and

S.

B.

Navathe

An

analytical

framework

for

the

redesign

of

distributed

databases

In

Proceedings

of

the

Adavanced

Database

Symposium

Tokyo

Japan.

data allocation
distributed database systems
optimal allocation
query processing

distributed database design
hill-climbing heuristics
max-flow mincut problem
network flow algorithms