Design

and

Evaluation

of

Data

Allocation

Algorithms

for

Distributed

Multimedia

Database

Systems

Yu-Kwong

Kwok

Kamalakar

Karlapalem

Ishfaq

Ahmad

and

Ng

Moon

Pun

Department

of

Computer

Science

The

Hong

Kong

University

of

Science

and

Technology

Clear

Water

Bay

Kowloon

Hong

Kong

Abstract

A

major

cost

in

retrieving

multimedia

data

from

multiple

sites

is

the

cost

incurred

in

transferring

multimedia

data

objects

-LRB-

MDOs

-RRB-

from

different

sites

to

the

site

where

the

query

is

initiated

The

objective

of

a

data

allocation

algorithm

is

to

locate

the

MDOs

at

different

sites

so

as

to

minimize

the

total

data

transfer

cost

incurred

in

executing

a

given

set

of

queries

There

is

a

mutual

dependency

between

data

allocation

and

query

execution

strategies

in

that

the

optimal

allocation

of

MDOs

depends

on

the

query

execution

strategy

employed

by

a

distributed

multimedia

system

while

the

query

execution

strategy

optimizes

a

query

based

on

this

allocation

In

this

paper

we

fix

the

query

execution

strategy

and

develop

a

siteindependent

MDO

dependency

graph

representation

to

model

the

dependencies

among

the

MDOs

accessed

by

a

query

Given

the

MDO

dependency

graphs

as

well

as

the

set

of

multimedia

database

sites

data

transfer

costs

between

the

sites

the

allocation

limit

on

the

number

of

MDOs

that

can

be

allocated

at

a

site

and

the

query

execution

frequencies

from

the

sites

an

allocation

scheme

is

generated

We

formulate

the

data

allocation

problem

as

an

optimization

problem

We

solve

this

problem

with

a

number

of

techniques

that

broadly

belong

to

three

classes

max-flow

min-cut

state-space

search

and

graph

partitioning

heuristics

The

max-flow

min-cut

technique

formulates

the

data

allocation

problem

as

a

network-flow

problem

and

uses

a

hill-climbing

approach

to

try

to

find

the

optimal

solution

For

the

state-space

search

approach

the

problem

is

solved

using

a

best-first

search

algorithm

The

graph

partitioning

approach

uses

two

clustering

heuristics

the

agglomerative

clustering

and

divisive

clustering

We

evaluate

and

compare

these

approaches

and

assess

their

costperformance

trade-offs

All

algorithms

are

also

compared

with

optimal

solutions

obtained

through

exhaustive

search

Conclusions

are

also

made

on

the

suitability

of

these

approaches

to

different

scenarios

Introduction

A

distributed

multimedia

database

system

-LSB-

-RSB-

-LSB-

-RSB-

is

a

database

system

loosely

coupled

with

a

multimedia

data

provider

as

shown

in

Figure

-LRB-

unlike

the

integrated

and

heterogeneous

computing

paradigm

based

distributed

multimedia

database

systems

-LSB-

-RSB-

-RRB-

Special-purpose

distributed

multimedia

database

system

architectures

-LSB-

-RSB-

-LSB-

-RSB-

have

been

proposed

The

aim

of

these

architectures

is

to

support

specific

application

domains

-LRB-

like

Medical

Databases

and

News-on-Demand

-RRB-

and

thus

they

are

tightly

integrated

to

favor

these

applications

We

consider

a

general-purpose

loosely

coupled

architecture

which

can

be

built

by

using

off-the-shelf

products

In

this

architecture

Multimedia

Data

Provider

-LRB-

MDP

-RRB-

enables

users

to

retrieve

multimedia

data

objects

-LRB-

MDOs

-RRB-

from

different

sites

A

Common

Multimedia

User

Interface

-LRB-

CMUI

-RRB-

enables

the

users

to

specify

queries

accessing

the

distributed

multimedia

database

system

and

presenting

the

result

to

the

user

The

synchronization

for

the

presentation

of

the

multimedia

data

is

handled

by

the

CMUI

-LSB-

-RSB-

Whereas

the

Multimedia

Data

Provider

-LRB-

MDP

-RRB-

identifies

the

relevant

multimedia

data

for

an

user

query

and

facilitates

shipping

of

the

multimedia

data

to

the

CMUI

The

CMUI

is

a

client

process

and

the

Distributed

Database

Management

System

-LRB-

DDBMS

-RRB-

and

the

MDP

are

server

processes

There

can

be

multiple

CMUI

processes

and

multiple

DDBMS

and

MDP

processes

An

user

can

use

any

one

of

the

CMUI

client

processes

to

query

the

distributed

multimedia

database

system

The

query

is

decomposed

into

two

parts

by

CMUI

one

accessing

the

multiple

DDBMS

servers

and

the

other

accessing

multiple

MDP

servers

After

which

the

DDBMS

and

MDP

servers

ship

the

result

data

to

the

CMUI

Finally

the

result

is

presented

by

the

CMUI

to

the

user

A

major

component

of

multimedia

query

execution

cost

is

the

data

transfer

cost

That

is

the

total

cost

involved

in

moving

the

MDOs

from

the

sites

where

they

are

located

to

the

site

where

the

query

is

issued

The

MDOs

are

made

of

two

kinds

of

data

The

first

is

the

singlemedia

data

that

is

managed

by

the

DDBMS

servers

such

as

relations

-LRB-

fragments

-RRB-

records

Common

Multimedia

Distributed

Database

Management

System

-LRB-

DDBMS

-RRB-

Multimedia

Data

Provider

-LRB-

MDP

-RRB-

Audio

Image

Video

USER

QUERIES

Database

Figure

Loosely

coupled

architecture

of

Distributed

Multimedia

Database

System

NETWORK

User

Interface

-LRB-

CMUI

-RRB-

Common

Multimedia

USER

QUERIES

User

Interface

-LRB-

CMUI

-RRB-

Common

Multimedia

USER

QUERIES

User

Interface

-LRB-

CMUI

-RRB-

etc.

The

second

is

the

multimedia

data

such

as

audio

video

and

image

managed

by

the

MDP

servers

These

two

types

of

data

are

managed

by

different

specialized

storage

managers

and

need

to

be

retrieved

for

the

user

queries

As

the

allocation

problem

arises

for

both

single-media

data

and

multimedia

data

we

develop

a

common

data

allocation

problem

formulation

and

a

solution

methodology

Since

the

CMUI

is

responsible

for

synchronization

before

presenting

the

data

to

the

user

the

major

issue

is

to

reduce

the

data

transfer

cost

in

accessing

MDOs

from

different

sites

This

can

be

done

by

optimally

allocating

the

MDOs

to

the

sites

of

the

distributed

multimedia

database

system

Optimal

allocation

of

MDOs

is

a

complex

problem

because

of

mutual

interdependency

between

allocation

scheme

-LRB-

which

gives

the

location

of

each

of

the

MDOs

at

various

sites

of

a

distributed

database

system

-RRB-

and

query

optimization

strategy

-LRB-

which

decides

how

a

query

can

be

optimally

executed

given

an

allocation

scheme

-RRB-

Given

a

query

execution

strategy

we

model

the

dependencies

between

the

MDOs

and

the

amount

of

data

transfer

incurred

to

execute

a

query

as

site

independent

MDO

dependency

graph

-LRB-

MDG

-RRB-

We

use

the

MDO

dependency

graph

to

solve

for

an

optimal

or

near

optimal

allocation

of

MDOs

The

processing

strategy

of

distributed

multimedia

objects

retrieval

involves

shipping

of

all

the

multimedia

objects

to

the

user?s

query

site

because

this

strategy

supports

efficient

access

for

synchronization

during

the

presentation

of

the

result

by

the

CMUI

Figure

shows

the

steps

in

data

allocation

strategy

developed

in

this

paper

Essentially

there

are

two

aspects

to

this

problem

The

first

aspect

is

to

represent

and

evaluate

the

set

of

queries

accessing

the

distributed

multimedia

database

The

second

aspect

is

to

use

this

information

to

come

up

with

a

formulation

and

solution

for

the

data

allocation

problem

The

query

processing

-LSB-

-RSB-

consists

of

decomposing

the

queries

and

data

localization

Data

localization

involves

identifying

the

MDOs

accessed

by

the

query

and

generating

the

MDO

query

operator

trees

These

MDO

query

operator

trees

are

further

processed

by

taking

into

consideration

a

given

distributed

query

execution

strategy

and

the

information

about

the

MDO

sizes

to

generate

MDO

dependency

graphs

The

MDO

dependency

graph

models

the

dependencies

between

the

MDOs

accessed

by

a

query

and

the

amount

of

data

transfer

Query

Evaluation

Input

Fragmentation

Schemes

Queries

and

MDO

Sizes

Query

Decomposer

and

Data

Localization

MDG

Generator

MDO

Allocation

Algorithm

Query

Execution

Strategy

Allocation

Algorithm

Input

C

unit

data

transfer

cost

A

query

frequency

l

allocation

limit/site

k

number

of

MDOs

m

number

of

sites

Queries

MDO

Query

Operator

Trees

MDO

Dependency

Graphs

Minimum

Total

Data

Transfer

Cost

Allocation

Scheme

Figure

Steps

in

MDO

allocation

incurred

to

execute

a

query

A

data

allocation

algorithm

takes

as

inputs

i

-RRB-

the

MDO

dependency

graphs

ii

-RRB-

unit

data

transfer

costs

between

sites

iii

-RRB-

the

allocation

limit

on

the

number

of

MDOs

that

can

be

allocated

at

a

site

iv

-RRB-

the

query

execution

frequencies

from

the

sites

v

-RRB-

the

number

of

MDOs

and

vi

-RRB-

the

number

of

sites

and

outputs

an

allocation

scheme

The

objective

of

this

work

is

to

design

efficient

algorithms

to

generate

minimum

total

data

transfer

cost

allocation

scheme

The

rest

of

the

paper

is

organized

as

follows

Section

further

elaborates

the

data

allocation

problem

Section

describes

a

cost

model

used

to

calculate

the

total

data

transfer

cost

incurred

to

execute

a

set

of

queries

Section

includes

the

algorithms

proposed

in

this

paper

The

experimental

results

for

these

algorithms

are

provided

in

Section

The

effectiveness

and

suitability

of

the

proposed

algorithms

to

different

scenarios

is

discussed

in

Section

The

related

work

is

presented

in

Section

and

Section

concludes

this

paper

The

Data

Allocation

Problem

In

this

section

we

describe

in

detail

the

inputs

to

the

data

allocation

problem

addressed

in

this

paper

These

inputs

characterize

the

underlying

distributed

multimedia

database

system

and

help

in

formulating

the

problem

We

also

introduce

a

number

of

notations

throughout

the

paper

which

are

summarized

in

Table

Consider

a

distributed

multimedia

database

system

with

m

sites

with

each

site

having

its

own

processing

power

memory

and

a

database

system

Let

be

the

name

of

site

i

where

The

m

sites

of

the

distributed

multimedia

database

system

are

connected

by

a

communication

network

A

link

between

two

sites

and

-LRB-

if

it

exists

-RRB-

has

a

positive

integer

associated

with

it

giving

the

cost

for

a

unit

data

transferred

from

site

to

site

If

two

sites

are

not

directly

connected

by

a

communication

link

then

the

cost

for

unit

data

transferred

is

given

by

the

sum

of

the

cost

of

links

of

a

chosen

path

from

site

to

site

Let

be

the

most

important

queries

accounting

for

say

more

than

of

Table

Symbols

and

their

meanings

Symbol

Meaning

The

jth

MDO

The

ith

site

The

set

of

queries

The

xth

query

m

The

number

of

sites

in

the

network

k

The

number

of

MDOs

in

the

distributed

database

system

n

The

number

of

queries

The

access

frequencies

matrix

The

access

frequency

of

the

xth

query

at

site

i

C

The

unit

transportation

cost

matrix

of

the

network

The

unit

transportation

cost

from

site

i

to

site

i

l

The

allocation

limit

vector

of

the

sites

The

allocation

limit

of

site

i

The

query

data

transfer

size

matrix

The

query

data

transfer

size

of

the

xth

query

of

MDO

j

U

The

site

data

transfer

size

matrix

The

site

data

transfer

size

of

MDO

j

to

site

j

D

The

MDO

dependency

matrix

The

size

of

the

data

from

MDO

i

to

the

site

where

MDO

j

is

located

t

The

total

data

transfer

cost

O

jS

iQ

qx

A

a

i

x

cii

liR

rxj

uj

j

di

j

Si

i

m

S

i

Si

ci

i

S

i

Si

Si

Si

Q

q

qn

-LCB-

-RCB-

the

processing

in

the

distributed

multimedia

database

system

Each

query

can

be

executed

from

any

site

with

a

certain

frequency

Let

be

the

frequency

with

which

query

is

executed

from

site

The

executions

frequencies

of

n

queries

at

m

sites

can

be

represented

by

a

matrix

A.

Let

there

be

k

MDOs

-LRB-

or

database

objects

or

relations

-RRB-

named

Query

Representation

Any

query

accessing

both

the

single-media

database

fragments

and

multimedia

objects

can

be

split

into

two

queries

one

which

accesses

only

single-media

fragments

and

one

which

accesses

only

multimedia

data

The

basis

for

this

split

is

provided

by

the

architecture

of

the

loosely

coupled

distributed

multimedia

database

system

The

queries

that

access

single

database

fragments

will

be

optimized

by

distributed

database

system

and

the

queries

that

access

multimedia

data

will

be

processed

by

the

multimedia

data

provider

at

the

site

As

mentioned

above

the

data

allocation

problem

is

complex

because

of

the

mutual

interdependency

between

the

query

execution

strategy

-LRB-

decided

by

the

query

optimizer

-RRB-

and

the

allocation

of

the

fragments

The

optimal

allocation

of

the

fragments

depends

on

a

given

query

execution

strategy

while

the

optimal

query

execution

strategy

depends

on

a

fixed

materialization

of

the

fragments

-LRB-

i.e.

fixing

the

location

of

the

fragments

accessed

by

the

query

-RRB-

The

main

problem

in

deciding

on

the

optimal

allocation

is

the

lack

of

a

representational

model

of

the

dependencies

among

the

data

fragments

accessed

by

the

query

These

dependencies

arise

because

of

the

partitioning

of

the

relations

into

data

fragments

-LRB-

by

using

methodologies

proposed

by

-LSB-

-RSB-

-RRB-

and/or

access

to

multiple

relations

by

a

query

We

use

the

distributed

query

decomposer

and

data

localization

algorithms

-LSB-

-RSB-

to

decompose

a

distributed

query

into

a

set

of

queries

This

decomposed

query

incorporates

the

dependencies

between

data

fragments

These

dependencies

model

binary

operations

-LRB-

like

join

union

-RRB-

between

the

data

fragments

that

need

to

be

processed

in

order

to

execute

the

distributed

query

We

estimate

the

sizes

of

the

intermediate

relations

-LSB-

-RSB-

generated

after

executing

the

unary

and

binary

operations

by

using

the

database

statistics

-LRB-

like

cardinality

and

lengths

of

tuples

of

the

fragments/relations

-RRB-

available

from

the

system

catalog

In

the

distributed

database

query

processing

the

optimal

binary

operation

ordering

is

based

on

a

fixed

query

execution

strategy

By

fixing

the

query

execution

strategy

we

eliminate

the

dependency

between

the

distributed

query

execution

strategy

and

data

allocation

A

query

execution

strategy

can

be

-RRB-

Move-Small

If

a

binary

operation

involves

two

data

fragments

located

at

two

different

sites

then

ship

the

smaller

data

fragment

to

the

site

of

the

larger

data

fragment

-RRB-

Query-Site

Ship

all

the

data

fragments

to

the

site

of

query

origin

and

execute

the

query

Usually

the

move-small

query

execution

strategy

is

used

by

the

distributed

database

system

because

it

gives

rise

to

lower

data

transfer

costs

for

query

execution

than

the

query-site

query

execution

strategy

As

mentioned

before

the

MDP

uses

the

query-site

query

execution

strategy

because

of

the

synchronization

requirements

imposed

on

accessing

the

multimedia

objects

and

the

need

to

present

each

multimedia

object

as

an

individual

data

stream

Thus

the

first

aim

of

data

allocation

is

to

maximize

the

locality

of

the

MDOs

-LRB-

consisting

of

singlemedia

or

multi

media

-RRB-

for

executing

the

queries

The

second

aim

is

to

use

the

query

execution

strategy

to

reduce

the

total

data

transfer

cost

when

a

query

needs

to

access

MDOs

from

q

x

ai

x

qx

S

i

m

n

O

k

-LCB-

-RCB-

different

sites

Query

Evaluation

Each

of

the

n

queries

on

the

distributed

multimedia

database

are

restructured

and

decomposed

to

generate

query

operator

trees

Example

Consider

the

following

query

adapted

from

-LSB-

-RSB-

accessing

relations

E

-LRB-

Eno

Ename

Title

-RRB-

G

-LRB-

Eno

Jno

Resp

Dur

-RRB-

and

J

-LRB-

Jno

Jname

Budget

-RRB-

and

multimedia

objects

representing

the

pictures

EPicture

-LRB-

Eno

Picture

-RRB-

and

the

speech

ESpeech

-LRB-

Eno

Speech

-RRB-

as

shown

in

Figure

-LRB-

a

-RRB-

and

Figure

-LRB-

b

-RRB-

respectively

For

enhancing

the

readability

we

use

nested

SQL

statement

to

exemplify

MDO

retrieval

specification

From

the

data

allocation

point

of

view

we

need

only

consider

the

shipping

of

the

retrieved

Picture

and

Speech

from

EPicture

and

ESpeech

respectively

given

the

Eno

values

The

relational

algebra

tree

for

the

query

in

Figure

-LRB-

a

-RRB-

after

query

decomposition

and

data

localization

is

shown

in

Figure

-LRB-

a

-RRB-

The

intermediate

relations

generated

after

the

query

restructuring

phase

are

J

G

E

G

and

J

Assume

that

the

size

of

G

is

greater

than

the

size

of

E

and

the

size

of

J

is

greater

than

the

size

of

G

By

using

the

move-small

query

execution

strategy

it

is

preferable

to

transfer

E

to

the

site

where

G

is

located

and

also

to

transfer

G

to

the

site

where

J

is

located

in

order

to

minimize

the

total

size

of

data

required

to

be

transferred

to

execute

the

query

The

move-small

query

execution

strategy

and

the

corresponding

fragment

dependency

graph

that

is

generated

is

illustrated

in

Figure

-LRB-

b

-RRB-

In

a

fragment

dependency

graph

the

fragment-nodes

-LRB-

like

Site

-LRB-

J

-RRB-

Site

-LRB-

G

-RRB-

etc.

-RRB-

represent

the

potential

sites

where

the

fragments

are

located

The

query-node

Site

-LRB-

Q

-RRB-

represents

the

site

where

the

query

is

initiated

-LRB-

i.e.

the

query

site

which

is

fixed

for

each

execution

of

the

query

-RRB-

There

is

a

cost

value

attached

to

each

edge

of

the

graph

corresponding

to

the

amount

of

data

that

may

be

transferred

if

the

fragments

corresponding

to

the

two

nodes

of

the

edge

are

located

at

different

sites

or

if

the

location

of

the

fragment

and

the

query

site

are

different

sites

The

fragment

dependency

graph

models

the

execution

strategy

used

by

the

query

optimizer

without

exactly

fixing

the

locations

of

the

fragments

This

enables

us

to

calculate

the

amount

of

data

transfer

incurred

to

process

a

query

under

different

data

allocation

schemes

For

example

referring

to

Figure

-LRB-

b

-RRB-

if

relations

E

and

G

are

located

at

different

-LRB-

a

-RRB-

SELECT

Ename

FROM

J

G

E

WHERE

G.Eno

E.Eno

AND

G.Jno

J.

Jno

AND

Ename

J.Doe

AND

J.Name

CAD

CAM

AND

-LRB-

Dur

OR

Dur

-RRB-

-LRB-

b

-RRB-

SELECT

Picture

Speech

FROM

EPicture

ESpeech

WHERE

Eno

in

-LRB-

SELECT

Eno

FROM

J

G

E

WHERE

G.Eno

E.Eno

AND

G.Jno

J.

Jno

AND

Ename

J.Doe

AND

J.Name

CAD

CAM

AND

-LRB-

Dur

OR

Dur

-RRB-

-RRB-

Figure

-LRB-

a

-RRB-

SQL

statement

on

distributed

relational

database

system

-LRB-

b

-RRB-

Query

statement

to

retrieve

MDOs

from

EPicture

and

ESpeech

sites

then

it

will

incur

Size

-LRB-

E

-RRB-

data

transfer

cost

to

process

the

join

between

relations

E

and

G

when

using

move-small

query

execution

strategy

Similarly

by

applying

the

fragment

dependency

graph

concept

to

multimedia

data

retrieval

we

generate

the

MDO

data

dependency

graph

accessing

the

speech

and

picture

from

ESpeech

and

EPicture

respectively

by

using

the

query-site

query

execution

strategy

-LRB-

see

Figure

-LRB-

c

-RRB-

-RRB-

The

Data

Transfer

Cost

Model

The

cost

model

developed

in

this

section

is

applicable

for

calculating

the

total

data

transfer

cost

incurred

to

process

both

single-media

distributed

database

query

and

multimedia

data

retrieval

query

We

will

not

make

any

further

distinction

between

single

media

database

fragments

and

multimedia

data

for

the

rest

of

the

paper

and

will

consider

the

problem

in

the

context

of

MDOs

the

retrieval

of

MDOs

will

be

modeled

by

the

MDO

dependency

graphs

that

can

arise

from

either

or

both

move-small

or

query

site

query

execution

strategy

There

are

two

aspects

of

the

data

transfer

cost

incurred

to

process

a

query

that

need

to

be

modelled

The

first

aspect

is

the

unit

data

transfer

cost

from

one

site

to

another

This

is

modeled

as

minimum

cost

path

and

the

corresponding

path

from

one

site

to

another

We

use

an

all-pairs

shortest

path

algorithm

to

generate

the

cost

matrix

C

where

is

cost

of

Figure

-LRB-

a

-RRB-

An

example

of

fragment

query

operator

tree

-LRB-

b

-RRB-

the

move-small

fragment

dependency

graph

-LRB-

c

-RRB-

query-site

MDO

data

dependency

graph

PEnam

e

P

Jn

o

En

ame

P

Jn

o

PJn

o

Eno

PEn

o

Ename

s

J

n

a

m

e

CAD/CAM

sDur

OR

Dur

sEn

ame

J.Doe

Jno

Eno

J

G

E

-LRB-

J

-RRB-

-LRB-

G

-RRB-

-LRB-

E

-RRB-

-LRB-

G

-RRB-

-LRB-

J

-RRB-

Site

-LRB-

Q

-RRB-

Site

-LRB-

G

-RRB-

Site

-LRB-

E

-RRB-

Site

-LRB-

J

-RRB-

Size

-LRB-

J

-RRB-

Size

-LRB-

E

-RRB-

Size

-LRB-

G

-RRB-

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Site

-LRB-

Q

-RRB-

Site

-LRB-

EPicture

-RRB-

Site

-LRB-

ESpeech

-RRB-

Size

-LRB-

Speech

-RRB-

Size

-LRB-

Picture

-RRB-

-LRB-

c

-RRB-

cij

transporting

a

single

unit

of

data

from

site

to

site

Note

that

even

if

the

network

is

fully

connected

the

shortest

path

between

two

sites

may

have

more

than

one

edge

In

order

to

find

the

best

allocation

of

a

set

of

MDOs

-LRB-

i.e.

the

allocation

which

minimizes

the

total

data

transfer

cost

-RRB-

it

is

enough

to

know

the

size

of

data

from

every

MDO

that

is

required

from

every

site

This

is

the

second

aspect

of

data

transfer

cost

that

needs

to

be

calculated

However

the

problem

is

that

as

represented

in

the

MDOs

dependency

graphs

the

size

of

data

from

a

MDO

that

is

required

from

a

site

varies

with

the

locations

of

other

MDOs

The

MDO

dependency

graph

of

every

query

models

two

types

of

data

transfer

cost

The

first

type

of

cost

is

due

to

moving

the

data

from

the

sites

where

the

MDOs

are

located

to

the

site

where

the

query

is

initiated

The

second

type

of

cost

is

due

to

moving

the

data

from

the

site

where

one

MDO

is

located

to

the

site

where

another

MDO

is

located

In

this

case

the

size

of

data

of

a

MDO

required

by

every

site

does

not

vary

with

the

location

of

other

MDOs

as

there

is

no

dependency

between

the

MDOs

accessed

by

the

query

This

is

true

in

the

case

of

query-site

query

processing

strategy

and

the

top

level

of

the

MDO

dependency

graphs

In

the

case

of

move-small

query

processing

strategy

-LRB-

for

example

in

Figure

-LRB-

b

-RRB-

-RRB-

the

edge

models

this

type

of

data

transfer

cost

Let

be

defined

as

the

size

of

data

of

MDO

needed

to

be

transported

to

the

site

where

is

initiated

The

corresponding

matrix

is

R

of

size

Note

that

this

incorporates

the

top

level

of

the

MDOs

dependency

graph

Let

there

be

a

query

initiated

from

site

times

in

an

unit

time

interval

And

let

request

MDO

and

each

request

require

amount

of

data

transfer

from

the

site

where

is

located

Let

U

be

a

matrix

where

gives

the

amount

of

data

needed

to

be

transferred

from

the

site

where

MDO

is

allocated

to

the

site

where

the

queries

are

initiated

That

is

And

in

matrix

representation

The

second

type

of

data

transfer

cost

corresponds

to

the

deeper

levels

of

the

MDOs

dependency

graph

The

data

is

transported

from

the

site

where

one

MDO

is

located

to

the

site

where

the

other

MDO

is

located

in

order

to

perform

binary

operation

involving

two

-LRB-

or

more

-RRB-

different

MDOs

In

this

case

the

amount

of

data

of

a

MDO

required

by

a

site

varies

with

the

allocation

of

other

MDOs

Let

define

the

size

of

data

from

MDO

that

needs

to

be

transported

to

the

site

where

is

located

so

as

to

execute

some

binary

operation

Let

the

corresponding

matrix

be

D

But

this

is

dependent

on

the

query

that

is

to

be

processed

Therefore

for

each

query

we

need

to

extract

the

information

about

how

much

data

needs

to

be

transferred

from

site

where

one

MDO

is

located

to

the

site

where

another

MDO

is

located

given

that

both

the

MDOs

are

accessed

by

the

query

This

information

is

extracted

by

the

MDOs

dependency

graph

generator

which

processes

the

query

operator

trees

on

MDOs

by

applying

a

query

execution

strategy

and

is

represented

in

the

MDO

dependency

graph

Let

be

the

data

size

of

needed

to

be

transported

to

the

site

where

is

located

to

process

And

let

the

corresponding

matrix

be

Then

the

amount

of

data

that

needs

to

be

Si

Sj

S

it

e

J

-LRB-

-RRB-

S

i

t

e

Q

-LRB-

-RRB-

fi

rxj

Oj

q

x

n

k

q

x

S

i

ai

x

qx

O

j

rxj

Oj

m

k

ui

j

O

j

Si

ui

j

a

ix

r

x

j

x

n

U

A

R

dj

j

Oj

Oj

k

k

dj

j?x

O

j

Oj

qx

x

transported

from

the

site

where

is

located

to

the

site

where

is

given

by

And

in

matrix

representation

Let

denote

the

site

where

MDO

is

located

Then

the

total

transportation

cost

T

is

given

by

where

the

first

term

gives

the

data

transfer

cost

incurred

to

process

the

binary

operations

between

the

MDOs

located

at

different

sites

and

the

second

term

gives

the

data

transfer

cost

incurred

to

transfer

the

results

of

the

binary

operations

of

MDOs

to

the

site

where

the

query

is

initiated

The

objective

in

data

allocation

problem

is

to

minimize

T

by

altering

the

function

-LRB-

which

maps

a

MDO

to

a

site

-RRB-

Proposed

Data

Allocation

Algorithms

Developing

an

efficient

solution

to

the

data

allocation

problem

highly

depends

on

the

query

execution

strategy

employed

by

the

distributed

database

system

This

is

because

different

query

execution

strategies

have

different

MDO

migration

patterns

The

data

allocation

algorithm

is

NP-complete

in

general

-LSB-

-RSB-

In

this

section

we

develop

solutions

for

the

allocation

problem

when

query-site

and

move-small

query

execution

strategies

are

respectively

used

by

the

distributed

database

management

system

and

the

multimedia

data

provider

As

will

be

described

below

optimal

solution

can

be

obtained

in

polynomial

time

if

the

query

execution

strategy

is

constrained

to

be

query-site

only

In

this

section

we

present

three

approaches

to

tackle

the

data

allocation

problem

using

four

different

algorithms

In

the

first

approach

we

employ

a

hill-climbing

algorithm

to

generate

a

near-optimal

solution

In

the

second

approach

we

formulate

the

problem

as

a

state-space

search

problem

and

solve

it

by

using

a

best-first

search

algorithm

The

third

approach

is

a

graph

partitioning

approach

and

is

based

on

two

clustering

techniques.These

algorithms

are

explained

below

Optimal

Data

Allocation

Max-flow

Min-Cut

Approach

We

first

describe

an

algorithm

that

generates

optimal

solutions

for

query-site

query

execution

strategy

We

use

Floyd-Warshall

s

algorithm

-LSB-

-RSB-

to

find

the

minimal

cost

communication

path

between

any

two

sites

This

gives

us

the

values

in

the

cost

matrix

C

The

running

time

of

this

algorithm

is

where

m

is

the

number

of

sites

in

the

distributed

database

system

In

this

case

all

the

MDOs

are

transferred

to

the

query

site

and

the

query

is

executed

at

the

site

of

its

origin

This

is

the

same

as

neglecting

the

dependency

among

MDOs

and

O

j

O

j

d

jj

ai

x

i

m

d

jj

x

x

n

D

a

ix

i

m

x

x

n

s

i

t

e

O

j

-LRB-

-RRB-

Oj

T

D

U

c

s

ite

Oj

-LRB-

-RRB-

si

t

e

Oj

-LRB-

-RRB-

d

jj

uij

c

i

si

t

e

Oj

-LRB-

-RRB-

j

k

i

m

j

k

j

k

s

it

e

Oj

-LRB-

-RRB-

c

ij

O

-LRB-

-RRB-

allocating

the

MDOs

when

move-small

query

execution

strategy

is

used

Let

MDO

be

allocated

at

then

let

W

be

a

matrix

where

represents

the

cost

of

data

transferred

from

to

site

-LRB-

i.e.

the

second

term

in

expression

for

the

total

transportation

cost

-RRB-

That

is

And

in

matrix

representation

Let

be

if

is

allocated

at

site

and

zero

otherwise

The

allocation

problem

is

formulated

as

assigning

zero-one

values

to

under

the

constraints

-RRB-

-RRB-

so

as

to

minimize

The

first

constraint

ensures

that

each

MDO

is

allocated

to

at

least

one

site

and

the

second

constraint

ensures

that

no

site

is

allocated

more

MDOs

than

the

maximum

number

that

can

be

allocated

at

that

site

Based

on

the

above

formalism

the

data

allocation

problem

in

this

case

can

be

viewed

as

a

mapping

problem

Given

the

costs

of

data

transfer

incurred

-LRB-

i.e.

-RRB-

when

a

MDO

is

allocated

at

site

the

problem

is

to

map

the

MDOs

to

sites

so

as

to

minimize

the

total

data

transfer

cost

Since

a

single

site

will

not

have

enough

storage

space

for

all

the

MDOs

there

is

a

limit

on

number

of

MDOs

that

can

be

allocated

at

a

site

This

problem

is

equivalent

to

a

maximum-flow

minimum-cost

problem

and

an

optimal

mapping

can

be

achieved

The

outline

of

the

solution

is

formulated

as

follows

Calculate

W

as

shown

previously

Then

transform

the

problem

into

a

maximum-flow

minimum-cost

problem

After

the

costs

of

allocating

MDOs

at

each

site

have

been

calculated

-LRB-

W

is

found

-RRB-

the

problem

of

allocating

MDOs

to

sites

in

order

to

get

minimum

transportation

cost

can

be

viewed

as

a

maximumflow

minimum-cost

problem

The

first

step

in

translating

allocation

problem

to

a

maximumflow

minimum-cost

problem

is

to

calculate

the

cost

of

allocating

a

MDO

to

a

particular

site

This

is

done

by

calculating

the

matrix

W

In

order

to

formulate

the

data

allocation

problem

as

a

maximum-flow

minimum-cut

problem

-LRB-

see

Figure

-RRB-

we

need

to

perform

following

steps

-RRB-

Two

nodes

a

source

and

a

sink

are

created

and

named

S

and

T

respectively

-RRB-

m

nodes

are

created

one

each

corresponding

to

a

site

named

-RRB-

k

nodes

are

created

one

each

corresponding

to

a

MDO

named

-RRB-

For

each

node

where

an

edge

from

S

to

it

is

created

and

the

capacity

and

cost

of

the

link

are

assigned

as

and

respectively

-RRB-

For

each

node

where

an

edge

from

it

to

T

is

created

and

the

capacity

and

cost

of

the

link

are

assigned

to

and

respectively

-RRB-

For

every

pair

of

nodes

and

an

edge

from

to

is

created

and

the

capacity

Oj

Si

m

k

w

ij

Oj

Si

wi

j

ci

i

u

i?j

i

m

W

C

U

Xi

j

Oj

S

i

Xi

j

Xij

i

m

j

k

Xi

j

j

k

l

i

i

m

Xi

j

-LRB-

-RRB-

wi

j

j

k

i

m

wi

j

Oj

Si

S

Sm

-LCB-

-RCB-

Ok

-LCB-

-RCB-

Si

i

m

l

i

Oj

j

k

S

i

Oj

Si

O

j

and

cost

of

the

link

are

assigned

as

and

respectively

The

problem

is

to

find

the

maximum

flow

with

minimum

cost

from

S

to

T

It

can

be

observed

that

the

maximum

flow

in

this

case

is

k

-LRB-

i.e.

the

number

of

MDOs

in

the

distributed

database

system

-RRB-

since

it

is

bounded

by

the

sum

of

capacity

of

incoming

edges

to

T

The

sum

of

capacity

of

outgoing

edges

from

S

must

be

bigger

than

k

-LRB-

the

number

of

MDOs

-RRB-

for

all

MDOs

to

be

allocated

This

will

be

true

if

At

the

point

of

maximum

flow

there

must

be

exactly

one

incoming

edge

-LRB-

say

from

node

-RRB-

to

with

flow

equal

to

and

all

other

incoming

edges

from

nodes

have

flow

equal

to

zero

for

and

This

is

equivalent

to

allocating

MDO

to

site

Since

the

capacity

of

flow

from

S

to

site

is

assigned

no

site

will

be

allocated

more

than

the

maximum

limit

on

the

number

of

MDOs

allowed

The

problem

then

is

to

achieve

maximum

flow

from

node

S

to

node

T

with

the

minimum

cost

This

is

equivalent

to

assigning

each

MDO

to

a

site

while

minimizing

the

total

data

transfer

cost

Lemma

The

maximum-flow

and

minimum

cost

formulation

of

data

allocation

problem

with

query-site

query

execution

strategy

generates

the

optimal

solution

Proof

The

data

allocation

problem

can

be

restated

as

a

mapping

problem

-LRB-

i.e.

mapping

a

MDO

to

a

site

-RRB-

When

the

query-site

query

execution

strategy

is

used

the

MDOs

dependency

graph

of

queries

has

only

one

level

This

implies

that

the

data

transfer

cost

incurred

to

execute

all

the

queries

when

a

MDO

is

located

at

a

site

is

fixed

-LRB-

i.e.

independent

of

the

allocation

of

other

MDOs

-RRB-

That

is

all

the

values

for

are

zero

for

The

data

transfer

cost

for

the

MDO

to

be

allocated

on

site

can

be

found

by

summing

up

which

is

exactly

the

values

of

the

elements

of

matrix

W

These

are

the

cost

values

on

the

edge

from

site

to

MDO

in

Figure

Therefore

by

finding

the

maximum

flows

of

this

graph

a

feasible

allocation

is

found

And

by

finding

the

minimumcost

maximum-flow

a

minimum

total

data

transfer

cost

allocation

is

found

which

implies

that

the

optimal

solution

is

achieved

The

value

of

the

minimum

total

data

transfer

cost

is

the

S

T

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Capacity

Cost

Figure

Formulating

the

data

allocation

problem

to

max-flow

min-cut

problem

wij

l

i

i

m

k

Si

Oj

Sp

p

m

p

i

Oj

Si

Si

li

i

k

dj

j

j

j

k

O

j

S

i

c

ii

ai

j

rj

j

-LRB-

-RRB-

j

n

i

m

S

i

Oj

sum

of

the

costs

on

the

edges

from

to

which

form

the

maximum-flow

and

minimum-cost

solution

-LRB-

QED

-RRB-

Example

Consider

a

distributed

multimedia

database

system

with

fully

connected

sites

and

and

two

MDOs

ESpeech

and

EPicture

Let

there

be

only

one

query

-LRB-

as

shown

in

Figure

-RRB-

accessing

the

multimedia

service

provider

The

MDO

dependency

graph

is

illustrated

in

Figure

-LRB-

c

-RRB-

Let

this

query

be

initiated

from

site

with

frequency

from

site

with

frequency

and

from

site

with

frequency

Let

the

sizes

of

intermediate

MDOs

be

size

-LRB-

ESpeech

-RRB-

and

size

-LRB-

EPicture

-RRB-

Since

there

is

only

one

query

the

matrix

corresponding

to

MDOs

ESpeech

and

EPicture

respectively

and

the

matrix

-LRB-

i.e.

a

column

vector

-RRB-

corresponding

to

sites

and

respectively

Let

the

limit

vector

l

constraining

the

number

of

MDOs

that

can

be

allocated

to

a

site

be

Then

the

matrix

is

as

follows

-LRB-

the

rows

correspond

to

sites

and

respectively

and

the

columns

correspond

to

MDOs

ESpeech

and

EPicture

respectively

-RRB-

Let

the

cost

matrix

for

unit

data

transfer

cost

from

one

site

to

another

be

-LRB-

the

rows

and

columns

correspond

to

sites

and

respectively

-RRB-

Then

the

total

data

transfer

from

a

MDO

to

a

site

is

given

by

the

matrix

as

follows

-LRB-

the

rows

correspond

to

sites

and

respectively

and

the

columns

correspond

to

MDOs

ESpeech

and

EPicture

respectively

-RRB-

Figure

illustrates

the

max-flow

min-cut

formulation

for

this

example

After

applying

the

Ford

and

Fulkerson

method

-LSB-

-RSB-

to

solve

for

maximum

flow

with

minimum

cost

we

get

the

solution

as

allocate

E

to

site

and

J

to

site

The

total

data

transfer

cost

is

The

Hill-Climbing

Approach

The

general

data

allocation

problem

has

been

proved

to

be

NP-complete

-LSB-

-RSB-

Finding

the

optimal

solution

by

exhaustive

search

would

require

in

the

worst

case

where

k

is

the

number

of

MDOs

and

m

is

the

number

of

sites

Therefore

a

heuristic

algorithm

based

on

hillclimbing

technique

is

developed

in

order

to

find

a

near

optimal

solution

The

general

data

allocation

problem

solution

consists

of

following

two

steps

-RRB-

As

a

first

step

neglect

the

dependencies

among

the

MDOs

and

come

up

with

a

initial

w

ij

Si

O

j

S

S

R

A

t

S

l

U

A

R

U

S

C

W

C

U

S

S

W

O

km

-LRB-

-RRB-

data

allocation

by

using

the

max-flow

min-cut

formulation

of

the

problem

-LRB-

as

described

in

Section

-RRB-

-RRB-

Iteratively

improve

the

initial

data

allocation

by

using

the

hill

climbing

heuristic

until

no

further

reduction

in

total

data

transfer

cost

can

be

achieved

This

is

done

by

applying

some

operations

on

the

initial

allocation

scheme

Since

there

are

finite

number

of

feasible

allocations

the

heuristic

algorithm

will

complete

its

execution

The

initial

solution

generated

by

the

max-flow

min-cut

formulation

is

refined

by

applying

some

operations

on

it

The

objective

for

applying

these

operations

is

to

reduce

the

total

data

transfer

cost

There

are

two

types

of

operations

that

are

defined

namely

migrate

-LRB-

migrate

MDOs

from

its

currently

allocated

site

to

the

newly

allocated

site

-RRB-

and

swap

-LRB-

swap

the

locations

of

one

set

of

MDOs

with

the

locations

of

new

set

of

MDOs

-RRB-

Note

that

a

migrate

operation

can

not

be

applied

if

the

limit

on

the

number

of

MDOs

that

can

be

located

at

a

site

is

exceeded

Whereas

the

swap

operation

does

not

change

the

number

of

MDOs

allocated

at

each

of

the

sites

These

operations

take

into

consideration

the

dependency

between

the

MDOs

as

modeled

by

MDO

dependency

graphs

In

order

to

come

up

with

a

general

solution

these

operations

are

independent

of

exact

structure

of

a

MDO

dependency

graph

but

are

based

on

the

type

of

processing

involved

in

executing

a

query

in

a

distributed

fashion

These

operations

are

iteratively

applied

until

no

more

reduction

is

observed

in

the

total

data

transfer

cost

Figure

shows

the

major

steps

in

the

hill-climbing

heuristic

algorithm

that

is

developed

in

this

section

The

set

of

migrate

and

swap

operations

are

as

follows

move

MDO

to

This

operation

can

be

applied

to

each

MDO

and

the

MDO

can

be

moved

to

any

one

of

sites

-LRB-

it

is

not

located

at

-RRB-

Therefore

there

can

be

maximum

of

migrate

operations

that

can

be

applied

during

each

iteration

The

migrate

operation

can

potentially

reduce

the

data

transfer

cost

involved

in

processing

the

binary

operation

over

two

MDOs

If

one

MDO

is

located

at

one

site

and

another

MDO

is

located

at

a

different

site

then

migrating

one

MDO

to

the

site

of

another

can

potentially

reduce

data

transfer

cost

move

MDO

to

site

and

MDO

to

site

This

operation

can

be

applied

to

each

pair

of

MDOs

and

they

can

each

be

moved

to

any

one

of

sites

where

they

are

not

located

Therefore

there

can

be

maximum

of

ESpeech

EPicture

S

T

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Capacity

Cost

Figure

Max-flow

min-cut

formulation

for

the

example

allocation

problem

-LCB-

-RCB-

m

igrate

O

j

S

i

-LRB-

-RRB-

O

j

Si

m

k

m

-LRB-

-RRB-

m

igra

t

O

j

S

i

Oj

Si

-LRB-

-RRB-

Oj

S

i

Oj

S

i

m

operations

that

can

be

applied

during

each

iteration

As

is

case

of

migrate

operation

the

operation

can

potentially

reduce

data

transfer

costs

in

processing

binary

operations

involving

three

MDOs

move

MDO

to

site

MDO

to

site

and

MDO

to

site

This

operation

can

be

applied

to

each

triplet

of

MDOs

and

they

can

each

be

moved

to

any

one

of

sites

where

they

are

not

located

Therefore

there

can

be

a

maximum

of

operations

that

can

be

applied

during

each

iteration

The

operations

can

potentially

reduce

data

transfer

costs

in

processing

binary

operations

over

four

MDOs

swap

the

location

of

MDO

with

the

location

of

MDO

This

operation

can

be

applied

to

each

distinct

pair

of

MDOs

Therefore

there

can

be

a

maximum

of

swap

operations

that

can

be

applied

during

each

iteration

The

swap

operation

is

useful

in

minimizing

the

total

data

transfer

cost

by

monitoring

the

data

transfer

between

two

sites

Since

there

is

a

limit

on

number

of

MDOs

that

can

be

allocated

at

a

site

by

swapping

MDOs

we

maintain

this

constraint

and

simultaneously

explore

if

we

can

improve

the

allocation

swap

the

location

of

MDO

with

the

location

of

MDO

and

location

of

with

location

of

This

operation

can

be

applied

to

each

of

the

distinct

pairs

of

MDOs

and

The

operation

is

also

useful

in

reducing

the

total

data

transfer

cost

while

maintaining

the

constraint

on

number

of

MDOs

that

can

be

allocated

to

a

site

swap

the

location

of

MDO

with

the

location

of

MDO

location

of

with

location

of

and

location

of

with

the

location

of

Max-flow

Min-cut

Algorithm

Apply

migrate

operations

Apply

swap

operations

Any

reduction

in

Total

Data

Transfer

Cost

Any

reduction

in

Total

Data

Transfer

Cost

No

No

Yes

Yes

Incrementally

improved

allocation

Incrementally

improved

allocation

Initial

allocation

Figure

Steps

in

the

Hill-Climbing

algorithm

for

data

allocation

Final

Allocation

k

k

-LRB-

-RRB-

-LRB-

-RRB-

m

-LRB-

-RRB-

migra

t

Oj

Si

O

j

Si

O

j

Si

-LRB-

-RRB-

O

j

Si

O

j

Si

O

j

S

i

m

k

k

-LRB-

-RRB-

k

-LRB-

-RRB-

-LRB-

-RRB-

m

-LRB-

-RRB-

swap

O

x

O

x

-LRB-

-RRB-

Ox

Ox

k

k

-LRB-

-RRB-

O

x

O

x

O

y

O

y

-LRB-

-RRB-

Ox

O

x

O

y

O

y

O

x

O

x

Oy

O

y

O

x

O

x

O

y

O

y

Oz

O

z

-LRB-

-RRB-

O

x

O

x

Oy

Oy

Oz

This

operation

can

be

applied

to

each

of

the

distinct

pairs

of

MDOs

and

The

operation

is

also

useful

in

reducing

the

total

data

transfer

cost

while

maintaining

the

constraint

on

number

of

MDOs

that

can

be

allocated

to

a

site

In

the

experiments

that

we

have

conducted

we

noticed

that

migrate

and

swap

operations

are

applied

most

often

followed

by

and

operations

The

least

frequent

occurrences

were

that

of

and

operations

In

our

extensive

experimental

studies

we

found

that

many

a

times

just

applying

migrate

and

swap

operations

would

not

give

the

global

optimal

solution

but

would

give

a

global

solution

when

and/or

operations

or

and/or

operations

were

applied

This

was

the

reason

for

us

to

define

three

different

types

of

migrate

and

swap

operations

The

run

time

complexity

was

higher

for

the

hill-climbing

algorithm

to

use

or

operations

Example

We

use

the

example

presented

in

Section

with

the

distributed

database

system

using

the

move-small

query

execution

strategy

The

MDO

dependency

graph

is

as

illustrated

in

Figure

-LRB-

b

-RRB-

Let

the

MDO

sizes

are

size

-LRB-

E

-RRB-

size

-LRB-

J

-RRB-

and

size

-LRB-

G

-RRB-

First

we

use

the

top-level

of

the

MDO

dependency

graph

i.e.

the

edge

from

to

solve

for

initial

allocation

scheme

by

using

maximum-flow

minimum

cut

formulation

Let

this

query

be

initiated

from

site

with

frequency

from

site

with

frequency

and

from

site

with

frequency

As

there

is

only

one

query

we

have

the

matrix

with

each

element

corresponding

to

relations

E

G

and

J

respectively

and

the

matrix

-LRB-

i.e.

a

column

vector

-RRB-

with

each

element

corresponding

to

sites

and

respectively

Let

the

limit

vector

l

constraining

the

number

of

MDOs

that

can

be

allocated

to

a

site

be

Let

the

cost

matrix

for

unit

data

transfer

cost

from

one

site

to

another

be

-LRB-

the

rows

and

columns

correspond

to

sites

and

respectively

-RRB-

After

formulating

the

problem

as

maximum-flow

and

minimum

cut

problem

and

solving

it

-LRB-

as

illustrated

in

Section

-RRB-

we

get

the

initial

solution

as

allocate

MDO

E

at

site

allocate

MDO

G

at

site

and

allocate

MDO

J

at

site

This

allocation

schema

is

represented

as

corresponding

to

the

sites

where

the

relations

E

G

and

J

are

to

be

located

respectively

The

matrix

giving

the

amount

of

data

needed

from

a

MDO

to

be

transported

to

site

of

another

MDO

derived

from

the

MDO

dependency

graph

-LRB-

shown

in

Figure

-LRB-

c

-RRB-

-RRB-

is

The

row

and

columns

of

matrix

correspond

to

MDOs

E

G

and

J

respectively

We

get

the

total

amount

of

data

that

must

be

transported

from

the

site

where

one

MDO

is

located

to

the

site

where

another

MDO

is

located

as

-LRB-

the

rows

and

columns

correspond

to

MDOs

E

O

z

Ox

Ox

Oy

O

y

O

z

O

z

S

i

t

e

J

-LRB-

-RRB-

Site

Q

-LRB-

-RRB-

fi

S

S

R

A

t

S

S

l

C

S

S

S

-LCB-

-RCB-

G

and

J

respectively

-RRB-

The

total

data

transfer

cost

given

for

the

initial

allocation

is

The

cost

value

in

first

parenthesis

corresponds

to

the

data

transfer

cost

incurred

in

transferring

E

to

site

where

G

is

located

and

transferring

G

to

the

site

where

J

is

located

The

second

cost

value

corresponds

to

the

data

transfer

incurred

in

transferring

the

result

J

to

the

query

site

We

now

apply

the

hill-climbing

heuristic

algorithm

to

improve

upon

the

initial

solution

so

as

to

further

reduce

the

total

data

transfer

cost

if

possible

Table

illustrates

the

working

of

hill-climbing

algorithm

Table

shows

the

migrate

and

swap

operations

applied

to

improve

the

initial

solution

provided

by

the

max-flow

min-cut

algorithm

Two

operations

were

applied

i

-RRB-

-LRB-

which

reduced

the

total

data

transfer

cost

from

to

-RRB-

and

ii

-RRB-

-LRB-

which

reduced

the

data

transfer

cost

from

to

-RRB-

The

solution

to

data

allocation

problem

generated

by

the

hill-climbing

algorithm

is

-LRB-

i.e.

allocate

relation

E

at

site

relation

G

at

site

and

relation

J

at

site

-RRB-

with

the

total

data

transfer

cost

incurred

to

execute

the

query

as

Table

shows

all

feasible

allocation

schemes

and

the

total

data

transfer

cost

incurred

for

each

of

them.Table

also

shows

the

itemized

costs

for

the

data

transfer

The

cost

involved

in

transferring

E

to

site

where

relation

G

is

located

is

given

by

the

column

the

column

gives

the

cost

of

transferring

to

the

site

where

relation

J

is

located

and

the

column

gives

the

cost

of

transferring

the

J

to

the

query

site

Following

points

can

be

noted

after

comparing

hill-climbing

algorithm

with

the

results

of

exhaustive

solution

Table

Output

of

the

Hill-Climbing

Algorithm

for

data

allocation

Current

Allocation

Operation

Type

Operation

Applied

New

Allocation

Total

Data

Transfer

Cost

migrate

migrate

applied

applied

migrate

applied

applied

applied

D

-LRB-

-RRB-

S

S

-LCB-

-RCB-

-LRB-

-RRB-

-LRB-

-RRB-

migrate

J

-LRB-

-RRB-

m

i

grate

E

G

J

S

-LRB-

-RRB-

S

-LCB-

-RCB-

S

S

S

S

i

ze

E

-LRB-

-RRB-

S

i

t

e

G

-LRB-

-RRB-

fi

Size

G

-LRB-

-RRB-

Site

J

-LRB-

-RRB-

fi

G

S

ize

J

-LRB-

-RRB-

S

it

e

Q

-LRB-

-RRB-

fi

-LCB-

-RCB-

m

i

grat

e

J

S

-LRB-

-RRB-

-LCB-

-RCB-

S

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

m

i

grat

e

E

S

G

S

J

S

-LRB-

-RRB-

S

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

-RRB-

The

hill-climbing

algorithm

generates

the

optimal

solution

-LRB-

which

is

allocation

number

-RRB-

in

Table

-RRB-

The

hill-climbing

algorithm

would

not

have

find

the

optimal

solution

if

only

migrate

operation

had

been

used

Note

that

only

after

applying

operation

was

optimal

solution

achieved

-RRB-

The

hill-climbing

algorithm

may

not

always

generate

an

optimal

solution

BFS/ESP

Algorithm

In

this

section

the

data

allocation

problem

is

formulated

as

a

state-space

search

problem

swap

applied

applied

applied

Table

Enumeration

of

feasible

allocation

schemes

and

the

respective

data

transfer

costs

Allocation

Number

Allocation

Scheme

Data

Transfer

Cost

-LRB-

Itemized

-RRB-

Total

Data

Transfer

Cost

Table

Output

of

the

Hill-Climbing

Algorithm

for

data

allocation

Current

Allocation

Operation

Type

Operation

Applied

New

Allocation

Total

Data

Transfer

Cost

-LCB-

-RCB-

-LCB-

-RCB-

-LCB-

-RCB-

Si

ze

E

-LRB-

-RRB-

S

it

e

G

-LRB-

-RRB-

fi

Size

G

-LRB-

-RRB-

S

i

t

e

G

-LRB-

-RRB-

fi

S

iz

e

J

-LRB-

-RRB-

S

it

e

Q

-LRB-

-RRB-

fi

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

S

-LCB-

-RCB-

-LRB-

-RRB-

A

best-first

search

algorithm

which

is

based

on

similar

principles

as

the

A

algorithm

is

used

for

solving

the

problem

-LSB-

-RSB-

In

a

state-space

search

problem

each

state

description

is

denoted

by

a

node

Operators

applicable

to

nodes

are

defined

for

generating

successors

of

nodes

called

node

expansion

A

solution

path

of

a

search

problem

is

a

path

in

the

state-space

defined

by

a

sequence

of

operators

which

leads

a

start

node

to

one

of

the

goal

nodes

The

A

algorithm

has

been

successfully

applied

to

the

tasks

to

processors

mapping

problem

in

parallel

processing

In

the

data

allocation

problem

a

solution

path

defines

an

allocation

which

has

the

minimum

cost

The

state-space

search

problem

is

formalized

as

follows

State

Description

Let

a

set

of

ordered

pairs

of

MDO

and

site

denote

the

partially

developed

allocation

corresponding

to

a

node

in

the

search

tree

Initial

State

The

initial

state

is

the

empty

set

Operators

An

operator

adds

a

new

pair

of

MDO

and

site

to

the

current

set

under

node

expansion

Every

unallocated

MDO-site

pair

is

a

candidate

Goal

State

The

goal

state

is

reached

if

every

the

MDO

is

allocated

to

a

site

The

above

formulation

just

offers

a

search

scheme

for

finding

a

solution

We

use

an

algorithm

called

Best-First

Search

with

Efficient

Search-space

Pruning

-LRB-

BFS/ESP

-RRB-

to

find

the

optimal

solution

The

BFS/ESP

algorithm

is

based

on

similar

principles

as

the

A

algorithm

in

that

it

uses

an

evaluation

function

to

judge

which

search

node

to

expand

next

In

an

A

algorithm

an

evaluation

function

is

used

to

order

nodes

for

expansion

and

is

guaranteed

to

find

a

optimal

solution

path

in

that

the

path

cost

is

minimized

if

the

evaluation

function

is

admissible

More

specifically

we

define

an

evaluation

function

as

where

is

the

minimum

path

cost

from

the

start

node

to

node

N

in

the

state-space

and

is

a

lower-bound

estimate

using

any

heuristic

information

available

of

the

minimum

path

cost

from

node

N

to

a

goal

node

As

long

as

for

all

N

-LRB-

i.e.

is

consistent

-RRB-

then

is

called

admissible

and

the

A

algorithm

using

such

an

evaluation

function

will

be

guaranteed

to

find

an

optimal

solution

path

in

the

state

space

after

expanding

fewer

nodes

during

the

search

than

any

uninformed

algorithms

That

is

such

a

heuristic

search

algorithm

can

speed

up

the

search

of

an

optimal

data

allocation

which

is

usually

time

consuming

for

graphs

with

large

numbers

of

MDOs

and

sites

In

order

to

speed

up

the

search

process

by

properly

pruning

the

search

space

the

heuristic

function

has

to

be

tight

If

its

value

is

too

small

-LRB-

e.g.

-RRB-

then

the

search

degenerates

to

an

exhaustive

search

If

its

value

is

too

large

then

it

may

not

be

admissible

and

can

not

guarantee

optimal

solution

Thus

such

heuristic

function

is

problem-dependent

and

is

usually

difficult

to

define

For

the

data

allocation

algorithm

of

a

search

node

N

is

simply

the

cost

of

the

partial

allocation

This

partial

cost

can

be

easily

computed

by

using

the

total

transfer

cost

expression

with

the

unallocated

MDOs

being

excluded

from

the

expression

As

to

the

heuristic

function

it

can

be

defined

by

several

different

approaches

The

simplest

way

is

to

set

for

all

N

and

the

resulting

search

is

a

uniform-cost

search

To

be

more

efficient

in

the

search

nonzero

should

be

used

In

our

approach

we

define

as

follows

-LRB-

-RRB-

Let

O

be

the

set

of

unallocated

MDOs

at

the

current

search

node

N.

-LRB-

-RRB-

Set

-LRB-

-RRB-

FOR

each

MDO

DO

f

N

-LRB-

-RRB-

g

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

g

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

g

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

o

O

-LRB-

-RRB-

Set

-LRB-

-RRB-

FOR

each

site

s

DO

-LRB-

-RRB-

Determine

ThisCost

to

be

the

additional

allocation

cost

if

o

is

allocated

to

s.

-LRB-

-RRB-

IF

THEN

Set

-LRB-

-RRB-

ENDFOR

-LRB-

-RRB-

-LRB-

-RRB-

ENDFOR

Using

the

above

procedure

however

it

is

not

difficult

to

see

that

is

not

always

a

consistent

lower

bound

of

for

all

N

In

other

words

the

search

algorithm

can

not

guarrantee

optimal

solutions

in

all

cases

Given

the

definitions

of

and

the

BFS

ESP

algorithm

for

solving

the

data

allocation

problem

is

briefly

described

below

BFS/ESP

Data

Allocation

Algorithm

-LRB-

-RRB-

Put

the

initial

search

node

on

a

list

called

OPEN

and

set

-LRB-

-RRB-

Remove

from

OPEN

the

search

node

n

with

the

smallest

f

and

put

it

on

a

list

called

CLOSED

-LRB-

-RRB-

If

N

is

a

goal

node

then

determine

the

final

allocation

and

stop

otherwise

go

to

the

next

step

-LRB-

-RRB-

Expand

search

node

n

by

considering

to

allocate

the

next

MDO

to

all

the

sites

Compute

for

each

successor

node

N

of

N.

Put

all

the

successors

on

OPEN

Go

to

step

-LRB-

-RRB-

In

the

worst

case

the

BFS/ESP

algorithm

may

explore

the

entire

search

tree

before

the

best

solution

is

located

Thus

the

worst

case

time

complexity

of

the

BFS/ESP

algorithm

is

However

it

should

be

noted

that

with

the

above

properly

defined

heuristic

node

evaluation

function

the

BFS/ESP

algorithm

can

take

much

less

time

than

the

worst

case

time

complexity

implies

-LSB-

-RSB-

Agglomerative

Clustering

In

this

section

we

describe

two

clustering

approaches

to

tackle

the

data

allocation

problem

In

general

search-based

methods

can

produce

better

solutions

at

the

expense

of

much

longer

execution

time

while

clustering

approaches

are

much

faster

but

may

generate

less

optimal

solutions

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Clustering

algorithms

can

be

broadly

divided

into

two

classes

-LSB-

-RSB-

Agglomerative

Algorithms

The

graph

to

be

clustered

is

initially

considered

to

have

N

unit

clusters

A

number

of

clustering

operations

are

then

performed

to

produce

a

certain

number

of

larger

clusters

Divisive

Algorithms

The

graph

is

initially

considered

to

be

a

single

cluster

The

large

cluster

is

then

incrementally

cracked

to

form

a

number

of

smaller

clusters

In

this

section

we

present

an

agglomerative

algorithm

that

performs

clustering

of

the

MDOs

followed

by

a

mapping

of

the

clusters

to

the

sites

In

this

algorithm

the

MDOs

are

considered

to

be

connected

as

a

clique

Each

edge

in

the

clique

is

associated

with

a

weight

denoted

by

representing

the

aggregate

amount

of

data

dependence

between

the

two

MDOs

i

and

i

across

all

the

queries

Each

node

in

the

clique

represents

the

maximum

allocation

cost

of

the

MDO

denoted

by

-LRB-

determined

by

the

query-site

strategy

-RRB-

across

all

sites

Given

the

MDO

dependency

graphs

and

the

queries

set

this

MDO

clique

can

be

M

i

nCost

MAXINT

Th

i

sCost

MinCost

MinCost

Th

isCost

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

M

i

nCost

h

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

g

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

I

f

f

I

-LRB-

-RRB-

f

N

-LRB-

-RRB-

g

N

-LRB-

-RRB-

h

N

-LRB-

-RRB-

O

m

k

-LRB-

-RRB-

ei

i

w

i

computed

before

the

clustering

algorithm

starts

The

clustering

algorithm

works

by

first

locating

a

pivot

MDO

defined

as

the

MDO

that

is

connected

by

the

heaviest

edge

and

then

incrementally

includes

neighbor

MDOs

until

the

cumulative

maximum

MDO

allocation

costs

exceeds

a

threshold

T

This

threshold

is

simply

the

sum

of

all

the

maximum

MDO

allocation

costs

divided

by

the

number

of

sites

in

the

system

Following

the

clustering

of

the

pivot

MDO

the

same

procedure

is

applied

to

a

new

pivot

MDO

selected

from

the

remaining

unclustered

MDOs

After

a

set

of

MDO

clusters

are

formed

each

cluster

is

mapped

to

a

site

The

criterion

of

choosing

a

site

for

a

MDO

cluster

is

simple

the

site

that

gives

the

least

increase

in

the

partial

allocation

cost

is

the

target

site

The

algorithm

is

briefly

presented

below

Agglomerative

Clustering

-LRB-

-RRB-

Construct

the

MDO

clique

by

For

each

MDO

i

compute

the

maximum

allocation

cost

i

across

all

sites

This

value

denoted

by

is

taken

as

the

node

weight

of

i

in

the

clique

For

each

pair

of

MDO

i

and

i

determine

the

aggregate

MDO

dependency

costs

between

i

and

i

across

all

queries

This

cost

is

denoted

by

and

is

taken

as

the

edge

weight

in

the

clique

-LRB-

-RRB-

Determine

a

threshold

T

defined

as

the

sum

of

all

weights

divided

by

the

number

of

sites

-LRB-

-RRB-

WHILE

not

all

MDOs

are

clustered

DO

-LRB-

-RRB-

Determine

the

pivot

MDO

defined

as

the

MDO

with

the

heaviest

edge

-LRB-

-RRB-

Incrementally

include

neighbor

MDOs

in

order

of

decreasing

edge

weight

until

the

total

node

weights

exceeds

T.

-LRB-

-RRB-

ENDWHILE

-LRB-

-RRB-

Set

-LRB-

-RRB-

FOR

each

MDO

cluster

C

in

decreasing

order

of

total

weights

DO

-LRB-

-RRB-

Map

C

to

a

site

s

such

that

the

increase

in

the

partial

allocation

cost

is

the

minimum

-LRB-

-RRB-

-LRB-

-RRB-

ENDFOR

Compared

with

the

previous

approaches

the

AC

clustering

algorithm

incurs

much

less

time

complexity

The

time

complexity

of

the

AC

algorithm

is

only

Divisive

Clustering

The

agglomerative

clustering

-LRB-

AC

-RRB-

algorithm

described

above

is

designed

to

be

a

MDO

dependency-oriented

algorithm

That

is

we

put

the

main

emphasis

on

the

dependency

among

MDOs

in

determining

the

target

data

allocation

In

this

section

we

present

a

divisive

clustering

-LRB-

DC

-RRB-

algorithm

which

is

designed

to

be

a

site-oriented

algorithm

That

is

we

consider

the

network

communication

costs

among

the

sites

as

more

important

factor

in

determining

a

better

allocation

More

specifically

the

DC

algorithm

works

by

first

allocating

all

the

MDOs

to

a

pivot

site

at

once

The

pivot

site

is

the

one

that

has

the

lowest

aggregate

communication

costs

among

all

the

sites

The

sum

of

all

the

communication

overhead

of

the

outgoing

links

of

the

pivot

site

to

other

sites

is

the

minimum

among

all

the

sites

Each

MDO

is

then

considered

to

be

migrated

to

another

site

which

can

lower

the

cost

of

the

allocation

All

sites

are

considered

for

the

migration

of

each

MDO

The

process

terminates

when

all

the

wi

ei

i

TotalCost

CostD

TotalCost

Tota

l

Cost

Cos

tD

O

k

m

-LRB-

-RRB-

-LRB-

-RRB-

MDOs

are

examined

Similar

to

the

AC

algorithm

the

design

of

this

DC

algorithm

is

again

simple

so

that

it

admits

very

efficient

implementation

The

algorithm

is

briefly

described

below

Divisive

Clustering

-LRB-

-RRB-

Find

out

the

site

that

has

the

lowest

aggregate

communication

overhead

across

all

channels

Let

it

be

the

pivot

site

-LRB-

-RRB-

Allocate

all

the

MDOs

to

the

pivot

site

Determine

the

cost

of

the

allocation

-LRB-

-RRB-

FOR

each

MDO

o

in

the

pivot

site

DO

-LRB-

-RRB-

Set

-LRB-

-RRB-

FOR

each

other

site

s

DO

-LRB-

-RRB-

Compute

if

o

is

migrated

to

s.

-LRB-

-RRB-

IF

THEN

set

-LRB-

-RRB-

ENDFOR

-LRB-

-RRB-

Migrate

o

to

BestSite

-LRB-

-RRB-

ENDFOR

Similar

to

the

AC

algorithm

the

time

complexity

of

the

DC

algorithm

is

also

much

less

than

the

search

based

methods

The

time

complexity

of

the

DC

algorithm

is

Results

In

this

section

we

present

the

experimental

results

for

the

data

allocation

algorithms

described

in

the

previous

sections

Comparisons

among

these

algorithms

will

be

made

by

considering

the

quality

of

solutions

and

the

algorithm

running

times

Workload

The

example

considered

in

the

previous

section

was

used

for

illustrating

how

the

hillclimbing

algorithm

works

But

it

had

only

one

query

three

MDOs

and

three

sites

The

number

of

feasible

allocations

were

only

with

data

transfer

cost

ranging

from

to

Since

the

solutions

were

to

be

compared

with

the

optimal

solutions

generated

by

exhaustive

search

and

the

exhaustive

search

method

takes

large

amount

of

time

to

experiment

for

a

distributed

database

system

with

even

moderate

number

of

sites

and

MDOs

-LRB-

for

k

MDOs

and

m

sites

there

are

allocation

schemes

and

for

each

allocation

scheme

the

data

transfer

cost

needs

to

be

calculated

-RRB-

we

conducted

sixteen

experiments

with

number

of

MDOs

ranging

from

to

and

number

of

sites

ranging

from

to

Each

experiment

consisted

of

allocation

problems

with

number

of

sites

and

number

of

MDOs

fixed

Each

allocation

problem

had

between

and

queries

and

each

query

had

a

MDO

dependency

graph

The

communication

network

the

relation

sizes

the

link

costs

and

the

structure

of

MDO

dependency

graph

were

randomly

generated

from

a

uniform

distribution

Each

data

allocation

algorithm

described

above

was

tested

for

every

case

and

statistics

were

collected

Comparison

of

Allocation

Costs

In

Table

to

we

list

for

each

of

the

experiments

conducted

in

a

column-wise

fashion

the

following

i

-RRB-

the

number

of

MDOs

ii

-RRB-

number

of

sites

iii

-RRB-

number

of

problems

iv

-RRB-

number

of

problems

for

which

the

hill-climbing

algorithm

generated

the

optimal

solution

v

-RRB-

the

average

deviation

in

percentage

of

near

optimal

solutions

from

optimal

solution

for

those

BestSite

p

i

vot

Cos

tD

Cos

t

D

Cos

t

CostD

D

Bes

t

S

i

t

e

s

CostD

Cost

D

O

k

m

k

m

-LRB-

-RRB-

-LRB-

-RRB-

km

allocation

problems

for

which

the

hill-climbing

algorithm

did

not

generate

optimal

solution

vi

-RRB-

number

of

near

optimal

solutions

with

deviation

less

than

vii

-RRB-

number

of

near

optimal

solutions

with

deviation

of

or

more

but

less

than

viii

-RRB-

number

of

near

optimal

solutions

with

deviation

of

or

more

but

less

than

and

ix

-RRB-

number

of

near

optimal

solutions

with

deviation

or

more

The

number

of

optimal

solutions

can

reflect

how

good

the

algorithm

is

whereas

the

average

deviation

shows

how

bad

the

algorithm

performs

when

it

can

not

generate

optimal

solutions

From

Table

we

note

that

the

hill-climbing

algorithm

generated

optimal

solution

for

a

large

number

of

problems

cases

out

of

a

total

of

cases

corresponding

to

about

of

the

test

cases

In

addition

the

overall

average

deviation

-LRB-

i.e.

the

mean

of

all

average

deviations

-RRB-

of

a

near

optimal

solution

from

the

optimal

solution

was

only

about

in

total

data

transfer

cost

Furthermore

for

the

tests

that

when

the

hill-climbing

algorithm

generates

suboptimal

solutions

almost

all

of

them

are

within

from

the

optimal

Thus

the

hillclimbing

approach

is

a

very

effective

algorithm

in

that

it

can

generate

optimal

solutions

for

most

cases

and

very

near

to

optimal

solutions

when

it

can

not

generate

optimal

solution

Table

includes

the

results

of

the

BFS/ESP

algorithm

Although

not

as

good

as

the

hillclimbing

approach

the

BFS/ESP

algorithm

performs

well

The

number

of

optimal

solutions

is

out

of

more

than

Moreover

more

than

of

the

remaining

cases

are

only

from

the

optimal

Only

cases

are

more

than

from

the

optimal

The

overall

average

deviation

from

optimal

is

small?about

The

BFS/ESP

algorithm

is

more

efficient

than

the

hill-climbing

approach

because

it

takes

order

of

magnitude

less

time

to

generate

a

solution

as

will

be

seen

in

the

following

subsection

Table

contains

the

performance

data

of

the

agglomerative

clustering

-LRB-

AC

-RRB-

algorithm

The

performance

of

the

AC

algorithm

is

comparable

to

that

of

the

BFS/ESP

algorithm

Despite

that

the

number

of

optimal

solutions

generated

is

fewer

-LRB-

out

of

-RRB-

the

number

of

solutions

with

costs

within

from

the

optimal

is

the

same

as

that

of

the

BFS/ESP

algorithm

This

can

be

noted

from

the

table

that

both

algorithms

have

only

cases

with

costs

more

than

from

the

optimal

The

overall

average

deviation

is

also

similar?about

Based

on

these

results

we

can

notice

that

the

agglomerative

clustering

algorithm

is

indeed

very

cost-effective

This

is

because

all

the

search

based

algorithms

take

exponential

time

in

No

of

No

of

Aver

Number

of

Sol

with

Deviation

in

range

Problems

Opt

Sol

Deviation

-LRB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

No

of

No

of

Sites

MDOs

Table

Experimental

results

of

the

Hill-Climbing

algorithm

the

worst

case

Table

contains

the

performance

results

of

the

divisive

clustering

-LRB-

DC

-RRB-

algorithm

The

performance

of

the

DC

algorithm

is

again

comparable

to

both

the

AC

and

BFS/ESP

algorithms

The

number

of

optimal

solutions

generated

is

out

of

And

almost

all

of

the

suboptimal

solutions

are

also

within

from

the

optimal

The

overall

average

deviation

is

also

similar

about

Thus

we

can

observe

that

for

the

data

allocation

problem

there

is

no

particular

preference

to

either

the

agglomerative

approach

or

the

divisive

approach

Comparison

of

Running

Times

Table

contains

the

average

running

times

of

all

algorithms

for

each

experiment

For

comparison

the

time

taken

to

generate

the

optimal

solutions

by

using

exhaustive

search

are

also

listed

All

the

algorithms

were

implemented

on

a

SPARC

IPX

workstation

and

the

time

data

were

measured

in

milli-seconds

As

can

be

seen

from

the

table

although

the

hillclimbing

approach

took

much

short

time

compared

with

exhaustive

search

it

took

order

of

magnitude

more

time

than

the

clustering

algorithms

and

the

BFS/ESP

algorithm

The

two

No

of

No

of

Aver

Number

of

Sol

with

Deviation

in

range

Problems

Opt

Sol

Deviation

-LRB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

No

of

No

of

Sites

MDOs

Table

Experimental

results

of

the

BFS/ESP

algorithm

No

of

No

of

Aver

Number

of

Sol

with

Deviation

in

range

Problems

Opt

Sol

Deviation

-LRB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

No

of

No

of

Sites

MDOs

Table

Experimental

results

of

the

Agglomerative

Clustering

algorithm

clustering

algorithms

were

found

to

be

the

fastest

algorithms

Indeed

they

are

the

more

costeffective

algorithms

when

compared

against

the

other

algorithms

Discussion

From

the

experiment

results

presented

in

the

previous

section

we

observe

that

the

clustering

algorithms

are

very

cost-effective

if

fast

execution

is

desired

Therefore

these

algorithms

can

be

applied

to

generate

allocation

schemes

for

limited

redesign

of

distributed

multimedia

databases

-LSB-

-RSB-

Limited

redesign

implies

change

in

allocation

scheme

but

not

in

fragmentation

scheme

for

a

distributed

database

The

algorithms

developed

in

this

paper

can

be

integrated

with

algorithms

for

affecting

the

limited

redesign

of

distributed

databases

-LSB-

-RSB-

and

thus

facilitating

on-line

tuning

of

a

distributed

multimedia

database

system

If

solution

quality

is

the

more

prominent

factor

the

hill-climbing

approach

is

a

viable

choice

for

an

offline

allocation

The

BFS/ESP

algorithm

whose

solution

quality

is

next

to

that

of

the

hillclimbing

approach

is

also

a

practical

approach

though

it

can

take

much

longer

time

than

the

clustering

algorithms

The

clustering

algorithms

can

be

useful

if

the

quality

of

the

solution

is

not

the

main

objective

but

the

algorithm

running

times

is

the

major

consideration

No

of

No

of

Aver

Number

of

Sol

with

Deviation

in

range

Problems

Opt

Sol

Deviation

-LRB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

-LSB-

-RRB-

No

of

No

of

Sites

MDOs

Table

Experimental

results

of

the

Divisive

Clustering

algorithm

Exhaustive

Hill

BFS/ESP

AC

DC

Search

Climbing

No

of

No

of

Sites

MDOs

Table

Average

running

times

-LRB-

msecs

-RRB-

of

all

the

algorithms

Related

Work

The

data

allocation

problem

has

been

first

studied

in

terms

of

file

allocation

problem

in

a

multi-computer

system

and

later

on

as

a

data

allocation

problem

in

distributed

database

system

The

file

allocation

problem

does

not

take

into

consideration

the

semantics

of

the

processing

being

done

of

files

whereas

data

allocation

problem

must

take

into

consideration

the

interdependencies

among

accesses

to

multiple

fragments

by

a

query

Chu

-LSB-

-RSB-

studied

the

problem

of

file

allocation

with

respect

to

multiple

files

on

a

multiprocessor

system

He

presented

a

global

optimization

model

to

minimize

overall

processing

costs

under

the

constraints

of

response

time

and

storage

capacity

with

a

fixed

number

of

copies

of

each

file

Casey

-LSB-

-RSB-

distinguished

between

updates

and

queries

on

files

Eswaran

-LSB-

-RSB-

proved

that

Casey?s

formulation

is

NP

complete

He

suggested

that

a

heuristic

rather

than

exhaustive

search

approach

is

more

suitable

Ramamoorthy

and

Wah

-LSB-

-RSB-

analyzed

a

file

allocation

problem

in

the

environment

of

a

distributed

database

and

developed

a

heuristic

approximation

algorithm

for

a

simple

file

allocation

problem

and

the

generalized

file

allocation

problem

They

also

proposed

a

model

for

file

migration

or

reallocation

that

is

identical

in

formulation

to

the

file

allocation

problem

Ceri

Martella

and

Pellagatti

-LSB-

-RSB-

considered

the

problem

of

file

allocation

for

typical

distributed

database

applications

with

a

simple

model

for

transaction

execution

taking

into

account

the

dependencies

between

accesses

to

multiple

fragments

Apers

-LSB-

-RSB-

considered

the

allocation

of

the

distributed

database

to

the

sites

so

as

to

minimize

total

data

transfer

cost

The

author

came

up

with

a

complicated

approach

to

allocate

relations

by

first

partitioning

them

into

innumerable

number

of

fragments

and

then

allocating

them

The

author

did

integrate

the

system

dependent

query

processing

strategy

with

the

logical

model

for

allocating

the

fragments

But

the

author

solves

the

complete

fragmentation

and

allocation

problem

That

is

the

fragmentation

schema

is

one

of

the

outputs

of

the

allocation

algorithm

This

curtails

the

applicability

of

this

methodology

when

fragmentation

schema

is

already

defined

and

allocation

scheme

must

be

generated

Cornell

and

Yu

-LSB-

-RSB-

proposed

a

strategy

to

integrate

the

treatment

of

relation

assignment

and

query

strategy

to

optimize

performance

of

a

distributed

database

system

Though

they

took

into

consideration

the

query

execution

strategy

the

solution

they

came

up

with

is

a

complicated

linear

programming

solution

The

main

problem

in

their

approach

is

the

lack

of

simplicity

in

both

incorporation

of

the

query

execution

strategy

and

the

solution

procedure

There

have

been

many

linear

programming

formulations

proposed

for

data

allocation

problem

-LSB-

-RSB-

-LSB-

-RSB-

The

main

problem

with

these

approaches

is

the

lack

of

modeling

of

the

query

execution

strategy

Lin

Orlawska

and

Zhang

-LSB-

-RSB-

also

developed

a

heuristic

algorithm

for

minimum

overall

data

transfer

cost

by

considering

replicated

allocation

of

fragments

and

both

read

and

update

transactions

Conclusions

The

problem

of

non-redundant

data

allocation

of

MDOs

in

distributed

database

system

is

addressed

in

this

paper

A

query

driven

data

allocation

approach

is

developed

by

integrating

the

query

execution

strategy

with

the

formulation

of

the

data

allocation

problem

We

studied

the

data

allocation

problem

for

two

popularly

used

query

execution

strategies

namely

query

site

and

move-small

We

formulated

the

data

allocation

problem

with

query-site

execution

strategy

as

a

max-flow

min-cut

problem

and

showed

that

it

can

be

solved

to

generate

an

optimal

solution

A

hill-climbing

heuristic

algorithm

was

developed

for

the

data

allocation

problem

with

move-small

query

execution

strategy

We

studied

the

effectiveness

of

this

algorithm

by

conducting

a

set

of

experiments

which

show

that

the

hill-climbing

heuristic

algorithm

generates

the

optimal

solution

for

a

large

percentage

of

allocation

problems

Moreover

the

near

optimal

solution

generated

by

the

algorithm

has

on

an

average

small

percentage

of

deviation

from

the

optimal

solution

in

terms

of

the

total

data

transfer

costs

incurred

to

process

all

the

queries

The

problem

was

also

formulated

as

a

state

space

searched

problem

and

was

solved

with

a

efficient

search

algorithm

called

BFS/ESP

Two

clustering

algorithms

were

also

proposed

which

have

much

shorter

execution

times

although

the

quality

of

solutions

are

slightly

worse

than

the

hill-climbing

approach

The

proposed

data

allocation

algorithms

can

be

used

in

practice

for

allocating

MDOs

in

distributed

multimedia

database

systems

A

major

contribution

of

this

work

is

the

development

of

a

site-independent

MDO

dependency

graph

representation

of

data

transfer

costs

incurred

in

executing

a

query

by

using

a

particular

query

execution

strategy

This

enables

the

application

of

query

driven

allocation

approach

to

allocate

database

objects

under

different

database

systems

-LRB-

like

relational

object-oriented

network

etc.

-RRB-

and

facilitates

the

development

of

an

uniform

framework

within

which

to

define

and

solve

data

allocation

problems

A

complicated

query

execution

strategy

employed

by

a

distributed

database

system

will

generate

complex

MDO

dependency

graphs

which

will

involve

development

of

novel

techniques

to

solve

for

optimal

allocation

References

-LSB-

-RSB-

P.M.G.

Apers

Data

Allocation

in

Distributed

Database

Systems

ACM

Trans

Database

Systems

-LRB-

-RRB-

Sep.

pp.

-LSB-

-RSB-

P.

B.

Berra

C.

Y.

R.

Chen

A.

Ghafoor

C.C.

Lin

T.

D.

C.

Little

and

D.

Shin

Architecture

for

Distributed

Multimedia

Systems

Computer

Communications

vol

no.

-LRB-

May

-RRB-

-LSB-

-RSB-

M.J.

Berger

and

S.H.

Bokhari

A

Partitioning

Strategy

for

Nonuniform

Problems

on

Multiprocessors

IEEE

Trans

Computers

vol

no.

May

pp.

-LSB-

-RSB-

N.S.

Bowen

C.N.

Nikolaou

and

A.

Ghafoor

On

the

Assignment

Problem

of

Arbitrary

Process

Systems

to

Heterogeneous

Distributed

Computer

Systems

IEEE

Trans

Computers

vol

no.

Mar.

pp.

-LSB-

-RSB-

R.G.

Casey

Allocation

of

Copies

of

a

File

in

an

Information

Network

Proc

Spring

Joint

Computer

Conf.

IFIPS

pp.

-LSB-

-RSB-

S.

Ceri

G.

Martella

and

G.

Pelagatti

Optimal

File

Allocation

for

a

Distributed

on

a

Network

of

Minicomputers

Proc

Int?l

Conf

Database

Aberdeen

Jul.

pp.

-LSB-

-RSB-

S.

Ceri

and

G.

Pelagatti

Distributed

Databases

Principles

and

Systems

McGraw

Hill

New

York

-LSB-

-RSB-

W.W.

Chu

Optimal

File

Allocation

in

a

Multiple

Computer

System

IEEE

Trans

Computers

vol

no.

-LSB-

-RSB-

D.

W.

Cornell

and

P.

S.

Yu

Site

Assignment

for

Relations

and

Join

Operations

in

the

Distributed

Transaction

Processing

Environment

Proc

Int?l

Conf

Data

Engineering

IEEE

Feb.

-LSB-

-RSB-

S.

Dutt

and

N.R.

Mahapatra

Parallel

A

Algorithms

and

their

Performance

on

Hypercube

Multiprocessors

Proc

Int?l

Parallel

Processing

Symposium

Apr.

pp.

-LSB-

-RSB-

K.P.

Eswaran

Placement

of

Records

in

a

File

and

File

Allocation

in

a

Computer

Network

Information

Processing

pp.

-LSB-

-RSB-

B.

Gavish

and

H.

Pirkul

Computer

and

Database

Location

in

Distributed

Computer

Systems

IEEE

Trans

Computers

vol

no

pp.

-LSB-

-RSB-

A.

Ghafoor

Multimedia

Database

Management

Systems

ACM

Computing

Surveys

vol

no.

Dec.

pp.

-LSB-

-RSB-

X.M.

Lin

M.E.

Orlowska

and

Y.C.

Zhang

Database

Placement

in

Communication

Networks

for

Minimizing

the

overall

Transmission

Cost

Mathematical

and

Computer

Modelling

-LRB-

-RRB-

Jan.

pp.

-LSB-

-RSB-

T.D.C.

Little

and

A.

Ghafoor

Synchronization

and

Storage

Models

for

Multimedia

Objects

IEEE

Journal

on

Selected

Areas

in

Communications

vol

no.

Apr.

pp.

-LSB-

-RSB-

V.M.

Lo

Heuristic

Algorithms

for

Task

Assignment

in

Distributed

Systems

IEEE

Trans

Computers

vol

no.

Nov.

pp

-LSB-

-RSB-

T.

zsu

and

P.

Valduriez

Principles

of

Database

Systems

Prentice-Hall

Inc.

-LSB-

-RSB-

M.T.

zsu

D.

Szafron

G.

El-Medani

and

C.

Vittal

An

Object-Oriented

Multimedia

Database

System

for

a

News-on-Demand

Application

to

appear

in

Multimedia

Systems

-LRB-

-RRB-

-LSB-

-RSB-

T.C.

Rakow

E.J.

Neuhold

and

M.

Lohr

Multimedia

Database

Systems

The

Notions

and

the

Issues

to

be

published

in

Tagungsband

GI-Fachtagung

Datenbanksystems

in

Buro

Technik

and

Wissenschaft

-LRB-

BTW

-RRB-

Dresden

Marz

Springer

Informatik

Aktuell

Berlin

-LSB-

-RSB-

S.

Ram

and

R.E.

Marsten

A

Model

for

Database

Allocation

Incorporating

a

Concurrency

Control

Mechanism

IEEE

Trans

Knowledge

and

Data

Engineering

-LRB-

-RRB-

Sep.

pp.

-LSB-

-RSB-

C.V.

Ramamoorthy

and

B.

Wah

The

Placement

of

Relations

on

a

Distributed

Relational

Database

Proc

Int?l

Conf

Distributed

Computing

Systems

Huntsville

Alabama

Oct.

pp.

-LSB-

-RSB-

P.I.

Rivera-Vega

R.

Varadarjan

and

S.B.

Navathe

Scheduling

Data

Redistribution

in

Distributed

Databases

Proc

Int?l

Conf

Data

Engineering

IEEE

Feb.

-LSB-

-RSB-

P.

Sadayappan

F.

Ercal

and

J.

Ramanujam

Cluster

Partitioning

Approaches

to

Mapping

Parallel

Programs

onto

a

Hypercube

Parallel

Computing

pp.

-LSB-

-RSB-

P.

Sadayappan

and

F.

Ercal

Nearest-Neighbor

Mapping

of

Finite

Element

Graphs

onto

Processor

Meshes

IEEE

Trans

Computers

vol

no.

Dec.

pp.

-LSB-

-RSB-

C.C.

Shen

and

W.H.

Tsai

A

Graph

Matching

Approach

to

Optimal

Task

Assignment

in

Distributed

Computing

Systems

Using

a

Minimax

Criterion

IEEE

Trans

Computers

vol

no.

Mar.

pp

-LSB-

-RSB-

R.E.

Tarjan

Data

Structures

and

Network

Algorithms

vol

CBMS-NSF

SIAM

-LSB-

-RSB-

B.

Wilson

and

S.B.

Navathe

An

Analytical

Framework

for

the

Redesign

of

Distributed

Databases

Proc

Advanced

Database

Symposium

Tokyo

Japan

clustering
data allocation
distributed database systems
multimedia database systems
optimal allocation
query processing

best-first search algorithm
hill-climbing heuristics
max-flow min-cut problem
network flow algorithm