Declustering

and

Load-Balancing

Methods

for

Parallelizing

Geographic

Information

Systems

Shashi

Shekhar

Sivakumar

Ravada

Vipin

Kumar

Douglas

Chubb

Greg

Turner

Abstract

Declustering

and

load-balancing

are

important

issues

in

designing

a

high

performance

geographic

information

system

-LRB-

HPGIS

-RRB-

which

is

a

central

component

of

many

interactive

applications

such

as

real-time

terrain

visualization

The

current

literature

provides

efficient

methods

for

declustering

spatial

point-data

However

there

has

been

little

work

towards

developing

efficient

declustering

methods

for

collections

of

extended

objects

like

chains

of

line-segments

and

polygons

In

this

paper

we

focus

on

the

data-partitioning

approach

to

parallelizing

GIS

operations

We

provide

a

framework

for

declustering

collections

of

extended

spatial

objects

by

identifying

the

following

key

issues

-LRB-

i

-RRB-

the

work-load

metric

-LRB-

ii

-RRB-

the

spatial-extent

of

the

work-load

-LRB-

iii

-RRB-

the

distribution

of

the

work-load

over

the

spatial-extent

and

-LRB-

iv

-RRB-

the

declustering

method

We

identify

and

experimentally

evaluate

alternatives

for

each

of

these

issues

In

addition

we

also

provide

a

framework

for

dynamically

balancing

the

load

between

different

processors

We

experimentally

evaluate

the

proposed

declustering

and

load-balancing

methods

on

a

distributed

memory

MIMD

machine

-LRB-

Cray

-RRB-

Experimental

results

show

that

the

spatial-extent

and

the

work-load

metric

are

important

issues

in

developing

a

declustering

method

Experiments

also

show

that

replication

of

data

is

usually

needed

to

facilitate

dynamic

load-balancing

as

the

cost

of

local

processing

is

often

less

than

the

cost

of

data

transfer

for

extended

spatial

objects

In

addition

we

also

show

that

the

effectiveness

of

dynamic

load-balancing

techniques

can

be

improved

by

using

declustering

methods

to

determine

the

subsets

of

spatial

objects

to

be

transferred

during

run-time

Author

Address

E-mail

Phone/Fax

Shashi

Shekhar

Sivakumar

Ravada

Vipin

Kumar

Department

of

Computer

Science

University

of

Minnesota

EE/CS

Union

St.

SE

Minneapolis

MN

shekhar@cs.umn.edu

siva@cs.umn.edu

kumar@cs.umn.edu

Douglas

Chubb

Research

and

Technology

Division

U.S.

Army

CECOM

RDEC

IEWD

Vint

Hill

Farms

Station

Warrenton

VA

Greg

Turner

Army

Research

Laboratory

Adelphi

MD

This

work

was

supported

by

the

Army

High

Performance

Computing

Research

Center

under

the

agreement

contract

Introduction

A

high

performance

geographic

information

system

-LRB-

HPGIS

-RRB-

is

a

central

component

of

many

interactive

applications

like

real-time

terrain

visualization

situation

assessment

and

spatial

decision

making

The

geographic

information

system

-LRB-

GIS

-RRB-

often

contains

large

amounts

of

geometric

and

feature

data

-LRB-

e.g.

location

elevation

soil

type

etc.

-RRB-

represented

as

large

sets

of

points

chains

of

line-segments

and

polygons

This

data

is

often

accessed

via

range

queries

and

map-overlay

queries

The

existing

sequential

methods

for

supporting

the

GIS

operations

do

not

meet

the

real-time

requirements

imposed

by

many

interactive

applications

Hence

parallelization

of

GIS

is

essential

in

meeting

the

high

performance

requirements

of

several

real-time

applications

A

GIS

operation

can

be

parallelized

either

by

function-partitioning

-LSB-

-RSB-

or

by

datapartitioning

-LSB-

-RSB-

Function-Partitioning

uses

specialized

data

structures

-LRB-

e.g.

distributed

data

structures

-RRB-

and

algorithms

which

may

be

different

from

their

sequential

counterparts

Data-Partitioning

techniques

divide

the

data

among

different

processors

and

independently

execute

the

sequential

algorithm

on

each

processor

Data-Partitioning

in

turn

is

achieved

by

declustering

-LSB-

-RSB-

the

spatial

data

If

the

static

declustering

methods

fail

to

equally

distribute

the

load

among

different

processors

the

load-balance

may

be

improved

by

redistributing

parts

of

the

data

to

idle

processors

using

dynamic

load-balancing

-LRB-

DLB

-RRB-

techniques

In

this

paper

we

focus

on

parallelizing

a

range-query

operation

for

GIS

data

using

the

data-partitioning

approach

Application

Domain

Real-Time

Terrain

Visualization

A

real-time

terrain-visualization

system

is

an

environment

that

lets

users

navigate

and

interact

with

a

three-dimensional

computer

generated

geographic

environment

in

real-time

like

other

virtual

environments

-LSB-

-RSB-

visualization

systems

-LSB-

-RSB-

and

distributed

interactive

simulation

systems

-LSB-

-RSB-

This

type

of

system

has

three

major

components

interaction

graphics

and

GIS

Figure

shows

the

different

components

of

a

terrain

visualization

system

for

a

typical

flight

simulator

The

HPGIS

component

of

the

system

contains

a

secondary

storage

unit

for

storing

the

entire

geographic

database

and

a

main

memory

for

storing

the

data

related

to

the

current

location

of

the

simulator

The

graphics

engine

receives

the

spatial

data

from

the

HPGIS

component

and

transforms

these

data

into

objects

which

are

then

sent

to

the

display

unit

As

the

user

moves

over

the

terrain

the

part

of

the

map

that

is

visible

to

the

user

changes

over

time

and

the

graphics

engine

has

to

be

fed

with

the

visible

subset

of

spatial

objects

for

a

given

location

and

user

's

viewport

The

graphics

engine

transforms

the

user

's

viewport

into

a

range

query

and

sends

it

to

the

HPGIS

unit

For

example

Figure

shows

a

polygonal

map

and

a

range

query

Polygons

in

the

map

are

shown

with

dotted

lines

The

range

query

is

represented

by

the

rectangle

and

the

result

of

the

range

query

is

shown

in

solid

lines

The

HPGIS

unit

retrieves

the

visible

subset

of

spatial

data

from

the

main

memory

and

computes

their

geometric

intersection

with

the

current

viewport

of

the

user

and

sends

the

results

back

to

the

graphics

engine

The

frequency

of

this

operation

depends

on

the

speed

at

which

the

user

is

moving

over

the

terrain

For

example

in

the

terrain

visualization

of

a

flight

simulator

a

new

range

query

may

be

generated

twice

a

second

which

leaves

less

than

half

a

second

for

intersection

computation

A

typical

map

used

in

this

application

contains

tens

of

thousands

of

polygons

-LRB-

i.e.

millions

of

edges

-RRB-

and

the

range-query

size

can

be

of

the

total

map

This

requires

millions

of

intersection-point

computations

in

less

than

half

a

second

In

order

to

meet

such

response-time

constraints

HPGIS

often

caches

a

subset

of

spatial

data

in

main

memory

The

main-memory

database

may

in

turn

query

the

secondary-storage

database

to

get

a

subset

of

data

to

be

cached

The

frequency

of

this

operation

should

be

very

small

for

the

caching

to

be

effective

SecondaryStorage

MainMemory

GraphicsAnalysis

Display

feed

back

view

range-query

ofPolygons

secondary

storagerange-querySet

ofPolygons

TerrainEngine

High

Performance

GIS

Component

Database

TerrainDatabase

Figure

Components

of

the

Terrain-Visualization

System

Figure

A

sample

polygonal

map

and

a

range-query

Problem

Formulation

The

range-query

problem

for

the

GIS

can

be

stated

as

follows

Given

a

rectangular

query

box

B

and

a

set

SP

of

extended

spatial

objects

-LRB-

e.g

polygons

chains

of

line

segments

-RRB-

the

result

of

a

range

query

over

SP

is

given

by

the

set

fxjx

Pi

B

and

Pi

SP

g

where

gives

the

geometric

intersection

of

two

extended

objects

We

call

this

problem

the

GIS-range-query

problem

The

GIS-range-query

problem

has

three

main

components

-LRB-

i

-RRB-

Approximate

filtering

at

the

polygon

level

-LRB-

ii

-RRB-

Intersection

computations

and

-LRB-

iii

-RRB-

Polygonization

of

the

result

-LRB-

See

-LSB-

-RSB-

for

a

detailed

discussion

of

a

sequential

algorithm

-RRB-

Note

that

this

problem

is

different

from

the

traditional

range

query

where

the

objects

in

the

given

range

are

retrieved

from

secondary

memory

-LRB-

disk

-RRB-

to

main

memory

without

clipping

the

objects

but

it

is

similar

to

the

polygon-clipping

problem

-LSB-

-RSB-

in

computer

graphics

The

existing

sequential

solutions

-LSB-

-RSB-

for

the

range-query

problem

can

not

always

be

directly

used

as

a

solution

to

the

GIS-range-query

problem

due

to

the

high

performance

requirements

of

many

applications

For

example

the

limit

on

response

time

-LRB-

i.e.

half

a

second

as

shown

in

Figure

-RRB-

for

solving

the

GIS-range-query

problem

allows

the

processing

of

maps

with

no

more

than

polygons

-LRB-

or

edges

-RRB-

on

many

of

the

latest

processors

available

today

like

the

IBM

and

DEC

Alpha

-LRB-

-RRB-

processors

However

the

maps

used

in

many

HPGIS

applications

are

at

least

an

order

of

magnitude

larger

than

these

simple

maps

Hence

we

need

to

consider

parallel

processing

to

deliver

the

required

performance

In

this

paper

we

focus

on

parallelizing

the

GIS-range-query

problem

over

a

set

of

processors

to

meet

the

high

performance

requirements

imposed

by

a

typical

HPGIS

application

The

goal

of

the

parallelization

is

to

achieve

the

minimum

possible

response

time

for

a

set

of

range

queries

We

use

datapartitioning

with

declustering

and

dynamic

load-balancing

for

parallelizing

a

sequential

algorithm

to

the

GIS-range-query

problem

Figure

describes

the

steps

in

this

scheme

The

bounding

box

is

initially

broadcast

to

all

processors

Each

processor

then

executes

the

sequential

GIS-range-query

algorithm

on

the

local

set

of

polygons

After

processing

the

local

data

a

processor

checks

for

any

load

imbalances

and

seeks

more

work

from

another

processor

which

has

not

yet

finished

its

work

DLB

methods

are

used

for

transferring

the

work

between

processors

during

run-time

Staticpartition

Apprx

Filtering

ComputationApprx

Filtering

Computation

Polygonization

Of

the

result

Polygonization

Of

the

result

Intersection

Computation

Intersection

Computation

DLB

DLB

SYNC

GetNextBbox

Figure

Different

modules

of

the

parallel

formulations

Related

work

and

Our

Contributions

Declustering

and

load-balancing

are

important

issues

in

parallelization

of

the

typical

HPGIS

operations

like

range-query

and

map-overlay

operations

Several

researchers

have

used

declustering

and

loadbalancing

towards

parallelization

of

the

traditional

range-query

problems

Kamel

and

Faloutsos

-LSB-

-RSB-

used

local

load-balancing-based

data

declustering

to

maximize

the

throughput

of

range

queries

over

data-sets

consisting

of

two-dimensional

rectangles

Zhou

et

al.

-LSB-

-RSB-

describe

mapping-function-based

declustering

methods

for

parallelizing

the

grid

files

in

the

context

of

traditional

range

queries

Brunetti

et

al.

-LSB-

-RSB-

used

row-wise

division

of

two-dimensional

regular

grids

in

parallel

algorithms

for

characterizing

terrain

data

Armstrong

et

al.

-LSB-

-RSB-

used

row-wise

partitioning

of

grids

for

parallelizing

an

algorithm

to

determine

the

spatial

association

measures

for

point

data

It

has

been

shown

that

customized

declustering

techniques

based

on

space-division

mapping

functions

-LSB-

-RSB-

proximity-based

local

load-balance

-LSB-

-RSB-

and

similarity

graph-partitioning

-LSB-

-RSB-

are

needed

to

effectively

partition

spatial

data

In

the

case

of

uniformly

distributed

point

data

it

has

been

shown

that

the

static

declustering

is

often

adequate

for

achieving

a

good

load-balance

by

formal

methods

-LSB-

-RSB-

as

well

as

by

experimental

studies

-LSB-

-RSB-

However

the

effective

declustering

of

sets

of

extended

objects

has

not

received

adequate

attention

in

the

literature

In

the

case

of

extended

spatial

objects

static-declustering

methods

alone

might

not

be

enough

to

achieve

good

load-balance

In

such

a

case

both

static

partitioning

and

DLB

techniques

can

be

used

Wang

-LSB-

-RSB-

used

dynamic

allocation

of

work

at

different

levels

-LRB-

e.g

polygons

edges

-RRB-

for

map-overlay

computation

In

addition

several

dynamic

load-balancing

methods

have

been

developed

-LSB-

-RSB-

for

load-balancing

in

different

applications

Data-Partitioning

for

map-overlay

-LSB-

-RSB-

spatial-join

and

access

methods

-LSB-

-RSB-

is

not

related

to

the

work

presented

in

this

paper

Declustering

and

dynamic

load-balancing

for

extended

spatial-data

have

not

received

adequate

attention

in

the

literature

In

this

paper

we

focus

on

static

data-declustering

and

dynamic

load-balancing

methods

for

parallelizing

the

GIS-range-query

problem

over

sets

of

extended

objects

like

line-segments

and

polygons

We

provide

a

framework

for

declustering

collections

of

extended

spatial

objects

by

identifying

the

following

issues

-LRB-

i

-RRB-

the

work-load

metric

-LRB-

ii

-RRB-

the

spatial

extent

of

an

object

's

work-load

-LRB-

iii

-RRB-

the

distribution

of

the

work-load

over

the

spatial

extent

of

the

object

and

-LRB-

iv

-RRB-

the

declustering

method

In

addition

we

also

provide

a

framework

for

dynamic

load-balancing

for

GIS

operations

by

identifying

the

issues

of

-LRB-

i

-RRB-

work

transfer

methods

-LRB-

ii

-RRB-

identifying

the

donor

processor

and

-LRB-

iii

-RRB-

the

granularity

of

work

transfer

We

identify

and

experimentally

evaluate

alternatives

for

each

of

these

issues

for

the

range

query

operation

using

vector

data

for

Killeen

Texas

The

experiments

are

carried

out

on

the

Cray

which

is

a

distributed

memory

MIMD

machine

consisting

of

DEC-Alpha

-LRB-

-RRB-

processors

interconnected

by

a

torus

network

We

show

that

the

traditional

declustering

methods

-LSB-

-RSB-

for

multi-dimensional

point

data

need

significant

extensions

to

be

applicable

for

extended

spatial

data

We

also

show

that

neither

declustering

nor

dynamic

load-balancing

alone

are

sufficient

by

themselves

for

achieving

good

speedups

beyond

processors

Static

declustering

of

extended

spatial

data

is

hard

due

to

highly

non-uniform

data

distribution

as

well

as

great

variation

in

the

size

and

extent

of

spatial

data

Experiments

show

that

the

spatial-extent

and

the

work-load

metric

are

important

measures

in

developing

a

declustering

method

We

show

that

data

replication

is

often

needed

for

dynamic

load-balancing

as

the

cost

of

local

processing

is

usually

less

than

the

cost

of

data

transfer

for

extended

objects

In

addition

experimental

results

also

show

that

the

effectiveness

of

dynamic

load-balancing

techniques

can

be

further

improved

by

using

declustering

methods

to

determine

the

subsets

of

spatial

objects

to

be

transferred

during

run-time

Scope

and

Outline

of

the

Paper

Figure

shows

two

types

of

queries

First

a

query

to

retrieve

data

from

secondary

storage

to

main

memory

Second

a

query

-LRB-

-RRB-

to

retrieve

data

from

main

memory

to

the

graphics

engine

In

this

paper

we

focus

on

the

latter

type

of

range-queries

where

the

data

is

assumed

to

be

in

the

main

memory

Several

techniques

like

preprocessing

the

spatial

data

can

be

used

to

reduce

the

sequential

cost

of

the

GIS-range-query

problem

The

cost

of

the

range-query

processing

can

also

be

reduced

by

noting

that

consecutive

range-queries

may

spatially

overlap

with

the

previous

range-queries

In

this

case

the

new

range

query

can

be

considered

as

an

increment

of

the

previous

range

query

and

hence

incremental

rangequery

methods

can

be

used

to

solve

this

problem

But

this

incremental

range-query

can

be

expressed

as

a

combination

of

one

or

more

smaller

range-queries

The

GIS-range-query

problem

can

also

be

solved

using

pre-computation

of

the

results

For

this

a

fine

grid

is

laid

on

top

of

the

data

and

the

intersections

of

all

the

spatial

objects

and

the

grid

cells

are

computed

and

stored

in

the

main

memory

Since

every

range-query

will

be

some

combination

of

the

gridcells

the

intersection

results

for

each

of

the

grid-cells

which

make

up

the

range-query

can

be

retrieved

and

sent

to

the

graphics

engine

On

the

other

hand

in

the

case

of

data-partitioning

approaches

large

objects

may

be

decomposed

into

smaller

objects

to

improve

the

load-balance

among

different

processors

thus

increasing

the

efficiency

of

the

solution

But

these

two

approaches

result

in

increased

total

work

for

the

graphics

engine

as

it

has

to

process

more

objects

in

the

same

amount

of

time

The

cost

of

rendering

at

the

graphics

engine

also

increases

with

the

increased

number

of

polygons

In

addition

the

decomposition

of

objects

requires

more

memory

to

store

the

objects

On

the

other

hand

if

the

smaller

pieces

are

to

be

merged

again

into

a

single

object

after

the

range-query

operation

the

merging

will

result

in

increased

total

work

for

the

HPGIS

component

as

merging

of

the

smaller

objects

increases

the

total

work

For

example

Figure

shows

different

combinations

for

partitioning

polygonal

data

into

smaller

sets

These

combinations

can

be

grouped

into

four

types

Type

I

has

no

division

of

data

Type

II

divides

the

set

of

polygons

into

subsets

of

polygons

However

each

polygon

is

treated

as

an

atomic

unit

and

sub-division

at

the

polygon

level

is

not

allowed

In

contrast

type

III

divides

the

areas

of

individual

polygons/bounding-boxes

among

different

processors

Type

IV

schemes

divide

both

the

areas

and

the

edges

of

individual

polygons

and

the

bounding

box

The

potential

advantage

of

type

III

and

IV

schemes

over

a

type

II

scheme

is

the

possibility

of

better

load-balance

and

less

processor

idling

resulting

in

reduced

parallel

computation

time

-LSB-

-RSB-

However

note

that

types

III

and

IV

schemes

result

either

in

increased

total

work

or

in

increased

work

for

the

polygonization

of

the

result

Options

for

Dividing

Bounding

Box

No

Division

subsets

subsets

of

NoDivisionintoDividedivideintoEdges

I

IV-d

III-aIII-dIV-eIV-c

small

boxes

of

polygons

small

polygons

subsetsof

edges

II

IV-a

III-cIII-b

IV-bIV-f

Options

for

Dividing

the

Polygon

Data

Figure

Alternatives

for

Polygon/Bounding-Box

division

among

processors

Let

Tcomm

be

the

response-time

overhead

due

to

additional

communication

cost

or

the

increased

cost

for

the

polygonization

of

the

resulting

polygons

for

type

III

and

IV

schemes

The

gain

in

parallelcomputation

time

due

to

improved

load-balancing

is

bounded

by

the

difference

between

the

ideal

value

-LRB-

Tseq

=P

-RRB-

and

the

actual

TP

value

achieved

by

a

type

II

scheme

The

net

gain

in

response

time

by

any

type

III

or

IV

scheme

over

a

type

II

scheme

is

bounded

by

-LSB-

TP

-LRB-

scheme

II

-RRB-

Tseq

P

Tcomm

-RSB-

This

gain

is

positive

only

when

polygon-size

distributions

are

extremely

skewed

leading

to

high

load

imbalances

for

type

II

schemes

Even

though

these

techniques

can

potentially

increase

the

load-balance

and

response

time

for

the

GIS-range-query

we

do

not

consider

these

techniques

in

this

paper

In

the

rest

of

this

paper

we

focus

only

on

type

II

schemes

The

rest

of

the

paper

is

organized

as

follows

In

Section

we

discuss

the

issues

in

declustering

extended

spatial

data

In

Section

we

present

the

experimental

results

for

different

issues

in

declustering

spatial

data

In

Section

we

discuss

the

dynamic

load-balancing

issues

in

GIS

In

Section

we

present

the

experimental

results

for

DLB

issues

in

GIS

Finally

in

Section

we

present

the

conclusions

and

future

work

Declustering

Spatial

Data

The

goal

of

a

declustering

method

is

to

partition

the

data

so

that

each

partition

imposes

exactly

the

same

load

for

any

range

query

Intuitively

the

polygons

close

to

each

other

should

be

scattered

among

different

processors

such

that

for

each

range

query

every

processor

has

an

equal

amount

of

work

For

example

consider

the

raster

representation

of

a

set

S

of

spatial

vector

objects

in

a

plane

Suppose

that

each

point

of

the

raster

representation

is

associated

with

the

work-load

of

the

vector

objects

that

pass

through

that

point

Now

consider

the

distribution

D

of

this

work-load

associated

with

each

point

For

example

the

distribution

might

look

like

the

surface

shown

in

Figure

Now

consider

another

distribution

DP

which

is

the

scaled

down

version

of

the

distribution

D

by

a

factor

of

P

Suppose

that

the

set

S

is

declustered

into

P

subsets

so

that

each

subset

is

assigned

to

a

different

processor

Then

if

each

of

the

P

subsets

has

the

work-load

distribution

DP

the

work-load

imposed

for

a

query

will

be

equal

at

all

the

processors

Hence

this

data-partitioning

achieves

the

goal

of

optimally

declustering

S

into

P

subsets

Distribution

D

Distributions

Figure

An

example

of

work-load

distributions

for

P

Optimal

declustering

is

not

achievable

in

all

cases

due

to

the

non-uniform

distribution

and

variable

sizes

of

polygons

-LRB-

or

chains

of

line-segments

-RRB-

In

addition

the

load

imposed

by

each

polygon

-LRB-

or

chain

-RRB-

for

a

query

operation

is

a

function

of

the

size

and

location

of

the

query

Since

the

location

of

the

query

is

not

known

a

priori

it

is

hard

to

develop

a

strategy

that

will

be

optimal

for

all

queries

In

general

there

exists

no

algorithm

which

can

achieve

the

ideal

declustering

for

all

range-queries

for

more

than

processors

-LSB-

-RSB-

Even

in

cases

where

it

is

possible

to

achieve

the

ideal

declustering

it

is

hard

to

determine

this

partitioning

since

the

declustering

problem

is

NP-Hard

as

shown

below

Definition

The

optimization

version

of

the

GIS-Declustering

problem

Given

a

set

S

of

extendedobjects

P

processors

and

a

set

Q

Qng

of

n

range-queries

partition

the

set

S

among

P

processors

such

that

the

load

at

each

processor

is

balanced

for

all

Qi

Q

The

load

of

an

object

x

S

for

a

given

range-query

Qi

is

given

by

a

function

f

iQ

S

Z

where

Z

is

the

set

of

non-negative

integers

Definition

The

decision

version

of

the

GIS-Declustering

problem

Given

a

set

S

of

extendedobjects

P

processors

and

a

set

Q

Qng

of

n

queries

is

there

a

partition

of

set

S

into

P

subsets

SP

such

that

-LSB-

-LSB-

-LSB-

SP

S

Si

Si

OE

for

i

-LRB-

P

-RRB-

and

X

f

iQ

-LRB-

a

-RRB-

X

f

iQ

-LRB-

a

-RRB-

X

f

iQ

-LRB-

a

-RRB-

for

i

n

-LRB-

-RRB-

Theorem

The

GIS-Declustering

problem

is

NP-Hard

Proof

We

reduce

the

PARTITION

problem

-LSB-

-RSB-

to

the

GIS-Declustering

problem

An

instance

of

the

PARTITION

problem

is

defined

as

follows

Given

a

finite

set

A

and

a

size

s

-LRB-

a

-RRB-

Z

for

each

a

A

is

there

a

subset

A

such

that

X

s

-LRB-

a

-RRB-

X

s

-LRB-

a

-RRB-

-LRB-

-RRB-

This

problem

can

be

transformed

in

polynomial

time

to

an

instance

of

the

decision

version

of

the

GIS-Declustering

problem

with

P

query

set

Q

and

-LRB-

a

-RRB-

s

-LRB-

a

-RRB-

Hence

we

conclude

that

the

GIS-Declustering

problem

is

NP-Hard

Since

the

declustering

problem

is

NP-Hard

heuristic

methods

are

used

in

practice

for

declustering

extended

spatial

data

In

this

section

we

identify

the

issues

for

declustering

sets

of

extended

spatial

objects

and

develop

heuristic

methods

for

declustering

maps

with

extended

objects

Issues

in

Declustering

Spatial-Data

There

are

three

major

issues

in

declustering

sets

of

extended

spatial

objects

the

work-load

metric

the

spatial-extent

of

work-load

and

the

load-density

over

the

spatial-extent

Work-Load

Metric

The

load

imposed

by

a

spatial

object

is

a

function

of

the

shape

and

extent

of

the

object

In

the

case

of

point

data

this

load

may

be

uniform

i.e.

the

same

for

all

spatial

points

In

the

case

of

chains

of

line-segments

this

load

may

be

a

function

of

the

number

of

edges

and

in

the

case

of

a

polygon

the

load

may

be

a

function

of

the

number

of

edges

and/or

the

area

of

the

polygon

For

example

as

the

number

of

edges

increases

the

work

for

each

range

query

also

increases

due

to

the

increase

in

intersection

point

computations

or

the

increase

in

size

of

the

result

Similarly

an

increase

in

the

area

of

a

polygon

-LRB-

with

the

number

of

edges

being

fixed

-RRB-

results

in

more

range

queries

intersecting

the

polygon

So

in

the

case

of

an

extended

spatial

object

A

either

the

area

the

number

of

edges

or

the

actual

intersection

points

with

the

query

boundary

can

be

used

in

estimating

the

work-load

-LRB-

denoted

by

load

-LRB-

A

-RRB-

-RRB-

for

A

We

note

that

for

extended

spatial

data

there

is

no

accurate

method

of

estimating

the

amount

of

work

other

than

to

actually

solve

the

problem

The

number

of

edges/points

in

a

spatial

object

may

not

accurately

reflect

the

amount

of

work

required

for

that

object

for

a

particular

range

query

and

we

can

only

get

a

rough

estimate

of

the

work

by

the

work-load

metric

The

Spatial

Extent

of

the

Work-Load

The

spatial

extent

of

the

work-load

is

defined

as

the

region

R

-LRB-

A

-RRB-

of

space

affected

by

an

object

A

i.e.

if

a

query

Q

overlaps

with

R

-LRB-

A

-RRB-

then

the

work

required

to

process

Q

is

influenced

by

the

object

A.

Usually

R

-LRB-

A

-RRB-

depends

on

the

space

occupied

by

object

A.

However

it

is

often

expensive

to

use

the

exact

geometry

of

each

spatial

object

in

estimating

the

extent

of

that

object

Thus

approximate

geometries

are

considered

in

estimating

the

spatial

extent

Spatial-Extent

R

-LRB-

A

-RRB-

is

often

approximated

with

R

-LRB-

A

-RRB-

point

-LRB-

A

-RRB-

if

A

is

approximated

with

a

point

bb

-LRB-

A

-RRB-

if

A

is

approximated

with

a

box

BBn

-LRB-

A

-RRB-

bbng

if

A

is

approximated

with

n

boxes

For

example

when

bb

-LRB-

A

-RRB-

is

the

smallest

rectangular

box

enclosing

the

object

A

and

is

represented

by

its

two

corners

-LRB-

xmin

ymin

-RRB-

and

-LRB-

xmax

ymax

-RRB-

a

point

function

may

be

defined

as

point

-LRB-

A

-RRB-

-LRB-

xmin

xmax

ymin

ymax

-RRB-

-LRB-

-RRB-

Figure

shows

some

example

polygons

with

different

approximations

of

the

extent

of

the

work-load

The

figure

also

shows

a

sample

range

query

in

dotted

lines

Polygon

A

is

approximated

with

a

point

which

is

shown

in

the

middle

of

the

polygon

The

main

drawback

of

the

point

approximation

is

that

even

though

the

object

is

in

the

region

of

interest

-LRB-

e.g

-RRB-

it

might

be

still

be

considered

to

be

outside

if

the

point

lies

outside

that

region

as

shown

in

the

case

of

polygon

A.

Alternatively

the

bounding

box

approximation

can

be

used

as

shown

in

Figure

for

polygons

A

B

C

and

E

The

drawback

with

this

approximation

is

that

even

though

the

polygon

is

not

in

the

region

of

interest

the

bounding

box

might

still

be

in

the

region

of

interest

as

shown

for

polygon

E.

Alternatively

multiple

bounding

boxes

may

be

used

to

represent

a

polygon

as

shown

for

polygon

D

But

note

that

even

though

a

greater

number

of

bounding

boxes

gives

a

better

representation

of

the

spatial

extent

of

the

work

it

is

also

more

expensive

to

construct

this

kind

of

representation

A

C

E

D

B

query

point

-LRB-

A

-RRB-

bb

-LRB-

A

-RRB-

bb

-LRB-

B

-RRB-

-LRB-

D

-RRB-

-LRB-

D

-RRB-

Figure

Examples

of

approximations

for

the

extent

of

the

work-load

Load

Density

for

Spatial

Extent

In

the

case

of

extended

objects

the

distribution

-LRB-

or

density

-RRB-

of

the

work-load

over

their

spatial

extent

affects

the

declustering

decisions

If

it

is

expensive

to

determine

the

actual

work-load

distribution

an

approximate

distribution

or

a

uniform

distribution

may

be

used

instead

of

the

actual

distribution

An

approximate

distribution

of

the

work

can

be

determined

by

considering

multiple

bounding

boxes

or

by

dividing

the

region

into

small

cells

and

counting

the

work

in

each

of

the

cells

For

example

in

the

case

of

polygon

B

shown

in

Figure

the

clipped

load

-LRB-

denoted

by

clipped

load

-LRB-

B

-RRB-

-RRB-

corresponding

to

query

-LRB-

shown

by

the

dotted

line

-RRB-

can

be

estimated

in

different

ways

If

we

assume

that

the

work-load

distribution

of

the

polygon

is

uniform

in

the

bounding

box

of

polygon

B

then

we

can

compute

the

clipped

load

as

clipped

load

-LRB-

B

-RRB-

area

-LRB-

intersection

-LRB-

bb

-LRB-

B

-RRB-

-RRB-

-RRB-

area

-LRB-

bb

-LRB-

B

-RRB-

-RRB-

load

-LRB-

B

-RRB-

-LRB-

-RRB-

Note

that

this

work

estimate

may

be

inaccurate

in

a

few

cases

For

example

an

edge-based

work-load

metric

coupled

with

an

assumed

uniform

work-load

distribution

overestimates

the

work

required

for

polygon

C

for

range-query

and

an

area-based

work-load

metric

coupled

with

a

uniform

work-load

distribution

overestimates

the

work

required

for

polygon

E

for

range-query

Declustering

Methods

Since

the

declustering

problem

is

NP-Hard

heuristic

methods

are

used

for

declustering

spatial

data

Here

we

describe

three

heuristic

methods

based

on

the

ideas

of

space-partitioning

with

mappingfunctions

local

load-balance

and

similarity-graph

In

addition

we

propose

a

new

populationdistribution-based

declustering

method

for

declustering

spatial

data

For

simplicity

we

describe

these

methods

for

polygon

data

but

they

can

be

applied

to

other

extended

spatial

objects

as

well

Space-Partitioning

Mapping

Functions

Space-Partitioning

mapping-function-based

methods

provides

a

mapping

function

from

the

domain

of

data

items

to

the

set

of

processor

IDs

For

example

a

mapping

function

can

be

based

on

the

Hilbert

Space-filling

curve

-LSB-

-RSB-

-LRB-

See

-LSB-

-RSB-

for

a

survey

of

other

mapping

functions

-RRB-

The

Hilbert

curve

gives

a

total

ordering

of

points

in

space

Polygons

can

be

declustered

using

the

Hilbert

method

as

follows

Let

Ls

be

the

set

of

input

objects

and

let

Lp

be

the

ordered

list

of

polygons

corresponding

to

the

Hilbert

order

for

the

set

fpoint

-LRB-

Ai

-RRB-

jAi

Lsg

and

let

n

be

the

number

of

polygons

in

the

list

The

polygons

in

the

list

are

then

assigned

to

each

processor

in

a

cyclic

manner

That

is

the

polygons

in

the

list

Lp

with

indices

i

i

P

i

n

P

are

assigned

to

the

ith

processor

Local

Load-Balance

-LRB-

LLB

-RRB-

Method

Local

load-balancing

methods

-LSB-

-RSB-

consider

a

sample

window

of

space

-LRB-

based

on

the

frequent

rangequeries

-RRB-

and

try

to

equally

distribute

the

load

in

that

window

to

all

the

processors

The

local

loadbalance

method

with

a

parameter

window

W

has

the

following

steps

-LRB-

i

-RRB-

From

the

set

of

polygons

M

assign

the

first

P

polygons

to

P

processors

-LRB-

ii

-RRB-

For

the

next

polygon

in

the

list

consider

the

load

corresponding

to

window

W

at

each

processor

and

select

the

processor

with

the

minimum

load

and

-LRB-

iii

-RRB-

Assign

the

next

polygon

to

that

processor

Repeat

the

steps

-LRB-

ii

-RRB-

to

-LRB-

iii

-RRB-

until

all

the

polygons

have

been

assigned

At

step

-LRB-

ii

-RRB-

of

the

above

method

a

processor

with

the

minimum

load

is

selected

as

follows

Let

weight

-LRB-

W

i

-RRB-

P

clipped

load

-LRB-

pj

W

-RRB-

such

that

pj

is

at

processor

i

Then

select

processor

k

such

that

weight

-LRB-

W

i

-RRB-

for

i

-LRB-

P

-RRB-

is

minimum

at

i

k.

Similarity-Graph

Method

The

similarity-graph

declustering

method

-LSB-

-RSB-

has

been

shown

to

outperform

other

methods

for

declustering

non-uniformly

distributed

data

This

is

a

heuristic

method

based

on

the

max-cut

graph-partitioning

of

a

weighted

similarity-graph

-LRB-

WSG

-RRB-

where

WSG

models

the

data

and

some

properties

of

the

queries

As

in

the

case

of

the

LLB

method

a

rectangular

window

W

can

be

used

as

a

sample

query

for

efficiency

The

WSG

is

then

constructed

w.r.t.

this

window

W

by

assigning

clipped

load

-LRB-

v

W

-RRB-

as

t

-LRB-

v

-RRB-

for

each

object

v

in

the

input

In

our

experimental

study

we

use

the

incremental

max-cut

partitioning

-LSB-

-RSB-

approach

for

declustering

the

spatial

data

See

Appendix

A

for

details

of

the

similarity-graph

declustering

method

and

how

it

can

be

applied

to

extended

spatial-data

Population

Distribution-Based

-LRB-

PDB

-RRB-

Method

The

goal

of

a

population-distribution-based

declustering

method

is

to

achieve

identical

load

distribution

on

each

partition

of

the

data

We

discuss

an

example

of

the

population-distribution-based

method

for

declustering

polygonal

data

The

basic

idea

behind

this

method

is

to

partition

the

data

sets

into

groups

of

similar

work-load

distribution

over

the

entire

space

as

shown

in

Figure

The

work-load

distributions

in

each

group

over

the

entire

space

are

compared

for

allocating

a

new

object

to

a

group

The

new

object

is

allocated

to

a

group

such

that

the

statistical

difference

between

the

different

groups

is

minimal

However

tracking

and

comparing

two

distributions

for

statistical

differences

is

expensive

An

economical

but

less

accurate

method

is

to

use

an

approximate

distribution

instead

of

the

actual

work-load

distribution

We

use

a

pair

of

discrete

distributions

to

approximate

the

actual

distribution

This

method

uses

the

actual

intersection

points

of

polygons

with

a

grid

consisting

of

vertical

and

horizontal

scan-lines

imposed

on

top

of

the

polygonal

data

as

shown

in

Figure

Assume

that

there

are

n

scan-lines

parallel

to

the

x-axis

and

m

scan-lines

parallel

to

the

y-axis

Then

let

f

-LRB-

xi

-RRB-

i

m

be

the

number

of

intersection

points

of

the

line

x

xi

with

all

the

polygons

in

the

input

Similarly

let

g

-LRB-

yj

-RRB-

j

n

be

the

number

of

intersection

points

with

the

line

y

yj

Without

loss

of

generality

let

the

polygons

in

the

input

be

pz

To

distribute

these

polygons

among

the

processors

allocate

the

first

P

polygons

P

processors

such

that

polygon

pi

is

assigned

to

the

ith

processor

For

the

next

polygon

pw

determine

the

distribution

of

intersection

points

for

all

the

assigned

polygons

plus

the

current

polygon

and

scale

down

the

distribution

by

P

Let

this

distribution

be

the

base-distribution

That

is

base

distributions

fw

-LRB-

xi

-RRB-

=P

and

gw

-LRB-

yi

-RRB-

=P

are

similar

to

f

-LRB-

xi

-RRB-

and

g

-LRB-

yi

-RRB-

f

-LRB-

-RRB-

f

-LRB-

-RRB-

f

-LRB-

-RRB-

f

-LRB-

-RRB-

f

-LRB-

-RRB-

f

-LRB-

-RRB-

f

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

g

-LRB-

-RRB-

Figure

Distribution-Based

Method

but

the

base

distributions

contain

the

intersection

points

of

polygons

pw

only

Then

consider

P

different

assignments

of

polygon

pw

to

P

processors

and

estimate

the

total

population

mismatch

due

to

each

assignment

The

total

population

mismatch

of

an

assignment

is

estimated

as

the

sum

of

the

squared

differences

of

the

distributions

at

each

processor

with

the

base-distribution

Then

select

the

processor

corresponding

to

the

minimum

population

mismatch

as

the

processor

for

assigning

the

current

polygon

The

minimization

function

for

assigning

polygon

pw

is

given

as

min

l

P

P

X

i

n

X

k

P

fw

-LRB-

xk

-RRB-

fi

-LRB-

xk

-RRB-

m

X

j

P

gw

-LRB-

yj

-RRB-

gi

-LRB-

yj

-RRB-

A

-LRB-

-RRB-

where

the

current

polygon

pw

is

temporarily

assigned

to

the

lth

processor

in

each

iteration

of

the

minimization

function

and

fi

and

gi

are

the

distribution

functions

-LRB-

corresponding

to

f

and

g

respectively

-RRB-

at

the

ith

processor

Note

that

fi

and

gi

contain

the

intersection

points

of

only

those

polygons

which

are

assigned

to

the

ith

processor

Complexity

of

the

PDB

Method

for

Allocating

Polygon

The

innermost

sum

of

Equation

takes

-LRB-

n

m

-RRB-

time

and

since

this

sum

is

computed

for

each

processor

it

takes

-LRB-

P

-LRB-

n

m

-RRB-

-RRB-

time

for

the

double

summation

Since

there

are

P

iterations

of

this

double

sum

-LRB-

i.e.

P

iterations

of

the

minimization

function

-RRB-

it

takes

a

total

of

-LRB-

P

-LRB-

n

m

-RRB-

-RRB-

time

for

a

brute

force

implementation

of

this

method

But

note

that

between

two

iterations

of

the

minimization

function

only

four

terms

of

the

innermost

summation

change

at

each

processor

Hence

we

need

not

compute

the

entire

sum

for

each

iteration

of

the

minimization

function

as

we

can

reuse

the

rest

of

the

terms

from

the

previous

iteration

Hence

after

the

first

iteration

of

the

minimization

function

each

further

iteration

takes

a

constant

amount

of

time

This

reduces

the

overall

complexity

of

the

PDB

method

to

-LRB-

P

-LRB-

n

m

-RRB-

-RRB-

Experimental

Evaluation

of

Declustering

Issues

We

compare

the

performance

of

different

alternatives

for

each

of

the

issues

in

declustering

extended

spatial

objects

for

a

range

of

map

sizes

and

for

different

number

of

processors

via

experiments

carried

out

on

the

Cray

parallel

computer

We

use

spatial

vector

data

for

Killeen

Texas

for

this

experimental

study

This

data

is

divided

into

seven

themes

representing

the

attributes

slope

vegetation

surface

material

hydrology

etc.

We

used

the

slope

attribute-data

map

with

polygons

and

edges

as

a

base

map

in

our

experiments

-LRB-

this

is

denoted

by

map

-RRB-

For

studying

the

effect

of

increased

map

size

we

derived

new

maps

from

this

base

map

using

the

following

method

Scaling

down

the

base

map

along

the

x-axis

by

two

and

combining

two

such

scaled-down

maps

by

translating

one

of

the

scaled-down

maps

along

the

x-axis

This

results

in

a

map

of

polygons

with

edges

-LRB-

map

-RRB-

A

similar

technique

is

used

by

alternately

scaling

down

along

the

y-axis

and

the

x-axis

to

get

maps

of

different

sizes

We

also

use

the

chain

data

from

Fort

Sill

which

has

creeks

with

edges

as

shown

in

Figure

Table

shows

the

details

of

the

maps

and

the

range

queries

Figure

Creek

data

map

with

a

sample

range

query

Table

Maps

and

range

queries

used

in

our

experiments

Map

#Objects

#edges

range-query

size

number

polygons

polygons

polygons

polygons

Creek

chains

Experimental

Methodology

The

issues

in

declustering

are

studied

by

comparing

the

performance

of

different

methods

for

a

set

of

range

queries

For

this

a

sequence

of

range

queries

is

constructed

such

that

the

sequence

of

the

center

points

of

the

range

query

represents

a

random

walk

on

the

data

set

Post-processing

is

done

on

this

sequence

to

ensure

that

all

range

queries

are

unique

and

that

the

range-query

lies

completely

within

the

map

The

size

of

each

range

query

is

approximately

of

the

total

area

of

the

map

In

all

our

measurements

we

obtain

the

run

time

of

the

program

for

each

of

the

queries

and

report

the

observed

mean

of

these

values

Figure

shows

our

experimental

methodology

The

number

of

different

options

we

tried

for

each

parameter

is

shown

in

parentheses

and

the

number

of

possible

combinations

after

each

module

is

also

shown

in

the

figure

We

restrict

our

experiments

to

P

and

due

to

the

memory

limitation

Individual

nodes

on

Cray

have

only

MBytes

of

main

memory

limiting

the

size

of

the

map

-LRB-

-RRB-

for

which

sequential

run-time

can

be

measured

directly

This

map

-LRB-

-RRB-

does

not

have

adequate

work

for

each

processor

beyond

P

as

is

evident

from

the

absolute

run-times

-LRB-

ss

sec

-RRB-

shown

in

Tables

and

generator

work-load

H

P

G

I

S

spatial-extent

-LRB-

b

box

point

-RRB-

Map

Generator

size

of

sample

window

-LRB-

-RRB-

load-density

-LRB-

uniform

apprx

-RRB-

range

queries

#of

range-queries

desired

size

of

range-query

Decluster

-LRB-

llb

sim

PDB

-RRB-

work-load

metric

-LRB-

#edges

area

-RRB-

P

-LRB-

-RRB-

size

Map

-LRB-

X

X

Creek

-RRB-

Base

map

declustering-method

options

measurements

Analysis

Data

collection

maps

Figure

Experimental

Method

for

Evaluating

Declustering

Methods

In

our

experiments

we

only

measure

and

analyze

the

cost

per

range-query

and

exclude

any

preprocessing

cost

This

preprocessing

cost

includes

the

cost

of

loading

the

data

into

main

memory

and

the

cost

of

declustering

the

data

among

different

processors

Note

that

this

preprocessing

cost

is

paid

only

once

for

each

data

set

that

corresponds

to

the

current

window

of

interest

As

the

query

range

moves

out

of

the

current

window

new

data

is

fetched

from

the

disk

discarding

data

for

the

old

window

Since

the

next

location

of

the

window

can

often

be

predetermined

preprocessing

the

new

data

need

not

affect

the

performance

of

the

rest

of

the

system

Moreover

once

a

new

data

set

is

loaded

into

the

main

memory

it

would

be

active

for

several

minutes

before

the

window

has

move

out

of

the

current

range

Thus

this

would

leave

several

minutes

for

preprocessing

the

next

data

set

Hence

in

this

study

we

are

only

interested

in

measuring

the

performance

of

our

algorithm

in

terms

of

the

variable

cost

per

range

query

for

the

preprocessed

data

Experimental

Results

We

conduct

experiments

to

study

alternatives

for

each

of

the

following

issues

the

work-load

metric

the

spatial

extent

of

the

work-load

and

the

load

density

over

the

spatial

extent

In

addition

we

compare

the

different

declustering

methods

Local

Load-Balance

Similarity-Graph

and

PDB

In

these

experiments

the

data

is

initially

distributed

among

different

processors

A

processor

acts

as

the

leader

processor

and

is

responsible

for

broadcasting

each

range

query

to

the

rest

of

the

processors

After

receiving

the

range-query

information

each

processor

works

only

on

its

local

data

until

all

the

local

data

is

exhausted

After

the

local

data

is

processed

the

processor

waits

for

the

next

range

query

to

be

processed

The

lead

processor

waits

for

all

the

processors

to

finish

the

work

before

broadcasting

the

next

range

query

Note

that

the

only

communication

required

for

each

bounding

box

is

a

broadcast

of

the

parameters

of

the

range

query

Comparison

of

Alternatives

for

Work-Load

Metrics

We

compare

the

area

and

the

number

of

edges

as

alternatives

for

the

work-load

metric

in

the

case

of

polygonal

data

The

spatial

extent

of

the

work-load

is

based

on

the

bounding-box

approximation

and

the

load

density

over

the

spatial

extent

is

assumed

to

be

uniform

Thus

the

clipped

load

-LRB-

polygonP

windowW

-RRB-

is

estimated

using

Equation

We

used

the

LLB

method

with

a

sample

window

of

as

the

declustering

metric

The

number

of

processors

P

varies

from

to

and

the

map

is

used

as

the

data

set

The

results

of

this

experiment

are

shown

in

Figure

-LRB-

a

-RRB-

The

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

average

speedups

for

range

queries

The

main

trends

observed

from

this

graph

are

-LRB-

i

-RRB-

Number

of

edges

as

the

work-load

metric

results

in

better

speedups

and

hence

appears

to

be

a

more

accurate

work-load

metric

for

P

-LRB-

ii

-RRB-

For

small

P

-LRB-

P

-RRB-

the

difference

between

the

two

work-load

metrics

is

negligible

Comparison

of

Alternatives

for

the

Spatial-Extent

of

the

Work-Load

We

compare

point

and

bounding-box

approximators

as

alternatives

for

the

spatial-extent

of

the

workload

in

the

case

of

polygonal

data

The

work-load

metric

is

fixed

to

be

the

number

of

edges

and

the

load

density

over

spatial

extent

is

assumed

to

be

uniform

We

used

the

LLB

method

with

a

sample

window

Speedup

Number

of

Processors

Speedup

Number

of

Processors

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Speedups

for

LLB

method

for

map

of

as

the

declustering

method

The

number

of

processors

P

ranges

from

to

and

map

is

used

as

the

data

set

The

results

of

this

experiment

are

shown

in

Figure

-LRB-

b

-RRB-

The

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

average

speedups

for

range-queries

The

main

trends

observed

from

this

graph

are

-LRB-

i

-RRB-

The

bounding-box

approximator

for

the

spatial

extent

results

in

better

speedups

and

hence

appears

to

be

a

more

accurate

estimator

for

P

-LRB-

ii

-RRB-

For

small

P

-LRB-

P

-RRB-

the

difference

between

the

two

estimators

is

negligible

Comparison

of

Different

Declustering

Methods

We

compare

the

performance

of

different

declustering

methods

Hilbert

LLB

similarity-graph

and

PDB

In

addition

we

compare

the

effect

of

the

size

of

the

sample

window

on

the

performance

of

similarity-graph

and

LLB

methods

For

simplicity

the

work-load

metric

is

fixed

to

be

the

number

of

edges

and

the

spatial

extent

is

assumed

to

be

a

point

The

load-density

over

the

spatial

extent

is

assumed

to

be

uniform

in

the

case

of

LLB

and

similarity-graph

methods

Figure

gives

the

results

showing

the

effect

of

sample

window

size

for

LLB

and

similarity-graph

methods

The

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

average

speedups

for

range-queries

In

Figure

-LRB-

-RRB-

refers

to

the

LLB

-LRB-

similarity-graph

-RRB-

method

with

a

sample

window

which

is

of

the

total

area

of

the

map

Similar

notation

is

used

for

a

window

for

both

methods

The

main

trends

observed

from

these

graphs

are

-LRB-

i

-RRB-

Increased

window

sizes

gives

increasing

speedups

-LRB-

ii

-RRB-

For

the

LLB

method

the

increase

in

speedup

from

a

window

to

a

window

is

negligible

Figures

and

show

a

comparison

of

different

declustering

methods

for

polygon

and

chain

data

respectively

In

these

figures

the

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

speedup

speedup

number

of

proceccors

Speedup

number

of

proceccors

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Speedups

for

LLB

and

Similarity-Graph

methods

for

different

window

sizes

Speedups

for

maps

and

are

given

in

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

respectively

value

The

main

trends

observed

from

these

graphs

are

-LRB-

i

-RRB-

Bigger

maps

lead

to

better

speedups

for

most

schemes

probably

due

to

the

improved

load-balance

-LRB-

ii

-RRB-

Similarity-Graph

and

PDB

methods

give

the

best

speedups

among

the

different

methods

-LRB-

iii

-RRB-

Speedups

are

better

for

the

chain

data

than

for

the

polygon

data

This

may

be

due

to

less

variance

in

work-loads

for

line

data

when

compared

to

the

polygon

data

-LRB-

iv

-RRB-

Mapping-Function-Based

methods

like

Hilbert

provide

inferior

speedups

beyond

processors

-LRB-

v

-RRB-

Even

the

best

declustering

method

does

not

provide

good

speedups

for

more

than

processors

for

the

maps

used

in

our

experiments

Comparison

of

Static

Load-Balancing

The

effectiveness

of

declustering

methods

in

achieving

load-balance

is

shown

in

Table

The

data

shown

in

Table

is

represented

as

Mean

SD

for

the

range

queries

used

in

our

experiment

The

column

Avg

Static

gives

the

average

static

execution

time

over

processors

and

range

queries

The

column

Max

Static

gives

the

maximum

static

execution

time

over

processors

averaged

over

range

queries

In

this

experiment

we

observe

that

the

static

declustering

alone

does

not

achieve

a

good

load-balance

and

that

the

static

methods

need

to

be

augmented

with

dynamic

load-balancing

Table

Performance

Evaluation

of

SLB

for

P

and

Map

Method

Avg

Static

Max

Static

Speedup

SIM

PDB

LLB

speedup

number

of

proceccors

hilbert

PDB

Speedup

number

of

proceccors

hilbert

PDB

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Speedups

for

different

static-declustering

methods

Speedups

for

maps

and

are

given

in

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

respectively

Dynamic

Load-Balancing

-LRB-

DLB

-RRB-

Techniques

If

static

declustering

methods

fail

to

equally

distribute

the

load

among

different

processors

the

loadbalance

may

be

improved

by

transferring

some

spatial

objects

to

idle

processors

using

dynamic

loadbalancing

techniques

DLB

Issues

in

GIS

A

typical

dynamic

load-balancing

technique

addresses

three

issues

-LRB-

i

-RRB-

what

methods

are

good

for

transferring

work

-LRB-

spatial

objects

-RRB-

between

two

processors

-LRB-

ii

-RRB-

how

much

more

work

should

an

idle

processor

fetch

and

-LRB-

iii

-RRB-

which

processor

should

an

idle

processor

ask

for

more

work

Methods

for

Transferring

the

Work

Extended

spatial

objects

are

large

-LRB-

e.g.

edges

on

average

in

maps

of

Killeen

Texas

-RRB-

in

size

and

require

special

data

structures

for

solving

the

range-query

problem

Hence

sometimes

it

may

be

more

expensive

to

send

the

complete

object

data

and

the

corresponding

data

structures

to

another

processor

than

to

solve

the

problem

locally

To

compare

the

relative

costs

of

local

processing

and

data

transfer

we

develop

cost

models

for

these

two

operations

The

cost

of

computing

the

intersection

of

range

query

Q

with

a

polygon

A

depends

on

whether

A

intersects

Q

or

not

For

example

if

A

is

completely

inside

Q

it

can

be

detected

in

a

constant

amount

of

time

On

the

other

hand

if

A

intersects

Q

the

cost

of

intersection

computation

and

polygonization

depends

on

the

number

of

intersection

points

and

the

size

of

the

result

Let

be

the

probability

that

speedup

number

of

proceccors

hilbert

sim

llb

PDB

Figure

Speedups

for

different

static-declustering

methods

for

line

data

a

polygon

intersects

at

least

one

of

the

edges

of

the

range

query

and

let

x

be

the

number

of

edges

of

A

Then

the

sequential

cost

Ts

-LRB-

A

-RRB-

is

given

by

Ts

-LRB-

A

-RRB-

-LRB-

-RRB-

tc

where

is

the

fraction

of

the

edges

of

A

that

actually

intersect

Q

and

tc

is

the

cost

of

one

step

of

computation

For

simplicity

we

assume

that

the

cost

of

the

intersection

computation

and

the

polygonization

of

the

result

is

a

linear

function

of

-LRB-

-RRB-

Constant

accounts

for

checking

if

the

bounding-box

of

a

polygon

is

completely

inside

or

completely

outside

the

query

box

Q

Since

this

test

can

be

performed

using

comparisons

Typically

for

the

data

used

in

our

experiments

Similarly

transfer

cost

Tt

-LRB-

A

-RRB-

can

be

modelled

as

a

linear

function

of

the

number

of

edges

x

as

Tt

-LRB-

A

-RRB-

ts

assuming

tc

ss

tw

Here

constant

is

included

to

account

for

the

transfer

packing

and

unpacking

of

the

datastructures

and

the

data

associated

with

A

and

typically

Assuming

ts

and

tc

ss

tw

transfer

cost

Tt

is

more

than

local

processing

cost

Ts

when

-LRB-

-RRB-

-LRB-

-RRB-

x

-LRB-

-RRB-

For

the

GIS-range-query

computation

the

value

of

x

is

small

-LRB-

close

to

-RRB-

This

implies

that

even

for

small

objects

the

transfer

cost

Tt

is

more

than

the

local

processing

cost

We

note

that

even

when

ts

this

relation

remains

the

same

This

drawback

may

be

overcome

by

selectively

duplicating

the

data

on

different

processors

and

exchanging

only

the

object

IDs

Since

the

object

ID

is

only

a

word

of

data

this

will

result

in

minimum

communication

overhead

for

each

data

transfer

Note

that

this

replication

of

data

at

different

processors

results

in

memory

overhead

Partitioning

Method

and

Granularity

of

Transfers

Granularity

of

work

division

determines

how

much

work

is

transferred

between

a

donor

processor

and

an

idle

processor

This

granularity

may

depend

on

the

size

of

the

remaining

work

the

number

of

processors

the

cost

of

the

work

transfer

and

the

accuracy

in

estimating

the

remaining

work

Several

strategies

like

self-scheduling

-LSB-

-RSB-

factoring

scheduling

-LSB-

-RSB-

and

chunk

scheduling

-LSB-

-RSB-

exist

for

determining

the

amount

of

work

to

be

transferred

Also

the

simplest

case

of

transferring

one

piece

of

work

at

a

time

is

also

considered

in

some

cases

If

communication

cost

is

negligible

or

very

small

when

compared

to

the

average

cost

of

solving

the

range-query

problem

for

a

set

of

objects

chunks

of

single

objects

may

yield

the

best

possible

loadbalance

On

the

other

hand

chunks

of

more

than

one

object

are

suitable

if

the

communication

cost

is

comparable

to

the

average

cost

of

solving

the

range-query

problem

for

a

set

of

objects

-LRB-

which

is

true

for

most

of

the

distributed

memory

systems

-RRB-

In

the

case

of

chunks

of

more

than

one

object

it

is

desirable

to

keep

a

comparable

amount

of

work

in

each

chunk

so

that

the

load-imbalance

can

be

kept

low

We

note

that

this

problem

of

dividing

work

into

chunks

of

equal

work

is

similar

to

the

static

declustering

problem

Even

though

the

traditional

DLB

methods

use

simple

methods

like

random

partitioning

round

robin

etc.

we

hypothesize

that

the

load-balance

of

any

DLB

method

can

be

improved

by

using

a

systematic

declustering

method

for

dividing

the

work

into

chunks

Since

the

declustering

operation

is

very

expensive

this

chunking

can

be

done

statically

Also

note

that

for

simplicity

we

do

not

consider

dynamically

variable

size

chunks

in

this

paper

Which

Processor

Should

an

Idle

Processor

ask

for

More

Work

Methods

to

decide

which

processors

an

idle

processor

should

ask

for

more

work

are

discussed

and

analyzed

in

-LSB-

-RSB-

These

methods

can

be

divided

into

two

categories

-LRB-

-RRB-

In

a

pool-based

method

-LRB-

PBM

-RRB-

a

fixed

processor

has

all

the

available

work

and

an

idle

processor

asks

this

fixed

processor

for

more

work

-LRB-

-RRB-

In

a

peer-based

method

all

the

work

is

initially

distributed

among

different

processors

and

an

idle

processor

selects

a

peer

processor

as

the

work

donor

using

random

polling

nearest

neighbor

and

global

round

robin

-LRB-

GRR

-RRB-

or

asynchronous

-LRB-

local

-RRB-

round

robin

-LRB-

ARR

-RRB-

Pool-Based

Method

The

structure

of

the

GIS-range-query

problem

imposes

a

limitation

on

the

amount

of

work

that

can

be

kept

in

the

shared

pool

If

all

the

work

is

initially

at

a

single

processor

the

approximate

filtering

computation

for

each

range

query

can

not

be

parallelized

As

a

result

of

this

non-parallelizable

work

the

rest

of

the

processors

have

to

wait

for

a

single

processor

to

finish

the

filtering

computation

before

fetching

the

objects

for

intersection

computation

This

processor

idling

can

be

avoided

by

initially

partitioning

the

data

into

two

parts

Static

and

Pool

Initially

the

Static

part

of

the

data

is

declustered

into

P

sets

SP

assigning

Si

to

ith

processor

for

i

P

The

Pool

part

of

the

data

is

then

assigned

to

a

leader

processor

-LRB-

processor

-RRB-

For

each

range

query

each

processor

other

than

the

leader

processor

starts

working

on

the

local

data

corresponding

to

the

Static

part

The

leader

processor

first

completes

filtering

the

Pool

and

then

starts

working

on

its

local

data

which

corresponds

to

the

Static

part

This

situation

is

shown

in

Figure

IDLE

Process

DLB

IDLE

Process

DLB

Process

Process

DLB

IDLEApprxFil

-LRB-

Pool

-RRB-

Process

DLB

ApprxFil

-LRB-

Pool

-RRB-

Process

DLB

Small

Pool

Large

Pool

IDLE

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

A

Small

pool

may

result

in

high

a

static

load

imbalance

A

Large

pool

may

result

in

processor

idling

If

any

of

the

processors

finish

work

on

their

local

data

before

the

filtering

step

for

the

Pool

part

is

finished

that

processor

would

have

to

wait

for

the

lead

processor

to

finish

the

filtering

work

with

the

Pool

part

of

the

data

as

shown

in

Figure

-LRB-

b

-RRB-

This

idling

in

turn

results

in

increased

run

time

which

decreases

the

performance

of

the

algorithm

Hence

there

should

be

enough

work

at

each

processor

so

that

the

filtering

step

for

the

Pool

can

be

completed

without

leading

to

any

processor

idling

But

on

the

other

hand

the

Static

work

at

each

processor

should

not

be

so

much

that

the

static

load-imbalance

is

too

high

A

high

static

load-imbalance

can

also

result

in

processor

idling

as

shown

in

Figure

-LRB-

a

-RRB-

Let

W

be

the

total

work

required

to

solve

the

range-query

problem

and

let

be

the

fraction

of

the

total

time

spent

in

approximate

filtering

-LRB-

i.e.

the

stage

of

range-query

computation

-RRB-

Also

let

be

the

load

imbalance

due

to

the

static

declustering

of

the

data

That

is

if

the

total

work

W

is

declustered

among

P

processors

the

maximum

time

taken

by

a

processor

is

W

-LRB-

-RRB-

=P

and

the

minimum

time

taken

is

W

-LRB-

-RRB-

=P

Further

assume

that

to

=P

is

the

overhead

incurred

due

to

the

parallelization

Here

to

is

the

increase

in

total

run-time

due

to

the

communication

overhead

and

processor

idling

Suppose

x

fraction

of

the

total

work

is

taken

as

the

Pool

data

Then

the

P

ool

should

be

large

enough

to

overcome

the

static

load

imbalance

incurred

due

to

the

Static

part

of

the

data

-LRB-

Figure

-LRB-

a

-RRB-

W

-LRB-

x

-RRB-

-LRB-

-RRB-

P

W

P

i.e.

x

-LRB-

W

-RRB-

W

-LRB-

-RRB-

-LRB-

-RRB-

Also

the

filtering

cost

xW

for

the

P

ool

should

be

less

than

the

maximum

time

corresponding

to

the

Static

work

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

WP

-LRB-

x

-RRB-

-LRB-

-RRB-

xW

i.e.

x

-LRB-

-RRB-

-LRB-

P

-RRB-

-LRB-

-RRB-

Combining

Equations

and

we

get

the

lower

and

upper

bounds

for

pool

size

as

-LRB-

W

to

-RRB-

W

-LRB-

-RRB-

x

-LRB-

-RRB-

-LRB-

P

-RRB-

-LRB-

-RRB-

Table

gives

sample

upper

and

lower

bounds

for

x

estimated

using

Equation

The

parallel

overhead

to

is

assumed

to

be

zero

and

is

assumed

to

be

Table

Estimated

lower

and

upper

bounds

for

pool

size

with

and

to

P

LowerBound

UpperBound

Peer-Based

Methods

In

peer-based

methods

data

is

divided

among

all

processors

with

no

common

pool

and

an

idle

processor

asks

another

peer-processor

for

more

work

In

this

paper

we

evaluate

the

global

round

robin

-LRB-

GRR

-RRB-

and

asynchronous

round

robin

-LRB-

ARR

-RRB-

methods

for

the

GIS-range-query

problem

See

-LSB-

-RSB-

for

a

complete

discussion

of

these

two

algorithms

In

GRR

a

single

processor

acts

as

the

scheduler

and

is

responsible

for

sending

the

ID

of

the

next

available

processor

with

work

to

a

requesting

idle

processor

The

idle

processor

then

requests

work

from

this

processor

which

has

more

work

The

main

drawback

of

such

a

scheme

is

that

the

scheduler

processor

may

become

a

bottleneck

as

the

number

of

processors

increases

In

our

experimental

study

this

bottleneck

is

not

significant

as

the

number

of

processors

is

relatively

small

i.e.

less

than

In

ARR

every

processor

maintains

a

local

target

pointer

Whenever

a

processor

runs

out

of

work

it

uses

the

target

pointer

as

the

label

of

a

donor

processor

and

sends

it

a

work

request

The

target

value

is

incremented

modulo

P

each

time

a

work

request

is

sent

If

the

processors

that

receives

the

request

has

more

work

it

sends

some

work

to

the

requesting

processor

Otherwise

the

requesting

processor

sends

another

request

to

the

next

processor

given

by

the

target

pointer

until

more

work

is

received

from

a

donor

processor

Note

that

of

these

two

methods

the

ARR

method

does

not

have

the

single

processor

bottleneck

as

in

the

case

of

GRR

But

the

ARR

method

needs

extra

work

to

check

for

termination

detection

since

there

is

no

single

source

of

information

about

the

remaining

work

for

each

range

query

Hence

the

advantage

of

this

method

over

GRR

may

be

offset

due

to

the

termination-detection

overhead

For

the

GIS-range-query

problem

the

performance

of

these

two

methods

may

be

comparable

for

up

to

processors

A

Framework

for

Parallel

Formulations

In

our

approach

we

use

declustering

at

the

static

and

dynamic

load-balancing

levels

We

present

a

general

framework

for

this

method

which

can

be

used

with

any

of

the

declustering

and

DLB

methods

discussed

so

far

This

is

a

two-phase

scheme

since

we

use

an

initial

static

declustering

of

the

data

and

use

additional

load-balancing

at

run

time

Pseudocode

for

this

general

method

is

given

in

Figure

In

the

following

discussion

let

P

be

the

number

of

processors

used

in

the

system

Initially

all

the

data

is

declustered

into

two

sets

Sa

and

Sb

The

set

Sa

is

used

as

the

static

data

Once

the

objects

from

this

set

are

allocated

to

a

processor

that

processor

alone

is

responsible

for

processing

these

objects

That

is

objects

from

this

set

are

never

transferred

between

processors

during

the

DLB

phase

Similarly

the

set

Sb

is

used

as

the

dynamic

data

Objects

from

this

set

can

be

transferred

between

processors

during

the

DLB

phase

We

call

the

set

Sb

the

shared

pool

of

data

since

the

objects

from

this

set

can

be

shared

between

processors

during

the

DLB

phase

This

initial

declustering

of

the

data

into

two

sets

is

done

depending

on

the

desired

size

of

the

shared

pool

of

polygons

and

on

the

number

of

processors

-LRB-

In

the

following

section

we

experimentally

show

the

variation

in

the

size

of

Sa

across

a

different

number

of

processors

-RRB-

The

data

in

Sa

is

statically

declustered

into

P

sets

Sia

and

each

processor

Pi

is

assigned

the

set

Sia

for

i

-LRB-

P

-RRB-

The

choice

of

the

declustering

method

is

determined

by

the

number

of

processors

the

type

of

data

and

the

data

distribution

The

data

in

Sb

is

also

statically

declustered

into

x

buckets

and

is

replicated

at

all

the

processors

Again

note

that

any

of

the

static-declustering

methods

discussed

so

far

can

be

used

for

this

staticdeclustering

purpose

The

value

of

x

is

dependent

on

the

size

of

Sb

the

number

of

processors

and

the

communication

cost

Hence

this

parameter

should

be

tuned

depending

on

the

data

When

a

bounding

box

for

the

next

range

query

is

received

a

designated

lead

processor

-LRB-

for

example

processor

-RRB-

broadcasts

the

bounding-box

parameters

to

all

the

other

processors

in

the

group

After

receiving

the

bounding-box

parameters

each

processor

Pi

performs

the

approximate

polygon-level

filtering

and

retrieves

the

candidate

polygons

from

its

local

data

set

Sia

and

places

the

result

in

set

Li

In

addition

each

of

the

processors

performs

the

approximate

filtering

for

the

data

from

set

Sb

and

keeps

the

resulting

object

IDs

in

a

dynamic

set

such

that

the

set

of

object

IDs

from

each

of

the

x

buckets

is

in

a

separate

bin

Each

processor

Pi

then

independently

works

on

data

from

the

set

Li

until

no

more

objects

are

left

in

this

set

When

a

processor

Pi

finishes

work

on

the

data

from

Li

it

goes

into

the

DLB

mode

In

this

mode

only

the

data

from

the

dynamic

set

are

used

for

dynamic

load-balancing

The

work

is

transferred

by

transferring

a

bin

of

object

IDs

between

processors

The

algorithm

terminates

when

the

DLB

method

terminates

TYPE

pidSet

P

VAR

local

data

Array

-LSB-

pidSet

-RSB-

of

objects

map

-LSB-

i

-RSB-

to

processor

i

using

DECLUSTER

-LRB-

-RRB-

corresponding

to

data

from

S

a

VAR

global

data

Array

-LSB-

pidSet

-RSB-

of

objects

map

-LSB-

i

-RSB-

to

processor

i

using

DECLUSTER

-LRB-

-RRB-

corresponding

to

data

from

S

b

BEGIN

one

to

all

broadcast

-LRB-

pidSet

bbox

-RRB-

static

phase

parallel

for

-LRB-

pid

in

pidSet

-RRB-

do

sequential

algorithm

-LRB-

local

data

-LSB-

pid

-RSB-

-RRB-

endfor

goto

the

DLB

phase

parallel

while

-LRB-

more

work

-RRB-

do

object

ids

fetch

next

-LRB-

object

IDs

from

next

unprocessed

bucket

id

-RRB-

sequential

algorithm

-LRB-

object

ids

-RRB-

endwhile

END

Figure

Pseudo-code

for

Parallel

Formulation

Experimental

Evaluation

of

DLB

methods

We

compare

different

DLB

methods

when

applied

to

the

range-query

problem

over

a

set

of

extended

spatial

objects

We

use

the

framework

given

in

Figure

for

implementing

the

the

parallel

range-query

algorithm

Our

experiments

are

carried

out

on

the

Cray

parallel

computer

using

the

polygonal

data

described

in

Table

The

alternatives

for

each

of

the

DLB

issues

are

evaluated

by

comparing

their

average

performance

over

a

set

of

range

queries

In

these

experiments

the

similarity-graph

method

with

a

window

is

used

as

the

static

declustering

method

unless

mentioned

otherwise

For

simplicity

number

of

edges

is

used

as

the

work-load

metric

in

the

static

declustering

of

data

Similarly

the

spatial

extent

is

assumed

to

be

a

point

and

the

load-density

is

assumed

to

be

uniform

Figure

shows

our

experimental

methodology

for

evaluating

the

DLB

issues

The

number

of

different

options

we

tried

for

each

parameter

is

shown

in

parentheses

and

the

number

of

possible

combinations

after

each

module

is

also

shown

in

the

figure

The

message

start-up

time

ts

for

the

Cray

is

about

nano

seconds

i.e.

micro

seconds

To

study

the

effect

of

parallel

formulations

on

different

communication

networks

we

simulate

the

different

networks

by

increasing

the

value

of

ts

-LRB-

and

sec

-RRB-

Evaluation

of

Work-Transfer

Strategies

Work-Transfer

strategies

can

be

compared

on

the

basis

of

following

two

parameters

-LRB-

i

-RRB-

The

average

cost

Tt

of

transferring

the

complete

object

data

including

the

data

structures

from

one

processor

to

another

processor

and

-LRB-

ii

-RRB-

the

average

cost

Ts

of

solving

the

GIS-range-query

-LRB-

after

polygon-level

filtering

-RRB-

on

a

single

processor

Here

Tt

includes

the

cost

of

packing

and

unpacking

any

data

structures

related

to

the

polygons

after

the

filtering

and

the

cost

of

sending

the

packed

data

from

one

processor

to

another

processor

Table

shows

actual

experimental

values

for

Ts

and

Tt

for

randomly

chosen

range

queries

Analysis

Data

collection

polygons/chunk

Map

Generator

Map

-LRB-

X

X

-RRB-

Base

mapsize

maps

range

queries

size

of

range-query

measurements

Decluster

pool

size

P

-LRB-

-RRB-

DLB

methods

-LRB-

GRR

ARR

PBM

-RRB-

options

partitioning/declustering

method

-LRB-

llb

sim

random

-RRB-

Parallel

HPGIS

generator

work-load

#range

queries

desired

-LRB-

-RRB-

Figure

Experimental

Method

for

Evaluating

DLB

Methods

for

the

parallel

GIS-range-query

-LRB-

size

-RRB-

over

the

polygonal

data

from

the

map

The

table

also

shows

the

average

values

of

Ts

and

Tt

over

range-queries

over

map

As

shown

in

Equation

note

that

Tt

is

consistently

more

than

Ts

in

all

these

cases

and

that

this

gap

will

be

more

for

other

parallel

computers

such

as

IBM

as

ts

is

substantially

higher

for

these

machines

This

result

is

consistent

with

the

analysis

shown

in

Equation

From

this

we

conclude

that

it

is

not

desirable

to

transfer

the

complete

polygon

data

between

processors

at

run

time

Instead

only

the

polygon

IDs

should

be

transferred

at

run

time

This

is

facilitated

by

selectively

duplicating

the

polygon

data

at

some

processors

In

the

rest

of

the

experiments

work

transfers

are

always

done

by

transferring

the

object

IDs

unless

otherwise

stated

Table

Cost

of

Transfer

vs

the

cost

of

Solving

the

Problem

at

a

Single

Processor

-LRB-

cost

in

seconds

-RRB-

Time

-LRB-

sec

-RRB-

Avg

over

queries

Tt

Ts

Declustering

for

DLB

Methods

In

this

experiment

the

effect

of

chunking

based

on

systematic

declustering

is

compared

to

that

of

random

declustering

for

the

DLB

method

We

used

GRR

as

the

DLB

method

and

compared

random

similaritygraph

and

LLB

methods

of

declustering

The

dynamic

data

is

declustered

with

polygons

per

chunk

Figure

shows

the

experimental

results

for

ts

and

seconds

The

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

average

speedup

over

queries

From

this

data

it

is

clear

that

random

declustering

of

data

is

not

as

effective

as

systematic

declustering

for

achieving

a

good

load-balance

for

the

GIS-range-query

problem

Moreover

the

ordering

of

the

methods

remains

the

same

as

for

the

static

case

This

shows

that

systematic

declustering

of

data

improves

the

load-balance

Also

the

load-balance

can

be

improved

by

using

more

information

during

Speedup

Number

of

Processors

Sim

LLB

Rand

Number

of

Processors

Sim

LLB

Rand

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Speedups

for

different

declustering

methods

for

the

GRR

method

-LRB-

Map

-RRB-

the

declustering

phase

Evaluation

of

the

Granularity

of

Work-Allocation

in

DLB

We

compare

the

effect

of

different

chunk

sizes

with

the

number

of

polygons

in

chunks

ranging

from

to

using

GRR

as

the

DLB

method

and

as

the

declustering

method

In

addition

we

compared

the

effect

of

increasing

the

value

of

ts

with

decreasing

size

of

the

chunk

-LRB-

i.e.

increasing

number

of

chunks

-RRB-

The

experiment

is

conducted

using

map

for

P

with

replicated

data

being

of

the

total

data

Chunks

of

single

polygons

usually

result

in

the

best

possible

load-balance

but

this

also

results

in

maximum

overhead

due

to

the

increased

number

of

chunks

Figure

shows

the

graph

for

this

experiment

When

the

ts

value

is

low

chunks

of

single

polygons

result

in

the

best

possible

speedups

As

the

value

of

ts

is

increased

the

maximum

speedup

is

achieved

for

some

chunk

size

other

than

single

polygon

chunks

This

is

due

to

the

increased

communication

overhead

as

the

increased

number

of

chunks

requires

the

exchange

of

more

messages

between

processors

Note

that

ts

ss

seconds

is

a

typical

value

seen

in

a

MIMD

message

passing

computer

like

the

IBM

Effect

of

the

Pool

Size

We

evaluate

the

effect

of

the

pool

size

using

the

Pool-Based

Method

for

varying

number

of

processors

and

varying

data

files

The

number

of

processors

is

varied

from

to

and

the

data

files

are

varied

from

map

to

map

The

pool

size

is

varied

from

to

of

the

total

data

Note

that

a

60 80 100 120

Speedup

Polygons

per

Chunk

Figure

Granularity

of

Work-Transfers

pool

refers

to

the

static

declustering

with

no

DLB

Work

transfers

are

done

by

transferring

the

polygon

IDs

and

we

used

one

polygon

per

chunk

Speedup

Pool

Size

Speedups

for

P

P

P

Pool

Size

Speedups

for

P

Map

Map

Map

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Speedups

for

different

pool

sizes

for

Pool-Based

method

Figure

shows

the

results

of

this

experiment

The

x-axis

gives

the

size

of

the

pool

as

a

percentage

of

the

total

data

and

the

y-axis

gives

the

average

speedups

over

range

queries

As

expected

the

speedups

increase

as

we

increase

the

pool

size

up

to

a

point

and

then

they

start

decreasing

The

initial

increase

in

speedup

may

be

due

to

the

increased

load-balance

The

decrease

in

speedup

after

a

achieving

a

maximum

value

is

due

to

the

non-parallelizable

overhead

of

the

approximate

filtering

as

shown

in

Equation

Note

that

this

decrease

is

greater

as

P

increases

This

is

due

to

the

increase

in

non

parallelizable

overhead

with

increasing

P

The

maximum

speedup

occurs

at

different

pool

sizes

for

different

number

of

processors

and

for

different

data

sets

Also

note

that

the

maximum

speedups

occur

in

the

ranges

predicted

in

Table

Comparison

of

DLB

methods

We

compare

the

performance

of

the

three

DLB

methods

-LRB-

GRR

ARR

and

PBM

-RRB-

for

ts

and

ts

The

number

of

processors

is

varied

from

to

and

the

map

is

used

as

the

input

data

The

number

of

polygons

per

chunk

is

for

ts

and

for

ts

Work

is

transferred

by

transferring

the

polygon

IDs

and

is

used

for

declustering

the

data

Speedup

Number

of

Processors

PBM

ARR

GRR

Number

of

Processors

PBM

ARR

GRR

-LRB-

a

-RRB-

ts

-LRB-

b

-RRB-

ts

Figure

Speedups

for

different

DLB

methods

Figure

shows

the

speedups

for

these

three

methods

The

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

average

speedups

over

range

queries

For

ts

both

GRR

and

ARR

have

comparable

performance

while

PBM

performs

better

than

these

two

methods

as

shown

Figure

-LRB-

a

-RRB-

However

GRR

has

inferior

speedups

relative

to

other

methods

for

ts

as

shown

in

Figure

-LRB-

b

-RRB-

This

may

be

attributed

to

the

centralized

overhead

of

maintaining

the

list

of

possible

donor

processors

in

GRR

Effectivness

of

Dynamic

Load-Balancing

The

effectiveness

of

DLB

methods

in

achieving

a

good

load-balance

is

shown

in

Table

The

data

is

collected

with

P

and

with

a

pool

for

PBM

and

replicated

data

for

GRR

and

ARR

is

used

as

the

declustering

method

with

one

polygon

per

chunk

for

the

shared

data

Work

transfers

are

done

by

transferring

the

polygon

IDs

The

data

shown

in

Table

is

represented

as

Mean

SD

for

the

range

queries

used

in

our

experiment

The

column

Avg

Static

gives

the

static

execution

time

averaged

over

processors

and

range

queries

The

column

Max

Static

gives

the

maximum

static

execution

time

over

processors

averaged

over

range

queries

Similarly

Avg

T

otal

time

is

the

average

total

time

over

processors

for

queries

and

T

otal

is

the

total

parallel

run

time

averaged

over

range

queries

In

this

experiment

we

observe

that

the

DLB

methods

have

achieved

a

good

load-balance

-LRB-

i.e.

the

percentage

difference

between

the

avg.

and

the

total

-RRB-

even

though

there

is

a

very

high

load-imbalance

after

the

static

part

Table

Performance

Evaluation

of

DLB

for

P

and

Map

Method

Avg

Static

Max

Static

Avg

Total

Max

Total

Speedup

PBM

GRR

ARR

Conclusions

and

Future

Work

Data-partitioning

is

an

effective

approach

towards

achieving

high

performance

in

GIS

We

parallelize

the

GIS-range-query

problem

using

data

partitioning

and

dynamic

load-balancing

techniques

Partitioning

extended

spatial-data

maps

is

difficult

due

to

the

varying

sizes

and

extents

of

the

polygons

and

the

difficulty

of

estimating

the

work

load

Hence

special

techniques

are

needed

to

parallelize

the

GIS-rangequery

problem

We

identify

the

main

issues

in

declustering

collections

of

extended

spatial

objects

like

chains

of

linesegments

and

polygons

We

experimentally

evaluate

several

alternatives

for

each

of

these

issues

on

a

distributed

memory

MIMD

machine

for

the

range-query

operation

Experimental

results

show

that

the

number

of

edges

is

a

better

load

estimator

than

the

area

of

the

object

The

bounding

box

approximator

for

the

spatial

extent

of

an

object

gives

more

information

than

the

point

estimator

But

going

to

a

higher

order

estimator

like

multiple

bounding

boxes

is

not

practical

as

these

estimators

are

expensive

to

obtain

and

are

expensive

to

use

for

declustering

extended

spatial

data

The

results

also

show

that

among

the

static

declustering

methods

similarity-graph

and

distribution

based

methods

outperform

other

static

declustering

methods

We

also

show

that

the

performance

of

DLB

methods

can

be

further

improved

by

using

the

declustering

methods

for

determining

the

subsets

of

polygons

to

be

transferred

during

run-time

In

the

proposed

approach

we

use

the

ideas

of

declustering

in

a

hierarchical

fashion

increasing

the

load

balance

over

purely

static

methods

and

decreasing

the

communication

cost

over

purely

dynamic

methods

In

our

future

work

we

are

planning

to

scale

up

our

methods

to

larger

numbers

of

processors

larger

maps

and

queries

We

also

plan

to

extend

our

work

to

map-overlay

problems

and

other

computationally

intensive

HP-GIS

operations

Another

major

effort

would

focus

on

high

performance

techniques

for

secondary

and

tertiary-storage

terrain

mapping

and

the

effect

of

I/O

-LRB-

e.g.

swapping

-RRB-

and

indexing

methods

Finally

we

would

like

to

evaluate

these

techniques

on

the

workstation

clusters

which

are

common

in

many

GIS

applications

Acknowledgments

This

work

is

sponsored

by

the

Army

High

Performance

Computing

Research

Center

under

the

auspices

of

the

Department

of

the

Army

Army

Research

Laboratory

cooperative

agreement

number

contract

number

ARO

contract

number

the

content

of

which

does

not

necessarily

reflect

the

position

or

the

policy

of

the

government

and

no

official

endorsement

should

be

inferred

This

work

is

also

supported

by

the

Federal

Highway

Authority

and

the

Minnesota

Department

of

Transportation

We

would

like

to

thank

the

AHPCRC

University

of

Minnesota

and

the

Pittsburgh

Super

Computing

Center

for

providing

us

with

access

to

the

Cray

We

would

also

like

to

thank

Minesh

Amin

and

Christiane

McCarthy

for

improving

the

readability

and

technical

accuracy

of

this

paper

References

-LSB-

-RSB-

DIS

Home

Page

-LSB-

-RSB-

A.

Aggarwal

B.

Chazelle

L.

Guibas

C.

O'Dunlaing

and

C.

Yap

Parallel

Computational

Geometry

In

Proceedings

of

the

IEEE

Symposium

on

Foundations

of

Computer

Science

pages

-LCB-

-LSB-

-RSB-

S.

G.

Akl

and

K.

A.

Lyons

Parallel

Computational

Geometry

Prentice

Hall

Englewood

Cliffs

-LSB-

-RSB-

M.

P.

Armstrong

C.

E.

Pavlik

and

R.

Marciano

Experiments

in

the

Measurement

of

Spatial

Association

Using

a

Parallel

Supercomputer

Geographical

Systems

-LCB-

-LSB-

-RSB-

M.

J.

Atallah

and

M.

T.

Goodrich

Efficient

Plane

Sweeping

in

Parallel

In

Proceedings

of

the

Annual

ACM

Symposium

on

Computational

Geometry

pages

-LCB-

-LSB-

-RSB-

J.

L.

Bentley

and

T.

A.

Ottmann

Algorithms

for

Reporting

and

Counting

Geometric

Intersections

IEEE

Transactions

on

Computers

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

T.

Bially

Space-Filling

Curves

Their

Generation

and

Their

Application

to

Bandwidth

Reduction

IEEE

Transactions

on

Information

Theory

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

G.

Brunetti

A.

Clematis

B.

Falcidieno

A.

Sanguineti

and

M.

Spagnuolo

Parallel

Processing

of

Spatial

Data

for

Terrain

Characterization

In

Proceedings

of

the

ACM

workshop

in

GIS

-LSB-

-RSB-

H.

C.

Du

and

J.

S.

Sobolewski

Disk

Allocation

for

Product

Files

on

Multiple

Disk

Systems

ACM

Transactions

on

Database

Systems

March

-LSB-

-RSB-

C.

Faloutsos

and

D.

Metaxas

Disk

Allocation

Methods

Using

Error

Correcting

Codes

IEEE

Transactions

on

Computers

August

-LSB-

-RSB-

M.

T.

Fang

R.

C.

T.

Lee

and

C.

C.

Chang

The

Idea

of

Declustering

and

its

Applications

In

Proceedings

of

the

International

Conference

on

Very

Large

Databases

-LSB-

-RSB-

Z.

Fang

P.-C

Yew

P.

Tang

and

C.-Q

Zhu

Dynamic

processor

self-scheduling

for

general

parallel

nested

loops

In

Proceedings

of

the

International

Conference

in

Parallel

Processing

August

-LSB-

-RSB-

W.

R.

Franklin

C.

Narayanaswami

M.

Kankanahalli

D.

Sun

M.

Zhou

and

P.

Y.

F.

Wu

Uniform

Grids

A

Technique

for

Intersection

Detection

on

Serial

and

Parallel

Machines

In

Proceedings

of

Auto-Carto

pages

-LCB-

-LSB-

-RSB-

M.

R.

Garey

and

D.

S.

Johnson

Computers

and

Intractability

A

Guide

to

the

Theory

of

NP-Completeness

Freeman

San

Francisco

-LSB-

-RSB-

A.

Guttman

R-trees

A

Dynamic

Index

Structure

for

Spatial

Searching

In

Proceedings

of

the

SIGMOD

Conference

pages

-LCB-

-LSB-

-RSB-

W.

Hibbard

and

D.

Santek

Visualizing

Large

Data

Sets

in

the

Earth

Sciences

IEEE

Computer

Special

Issue

on

Vualization

in

Scientific

Computing

August

-LSB-

-RSB-

E.

G.

Hoel

and

H.

Samet

Data

Parallel

R-Tree

Algorithms

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

-LSB-

-RSB-

E.

G.

Hoel

and

H.

Samet

Data

Parallel

Spatial

Join

Algorithms

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

-LSB-

-RSB-

E.

G.

Hoel

and

H.

Samet

Performance

of

Data-Parallel

Spatial

Operations

In

Proceedings

of

the

VLDB

Conference

pages

-LCB-

-LSB-

-RSB-

S.

F.

Hummel

E.

Schonberg

and

L.

E.

Flynn

Factoring

a

method

for

scheduling

parallel

loops

Communications

of

the

ACM

pages

-LCB-

August

-LSB-

-RSB-

H.

V.

Jagadish

Linear

Clustering

of

Objects

with

Multiple

Attributes

In

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

-LSB-

-RSB-

I.

Kamel

and

C.

Faloutsos

Parallel

R-Trees

In

Proceedings

of

the

International

Conference

on

Management

of

Data

ACM

SIGMOD

-LSB-

-RSB-

C.

Kruskal

and

A.

Weiss

Allocating

independent

subtasks

on

parallel

processors

IEEE

Transactions

on

Software

Engineering

pages

-LCB-

October

-LSB-

-RSB-

V.

Kumar

A.

Grama

A.

Gupta

and

G.

Karypis

Introduction

to

Parallel

Computing

Design

and

Analysis

of

Algorithms

The

Benjamin/Cummings

Publishing

Company

Inc.

-LSB-

-RSB-

V.

Kumar

A.

Grama

and

V.

N.

Rao

Scalable

load

balancing

techniques

for

parallel

computers

Journal

of

Distributed

Computing

March

-LSB-

-RSB-

Y.

Liang

and

B.

A.

Barsky

An

Analysis

and

Algorithm

for

Polygon

Clipping

Communications

of

the

ACM

November

-LSB-

-RSB-

D.

R.

Liu

and

S.

Shekhar

A

Similarity

Graph-Based

Approach

to

Declustering

Problem

and

its

Applications

In

Proceedings

of

the

Eleventh

International

Conference

on

Data

Engineering

IEEE

-LSB-

-RSB-

D.

R.

Pratt

M.

Zyda

and

K.

Kelleher

Guest

Editor

's

Introduction

Virtual

Reality-In

the

Mind

of

the

Beholder

IEEE

Computer

Special

Issue

on

Virtual

Environments

July

-LSB-

-RSB-

S.

Shekhar

S.

Ravada

V.

Kumar

D.

Chubb

and

G.

Turner

Load-Balancing

in

High

Performance

GIS

Declustering

Polygonal

Maps

In

Proceedings

of

the

International

Symposium

on

Large

Spatial

Databases

August

-LSB-

-RSB-

R.

Sridhar

S.

S.

Iyengar

and

S.

Rajanarayanan

Range

Search

In

Parallel

Using

Distributed

Data

Structures

In

International

Conference

on

Databases

Parallel

Architectures

and

Their

Applications

Miami

Beach

Florida

pages

-LCB-

-LSB-

-RSB-

B.

R.

Vatti

A

Generic

Solution

to

Polygon

Clipping

Communications

of

the

ACM

July

-LSB-

-RSB-

F.

Wang

A

Parallel

Intersection

Algorithm

for

Vector

Polygon

Overlay

IEEE

Computer

Graphics

Applications

March

-LSB-

-RSB-

Y.

Zhou

S.

Shekhar

and

M.

Coyle

Disk

Allocation

Methods

for

Parallelizing

Grid

Files

In

Proceedings

of

the

Tenth

International

Conference

on

Data

Engineering

IEEE

A

Similarity-Graph

Method

The

similarity-graph

max-cut

approach

to

declustering

is

described

in

detail

in

-LSB-

-RSB-

Here

we

extend

this

concept

for

declustering

extended

spatial

objects

to

the

case

of

multiple

processors

The

weighted

similarity

graph

which

is

used

for

declustering

the

spatial

data

is

defined

as

follows

Definition

Weighted

Similarity

Graph

-LRB-

WSG

-RRB-

-LSB-

-RSB-

Let

WSG

-LRB-

V

E

-RRB-

be

a

weighted

similarity

graph

where

V

is

a

data

set

and

E

fe

-LRB-

u

v

-RRB-

ju

V

v

V

and

u

and

v

are

qualified

to

be

accessed

together

in

a

queryg

Each

edge

e

-LRB-

u

v

-RRB-

in

E

is

associated

with

a

weight

w

-LRB-

u

v

-RRB-

which

represents

the

relative

frequency

with

which

data

items

u

and

v

are

likely

to

be

accessed

together

by

a

query

of

interest

In

addition

each

v

V

is

associated

with

a

weight

t

-LRB-

v

-RRB-

which

represents

the

load

imposed

by

v

for

a

query

of

interest

For

example

t

-LRB-

v

-RRB-

may

be

load

-LRB-

v

-RRB-

or

clipped

load

-LRB-

v

-RRB-

As

in

the

case

of

the

LLB

method

a

rectangular

window

W

can

be

used

as

a

sample

query

for

efficiency

The

WSG

is

then

constructed

w.r.t.

this

window

W

by

assigning

clipped

load

-LRB-

v

W

-RRB-

as

t

-LRB-

v

-RRB-

for

each

object

v

in

the

input

In

addition

the

weight

-LRB-

relative

frequency

-RRB-

between

two

objects

is

assigned

as

follows

The

weight

w

-LRB-

u

v

-RRB-

between

two

data

objects

u

and

v

is

zero

if

one

object

is

inside

-LRB-

or

intersects

-RRB-

the

region

of

interest

and

the

other

object

is

outside

That

is

w

-LRB-

u

v

-RRB-

is

zero

if

u

and

v

are

never

accessed

together

by

a

range-query

For

example

in

Figure

w

-LRB-

u

v

-RRB-

is

zero

since

u

is

inside

the

region

of

interest

and

v

is

outside

When

the

edge

weight

between

two

objects

is

not

zero

the

relative

frequency

with

which

these

two

objects

are

accessed

together

depends

on

the

spatial

extent

of

the

work-load

Kamel

and

Faloutsos

-LSB-

-RSB-

give

a

method

for

finding

this

relative

frequency

when

the

objects

are

approximated

by

bounding

boxes

-LRB-

or

by

points

-RRB-

We

use

their

method

for

determining

the

number

of

range

queries

that

will

access

a

pair

of

spatial

objects

Since

a

part

of

the

polygon

may

be

outside

the

region

of

interest

we

need

to

use

only

that

part

of

the

polygon

which

is

inside

the

region

of

interest

in

computing

the

relative

frequency

For

example

polygon

b

in

Figure

is

partly

outside

the

region

of

interest

Then

we

only

use

the

part

of

the

bounding

box

which

is

inside

the

region

of

interest

as

the

bounding

box

approximation

for

polygon

b

In

the

case

of

multiple

bounding-box

approximation

for

spatial

extent

there

exists

no

simple

method

to

determine

the

relative

frequency

and

we

intend

to

investigate

this

issue

in

our

future

work

Figure

shows

two

possible

geometries

between

the

lines

R

and

S.

Let

proximity

-LRB-

R

S

-RRB-

be

the

probability

that

a

query

retrieves

both

R

and

S.

Then

when

the

two

lines

R

and

S

intersect

the

fraction

of

the

queries

retrieving

both

the

line-segments

is

given

by

proximity

-LRB-

R

S

-RRB-

-LRB-

ffi

-RRB-

When

the

two

lines

are

non

intersecting

the

fraction

of

the

queries

retrieving

both

the

line-segments

is

u

v

b

Figure

Different

geometries

between

polygons

and

the

region

of

interest

given

by

proximity

-LRB-

R

S

-RRB-

-LRB-

ffi

-RRB-

Since

x

and

y

extents

of

the

bounding

boxes

are

independent

the

fraction

of

the

queries

that

retrieve

two

bounding

boxes

BR

and

BS

is

given

by

proximity

-LRB-

BR

BS

-RRB-

proximity

-LRB-

BxR

BxS

-RRB-

proximity

-LRB-

ByR

ByS

-RRB-

where

BxR

and

ByR

are

the

projections

of

the

box

BR

on

x-axis

and

y-axis

respectively

Similar

notation

is

used

for

the

box

BS

R

S

R

S

Dd

Figure

Two

different

geometries

for

two

intersecting

lines

To

decluster

the

data

using

the

WSG

a

new

weighted

graph

G

is

constructed

from

WSG

as

follows

A

graph

G

fEG

VGg

is

constructed

from

WSG

fE

V

g

such

that

EG

E

and

VG

V

The

new

weight

weight

-LRB-

u

v

-RRB-

between

two

nodes

u

VG

and

v

VG

is

computed

as

weight

-LRB-

u

v

-RRB-

w

-LRB-

u

v

-RRB-

minft

-LRB-

u

-RRB-

t

-LRB-

v

-RRB-

g

where

w

-LRB-

u

v

-RRB-

is

the

weight

between

u

and

v

in

WSG

Once

graph

G

is

constructed

max-cut

partitioning

of

G

is

used

for

declustering

the

data

Since

the

max-cut

graph-partitioning

problem

is

NP-complete

heuristic

algorithms

are

used

for

partitioning

graph

G

In

our

study

we

use

the

incremental

max-cut

partitioning

-LSB-

-RSB-

approach

declustering methods
geographic information systems
high performance
loadbalancing
polygon clipping
range query