Data

Partitioning

and

Load

Balancing

in

Parallel

Disk

Systems

P.

G.

P.

Technical

Report

Department

of

Computer

Science

University

of

the

Saarland

April

Peter

Scheuermann

Department

of

Electrical

Engineering

and

Computer

Science

Northwestern

University

Evanston

IL

U.S.A.

E-mail

peters@eecs.nwu.edu

Gerhard

Weikum

Department

of

Computer

Science

University

of

the

Saarland

P.O.

Box

Saarbr?ucken

Germany

E-mail

weikum@cs.uni-sb.de

WWW

Peter

Zabback

Tandem

Computers

Incorporated

North

Tantau

Avenue

Cupertino

CA

U.S.A.

E-mail

zabback@patch.tandem.com

This

research

has

been

partially

supported

by

NASA-Ames

grant

grant

NSF

grant

and

the

ESPRIT

LTR

project

-LRB-

HERMES

-RRB-

Contents

Introduction

Tuning

Issues

in

Parallel

Disk

Systems

Tuning

Issues

in

Data

Partitioning

Tuning

Issues

in

Data

Allocation

and

Load

Balancing

Contribution

and

Outline

of

the

Paper

Data

Partitioning

Phase

A

Minimizing

Service

Time

Phase

B

Minimizing

Response

Time

by

Considering

Through

put

and

Queueing

Delay

Putting

It

All

Together

the

Algorithm

for

Data

Partitioning

Load

Balancing

Data

Allocation

Disk

Cooling

Heat

Tracking

Experimental

Results

Experiments

With

Synthetic

Workload

Workload

With

Uniform

Access

Frequencies

Workload

With

Skewed

Access

Frequencies

Experiments

With

Application

Traces

World-Wide-Web

Server

On-line

Transaction

Processing

Conclusion

Abstract

Parallel

disk

systems

provide

opportunities

for

exploiting

I/O

parallelism

in

two

possible

ways

namely

via

inter-request

and

intra-request

parallelism

In

this

paper

we

discuss

the

main

issues

in

performance

tuning

of

such

systems

namely

striping

and

load

balancing

and

show

their

relationship

to

response

time

and

throughput

We

outline

the

main

components

of

an

intelligent

file

system

that

optimizes

striping

by

taking

into

account

the

requirements

of

the

applications

and

performs

load

balancing

by

judicious

file

allocation

and

dynamic

redistributions

of

the

data

when

access

patterns

change

Our

system

uses

simple

but

effective

heuristics

that

incur

only

little

overhead

We

present

performance

experiments

based

on

synthetic

workloads

and

real-life

traces

Introduction

Tuning

Issues

in

Parallel

Disk

Systems

Parallel

disk

systems

are

of

great

importance

to

massively

parallel

computers

since

they

are

scalable

and

they

can

ensure

that

I/O

is

not

the

limiting

factor

in

achieving

high

speedup

-LSB-

-RSB-

However

to

make

effective

use

of

the

commercially

available

architectures

it

is

necessary

to

develop

intelligent

software

tools

that

allow

automatic

tuning

of

the

parallel

disk

system

to

varying

workloads

The

choice

of

a

striping

unit

and

whether

to

choose

a

file-specific

striping

unit

versus

a

global

striping

unit

are

important

parameters

that

affect

the

response

time

and

throughput

of

the

system

Equally

important

is

the

decision

of

how

to

allocate

the

data

on

the

actual

disks

and

how

to

perform

redistribution

of

the

data

when

access

patterns

change

These

tuning

options

need

to

be

performed

dynamically

using

simple

but

effective

heuristics

that

incur

only

little

overhead

This

paper

presents

a

set

of

performance

tuning

techniques

for

parallel

disk

systems

These

techniques

are

orthogonal

to

the

techniques

for

high

availability

that

are

typically

employed

in

parallel

disk

systems

-LRB-

e.g.

RAID

levels

-RRB-

and

can

be

applied

to

a

wide

spectrum

of

applications

ranging

from

conventional

file

systems

and

WWW

servers

to

database

systems

Throughout

the

paper

we

assume

that

the

underlying

computer

architecture

is

that

of

a

shared-memory

multiprocessor

extensions

to

distributed-memory

architectures

are

conceivable

but

are

not

considered

in

this

paper

In

order

to

effectively

exploit

the

potential

for

I/O

parallelism

in

parallel

disk

systems

data

must

be

partitioned

and

distributed

across

disks

The

partitioning

can

be

performed

at

two

levels

The

physical

-LRB-

block

or

byte

-RRB-

level

The

term

striping

is

used

for

this

variant

of

partitioning

schemes

which

divides

a

file

into

fixed-size

runs

of

logically

consecutive

data

units

that

are

assigned

to

disks

in

a

roundrobin

manner

-LSB-

-RSB-

The

striping

unit

denotes

the

number

of

consecutive

data

bytes

or

blocks

stored

per

disk

The

application

level

The

term

declustering

has

been

employed

in

relational

database

systems

to

denote

partitioning

schemes

that

perform

a

horizontal

division

of

a

relation

into

fragments

based

on

the

values

of

one

or

several

attributes

Among

the

schemes

employed

for

single

attribute

partitioning

are

hashing

and

range

partitioning

-LSB-

-RSB-

while

techniques

based

on

Cartesian

product

files

have

been

advocated

for

multiple

attribute

declustering

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Striping

has

an

advantage

over

application-level

methods

in

that

it

can

be

applied

as

a

generic

low-level

method

for

a

wide

spectrum

of

data

types

-LRB-

all

of

which

are

ultimately

mapped

into

block-structured

files

-RRB-

In

this

paper

we

therefore

restrict

our

attention

to

file

striping

for

data

partitioning

In

the

following

a

file

may

denote

a

tablespace

or

indexspace

in

a

relational

database

a

logical

object

cluster

in

an

object-oriented

database

a

document

such

as

WWW

pages

in

a

multimedia

information

system

or

indeed

simply

a

Unix-like

sequence-of-blocks

file

We

shall

use

the

term

striping

width

of

a

file

to

denote

the

number

of

disks

over

which

a

file

is

spread

A

logically

consecutive

portion

of

the

file

that

resides

on

one

disk

and

whose

size

is

a

striping

unit

is

called

a

run

All

runs

of

a

file

that

are

mapped

to

the

same

disk

are

combined

into

a

single

allocation

unit

called

an

extent

Striping

provides

opportunities

for

exploiting

I/O

parallelism

in

two

possible

ways

Intra-request

-LRB-

intra-operation

-RRB-

parallelism

allows

the

parallel

execution

of

a

single

request

by

multiple

disks

Inter-request

-LRB-

inter-operation

-RRB-

parallelism

can

be

achieved

if

independent

requests

are

being

served

by

the

disk

system

at

the

same

time

The

degree

of

parallelism

in

serving

a

single

data

request

is

the

number

of

different

disks

on

which

the

requested

data

resides

Tuning

Issues

in

Data

Partitioning

The

striping

unit

is

an

important

parameter

that

must

be

chosen

judiciously

in

order

to

reduce

the

service

time

of

a

single

request

or

to

improve

the

throughput

of

multiple

requests

-LSB-

-RSB-

A

large

striping

unit

tends

to

cluster

a

file

on

one

disk

which

does

not

allow

any

degree

of

intra-request

parallelism

In

consequence

the

service

time

of

a

request

is

not

improved

but

the

throughput

is

optimal

if

the

requests

are

uniformly

spread

across

all

disks

At

the

other

end

of

the

spectrum

a

small

striping

unit

provides

very

good

response

time

for

light

load

but

severely

limits

the

throughput

as

the

total

amount

of

device-busy

time

consumed

in

serving

a

single

request

is

increasing

with

decreasing

striping

unit

Consequently

for

small

striping

units

the

response

time

may

deteriorate

under

heavy

load

due

to

queueing

delays

In

practice

it

is

necessary

to

choose

the

striping

unit

such

that

a

certain

objective

function

is

minimized

One

such

objective

function

aims

at

minimizing

the

response

time

subject

to

a

constraint

on

the

achievable

throughput

In

-LSB-

-RSB-

heuristic

methods

are

proposed

to

determine

the

striping

unit

of

a

disk

array

based

on

the

knowledge

of

the

average

request

size

and

the

application

's

expected

multiprogramming

level

under

the

assumption

of

a

closed

queueing

model

While

these

assumptions

may

be

valid

for

relatively

small

multiprogramming

levels

they

do

not

scale

up

to

data

management

systems

with

large

numbers

of

concurrent

users

which

translates

to

high

arrival

rates

with

stochastic

load

fluctuation

It

is

most

crucial

for

such

systems

to

guarantee

a

certain

level

of

performance

during

these

peak

periods

Most

heuristic

methods

also

advocate

choosing

a

global

striping

unit

i.e.

the

same

striping

unit

for

all

files

in

the

system

-LSB-

-RSB-

However

many

applications

such

as

multimedia

information

systems

-LRB-

e.g.

in

digital

libraries

or

medical

applications

-RRB-

exhibit

highly

diverse

file

characteristics

making

it

desirable

to

be

able

to

tune

the

striping

unit

individually

for

each

file

Consider

for

example

the

case

where

a

global

striping

unit

may

be

appropriate

at

some

moment

in

time

but

later

on

the

load

is

shifting

causing

some

crucial

files

to

exhibit

unusually

high

response

time

File-specific

striping

enables

us

to

incrementally

restripe

only

these

crucial

files

leaving

the

other

files

with

the

-LRB-

old

-RRB-

global

striping

unit

After

the

striping

unit

has

been

determined

-LRB-

globally

or

on

file-specific

basis

-RRB-

the

file

system

must

derive

the

striping

width

i.e.

the

number

of

disks

across

which

the

file

-LRB-

s

-RRB-

is

-LRB-

are

-RRB-

spread

In

our

model

a

file

is

either

spread

across

all

disks

or

if

the

file

is

relatively

small

its

width

is

obtained

as

the

quotient

of

the

file

size

and

the

striping

unit

Similar

response-time

constraints

to

the

ones

discussed

above

may

justify

however

that

some

files

be

stored

on

a

dedicated

subset

of

disks

in

order

to

avoid

contention

and

hence

their

striping

width

would

be

limited

by

the

number

of

disks

in

this

subset

This

specific

consideration

of

incorporating

response-time

constraints

is

not

pursued

in

this

paper

however

Tuning

Issues

in

Data

Allocation

and

Load

Bal

ancing

The

striping

unit

-LRB-

s

-RRB-

and

striping

width

-LRB-

s

-RRB-

are

only

some

of

the

parameters

that

affect

the

response

time

or

throughput

of

a

parallel

disk

system

The

decision

of

how

to

allocate

or

place

the

files

on

the

actual

disks

is

an

equally

important

one

in

order

to

obtain

good

load

balancing

Load

refers

to

the

amount

of

work

done

by

each

disk

and

it

affects

both

the

response

time

and

throughput

Balancing

the

load

contributes

towards

minimizing

the

average

length

of

the

queues

associated

with

the

disk

-LRB-

minimizing

the

service

time

variance

per

disk

would

be

another

factor

-LSB-

-RSB-

which

is

not

considered

in

this

paper

however

-RRB-

Very

small

striping

units

lead

to

very

good

load

balancing

in

the

extreme

case

each

request

involves

all

the

disks

in

the

system

so

that

the

load

is

perfectly

balanced

But

throughput

considerations

require

for

many

applications

that

we

choose

large

striping

units

-LRB-

e.g.

the

size

of

a

cylinder

-RRB-

-LSB-

-RSB-

Thus

load

balancing

needs

to

be

performed

even

if

striping

is

employed

In

order

to

perform

disk

load

balancing

it

is

necessary

to

take

into

account

the

frequencies

of

the

requests

to

the

various

files

or

data

partitions

as

well

as

the

request

sizes

To

account

for

these

parameters

the

file

system

has

to

keep

track

of

the

following

related

statistics

ffl

the

heat

of

extents

and

disks

where

the

heat

is

defined

as

the

sum

of

the

number

of

block

accesses

of

an

extent

or

disk

per

time

unit

as

determined

by

statistical

observation

over

a

certain

period

of

time

ffl

and

the

temperature

of

extents

which

is

defined

as

the

ratio

between

heat

and

size

-LSB-

-RSB-

If

the

striping

unit

is

a

byte

and

all

files

are

partitioned

across

all

disks

in

the

system

then

we

obtain

a

well

balanced

I/O

load

While

this

approach

may

be

adequate

for

supercomputer

applications

characterized

by

very

large

request

sizes

-LSB-

-RSB-

-LRB-

i.e.

a

high

data

rate

-RRB-

it

certainly

limits

the

throughput

of

transaction

processing

applications

characterized

by

a

high

rate

of

read

and

write

requests

to

small

amounts

of

random

information

-LSB-

-RSB-

-LRB-

i.e.

a

high

I/O

rate

-RRB-

As

soon

as

the

striping

unit

is

relatively

large

-LRB-

e.g.

a

track

or

cylinder

-RRB-

the

need

for

load

balancing

reappears

immediately

even

if

the

files

are

partitioned

across

all

disks

This

is

due

to

the

fact

that

the

heat

of

the

various

blocks

or

extents

are

often

distributed

in

a

highly

non-uniform

manner

Contribution

and

Outline

of

the

Paper

This

paper

presents

viable

methods

for

several

key

issues

in

the

automatic

tuning

of

parallel

storage

systems

Various

aspects

of

earlier

versions

of

our

approach

have

been

published

in

-LSB-

-RSB-

and

-LSB-

-RSB-

give

an

account

of

the

more

general

project

where

this

work

has

been

embedded

In

this

paper

we

emphasize

the

system

perspective

and

we

describe

more

advanced

automatic

tuning

methods

We

provide

guidelines

for

potential

system

architects

of

self-reliant

storage

systems

and

we

give

a

comprehensive

experimental

evidence

of

the

viability

and

benefits

of

our

approach

We

present

a

procedure

for

performing

data

partitioning

that

is

a

significant

extension

to

the

algorithm

outlined

in

-LSB-

-RSB-

In

particular

our

new

algorithm

takes

into

account

queueing

delays

explicitly

by

providing

a

computationally

tractable

analytical

approximation

for

the

fork-join

queueing

model

-LSB-

-RSB-

We

describe

in

detail

an

effective

method

for

heat

bookeeping

and

our

demon-based

dynamic

migration

procedure

based

on

disk

cooling

Although

the

problems

of

data

partitioning

and

load

balancing

are

orthogonal

issues

they

are

not

independent

The

performance

experiments

reported

in

this

paper

clearly

illustrate

this

issue

and

show

that

the

combined

effects

of

data

partitioning

and

load

balancing

produce

significant

advantages

over

conventional

striping

methods

based

on

physical

device

units

-LRB-

e.g

block

track

-RRB-

The

remainder

of

this

paper

is

organized

as

follows

We

will

describe

in

Sections

and

the

main

components

of

an

intelligent

file

manager

for

parallel

disk

systems

that

performs

automatic

data

partitioning

data

allocation

and

load

balancing

by

incremental

reorganization

steps

In

Section

we

report

on

performance

studies

of

our

file

system

based

on

synthetic

workloads

and

real-life

traces

We

conclude

with

an

outlook

on

additional

research

avenues

that

we

plan

to

explore

in

order

to

generalize

our

approach

Data

Partitioning

We

have

developed

an

analytic

model

to

determine

heuristically

the

optimal

striping

unit

and

striping

width

on

an

individual

file

basis

or

on

a

global

basis

These

parameters

can

be

chosen

for

each

file

individually

based

upon

the

file

's

estimated

average

request

size

R

or

globally

by

using

instead

the

average

request

size

over

all

files

denoted

by

R

In

either

case

the

optimization

can

be

carried

out

in

one

or

two

phases

A

and

B

depending

upon

the

anticipated

arrival

rate

of

requests

For

low

arrival

rates

of

requests

where

we

can

assume

that

no

queueing

delays

occur

Phase

A

chooses

a

degree

of

parallelism

that

minimizes

the

service

time

of

an

average

request

of

size

R

-LRB-

or

R

-RRB-

which

is

equivalent

to

minimizing

the

response

time

if

the

system

operates

in

single

user

mode

Phase

B

chooses

a

degree

of

parallelism

that

minimizes

the

-LRB-

multi-user

-RRB-

response

time

subject

to

the

constraint

that

the

achievable

throughput

is

at

least

as

high

as

the

application

's

average

arrival

rate

of

requests

to

all

files

denoted

by

The

optimal

degree

of

parallelism

Peff

chosen

for

an

average

request

is

then

adjusted

by

choosing

the

minimum

-LRB-

normalized

as

explained

below

-RRB-

between

the

outcomes

of

Phases

A

and

B

The

striping

unit

and

width

are

then

derived

from

the

optimal

degree

of

parallelism

Peff

Our

Phase

B

optimization

uses

an

open

queueing

model

in

order

to

take

into

account

explicitly

the

throughput

considerations

and

queueing

delays

As

mentioned

before

the

striping

method

proposed

by

Chen

et

al.

-LSB-

-RSB-

is

based

on

a

closed

queueing

model

There

a

heuristic

formula

is

derived

from

experiments

as

well

as

approximative

analytical

treatment

which

suggests

a

global

striping

unit

of

SU

s

-LRB-

L

X

-LRB-

M

-RRB-

R

D

-LRB-

-RRB-

where

L

is

the

average

latency

-LRB-

sum

of

seek

and

rotational

delays

-RRB-

of

a

disk

X

is

the

transfer

rate

of

a

disk

M

is

the

multiprogramming

level

of

the

application

R

is

the

average

request

size

and

D

is

the

number

of

disks

-LSB-

-RSB-

further

extends

this

approach

by

considering

the

impact

of

parity

writes

in

a

RAID

level

system

Chen

et

al.

-LSB-

-RSB-

also

discuss

the

difficulty

of

estimating

the

multiprogramming

level

As

we

pointed

out

earlier

we

consider

an

open

queueing

model

to

be

more

appropriate

for

a

data

management

system

with

a

large

number

of

users

-LRB-

as

opposed

for

example

to

a

file

server

in

a

LAN

of

workstations

-RRB-

In

addition

an

estimate

of

the

average

arrival

rate

of

requests

to

all

files

which

is

used

in

our

model

is

generally

much

easier

to

obtain

than

an

estimate

of

M

the

multiprogramming

level

Phase

A

Minimizing

Service

Time

Given

a

number

of

files

to

be

allocated

Phase

A

determines

the

optimal

degree

of

partitioning

on

a

file-specific

-LRB-

global

-RRB-

basis

based

on

average

request

request

size

R

-LRB-

R

-RRB-

This

estimate

can

be

derived

in

many

cases

from

the

file

's

type

information

For

example

in

an

OLTP

system

such

as

airline

reservation

or

phone

call

switching

and

accounting

one

can

typically

expect

an

average

request

size

of

a

block

On

the

other

hand

in

a

multimedia

application

such

as

digital

libraries

or

medical

archiving

we

can

expect

that

all

requests

will

require

access

to

an

entire

document

-LRB-

e.g.

an

image

-RRB-

and

hence

R

would

be

the

file

size

Let

P

be

the

degree

of

parallelism

involved

in

serving

an

average

request

of

size

R

i.e.

the

number

of

disks

involved

in

serving

this

request

In

the

absence

of

queueing

delays

the

expected

service

time

to

be

denoted

by

Tserv

-LRB-

R

P

-RRB-

is

in

fact

equal

to

the

expected

response

time

to

be

denoted

by

Tresp

-LRB-

R

P

-RRB-

The

expected

service

time

is

given

by

Tserv

-LRB-

R

P

-RRB-

max

i

-LRB-

tseek

i

trot

i

-RRB-

ttrans

-LRB-

R

P

-RRB-

-LRB-

-RRB-

where

tseek

i

and

trot

i

-LRB-

i

P

-RRB-

denote

the

seek

time

and

rotation

time

respectively

of

disk

i

involved

in

serving

the

request

For

tractability

purposes

we

replace

the

right

hand

side

by

the

following

approximation

Tserv

-LRB-

R

P

-RRB-

max

i

-LRB-

tseek

i

-RRB-

max

i

-LRB-

trot

i

-RRB-

ttrans

-LRB-

R

P

-RRB-

-LRB-

-RRB-

Thus

we

note

that

the

solution

to

equation

-LRB-

-RRB-

provides

in

fact

an

upper

bound

for

Tserv

-LRB-

R

P

-RRB-

In

order

to

obtain

approximate

distributions

for

Tseek

maxi

-LRB-

tseek

i

-RRB-

and

Trot

maxi

-LRB-

trot

i

-RRB-

we

make

the

standard

assumption

that

the

delays

at

each

disk

i.e.

seek

times

and

rotation

times

are

independent

and

identically

distributed

random

variables

-LSB-

-RSB-

In

addition

we

assume

that

the

delay

probabilities

are

unconditional

i.e.

the

probability

of

a

delay

does

not

depend

upon

the

probabilities

of

previous

delays

In

reality

there

may

be

a

certain

degree

of

correlation

among

these

variables

for

example

in

the

case

of

a

synchronized

disk

array

where

all

disks

heads

move

in

tandem

Also

in

some

applications

it

is

possible

to

have

a

sequence

of

requests

to

successive

blocks

on

a

disk

in

other

words

the

probability

of

a

seek

distance

is

conditional

upon

the

probabilities

of

previous

seek

distances

Let

us

denote

by

dseek

i

and

drot

i

-LRB-

i

:P

-RRB-

the

dual

random

variables

that

give

us

the

distances

traveled

on

disk

i

by

the

head

or

arm

respectively

from

the

current

location

to

the

requested

one

We

shall

compute

first

the

expected

values

of

Dseek

maxi

-LRB-

dseek

i

-RRB-

and

Drot

maxi

-LRB-

drot

i

-RRB-

and

use

these

values

to

derive

the

expected

values

of

Tseek

and

Trot

Under

the

assumptions

given

above

the

cumulative

distribution

functions

for

Dseek

and

Drot

can

be

computed

respectively

as

the

product

of

P

cumulative

distribution

functions

of

the

random

variables

dseek

i

and

drot

i

corresponding

to

the

P

disks

involved

in

serving

the

request

We

compute

first

the

probability

mass

function

of

dseek

i

P

rob

-LSB-

dseek

i

z

-RSB-

-LRB-

C

z

-RRB-

-LRB-

-RRB-

where

C

denotes

the

number

of

cylinders

on

one

disk

From

here

be

obtain

the

cumulative

distribution

function

P

rob

-LSB-

dseek

i

z

-RSB-

-LRB-

z

C

-RRB-

-LRB-

-RRB-

It

was

shown

in

-LSB-

-RSB-

that

the

expected

value

E

-LSB-

Dseek

-RSB-

is

given

by

E

-LSB-

Dseek

-RSB-

C

-LRB-

PY

i

-RRB-

-LRB-

-RRB-

The

product

in

equation

-LRB-

-RRB-

can

be

approximated

by

the

following

expression

with

constants

a

and

b

E

-LSB-

Dseek

-RSB-

C

-LRB-

a

b

ln

-LRB-

P

-RRB-

-RRB-

-LRB-

-RRB-

From

here

the

expected

value

of

Tseek

can

be

approximated

by

the

following

linear

equation

with

appropriate

-LRB-

disk-type-dependent

-RRB-

constants

e

and

f

E

-LSB-

Tseek

-RSB-

eE

-LSB-

Dseek

-RSB-

f

-LRB-

-RRB-

We

note

here

that

the

equation

which

converts

seek

distance

in

cylinders

to

seek

time

consists

in

fact

of

two

components

a

non-linear

one

and

a

linear

one

-LSB-

-RSB-

However

in

our

model

we

are

interested

only

in

the

expected

value

of

the

seek

distance

and

the

corresponding

expected

value

of

the

seek

time

and

the

expected

value

of

the

seek

distance

lies

in

the

linear

part

of

the

distance-time

equation

The

rotation

distance

on

a

given

disk

i

drot

i

gives

the

fraction

of

a

full

rotation

that

is

necessary

in

order

to

position

the

arm

on

the

first

block

of

the

current

request

In

order

to

compute

E

-LSB-

Drot

-RSB-

we

make

the

common

assumption

that

drot

i

is

a

uniformly

distributed

variable

in

the

range

-LSB-

-RSB-

thus

P

rob

-LSB-

drot

i

r

-RSB-

r

-LRB-

-RRB-

From

here

we

obtain

the

cumulative

distribution

function

of

Drot

maxi

-LRB-

drot

i

-RRB-

as

P

rob

-LSB-

Drot

r

-RSB-

PY

i

P

rob

-LSB-

drot

i

r

-RSB-

rP

-LRB-

-RRB-

and

furthermore

E

-LSB-

Drot

-RSB-

PP

It

follows

that

the

expected

value

of

Trot

is

given

by

E

-LSB-

Trot

-RSB-

P

P

-LRB-

-RRB-

where

ROT

denotes

the

rotation

time

of

a

disk

It

can

be

seen

from

these

equations

that

as

the

degree

of

parallelism

P

increases

both

the

expected

seek

time

E

-LSB-

Tseek

-RSB-

and

the

expected

rotation

time

E

-LSB-

Trot

-RSB-

increase

also

For

small

requests

these

two

components

of

service

time

are

the

dominant

ones

hence

the

service

time

increases

also

The

only

component

of

the

service

time

that

decreases

with

an

increased

degree

of

parallelism

is

the

transfer

time

ttrans

-LRB-

R

P

-RRB-

Each

disk

transfers

RP

blocks

-LRB-

assuming

for

simplicity

identical

subrequest

sizes

on

the

disks

-RRB-

-RRB-

and

if

we

ignore

cylinder

and

head

switches

the

transfer

time

can

be

estimated

as

RP

ROT

B

where

B

is

the

number

of

blocks

in

a

track

However

in

order

to

account

for

the

fact

that

these

RP

blocks

span

over

track

and

cylinder

boundaries

we

add

corresponding

correction

terms

and

obtain

E

-LSB-

ttrans

-LRB-

R

P

-RRB-

-RSB-

-LRB-

nhs

ncs

-RRB-

ths

ncstcs

R

PBROT

-LRB-

-RRB-

where

nhs

is

the

number

of

head

switches

-LRB-

including

cylinder

switches

-RRB-

ncs

is

the

number

of

cylinder

switches

ths

is

the

head

switch

delay

and

tcs

is

the

cylinder

switch

time

-LRB-

time

for

a

seek

of

distance

-RRB-

Using

simple

probability

arguments

we

estimate

nhs

dR

=P

B

e

B

-LRB-

R

=P

-LRB-

dR

PB

e

-RRB-

B

-RRB-

B

ss

RP

B

-LRB-

-RRB-

ncs

dR

=P

TB

e

TB

-LRB-

R

=P

-LRB-

dR

=P

TB

e

-RRB-

B

-RRB-

TB

ss

RP

TB

-LRB-

-RRB-

and

we

obtain

E

-LSB-

ttrans

-LRB-

R

P

-RRB-

-RSB-

-LRB-

RP

B

RP

TB

-RRB-

ths

RP

TB

tcs

R

PBROT

-LRB-

-RRB-

where

T

is

the

number

of

tracks

in

a

cylinder

Combining

the

above

results

we

obtain

the

following

formula

for

the

expected

service

time

E

-LSB-

Tserv

-LRB-

R

P

-RRB-

-RSB-

eC

-LRB-

a

bln

-LRB-

P

-RRB-

-RRB-

f

P

P

-LRB-

-RRB-

-LRB-

RP

B

RP

TB

-RRB-

ths

RP

TB

tcs

R

PBROT

The

trade-offs

between

increased

seek

and

rotation

time

on

one

hand

and

reduced

transfer

time

on

the

other

hand

for

various

degrees

of

parallelism

are

illustrated

also

in

Figure

This

example

considers

a

file

that

is

being

striped

across

disks

with

three

different

striping

units

and

resulting

degrees

of

intra-request

parallelism

The

figure

traces

the

execution

of

an

I/O

request

of

size

blocks

for

the

three

configurations

In

addition

to

the

service

time

the

figure

also

illustrates

the

device-busy

time

for

the

given

request

which

is

the

sum

of

the

times

that

the

disks

are

involved

in

the

request

For

illustration

purposes

the

seek

and

rotation

times

are

combined

together

into

latency

time

The

optimal

degree

of

parallelism

Popt

can

be

determined

by

finding

the

minimum

of

the

function

E

-LSB-

Tserv

-LRB-

R

P

-RRB-

-RSB-

i.e.

by

solving

the

following

cubic

equation

for

P

dE

-LSB-

Tserv

-LRB-

R

P

-RRB-

-RSB-

dP

ROT

P

eCb

P

PROT

-LRB-

P

-RRB-

-LRB-

-RRB-

-LRB-

R

P

R

P

-RRB-

ths

R

P

tcs

RROT

P

Phase

B

Minimizing

Response

Time

by

Consider

ing

Throughput

and

Queueing

Delay

An

increased

degree

of

parallelism

leads

not

only

to

trade-offs

between

seek

and

rotation

time

on

one

hand

and

reduced

transfer

time

on

the

other

hand

but

also

affects

adversely

the

device-busy

time

of

a

request

i.e.

the

sum

of

the

times

that

the

disks

are

involved

in

the

request

and

hence

are

not

available

for

other

requests

The

relationship

between

the

device-busy

time

and

the

various

components

of

the

response

time

is

illustrated

in

the

execution

of

requests

in

Figure

The

throughput

measured

as

the

number

of

requests

completed

per

time

unit

is

inversely

proportional

to

the

average

device-busy

time

of

a

request

Thus

higher

degrees

of

parallelism

lead

to

unproductive

positioning

times

and

hence

to

lower

throughput

The

Phase

A

model

for

service

time

minimization

has

assumed

that

there

are

no

interferences

among

the

various

requests

and

that

no

queueing

delays

occur

This

is

obviously

not

the

situation

in

a

multiprogramming

environment

especially

under

heavy

load

i.e.

a

high

arrival

rate

queueing

delays

play

an

important

role

The

scenario

where

each

I/O

request

is

served

by

a

single

disk

is

well

understood

and

can

be

modeled

via

an

queueing

model

-LSB-

-RSB-

We

observe

however

that

no

general

analytical

model

is

known

for

so

called

fork-join

queueing

model

-LSB-

-RSB-

i.e.

for

the

case

when

I/O

requests

are

served

by

multiple

disks

and

the

number

of

disks

involved

varies

from

request

to

request

An

exception

is

the

case

when

exactly

two

disks

are

being

involved

in

serving

every

request

-LSB-

-RSB-

We

present

in

this

section

a

simplified

and

computationally

tractable

analytic

approximation

to

a

fork-join

model

under

the

assumption

of

perfect

load

balance

More

specifically

we

compute

first

the

mean

response

time

on

each

disk

assuming

the

requests

are

equally

distributed

among

the

disks

and

that

each

disk

can

be

represented

as

an

system

Then

we

use

an

approximation

method

outlined

in

-LSB-

-RSB-

in

order

to

compute

the

expected

response

time

for

requests

with

degree

of

parallelism

P

-LRB-

averaged

over

the

requests

to

all

files

-RRB-

as

the

maximum

among

the

response

times

of

the

P

participating

disks

Our

analytic

approximation

to

the

queueing

model

requires

that

we

provide

an

estimate

of

the

average

arrival

rate

to

all

files

in

the

system

denoted

as

in

addition

to

the

average

request

size

across

all

files

denoted

as

R.

Note

that

the

value

of

R

can

be

derived

by

sampling

or

alternatively

it

can

be

computed

from

the

average

request

sizes

Ri

to

the

individual

files

and

the

access

frequencies

of

the

files

The

objective

of

Phase

B

is

to

compute

the

optimal

value

for

P

the

average

degree

of

parallelism

Given

that

requests

in

our

system

have

an

average

arrival

rate

of

and

an

average

degree

of

parallelism

of

P

we

obtain

the

overall

arrival

rate

for

the

constituent

subrequests

as

P

Under

the

assumption

of

a

perfectly

balanced

system

where

the

subrequests

are

equally

distributed

among

the

disks

the

subrequest

arrival

rate

to

a

given

disk

i

-LRB-

i

D

-RRB-

to

be

denoted

as

i

can

be

computed

as

i

P

D

-LRB-

-RRB-

with

D

standing

for

the

number

of

disks

in

the

system

The

average

subrequest

size

to

be

denoted

as

S

can

be

derived

as

S

R

P

-LRB-

-RRB-

The

service

time

for

an

individual

subrequest

to

disk

i

to

be

denoted

by

tserv

i

-LRB-

S

-RRB-

can

be

computed

by

using

the

standard

formulae

for

the

service

time

of

a

single

disk

We

can

express

the

utilization

of

disk

i

aei

as

aei

i

tserv

i

-LRB-

S

-RRB-

-LRB-

-RRB-

Using

our

assumption

that

each

disk

can

be

viewed

as

an

queue

the

expected

value

of

tresp

i

-LRB-

S

-RRB-

the

response

time

of

the

subrequests

served

at

disk

i

is

given

as

-LSB-

-RSB-

E

-LSB-

tresp

i

-LRB-

S

-RRB-

-RSB-

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

aei

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

-LRB-

aei

-RRB-

-LRB-

-RRB-

where

stands

for

the

squared

coefficient

of

variation

of

the

service

time

of

subrequests

at

disk

i.

is

defined

as

the

ratio

of

the

corresponding

variance

-LRB-

V

AR

-RRB-

and

expected

service

time

V

AR

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

-LRB-

-RRB-

Also

from

queueing

theory

we

obtain

the

formula

below

which

relates

the

variance

of

the

response

time

of

individual

subrequests

on

a

disk

i

to

the

first

three

moments

of

their

service

time

VAR

-LSB-

tresp

i

-LRB-

S

-RRB-

-RSB-

VAR

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

iE

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

-LRB-

aei

-RRB-

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

-LRB-

aei

-RRB-

-LRB-

-RRB-

The

response

time

for

requests

of

size

R

served

by

P

disks

to

be

denoted

as

Tresp

-LRB-

R

P

-RRB-

satisfies

the

equality

E

-LSB-

Tresp

-LRB-

R

P

-RRB-

-RSB-

max

i

-LRB-

tresp

i

-LRB-

S

-RRB-

-RRB-

-LRB-

-RRB-

In

order

to

derive

an

analytic

expression

for

the

above

equation

we

make

use

of

an

approximation

method

presented

in

-LSB-

-RSB-

which

has

been

shown

to

be

quite

accurate

if

the

response

times

of

the

individual

subrequests

i.e.

tresp

i

obey

a

normal

distribution

This

approximation

states

that

the

expected

response

time

for

a

request

can

be

estimated

as

the

response

time

of

an

individual

subrequest

plus

a

correction

factor

which

accounts

for

the

slowest

subrequest

E

-LSB-

Tresp

-LRB-

R

P

-RRB-

-RSB-

ss

E

-LSB-

tresp

i

-LRB-

S

-RRB-

-RSB-

q

VAR

-LSB-

tresp

i

-LRB-

S

-RRB-

-RSB-

P

for

P

E

-LSB-

tresp

i

-LRB-

S

-RRB-

-RSB-

q

VAR

-LSB-

tresp

i

-LRB-

S

-RRB-

-RSB-

q

log

P

for

P

-LRB-

-RRB-

We

have

conducted

a

series

of

experiments

and

these

have

shown

that

the

assumption

of

normally

distributed

response

times

for

the

individual

subrequests

is

a

valid

one

We

observe

that

with

an

increase

in

the

variance

of

the

response

time

of

individual

subrequests

the

correction

factor

increases

correspondingly

The

impact

of

the

degree

of

parallelism

P

on

the

different

components

of

the

response

time

which

we

described

informally

in

Figure

is

taken

into

account

implicitly

by

the

correction

factor

in

equation

-LRB-

-RRB-

In

order

to

calculate

E

-LSB-

Tresp

-RSB-

in

equation

-LRB-

-RRB-

it

is

necessary

to

compute

the

first

three

moments

of

the

subrequests

service

time

distribution

namely

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

and

E

-LSB-

tserv

i

-LRB-

S

-RRB-

-RSB-

For

this

calculation

we

need

to

derive

the

probability

density

function

of

tserv

i

-LRB-

S

-RRB-

The

probability

density

functions

of

the

corresponding

seek

and

rotation

times

i.e.

fseek

i

and

frot

i

can

be

derived

from

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

the

probability

density

function

of

the

transfer

time

i.e.

ftrans

i

is

a

constant

whose

value

is

obtained

by

setting

P

in

equation

-LRB-

-RRB-

Finally

the

probability

density

function

for

tserv

i

-LRB-

S

-RRB-

can

be

obtained

by

convoluting

the

probability

density

functions

fseek

i

frot

i

and

ftrans

i

The

full

details

of

this

derivation

are

given

in

-LSB-

-RSB-

The

value

P

which

minimizes

equation

-LRB-

-RRB-

can

be

found

iteratively

by

going

through

the

range

of

possible

values

for

P

-LRB-

this

is

obviously

bounded

by

D

the

number

of

disks

in

the

system

-RRB-

We

choose

this

approach

since

equation

-LRB-

-RRB-

is

not

easily

differentiable

unlike

its

counterpart

in

Phase

A

namely

equation

-LRB-

-RRB-

Putting

It

All

Together

the

Algorithm

for

Data

Partitioning

The

complete

algorithm

for

data

partitioning

is

outlined

in

Figure

below

If

we

anticipate

a

low

arrival

rate

of

requests

and

desire

to

perform

optimization

only

by

using

Phase

A

then

Steps

and

are

omitted

For

file-specific

partitioning

Steps

and

need

to

be

iterated

over

the

number

of

files

in

the

system

On

the

other

hand

for

global

partitioning

the

above

steps

need

to

be

executed

only

once

with

one

exception

as

explained

below

The

effective

degree

of

parallelism

Peff

is

computed

in

Step

by

choosing

the

minimum

between

the

the

optimal

degrees

of

parallelism

computed

in

Steps

and

The

factor

Ri

R

is

used

to

normalize

the

outcome

of

Step

This

is

due

to

the

fact

that

for

requests

larger

than

R

we

want

the

degree

of

parallelism

of

file

i

to

exceed

P

and

if

Ri

is

smaller

than

R

then

Peff

i

should

be

smaller

than

P

The

optimal

striping

unit

and

striping

width

are

then

derived

from

Peff

i

-LRB-

or

Peff

respectively

-RRB-

in

Step

If

all

I/O

requests

start

at

run

boundaries

then

the

striping

unit

of

a

file

SUi

can

be

derived

by

using

the

formula

d

Ri

Peff

i

e

This

is

also

the

case

when

the

requests

are

for

individual

blocks

i.e.

Ri

or

for

the

entire

file

i.e.

Ri

Li

with

Li

being

the

file

size

On

the

other

hand

if

the

requests

can

start

at

any

block

inside

a

run

the

formula

above

yields

a

striping

unit

which

can

not

support

in

most

cases

the

degree

of

parallelism

Peff

i

this

in

fact

increases

Peff

i

by

one

In

order

to

cover

this

case

the

striping

unit

is

derived

by

the

alternative

formula

d

Ri

Peff

i

which

guarantees

a

degree

of

parallelism

of

Peff

i

in

all

cases

In

the

case

of

Peff

i

the

striping

unit

should

be

chosen

as

large

as

possible

i.e.

SUi

Li

with

Li

being

the

file

size

Finally

the

striping

width

denoted

as

SWi

is

chosen

as

high

as

possible

in

order

to

support

inter-request

parallelism

also

in

addition

to

the

intra-request

parallelism

optimized

by

the

above

steps

Notice

that

the

striping

width

SWi

needs

to

be

computed

individually

also

in

the

case

of

global

partitioning

since

some

files

may

be

too

small

to

be

spread

over

all

the

disks

The

algorithm

outlined

in

Figure

accomplishes

static

partitioning

since

all

the

files

are

allocated

at

the

same

time

However

the

algorithm

can

be

extended

easily

to

perform

dynamic

partitioning

Dynamic

partitioning

and

the

complementary

procedure

of

incremental

repartitioning

need

to

be

performed

when

new

files

are

added

old

files

are

deleted

or

when

the

access

characteristics

of

some

files

change

substantially

Let

us

discuss

here

the

case

when

a

new

file

needs

to

be

added

to

the

system

We

need

to

recompute

first

the

access

characteristics

specified

in

the

input

to

the

partitioning

algorithm

i.e.

to

readjust

R

S

and

in

order

to

account

for

the

addition

of

the

new

file

In

order

to

perform

these

calculations

we

need

to

estimate

Ri

the

average

file

request

size

as

well

as

i

the

average

arrival

rate

of

requests

to

the

new

file

As

discussed

before

this

information

may

be

derived

by

sampling

existing

files

of

the

same

type

or

may

be

provided

as

an

administrator

hint

-LRB-

e.g.

when

we

consider

a

large

database

application

-RRB-

We

then

invoke

the

static

partitioning

algorithm

given

above

on

the

new

file

in

order

to

determine

its

effective

degree

of

parallelism

Peff

i

and

its

striping

unit

and

width

A

companion

incremental

repartitioning

procedure

is

invoked

periodically

This

procedure

checks

first

if

a

trigger

condition

is

satisfied

in

order

to

warrant

incremental

repartitioning

The

trigger

condition

consists

of

two

parts

-LRB-

-RRB-

P

new

P

old

and

-LRB-

-RRB-

E

-LSB-

tresp

-LRB-

Rnew

P

new

-RRB-

-RSB-

E

-LSB-

tresp

-LRB-

Rold

P

old

-RRB-

-RSB-

ffl

with

ffl

being

a

system

determined

parameter

The

new

set

of

statistics

is

computed

by

performing

Steps

through

of

the

static

partitioning

algorithm

the

old

set

of

statistics

is

the

one

computed

at

the

last

invocation

of

this

procedure

If

the

trigger

condition

is

satisfied

then

we

proceed

to

do

incremental

repartitioning

of

k

files

The

procedure

considers

candidate

files

for

reorganization

by

using

a

list

in

which

the

files

are

sorted

in

descending

order

of

heat

We

use

heat

as

an

ordering

criterium

since

this

measures

the

product

of

arrival

rate

and

file

size

an

early

repartitioning

of

the

hottest

files

will

make

the

biggest

contribution

to

the

average

degree

of

parallelism

Popt

Note

that

although

P

new

may

be

different

from

P

old

a

particular

file

i

may

not

need

to

be

reorganized

if

the

value

of

Peff

i

does

not

change

Load

Balancing

The

need

for

load

balancing

was

mentioned

already

in

Section

in

the

context

of

data

allocation

Recall

that

load

balancing

does

not

become

obsolete

when

striping

is

employed

Many

applications

require

that

we

choose

large

striping

units

in

order

to

achieve

a

certain

throughput

with

multi-block

requests

For

example

Gray

et

al.

have

proposed

the

parity

striping

scheme

-LSB-

-RSB-

where

the

distribution

of

data

blocks

is

based

on

a

very

large

-LRB-

possibly

infinite

-RRB-

striping

unit

and

similar

results

on

the

throughput

limits

of

fine-grained

striping

have

been

stated

in

-LSB-

-RSB-

However

a

coarser

striping

unit

increases

the

probability

of

load

imbalance

under

a

skewed

workload

-LSB-

-RSB-

Addressing

this

tradeoff

solely

by

tuning

the

striping

unit

is

only

a

-LRB-

bad

-RRB-

compromise

Thus

additional

methods

for

load

balancing

are

called

for

regardless

of

whether

data

is

partitioned

or

not

Obviously

the

load

balance

of

a

disk

system

depends

on

the

placement

of

data

regardless

of

whether

the

files

are

partitioned

or

not

The

data

placement

problem

is

similar

to

the

file

allocation

problem

in

distributed

systems

-LSB-

-RSB-

and

falls

in

the

class

of

NP-hard

problems

-LRB-

the

simplest

case

is

equivalent

to

the

NP-complete

problem

of

multiprocessor

scheduling

-LCB-

see

problem

-LSB-

-RSB-

in

-LSB-

-RSB-

-RRB-

Hence

viable

solutions

must

be

based

on

heuristics

The

worst-case

performance

of

these

heuristics

methods

can

be

measured

in

terms

of

their

competitive

ratio

which

is

defined

as

the

ratio

between

the

heat

of

the

hottest

disk

under

a

given

heuristic

placement

and

the

heat

of

the

hottest

disk

under

an

optimal

placement

Good

heuristics

based

on

greedy

placement

-LSB-

-RSB-

or

iterated

bin-packing

-LSB-

-RSB-

are

well

understood

for

the

static

file

allocation

problem

with

non-partitioned

files

where

the

heat

of

each

file

is

known

in

advance

In

the

greedy

algorithm

which

was

adopted

in

the

Bubba

parallel

database

machine

-LSB-

-RSB-

the

files

are

first

sorted

by

descending

heat

and

then

they

are

allocated

in

this

order

where

in

each

step

the

disk

with

the

lowest

accumulated

heat

is

selected

Under

this

greedy

heuristic

the

competitive

ratio

is

bounded

by

number

of

disks

while

for

the

iterated

bin-packing

algorithm

of

-LSB-

-RSB-

the

corresponding

competitive

ratio

is

approximately

We

observe

here

that

these

results

are

derived

from

specifically

constructed

adversary

inputs

and

there

is

experimental

evidence

that

these

heuristic

allocation

algorithms

perform

better

for

most

realistic

inputs

In

practice

realistic

algorithms

for

static

allocation

of

non-partitioned

files

need

to

consider

additional

parameters

and

system

constraints

such

as

controller

contention

and

storage

space

limitation

A

comprehensive

heuristic

optimization

method

which

considers

some

of

these

constrained

is

presented

in

-LSB-

-RSB-

where

a

non-linear

programming

solution

embedded

in

a

queueing

network

model

is

described

Moreover

in

many

application

environments

the

files

are

not

allocated

all

at

the

same

time

but

rather

some

files

are

allocated

dynamically

For

this

dynamic

case

the

following

canonical

extension

of

the

greedy

heuristic

mentioned

above

has

been

studied

intensively

in

the

theory

of

online

algorithms

a

new

file

is

placed

on

the

disk

with

the

currently

lowest

accumulated

heat

and

the

heat

of

the

target

disk

is

then

incremented

by

the

heat

of

the

new

file

It

has

been

shown

that

this

online

greedy

method

guarantees

a

competitive

ratio

of

r

number

of

disks

-LSB-

-RSB-

This

worst-case

bound

can

be

further

improved

to

a

minor

extent

by

more

sophisticated

allocation

heuristics

-LSB-

-RSB-

However

it

has

also

been

shown

that

no

online

algorithm

can

achieve

a

competitive

ratio

better

than

ss

-LSB-

-RSB-

When

additional

constraints

on

the

set

of

eligible

disks

are

taken

into

account

the

best

possible

competitive

ratio

is

bounded

-LRB-

from

below

-RRB-

by

-LRB-

number

of

disks

-RRB-

e

-LSB-

-RSB-

The

problem

of

data

allocation

in

parallel

disk

systems

has

an

additional

constraint

that

is

not

considered

in

any

of

the

works

mentioned

above

Namely

in

order

to

support

intra-request

parallelism

it

is

necessary

to

allocate

the

extents

of

a

file

on

different

disks

Not

only

are

files

to

be

created

or

deleted

dynamically

but

files

can

grow

or

shrink

In

addition

the

access

characteristics

of

files

can

change

over

time

and

what

was

originally

a

good

allocation

under

a

certain

workload

may

not

be

any

longer

the

case

later

in

time

In

order

to

deal

with

all

these

dynamics

of

change

it

is

necessary

to

incorporate

into

a

file

manager

another

tuning

component

that

can

redistribute

the

load

by

migrating

data

from

one

disk

to

another

at

any

time

a

certain

imbalance

in

load

is

detected

Migration

of

entire

files

has

been

considered

in

the

context

of

replicated

file

systems

On

the

other

hand

migration

of

file

portions

has

been

considered

for

scalable

distributed

hashing

schemes

but

with

different

objective

functions

-LSB-

-RSB-

The

only

work

that

considers

data

migration

in

the

context

of

disk

load

balancing

is

-LSB-

-RSB-

however

this

work

is

restricted

to

off-line

and

monolithic

-LRB-

i.e.

non-incremental

-RRB-

reorganization

The

load

balancing

component

of

our

intelligent

file

system

consists

of

two

independent

modules

one

that

performs

file

allocation

and

a

second

one

that

performs

dynamic

redistribution

of

data

These

components

are

described

in

Subsections

and

Subsection

explains

how

our

system

keeps

track

of

the

heat

and

temperature

of

extents

and

disks

Data

Allocation

We

have

extended

the

greedy

algorithm

of

-LSB-

-RSB-

in

order

to

deal

with

-LRB-

dynamic

-RRB-

allocation

of

partitioned

files

-LSB-

-RSB-

In

the

static

case

where

all

files

are

given

in

advance

the

algorithm

first

sorts

all

extents

by

descending

heat

and

the

extents

are

allocated

in

sort

order

For

each

extent

to

be

allocated

the

algorithm

selects

the

disk

with

the

lowest

accumulated

heat

among

the

disks

which

have

not

yet

been

assigned

another

extent

of

the

same

file

This

method

is

illustrated

in

Figure

and

is

contrasted

with

a

standard

roundrobin

scheme

The

figure

shows

the

placement

of

three

files

each

consisting

of

three

extents

with

heat

proportional

to

the

height

of

the

corresponding

boxes

We

denote

by

i.j

the

extent

j

of

file

i.

Observe

that

in

Figure

extents

and

are

allocated

in

this

order

to

the

current

disk

with

the

lowest

accumulated

disk

however

when

extent

is

to

be

allocated

we

do

not

choose

disk

since

it

holds

already

an

extent

of

file

but

instead

of

this

allocate

it

on

disk

In

the

dynamic

case

the

sorting

step

is

eliminated

and

the

algorithm

uses

only

the

information

about

the

heat

of

the

files

which

have

been

allocated

and

for

which

statistics

are

collected

already

Thus

as

compared

to

the

canonical

extension

discussed

in

the

previous

section

the

heat

of

the

target

disk

remains

unchanged

at

the

time

of

an

extent

allocation

The

heat

will

adjusted

correspondingly

only

after

enough

accesses

to

the

newly

allocated

extent

have

been

recorded

The

disk

selection

can

be

made

in

such

a

way

as

to

consider

also

if

so

desired

the

cost

of

additional

I/Os

necessary

to

perform

partial

disk

reorganization

Partial

disk

reorganization

may

have

to

be

performed

if

due

to

file

additions

and

deletions

there

is

room

to

store

an

extent

on

a

disk

but

the

space

is

not

contiguous

Even

more

expensive

is

the

situation

when

disk

i

has

the

lowest

heat

and

may

appear

as

the

obvious

choice

to

store

a

new

extent

of

a

file

but

this

disk

does

not

have

enough

free

space

In

order

to

make

room

for

the

new

extent

we

have

to

migrate

one

or

more

extents

to

a

different

disk

In

order

to

account

for

these

reorganization

costs

we

associate

with

every

disk

a

status

variable

with

regard

to

the

extent

chosen

for

allocation

The

status

variable

can

take

the

values

FREE

FRAG

and

FULL

depending

upon

whether

the

disk

-LRB-

-RRB-

has

enough

free

space

for

the

extent

-LRB-

-RRB-

has

enough

space

but

the

the

space

is

fragmented

or

-LRB-

-RRB-

does

not

have

enough

free

space

Our

file

allocation

algorithm

has

the

option

of

selecting

disks

in

increasing

heat

order

without

regard

to

their

status

Alternatively

we

may

select

the

disks

in

multiple

passes

where

in

the

first

pass

we

only

choose

those

that

have

status

FREE

More

details

and

experimental

studies

on

this

combined

free-space

management

and

data

allocation

method

are

given

in

-LSB-

-RSB-

In

the

current

paper

we

do

not

further

consider

the

impact

of

fragmented

or

full

disks

Disk

Cooling

In

order

to

perform

dynamic

heat

redistribution

we

employ

in

our

system

a

dynamic

load

balancing

step

called

disk

cooling

Basically

disk

cooling

is

a

greedy

procedure

which

tries

to

determine

the

best

candidate

i.e.

extent

to

remove

from

the

hottest

disk

in

order

to

minimize

the

amount

of

data

that

is

moved

while

obtaining

the

maximal

gain

The

temperature

metric

is

used

as

the

criterion

for

selecting

the

extents

to

be

reallocated

because

temperature

reflects

the

benefit/cost

ratio

of

the

reallocation

since

benefit

is

proportional

to

heat

-LRB-

i.e.

reduction

of

heat

-RRB-

and

cost

is

proportional

to

size

-LRB-

of

the

reallocated

extents

-RRB-

This

approach

is

illustrated

in

Figure

the

basic

disk

cooling

algorithm

is

given

in

Figure

The

extent

to

be

moved

denoted

by

e

is

reallocated

on

the

coolest

disk

denoted

by

t

such

that

t

does

not

hold

already

an

extent

of

the

corresponding

file

and

t

has

enough

contiguous

free

space

In

our

system

the

disk

cooling

procedure

is

implemented

as

a

background

demon

which

is

invoked

at

fixed

intervals

in

time

The

procedure

checks

first

if

the

trigger

condition

is

satisfied

or

not

-LRB-

Steps

and

in

Figure

-RRB-

If

the

trigger

condition

is

false

the

system

is

considered

load

balanced

and

no

cooling

action

is

performed

In

the

basic

disk

cooling

procedure

the

system

is

not

considered

load

balanced

if

the

heat

of

the

hottest

disk

exceeds

the

average

disk

heat

by

a

certain

quantity

ffi

It

is

important

to

observe

that

during

each

invocation

of

the

procedure

different

disks

can

be

selected

as

candidates

for

cooling

after

each

cooling

step

Our

procedure

considers

implicitly

the

cost/benefit

ratio

of

a

considered

cooling

action

and

only

schedules

it

for

execution

if

is

considered

beneficial

These

cost

considerations

are

reflected

in

Step

of

the

algorithm

The

hottest

disk

is

likely

to

have

already

a

heavy

share

of

the

load

which

we

can

measure

by

observing

if

its

queue

is

non-empty

A

cooling

action

would

most

likely

increase

the

load

imbalance

if

a

queue

is

present

at

the

source

disk

since

it

implies

additional

I/Os

for

the

reorganization

process

Hence

we

choose

not

to

schedule

the

cooling

action

if

this

condition

is

satisfied

We

also

consider

the

cooling

move

not

to

be

cost-beneficial

if

would

it

be

executed

the

heat

of

the

target

disk

would

exceed

the

heat

of

the

source

disk

Hence

although

our

background

demon

is

invoked

a

fixed

number

of

times

only

a

fraction

of

these

invocations

result

in

data

migration

Our

generic

disk

cooling

procedure

can

be

generalized

in

a

number

of

ways

In

-LSB-

-RSB-

we

have

shown

how

an

explicit

objective

function

based

on

disk

heat

variance

-LRB-

DHV

-RRB-

can

be

used

in

a

more

general

test

for

the

cost/benefit

of

a

cooling

action

Thus

the

benefit

is

computed

by

comparing

the

DHV

after

the

potential

cooling

step

with

the

DHV

before

the

potential

cooling

step

In

addition

we

can

consider

also

explicitly

the

cost

of

performing

the

cooling

Thus

a

more

accurate

calculation

of

benefit

and

cost

would

consider

not

only

the

reduction

in

heat

on

the

origin

disk

and

the

increase

in

heat

on

the

target

disk

but

also

the

additional

heat

caused

by

the

reorganization

process

itself

The

cooling

process

is

executed

during

two

intervals

of

time

the

first

corresponding

to

the

read

phase

of

the

action

and

the

second

corresponding

to

the

write

phase

of

the

action

The

additional

heat

generated

during

these

phases

can

be

computed

by

dividing

the

size

of

the

extent

to

be

moved

by

the

corresponding

duration

of

the

phase

The

duration

times

of

the

read

and

write

phase

of

a

cooling

action

can

be

estimated

by

using

a

queueing

model

as

shown

in

-LSB-

-RSB-

Our

disk

cooling

procedure

can

be

fine-tuned

so

that

the

unit

of

reallocation

is

chosen

dynamically

in

order

to

increase

the

potential

of

a

positive

cost/benefit

ratio

In

the

basic

procedure

given

in

Figure

the

unit

of

redistribution

is

assumed

to

be

an

extent

However

in

the

case

of

large

extents

that

are

very

hot

the

cost

of

a

redistribution

may

be

prohibitive

In

this

case

we

can

subdivide

further

an

extent

into

a

number

of

fixed-size

fragments

and

use

a

fragment

as

the

unit

of

redistribution

Since

all

fragments

of

an

extent

are

of

the

same

size

we

can

now

base

the

choice

of

the

migration

candidates

-LRB-

see

Step

in

Figure

-RRB-

on

the

heat

statistic

instead

of

temperature

In

addition

the

increase

in

the

number

of

allocation

units

of

a

file

also

requires

that

we

remove

the

allocation

constraint

on

the

target

disk

namely

we

do

not

require

anymore

that

the

disk

should

hold

only

one

fragment

of

a

file

Hence

we

put

here

the

objective

of

a

balanced

load

above

the

requirement

that

the

file

partitioning

is

optimal

Heat

Tracking

The

dynamic

tracking

of

the

heat

of

blocks

is

implemented

based

on

a

moving

average

of

the

interarrival

time

of

requests

to

the

same

block

Conceptually

we

keep

track

of

the

times

when

the

last

k

requests

to

each

block

occurred

where

k

is

a

fine-tuning

parameter

-LRB-

in

the

range

from

to

-RRB-

To

illustrate

this

bookkeeping

procedure

assume

that

a

block

is

accessed

at

the

points

in

time

tn

-LRB-

n

k

-RRB-

Then

the

average

interarrival

time

of

the

k

last

requests

is

tn?tn?k

k

and

the

estimated

heat

of

the

block

is

the

corresponding

reciprocal

k

tn?tn?k

Upon

the

next

access

to

this

block

say

at

time

tn

the

block

heat

is

re-estimated

as

k

tn

tn?k

One

may

conceive

an

alternative

method

for

heat

tracking

that

keeps

a

count

of

the

number

of

requests

to

a

block

within

the

last

T

seconds

where

T

would

be

a

global

tuning

parameter

The

problem

with

such

a

global

approach

is

that

it

can

not

track

the

heat

of

both

hot

and

cold

blocks

in

an

equally

responsive

manner

Hot

blocks

would

need

a

relatively

short

value

of

T

to

ensure

that

we

become

aware

of

heat

variations

quickly

enough

Cold

blocks

on

the

other

hand

would

need

a

large

value

of

T

to

ensure

that

we

see

a

sufficient

number

of

requests

to

smooth

out

stochastic

fluctuations

The

moving-average

method

for

the

interarrival

time

does

not

have

this

problem

since

a

fixed

value

of

k

actually

implies

a

short

observation

time

window

for

hot

blocks

and

a

long

window

for

cold

blocks

Moreover

extensive

experimentation

with

traces

from

real

applications

with

evolving

access

patterns

has

shown

that

our

tracking

method

works

well

for

a

wide

spectrum

of

k

values

the

heat

estimation

is

fairly

insensitive

to

the

exact

choice

of

k

-LSB-

-RSB-

Furthermore

under

the

assumption

that

requests

to

a

block

arrive

according

to

a

Poisson

process

-LRB-

i.e.

with

exponentially

distributed

interarrival

time

-RRB-

the

heat

estimate

would

be

Erlang-k

distributed

and

the

minimum

k

for

achieving

a

desired

statistical

confidence

in

the

heat

estimate

can

be

derived

analytically

-LSB-

-RSB-

The

adopted

heat

tracking

method

is

very

responsive

to

sudden

increases

of

a

block

's

heat

the

new

access

frequency

is

fully

reflected

in

the

heat

estimate

after

k

requests

which

would

take

only

a

short

while

for

hot

blocks

-LRB-

and

reasonable

values

of

k

-RRB-

However

the

method

adapts

the

heat

estimate

more

slowly

when

a

block

exhibits

a

sudden

drop

of

its

heat

In

the

extreme

case

a

hot

block

may

suddenly

cease

to

be

accessed

at

all

In

this

case

we

would

continue

to

keep

the

block

's

old

heat

estimate

as

there

are

no

more

new

requests

to

the

block

To

counteract

this

form

of

erroneous

heat

estimation

we

employ

an

additional

aging

method

for

the

heat

estimates

The

aging

is

implemented

by

periodically

invoking

a

demon

process

that

simulates

pseudo

requests

to

all

blocks

Whenever

such

a

pseudo

request

would

lead

to

a

heat

reduction

the

block

's

heat

estimate

is

updated

otherwise

the

pseudo

request

is

ignored

For

example

assume

that

there

is

a

pseudo

request

at

time

and

consider

a

block

with

heat

H

We

compute

tentatively

the

new

heat

of

the

block

as

H

k

but

we

update

the

heat

bookkeeping

only

if

H

H

We

notice

that

this

selective

aging

method

is

much

more

effective

than

the

global

one

proposed

in

-LSB-

-RSB-

The

complete

heat

tracking

method

is

illustrated

in

Figure

The

described

heat

tracking

method

requires

a

space

overhead

of

-LRB-

k

-RRB-

floating-point

numbers

per

block

Since

we

want

to

keep

this

bookkeeping

information

in

memory

for

fast

cooling

decisions

it

is

usually

unacceptable

to

track

the

heat

of

each

individual

block

In

order

to

reduce

the

overhead

involved

in

heat

tracking

we

actually

apply

the

heat

estimation

procedure

to

entire

extents

-LRB-

or

fragments

of

a

specified

size

-RRB-

We

keep

track

of

the

times

tn

tn?k

of

the

last

k

requests

that

involve

any

blocks

of

the

extent

in

the

manner

described

above

and

also

we

keep

the

number

of

accessed

blocks

within

the

extent

for

each

of

the

last

k

requests

Assume

that

the

average

number

of

accessed

blocks

is

R

Then

the

heat

of

the

extent

is

estimated

by

kR

tn?tn?k

Finally

we

estimate

the

heat

of

a

fraction

of

an

extent

by

assuming

that

each

block

in

the

extent

has

the

same

heat

-LRB-

which

is

extent

heat

divided

by

extent

size

-RRB-

This

extent-based

heat

tracking

method

reduces

substantially

the

space

overhead

of

the

block-based

estimation

procedure.y

On

the

other

hand

our

experimental

studies

-LRB-

including

studies

with

application

traces

-RRB-

have

shown

that

the

loss

in

accuracy

versus

block-based

heat

tracking

is

minimal

Experimental

Results

In

this

section

we

present

an

experimental

performance

evaluation

of

the

file

striping

and

allocation

and

load

balancing

algorithms

presented

above

The

testbed

for

these

experiments

was

built

on

top

of

the

file

system

prototype

FIVE

-LSB-

-RSB-

FIVE

runs

on

shared-memory

multiprocessors

under

Solaris

and

a

few

other

Unix

versions

It

can

manage

either

real

data

on

real

disks

-LRB-

i.e.

raw

partitions

-RRB-

or

it

can

interact

with

a

simulated

disk

system

to

estimate

the

impact

the

virtual

resources

The

disk

simulation

keeps

track

of

exact

arm

positions

as

well

as

rotational

positions

of

the

disk

head

Our

simulator

considers

head

switch

delays

and

incorporates

a

realistic

estimation

of

the

seek

time

as

a

nonlinear

function

of

the

seek

distance

as

well

as

other

details

of

real

disks

-LSB-

-RSB-

In

the

simulation

mode

FIVE

makes

use

of

the

processoriented

simulation

library

CSIM

-LSB-

-RSB-

which

manages

the

bookkeeping

for

the

virtual

disks

-LRB-

e.g.

disk

queues

-RRB-

For

the

experiments

reported

here

we

used

yAdditional

approximation

techniques

to

further

decrease

the

space

overhead

are

described

in

-LSB-

-RSB-

When

memory

consumption

is

extremely

critical

one

can

even

employ

an

approximation

that

requires

only

keeping

the

values

of

tn

and

tn?k

and

thus

has

constant

space

overhead

independently

of

k.

a

simulated

parallel

disk

system

whose

parameters

are

described

in

Table

FIVE

allows

for

the

striping

of

files

on

an

individual

or

global

basis

and

incorporates

heuristic

algorithms

for

file

striping

allocation

and

dynamic

load

balancing

as

described

in

Sections

and

These

algorithms

can

be

invoked

on-line

i.e.

concurrently

with

regular

requests

to

existing

files

We

have

implemented

a

load

generator

that

can

generate

synthetic

workloads

according

to

specified

parameter

distributions

or

analyze

-LRB-

and

filter

-RRB-

existing

traces

and

feed

them

as

input

to

FIVE

For

the

performance

studies

reported

here

we

mostly

relied

on

synthetic

workloads

for

which

we

could

control

and

systematically

vary

all

relevant

parameters

A

representative

set

of

experiments

with

a

synthetic

workload

is

described

in

Subsection

We

also

report

on

disk

cooling

studies

using

two

trace-based

experiments

in

Subsection

Further

trace-based

performance

studies

with

FIVE

can

be

found

in

-LSB-

-RSB-

disks

capacity

of

one

disk

MBytes

block

size

KByte

capacity

of

the

disk

system

GBytes

track

size

blocks

revolutions

per

minute

rpm

tracks

per

cylinder

average

seek

time

ms

cylinders

per

disk

transfer

rate

per

disk

MBytes/s

Table

Hardware

characteristics

of

the

simulated

disk

system

Experiments

With

Synthetic

Workload

For

these

experiments

we

generated

a

set

of

files

and

two

types

of

workloads

one

with

a

uniform

access

pattern

and

the

second

with

a

skewed

access

pattern

as

we

shall

describe

in

more

detail

below

The

files

themselves

were

identical

for

both

workloads

and

in

both

cases

each

-LRB-

read

or

write

-RRB-

request

accessed

an

entire

file

The

file

sizes

were

hyperexponentially

distributed

such

that

each

file

belongs

to

one

of

three

different

classes

with

certain

mean

values

-LRB-

and

exponential

distribution

of

file

sizes

within

each

class

-RRB-

Files

of

class

A

had

a

mean

size

of

KBytes

files

of

class

B

had

a

mean

size

of

KBytes

and

files

of

class

C

had

a

mean

size

of

KBytes

Class

C

files

were

not

accessed

in

the

generated

workload

they

represent

passive

data

that

occupies

disk

space

and

thus

influence

seek

times

Class

A

files

represent

relatively

small

data

objects

e.g.

simple

HTML

documents

on

the

WWW

Class

B

files

on

the

other

hand

represent

relatively

large

multimedia

data

objects

The

important

point

here

is

that

the

workload

covered

a

wide

spectrum

of

request

sizes

which

we

consider

to

be

a

particular

challenge

of

advanced

applications

such

as

HTTP

servers

multimedia

information

systems

and

object-oriented

database

systems

In

both

workloads

we

assigned

the

same

probability

of

selection

to

files

from

the

two

classes

A

and

B.

Table

summarizes

the

common

characteristics

of

both

synthetic

workloads

of

files

of

class

A

of

files

of

class

B

fraction

of

files

of

class

A

fraction

of

files

of

class

B

average

size

of

file

class

A

KBytes

average

size

of

file

class

B

KBytes

overall

average

request

size

KBytes

standard

deviation

of

request

size

KBytes

read

fraction

Table

Characteristics

of

the

synthetic

workload

Workload

With

Uniform

Access

Frequencies

In

this

subsection

we

consider

a

workload

with

uniform

access

frequencies

read

and

write

accesses

are

generated

to

file

classes

of

type

A

or

B

such

that

each

file

within

a

class

has

the

same

probability

of

selection

We

generated

a

sequence

of

million

file

requests

with

exponentially

distributed

interarrival

times

We

compared

first

the

response

time

of

five

different

striping

strategies

namely

a

file-specific

one

-LRB-

Opt

-RRB-

and

four

global

strategies

-LRB-

Gopt

Block

Track

Cylinder

-RRB-

under

light

load

-LRB-

i.e.

an

arrival

rate

of

request/second

-RRB-

so

that

queueing

effects

were

negligible

These

striping

strategies

are

Opt

files

are

partitioned

based

on

the

first

step

of

the

heuristic

approach

described

in

Section

that

minimizes

response

time

in

singleuser

mode

Gopt

the

striping

unit

for

each

file

is

the

global

optimum

of

KBytes

which

was

determined

by

using

the

first

step

of

the

heuristic

method

of

Section

under

the

assumption

that

all

files

have

the

same

average

request

size

R

KBytes

Block

the

striping

unit

for

each

file

is

a

block

-LRB-

i.e.

KByte

-RRB-

Track

the

striping

unit

for

each

file

is

a

track

-LRB-

i.e.

KBytes

-RRB-

Cyl

the

striping

unit

for

each

file

is

a

cylinder

-LRB-

i.e.

KBytes

-RRB-

Note

that

for

this

first

set

of

experiments

we

assumed

a

light

load

hence

the

striping

unit

for

Opt

and

Gopt

was

computed

without

regard

to

throughput

and

queueing

delay

considerations

Table

shows

the

average

response

time

for

the

five

different

striping

methods

these

performance

figures

are

further

broken

down

into

different

categories

of

request

sizes

in

Table

The

Opt

method

outperforms

all

other

methods

for

almost

all

request

size

categories

However

the

advantage

over

Block

striping

is

marginal

at

best

The

improvement

is

significant

only

for

request

sizes

up

to

-LRB-

or

possibly

-RRB-

KBytes

For

larger

requests

the

latency

of

the

slowest

disk

rapidly

approaches

the

maximum

latency

under

both

Opt

and

Block

so

that

the

aggressive

intra-request

parallelism

of

the

Block

method

does

not

incur

an

additional

penalty

once

the

degree

of

parallelism

exceeds

a

certain

number

As

we

will

see

below

the

Block

method

exhibits

severe

drawbacks

when

the

request

arrival

rate

is

increased

so

that

disk

arm

contention

and

the

resulting

queueing

delays

become

a

factor

whereas

the

Opt

method

scales

much

better

with

increasing

load

Compared

to

the

Track

and

Cyl

methods

Opt

achieves

significant

improvements

in

the

order

of

percent

-LRB-

in

the

case

of

Track

-RRB-

for

medium

to

large

requests

between

and

KBytes

For

very

large

requests

all

methods

-LRB-

except

Cyl

-RRB-

spread

a

file

across

all

disk

so

that

the

performance

differences

eventually

become

negligible

when

the

request

size

is

further

increased

beyond

MByte

The

global

striping

method

Gopt

turned

out

to

be

very

competitive

to

the

file-specific

striping

method

Opt

the

advantage

of

Opt

is

more

or

less

negligible

throughout

the

spectrum

of

request

sizes

We

also

compared

these

two

methods

with

Gbest

the

best

possible

global

striping

strategy

whose

striping

unit

was

found

through

exhaustive

trials

For

this

particular

workload

the

Gbest

method

has

a

striping

unit

of

KBytes

and

its

performance

was

almost

identical

to

that

of

Opt

and

Gopt

So

although

file-specific

striping

did

not

prove

to

be

truly

superior

to

global

striping

in

these

experiments

the

positive

conclusion

from

these

light-load

experiments

is

that

our

heuristic

optimization

method

did

indeed

approximate

the

real

optimum

very

well

Opt

Gopt

-LRB-

KB

-RRB-

Block

Track

Cyl

Table

Average

response

time

in

milliseconds

of

the

synthetic

workload

under

light

load

-LRB-

-RRB-

In

order

to

take

into

account

throughput

requirements

and

queueing

delays

we

performed

a

second

set

of

experiments

in

which

we

varied

the

request

request

size

-LSB-

KB

-RSB-

Opt

Gopt

-LRB-

-RRB-

Block

Track

Cyl

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

Table

Average

response

time

in

milliseconds

for

different

request

sizes

of

the

synthetic

workload

under

light

load

-LRB-

-RRB-

arrival

rate

For

this

set

of

experiments

we

also

considered

two

additional

striping

strategies

namely

the

optimal

file-specific

striping

unit

is

computed

with

the

additional

constraint

that

a

request

arrival

rate

of

must

be

supported

Accordingly

files

are

partitioned

subject

to

the

constraint

that

the

average

degree

of

intra-request

parallelism

is

bounded

by

-LRB-

as

computed

by

the

heuristics

described

in

Section

-RRB-

the

optimal

global

striping

unit

is

computed

for

a

request

arrival

rate

of

The

corresponding

striping

unit

size

is

KBytes

Table

shows

the

average

response

times

of

the

various

striping

methods

as

a

function

of

the

request

arrival

rate

which

was

varied

from

up

to

requests

per

second

Note

that

although

the

figures

show

explicitly

only

response

time

a

fast

growing

curve

for

response

time

implies

that

beyond

a

relatively

small

value

for

the

arrival

rate

the

throughput

reaches

saturation

This

also

explains

the

entries

in

Table

they

denote

those

experiments

where

the

arrival

rate

exceeded

the

sustainable

throughput

and

thus

led

to

excessive

queueing

and

a

continuously

growing

backlog

of

requests

As

Table

shows

the

Opt

method

scales

up

with

increasing

arrival

rate

much

better

than

Block

striping

However

for

sufficiently

high

arrival

rate

Opt

is

clearly

outperformed

by

Track

and

Cyl

striping

the

reason

being

that

the

latter

two

methods

employ

lower

degrees

of

intra-request

parallelism

and

can

thus

sustain

higher

load

The

figures

also

show

the

trend

that

Cyl

will

eventually

pass

Track

as

it

is

even

more

conservative

in

terms

of

parallelism

and

resource

consumption

The

striping

methods

that

are

specifically

tuned

for

a

particular

arrival

rate

outperform

both

Track

and

Cyl

by

almost

a

factor

of

two

-LRB-

in

the

case

of

-RRB-

This

demonstrates

very

nicely

the

need

for

application-specific

tuning

of

striping

units

We

also

determined

through

exhaustive

trials

the

best

possible

striping

units

for

the

Gbest

method

under

different

arrival

rates

For

the

response

time

of

Gbest

was

approximately

ms

and

this

was

obtained

for

a

global

striping

unit

of

KBytes

We

note

however

that

such

a

tuning

method

that

is

based

on

exhaustive

trials

is

completely

infeasible

in

practice

Thus

the

fact

that

both

the

Opt

and

the

Gopt

methods

approached

the

real

optima

within

approximately

to

percent

is

indeed

a

successful

result

and

demonstrates

the

viability

of

our

tuning

heuristics

Opt

Gopt

Block

Track

Cyl

-LRB-

-RRB-

-LRB-

-RRB-

Table

Average

response

time

in

milliseconds

for

the

synthetic

workload

as

a

function

of

the

request

arrival

rate

In

the

above

experiment

the

Opt

methods

achieved

only

very

small

improvements

over

the

corresponding

Gopt

methods

This

almost

negligible

advantage

of

Opt

over

Gopt

does

not

seem

to

justify

the

increased

software

complexity

of

file-specific

striping

However

file-specific

striping

allows

for

incremental

restriping

of

individual

files

when

changing

workload

characteristics

require

higher

I/O

rates

or

data

rates

for

some

crucial

files

A

global

striping

unit

strategy

does

not

support

this

type

of

reconfiguration

Thus

for

global

striping

a

change

of

the

striping

unit

requires

unloading

all

files

reinitializing

the

disk

system

with

the

new

striping

unit

and

reloading

the

data

This

costly

procedure

leads

to

a

significant

downtime

of

the

system

For

this

reason

we

still

believe

that

file-specific

striping

is

an

essential

requirement

for

data

management

in

parallel

disk

systems

Workload

With

Skewed

Access

Frequencies

In

order

to

study

the

influence

of

data

access

skew

and

the

effectiveness

of

our

disk

cooling

procedure

we

have

modified

the

synthetic

workload

of

the

previous

subsection

so

that

the

distribution

of

file

access

frequencies

followed

a

Zipf-like

curve

-LRB-

everything

else

was

identical

to

the

previous

setup

-RRB-

Thus

if

the

files

are

numbered

from

to

N

the

probability

of

accessing

a

file

numbered

i

with

i

N

is

given

by

the

formula

-LSB-

-RSB-

P

rob

-LSB-

i

s

-RSB-

-LRB-

s

N

-RRB-

log

-LRB-

X

-RRB-

log

-LRB-

Y

-RRB-

-LRB-

-RRB-

where

X

and

Y

are

parameters

that

were

set

to

and

respectively

The

parameter

N

denotes

the

number

of

active

files

i.e.

files

in

classes

A

and

B

which

is

set

to

in

our

experiments

This

probability

distribution

results

in

a

self-similar

skewed

access

pattern

where

a

fraction

X

of

the

requests

refers

to

a

fraction

Y

of

the

files

and

this

skew

is

recursively

repeated

within

the

fraction

X

of

hot

files

Such

skew

patterns

are

common

in

many

OLTP

and

database

applications

and

they

have

been

observed

for

WWW

servers

as

well

-LSB-

-RSB-

In

order

to

study

the

effects

of

load

balancing

in

isolation

we

did

not

perform

any

caching

of

the

data

in

these

experiments

Note

that

load

balancing

is

still

a

crucial

problem

even

if

caching

is

used

Caching

would

keep

the

hottest

blocks

in

main

memory

but

the

remaining

blocks

can

still

exhibit

a

significant

access

skew

Table

shows

the

average

response

time

results

for

this

experiment

as

a

function

of

the

arrival

rate

We

considered

three

different

striping

strategies

namely

Track

and

Cyl

We

do

not

show

explicitly

the

results

for

the

strategies

Block

and

the

Block

strategy

could

sustain

only

a

throughput

of

about

requests

per

second

and

started

thrashing

at

this

point

while

the

performance

of

the

strategy

was

almost

identical

to

that

of

All

files

were

pre-allocated

based

on

a

round-robin

scheme

and

we

compared

the

case

without

cooling

against

the

case

with

cooling

switched

on

The

latter

case

is

denoted

by

the

C

suffix

in

Table

A

cooling

step

was

attempted

every

seconds

-LRB-

i.e.

equivalently

every

regular

requests

-RRB-

the

migration

units

were

entire

extents

and

the

load

imbalance

threshold

ffi

was

set

to

percent

-LRB-

see

Section

-RRB-

The

response

time

figures

demonstrate

that

access

skew

does

have

a

disastrous

effect

on

performance

unless

it

is

counteracted

by

load

balancing

For

example

at

an

arrival

rate

of

requests

per

second

the

average

response

time

of

Track

striping

without

cooling

degrades

by

a

factor

of

compared

to

the

workload

with

uniform

access

frequencies

The

underlying

reason

is

that

under

the

skewed

load

the

hottest

disk

had

a

much

higher

utilization

-LRB-

and

corresponding

average

queue

length

-RRB-

than

the

overall

disk

system

and

thus

formed

a

premature

bottleneck

at

the

hottest

disk

had

a

utilization

of

and

an

average

queue

length

of

while

the

average

disk

utilization

and

queue

length

-LRB-

averaged

over

all

disks

-RRB-

were

and

respectively

-LRB-

under

Track

striping

-RRB-

In

fact

all

methods

without

cooling

started

thrashing

at

an

arrival

rate

of

requests

per

second

or

earlier

The

cooling

procedure

was

able

to

reduce

the

utilization

and

the

average

queue

length

Track

Cyl

Track-C

Cyl-C

-LRB-

-RRB-

Table

Average

response

time

in

milliseconds

for

the

skewed

synthetic

workload

as

a

function

of

the

request

arrival

rate

of

the

hottest

disk

down

to

and

respectively

at

and

could

thus

improve

the

average

response

time

significantly

When

approaching

the

thrashing

region

the

response

times

of

any

striping

strategy

with

cooling

switched

on

are

an

order

of

magnitude

lower

compared

with

the

same

strategy

with

cooling

turned

off

Note

that

cooling

does

incur

a

certain

overhead

by

migrating

extents

between

disks

This

leads

to

a

small

increase

of

the

overall

disk

utilization

and

this

is

why

the

cooling

methods

exhibit

a

slightly

higher

response

time

than

the

no-cooling

methods

under

light

load

However

when

the

extra

load

due

to

cooling

becomes

a

critical

factor

cooling

is

inactivated

automatically

as

described

in

Section

An

analysis

of

the

invocation

frequency

distribution

of

cooling

steps

over

the

duration

of

an

experiment

shows

that

the

cooling

frequency

is

high

in

the

first

tenth

of

the

experiment

and

as

soon

as

the

load

is

sufficiently

balanced

-LRB-

as

estimated

by

the

heat

bookkeeping

-RRB-

cooling

is

invoked

only

very

infrequently

due

to

occasional

load

fluctuations

that

exceed

the

imbalance

threshold

Among

the

three

cooling

variants

that

are

shown

in

Table

the

Gopt

C

method

showed

significant

advantages

over

Track-C

striping

under

high

load

with

response

time

improvements

up

to

a

factor

of

two

This

demonstrates

that

although

load

balancing

and

striping

are

orthogonal

strategies

they

are

not

independent

rather

well

tuned

striping

units

and

the

cooling

procedure

exhibit

synergetic

effects

Note

that

under

the

skewed

load

the

Gopt

method

without

cooling

could

not

sustain

a

throughput

of

as

all

our

heuristic

calculations

for

the

derivation

of

striping

units

are

based

on

uniform

access

frequencies

-LRB-

i.e.

overly

optimistic

assumptions

-RRB-

Track

striping

on

the

other

hand

achieves

a

better

load

balance

because

of

its

finer

striping

units

but

is

still

much

inferior

to

the

case

with

both

tuned

striping

units

and

cooling

In

summary

application-specifically

tuned

striping

in

combination

with

cooling

shows

significant

performance

advantages

over

conventional

methods

Not

surprisingly

load

imbalance

is

only

an

issue

under

high

load

when

queueing

delays

start

becoming

a

factor

One

may

argue

that

an

easy

cure

against

load

imbalance

thus

is

to

keep

disk

utilization

low

However

for

many

applications

this

implies

unnecessarily

high

costs

as

their

performance

requirements

could

be

met

with

fewer

disks

at

higher

utilization

Furthermore

although

system

administration

rules

of

thumb

dictate

that

the

disk

utilization

should

generally

be

kept

below

percent

this

is

often

impossible

during

load

peaks

or

when

user

demands

grow

faster

than

one

can

purchase

additional

disks

In

fact

it

is

often

exactly

during

load

peaks

e.g.

the

Monday

morning

rush

hour

for

retail

banking

or

the

hours

right

after

an

important

sports

event

for

a

WWW

server

when

good

response

time

matters

most

Experiments

With

Application

Traces

To

study

the

viability

of

the

developed

tuning

procedures

in

a

realistic

application

setting

we

also

conducted

extensive

experiments

based

on

block

access

traces

from

a

variety

of

applications

including

on-line

transaction

processing

file

systems

office

document

management

and

WWW

servers

Most

of

these

experiments

confirmed

the

results

of

the

previous

subsection

However

while

such

traces

capture

several

essential

characteristics

of

real-life

application

workloads

-LRB-

e.g.

workload

evolution

over

time

including

transient

load

peaks

-RRB-

one

has

to

be

extremely

careful

about

generalizing

trace-based

results

Traces

constitute

short-term

snapshots

with

certain

peculiarities

that

are

not

necessarily

of

fundamental

nature

For

this

reason

we

preferred

deriving

our

basic

performance

results

from

a

precisely

controllable

synthetic

workload

as

discussed

in

the

previous

subsection

and

we

restrict

ourselves

in

this

subsection

to

two

sample

results

that

were

obtained

with

a

WWW

server

trace

and

a

trace

from

a

bank

's

on-line

transaction

processing

system

World-Wide-Web

Server

This

study

is

based

on

a

trace

that

was

recorded

with

the

httpd

logging

facility

on

the

WWW

server

of

the

UC

Museum

of

Paleontology

at

Berkeley

over

a

time

period

of

hours

Note

that

the

fact

that

the

requests

were

traced

at

the

server

site

automatically

factors

out

-LRB-

client

-RRB-

caching

The

trace

contains

read

accesses

to

an

entirety

of

HTML

and

other

files

with

heavily

skewed

access

frequencies

The

average

request

size

was

KBytes

and

the

standard

deviation

of

the

requests

size

distribution

was

KBytes

We

studied

this

trace

under

a

spectrum

of

load

levels

This

was

done

by

speeding

up

the

arrivals

in

the

original

trace

in

the

following

way

Consider

two

requests

ri

and

ri

in

the

original

trace

which

have

an

interarrival

time

of

ffii

Using

a

speed-up

factor

of

ff

the

interarrival

time

between

the

requests

becomes

ffii

ff

Thus

in

more

general

terms

if

the

original

trace

has

an

average

interarrival

time

of

a

trace

with

speed-up

factor

ff

has

an

average

interarrival

time

of

=-LRB-

ff

-RRB-

Note

that

this

method

of

speeding

up

a

trace

albeit

somewhat

speculative

preserves

all

access

characteristics

of

the

original

workload

other

than

its

arrival

rate

particularly

the

relative

interarrival

times

between

requests

are

preserved

which

is

essential

to

capture

load

bursts

The

only

case

where

the

speed-up

transformation

would

seriously

distort

the

workload

is

when

a

large

number

of

consecutive

requests

are

correlated

and

must

have

a

certain

interarrival

time

But

this

case

is

rather

unlikely

given

that

a

WWW

server

trace

is

typically

based

on

a

high

number

of

concurrent

users

Because

of

the

small

average

request

size

and

the

moderate

variance

of

request

sizes

tuning

the

striping

unit

was

not

really

an

issue

for

this

workload

Rather

the

challenge

in

this

trace

was

to

cope

well

with

the

access

skew

in

combination

with

the

dynamic

load

fluctuations

So

we

concentrated

ourselves

on

the

impact

of

cooling

and

compared

a

round-robin

allocation

for

a

striping

unit

of

one

track

-LRB-

i.e.

KBytes

-RRB-

without

cooling

labeled

Track

versus

the

case

with

cooling

labeled

Track-C

Cooling

was

invoked

every

seconds

of

the

original

time

scale

-LRB-

or

equivalently

every

seconds

of

the

accelerated

trace

-RRB-

with

entire

extents

as

migration

units

and

an

imbalance

threshold

of

ffi

Table

shows

the

average

response

time

of

Track

versus

Track-C

as

a

function

of

the

acceleration

factor

ff

ff

Track

Track-C

Table

Average

response

time

in

milliseconds

for

the

WWW

workload

as

a

function

of

the

request

arrival

rate

Cooling

exhibits

noticeable

performance

even

under

medium

load

and

dramatically

improves

response

time

by

an

order

of

magnitude

for

the

highest

measured

load

For

ff

the

average

disk

utilization

was

percent

and

the

utilization

of

the

hottest

disk

was

percent

without

cooling

With

cooling

the

average

utilization

increased

slightly

up

to

percent

because

of

the

additional

load

incurred

by

data

migrations

but

the

utilization

of

the

hottest

disk

was

reduced

down

to

percent

which

accounted

for

the

dramatic

performance

gain

Note

that

an

average

utilization

of

percent

appears

to

be

a

very

light

load

however

one

has

to

take

into

account

that

the

load

fluctuates

heavily

over

time

with

very

long

disk

queues

built

up

during

the

load

peaks

In

terms

of

average

disk

queue

lengths

the

improvement

by

cooling

was

even

more

impressive

without

cooling

the

average

queue

length

of

the

hottest

disk

-LRB-

averaged

over

all

points

of

time

when

a

request

was

enqueued

-RRB-

was

whereas

with

cooling

this

measure

was

-LRB-

i.e.

request

in

service

and

an

expected

value

of

for

the

number

of

requests

that

wait

in

the

queue

-RRB-

This

effect

is

illustrated

in

Figure

which

shows

the

response

time

and

the

cooling

frequency

as

they

vary

over

the

duration

of

the

experiment

for

the

case

of

ff

The

improvement

of

response

time

due

to

cooling

even

exceeds

a

factor

of

during

the

load

peak

On-line

Transaction

Processing

A

second

study

with

real

application

workloads

was

based

on

an

I/O

trace

from

the

OLTP

system

of

a

large

Swiss

bank

-LRB-

Union

Bank

of

Switzerland

-RRB-

The

database

for

this

study

consists

of

files

with

a

total

size

of

GBytes

The

I/O

trace

contains

approximately

I/O

requests

to

these

files

recorded

during

one

hour

As

in

a

typical

OLTP

application

most

requests

read

or

write

a

single

block

-LRB-

of

size

KBytes

in

this

application

-RRB-

the

average

request

size

is

approximately

KBytes

with

low

variance

Thus

this

workload

does

not

warrant

any

specific

tuning

of

the

striping

unit

so

that

we

chose

Track

striping

as

the

partitioning

method

All

files

were

allocated

using

a

round-robin

scheme

The

workload

exhibits

heavily

skewed

access

frequencies

both

across

files

and

within

the

hot

files

In

addition

the

trace

contains

significant

fluctuations

in

the

access

frequencies

and

in

the

overall

arrival

rate

of

requests

We

compared

the

performance

of

round-robin

placement

without

cooling

to

round-robin

allocation

augmented

with

the

cooling

procedure

The

cooling

method

improved

the

average

response

time

of

the

requests

by

approximately

a

factor

of

under

high

load

As

with

the

WWW

experiment

we

measured

response

time

versus

different

speed-up

factors

of

the

arrival

rate

The

results

in

Figure

are

based

on

an

arrival

rate

speed-up

factor

of

As

Figure

shows

the

cooling

method

could

not

improve

response

time

in

the

initial

light-load

phase

since

the

load

imbalance

of

the

vanilla

method

did

not

yet

incur

any

severe

queueing

However

the

cooling

method

did

collect

heat

statistics

during

this

phase

This

enabled

the

cooling

method

to

rebalance

the

disk

load

by

data

migration

Then

during

the

load

peak

-LRB-

represented

in

Figure

by

the

sharp

increase

of

response

time

-RRB-

the

cooling

method

achieved

a

response

time

improvement

by

a

factor

of

Note

that

many

OLTP

applications

have

soft

response

time

constraints

such

as

ensuring

a

certain

response

time

for

percent

of

the

transactions

Thus

it

is

crucial

to

guarantee

acceptable

response

time

even

during

load

peaks

Figure

also

shows

the

frequency

of

the

data

migration

steps

invoked

by

our

cooling

method

varying

over

time

The

figure

shows

that

our

algorithm

was

careful

enough

so

as

not

to

initiate

too

many

cooling

steps

during

the

high-load

phases

rather

the

data

migrations

were

performed

mostly

during

the

low-load

phases

thus

improving

the

load

balance

for

the

next

high-load

phase

at

low

cost

This

robustness

is

achieved

by

explicitly

trading

off

the

benefit

of

cooling

versus

its

additional

cost

as

discussed

in

Section

Conclusion

We

have

demonstrated

the

need

for

tuning

the

data

placement

in

parallel

disk

systems

and

we

have

presented

various

tuning

heuristics

for

data

partitioning

data

allocation

and

load

balancing

The

feasibility

of

the

developed

methods

has

been

shown

in

a

number

of

performance

experiments

including

simulations

based

on

real-life

traces

We

have

developed

an

extended

optimization

procedure

for

file

striping

that

takes

into

account

explicitly

throughput

requirements

and

queueing

delays

and

in

the

process

we

have

developed

an

analytical

approximation

to

the

well

known

fork-join

problem

-LSB-

-RSB-

in

the

specific

setting

of

parallel

disk

systems

We

have

shown

that

our

procedure

for

tuning

the

striping

unit

-LRB-

s

-RRB-

of

files

is

a

very

effective

method

for

workloads

with

large

variations

in

request

sizes

Such

workloads

arise

for

example

in

combined

OLTP/decision-support

applications

multimedia

information

systems

and

many

other

advanced

database

applications

Our

extended

optimization

that

considers

both

steps

of

our

heuristic

outperforms

all

other

striping

methods

for

the

specified

load

value

or

higher

while

being

competitive

also

for

lighter

load

While

file-specific

striping

at

best

provides

marginal

performance

gains

over

a

properly

chosen

global

striping

unit

we

nevertheless

believe

that

filespecific

striping

is

important

as

a

prerequisite

for

incremental

repartitioning

of

files

Incremental

repartitioning

is

crucial

in

order

to

cope

with

evolving

performance

requirements

and

in

order

to

support

system

scalability

For

example

when

the

throughput

requirements

of

an

application

increase

we

can

repartition

merely

the

hottest

files

in

order

to

meet

the

new

throughput

goal

which

is

possible

since

our

approach

supports

file-specific

striping

units

Similarly

if

more

disks

are

added

to

a

system

restriping

of

the

most

crucial

files

allows

us

to

take

advantage

of

the

additional

resources

The

methods

for

data

allocation

and

redistribution

complement

the

data

partitioning

objective

of

minimizing

queueing

delays

at

the

disks

under

heavy

load

by

distributing

the

load

across

the

disks

as

evenly

as

possible

and

by

selectively

redistributing

the

load

dynamically

by

means

of

disk

cooling

steps

Since

our

optimization

procedure

for

data

partitioning

is

based

on

uniform

access

frequencies

the

combination

of

appropriately

tuned

striping

and

disk

cooling

is

necessary

to

deal

with

skews

in

data

access

By

coupling

these

two

procedures

our

experiments

have

shown

that

at

high

loads

we

can

obtain

substantial

performance

gains

The

dynamic

load

redistribution

procedure

has

been

shown

to

be

efficient

and

robust

i.e.

it

performs

disk

cooling

at

a

small

cost

and

very

selectively

i.e

only

during

periods

of

low

activity

We

observe

here

that

our

procedures

for

data

allocation

and

redistribution

can

be

integrated

with

techniques

for

clustering

the

hottest

files

-LRB-

extents

-RRB-

on

each

disk

in

its

center

-LSB-

-RSB-

and

with

disk

scheduling

algorithms

that

reorder

the

requests

in

a

queue

-LRB-

e.g.

an

elevator

algorithm

-RRB-

Our

future

work

will

be

centered

around

the

following

two

major

issues

combining

the

developed

data

placement

methods

with

techniques

for

providing

fault

tolerance

and

high

availability

and

generalizing

our

approach

towards

shared-nothing

parallel

database

systems

and

systems

based

on

networks

of

workstations

Our

placement

methods

are

orthogonal

to

the

proposed

fault

tolerance

techniques

in

that

they

can

be

combined

in

a

straightforward

manner

with

arbitrary

variants

of

either

mirroring

-LRB-

e.g.

mirrored

disks

interleaved

declustering

or

chained

declustering

-LSB-

-RSB-

-RRB-

or

error-correcting

codes

-LRB-

e.g.

parity

groups

of

some

type

-LSB-

-RSB-

-RRB-

or

simply

conventional

logging

-LSB-

-RSB-

However

the

placement

of

data

replicas

or

error-correcting

information

does

itself

provide

additional

degrees

of

freedom

that

should

be

taken

into

account

by

an

integrated

approach

in

order

to

ensure

the

best

possible

performance

and

availability

for

given

system

costs

-LSB-

-RSB-

In

order

to

generalize

our

approach

to

a

general

shared-nothing

parallel

database

system

we

need

to

consider

the

impact

of

communication

and

CPU

costs

in

addition

to

the

disk

I/O

service

time

For

the

partitioning

problem

the

optimal

partition

size

-LRB-

e.g.

the

interval

width

in

an

interleaved

rangepartitioning

scheme

for

relational

data

-LSB-

-RSB-

-RRB-

would

again

be

derived

from

the

optimal

degree

of

parallelism

in

analogy

to

our

approach

for

striping

However

the

operations

under

consideration

are

more

complex

-LRB-

e.g.

relational

operators

such

as

selection

or

join

-RRB-

and

the

performance

for

a

given

degree

of

parallelism

depends

also

on

communication

overhead

and

startup

costs

-LRB-

thus

requiring

a

generalization

of

our

notion

of

latency

-RRB-

as

well

as

on

the

operations

CPU

time

consumption

-LRB-

hence

requiring

a

generalization

of

our

notion

of

transfer

time

-RRB-

In

all

these

considerations

an

underlying

assumption

is

that

the

system

consists

of

homogeneous

processing

nodes

-LRB-

e.g.

the

same

disk

type

is

used

in

the

entire

system

-RRB-

A

further

even

more

challenging

step

would

be

to

consider

also

heterogeneous

systems

where

the

processing

nodes

can

differ

in

their

performance

characteristics

-LRB-

processor

speed

memory

size

disk

storage

and

performance

capacity

-RRB-

For

example

networks

of

workstations

also

known

as

NOW

-LSB-

-RSB-

are

evolving

as

a

paradigm

for

high

performance

computing

In

order

to

make

NOW

a

viable

approach

for

large-scale

data

management

it

is

crucial

to

develop

appropriate

self-tuning

and

self-reliant

data

placement

and

storage

techniques

A

first

approach

along

these

lines

with

specific

consideration

to

load

balancing

is

presented

in

-LSB-

-RSB-

References

-LSB-

-RSB-

T.E.

Anderson

D.E.

Culler

D.A.

Patterson

and

the

NOW

Team

A

Case

for

NOW

-LRB-

Networks

of

Workstations

-RRB-

to

appear

in

IEEE

Micro

-LSB-

-RSB-

B.

Awerbuch

Y.

Bartal

A.

Fiat

Competitive

Distributed

File

Allocation

ACM

Symposium

on

Theory

of

Computing

-LSB-

-RSB-

Y.

Azar

J.

Naor

R.

Rom

The

Competitiveness

of

Online

Assignment

ACM/SIAM

Symposium

on

Discrete

Algorithms

-LSB-

-RSB-

S.

Aky?urek

K.

Salem

Adaptive

Block

Rearrangement

ACM

Transactions

on

Computer

Systems

Vol

No.

pp.

-LSB-

-RSB-

Y.

Bartal

A.

Fiat

H.

Karloff

R.

Vohra

New

Algorithms

for

an

Ancient

Scheduling

Problem

Proceedings

of

the

ACM

Symposium

on

Theory

of

Computing

pp.

-LSB-

-RSB-

Y.

Bartal

A.

Fiat

Y.

Rabani

Competitive

Algorithms

for

Distributed

Data

Management

ACM

Symposium

on

Theory

of

Computing

-LSB-

-RSB-

A.

Bestavros

Demand-based

Document

Dissemination

to

Reduce

Traffic

and

Balance

Load

in

Distributed

Information

Systems

Proceedings

of

the

IEEE

Symposium

on

Parallel

and

Distributed

Processing

-LSB-

-RSB-

D.

Bitton

and

J.N.

Gray

Disk

Shadowing

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pp.

-LCB-

-LSB-

-RSB-

D.D.

Chamberlin

F.B.

Schmuck

Dynamic

Data

Distribution

-LRB-

-RRB-

in

a

Shared-Nothing

Multiprocessor

Data

Store

International

Conference

on

Very

Large

Data

Bases

Vancouver

pp.

-LCB-

-LSB-

-RSB-

P.M.

Chen

E.K.

Lee

G.A.

Gibson

R.H.

Katz

and

D.A.

Patterson

RAID

High-Performance

Reliable

Secondary

Storage

ACM

Computing

Surveys

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

P.M.

Chen

E.K.

Lee

Striping

in

a

RAID

Level

Disk

Array

ACM

SIGMETRICS

Conference

pp.

-LSB-

-RSB-

P.M.

Chen

and

D.A.

Patterson

Maximizing

Performance

in

a

Striped

Disk

Array

Proceedings

of

the

International

Symposium

on

Computer

Architecture

-LRB-

SIGARCH

-RRB-

pp.

-LCB-

-LSB-

-RSB-

S.

Chen

and

D.

Towsley

The

Design

and

Evaluation

of

RAID

and

Parity

Striping

Disk

Array

Architectures

Journal

of

Parallel

and

Distributed

Computing

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

G.

Copeland

W.

Alexander

E.

Boughter

and

T.

Keller

Data

Placement

in

Bubba

Proceedings

of

the

SIGMOD

International

Conference

on

Management

of

Data

pp.

-LCB-

-LSB-

-RSB-

G.

Copeland

and

T.

Keller

A

Comparison

of

High-Availability

Media

Recovery

Techniques

Proceedings

of

the

SIGMOD

International

Conference

on

Management

of

Data

pp.

-LCB-

-LSB-

-RSB-

G.

Copeland

T.

Keller

and

M.

Smith

Database

Buffer

and

Disk

Configuring

and

the

Battle

of

the

Bottlenecks

International

Workshop

on

High

Performance

Transaction

Systems

-LSB-

-RSB-

E.G.

Coffman

Jr.

M.R.

Garey

D.S.

Johnson

An

Application

of

BinPacking

to

Multiprocessor

Scheduling

SIAM

Journal

of

Computing

Vol

No.

pp.

-LSB-

-RSB-

D.J.

DeWitt

and

J.N.

Gray

Parallel

Database

Systems

The

Future

of

High

Performance

Database

Systems

Communications

of

the

ACM

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

W.

Dowdy

and

D.V.

Foster

Comparative

Models

of

the

File

Assignment

Problem

ACM

Computing

Surveys

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

Du

H.C.

Sobolewski

J.S.

Disk

Allocation

for

Cartesian

Product

Files

on

Multiple

Disk

Systems

ACM

Transactions

on

Database

Systems

Vol

No.

pp.

-LSB-

-RSB-

C.

Faloutsos

and

D.

Metaxas

Disk

Allocation

Methods

Using

Error

Correcting

Codes

IEEE

Transactions

on

Computers

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

U.

Faigle

W.

Kern

G.

Turan

On

the

Performance

of

On-line

Algorithms

for

Particular

Problems

Acta

Cybernetica

Vol

pp

-LSB-

-RSB-

L.

Fatto

S.

Hahn

Two

Parallel

Queues

Created

By

Arrivals

With

Two

Demands

I

SIAM

Journal

of

Applied

Mathematics

Vol

pp

-LSB-

-RSB-

L.

Fatto

Two

Parallel

Queues

Created

By

Arrivals

With

Two

Demands

II

SIAM

Journal

of

Applied

Mathematics

Vol

pp

-LSB-

-RSB-

G.R.

Ganger

B.L.

Worthington

R.Y.

Hou

Y.N.

Patt

Disk

Arrays

High-Performance

High-Reliability

Storage

Subsystems

IEEE

Computer

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

M.R.

Garey

D.S.

Johnson

Computers

and

Intractability

W.H.

Freeman

and

Company

-LSB-

-RSB-

Ghandeharizadeh

S.

DeWitt

D.J.

A

Multiuser

Performance

Analysis

of

Alternative

Declustering

Strategies

IEEE

International

Conference

on

Data

Engineering

Los

Angeles

-LSB-

-RSB-

Ghandeharizadeh

S.

DeWitt

D.J.

Hybrid-range

Partitioning

Strategy

A

New

Declustering

Strategy

for

Multiprocessor

Database

Machines

International

Conference

on

Very

Large

Data

Bases

Brisbane

-LSB-

-RSB-

Ghandeharizadeh

S.

DeWitt

D.J.

MAGIC

A

Multiattribute

Declustering

Mechanism

for

Multiprocessor

Database

Machines

IEEE

Transactions

on

Parallel

and

Distributed

Systems

Vol

No.

pp

-LSB-

-RSB-

G.A.

Gibson

L.

Hellerstein

R.M.

Karp

R.H.

Katz

and

D.A.

Patterson

Failure

Correction

Techniques

for

Large

Disk

Arrays

Proceedings

of

the

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pp

-LCB-

-LSB-

-RSB-

G.A.

Gibson

Redundant

Disk

Arrays

Reliable

Parallel

Secondary

Storage

MIT

Press

-LSB-

-RSB-

R.L.

Graham

Bounds

on

Certain

Multiprocessing

Anomalies

SIAM

Journal

of

Applied

Mathematics

Vol

pp.

-LSB-

-RSB-

J.N.

Gray

B.

Horst

and

M.

Walker

Parity

Striping

of

Disk

Arrays

Low-Cost

Reliable

Storage

with

Acceptable

Throughput

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pp.

-LCB-

-LSB-

-RSB-

J.

Gray

A.

Reuter

Transaction

Processing

Concepts

and

Techniques

Morgan

Kaufmann

Publ.

-LSB-

-RSB-

H.

Hsiao

and

D.J.

DeWitt

A

Performance

Study

of

Three

HighAvailability

Data

Replication

Strategies

International

Journal

on

Distributed

and

Parallel

Databases

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

M.

Holland

and

G.A.

Gibson

Parity

Declustering

for

Continuous

Operation

in

Redundant

Disk

Arrays

Proceedings

of

the

International

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pp.

-LCB-

-LSB-

-RSB-

M.

Holland

G.A.

Gibson

D.P.

Siewiorek

Architectures

and

Algorithms

for

On-Line

Failure

Recovery

in

Redundant

Disk

Arrays

Distributed

and

Parallel

Databases

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

K.A.

Hua

C.

Lee

H.C.

Young

Data

Partitioning

for

Multicomputer

Database

Systems

A

Cell-based

Approach

Information

Systems

Vol

No.

pp.

-LSB-

-RSB-

R.

Jain

The

Art

of

Computer

Systems

Performance

Analysis

John

Wiley

Sons

-LSB-

-RSB-

D.R.

Karger

S.J.

Phillips

E.

Torng

A

Better

Algorithm

for

an

Ancient

Scheduling

Problem

ACM/SIAM

Symposium

on

Discrete

Algorithms

-LSB-

-RSB-

R.H.

Katz

G.A.

Gibson

and

D.A.

Patterson

Disk

System

Architectures

for

High

Performance

Computing

Proceedings

of

the

IEEE

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

R.H.

Katz

and

W.

Hong

The

Performance

of

Disk

Arrays

in

Shared

Memory

Database

Machines

Distributed

and

Parallel

Databases

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

M.Y.

Kim

Synchronized

Disk

Interleaving

IEEE

Transactions

on

Computers

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

M.Y.

Kim

and

S.

Pramanik

Optimal

File

Distribution

for

Partial

Match

Retrieval

Proceedings

of

the

SIGMOD

International

Conference

on

Management

of

Data

pp.

-LCB-

-LSB-

-RSB-

M.Y.

Kim

and

A.N.

Tantawi

Asynchronous

Disk

Interleaving

Approximating

Access

Delays

IEEE

Transactions

on

Computers

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

L.

Kleinrock

Queueing

Systems

John

Wiley

-LSB-

-RSB-

Knuth

D.E.

The

Art

of

Computer

Programming

Vol

Sorting

and

Searching

Addison-Wesley

-LSB-

-RSB-

E.K.

Lee

and

R.H.

Katz

An

Analytic

Performance

Model

of

Disk

Arrays

Proceedings

of

the

International

Conference

on

Measurement

and

Modeling

of

Computer

Systems

-LRB-

ACM

SIGMETRICS

-RRB-

pp.

-LCB-

-LSB-

-RSB-

Lee

L.-W.

Optimization

of

Load-Balanced

File

Allocation

Doctoral

Thesis

Department

of

Electrical

Engineering

and

Computer

Science

Northwestern

University

Evanston

Illinois

-LSB-

-RSB-

W.

Litwin

M.-A

Neimat

D.A.

Schneider

LH

Linear

Hashing

for

Distributed

Files

ACM

SIGMOD

Conference

Washington

-LSB-

-RSB-

M.

Livny

S.

Khoshafian

and

H.

Boral

Multi-Disk

Management

Algorithms

Proceedings

of

the

International

Conference

on

Measurement

and

Modeling

of

Computer

Systems

-LRB-

ACM

SIGMETRICS

-RRB-

pp.

-LCB-

-LSB-

-RSB-

J.C.S.

Lui

R.R.

Muntz

D.

Towsley

Computing

Performance

Bounds

for

Fork-Join

Queueing

Models

Technical

Report

Computer

Science

Department

UCLA

-LSB-

-RSB-

J.

Menon

Performance

of

Disk

Arrays

with

Read

and

Write

Caching

Distributed

and

Parallel

Databases

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

J.

Menon

and

J.

Cortney

The

Architecture

of

a

Fault-Tolerant

Cached

RAID

Controller

Proceedings

of

the

Symposium

on

Computer

Architecture

-LRB-

ACM

SIGARCH

-RRB-

pp.

-LCB-

-LSB-

-RSB-

C.

Mohan

H.

Pirahesh

W.G.

Tang

Y.

Wang

Parallelism

in

Relational

Database

Management

Systems

IBM

Systems

Journal

Vol

No.

pp.

-LSB-

-RSB-

J.

Menon

J.

Roche

and

J.

Kasson

Floating

Parity

and

Data

Disk

Arrays

Journal

of

Parallel

and

Distributed

Computing

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

R.R.

Muntz

and

J.C.S.

Lui

Performance

Analysis

of

Disk

Arrays

Under

Failure

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pp.

-LCB-

-LSB-

-RSB-

A.

Merchant

P.S.

Yu

Design

and

Modeling

of

Clustered

RAID

Proceedings

of

the

Annual

Symposium

on

Fault-Tolerant

Computing

pp.

-LCB-

-LSB-

-RSB-

A.

Merchant

P.S.

Yu

Analytic

Modeling

and

Comparisons

of

Striping

Strategies

for

Replicated

Disk

Arrays

IEEE

Transactions

on

Computers

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

K.

Mogi

M.

Kitsuregawa

Dynamic

Parity

Stripe

Reorganizations

for

Disk

Arrays

Proceedings

of

the

International

Conference

on

Parallel

and

Distributed

Information

Systems

Austin

-LSB-

-RSB-

K.

Mogi

M.

Kitsuregawa

Hot

Block

Clustering

for

Disk

Arrays

with

Dynamic

Striping

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

Zurich

-LSB-

-RSB-

R.

Nelson

and

A.N.

Tantawi

Approximate

Analysis

of

Fork/Join

Synchronization

in

Parallel

Queues

IEEE

Transactions

on

Computers

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

Y.N.

Patt

-LRB-

Guest

Editor

-RRB-

The

I/O

Subsystem

A

Candidate

for

Improvement

IEEE

Computer

Vol

No.

March

-LSB-

-RSB-

C.A.

Polyzois

A.

Bhide

and

D.M.

Dias

Disk

Mirroring

with

Alternating

Deferred

Updates

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pp.

-LCB-

-LSB-

-RSB-

D.A.

Patterson

G.A.

Gibson

and

R.H.

Katz

A

Case

for

Redundant

Arrays

of

Inexpensive

Disks

-LRB-

RAID

-RRB-

Proceedings

of

the

SIGMOD

International

Conference

on

Management

of

Data

pp.

-LCB-

-LSB-

-RSB-

A.L.N.

Reddy

J.

Chandy

and

P.

Banerjee

Design

and

Evaluation

of

Gracefully

Degradable

Disk

Arrays

Journal

of

Parallel

and

Distributed

Computing

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

C.

Ruemmler

J.

Wilkes

An

Introduction

to

Disk

Drive

Modeling

IEEE

Computer

Vol

No.

pp.

-LSB-

-RSB-

K.

Salem

and

H.

Garcia-Molina

Disk

Striping

Proceedings

of

the

International

Conference

on

Data

Engineering

pp.

-LCB-

-LSB-

-RSB-

C.

Staelin

H.

Garcia-Molina

Clustering

Active

Disk

Data

to

Improve

Disk

Performance

Technical

Report

Department

of

Computer

Science

Princeton

University

-LSB-

-RSB-

D.

Stodolsky

G.

Gibson

and

M.

Holland

Parity

Logging

Overcoming

the

Small

Write

Problem

in

Redundant

Disk

Arrays

Proceedings

of

the

Symposium

on

Computer

Architecture

-LRB-

ACM

SIGARCH

-RRB-

pp.

-LCB-

-LSB-

-RSB-

P.

Scheuermann

G.

Weikum

and

P.

Zabback

Automatic

Tuning

of

Data

Placement

and

Load

Balancing

in

Disk

Arrays

In

Database

Systems

for

Next-Generation

Applications

Principles

and

Practice

Advanced

Database

Research

and

Development

Series

World

Scientific

Publications

-LSB-

-RSB-

P.

Scheuermann

G.

Weikum

and

P.

Zabback

Adaptive

Load

Balancing

in

Disk

Arrays

Proceedings

of

the

International

Conference

on

Foundations

of

Data

Organization

and

Algorithms

-LRB-

FODO

-RRB-

pp.

-LCB-

-LSB-

-RSB-

H.

Schwetman

CSIM

Reference

Manual

-LRB-

Revision

-RRB-

MCC

Technical

Report

Microelectronics

and

Computer

Technology

Corporation

Austin

-LSB-

-RSB-

J.A.

Solworth

and

C.U.

Orji

Distorted

Mapping

Techniques

to

Achieve

High

Performance

in

Mirrored

Disk

Systems

International

Journal

on

Distributed

and

Parallel

Databases

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

M.

Stonebraker

P.M.

Aoki

A.

Pfeffer

A.

Sah

J.

Sidell

C.

Staelin

A.

Yu

Mariposa

A

Wide-Area

Distributed

Database

System

The

VLDB

Journal

Vol

No.

pp.

-LCB-

-LSB-

-RSB-

R.

Vingralek

Y.

Breitbart

G.

Weikum

Distributed

File

Organization

with

Scalable

Cost/Performance

ACM

SIGMOD

Conference

Minneapolis

-LSB-

-RSB-

R.

Vingralek

Y.

Breitbart

G.

Weikum

SNOWBALL

Scalable

Storage

on

Networks

of

Workstations

with

Balanced

Load

Technical

Report

Department

of

Computer

Science

University

of

Kentucky

-LSB-

-RSB-

G.

Weikum

C.

Hasse

A.

M?onkeberg

P.

Zabback

The

COMFORT

Automatic

Tuning

Project

Information

Systems

Vol

No.

pp

-LSB-

-RSB-

G.

Weikum

P.

Zabback

and

P.

Scheuermann

Dynamic

File

Allocation

in

Disk

Arrays

Proceedings

of

the

SIGMOD

International

Conference

on

Management

of

Data

pp.

-LCB-

extended

version

available

as

Technical

Report

No.

Computer

Science

Dept.

ETH

Z?urich

-LSB-

-RSB-

G.

Weikum

and

P.

Zabback

Tuning

of

Striping

Units

in

Disk-ArrayBased

File

Systems

Proceedings

of

the

International

Workshop

on

Research

Issues

on

Data

Engineering

Transaction

and

Query

Processing

-LRB-

RIDE-TQP

-RRB-

pp.

-LCB-

-LSB-

-RSB-

J.

Wilkes

R.

Golding

C.

Staelin

T.

Sullivan

The

HP

AutoRAID

Hierarchical

Storage

System

Proceedings

of

the

ACM

Symposium

on

Operating

Systems

Principles

-LSB-

-RSB-

J.

Wolf

The

Placement

Optimization

Program

A

Practical

Solution

to

the

Disk

File

Assignment

Problem

Proceedings

of

the

International

Conference

on

Measurement

and

Modeling

of

Computer

Systems

-LRB-

ACM

SIGMETRICS

-RRB-

pp.

-LCB-

-LSB-

-RSB-

O.

Wolfson

and

S.

Jajodia

Distributed

Algorithms

for

Dynamic

Replication

of

Data

Proceedings

of

the

ACM

SIGACT-SIGMOD-SIGART

Symposium

on

Principles

of

Database

Systems

-LRB-

PODS

-RRB-

pp.

-LCB-

-LSB-

-RSB-

P.

Zabback

I/O

Parallelism

in

Database

Systems

Design

Implementation

and

Evaluation

of

a

Storage

System

for

Parallel

Disks

-LRB-

in

German

-RRB-

Doctoral

Thesis

Department

of

Computer

Science

ETH

Z?urich

ETH-NR.

ss

oe

ae

OE?oeO

OE

ae

OE

OAEooeO

ae

O

ae

ffi

OE?AE?oe

ffloe

ffsso

O

fi

OE

o

fl?OE?ssss

ss?Oae

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

fi?O

ffi

OE?AE?oe

ffloe

ffsso

O

fi

OE

o

fl?OE?ssss

ss?Oae

ffi

OE?AE?oe

ffloe

ffsso

fi

OE

o

fl?OE?ssss

ss?Oae

Figure

Striping

with

different

striping

units

Input

D

number

of

disks

R

average

request

size

over

all

files

Ri

average

request

size

of

file

i

-LCB-

for

file-specific

partitioning

L

average

file

size

Li

size

of

file

i

-LCB-

for

file-specific

partitioning

average

arrival

rate

of

requests

Output

SUi

optimal

striping

unit

of

file

i

-LCB-

for

file-specific

partitioning

SU

optimal

global

striping

unit

SWi

optimal

striping

width

of

file

i

Step

Apply

Phase

A

optimization

with

respect

to

service

time

a.

File

specific

partitioning

determine

Popt

i

the

optimal

degree

of

parallelism

for

file

i

by

setting

R

Ri

in

equation

-LRB-

-RRB-

b.

Global

partitioning

determine

Popt

the

optimal

average

degree

of

parallelism

by

setting

R

R

in

equation

-LRB-

-RRB-

Step

Apply

Phase

B

optimization

to

determine

P

the

optimal

-LRB-

average

-RRB-

degree

of

parallelism

for

the

requested

throughput

Step

Determine

the

effective

degree

of

parallelism

a.

File

specific

partitioning

Peff

i

mini

-LRB-

Popt

i

PRi

R

-RRB-

b.

Global

partitioning

Peff

min

-LRB-

Popt

P

-RRB-

Step

Determine

optimal

striping

unit

and

width

a.

File

specific

partitioning

SUi

d

-LRB-

Ri

-RRB-

-LRB-

Peff

i

-RRB-

e

for

Ri

Li

dRi

Peff

ie

otherwise

SWi

mini

-LRB-

D

dLi

SUie

-RRB-

b.

Global

partitioning

Compute

SU

as

in

step

by

replacing

Ri

by

R

Li

by

L

and

Peff

i

by

Peff

Compute

SWi

as

in

step

Figure

Data

partitioning

algorithm

File

File

File

File

Greedy

heuristics

for

heat

balancingRound-Robin

Figure

Illustration

of

static

allocation

heuristics

disk

cooling

Figure

Illustration

of

disk

cooling

Input

D

number

of

disks

Hj

heat

of

extent

j

H?i

heat

of

disk

i

H

average

disk

heat

Ei

list

of

extents

on

disk

i

sorted

in

descending

temperature

order

D

list

of

disks

sorted

in

ascending

heat

order

Step

Initialization

target

not

found

Step

Select

the

hottest

disk

s

Step

Check

trigger

condition

if

Hs

H

-LRB-

ffi

-RRB-

then

Step

while

-LRB-

Es

not

exhausted

-RRB-

and

-LRB-

target

not

found

-RRB-

do

Select

next

extent

e

in

Es

Step

while

-LRB-

D

not

exhausted

-RRB-

and

-LRB-

target

not

found

-RRB-

do

Select

next

disk

t

in

D

in

ascending

heat

order

if

-LRB-

t

does

not

hold

an

extent

of

the

file

to

which

e

belongs

-RRB-

and

STATUS

-LRB-

t

-RRB-

FREE

then

target

found

fi

endwhile

endwhile

Step

if

s

has

no

queue

then

H

H?s

He

H

H?t

He

if

H

He

then

reallocate

extent

e

from

disk

s

to

disk

t

update

heat

of

disks

s

and

t

H?s

H

H?t

H

fi

fi

fi

Figure

Basic

disk

cooling

algorithm

ABC

pseudo

requeststo

all

blocks

requests

to

block

time

Figure

Illustration

of

the

heat

tracking

method

for

k

The

relevant

interarrival

times

are

shown

by

the

double-ended

arrows

without

cooling

with

cooling

Figure

Response

time

and

cooling

frequency

for

the

WWW

workload

varying

over

time

average

response

time

-LSB-

s

-RSB-

with

coolingwithout

of

cooling

steps

Figure

Average

response

time

and

cooling

frequency

for

the

OLTP

workload

varying

over

time

data allocation
disk cooling
file striping
load balancing
parallel disk systems
performance tuning