Differential

Evaluation

of

Continual

Queries

Ling

Liu

University

of

Alberta

Department

of

Computing

Science

GSB

Edmonton

Alberta

Canada

email

lingliu@cs.ualberta.ca

Calton

Pu

Roger

Barga

Tong

Zhou

Oregon

Graduate

Institute

of

Science

and

Technology

Department

of

Computer

Science

P.O.Box

Portland

Oregon

USA

email

calton,roger,tong@cse.ogi.edu

Abstract

Information

Superhighway

environments

such

as

the

Internet

have

brought

us

ready

access

to

large

amount

of

information

However

Internet

data

is

notoriously

unorganized

and

autonomously

managed

in

a

distributed

fashion

Large

scale

information

monitoring

in

the

Internet

environment

requires

support

beyond

traditional

database

techniques

Two

of

the

key

issues

are

the

increasing

reward

in

monitoring

a

fast

growing

information

base

and

the

similarly

increasing

processing

cost

To

improve

the

expressiveness

of

queries

for

information

monitoring

we

define

continual

queries

as

a

useful

tool

for

monitoring

of

updated

information

Continual

queries

are

standing

queries

that

monitor

the

source

data

and

notify

the

users

whenever

new

data

matches

the

query

In

addition

to

periodic

refresh

-LSB-

-RSB-

continual

queries

include

Epsilon

Transaction

-LSB-

-RSB-

concepts

to

allow

users

to

specify

query

refresh

based

on

the

magnitude

of

updates

To

support

efficient

processing

of

continual

queries

we

propose

a

differential

re-evaluation

algorithm

-LRB-

DRA

-RRB-

which

exploits

the

structure

and

information

contained

in

both

the

query

expressions

and

the

database

update

operations

The

DRA

design

can

be

seen

as

a

synthesis

of

previous

research

on

differential

files

incremental

view

maintenance

and

active

databases

Introduction

The

exponential

growth

of

the

Internet

-LSB-

-RSB-

makes

an

unprecedented

amount

of

new

information

available

Large

scale

information

monitoring

has

become

tantalizingly

close

to

reality

However

this

flood

of

information

can

easily

overwhelm

the

users

if

queries

are

formulated

naively

Furthermore

naive

query

processing

can

easily

overload

the

database

system

and

network

For

example

a

user

monitoring

new

multimedia

research

may

ask

the

system

to

show

all

the

multimedia

projects

described

in

WWW

pages

of

universities

and

companies

-LRB-

-RRB-

this

is

a

query

that

may

return

a

tremendous

amount

of

information

if

the

query

is

submitted

to

all

the

edu

and

com

sites

running

WWW

servers

The

problem

can

be

aggravated

when

constant

addition

and

updates

of

information

sources

on

the

Internet

force

users

to

re-issue

their

queries

frequently

to

obtain

the

new

data

that

match

their

queries

Trying

to

monitor

the

constant

information

flow

in

the

Internet

is

both

labor

intensive

for

the

user

and

resource

intensive

for

the

system

Therefore

the

support

for

flexible

and

efficient

monitoring

of

information

updates

is

an

important

goal

when

accessing

data

in

the

Internet

Traditional

database

techniques

such

as

view

materialization

-LSB-

-RSB-

active

databases

-LSB-

-RSB-

and

query

optimization

can

not

be

directly

applied

in

distributed

interoperable

environments

such

as

the

Internet

The

reasons

include

the

lack

of

integration

of

information

sources

autonomy

and

heterogeneity

in

the

Internet

and

the

relatively

unorganized

storage

and

management

of

data

such

as

WWW

pages

-LSB-

-RSB-

A

promising

proposal

is

Terry

et

al

's

continuous

queries

-LSB-

-RSB-

which

can

be

processed

incrementally

to

reduce

both

system

overhead

and

user

overload

However

the

severe

assumptions

they

made

about

their

environment

such

as

append-only

databases

with

no

deletions

render

their

work

unsuitable

for

many

kinds

of

queries

and

information

sources

in

the

Internet

The

challenges

presented

by

the

Internet

starts

with

the

formulation

of

queries

that

express

the

different

ways

users

want

to

be

notified

of

the

information

changes

Once

queries

are

formulated

optimization

in

monitoring

processing

and

transmission

is

needed

After

these

necessary

two

steps

integration

of

heterogeneous

data

sources

legacy

databases

file

systems

and

unorganized

data

are

important

for

enlarging

the

data

available

Further

refinements

include

approximate

queries

partial

matching

and

correlations

and

other

deep

information

semantics

In

this

paper

we

focus

our

attention

on

the

first

two

issues

We

make

two

contributions

towards

flexible

and

efficient

monitoring

of

information

updates

motivated

by

a

distributed

interoperable

environment

such

as

DIOM

-LSB-

-RSB-

First

we

describe

continual

queries

with

epsilon

specification

namely

epsilon

continual

queries

-LRB-

ECQ

-RRB-

to

simplify

continual

query

formulation

on

the

user

side

In

addition

to

triggering

query

refresh

periodically

as

proposed

in

previous

work

-LSB-

-RSB-

ECQs

allow

the

users

to

specify

and

activate

query

refresh

based

on

the

magnitude

of

updates

For

example

the

query

show

all

the

new

multimedia

projects

and

those

that

have

their

project

description

text

changed

by

more

than

since

last

week

-LRB-

-RRB-

would

bring

more

significant

information

to

the

user

monitoring

new

multimedia

research

compared

to

above

Another

example

of

new

ECQ

capability

is

the

query

show

the

IBM

stock

transactions

that

differ

by

more

than

from

per

share

-LRB-

-RRB-

Our

second

contribution

is

aimed

at

reducing

query

processing

cost

We

develop

a

differential

strategy

to

evaluate

ECQs

incrementally

and

prove

that

the

differential

re-evaluation

of

a

ECQ

is

functionally

equivalent

to

the

complete

re-evaluation

solution

This

differential

re-evaluation

algorithm

reduces

data

transmission

by

processing

the

query

primarily

on

the

changed

data

rather

than

re-evaluating

the

entire

database

Furthermore

differential

re-evaluation

supports

general

updates

including

insertions

deletions

and

modifications

This

is

a

significant

improvement

over

the

no-deletion

assumption

in

-LSB-

-RSB-

The

rest

of

the

paper

is

organized

as

follows

In

Section

we

discuss

the

motivations

of

our

work

and

the

related

research

Section

presents

the

semantics

of

continual

queries

with

epsilon

specification

Section

summarizes

the

notations

and

terminology

used

in

the

processing

of

continual

queries

Section

describes

a

method

for

differentially

evaluating

continual

queries

Section

discusses

some

implementation

issues

and

topics

of

future

research

Section

concludes

the

paper

Motivation

and

Related

Work

There

has

been

considerable

research

done

in

the

monitoring

of

information

change

in

databases

The

following

discussion

should

not

be

seen

as

a

critique

of

these

techniques

Rather

these

techniques

have

been

proposed

primarily

for

data-centric

environments

where

data

is

well

organized

and

controlled

When

applied

to

an

Information

Superhighway

environment

-LRB-

for

brevity

and

concreteness

we

refer

to

this

environment

as

the

Internet

-RRB-

these

assumptions

no

longer

hold

-LRB-

see

-LSB-

-RSB-

for

a

summary

of

desired

system

properties

in

the

Internet

-RRB-

Therefore

we

discuss

these

assumptions

before

we

launch

into

our

definition

of

continual

queries

-LRB-

Section

-RRB-

and

a

technique

-LRB-

Section

-RRB-

for

efficient

processing

of

continual

queries

Continuous

Queries

Terry

et

al

-LSB-

-RSB-

proposed

continuous

queries

for

monitoring

information

change

However

their

proposal

made

several

assumptions

that

seriously

restricted

the

applicability

of

their

technique

to

the

Internet

The

first

such

assumption

is

the

limitation

of

database

updates

to

append-only

disallowing

deletions

and

modifications

Since

this

assumption

is

used

in

their

query

transformation

algorithm

it

has

been

difficult

to

relax

it

-LSB-

-RSB-

when

following

their

definition

of

continuous

queries

This

is

one

of

the

motivations

for

our

new

definition

in

Section

under

a

new

name

continual

queries

Another

problem

discussed

in

-LSB-

-RSB-

is

the

non-determinism

in

continuous

query

results

where

two

users

with

exactly

the

same

continuous

query

expression

could

be

presented

with

different

results

Consider

the

query

for

a

product-purchase

database

select

the

products

which

have

received

no

orders

so

far

Whenever

a

new

product

is

inserted

it

matches

the

query

until

an

order

for

this

product

arrives

Suppose

a

particular

product

say

Simpson

Soap

is

inserted

into

the

database

at

and

an

order

for

it

arrived

at

then

the

query

would

not

return

Simpson

Soap

if

the

continuous

query

runs

every

hour

on

the

hour

but

Simpson

Soap

would

be

returned

by

a

system

that

runs

it

every

hour

on

the

half

hour

since

the

product

would

match

the

query

at

In

our

view

the

main

cause

of

this

non-determinism

comes

from

the

different

database

states

due

to

updates

which

affect

query

results

as

they

should

Therefore

instead

of

rejecting

it

as

an

anomaly

we

capture

it

In

Section

we

define

a

continual

query

as

a

sequence

of

executions

of

a

conventional

database

query

which

is

re-executed

when

a

triggering

condition

becomes

true

Two

continual

queries

with

the

same

query

expression

and

triggering

conditions

should

observe

the

same

query

results

over

time

while

continual

queries

with

the

same

query

expression

but

different

triggering

conditions

should

be

considered

different

Other

advantages

of

our

semantics

will

be

discussed

in

Section

Active

Databases

Active

databases

-LSB-

-RSB-

allow

users

to

specify

in

the

form

of

rules

actions

to

be

performed

following

changes

of

database

state

Despite

their

conceptual

generality

rules

have

been

so

far

supported

in

a

fairly

restrictive

form

in

practical

systems

for

example

by

triggers

-LSB-

-RSB-

in

relational

database

management

systems

such

as

Oracle

Sybase

and

Informix

A

trigger

is

an

event

-LCB-

condition

-LCB-

action

-LRB-

ECA

-RRB-

rule

in

a

restrictive

form

Typically

triggers

can

be

specified

only

on

a

single

base

table

Active

query

introduced

in

Alert

-LSB-

-RSB-

is

yet

another

form

of

ECA

rules

similar

to

continuous

queries

-LRB-

Section

-RRB-

and

usually

assumed

to

work

in

an

append

-LCB-

only

environment

Active

queries

are

more

sophisticated

than

triggers

since

they

can

be

defined

on

multiple

tables

on

views

and

can

be

nested

within

other

active

queries

However

like

continuous

queries

-LSB-

-RSB-

the

append

-LCB-

only

assumption

seriously

restricts

the

applicability

of

active

queries

to

the

Internet

environment

where

data

is

appended

removed

or

replaced

constantly

Maintenance

of

Materialized

Views

Another

related

area

of

research

is

the

incremental

maintenance

of

materialized

views

which

store

a

snapshot

of

selected

database

state

As

a

database

is

updated

the

materialized

view

must

be

refreshed

to

reflect

the

updates

A

naive

solution

is

to

rematerialize

the

view

from

the

base

data

In

contrast

incremental

update

algorithms

are

believed

to

carry

lower

execution

cost

if

changes

to

the

database

are

moderate

-LSB-

-RSB-

Three

approaches

have

been

described

previously

The

first

approach

refreshes

the

view

immediately

after

each

update

to

the

base

table

-LSB-

-RSB-

The

second

defers

the

refresh

until

a

query

is

issued

against

the

view

-LSB-

-RSB-

The

third

refreshes

the

view

periodically

-LSB-

-RSB-

The

main

tradeoff

in

choosing

among

these

approaches

is

the

staleness

of

the

view

data

vs.

the

cost

of

updating

it

Most

of

the

algorithms

in

the

literature

-LSB-

-RSB-

work

in

a

centralized

database

environment

in

which

the

materialized

view

and

its

base

tables

co-reside

The

study

on

distributed

materialized

view

management

has

been

primarily

focused

on

determining

the

optimal

refresh

sources

and

timing

for

multiple

views

defined

on

the

same

base

data

-LSB-

-RSB-

In

contrast

to

database

view

work

which

assumes

well

organized

data

and

schema

as

well

as

systemdefined

periodic

refresh

continual

queries

should

give

users

more

flexibility

in

specifying

the

conditions

for

refresh

and

tighter

control

over

the

distance

between

view

result

and

the

current

database

state

This

is

similar

to

the

specification

of

update

propagation

in

quasi-copies

-LSB-

-RSB-

A

recent

paper

-LSB-

-RSB-

has

identified

the

problem

of

using

incremental

view

maintenance

approaches

in

a

data

warehousing

environment

where

the

materialized

view

and

its

base

relations

are

loosely

coupled

While

the

warehouse

attempts

to

update

a

view

the

base

data

at

source

may

keep

changing

An

anomaly

may

occur

whenever

the

warehouse

performs

view

update

operations

using

a

state

of

the

base

relations

which

differs

from

the

state

at

which

this

view

update

was

issued

An

eager

compensating

algorithm

is

developed

in

-LSB-

-RSB-

to

address

this

problem

Other

possible

solutions

include

locking

the

relevant

data

at

source

while

the

results

of

CQs

are

being

refreshed

or

the

source

maintains

timestamps

for

its

data

Research

Challenges

An

important

distinction

between

CQs

and

previous

work

is

the

handling

of

general

updates

including

insertions

deletions

and

modifications

For

example

let

us

assume

that

a

job

opening

database

is

updated

whenever

a

posted

position

is

filled

-LRB-

deletion

-RRB-

or

a

new

job

posting

arrives

-LRB-

insertion

-RRB-

Some

warehouse

is

a

repository

of

integrated

information

gathered

from

remote

data

sources

for

queries

and

analysis

-LRB-

e.g.

decision

support

or

data

mining

-RRB-

users

may

desire

to

be

informed

of

both

the

incoming

new

openings

from

some

particular

type

of

companies

and

the

positions

that

were

available

before

-LRB-

e.g.

in

the

previous

execution

of

the

query

-RRB-

but

have

been

recently

filled

or

canceled

due

to

budget

disapproval

Simply

recomputing

the

query

for

each

execution

does

not

guarantee

the

correct

results

because

conventional

SQL

query

manager

can

only

filter

data

currently

matching

the

query

To

provide

adequate

support

for

flexible

incorporation

of

updates

the

CQ

manager

must

exploit

the

knowledge

implied

by

both

the

query

expressions

and

database

update

operations

Another

important

problem

in

monitoring

of

information

updates

is

the

cost

of

query

evaluation

A

naive

query

evaluation

strategy

would

recompute

the

same

query

from

scratch

for

each

execution

request

and

highlight

the

difference

In

the

Internet

environment

such

strategy

can

easily

overwhelm

the

users

and

overload

the

system

and

networks

especially

when

a

large

number

of

records

match

the

query

but

only

a

small

fraction

is

changed

each

time

or

when

some

users

want

to

see

only

the

data

that

have

been

updated

since

the

last

query

result

This

is

our

motivation

for

proposing

the

differential

re-evaluation

algorithm

-LRB-

DRA

-RRB-

described

in

Section

for

processing

ECQs

The

solution

we

promote

avoids

the

overhead

of

recomputing

the

query

from

scratch

and

relaxes

the

restrictions

of

appendonly

database

and

monotonic-transformable

queries

Our

design

builds

on

the

research

in

differential

files

-LSB-

-RSB-

hypothetical

relations

-LSB-

-RSB-

and

incremental

updating

materialized

views

-LSB-

-RSB-

and

offers

several

advantages

over

the

previous

solutions

-LSB-

-RSB-

The

basic

idea

is

to

compute

the

result

of

the

current

execution

of

a

continual

query

by

combining

the

result

produced

by

the

most

recent

execution

with

the

updates

occurred

in

between

these

two

consecutive

executions

We

use

differential

relations

to

record

changes

to

the

source

relations

-LRB-

or

materialized

views

-RRB-

We

define

a

set

of

differential

operators

to

compute

the

updates

occurred

in

between

two

consecutive

executions

of

a

continual

query

Continual

Query

Semantics

Basic

Semantics

A

continual

query

CQ

is

a

triple

-LRB-

Q

TCQ

Stop

-RRB-

consisting

of

a

normal

query

Q

-LRB-

e.g.

written

in

SQL

-RRB-

a

triggering

condition

TCQ

as

specified

below

and

a

termination

condition

Stop

-LRB-

Since

TCQ

and

Stop

in

general

may

depend

on

many

different

parameters

we

usually

omit

their

parameters

for

clarity

-RRB-

Let

us

denote

the

result

of

running

query

Q

on

database

state

Si

as

Q

-LRB-

Si

-RRB-

We

define

the

result

of

running

a

continual

query

CQ

as

a

sequence

of

query

answers

fQ

-LRB-

-RRB-

Q

-LRB-

-RRB-

Q

-LRB-

Sn

-RRB-

g

obtained

by

running

query

Q

on

the

sequence

of

database

states

Si

i

n

each

time

triggered

by

TCQ

i.e.

TCQ

Stop

We

assume

that

there

is

a

mapping

from

the

sequence

of

database

states

Si

i

n

and

n

into

a

sequence

of

monotonically

increasing

timestamps

-LRB-

tn

-RRB-

i.e.

when

given

a

state

Si

we

can

refer

to

its

time

ti

Each

CQ

has

its

own

sequence

of

database

states

defined

by

the

triggering

condition

TCQ

The

triggering

condition

TCQ

is

a

specification

of

when

the

query

Q

should

be

executed

It

can

take

several

forms

ffl

A

direct

specification

of

time

For

example

Q

should

be

executed

once

every

Monday

This

is

the

policy

adopted

by

the

Harvest

information

gatherers

-LSB-

-RSB-

ffl

A

specification

of

time

interval

from

a

previous

query

result

For

example

a

week

since

Q

-LRB-

Sn

-RRB-

was

produced

-LRB-

This

can

be

interpreted

as

a

time

specification

relative

to

an

event

-RRB-

ffl

A

condition

on

the

database

state

For

example

Q

should

be

executed

whenever

a

deposit

of

one

million

dollars

is

made

This

is

supported

by

epsilon

specifications

-LRB-

Section

-RRB-

ffl

A

relationship

between

a

previous

query

result

and

the

current

database

state

For

example

Q

should

be

executed

when

a

total

of

million

dollars

worth

of

deposits

have

been

made

since

Q

-LRB-

Sn

-RRB-

was

produced

This

is

also

supported

by

epsilon

specifications

-LRB-

Section

-RRB-

If

the

termination

condition

Stop

is

CQ

will

produce

results

from

Q

-LRB-

-RRB-

to

Q

-LRB-

-RRB-

Otherwise

CQ

will

produce

results

from

a

starting

time

to

a

final

time

tn

when

Stop

becomes

true

In

other

words

CQ

-LRB-

the

sequence

-RRB-

ends

when

the

termination

condition

becomes

true

Each

time

TCQ

is

triggered

conceptually

Q

is

run

against

the

current

state

of

the

database

Sn

and

the

result

Q

-LRB-

Sn

-RRB-

is

sent

to

the

user

who

issued

CQ

In

general

it

is

expensive

to

re-evaluate

the

whole

query

over

the

entire

source

data

for

each

execution

of

a

continual

query

although

in

some

circumstances

-LRB-

e.g.

legacy

databases

and

some

file

systems

-RRB-

it

may

be

unavoidable

to

reprocess

the

query

from

scratch

Therefore

it

is

important

to

find

optimization

steps

that

can

bypass

the

complete

re-evaluation

and

thus

avoid

the

duplicate

computation

and

unnecessary

data

transmission

In

Section

we

describe

a

strategy

to

generate

Q

-LRB-

Sn

-RRB-

from

Q

-LRB-

Sn

-RRB-

incrementally

thus

reducing

both

processing

time

and

network

transmission

bandwidth

Epsilon

Specifications

Epsilon

specifications

have

been

introduced

as

part

of

the

work

on

Epsilon

Serializability

-LRB-

ESR

-RRB-

-LSB-

-RSB-

ESR

was

originally

proposed

as

a

correctness

criterion

to

generalize

concurrency

control

in

extended

transaction

processing

The

key

idea

of

ESR

is

that

we

can

generalize

serializability

by

using

the

notion

of

distance

to

a

serializable

state

in

database

state

spaces

that

define

such

a

distance

-LRB-

e.g.

numerical

state

spaces

-RRB-

A

simple

example

of

an

epsilon

query

is

the

checking

account

sum-up

query

a

bank

manager

wants

to

know

how

many

millions

of

dollars

she

has

in

all

the

checking

accounts

with

the

error

tolerance

rate

of

half

a

million

Since

the

round-off

error

of

the

query

is

half

a

million

dollars

the

query

could

contain

other

errors

-LRB-

say

due

to

non-serializable

conflicts

with

some

updates

-RRB-

up

to

half

a

million

dollars

and

still

return

a

meaningful

result

The

specification

of

the

error

tolerance

in

the

above

example

i.e.

half

a

million

dollars

is

called

an

epsilon

specification

-LRB-

ffl-spec

-RRB-

A

formal

characterization

of

ESR

is

presented

in

-LSB-

-RSB-

Divergence

control

algorithms

that

guarantee

ESR

efficiently

have

been

devised

-LSB-

-RSB-

and

a

prototype

implementation

-LSB-

-RSB-

done

on

top

of

Transarc

Encina

a

commercial

transaction

processing

monitor

Although

ESR

was

originally

proposed

for

transaction

processing

we

have

found

an

interesting

application

of

epsilon

queries

in

the

context

of

CQs

We

can

model

an

epsilon

query

as

a

pair

-LRB-

Q

ffl-spec

-RRB-

where

the

ffl-spec

is

used

as

the

triggering

condition

TCQ

This

way

we

can

define

an

ECQ

from

an

epsilon

query

plus

a

termination

condition

Intuitively

the

original

epsilon

query

was

defined

to

allow

a

bounded

amount

of

inconsistency

to

be

introduced

into

the

query

result

For

example

divergence

control

algorithms

-LSB-

-RSB-

allow

limited

non-serializable

conflicts

between

updates

and

the

epsilon

query

to

happen

to

increase

system

execution

flexibility

and

concurrency

In

an

ECQ

environment

we

can

consider

the

ffl-spec

as

the

triggering

condition

that

bounds

the

distance

between

the

previous

element

in

the

sequence

of

ECQ

results

to

the

next

Each

time

the

current

ECQ

query

result

is

about

to

violate

the

ffl-spec

the

system

generates

the

next

ECQ

result

in

the

sequence

For

example

suppose

the

last

execution

of

the

checking

account

sum-up

epsilon

query

returned

a

result

of

million

If

the

accumulated

amount

of

withdrawals

from

and

deposits

to

the

checking

accounts

have

exceeded

half

a

million

then

the

system

generates

the

next

sum-up

result

Instead

of

re-evaluation

of

complete

sum-up

query

a

differential

re-evaluation

algorithm

-LRB-

DRA

-RRB-

described

in

Section

can

be

used

If

a

triggering

condition

or

a

termination

condition

contains

references

to

the

database

state

then

updates

in

the

database

need

to

trigger

the

evaluation

of

these

conditions

Differential

Processing

of

CQ

Although

we

have

defined

CQ

as

a

sequence

of

query

results

there

are

situations

where

users

are

more

interested

in

the

difference

between

Q

-LRB-

Si

-RRB-

and

Q

-LRB-

Si

-RRB-

Typically

when

Q

-LRB-

Si

-RRB-

is

relatively

large

and

only

a

small

percentage

of

the

result

changes

from

Q

-LRB-

Si

-RRB-

to

Q

-LRB-

Si

-RRB-

then

users

would

want

to

have

the

difference

emphasized

This

can

be

accomplished

by

naively

executing

the

entire

query

and

then

filtering

out

the

part

of

the

query

result

that

is

the

same

as

the

previous

result

This

simple

and

straightforward

approach

can

be

quite

expensive

especially

in

the

Internet

environment

where

query

results

need

to

be

gathered

from

multiple

source

data

repositories

This

is

a

motivating

example

for

differential

query

processing

and

result

presentation

Using

differential

re-evaluation

-LRB-

Section

-RRB-

we

produce

Q

-LRB-

Si

-RRB-

by

incrementally

updating

Q

-LRB-

Si

-RRB-

This

way

we

avoid

reprocessing

the

entire

query

from

scratch

When

the

changes

are

substantially

smaller

compared

with

the

latest

query

execution

result

this

differential

update

will

be

more

efficient

than

reprocessing

the

entire

query

Since

a

differential

update

is

performed

on

top

of

Q

-LRB-

Si

-RRB-

to

produce

Q

-LRB-

Si

-RRB-

we

assume

that

the

underlying

system

maintains

a

copy

of

Q

-LRB-

Si

-RRB-

i.e.

it

is

not

deleted

as

the

result

is

scrolled

off

the

user

's

screen

The

copy

is

maintained

at

the

site

where

the

differential

query

refresh

is

carried

out

If

the

user

is

interested

only

in

being

notified

of

the

difference

between

the

current

query

result

and

the

next

query

result

-LRB-

called

differential

query

result

-RRB-

then

we

may

skip

the

maintenance

of

a

copy

of

the

complete

result

of

the

previous

query

But

if

the

user

requests

a

complete

query

result

we

may

have

to

reprocess

the

query

upon

such

requests

The

trade-off

in

choosing

the

best

strategy

is

analogous

to

the

optimization

trade-off

in

choosing

between

complete

system

backups

and

incremental

backups

Notations

and

Terminology

The

relational

model

is

used

in

this

paper

to

specify

continual

queries

to

record

and

manipulate

changes

by

differential

relations

and

associated

operators

We

assume

that

the

reader

is

familiar

with

the

basic

concepts

and

notation

concerning

relational

database

as

described

in

-LSB-

-RSB-

In

this

paper

we

refer

to

relational

selection

projection

join

outerjoin

union

and

difference

operators

by

oe

S

and

respectively

For

presentation

convenience

we

sometimes

use

-LRB-

R

X

-RRB-

to

denote

X

-LRB-

R

-RRB-

oe

-LRB-

R

F

-RRB-

to

denote

oeF

-LRB-

R

-RRB-

and

a

SPJ

expression

to

denote

a

Select-Project-Join

expression

We

use

differential

relations

to

represent

the

net

effect

of

a

collection

of

updates

to

a

relation

either

stored

or

derived

The

differential

relation

for

a

stored

relation

is

instantiated

by

the

system

when

the

source

is

updated

by

insertion

deletion

or

modification

-LRB-

see

Example

-RRB-

We

define

an

operator

that

computes

differential

relations

for

arbitrary

SPJ

expressions

The

concept

of

differential

relations

is

to

some

extent

similar

to

the

concept

of

hypothetical

relations

used

for

incremental

updating

materialized

views

-LSB-

-RSB-

The

difference

lies

in

the

usage

and

the

detailed

structure

In

the

eager

mechanism

for

materialized

view

update

a

hypothetical

relation

represents

the

net

changes

made

by

a

single

transaction

keeps

all

tuples

in

both

the

left

and

right

relations

when

no

matching

tuples

are

found

padding

them

with

values

as

needed

-LSB-

-RSB-

to

a

base

relation

and

can

be

dropped

after

the

transaction

is

committed

and

the

materialized

view

is

updated

In

the

continual

query

refresh

method

a

differential

relation

actually

maintains

changes

made

by

several

transactions

to

a

base

relation

Data

in

the

differential

relation

can

only

be

dropped

when

their

timestamps

are

older

than

the

timestamp

of

the

latest

execution

of

every

relevant

continual

query

We

would

like

to

mention

that

the

relational

model

is

not

essential

to

our

approach

but

it

simplifies

the

representation

of

database

changes

allows

use

of

the

relational

algebra

and

avoids

the

need

to

explain

the

semantics

of

a

particular

object

model

Differential

Relations

We

introduce

the

concept

of

differential

relation

a

relation

that

can

represent

changes

to

another

relation

and

design

a

set

of

basic

operations

to

facilitate

the

manipulation

of

such

relations

The

goal

for

defining

differential

relations

instead

of

using

hypothetical

relations

described

in

-LSB-

-RSB-

is

to

provide

a

unified

treatment

of

changes

not

separate

treatments

of

insertions

deletions

and

modifications

resulting

in

several

algorithms

for

generating

and

combining

individual

results

Let

R

denote

a

relation

scheme

described

by

a

set

of

attributes

An

Let

R

denote

a

relation

instance

of

R

consisting

of

a

collection

of

tuples

whose

values

are

taken

from

the

domains

of

the

set

of

attributes

of

R.

t

Ai

-LRB-

i

n

-RRB-

denotes

the

value

of

attribute

Ai

for

tuple

t.

Each

tuple

has

an

attribute

denoted

as

tid

which

provides

a

unique

immutable

identifier

When

a

tuple

is

deleted

and

later

re-appended

to

R

it

will

have

a

new

tid

assigned

The

unique

tuple

identifier

tid

makes

it

easier

to

connect

tuples

that

hold

values

of

the

same

object

before

and

after

changes

In

fact

the

primary

key

can

be

used

as

the

unique

identifier

for

each

tuple

of

R.

Definition

-LRB-

Differential

relation

-RRB-

Let

R

-LRB-

tid

An

-RRB-

be

a

relation

scheme

For

each

relation

R

of

scheme

R

we

define

a

differential

relation

denoted

by

R

to

represent

changes

R

-LRB-

tid

<old>

A

<old>

A

<old>

n

tid

<new>

A

<new>

A

<new>

n

timestamp

-RRB-

where

ffl

A

<old>

i

refers

to

old

attribute

values

and

ffl

A

<new>

i

-LRB-

i

n

-RRB-

refers

to

new

attribute

values

ffl

For

any

tuple

t

R

-LCB-

tid

<old>

and

tid

<new>

can

not

both

be

-LCB-

If

tid

<old>

is

so

are

all

t

A

<old>

i

-LRB-

i

n

-RRB-

-LCB-

Similarly

if

tid

<new>

is

so

are

all

t

A

<new>

i

-LRB-

i

n

-RRB-

-LCB-

If

both

tid

<new>

and

tid

<old>

are

not

then

tid

<new>

tid

<old>

ffl

For

each

tuple

in

R

the

system

assigns

a

timestamp

at

its

creation

time

as

its

identifier

Differential

relations

can

represent

tuples

-LRB-

or

objects

-RRB-

where

only

the

tid

field

is

nonnull

No

tid

can

appear

in

multiple

rows

For

addition

the

attributes

A

<old>

i

-LRB-

i

n

-RRB-

are

For

deletion

the

attributes

A

<new>

i

are

For

modification

attributes

A

<old>

i

hold

old

values

and

attributes

A

<new>

i

hold

newly

modified

values

The

timestamp

field

is

set

to

the

current

time

-LRB-

from

a

system

clock

or

any

other

monotonically

increasing

source

of

timestamps

-RRB-

whenever

a

tuple

is

appended

to

R.

Example

Consider

the

relation

Stocks

with

attributes

such

as

name

and

price

per

units

Stocks

tid

Name

Price

DEC

OLI

ODI

USL

Assume

that

the

transaction

T

updates

the

Stocks

relation

by

insertion

deletion

and

modification

Transaction

T

-LCB-

Insert

-LRB-

MAC

-RRB-

Modify

-LRB-

DEC

-RRB-

-LRB-

DEC

-RRB-

Delete

-LRB-

-RRB-

-RCB-

The

following

differential

relation

Stocks

captures

the

changes

that

the

transaction

T

made

to

Stocks

Stocks

tid

<old>

Name

<old>

Price

<old>

tid

<new>

Name

<new>

Price

<new>

timestamp

MAC

DEC

150 120992

DEC

OLI

In

the

rest

of

this

section

for

presentation

clarity

the

field

timestamp

is

omitted

when

no

confusion

is

incurred

We

sometimes

simply

use

delta

relations

to

refer

to

differential

relations

Basic

Operations

We

first

define

renaming

functions

that

add

or

remove

the

superscripts

old

and

new

from

attribute

names

in

a

relation

scheme

R.

Definition

Let

R

-LRB-

tid

An

-RRB-

be

a

relation

scheme

and

R

be

a

relation

of

scheme

R.

Let

the

relation

S

be

of

relation

scheme

S

-LRB-

tid

<old>

A

<old>

A

<old>

n

-RRB-

The

differential

relation

schema

of

R

is

denoted

by

R.

old

-LRB-

R

-RRB-

ftid

<old>

A

<old>

A

<old>

n

g.

new

-LRB-

R

-RRB-

ftid

<new>

A

<new>

A

<new>

n

g.

detach

-LSB-

S

old

-LRB-

S

-RRB-

-RSB-

returns

the

same

relation

with

each

attribute

name

detached

from

the

superscript

<old>

attach

-LSB-

R

old

-LRB-

R

-RRB-

-RSB-

returns

the

same

relation

with

each

attribute

name

attached

by

the

superscript

<old>

The

second

group

of

operators

is

used

to

project

the

old

or

new

values

of

a

differential

relation

and

to

compute

the

updated

relation

say

by

combining

the

relation

R

with

its

differential

relation

R.

Definition

-LRB-

High-level

projection

operators

-RRB-

Let

R

be

a

relation

of

R

-LRB-

tid

An

-RRB-

and

R

be

a

relation

of

the

differential

relation

schema

R

We

define

the

operators

deletions

additions

and

combine

as

follows

deleteLog

-LRB-

R

-RRB-

-LRB-

R

old

-LRB-

R

-RRB-

-RRB-

insertLog

-LRB-

R

-RRB-

-LRB-

R

new

-LRB-

R

-RRB-

-RRB-

deletions

-LRB-

R

-RRB-

detach

-LSB-

deleteLog

-LRB-

R

-RRB-

old

-LRB-

R

-RRB-

-RSB-

insertions

-LRB-

R

-RRB-

detach

-LSB-

insertLog

-LRB-

R

-RRB-

new

-LRB-

R

-RRB-

-RSB-

combine

-LRB-

R

R

-RRB-

-LRB-

R

deletions

-LRB-

R

-RRB-

-RRB-

S

insertions

-LRB-

R

-RRB-

Example

Consider

the

relation

Stocks

and

the

differential

relation

Stocks

in

Example

deleteLog

-LRB-

Stocks

-RRB-

tid

<old>

Name

<old>

Price

<old>

DEC

OLI

insertions

-LRB-

Stocks

-RRB-

tid

Name

Price

MAC

DEC

Differential

Evaluation

of

Continual

Queries

In

this

section

we

present

a

differential

re-evaluation

algorithm

-LRB-

DRA

-RRB-

for

processing

a

continual

query

efficiently

In

contrast

to

a

complete

re-evaluation

differential

re-evaluation

means

that

after

the

initial

execution

of

a

CQ

the

re-evaluation

of

each

subsequent

execution

of

this

CQ

will

be

performed

by

using

the

differential

form

of

the

query

The

DRA

is

invoked

by

the

CQ

manager

based

on

the

epsilon

specification

associated

with

the

given

CQ

We

assume

that

the

information

available

when

the

DRA

is

invoked

includes

-LRB-

i

-RRB-

the

CQ

definition

-LRB-

ii

-RRB-

the

contents

of

each

base

relation

after

the

last

execution

of

the

CQ

-LRB-

iii

-RRB-

the

differential

relations

for

each

of

those

operand

relations

that

have

been

changed

since

the

last

execution

of

the

CQ

-LRB-

iv

-RRB-

the

timestamp

of

the

last

execution

of

the

CQ

-LRB-

v

-RRB-

the

complete

set

of

the

result

of

the

CQ

produced

by

the

last

execution

Note

that

the

CQ

manager

will

use

-LRB-

iv

-RRB-

to

append

the

proper

timestamp

predicate

-LRB-

s

-RRB-

into

the

differential

form

of

the

CQ

which

limits

the

search

space

to

only

those

tuples

that

were

written

to

the

differential

relations

by

the

updates

occurred

after

the

last

execution

of

this

CQ

-LRB-

recall

Example

for

an

illustration

-RRB-

We

also

formally

study

the

correctness

of

the

DRA

with

respect

to

the

complete

re-evaluation

solution

In

what

follows

we

first

formally

define

an

operator

that

computes

the

differential

relations

for

the

data

derived

by

arbitrary

query

expressions

in

Section

In

Section

we

introduce

the

differential

forms

for

the

three

basic

relational

algebraic

operations

Select

Project

and

Join

We

prove

that

instantiation

of

Propagate

-LRB-

Q

PL

-RRB-

for

relational

select

project

and

join

are

functionally

equivalent

to

their

differential

forms

DiffSelect

DiffProj

and

DiffJoin

The

differential

re-evaluation

algorithm

is

described

in

Section

Computing

the

Differential

Results

for

Continual

Queries

In

this

section

we

introduce

a

high

level

operator

called

Propagate

to

describe

how

the

result

relation

of

a

continual

query

changes

when

at

least

one

of

its

operand

relations

changes

This

operator

computes

the

difference

between

two

consecutive

executions

of

a

CQ

by

complete

re-evaluation

of

the

query

for

each

execution

It

can

be

considered

as

an

instantiation

of

complete

re-evaluation

solution

The

main

purpose

of

introducing

the

operator

Propagate

is

to

formally

prove

that

our

differential

re-evaluation

algorithm

to

continual

queries

is

functionally

equivalent

to

the

recompute

the

query

from

scratch

solution

and

in

many

situations

is

more

efficient

Before

giving

the

definition

of

the

operator

propagate

we

define

an

operator

that

compute

the

difference

of

two

relations

of

the

same

scheme

Definition

-LRB-

The

operator

Diff

-RRB-

Let

and

be

two

relations

of

the

same

scheme

R

The

operator

Diff

-LRB-

-RRB-

is

defined

as

the

Outerjointid

of

tuples

in

and

under

the

join

condition

tid

<old>

tid

<new>

Diff

-LRB-

-RRB-

Outerjointid

-LRB-

attach

-LSB-

-LRB-

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

-LRB-

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

It

returns

a

differential

relation

R

that

describes

their

differences

values

are

used

to

pad

tuples

that

appear

in

only

or

The

relation

scheme

of

R

is

old

-LRB-

R

-RRB-

-LSB-

new

-LRB-

R

-RRB-

Definition

-LRB-

The

Propagate

operator

-RRB-

Let

Ri

be

a

relation

of

scheme

Ri

-LRB-

i

n

-RRB-

Ri

be

a

differential

relation

of

Ri

and

denote

combine

-LRB-

Ri

Ri

-RRB-

Let

Q

-LRB-

Rn

-RRB-

denote

an

arbitrary

continual

query

and

SL

f

-LSB-

Ri

Ri

-RSB-

j

-LRB-

i

n

-RRB-

g

denote

the

differential

substitution

list

The

Propagate

operator

is

defined

as

follows

Propagate

-LRB-

Q

-LRB-

Rn

-RRB-

SL

-RRB-

Diff

-LRB-

Q

-LRB-

Rn

-RRB-

Q

-LRB-

-RRB-

-RRB-

-LRB-

n

-RRB-

It

returns

the

differential

relation

of

query

Q

denoted

by

RQ

with

the

scheme

RQ

We

may

also

denote

Propagate

-LRB-

Q

-LRB-

Rn

-RRB-

SL

-RRB-

by

Q

-LRB-

Rn

-RRB-

Example

Consider

the

query

Q

oeprice

-LRB-

Stocks

-RRB-

as

a

continual

query

Let

Ei

be

the

ith

execution

of

Q

at

time

ti

and

Ei

-LRB-

Q

ti

-RRB-

denote

the

result

of

the

ith

execution

of

Q

-LRB-

i

-RRB-

Now

assume

that

the

base

relation

Stocks

is

changed

after

the

last

execution

Ei

and

before

the

current

execution

Ei

by

the

update

transaction

T

given

in

Example

Let

Q

-LRB-

Stocks

-RRB-

Ei

-LRB-

Q

ti

-RRB-

denote

the

result

of

the

ith

execution

of

Q

over

Stocks

Let

Stocks

denote

the

base

relation

after

updates

to

Stocks

by

transaction

T

and

Q

-LRB-

Stocks

-RRB-

Ei

-LRB-

Q

ti

-RRB-

denote

the

result

of

the

current

execution

of

Q

over

the

relation

Stocks

Based

on

Definition

to

express

how

the

result

Ei

-LRB-

Q

ti

-RRB-

may

change

after

the

updates

by

the

transaction

T

we

may

simply

compute

Propagate

-LRB-

Q

-LRB-

Stocks

-RRB-

-LSB-

Stocks

Stocks

-RSB-

-RRB-

the

difference

between

the

result

relation

before

the

updates

Q

-LRB-

Stocks

-RRB-

and

the

result

relation

after

the

update

Q

-LRB-

Stocks

-RRB-

Q

-LRB-

Stocks

-RRB-

oeprice

-LRB-

Stocks

-RRB-

f

-LRB-

DEC

-RRB-

-LRB-

OLI

-RRB-

g.

Q

-LRB-

Stocks

-RRB-

oeprice

-LRB-

Stocks

-RRB-

f

-LRB-

DEC

-RRB-

g.

Q

-LRB-

Stocks

-RRB-

Q

-LRB-

Stocks

-RRB-

f

-LRB-

DEC

-RRB-

-LRB-

OLI

-RRB-

g.

Q

-LRB-

Stocks

-RRB-

Q

-LRB-

Stocks

-RRB-

f

-LRB-

DEC

-RRB-

g

The

differential

result

Q

-LRB-

Stocks

-RRB-

below

represents

the

net

effect

made

by

all

the

updates

occurred

between

the

the

last

execution

Ei

and

the

current

execution

Ei

Q

-LRB-

Stocks

-RRB-

Diff

-LRB-

Q

-LRB-

Stocks

-RRB-

Q

-LRB-

Stocks

-RRB-

-RRB-

tid

<old>

Name

<old>

Price

<old>

tid

<new>

Name

<new>

Price

<new>

DEC

150 120992

DEC

OLI

In

this

example

the

differential

result

of

the

query

oeprice

-LRB-

Stocks

-RRB-

since

the

last

execution

of

Q

is

computed

by

directly

evaluating

the

function

Propagate

-LRB-

oeprice

-LRB-

Stocks

-RRB-

-LSB-

Stocks

Stocks

-RSB-

-RRB-

Assume

the

previous

execution

result

Ei

-LRB-

Q

ti

-RRB-

is

saved

The

evaluation

of

Propagate

directly

from

its

definition

requires

first

a

scan

of

the

modified

relation

Stocks

in

order

to

compute

Q

-LRB-

Stocks

-RRB-

Such

recomputing

from

scratch

is

often

wasteful

and

in

many

cases

unacceptable

Observe

that

when

-LRB-

i

-RRB-

the

size

of

the

source

relation

Stocks

is

large

-LRB-

ii

-RRB-

the

selectivity

factor

of

the

query

Q

over

Stocks

is

not

high

and

-LRB-

iii

-RRB-

the

number

of

tuples

written

by

the

updates

occurred

between

the

two

consecutive

executions

Ei

and

Ei

is

relatively

small

it

would

be

more

efficient

to

compute

the

differential

result

of

the

query

Q

before

and

after

the

updates

by

evaluating

the

query

over

the

differential

relation

Stocks

instead

It

is

because

computing

Propagate

-LRB-

oeprice

-LRB-

Stocks

-RRB-

-LSB-

Stocks

Stocks

-RSB-

-RRB-

is

functionally

equivalent

to

the

evaluation

of

the

differential

query

oeF

-LRB-

Stocks

-RRB-

where

F

denotes

the

predicate

price

<old>

price

<new>

timestamp

This

is

true

even

when

the

user

needs

the

complete

composite

set

of

the

query

result

because

computing

the

union

of

Q

-LRB-

Stocks

-RRB-

and

insertions

-LRB-

oeF

-LRB-

Stocks

-RRB-

-RRB-

is

cheaper

than

recomputing

the

expression

oeprice

-LRB-

Stocks

-RRB-

from

scratch

Furthermore

using

a

differential

evaluation

approach

we

can

show

those

tuples

that

were

removed

between

the

two

consecutive

executions

of

Q

simply

by

computing

deletions

-LRB-

oeF

-LRB-

Stocks

-RRB-

-RRB-

In

general

for

any

continual

query

Q

over

the

relation

R

let

Ei

-LRB-

Q

ti

-RRB-

be

the

last

execution

of

Q

at

time

ti

A

complete

set

of

the

result

of

current

execution

of

Q

can

be

obtained

by

computing

the

expression

-LRB-

Ei

-LRB-

Q

ti

-RRB-

oetimestamp

ti

-LRB-

deletions

-LRB-

R

-RRB-

-RRB-

-RRB-

-LSB-

oetimestamp

ti

-LRB-

insertions

-LRB-

R

-RRB-

-RRB-

The

expression

oetimestamp

ti

-LRB-

insertions

-LRB-

R

-RRB-

returns

all

the

records

that

have

been

appended

to

R

since

the

last

execution

of

Q

whereas

the

expression

oetimestamp

ti

-LRB-

deletions

-LRB-

R

-RRB-

returns

all

the

records

removed

from

R

since

the

last

execution

of

Q.

Optimization

based

on

Differential

Operators

For

many

forms

of

queries

Propagate

-LRB-

Q

PL

-RRB-

can

be

computed

more

efficiently

than

by

directly

evaluating

the

definition

of

the

Propagate

operator

These

computations

use

the

differential

relations

heavily

and

sometimes

exclusively

instead

of

computing

on

the

base

relations

which

tend

to

be

much

larger

In

this

section

we

define

the

differential

forms

for

the

three

basic

relational

algebraic

operations

Select

Project

and

Join

We

prove

that

instantiation

of

Propagate

-LRB-

Q

PL

-RRB-

for

relational

select

project

and

join

are

functionally

equivalent

to

their

differential

forms

DiffSelect

DiffProj

and

DiffJoin

The

predicates

contained

in

these

operators

can

be

atomic

or

composite

by

logical

AND

and

logical

OR

The

attributes

involved

in

the

predicates

or

the

projection

list

are

single

valued

attributes

Aggregate

functions

such

as

SUM

COUNT

MAX

MIN

are

allowed

in

this

framework

with

some

additional

consideration

For

example

when

the

query

expression

contains

the

aggregate

function

MAX

-LRB-

A

-RRB-

we

need

to

compare

the

A

field

of

each

new

tuple

in

the

differential

relation

with

the

value

of

MAX

-LRB-

A

-RRB-

and

to

reset

MAX

-LRB-

A

-RRB-

if

necessary

Similar

treatment

applies

to

SUM

COUNT

MIN

However

when

the

query

condition

timestamp

ti

is

appended

by

the

CQ

manager

for

incorporating

correct

amount

of

updates

and

limiting

the

query

search

space

Section

provides

some

further

discussion

contains

the

aggregate

function

AVG

the

recomputation

of

the

query

is

needed

The

Differential

Select

Operator

Definition

-LRB-

Differential

select

-RRB-

Let

F

denote

a

predicate

defined

on

the

relation

R

of

scheme

R

and

oe

-LSB-

R

F

-RSB-

denote

the

associated

relational

selection

operator

Let

F

<old>

and

F

<new>

denote

the

same

predicate

with

all

attribute

names

superscripted

accordingly

We

define

the

operator

DiffSelect

as

follows

DiffSelect

-LSB-

R

F

-RSB-

Outerjointid

-LRB-

oe

-LSB-

deleteLog

-LRB-

R

-RRB-

F

<old>

-RSB-

oe

-LSB-

insertLog

-LRB-

R

-RRB-

F

<new>

-RSB-

-RRB-

Proposition

Propagate

-LRB-

oe

-LSB-

R

F

-RSB-

-LRB-

R

R

-RRB-

-RRB-

DiffSelect

-LSB-

R

F

-RSB-

Proof

-LRB-

see

Appendix

A

-RRB-

This

proposition

shows

that

Definition

gives

an

implementation

of

DiffSelect

which

does

not

reference

the

base

relation

and

which

can

be

implemented

by

one

pass

over

R

determining

for

each

tuple

of

R

whether

it

induces

an

insertion

deletion

or

modification

to

oe

-LSB-

R

F

-RSB-

When

jRj

j?Rj

it

is

cheaper

to

re-evaluate

the

query

expression

oeP

-LRB-

R

-RRB-

by

using

the

differential

select

operator

DiffSelect

-LRB-

R

P

-RRB-

rather

than

recomputing

the

expression

oeP

-LRB-

R

-RRB-

from

scratch

The

Differential

Project

Operator

Definition

-LRB-

Differential

project

-RRB-

Let

R

be

a

relation

-LRB-

base

or

derived

-RRB-

of

scheme

R

-LRB-

An

-RRB-

Let

X

Aikg

denotes

a

subset

of

the

attributes

in

R

X

and

ij

ng

-LRB-

j

k

-RRB-

Let

P

denote

the

predicate

Wkj

-LRB-

A

<old>

ij

A

<new>

ij

-RRB-

if

R

is

a

-LRB-

persistent

-RRB-

base

relation

otherwise

P

denote

the

truth

value

true

The

differential

project

operator

DiffProj

is

defined

as

follows

DiffProj

-LSB-

R

X

-RSB-

oeP

-LRB-

-LSB-

R

old

-LRB-

X

-RRB-

Snew

-LRB-

X

-RRB-

-RSB-

-RRB-

The

following

proposition

shows

that

DiffProj

is

an

efficient

differential

form

that

can

be

used

to

implement

Propagate

-LRB-

-LSB-

R

X

-RSB-

-LRB-

R

R

-RRB-

-RRB-

because

they

are

functionally

equivalent

DiffProj

runs

one

pass

over

R

and

the

Propagate

runs

over

which

tends

to

be

much

larger

than

R.

Proposition

Propagate

-LRB-

-LSB-

R

X

-RSB-

-LRB-

R

R

-RRB-

-RRB-

DiffProj

-LSB-

R

old

-LRB-

X

-RRB-

Snew

-LRB-

X

-RRB-

-RSB-

Proof

-LRB-

see

Appendix

A

-RRB-

Example

Consider

the

following

two

queries

over

the

relation

Stocks

in

Example

-LSB-

Stocks

Name

-RSB-

-LSB-

oeprice

-LRB-

Stocks

-RRB-

Name

-RSB-

Let

HighStocks

oe

-LRB-

Stocks

Price

-RRB-

By

Definition

the

evaluation

of

Propagate

-LRB-

oeprice

-LRB-

Stocks

-RRB-

-LSB-

Stocks

Stocks

-RSB-

-RRB-

requires

as

the

inputs

both

the

base

relation

Stocks

before

the

updates

and

the

base

relation

after

the

updates

which

is

Stocks

According

to

Proposition

and

Definition

it

would

be

more

efficient

to

process

both

queries

-LRB-

and

-RRB-

using

the

differential

operator

DiffProj

than

directly

using

the

definition

of

Propagate

ffl

Propagate

-LRB-

-LSB-

Stocks

Name

-RSB-

-LSB-

Stocks

Stocks

-RSB-

-RRB-

DiffProj

-LSB-

Stocks

Name

-RSB-

tid

<old>

Name

<old>

tid

<new>

Name

<new>

MAC

OLI

ffl

Propagate

-LRB-

-LSB-

HighStocks

old

-LRB-

Name

-RRB-

-LSB-

new

-LRB-

Name

-RRB-

-RSB-

-LSB-

HighStocks

HighStocks

-RSB-

-RRB-

DiffProj

-LSB-

HighStocks

old

-LRB-

Name

-RRB-

-LSB-

new

-LRB-

Name

-RRB-

-RSB-

-LRB-

by

Proposition

-RRB-

DiffProj

-LSB-

Propagate

-LRB-

HighStocks

-LSB-

Stocks

Stocks

-RSB-

-RRB-

old

-LRB-

Name

-RRB-

-LSB-

new

-LRB-

Name

-RRB-

-RSB-

-LRB-

by

Lemma

-RRB-

DiffProj

-LSB-

DiffSelect

-LRB-

Stocks

-RRB-

old

-LRB-

Name

-RRB-

-LSB-

new

-LRB-

Name

-RRB-

-RSB-

-LRB-

by

Proposition

-RRB-

tid

<old>

Name

<old>

tid

<new>

Name

<new>

DEC

DEC

OLI

The

Differential

Join

Operator

We

first

consider

the

join

over

two

relations

and

-LRB-

i.e.

n

-RRB-

In

this

case

changes

to

the

resulting

relation

can

be

induced

by

changes

to

either

input

operand

or

both

of

them

Therefore

the

differential

join

operator

DiffJoin

should

consider

the

following

three

cases

when

computing

the

total

changes

to

the

result

relation

of

-LRB-

i

-RRB-

only

changes

-LRB-

ii

-RRB-

only

changes

and

-LRB-

iii

-RRB-

both

and

change

Definition

-LRB-

Differential

join

-RRB-

Let

Pjoin

be

a

predicate

on

and

and

let

and

respectively

Let

P

<old>

join

and

P

<new>

join

denote

the

predicates

obtained

from

Pjoin

by

attaching

each

attribute

name

with

superscript

old

and

new

respectively

Let

P

-LRB-

-RRB-

denote

the

predicate

-LRB-

tid

<old>

tid

<new>

-RRB-

AND

-LRB-

tid

<old>

tid

<new>

-RRB-

Let

R

denote

-LSB-

We

define

the

operator

DiffJoin

as

follows

DiffJoinP

-LRB-

-RRB-

-LRB-

-RRB-

DJoinPjoin

-LRB-

-RRB-

SDJoinPjoin

-LRB-

-RRB-

SDJoinPjoin

-LRB-

-RRB-

where

ffl

DJoinPjoin

-LRB-

-RRB-

OuterjoinP

-LRB-

-RRB-

-LRB-

attach

-LSB-

P

<old>

join

-LRB-

deletions

-LRB-

-RRB-

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

P

<new>

join

-LRB-

insertions

-LRB-

-RRB-

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

ffl

DJoinPjoin

-LRB-

-RRB-

OuterjoinP

-LRB-

-RRB-

-LRB-

attach

-LSB-

P

<old>

join

-LRB-

deletions

-LRB-

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

P

<new>

join

-LRB-

insertions

-LRB-

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

ffl

DJoinPjoin

-LRB-

-RRB-

OuterjoinP

-LRB-

-RRB-

-LRB-

P

<old>

join

-LRB-

deleteLog

-LRB-

-RRB-

deleteLog

-LRB-

-RRB-

-RRB-

P

<new>

join

-LRB-

insertLog

-LRB-

-RRB-

insertLog

-LRB-

-RRB-

-RRB-

Proposition

Propagate

-LRB-

Pjoin

-LRB-

-RRB-

-LSB-

-LRB-

-RRB-

-LRB-

-RRB-

-RSB-

-RRB-

DiffJoinP

-LRB-

-RRB-

-LRB-

-RRB-

Proof

-LRB-

see

Appendix

A

-RRB-

This

development

can

be

generalized

to

the

join

of

an

arbitrary

number

of

base

relations

Let

Q

Rn

-LRB-

n

-RRB-

denote

an

arbitrary

join

expression

The

equation

Propagate

-LRB-

Rn

SL

-RRB-

tidn

-LRB-

Rn

-RRB-

holds

in

general

Also

when

the

number

k

-LRB-

k

n

-RRB-

relations

have

been

changed

since

the

last

execution

of

Q

to

evaluate

tidn

-LRB-

Rn

-RRB-

we

need

to

consider

only

cases

each

representing

one

type

of

change

effects

The

total

changes

to

the

result

of

Q

will

be

the

union

of

these

cases

According

to

the

associative

and

symmetric

property

of

relational

join

we

may

assume

that

the

first

k

relations

-LRB-

i.e.

Rk

k

n

-RRB-

are

those

that

have

been

changed

without

loss

of

generality

By

associating

a

truth

table

T

with

k

columns

and

p

rows

to

the

query

Q

it

is

easy

to

compute

each

possible

combination

of

joins

which

needs

to

be

considered

when

computing

the

changes

to

the

result

of

Q

after

k

operand

relations

have

been

changed

Each

column

of

the

T

table

corresponds

to

an

updated

relation

in

Q

since

the

last

execution

of

Q.

Each

row

represents

one

possible

case

that

the

computation

of

DiffJoin

-LRB-

Rn

-RRB-

considers

Example

For

a

query

Q

Rn

-LRB-

n

-RRB-

consider

the

case

when

k

and

k

n.

Based

on

the

T

table

associated

with

Q

below

we

need

to

consider

only

seven

cases

-LRB-

-RRB-

all

change

-LRB-

-RRB-

only

change

-LRB-

-RRB-

only

change

-LRB-

-RRB-

only

change

-LRB-

-RRB-

only

changes

-LRB-

-RRB-

only

changes

-LRB-

-RRB-

only

changes

Each

row

in

the

T

table

corresponds

to

one

possible

combination

of

join

required

to

compute

the

changes

to

the

result

of

Q.

DiffJoin

-LRB-

Rn

-RRB-

DJoin

-LRB-

Rn

-RRB-

-LSB-

DiffJoin

-LRB-

Rn

-RRB-

-LSB-

DJoin

-LRB-

Rn

-RRB-

-LSB-

DJoin

-LRB-

Rn

-RRB-

-LSB-

DJoin

-LRB-

Rn

-RRB-

-LSB-

DJoin

-LRB-

Rn

-RRB-

-LSB-

DJoin

-LRB-

Rn

-RRB-

Let

Ri

be

the

scheme

of

relation

Ri

-LRB-

i

-RRB-

and

R

denote

-LSB-

ni

The

definitions

for

the

above

DJoins

are

similar

to

those

in

Definition

For

instance

DJoinPjoin

-LRB-

Rn

-RRB-

Outerjointid

-LRB-

attach

-LSB-

deletions

-LRB-

-RRB-

Rn

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

insertions

-LRB-

-RRB-

Rn

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

DJoinPjoin

-LRB-

Rn

-RRB-

Outerjointid

-LRB-

attach

-LSB-

deletions

-LRB-

-RRB-

deletions

-LRB-

-RRB-

Rn

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

insertions

-LRB-

-RRB-

insertions

-LRB-

-RRB-

Rn

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

Observe

that

the

above

example

exhibits

an

interesting

optimization

problem

namely

the

efficient

execution

of

a

set

of

n-ary

join

expressions

in

which

intermediate

results

can

be

reused

among

several

SPJ

expressions

For

instance

when

n

in

the

above

query

Q

let

Rn

and

Saving

and

as

intermediate

results

and

then

re-using

them

in

the

evaluation

of

each

of

the

seven

DJoin

expressions

above

we

may

easily

speed

up

the

processing

of

DiffJoin

This

mechanism

works

effectively

when

n

is

larger

than

k

The

idea

of

using

the

truth

table

to

facilitate

the

combination

of

possible

joins

was

borrowed

from

the

research

in

updating

materialized

view

-LSB-

-RSB-

We

minimize

the

cost

of

constructing

such

a

table

by

using

as

the

columns

of

the

table

only

the

number

of

changed

relations

instead

of

the

n

operand

relations

in

the

query

expression

X

-LRB-

oeF

-LRB-

Rn

-RRB-

-RRB-

The

Differential

Re-evaluation

Algorithm

We

now

outline

an

algorithm

for

re-evaluating

continual

queries

-LRB-

limited

to

SPJ

expressions

-RRB-

differentially

Algorithm

-LRB-

The

DRA

algorithm

-RRB-

Input

ffl

the

SPJ

definition

of

the

continual

query

Q

i.e.

Q

X

-LRB-

oeF

-LRB-

Rn

-RRB-

-RRB-

where

X

denotes

the

projection

list

and

F

denotes

the

selection

predicate

over

Rn

ffl

the

contents

of

the

base

relations

Ri

-LRB-

i

n

-RRB-

after

the

last

execution

of

the

CQ

ffl

the

differential

relations

Ri

-LRB-

i

n

-RRB-

ffl

the

timestamp

of

the

last

execution

of

this

CQ

say

Ei

ffl

the

complete

set

of

the

result

produced

by

the

last

execution

of

the

CQ

Output

the

result

of

the

current

execution

of

the

query

Q.

Procedural

Steps

Build

the

truth

table

T

with

k

columns

-LRB-

k

n

-RRB-

and

p

rows

p

Each

column

is

corresponding

to

a

relation

in

the

SPJ

expression

which

has

been

changed

since

the

last

execution

of

Q.

For

each

row

i

-LRB-

i

p

-RRB-

of

the

table

T

construct

the

associated

SPJ

expression

by

substituting

Ri

in

Q

with

Ri

when

the

binary

variable

Tij

For

each

of

these

SPJ

expression

denoted

by

G

Sn

evaluate

G

by

its

differential

form

DJoin

-LRB-

Sn

-RRB-

Perform

the

union

of

the

results

obtained

from

each

computation

in

Step

Based

on

the

epsilon

specification

of

the

CQ

assemble

the

final

set

of

the

result

to

be

returned

to

the

users

For

example

let

RQ

denote

the

result

generated

by

Step

ffl

If

the

user

wishes

to

see

only

the

differential

result

since

the

last

execution

of

the

Q

say

Ei

-LRB-

Q

ti

-RRB-

without

deletion

notification

the

result

to

be

returned

can

be

computed

by

oetimestamp

ti

-LRB-

insertions

-LRB-

RQ

-RRB-

-RRB-

ffl

If

the

user

needs

to

see

the

complete

set

of

the

result

matching

the

query

we

return

Ei

-LRB-

Q

ti

-RRB-

S

oetimestamp

ti

-LRB-

insertions

-LRB-

RQ

-RRB-

-RRB-

ffl

If

the

user

wants

to

be

notified

all

the

deleted

tuples

since

the

last

execution

of

the

CQ

we

simply

compute

the

oetimestamp

ti

-LRB-

deletions

-LRB-

RQ

-RRB-

-RRB-

Discussion

In

previous

sections

we

have

defined

the

differential

forms

for

the

three

basic

relational

algebraic

operations

Select

Project

and

Join

and

developed

a

differential

re-evaluation

algorithm

-LRB-

DRA

-RRB-

The

key

idea

of

the

DRA

method

is

to

transform

a

continual

query

over

the

source

data

-LRB-

base

relations

-RRB-

into

a

differential

query

that

runs

over

the

corresponding

delta

relations

We

have

proved

that

for

any

SPJ

expression

using

differential

re-evaluation

method

is

functionally

equivalent

to

the

complete

re-evaluation

of

the

query

In

this

section

we

will

discuss

a

number

of

issues

related

to

the

implementation

of

continual

queries

in

the

Diorama

project

-LSB-

-RSB-

Improving

Performance

in

Compilation

and

Execution

Our

differential

algorithm

for

query

refresh

does

not

automatically

provide

the

most

efficient

way

of

processing

CQs

There

are

a

number

of

possibilities

to

further

improve

the

performance

of

differential

processing

of

continual

queries

First

we

should

test

the

CQ

condition

based

on

the

differential

updates

before

every

execution

If

the

updates

occurred

in

between

of

the

two

consecutive

executions

have

no

impact

on

the

previous

query

result

set

we

consider

them

as

irrelevant

updates

to

the

continual

query

Thus

no

computation

is

performed

for

this

CQ

because

in

this

case

nothing

needs

to

be

returned

if

the

user

is

only

interested

in

the

differential

result

When

the

user

asks

for

the

complete

answer

we

simply

return

the

result

of

previous

execution

In

addition

for

each

SPJ

expression

in

step

of

the

DRA

algorithm

it

is

necessary

to

determine

its

execution

strategy

One

way

to

find

a

good

execution

strategy

is

simply

to

use

the

heuristics

such

as

Select

before

Join

extracting

common

subexpressions

cheaper

selection

predicate

before

expensive

ones

-RRB-

This

approach

might

be

most

appropriate

if

one

does

not

have

access

to

an

appropriate

query

optimizer

An

alternative

approach

is

to

have

a

DBMS

query

optimizer

generate

the

strategies

The

differential

form

of

a

query

can

be

regarded

as

a

query

to

a

database

that

consists

of

base

relations

and

differential

relations

Another

benefit

is

that

the

DBMS

supports

relational

operators

-LRB-

select

project

join

etc.

-RRB-

and

an

interpreter

for

the

strategies

that

are

produced

Otherwise

we

need

to

implement

these

capabilities

as

part

of

the

CQ

manager

However

there

are

also

serious

drawbacks

to

using

a

conventional

DBMS

that

is

non

extensible

In

this

case

integration

of

the

CQ

manager

may

be

difficult

or

impossible

One

must

find

a

way

to

add

operators

that

manipulate

differential

relations

Differential

Evaluation

of

Continual

Query

Conditions

As

described

in

Section

a

continual

query

CQ

consists

of

a

normal

query

Q

-LRB-

e.g.

written

in

SQL

-RRB-

an

epsilon

specification

-LRB-

ffl-spec

-RRB-

as

the

triggering

condition

and

a

termination

condition

Stop

We

use

the

term

continual

query

condition

to

refer

to

the

refresh

condition

TCQ

-LRB-

ffl-spec

-RRB-

and

the

termination

condition

Stop

Whenever

the

ffl-spec

of

a

given

ECQ

is

satisfied

the

CQ

manager

will

invoke

the

differential

re-evaluation

algorithm

to

execute

the

query

Q

A

continual

query

ceases

to

exist

when

its

termination

condition

Stop

becomes

true

The

next

question

that

needs

to

be

answered

is

when

and

how

the

CQ

manager

checks

the

refresh

condition

TCQ

usually

an

ffl-spec

and

the

termination

condition

Stop

Given

an

ECQ

if

its

ffl-spec

is

a

direct

specification

of

time

-LRB-

e.g.

every

Monday

-RRB-

or

a

time

interval

with

respect

to

the

previous

query

result

-LRB-

e.g.

a

week

since

the

last

execution

of

query

Q

-RRB-

then

the

CQ

manager

will

use

the

ffl-spec

as

the

default

for

triggering

the

CQ

and

testing

the

Stop

condition

However

if

the

ffl-spec

and

the

Stop

condition

are

bounded

to

a

database

state

or

a

relationship

between

a

previous

query

result

and

the

current

database

state

-LRB-

recall

examples

in

Section

-RRB-

then

the

CQ

manager

will

decide

when

and

how

to

evaluate

TCQ

There

are

two

general

strategies

concerning

when

to

test

the

TCQ

The

first

is

to

evaluate

TCQ

immediately

after

each

update

to

the

source

data

-LRB-

base

relations

-RRB-

The

second

is

to

evaluate

TCQ

periodically

In

the

Diorama

system

both

can

be

implemented

For

concreteness

we

discuss

the

second

choice

here

Consider

the

checking

account

example

discussed

in

Section

a

bank

manager

wants

to

know

how

many

millions

of

dollars

she

has

in

all

the

checking

accounts

This

query

Q

can

be

expressed

in

SQL

as

SELECT

SUM

-LRB-

amount

-RRB-

FROM

CheckAcc

Suppose

it

is

installed

as

a

continual

query

with

the

following

TCQ

-LRB-

and

ffl-spec

-RRB-

jDeposits

Withdrawalsj

M

and

Stop

Deposits

and

Withdrawals

denote

the

total

amount

of

deposits

and

withdrawals

since

the

previous

execution

of

the

query

respectively

Since

there

is

no

specific

time

in

the

ffl-spec

of

this

query

the

CQ

manager

can

decide

when

to

evaluate

TCQ

by

a

system-defined

default

interval

say

every

day

at

midnight

If

the

ffl-spec

is

greater

than

half

a

million

the

query

Q

is

executed

In

the

Diorama

system

the

differential

evaluation

method

will

be

used

to

verify

TCQ

whenever

possible

In

this

example

the

CQ

manager

first

rewrites

TCQ

originally

specified

as

jDeposits

Withdrawalsj

M

which

requires

testing

of

the

base

relation

CheckAcc

The

new

TCQ

has

a

differential

form

that

requires

only

scanning

CheckAcc

Deposits

SELECT

SUM

-LRB-

account

-RRB-

FROM

insertions

-LRB-

CheckAcc

-RRB-

WHERE

timestamp

ti

Withdrawals

SELECT

SUM

-LRB-

account

-RRB-

FROM

deletions

-LRB-

CheckAcc

-RRB-

WHERE

timestamp

ti

where

ti

denotes

the

timestamp

of

the

previous

execution

Q

-LRB-

Si

-RRB-

In

general

the

cost

of

evaluating

the

differential

form

of

TCQ

is

cheaper

than

the

complete

re-evaluation

of

TCQ

over

the

entire

base

relations

In

this

example

it

is

true

when

jCheckAccj

j?CheckAccj

Garbage

Collection

of

Differential

Relations

As

the

source

data

changes

their

differential

relations

grow

accordingly

To

keep

the

differential

relations

to

a

bounded

size

we

need

to

garbage

collect

the

portions

of

the

differential

relations

that

are

no

longer

useful

The

technical

details

of

the

solution

are

beyond

the

scope

of

this

paper

We

outline

the

basic

idea

here

First

let

us

consider

the

case

of

a

single

active

CQ

in

the

system

Each

time

a

new

query

result

Q

-LRB-

Si

-RRB-

is

obtained

we

can

retire

the

differential

relations

referring

to

database

states

prior

to

ti

This

is

intuitively

easy

to

understand

since

only

the

data

in

the

differential

relations

with

the

timestamps

later

than

ti

will

be

needed

for

the

processing

of

Q

-LRB-

Si

-RRB-

Informally

we

call

these

portions

of

the

differential

relations

active

delta

zone

With

multiple

active

CQs

in

the

system

the

garbage

collection

algorithm

is

an

extension

of

the

basic

idea

outlined

above

For

each

CQ

we

define

its

active

delta

zone

For

the

whole

system

we

define

the

system

active

delta

zone

as

the

union

of

the

active

delta

zones

of

all

CQs

Assuming

that

each

CQ

will

make

progress

its

active

delta

zone

will

move

forward

in

time

The

system

active

delta

zone

will

move

forward

as

a

consequence

with

its

boundary

delimited

by

the

oldest

active

delta

zone

All

the

data

in

the

differential

relations

that

fall

outside

the

system

active

delta

zone

can

be

garbage

collected

since

they

will

not

be

used

by

any

active

CQ

Conclusion

Information

monitoring

in

the

Internet

is

an

increasingly

important

research

topic

due

to

two

conflicting

issues

On

the

positive

side

the

exponential

growth

of

Internet

makes

exponentially

more

data

available

and

information

monitoring

similarly

more

rewarding

On

the

negative

side

the

same

growth

increases

overhead

similarly

making

information

monitoring

more

difficult

and

expensive

In

this

paper

we

address

the

two

issues

with

two

complementary

ideas

First

we

define

Continual

Queries

-LRB-

CQ

-RRB-

to

facilitate

the

query

specification

for

information

monitoring

Each

CQ

is

a

sequence

of

query

executions

marked

by

a

normal

database

query

for

example

formulated

using

SQL

The

sequence

is

started

at

CQ

submission

time

refreshed

from

time

to

time

by

a

triggering

condition

TCQ

and

terminated

by

a

stopping

condition

Our

triggering

conditions

are

more

expressive

than

previous

proposals

such

as

continuous

queries

-LSB-

-RSB-

since

TCQ

can

describe

time

of

day

intervals

of

time

the

magnitude

of

updates

or

accumulated

updates

in

a

way

similar

to

epsilon

transactions

-LSB-

-RSB-

These

new

CQs

are

called

Epsilon

CQs

-LRB-

ECQs

-RRB-

Second

we

describe

a

differential

re-evaluation

algorithm

to

support

efficient

processing

of

ECQs

Given

the

distributed

nature

of

Internet

complete

reprocessing

of

the

ECQ

query

each

time

it

is

triggered

would

be

prohibitively

expensive

The

differential

re-evaluation

algorithm

is

based

on

differential

relations

summaries

of

updates

that

allow

efficient

incremental

processing

of

the

next

result

in

the

ECQ

sequence

by

combining

the

differential

relations

with

the

current

ECQ

result

We

have

defined

a

high

level

operator

Propagate

as

an

instantiation

of

complete

re-evaluation

solution

and

formally

prove

that

our

differential

re-evaluation

algorithm

to

continual

queries

is

functionally

equivalent

to

the

recompute

the

query

from

scratch

solution

and

in

many

situations

is

more

efficient

In

addition

to

a

formal

description

of

delta

relations

-LRB-

Section

-RRB-

and

the

differential

re-evaluation

algorithm

-LRB-

Section

-RRB-

we

also

discuss

a

number

of

implementation

issues

such

as

differential

evaluation

of

continual

query

conditions

and

strategies

for

garbage

collection

of

delta

relations

-LRB-

Section

-RRB-

Although

we

have

addressed

both

the

positive

and

negative

aspects

of

differential

evaluation

of

continual

queries

for

large-scale

information

monitoring

by

no

means

have

we

exhausted

the

topic

Many

interesting

challenges

remain

in

the

area

of

information

monitoring

on

the

Internet

for

example

integration

of

information

from

heterogeneous

sources

other

algorithms

for

differential

or

incremental

evaluation

of

ECQs

performance

evaluation

of

the

query

processing

techniques

and

the

implementation

of

the

research

ideas

into

actual

software

tools

Our

current

research

is

focused

on

the

refinement

of

algorithms

and

their

implementation

References

-LSB-

-RSB-

R.

Alonso

D.

Barbara

and

H.

Garcia-Molina

Data

caching

issues

in

an

information

retrieval

system

ACM

Transactions

on

Database

Systems

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

D.

Barbara

and

R.

Alonso

Processing

continuous

queries

in

general

environments

Technical

report

Matsushita

Information

Technology

Laboratory

Princeton

NJ

June

-LSB-

-RSB-

J.

Blakeley

P.

Larson

and

F.

Tompa

Efficiently

updating

materialized

views

In

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

Washington

DC

May

-LSB-

-RSB-

C.

Bowman

P.

Danzig

D.

Hardy

U.

Manber

and

M.

F.

Schwartz

The

harvest

information

discovery

and

access

system

In

Proceedings

of

the

Second

International

World

Wide

Web

Conference

pages

-LCB-

Chicago

Illinois

October

-LSB-

-RSB-

C.

Bowman

P.

Danzig

D.

Hardy

U.

Manber

M.

F.

Schwartz

and

D.

P.

Wessels

Harvest

A

scalable

customizable

discovery

and

access

system

Technical

report

Department

of

Computer

Science

University

of

Colorado

Technical

Report

August

-LSB-

-RSB-

S.

Chakravarthy

Architectures

and

monitoring

techniques

for

active

databases

An

evaluation

In

Technical

Report

University

of

Florida

Gainesville

FL

-LSB-

-RSB-

A.

Gupta

I.

Mumick

and

V.

Subrahmanian

Maintaining

views

incrementally

In

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

Washington

DC

May

-LSB-

-RSB-

E.

N.

Hanson

A

performance

analysis

of

view

materialization

strategies

In

Proceedings

of

the

ACM-SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

San

Francisco

CA

May

-LSB-

-RSB-

Informix

Software

Inc.

Informix

Guide

to

SQL

Syntax

-LRB-

Version

-RRB-

-LSB-

-RSB-

B.

Kahler

and

O.

Risnes

Extending

logging

for

database

snapshot

refresh

In

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pages

-LCB-

Brighton

England

September

-LSB-

-RSB-

B.

Lindsay

L.

Haas

and

C.

Mohan

A

snapshot

differential

refresh

algorithm

In

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

Washington

DC

May

-LSB-

-RSB-

L.

Liu

and

C.

Pu

The

distributed

interoperable

object

model

and

its

application

to

large-scale

interoperable

database

systems

Technical

report

Portland

OR

March

-LSB-

-RSB-

L.

Liu

and

C.

Pu

The

distributed

interoperable

object

model

and

its

application

to

large-scale

interoperable

database

systems

Technical

Report

Department

of

Computer

Science

and

Engineering

Oregon

Graduate

Institute

February

-LSB-

-RSB-

D.

Maier

The

Theory

of

Relational

Databases

Computer

Science

Press

-LSB-

-RSB-

D.

McCarthy

and

U.

Dayal

The

architecture

of

an

active

database

management

system

In

Proceedings

of

the

ACM-SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

May

-LSB-

-RSB-

C.

Pu

and

S.

Chen

ACID

properties

need

fast

relief

Relaxing

consistency

using

epsilon

serializability

In

Proceedings

of

Fifth

International

Workshop

on

High

Performance

Transaction

Systems

Asilomar

California

September

-LSB-

-RSB-

C.

Pu

W.

Hseush

G.

Kaiser

K.-L

Wu

and

P.

Yu

Divergence

control

for

distributed

database

systems

International

Journal

on

Distributed

and

Parallel

Databases

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

C.

Pu

and

A.

Leff

Replica

control

in

distributed

systems

An

asynchronous

approach

In

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

Denver

May

-LSB-

-RSB-

K.

Ramamrithan

and

C.

Pu

A

formal

characterization

of

epsilon

serializability

IEEE

Transactions

on

Knowledge

and

Data

Engineering

June

-LSB-

-RSB-

A.

Rosenthal

S.

Chakravarthy

B.

Blaustein

and

J.

Blakely

Situation

monitoring

for

active

databases

In

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pages

-LCB-

Amsterdam

Holland

September

-LSB-

-RSB-

N.

Roussopoulos

and

H.

Kang

Preliminary

design

of

adms

A

workstation-mainframe

integrated

architecture

fopr

database

management

systems

In

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pages

-LCB-

Kyoto

Japan

August

-LSB-

-RSB-

U.

Schreier

H.

Pirahesh

R.

Agrawal

and

C.

Mohan

Alert

An

architecture

for

transforming

a

passive

dbms

into

an

active

dbms

In

Proceedings

of

the

International

Conference

on

Very

Large

Data

Bases

pages

-LCB-

Barcelona

Spain

September

-LSB-

-RSB-

A.

Segev

and

W.

Fang

Currency-based

updates

to

distributed

materialized

views

In

Proceedings

of

the

International

Conference

on

Data

Engineering

pages

-LCB-

Los

Alamitos

February

-LSB-

-RSB-

A.

Segev

and

J.

Park

Maintaining

materialized

views

in

distributed

databases

In

Proceedings

of

the

International

Conference

on

Data

Engineering

pages

-LCB-

Los

Angeles

February

-LSB-

-RSB-

D.

G.

Severance

and

G.

M.

Lohman

Differential

files

Their

application

to

the

maintenance

of

large

databases

ACM

Transactions

on

Database

Systems

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

M.

Stonebreaker

A.

Jhingran

J.

Goh

and

S.

Potamianos

On

rules

procedures

caching

and

views

in

database

systems

In

Proceedings

of

the

ACM-SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

Barcelona

Spain

-LSB-

-RSB-

D.

Terry

D.

Goldberg

D.

Nichols

and

B.

Oki

Continuous

queries

over

append-only

databases

In

Proceedings

of

the

ACM-SIGMOD

International

Conference

on

Management

of

Data

pages

-LCB-

San

Diego

CA

January

-LSB-

-RSB-

K.

Wu

P.

S.

Yu

and

C.

Pu

Divergence

control

for

epsilon-serializability

In

Proceedings

of

Eighth

International

Conference

on

Data

Engineering

pages

-LCB-

Phoenix

February

IEEE/Computer

Society

-LSB-

-RSB-

Y.

Zhuge

H.

Garcia-Molina

J.

Hammer

and

J.

Widom

View

maintenance

in

a

warehousing

environment

In

Proceedings

of

the

ACM-SIGMOD

International

Conference

on

Management

of

Data

San

Jose

CA

May

Appendix

A

Lemmas

and

Proposition

Proofs

Lemma

The

following

equations

hold

attach

-LSB-

additions

-LRB-

R

-RRB-

new

-LRB-

R

-RRB-

-RSB-

insertLog

-LRB-

R

-RRB-

attach

-LSB-

deletions

-LRB-

R

-RRB-

old

-LRB-

R

-RRB-

-RSB-

deleteLog

-LRB-

R

-RRB-

Outerjointid

-LRB-

deleteLog

-LRB-

R

-RRB-

insertLog

-LRB-

R

-RRB-

-RRB-

R.

R

combine

-LRB-

R

R

-RRB-

deletions

-LRB-

R

-RRB-

combine

-LRB-

R

R

-RRB-

R

insertions

-LRB-

R

-RRB-

The

proof

of

the

equations

in

Lemma

follows

straightforward

from

Definition

and

Definition

Lemma

The

following

equations

hold

Propagate

-LRB-

R

-LSB-

R

R

-RSB-

-RRB-

R.

combine

-LRB-

Q

-LRB-

Rn

-RRB-

Propagate

-LRB-

Q

PL

-RRB-

-RRB-

Q

-LRB-

-RRB-

Proposition

Propagate

-LRB-

oe

-LSB-

R

F

-RSB-

-LRB-

R

R

-RRB-

-RRB-

DiffSelect

-LSB-

R

F

-RSB-

Proof

Let

RHS

and

LHS

represent

the

right

and

left

hand

side

of

the

equation

Let

the

condition

tid

of

the

Outerjointid

denote

tid

<old>

tid

<new>

F

<old>

and

F

<new>

denote

the

same

predicate

with

all

attribute

names

superscripted

accordingly

By

Definition

RHS

Outerjointid

-LRB-

oe

-LSB-

deleteLog

-LRB-

R

-RRB-

F

<old>

-RSB-

oe

-LSB-

insertLog

-LRB-

R

-RRB-

F

<new>

-RSB-

-RRB-

Let

oe

-LSB-

R

F

-RSB-

oe

-LSB-

combine

-LRB-

R

R

-RRB-

F

-RSB-

oe

-LSB-

R?combine

-LRB-

R

R

-RRB-

F

-RSB-

oe

-LSB-

deletions

-LRB-

R

-RRB-

F

-RSB-

-LRB-

by

Lemma

-LRB-

-RRB-

-RRB-

oe

-LSB-

combine

-LRB-

R

R

-RRB-

F

-RSB-

oe

-LSB-

R

F

-RSB-

oe

-LSB-

combine

-LRB-

R

R

-RRB-

R

F

-RSB-

oe

-LSB-

insertions

-LRB-

R

-RRB-

F

-RSB-

-LRB-

by

Lemma

-LRB-

-RRB-

-RRB-

By

Definition

-LRB-

Propagate

-RRB-

and

Definition

-LRB-

Diff

-RRB-

LHS

Diff

-LRB-

oe

-LSB-

R

F

-RSB-

oe

-LSB-

combine

-LRB-

R

R

-RRB-

F

-RSB-

-RRB-

Outerjointid

-LRB-

attach

-LSB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

Outerjointid

-LRB-

oe

-LSB-

deleteLog

-LRB-

R

-RRB-

F

<old>

-RSB-

oe

-LSB-

insertLog

-LRB-

R

-RRB-

F

<new>

-RSB-

-RRB-

-LRB-

Definition

-RRB-

Proposition

Propagate

-LRB-

-LSB-

R

X

-RSB-

-LRB-

R

R

-RRB-

-RRB-

DiffProj

-LSB-

R

old

-LRB-

X

-RRB-

Snew

-LRB-

X

-RRB-

-RSB-

Proof

Let

-LSB-

R

X

-RSB-

-LSB-

combine

-LRB-

R

R

-RRB-

X

-RSB-

-LSB-

R?combine

-LRB-

R

R

-RRB-

X

-RSB-

-LSB-

deletions

-LRB-

R

-RRB-

X

-RSB-

-LRB-

Lemma

-LRB-

-RRB-

-RRB-

-LSB-

combine

-LRB-

R

R

-RRB-

X

-RSB-

-LSB-

R

X

-RSB-

-LSB-

combine

-LRB-

R

R

-RRB-

R

X

-RSB-

-LSB-

insertions

-LRB-

R

-RRB-

X

-RSB-

-LRB-

Lemma

-LRB-

-RRB-

-RRB-

The

expression

-LSB-

RTcombine

-LRB-

R

R

-RRB-

X

-RSB-

returns

the

set

of

X

tuples

for

which

their

attributes

are

unchanged

By

Definition

LHS

Diff

-LRB-

-LSB-

R

X

-RSB-

-LSB-

combine

-LRB-

R

R

-RRB-

X

-RSB-

-RRB-

Outerjointid

-LRB-

attach

-LSB-

old

-LRB-

X

-RRB-

-RSB-

attach

-LSB-

new

-LRB-

X

-RRB-

-RSB-

-RRB-

Outerjointid

-LRB-

-LSB-

deleteLog

-LRB-

R

-RRB-

old

-LRB-

X

-RRB-

-RSB-

-LSB-

insertLog

-LRB-

R

-RRB-

new

-LRB-

X

-RRB-

-RSB-

-RRB-

-LRB-

by

Lemma

-LRB-

-RRB-

-LRB-

-RRB-

and

Definition

-RRB-

oeP

-LRB-

-LSB-

R

old

-LRB-

X

-RRB-

Snew

-LRB-

X

-RRB-

-RSB-

-RRB-

-LRB-

by

Lemma

-LRB-

-RRB-

and

Definition

-RRB-

RHL

In

the

proof

we

have

used

the

distributive

property

of

projection

over

difference

-LRB-

i.e.

X

-LRB-

-RRB-

X

-LRB-

-RRB-

X

-LRB-

-RRB-

-RRB-

which

holds

under

the

assumption

that

each

tuple

has

a

unique

immutable

identifier

-LRB-

recall

Section

-RRB-

This

is

because

under

this

assumption

the

projection

is

re-defined

as

X

-LRB-

R

-RRB-

tid

t

tid

-LSB-

X

-RSB-

t

-LSB-

X

-RSB-

g

and

the

result

relation

scheme

is

ftidg

-LSB-

X

instead

of

X

However

the

distributive

property

of

projection

over

difference

does

not

hold

for

the

traditional

relational

project

operator

which

eliminates

the

duplicates

in

its

result

relation

-LSB-

-RSB-

That

is

X

-LRB-

-RRB-

X

-LRB-

-RRB-

X

-LRB-

-RRB-

An

alternative

way

to

maintain

the

distributive

property

of

projection

over

difference

is

to

associate

a

multiplicity

counter

to

each

tuple

in

a

relation

This

approach

has

been

used

by

many

algorithms

for

updating

materialized

views

-LSB-

-RSB-

Proposition

Propagate

-LRB-

Pjoin

-LRB-

-RRB-

-LSB-

-LRB-

-RRB-

-LRB-

-RRB-

-RSB-

-RRB-

DiffJoinP

-LRB-

-RRB-

-LRB-

-RRB-

Proof

Let

Pjoin

-LRB-

-RRB-

Pjoin

-LRB-

combine

-LSB-

-RSB-

combine

-LSB-

-RSB-

-RRB-

-RRB-

Pjoin

-LRB-

deletions

-LRB-

-RRB-

-RRB-

S

Pjoin

-LRB-

deletions

-LRB-

-RRB-

-RRB-

S

Pjoin

-LRB-

deletions

-LRB-

-RRB-

deletions

-LRB-

-RRB-

-RRB-

Pjoin

-LRB-

combine

-LSB-

-RSB-

combine

-LSB-

-RSB-

-RRB-

-RRB-

Pjoin

-LRB-

-RRB-

Pjoin

-LRB-

insertions

-LRB-

-RRB-

-RRB-

S

Pjoin

-LRB-

insertions

-LRB-

-RRB-

-RRB-

S

Pjoin

-LRB-

insertions

-LRB-

-RRB-

insertions

-LRB-

-RRB-

-RRB-

By

Definition

and

Definition

LHS

Diff

-LRB-

Pjoin

-LRB-

-RRB-

Pjoin

-LRB-

combine

-LSB-

-RSB-

combine

-LSB-

-RSB-

-RRB-

-RRB-

OuterjoinP

-LRB-

-RRB-

-LRB-

attach

-LSB-

old

-LRB-

R

-RRB-

Snew

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

old

-LRB-

R

-RRB-

Snew

-LRB-

R

-RRB-

-RSB-

-RRB-

OuterjoinP

-LRB-

-RRB-

-LRB-

attach

-LSB-

P

<old>

join

-LRB-

deletions

-LRB-

-RRB-

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

P

<new>

join

-LRB-

insertions

-LRB-

-RRB-

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

S

OuterjoinP

-LRB-

-RRB-

-LRB-

attach

-LSB-

P

<old>

join

-LRB-

deletions

-LRB-

-RRB-

-RRB-

old

-LRB-

R

-RRB-

-RSB-

attach

-LSB-

P

<new>

join

-LRB-

insertions

-LRB-

-RRB-

-RRB-

new

-LRB-

R

-RRB-

-RSB-

-RRB-

S

OuterjoinP

-LRB-

-RRB-

-LRB-

P

<old>

join

-LRB-

deleteLog

-LRB-

-RRB-

deleteLog

-LRB-

-RRB-

-RRB-

P

<new>

join

-LRB-

insertLog

-LRB-

-RRB-

insertLog

-LRB-

-RRB-

-RRB-

RHS

continual queries
epsilon serializability
information monitoring
internet

differential data
incremental query processing
information superhighways