Implementation of the Pan-European Academic Research Network: TEN-155 AbstractThis paper addresses the implementation of a pan-European The success of EuropaNet [1] and TEN-34 [2] has demonstrated that a dedicated networking infrastructure to the European research community is essential for successful co-operation amongst European researchers. Both these networks have had a short life-time, mainly due to the excessive cost of international capacity. Nevertheless, with the help of funding from the European Commission (EC) it has been possible to deploy them and demonstrate their vital importance to the research community. The paper will outline the Quantum project and describe in detail the resulting TEN-155 network which supersedes TEN-34 in terms of available bandwidth and above all in the ability to offer a Managed Bandwidth Service to guarantee end-to-end Quality of Service. TEN-155 makes use of ATM, considered the most effective technology to offer guaranteed capacity end-to-end. A section is dedicated to the Quantum Test Programme, the purpose of which is to evaluate new and emerging technologies and investigate the possibility of their deployment on the production network. Finally, the paper will focus on a description of the Managed Bandwidth Service (MBS), in terms of its organisation and development phase. The Quantum project has been successful in deploying a pan-European network for research, providing a huge increase in capacity when compared to its predecessors and at a similar price. This is mainly due to developments in the European telecommunications market. From the technical point of view, TEN-155 has been designed to make efficient use of the available bandwidth and fair sharing of the bandwidth in situations of congestion. These targets have been met by combining different ATCs, and the results obtained in test laboratories confirmed the theoretical expectations. Rollout of the MBS and the work carried out within the Quantum Test Programme is expected to enhance even further the capabilities of the network, with the deployment of native multicast and developments in the area of the co-existence of differentiated and integrated services.

O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


MGR:

An

information

theory

based

hierarchical

divisive

clustering

algorithm

for

categorical

data

Categorical

data

clustering

has

attracted

much

attention

recently

due

to

the

fact

that

much

of

the

data

contained

in

today's

databases

is

categorical

in

nature.

While

many

algorithms

for

clustering

categorical

data

have

been

proposed,

some

have

low

clustering

accuracy

while

others

have

high

computational

complexity.

This

research

proposes

mean

gain

ratio

(MGR),

a

new

information

theory

based

hierarchical

divisive

clustering

algorithm

for

categorical

data.

MGR

implements

clustering

from

the

attributes

viewpoint

which

includes

selecting

a

clustering

attribute

using

mean

gain

ratio

and

selecting

an

equivalence

class

on

the

clustering

attribute

using

entropy

of

clusters.

It

can

be

run

with

or

without

specifying

the

number

of

clusters

while

few

existing

clustering

algorithms

for

categorical

data

can

be

run

without

specifying

the

number

of

clusters.

Experimental

results

on

nine

University

of

California

at

Irvine

(UCI)

benchmark

and

ten

synthetic

data

sets

show

that

MGR

performs

better

as

compared

to

baseline

algorithms

in

terms

of

its

performance

and

efficiency

of

clustering.

Clustering

is

an

important

data

mining

technique

which

partitions

a

set

of

objects

into

clusters

such

that

objects

in

the

same

cluster

are

more

similar

to

each

other

than

objects

in

different

clusters

Most

previous

clustering

algorithms

focus

on

numerical

data

whose

inherent

geometric

properties

can

be

exploited

naturally

to

define

distance

functions

between

objects.

However,

many

fields,

from

statistics

to

psychology,

deal

with

categorical

data.

Unlike

numerical

data,

it

cannot

be

naturally

ordered.

An

example

of

categorical

attribute

is

color

whose

values

include

red,

green,

blue,

etc.

Therefore,

those

clustering

algorithms

dealing

with

numerical

data

cannot

be

used

to

cluster

categorical

data.

Recently,

the

problem

of

clustering

categorical

data

has

received

much

attention..

A

number

of

algorithms

have

been

proposed

for

clustering

categorical

data

Similar

to

other

clustering

problems,

categorical

data

clustering

can

also

be

considered

as

an

optimization

problem

thus

a

typical

method

for

clustering

categorical

data

is

to

define

a

dissimilarity

measure

between

objects,

an

objective

function,

and

then

iteratively

minimize

or

maximize

the

objective

function

until

a

solution

is

found.

Unfortunately,

this

optimization

problem

is

NP-complete.

Therefore

most

researchers

resort

to

heuristic

methods

to

solve

it.

ROCK

kmodes

and

k-ANMI

are

representative

examples

of

such

type

of

methods.

These

methods

require

the

user

to

specify

the

number

of

clusters

first

and

then

conduct

the

processes

of

initialization,

iteration,

and

so

on.

They

focus

on

the

relationship

between

the

objects

and

clusters

during

the

process

of

clustering,

as

a

result,

their

time

complexity

increases

greatly

with

the

increase

in

the

number

of

objects.

We

can

say

these

methods

have

implemented

clustering

from

the

viewpoint

of

objects.

As

we

know,

a

data

set

consists

of

two

elements:

objects

and

attributes.

Besides

objects,

attributes

are

also

an

important

aspect

to

be

considered

for

clustering.

Generally,

the

number

of

attributes

is

much

less

than

the

number

of

objects

in

a

data

set,

thus

it

is

possible

to

improve

the

clustering

efficiency

if

we

employ

attributes

for

clustering.

The

following

example

reveals

the

potential

of

attributes

for

categorical

data

clustering.

Table

shows

a

categorical

data

set

with

ten

objects

and

five

attributes.

The

column

of

real

classes

implies

that

the

set

of

objects

can

be

partitioned

into

three

classes.

We

assume

that

the

objects

in

each

class

are

the

same

while

completely

distinct

from

the

objects

Elsevier

B.V.

All

rights

reserved.

in

other

classes.

A

i

B

i

and

C

i

for

i

denote

different

categories

on

ith

attribute.

The

user

is

required

to

cluster

the

data

set

without

knowing

the

real

classes

in

advance..

Using

the

methods

mentioned

above,

the

user

has

to

specify

the

number

of

clusters

first.

Imagine

the

number

of

clusters

is

set

to

two,

the

accuracy

of

clustering

will

be

affected.

In

fact,

from

the

viewpoint

of

attributes,

it

can

be

seen

that

each

attribute

partitions

the

data

set

in

the

same

way.

If

we

can

find

such

relation

between

the

attributes,

a

perfect

clustering

of

the

data

set

including

three

clusters

will

be

obtained

by

using

the

partition

defined

by

any

attribute

without

specifying

the

number

of

clusters

in

advance.

Obviously,

using

attributes

to

cluster

the

data

set

in

this

example

is

a

more

natural

way..

In

a

real

life

categorical

data

set,

the

partitions

defined

by

attributes

are

not

as

perfect

as

that

in

the

above

example

(i.e.

the

partitions

defined

by

attributes

are

not

always

the

same);

however,

if

the

real

classes

are

sufficiently

distinguishable

from

each

other,

the

objects

in

the

same

real

classes

will

create

distinct

values

on

some

attributes

from

the

objects

in

the

other

real

classes,

consequently,

there

exist

some

partitions

defined

by

attributes

which

are

similar

to

the

real

clustering

of

objects;

at

least,

there

exist

some

equivalence

classes

(the

set

of

objects

which

has

the

same

value

of

the

attribute)

in

these

partitions

which

are

similar

to

the

real

classes.

Our

goal

is

to

find

such

partitions

and

equivalence

classes

to

construct

the

clustering

of

the

objects..

In

this

paper,

a

novel

information

theory

based

hierarchical

divisive

clustering

algorithm

for

categorical

data,

namely

MGR,

is

proposed.

MGR

iteratively

performs

two

steps

on

the

current

data

set:

selecting

a

clustering

attribute

and

an

equivalence

class

on

the

clustering

attribute.

Information

theory

based

concepts

of

mean

gain

ratio

and

entropy

of

clusters

are

used

to

implement

these

two

steps

respectively.

Experimental

results

on

nine

UCI

real

life

and

ten

synthetic

data

sets

show

that

our

algorithm

has

lower

computational

complexity

and

comparable

clustering

results..

The

rest

of

the

paper

is

organized

as

follows.

Section

briefly

reviews

the

related

work.

Section

describes

our

algorithm

MGR,

with

an

illustrative

example.

Section

analyzes

the

limitations

of

MMR

algorithm,

the

most

similar

work

to

our

method,

and

compares

it

with

MGR.

Section

presents

experimental

results,

with

a

comparison

with

other

algorithms.

Finally,

Section

presents

conclusions

and

recommendations

for

future

work.

This

paper

has

proposed

a

new

information

theory

based

hierarchical

divisive

clustering

algorithm,

namely

the

MGR

for

categorical

data,

which

implements

clustering

from

the

viewpoint

of

attributes.

Information

theory

based

concepts

of

MGR

and

entropy

of

clusters

are

introduced

in

the

algorithm

to

select

clustering

attribute

and

divide

the

objects

on

the

clustering

attribute..

Experimental

results

on

nine

UCI

real

life

data

sets

show

that

MGR

has

better

clustering

results

and

stability.

It

can

be

applied

to

the

data

sets

which

have

balanced

class

distribution,

such

as

Votes

and

Breast

Cancer,

as

well

as

those

which

have

unbalanced

class

distribution

like

the

Zoo

data

set.

Experimental

results

on

ten

synthetic

data

sets

show

that

MGR

has

better

clustering

efficiency

and

scalability.

It

can

be

applied

to

small

and

large

categorical

data

sets.

Another

advantage

of

MGR

is

that

it

can

be

run

without

specifying

the

number

of

clusters.

This

is

a

more

natural

way

especially

when

the

user

experiences

difficulties

in

identifying

the

number

of

clusters..

For

future

work,

first,

we

are

planning

to

combine

the

advantages

of

MGR

and

G-ANMI

algorithms

to

improve

the

clustering

accuracy

of

MGR,

at

the

same

time

keep

the

running

time

acceptable.

Second,

we

are

trying

to

introduce

a

reprocess

procedure

like

that

in

the

COOLCAT

algorithm

for

further

improvement

of

the

clustering

accuracy

of

MGR.

Ralambondrainy

proposes

a

method

to

convert

multiple

categories

attributes

into

binary

attributes

using

and

to

represent

either

a

category

absence

or

presence,

and

to

treat

the

binary

attributes

as

numeric

in

the

k-means

algorithm.

ROCK

algorithm

is

an

adaptation

of

agglomerative

hierarchical

clustering

algorithm

in

which

the

notion

of

''links''

is

defined

to

measure

the

closeness

between

clusters.

STIRR

is

an

iterative

algorithm,

which

maps

categorical

data

to

non-linear

dynamic

systems.

If

the

dynamic

system

converges,

the

categorical

data

can

be

clustered.

Based

on

a

novel

formalization

of

a

cluster

for

categorical

data,

a

fast

summarization

based

algorithm,

CACTUS,

is

presented

in

CACTUS

finds

clusters

in

subsets

of

all

attributes

and

thus

performs

a

subspace

clustering

of

the

data..

The

k-modes

algorithm

extends

the

k-means

paradigm

to

categorical

domain

by

using

a

simple

matching

dissimilarity

measure

for

categorical

objects,

i.e.,

modes

instead

of

means

for

clusters,

and

a

frequency-based

method

to

update

modes.

Subsequently,

based

on

k-modes,

many

algorithms

are

proposed

including

adapted

mixture

model

fuzzy

k-modes

tabu

search

technique

iterative

initial

points

refinement

algorithm

for

k-modes

clustering

an

extension

of

k-modes

algorithm

to

transactional

data

fuzzy

centroids

initialization

methods

for

k-modes

and

fuzzy

k-modes

a

dissimilarity

measure

for

k-modes

attribute

value

weighting

in

k-modes

clustering

and

genetic

fuzzy

k-modes

k-ANMI

is

also

a

k-means

like

clustering

algorithm

for

categorical

data

that

optimizes

the

mutual

information

sharing

based

objective

function..

Besides

k-means,

classical

information

theory

is

another

widely

used

technique

in

categorical

data

clustering.

COOLCAT

explores

the

connection

between

clustering

and

entropy:

clusters

of

similar

points

have

lower

entropy

than

those

of

dissimilar

ones.

LIMBO

is

a

hierarchical

algorithm

that

builds

on

the

Information

Bottleneck

(IB)

framework

to

detect

the

clustering

structure

in

a

data

set.

''Best

K''

proposes

a

BkPlot

method

to

determine

the

best

K

number

of

clusters

for

a

categorical

data

set..

He

et

al.

formally

define

the

categorical

data

clustering

problem

as

an

optimization

problem

from

the

viewpoint

of

cluster

ensemble,

and

apply

cluster

ensemble

approach

for

clustering

categorical

data.

Simultaneously,

Gionis

et

al.

use

disagreement

measure

based

cluster

ensemble

method

to

solve

the

problem

of

categorical

data

clustering..

Recently,

several

works

try

to

solve

the

problem

of

categorical

data

clustering

by

direct

optimization.

In

algorithms

ALG-RAND

G-ANMI

and

the

iterative

Monte-Carlo

procedure

in

some

concepts

of

information

theory,

such

as

generalized

conditional

entropy,

mutual

information

are

used

to

define

the

objective

function

and

some

optimization

methods

like

Genetics

are

used

to

solve

the

problem.

While

these

algorithms

improve

clustering

accuracy

on

some

data

sets,

as

pointed

out

in

considerable

obstacles

still

remain

before

they

can

be

widely

used

in

practice.

One

main

obstacle

is

the

efficiency

of

the

optimization

algorithms

like

Genetics..

In

addition,

He

et

al.

propose

TCSOM

algorithm

for

clustering

binary

data

by

extending

traditional

self-organizing

map

(SOM).

The

same

authors

also

propose

Squeezer

algorithm

Squeezer

is

a

threshold

based

one-pass

algorithm

which

is

also

suitable

for

clustering

categorical

data

streams.

Chen

and

Chuang

investigate

the

correlation

between

attribute

values

and

develop

CORE

algorithm

by

employing

the

concept

of

correlated-force

ensemble.

Abdu

and

Salane

proposed

a

spectral-based

clustering

algorithm

for

categorical

data

using

data

summaries.

There

also

exist

some

algorithms

focusing

on

transaction

data

clustering.

Wang

et

al.

propose

the

notion

of

large

item

and

develop

an

allocation

and

refinement

strategy

based

algorithm

for

clustering

transactions.

Following

the

large

item

method,

another

measurement,

called

the

small-large

ratio

is

proposed

and

utilized

to

perform

the

clustering

of

market

basket

data

Yun

et

al.

consider

the

item

taxonomy

in

performing

cluster

analysis.

Xu

and

Sung

explore

the

purchase

features

of

customers

and

propose

an

algorithm

based

on

''caucus''

which

is

known

as

the

fine-partitioned

demographic

group.

O O B I O O O B O O B I B I B O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O B O O O O O O O O O O O B I O O O B I O O O B O O B I O O B O O O O O O O O B O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O B B I O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O B O B I B O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O B I B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O B O O B I O O O O O O O O O O O O O O O O B O O O O O O O B O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O B I O O B O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O B I O O O O O O B I B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I B O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O B I B O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Can we build software faster and better and cheaper? ABSTRACT "Faster, Better, Cheaper" (FBC) was a development philosophy adopted by the NASA administration in the mid to late 1990s. that lead to some some dramatic successes such as Mars Pathfinder as well as a number highly publicized mission failures, such as the Mars Climate Orbiter & Polar Lander.The general consensus on FBC was "Faster, Better, Cheaper? Pick any two". According to that view, is impossibly to optimize on all three criteria without compromising the third. This paper checks that view using an AI search tool called STAR. We show that FBC is indeed feasible and produces similar or better results when compared to other methods However, for FBC to work, there must be a balanced concern and concentration on the quality aspects of a project. If not, "FBC" becomes "CF" (cheaper and faster) with the inevitable lose in project quality. Previously, PROMISE researchers have used induction (summary of data into a model) to generate predictor models for software engineering. While useful, induction has several drawbacks:. • It only explores half the story. As Murray Cantor observed in his PROMISE 2008 keynote, once predictive models are generated, they are used within some business context. While we need more papers on predictive model generation, it may be time to ask the PROMISE community to write more papers on predictive model usage.. .. • Automatic induction assumes the existence of data. Many domains are data starved where it is hard to obtain local data.. • More generally, we ask the question: why do we persist on generating new predictive models all the time? If there is any generality in software engineering it should be possible to reuse predictive models. We should at least experiment with this approach rather than always assuming the best predictive model is a new model.. Hence, we explore predictive model use and reuse rather than predictive model generation. Model reuse is complicated by extreme data starvation: the tuning variance problem arises if there is insufficient data to generate precise tunings. Our STAR tool [8, 23, 27] uses AI methods to conduct large scale "what-if" queries over software predictive models. If parts of a predictive models are not known with certainty, STAR seeks stable conclusions within the space of possible values.. In previous publications [24] , we had explored the benefits to using STAR for the purpose of presenting a concise strategy to software project managers. This paper applies STAR to assess the infamous "Faster, Better, Cheaper" (FBC) development practices used at NASA in the 1990s. FBC was advocated in the 1990s by the then-administrator of NASA, Daniel Goldin, as a method for reducing the expenditure of NASA. FBC was in-line with the direction that the Clinton administration's approach of doing more for less. FBC was initially successful: projects that usually cost over a billion were implemented at 1 4 th of that cost (e.g. Mars Pathfinder). However, subsequent failures (Mars Climate Orbiter and Polar Lander; the Columbia Shuttle disaster) lead to much criticism of FBC.. STAR is a natural tool for assessing FBC, since it uses CO-COMO based software engineering models that have already been usesd within NASA. These COCOMO models represent FBC in the following manner:. • Faster is represented by the Months model, which estimates the total development months needed for a software project.. • Better is represented by the Defects model, which estimates the number of delivered defects per KLOC (thousand lines of code).. • Cheaper is represented by the Effort model, which estimates the effort that is needed for a software project in personmonths, and hence can be used to estimate the cost of the development of the project.. Note that for all the models, lower is better. The tool combines estimates with utility weights {f, b, c} (short for Faster, Better, Permission to make digital or hard copies of part or all of this work or personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. © ACM 2009 ISBN: 978-1-60558-634-2...$10.00. Cheaper) using the following formula:. This score value represents the Euclidean distance to the normalized values of the predictions of development effort "E"; total development months "M "; and "D", the number of delivered defects per KLOC. This is the utility function that is used in order to assess any given set of "policies" that might be presented to be implemented in a given software project. Given that we normalize the predictions min..max to 0..1 then Equation 1 has the range one to zero and lower scores are better. STAR searches for the minimal set of project changes that most reduces this score. By adjusting the various values of (f, b, c), we can compare the effects of methodologies that emphasize different project goals:. • BF = "better, faster" i.e. c = 0 and b = f = 1;. • BC = "better, cheaper" i.e. f = 0 and b = c = 1;. • CF = "cheaper, faster" i.e. b = 0 and f = c = 1;. • FBC = "faster, better, cheaper" i.e. b = f = c = 1.. This paper reports the results of such a comparison. After a discussion of related work, we will review the sad history of "Better, Faster, Cheaper" at NASA. We will then describe the case studies will be explore, as well as the STAR systems. In the subsequent results section we will offer the surprising observation that, contrary to the prevailing wisdom at NASA, FBC is not necessarily a disastrous development methodology. In fact, in 7 9 of our studies, FBC produces results that are within 5% of the minimum estimates generated by any of BF,BC,CF.. Our results suggest that (a) FBC is in fact a viable development methodology but (b) what really went wrong with FBC at NASA was that it was used as a front for CF; i.e. built it faster and cheaper, while almost disregarding quality considerations.. Our more general conclusion is that we can make more use and reuse of predictive models. If stable conclusions exist across the tuning space, then AI tools like STAR can offer conclusions from predictive models, even when their is insufficient data for tuning. This is an powerful aspect of AI tools like STAR since we often lack the data required to precisely tune an imported model to a local domains. It is hoped that that this paper will inspire more papers at future PROMISE conferences that explore not just model generation, but also the reuse of those generated model in new domains. To facilitate that discussion, we have made all our code openly available 1 and we are building a web interface to an executable version of STAR 2 . That interface will be on-line by the time of the PROMISE'09 conference. Figure 1 places STAR in the context of related work. Predictive models can be build automatically or manually. Automatically generated predictive models can be automatically validated using many methods including (e.g.) the cross-validation procedures used in much data mining research [37] . Manually generated predictive models, built using human expertise, can be remarkably effective: see Fenton's PROMISE'07 paper reported a validation study of one such predictive model [9] build predictive models manually; tune with local data; e.g. [9] reuse predictive models from other sites; tune with local data; E.g. Our results show that FBC can be achieved with minimal compromises on individual criteria. How can we reconcile this result with NASA very negative experience with FBC?. One thing forgotten about FBC is that, usually, it worked. Despite all the criticism against it, FBC successful/partially successful in 136 of the total of 146 missions launched during the period that Goldin was administrator. This would be called an overall success if it hadn't been for the largely publicized failures. That is, FBC was mostly a technical success, but a PR failure [35] .. However, one way to explain the very large failures within the FBC program is to speculate that, sometimes, FBC was a front for CF (i.e.. cheaper and faster, as the expense of quality). Figure 7 shows the disastrous effects of CF:. • In the case of OSP CF's defects where 62 9 = 689% worse than the worst policy seen using any of BC, BF, or BFC.. • In the case of OSP2 CF's defects where 97 11 = 881% worse than the worst policy seen using any of BC, BF, or BFC.. Some of the decisions made under the banner of FBC are questionable; i.e. staff reductions leading to loss in veteran engineers and managers to retirement and causing experienced managerial staff to be stretched too thin given tight scheduling [38] . This forced projects to use inexperienced managers which caused management mix ups and human error.. Like Spear [33] , we would endorse FBC, but under the condition that it is better managed. Tony Spear, a JPL veteran engineer from 1962 to 1998, testified to the possible effectiveness of FBC. Despite mentioning problems with FBC (a fixation on cost, causing cost cuts that were too much for 2 nd generation FBC projects), he recommended not to discard it. Rather, he argued for a more focused way of implementing it, concentrating on aspects such as building and retaining talent, taking advantage of advancements in technology such as the Internet, and advancing methods used in project development and verification [32, 33] .. To Spear's recommendation we would add that when applying FBC, never surrender the quest of "better". Observe how, in Figure 7 , whenever we optimize for "Better" using b = 1, we always reduce defects by about an order of magnitude over the baseline "do nothing" result. The only time that our optimizer does not reduce defects is when the "better" utility is set to zero (see the CF detect results). Hence, we recommend always setting b = 1. Boehm's local calibration of CO-COMO models [2] . data rich (much data) build predictive models automatically e.g. [5, 14, 19, 21, 26] . • Metrics-guru Norman Fenton study data collection [10] for many years. Recently, he has despaired of that approach. At a keynote address PROMISE 2007 3 , Fenton shocked his audience by saying:. "....much of the current software metrics research is inherently irrelevant to the industrial mix ... any software metrics program that depends on some extensive metrics collection is doomed to failure.". • The COCOMO experience is similar to that of Fenton.. After 26 years of trying, the COCOMO team has collected less than 200 sample projects for the COCOMO database. Also, even after two years of effort we were only able to add 7 records to a NASA-wide software cost metrics repository [19] . As shown top right of Figure 1 , STAR is appropriate in domains that lack both domain experts and data. Many domains are data starved (see Figure 2 ) in which case there is neither sufficient data to automatically learn predictive models .. In the completely opposite circumstance, a domain is data rich. In such domains, automatic methods can learn predictive models. There are many examples of work in this area including [5, 14, 19, 21, 26] . We recommend automatic methods for data rich domains since manual modeling can be very expensive (for example, Fenton manually built his Bayes nets over a two year period [9] ).. In the middle case of Figure 1 , small amounts of local data can be used to tune either:. • Local predictive models either built manually; or. • Predictive models imported from other sites.. For example, Boehm et at. [4] advocate a certain functional form for generating software development effort estimates. In that form, the development effort is linear on a set of effort multipliers EMi and exponential on a set of scale factors SFj: A problem that has been under-explored in the literature is tuning variance. In data starved domains, there is insufficient data to produce precise tunings. For example, At PROMISE 2005, we have reported very large tuning variance in the post-tuning values of α and β [25] . Baker [1] offers a similar finding. After thirty 90% random samples of that data, the A, B ranges found during tuning were surprisingly wide:. We are not the only research group to be concerns about tuning 4 We will use uppercase B to denote the COCOMO linear tuning variable of Equation 2 and lower b to denote the business utility associated with defect predictions of Equation 1. variance. At PROMISE 2007, Korte & Port [17] explore the variance of automatically learned effort predictors. They comment that this variance is large enough to confuse standard methods for assessing different predictive model generators.. Since 2005 [5, 20] , we have been trying to reduce tuning variance. using feature subset selection (FSS). However, despite years of work, we now report that FSS reduces but does not tame the variance of A, B, α, β.. Having failed to tame tuning variance, we have been exploring a new approach. The STAR tool [8, 23, 27] . that we describe below checks for stable conclusions within the space of possible tunings.

O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Rewriting

nested

XML

queries

using

nested

views

We

present

and

analyze

an

algorithm

for

equivalent

rewriting

of

XQuery

queries

using

XQuery

views,

which

is

complete

for

a

large

class

of

XQueries

featuring

nested

FLWR

blocks,

XML

construction

and

join

equalities

by

value

and

identity.

These

features

pose

significant

challenges

which

lead

to

fundamental

extension

of

prior

work

on

the

problems

of

rewriting

conjunctive

and

tree

pattern

queries.

Our

solution

exploits

the

Nested

XML

Tableaux

(NEXT)

notation

which

enables

a

logical

foundation

for

specifying

XQuery

semantics.

We

present

a

tool

which

inputs

XQuery

queries

and

views

and

outputs

an

XQuery

rewriting,

thus

being

usable

on

top

of

any

of

the

existing

XQuery

processing

engines.

Our

experimental

evaluation

shows

that

the

tool

scales

well

for

large

numbers

of

views

and

complex

queries.

The

ability

to

equivalently

rewrite

queries

using

views

is

required

by

multiple

data

management

tasks.

For

example,

a

query

processor

can

speed

up

the

processing

of

a

query

when

part

of

the

computation

needed

by

the

query

has

already

been

performed

by

the

materialized

views

or

cached

queries.

Another

application

comes

from

the

field

of

privacy-preserving

data

publishing,

in

which

a

data

source

answers

only

those

client

queries

which

can

be

rewritten

using

exclusively

views

the

data

owner

agrees

to

publish

Finally,

in

data

integration

views

have

been

used

to

describe

the

source

content

(in

Local-As-View

architectures)

and

the

source

capabilities.

We

point

the

reader

to

the

survey

for

a

comprehensive

list

of

applications

of

rewriting

queries

using

views..

The

XML

data

model

emerges

as

equally

important

to

the

relational

model

for

many

of

the

data

management

problems

that

require

answering

queries

using

views.

At

the

same

time,

solving

the

problem

in

the

context

of

XML

and

XQuery

presents

a

set

of

novel

challenges.

First,

data

and

queries

are

nested.

Second,

XQuery

has

list

semantics,

which

degenerates

to

bag

semantics

if

the

unordered

keyword

is

used,

and

to

set

semantics

in

the

presence

of

the

duplicate-eliminating

primitive

distinct-values

(as

in

Example

below).

Any

rewriting

algorithm

must

be

equipped

to

uni-formly

test

the

equivalence

of

the

produced

rewriting

to

the

original

query

under

any

of

these

semantics.

Contrast

this

with

prior

work

on

XPath

(and

relational)

rewriting,

which

use

only

set

semantics.

Third,

XML

elements

may

be

compared

for

equality

on

either

their

value

or

their

identity;

no

such

distinction

comes

up

in

conjunctive

queries

and

XPath.

Finally,

XQuery

copies

XML

subtrees

in

the

result

construction

phase,

precluding

the

use

of

XML

node

ids

for

assembling

the

view

data

into

the

query

result..

We

provide

an

algorithm

which,

given

an

XQuery

Q

and

a

set

of

XQuery

views

V

Vn},

discovers

a

rewriting

query

RW

of

Q

using

V.

We

follow

the

classic

definition

of

rewriting

query

RW

is

evaluated

on

the

views

and

for

all

possible

database

instances

D

its

result

RW

Vn(D))

coincides

with

the

result

Q(D)

of

Q.

The

algorithm

is

sound

for

full

XQuery

queries

and

views,

and

is

complete

for

an

expressive

subset

of

XQuery

called

OptXQuery

which

includes

nesting,

optional

use

of

duplicate

elimination

(using

the

distinct-values

keyword),

and

allows

in

the

where

clause

conjunctions

of

idbased

and

value-based

equality

conditions,

as

well

as

existential

quantification

(some

clauses).

EXAMPLE

Consider

the

sample

data

of

Figure

and

the

following

query

Q

that

groups

paper

reviews

by

the

papers'

authors;

it

is

a

minor

variation

of

query

from

XMP

use

case

The

distinct-values

function

eliminates

duplicates,

comparing

elements

by

value-based

equality

The

for

loop

binding

$a

(called

the

$a

loop)

has

set

semantics

since

the

output

list

of

distinct-values

has

no

duplicates

and

the

order

of

its

elements

is

non-deterministic.

The

inner

for

loop

has

bag

semantics

i.e.,

duplicates

are

not

removed

but

the

order

is

non-deterministic

since

the

loop

is

in

the

context

of

the

unordered

$a

loop.

let

$doc

document('DBLP.xml')

for

$p

in

$doc//paper,

$r

in

$p/review

return

feedback

$r,

(V

authors

$p/author

/authors

/feedback.

Since

the

view

V

returns

all

author

and

review

information

which

is

pertinent

to

the

query

Q,

there

is

a

rewriting

RW

of

Q

using

V

let

$doc

document(V

for

in

distinct-values($doc/feedback[review]/authors/author)

return

evaluation

$a,

(RW

for

$f

in

$doc/feedback

$r

in

$f/review

where

some

in

$f/authors/author

satisfies

eq

$a

return

$r

/evaluation.

Our

query

processor

discovers

rewritings

that

use

the

views

in

order

to

obtain

the

variable

bindings

generated

by

the

query..

To

the

best

of

our

knowledge

this

is

the

first

work

on

rewriting

using

views

for

XQuery,

which

also

provides

formal

guarantees

of

completeness

for

a

large

subset

of

XQuery..

Prior

work

on

rewriting

using

views

for

XML

queries

(reviewed

in

Section

focused

on

the

XPath

language

The

resulting

algorithms

involve

detecting

query

subexpressions

subsumed

by

the

view.

The

subsumption

test

is

enabled

by

a

pattern-based

representation

of

XPath

expressions

called

tree

patterns

which

reduce

the

test

to

matching

the

view

pattern

against

the

query

pattern.

The

standard

specification

of

XQuery

semantics

does

not

support

the

extension

of

XPath

rewriting

techniques

as

it

provides

no

pattern-based

query

representation.

We

therefore

adopted

NEsted

XML

Tableaux

(NEXT)

a

pattern-based

notation,

which

consolidates

navigation

in

the

minimum

number

of

tree

patters

and

hence

maximizes

rewriting

opportunities.

NEXT

can

represent

a

large

subset

of

XQuery,

called

OptXQuery

and

introduced

in

OptXQuery

is

a

natural

boundary

within

which

the

rewriting

algorithm

is

guaranteed

to

find

rewritings

whenever

they

exist.

The

OptXQuery

subset

includes

the

XPath

language,

and

it

fully

subsumes

the

c-XQuery

subset

of

conjunctive,

nested

XQueries

whose

containment

is

studied

in

Our

contributions

are:.

An

algorithm

for

rewriting

queries

from

the

OptXQuery

subset

of

XQuery

using

exclusively

views

from

the

same

sublanguage.

The

algorithm

equivalently

rewrites

the

variable

binding

stage

of

the

query,

which

is

where

the

large

costs

of

navigations,

joins

and

selections

of

data

intensive

applications

are

incurred.

If

the

document

order

of

the

query

result

is

immaterial

(due

to

the

use

of

distinct-values

and

unordered

keywords),

then

the

algorithm

is

complete,

i.e.,

it

always

finds

an

equivalent

rewriting,

if

such

exists

(Theorem

We

analyze

the

complexity

of

checking

the

existence

of

a

rewriting,

showing

NP-completeness

in

the

width

of

the

query's

NEXT

pattern

representation.

This

measure

(defined

in

Section

is

typically

much

smaller

than

the

query

size,

depending

only

on

the

number

of

variables

shared

across

nested

query

blocks

and

the

number

of

variables

involved

in

equality

conditions.

For

instance,

the

width

of

query

Q

in

Example

is

Our

rewriting

algorithm

is

worst-case

exponential

in

the

query

width,

which

is

optimal

behavior:

it

runs

in

PTIME

when

the

query

is

acyclic,

(for

which

it

turns

out

that

the

width

is

its

performance

degrading

gracefully

with

increasing

query

width.

The

acyclic

case

includes

XPath

tree

patterns..

We

introduce

a

technique

for

extending

the

rewriting

algorithm

from

OptXQuery

to

arbitrary

XQuery,

by

identifying

and

rewriting

the

OptXQuery

subexpressions.

The

technique

is

sound,

in

the

sense

that

it

creates

only

equivalent

rewritings.

It

is

not

complete,

but

this

is

an

unavoidable

consequence

of

the

undecidability

of

checking

rewriting

existence

for

(even

slightly)

more

expressive

queries

and

views

than

OptXQuery.

Indeed,

even

relational

rewriting

algorithms

perform

a

best-effort

approach

and

are

incomplete

for

full

SQL.

Our

approach

combines

the

benefits

of

complete

rewriting

feasible

for

NEXT/OptXQuery

with

the

easy-to-engineer

but

incomplete

techniques

based

on

isomorphically

matching

common

sub-expressions

between

query

and

view

We

provide

a

tool

which

inputs

XQuery

queries

and

views

and

outputs

an

XQuery

rewriting,

thus

being

usable

on

top

of

any

of

the

existing

XQuery

processing

engines.

A

demo

of

the

tool

is

available

at

We

report

on

our

experimental

evaluation,

which

measured

rewriting

times

of

around

second

for

as

many

as

views,

for

queries

of

up

to

nesting

levels,

variables

per

level..

The

remainder

of

this

paper

is

organized

as

follows.

Section

describes

the

system

architecture,

OptXQuery

and

its

corresponding

NEXT

notation.

Section

presents

the

rewriting

algorithm,

its

implementation,

and

analyzes

its

complexity.

The

extension

of

the

rewriting

algorithm

to

arbitrary

XQueries

is

given

in

Section

We

report

on

the

experimental

evaluation

in

Section

Related

work

is

discussed

in

Section

In

this

paper,

we

present

a

sound

algorithm

for

rewriting

using

views

for

XQuery.

The

algorithm

is

complete

for

the

large

sublanguage

OptXQuery,

when

unordered

rewritings

are

of

interest.

We

obtain

good

performance

by

exploiting

the

tree-like

structure

of

OptXQuery

navigation

patterns..

Our

rewriting

algorithm

was

enabled

by

the

integration

of

a

proven

relational

approach

(use

a

pattern-based

representation

of

queries

and

views

to

detect

subsumed

query

sub-expressions

via

pattern

mappings)

with

solutions

for

the

special

XQuery

challenges

such

as

nested

return

clauses,

two

types

of

equalities,

unrestricted

mix

of

loops

with

set,

bag

and

list

semantics,

and

the

copy

semantics

of

XML

construction.

of

these

challenges

arose

in

prior

XPath

rewriting

work..

A

note

about

the

copy

semantics:

our

example

shows

that

despite

the

fact

that

XQuery

loses

node

identities

when

it

constructs

the

result,

there

are

interesting

rewriting

opportunities,

which

our

algorithm

discovers.

Given

(XQuery-inexpressible)

views

which

preserve

node

identities,

even

more

such

opportunities

would

be

exposed.

A

simple

extension

of

our

algorithm

would

easily

exploit

these

opportunities:

during

query

expansion

(Phase

connect

query

and

view

variables

with

id-equality

edges

(as

opposed

to

only

value-equality

in

the

current

form).

We

can

show

that,

with

this

modification,

the

previous

soundness

and

completeness

guarantees

apply

to

id-preserving

views..

Note

that

our

algorithm

yields

a

single

rewriting,

obtained

by

finding

all

(possibly

redundant)

alternate

data

accesses

through

views.

This

is

already

sufficient

in

certain

scenarios,

such

as

privacy-preserving

publishing

(where

one

only

cares

about

the

existence

of

a

rewriting),

or

in

distributed

processing

in

which

evaluating

a

redundant

query

over

the

local

cache

is

likely

preferable

to

sending

even

non-redundant

queries

to

remote

sites.

We

are

working

on

integrating

the

rewriting

algorithm

with

the

NEXT

query

minimization

techniques

of

and

the

work

on

cost-based

pruning

during

minimization

described

in

the

MARS

system

Finally,

we

are

working

on

extending

the

rewriting

in

the

presence

of

XML

Schema

and

DTD

constraints,

which

create

even

more

rewriting

opportunities.

Our

NEXT-based

rewriting

approach

is

compatible

with

this

extension,

exploiting

pattern

mappings

to

find

the

query

sub-expressions

affected

by

constraints.

In

XQuery

stream

processing,

identifies

common

XQuery

subexpressions

and

memoizes

in

a

cache

to

avoid

redundant

evaluation.

This

can

be

seen

as

rewriting

the

original

XQuery

using

the

views

in

the

cache.

The

test

for

pattern

equivalence

is

based

on

expression

isomorphism

and

thus

trades

rewriting

opportunities

for

engineering

simplicity.

As

we

discussed,

one

can

maximize

rewriting

opportunities

within

the

NEXT/OptXQuery

set,

while

retaining

syntactic

isomorphism

for

NEXT+/XQuery.

NEXT

rewriting

allows

us

to

match

query

and

view

navigation

across

XPath

sub-expressions,

regardless

of

whether

it

appears

in

the

for

or

the

where

clause

of

a

query,

or

as

distinct-values

argument.

of

these

matches

are

supported

by

pure

syntactic

isomorphism.

rewrite

only

the

XPath

subexpressions

of

XQueries

using

materialized

XPath

indexes

and

thus

do

not

face

the

problems

caused

by

nesting,

equalities,

XML

construction

in

the

view

output,

all

of

which

we

address

in

this

paper..

The

type

of

nesting

we

address

here

corresponds

in

the

SQL

relational

case

to

the

illegal

nesting

in

the

SELECT

clause,

and

was

therefore

not

a

focus

of

SQL

optimization.

While

this

nesting

is

allowed

in

OQL,

the

development

of

complete

algorithms

for

rewriting

OQL

queries

using

views

is

precluded

by

the

fact

that

checking

equivalence

of

nested

OQL

queries

is

an

open

problem

even

for

the

idealized

conjunctive

OQL

sublanguage

of

We

are

aware

of

only

one

particular

case

of

view-based

OQL

rewriting

(which

however

does

not

involve

nesting),

namely

rewriting

OQL

paths

using

path

indexes

Note

that

other

kinds

of

XQuery

or

OQL

nesting

(within

the

FOR

and

WHERE

clauses)

are

easier

to

deal

with,

as

it

is

in

most

cases

translated

away

using

normalization

rules

such

as

in

Agora

(adopted

in

our

work

as

well),

and

the

normalization

from

for

moving

nested

some

loops

from

the

where

clause

into

the

for

clause

(see

NEXTQ)..

rewrites

semistructured

queries

using

semistructured

views

in

the

context

of

the

semistructured

OEM

data

model.

The

copy

semantics

of

XQuery's

construction

operators

rule

out

the

use

of

database

id's

for

assembling

the

view

data

into

the

query

result,

as

done

in

In

the

context

of

data

integration,

rewrites

XQueries

under

open-world,

"certain

answer"

semantics

using

source-to-target

constraints.

The

work

does

not

address

equivalent

rewritings

in

a

closed

world,

which

is

our

setting.

The

two

settings

requires

very

different

algorithms

even

in

the

relational

case..

In

XML

publishing,

the

Agora

and

MARS

systems

both

rewrite

XQueries

using

publishing

views,

but

indirectly

via

a

reduction

to

rewriting

relational

queries

using

relational

views

(Agora)

or

constraints

(MARS).

Consequently,

they

do

not

address

list

and

bag

semantics.

The

strength

of

the

two

approaches

is

to

allow

mixing

of

relational

and

XML

models

by

reducing

the

XQuery

treatment

to

a

relational

one.

A

benefit

of

our

NEXT-based

approach

is

that,

by

exploiting

the

underlying

tree

pattern

structure,

we

achieve

faster

algorithms

for

pattern

matching.

The

relational

reduction

misses

this

opportunity.

Moreover,

Agora

and

MARS

first

decorrelate

queries

into

unnested

queries.

Each

unnested

query

is

then

individually

rewritten

and

evaluated

using

relational

techniques

and

the

results

are

put

together

using

an

outer

join.

Hence

the

solution

forces

the

processor

to

use

decorrelation

and

outer

joins

at

the

physical

level.

In

contrast,

we

provide

a

tool

which

inputs

XQuery

and

outputs

an

XQuery

rewriting,

thus

imposing

no

requirements

on

the

underlying

engine,

with

obvious

benefits

of

portability

and

of

allowing

the

cost-based

optimizer

to

subsequently

choose

a

physical

level

plan..

The

equivalence

checker

is

a

basic

building

block

for

any

rewriting

algorithm.

There

is

a

significant

body

of

work

on

equivalence

of

XPath

(tree

patterns)

The

only

work

we

are

aware

of

on

equivalence

of

nested

XML

queries

was

reported

in

and

adopted

in

this

paper.

While

not

complete

for

checking

equivalence

of

entire

queries,

the

algorithm

of

is

complete

for

checking

equivalence

of

binding

stages,

which

is

ultimately

what

we

want

to

rewrite.

completely

solves

checking

containment

for

a

class

of

nested

XML

queries

subsumed

by

ours.

However,

it

addresses

a

containment

flavor

based

on

homomorphic

embedding

of

the

resulting

XML

trees.

Unfortunately,

equivalence

is

not

reducible

to

this

kind

of

containment

(two

queries

may

be

contained

in

each

other

without

being

equivalent).

Moreover,

the

test

for

this

containment

flavor

has

inherently

higher

complexity

(Π

p

-complete

if

applied

to

NEXT

patterns)

than

for

the

binding

stage

containment

we

use

(NP-complete

in

the

pattern

width).

One

may

wonder

how

we

check

binding

stage

equivalence

under

bag

semantics,

given

that

containment

under

bag

semantics

is

open

for

conjunctive

queries

(but

Π

p

-hard)

and

undecidable

for

unions

thereof

Fortunately,

in

XML

nodes

have

unique

identities,

and

for

a

NEXT

query,

a

bag

of

element

values

corresponds

to

a

set

of

their

identities,

which

allows

us

to

reduce

the

test

to

set

containment

of

tuples

consisting

of

values

and

identities.

Finally,

we

do

not

inherit

the

open

problem

of

checking

equivalence

for

OQL

As

shown

in

equivalence

and

containment

are

not

inter-reducible,

except

for

OQL

queries

yielding

VERSO

relations

These

are

essentially

obtained

as

the

result

of

grouping,

and

therefore

are

produced

by

the

bind

stage

of

NEXT

queries.

B O O O O O B O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Relating average and discounted costs for quantitative analysis of timed systems Quantitative analysis and controller synthesis problems for reactive real-time systems can be formalized as optimization problems on timed automata, timed games, and their probabilistic extensions. The limiting average cost and the discounted cost are two standard criteria for such optimization problems. In theory of finite-state probabilistic systems, a number of interesting results are available relating the optimal values according to these two different performance objectives. These results, however, do not directly apply to timed systems due to the infinite state-space of clock valuations. In this paper, we present some conditions under which the existence of the limit of optimal discounted cost objective implies the the existence of limiting average cost to the same value. Using these results we answer an open question posed by Fahrenberg and Larsen, and give simpler proofs of some known decidability results on (probabilistic) timed automata. We also show the determinacy and decidability of average-time games on timed automata, and expected average-time games on probabilistic timed automata. A number of recent case-studies [8, 15, 14, 4] demonstrate the applicability of timed automata [1] for model based design and analysis of embedded real-time systems. Probabilistic extensions of timed automata, such as probabilistic timed automata [21] , and duration probabilistic automata [23] are useful in modeling uncertainties (component failure, probability distribution on delays etc.), Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. EMSOFT '11, October 9-14, 2011 , Taipei, Taiwan. Copyright 2011 ACM 978-1-4503-0714-7/11/10 ...$10.00.. while two-player games [3, 7] on timed automata are natural candidates for modeling open real-time systems. The success of timed automata as a model of real-time systems can be partly attributed to its theoretical properties, such as decidable model checking problems, and partly to the availability of an excellent set of tools such as UPPAAL [29] , Kronos [20] , RED [27] , and, PRISM [25] .. Performance analysis and optimal controller synthesis questions for real-time embedded systems can be reduced to optimization problems on priced extensions of (probabilistic) timed automata for a given performance metric [4, 2] . The limiting average cost and the discounted cost are two standard criteria for associating a performance metric with an (infinite) execution of a reactive system. For a given execution r let π(r) = p i ∈ R ∞ i=0 be the sequence of numbers such that p i is the price at step i. Then, the limiting average cost of the execution r is the limit, if exists, of finite length averages:. Avg(π(r)) def = lim. On the other hand, the discounted cost of the execution r for a discount factor λ ∈ (0, 1] is the total discounted sum of prices, where the price at step i is discounted by (1 − λ) i :. The notions of limiting average cost and discounted cost are, loosely speaking, quite opposite to each other: while the limiting average cost criterion is insensitive to prices appearing in the initial prefixes, the discounted cost criterion gives more weight to prices appearing early in a sequence than the later ones. Hence it is perhaps surprising at first to learn that these two criteria are related to each-other. Hardy and Littlewood [11] showed that for a bounded (or positive) sequence π, if the limit lim λ→0 + Disct λ (π) exists then the average cost Avg(π) exists as well, and it is equal to lim λ→0 + Disct λ (π). In this paper we study optimization problems on timed automata (and its generalizations) with discounted cost and average cost criteria, and show the relation between their optimal values. In addition to their purely theoretical interest, such relations yield new proofs of decidability of certain average cost games on timed automata.. Connections between optimal discounted cost and average cost criteria are well-studied for probabilistic systems. Mertens and Neyman [24] showed that for two-player zero-sum stochastic games (and hence for Markov decision processes (MDPs)) with finitely many states and actions, the value of the game with average cost objective is equal to the limit of values of games with discounted cost objectives. Lehrer and Sorin [22] showed a more general result in the context of MDPs with infinite state spaces. They proved that uniform convergence of the limit of optimal discounted cost objective implies uniform convergence of the average reward objective to the same value. The concept of Blackwell-optimal strategy is tightly coupled with the relation between the limit of optimal discounted cost and the optimal average cost. Blackwell [5] showed the existence of strategies in finite Markov decision processes that are optimal for all discount factors sufficiently close to 0. It is well known [11, 26] that every positional Blackwell-optimal strategy is also average optimal, while the opposite is not true. The result of Mertens and Neyman show the existence of Blackwell-optimal strategies on finite stochastic games. To the best of our knowledge no analog of such results is known for timed automata, and the results from finite state probabilistic systems do not automatically generalize to timed systems due to infinite state space of clock valuations.. The main contribution of this paper are the theorems relating the limit of optimal discounted cost and the optimal average cost criteria on timed automata. We also show the existence of ε-Blackwell optimal (Blackwell optimal up to a given precision ε > 0) strategies in priced timed automata. We generalize these results for probabilistic timed automata, and games on probabilistic timed automata. Our results yield a simpler proof of decidability and determinacy of optimal average-time games on timed automata, and the first proof of decidability and determinacy for expected averagetime games on probabilistic timed automata. We also prove a conjecture of Fahrenberg and Larsen [9] on the connection between continuously-discounted cost and average cost (per time-unit) optimization problem on timed automata.. The rest of the paper is organized in the following manner. We begin the technical part of the paper by introducing the concepts and the notations in a general setting of MDPs and stochastic games with infinite states and actions. In Section 3 we introduce priced timed automata and present our main result. In Section 4 and 5 we discuss the extensions of our results for probabilistic timed automata and two player games, respectively. Competitive optimization problems on timed automata are central to model based performance evaluation and optimal controller design for embedded systems. Discounted cost and average cost criteria are two standard performance metrics for reactive systems. We studied the limits of λ-discounted cost and N -average cost performance criteria for timed automata and its generalizations. We showed the equivalence of these limits to the optimal average cost criterion for priced timed automata, priced probabilistic timed automata, and probabilistic timed games. Our results show a new proof of determinacy and decidability of expected average-time games on probabilistic timed automata, while simplifying an already known result for average-time games on timed automata.

O O O O O O B I O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I


Frank Miller: Inventor of the One-Time Pad The invention of the one-time pad is generally credited to Gilbert S. Vernam and Joseph O. Mauborgne. We show that it was invented about 35 years earlier by a Sacramento banker named Frank Miller. We identify which Frank Miller it was, and speculate on what might have led him to his idea. We also discuss whether or not Mauborgne might have known of Miller's work, especially via his colleague Parker Hitt.Mauborgne. It thus remains unclear who should be credited with effectively inventing the one-time pad.We first give an overview of the state of cryptography at the time. We continue by describing and analyzing Miller's codebook. We then explain our tentative identification of the author, and speculate on possible linkages to Mauborgne. One-time pads are in theory the strongest possible algorithmic cipher: if the key is used properly, they cannot be broken, even in theory. The invention of the one-time pad is generally credited to Gilbert S. Vernam of Bell Telephone Laboratories and Joseph O. Mauborgne of the U.S. Army Signal Corps [38, 72] . Vernam invented a device that would exclusive-OR keystream bits from a paper tape with the Baudot code generated by letters typed on a keyboard; he and Mauborgne realized that if the keystream tape characters were (a) perfectly random, and (b) never reused, ''the messages are rendered entirely secret, and are impossible to analyze without the key'' [72] . (To be sure, others have made similar assertions about their ciphers; Vernam and Mauborgne's claim has the advantages of being both correct and mathematically provable.) Shortly afterwards, three Germans, Werner Kunze, Rudolf Schauffler, and Erich Langlotz, adapted the principle to superencipherment of codebooks [38] .. In fact, they were anticipated by about 35 years. In 1882, a California banker named Frank Miller published Telegraphic Code to Insure Privacy and Secrecy in the Transmission of Telegrams. In it, he describes the first one-time pad system, as a superencipherment mechanism for his telegraph code [51] . 1 If used properly, it would have had the same property of absolute security.. Although in theory Miller can claim priority, reality is more complex. As will be explained below, it is quite unlikely that either he or anyone else ever used his system for real messages; in fact, it is unclear if anyone other than he and his friends and family ever knew of its existence. That said, there are some possible links to For instances, instead of confining the pre-agreements of the parties, to the deduction or addition of one number only, let one number be agreed on, for every alternate word sent in figures, and another number for every other word; or, by the same rule, let several numbers be used for addition or deduction, in any order that may be agreed on.. In addition, he suggested the use of a monoalphabetic substitution of the code letter.. Few late-19th century codebooks were much better at superencryption. The 1899 U.S. War Department code [32] , though intended primarily for administrative use, did provide for superencipherment; it was less sophisticated than Smith's. It suggested adding or subtracting a ''key number or series of key numbers,'' and gave as examples the sequence ' '25, 50, 75, 100 .'' It did, however, note that ''the use of 50 or 100, while easy to remember, should be avoided.'' Wells, Fargo did use good cryptography internally. Their codebooks contained homophones [22] ; they also used a new codebook every year [29] . In addition, they employed a double transposition as a superencipherment. While protecting commercial messages was important, a more common use was in reporting robberies and plans for pursuing robbers. Their cipher book was first used in 1874 [22] , a year after 204 S. M. Bellovin James B. Hume joined the company as chief detective [28] . We suspect (but cannot prove) that Hume instigated the deployment; the code was used primarily when dealing with robberies [22] , and half of his cases involved inside jobs [28, p. 132 ]. Hume would have wanted a way to protect his communications against rogue telegraph operators and clerks. In [36, p. 59 ], Hume and his colleague John Thacker do describe one robbery perpetrated by a telegraph operator. They do not mention any use of the cipher book. (The threat from corrupt employees goes a long way towards explaining the strong cryptography: few bandit gangs have strong SIGINT capabilities. . . .) Interestingly, on p. 83 the book talks about a robber learning that the code phrase ''Red Coat'' meant that a large payroll shipment was being sent to a particular mine; he then sent a ''ciphered message''-prearranged words [77] -to his confederates who would actually rob the train. One well-known, public codebook stood out. Slater's Telegraphic Code, to Ensure Secresy in the Transmission of Telegrams [sic] [61] talked about simple additives, but he also described a variety of more complex transforms. For example, he suggested transposing digits in code numbers, or regrouping the 5-digit code numbers into 4-digit ones. In addition, he suggested using combinations of these methods. Despite its limitations, Slater's codebook became the basis of a War Department code [19, 75] . (Curiously, Friedman's discussion of Slater's superencipherment focuses only on the use of additives, and misses the more complex transformations [19, p. 96].) None of Slater's schemes were that sophisticated. Friedman conjectures that European cryptanalysts were able to read American State Department correspondence with little trouble; this is certainly supported by Kahn. It is clear that Miller invented the one-time pad: res ipsa loquitur. His work was remarkable, not just because he invented a secure encryption scheme 35 years before it was reinvented, but also because it came out of the blue. Vernam and Mauborgne were working in the field; Vernam was trying to build a secure communications system and Mauborgne was the head of research and engineering for the U.S. Army Signal Corps. They were the sort of people one would expect to invent such a thing. Miller, on the other hand, was a banker, whose only first-hand experience with cryptology (if any) was likely 16 years earlier. Still, he was not the first amateur to make a significant contribution to the field. Indeed, Kahn has a chapter entitled ''The Contribution of the Dilettantes''. The most remarkable tale in that chapter is that of Thomas Jefferson, who invented a wheel cipher in the 1790s, an invention that was promptly forgotten and not rediscovered in his papers until 1922. Ironically enough, Jefferson's cipher was reinvented a few years before that by French cryptologist É tienne Bazeries; it was brought to the U.S. and refined by none other than Parker Hitt and Joseph Mauborgne. 8 One can also speculate on the risks of tying a bad system-the conventional codebook Miller authored-to a good one. Had Miller published-and marketed-his superencipherment independently from his codebook, it might have succeeded. Robert Slater's codebook [61] , intended only for confidentiality, was even worse than Miller's for compression (though it had 25,000 words, compared with Miller's 12,800); despite that, the work went through nine editions, the last published in 1938 [63] , and was even the basis for a U.S. government codebook [19, 75] . The difference was likely marketing-Slater was secretary of the French Atlantic Telegraph Company-but regardless, Slater's work survived and was influential; Miller's work has been forgotten.. It is tempting to speculate on how history may have been different had people paid attention. Almost certainly, one-time pads would never have been employed 8 One should not draw the conclusion that Hitt was a habitual plagiarist who stole from both Miller and Bazeries. Kahn quotes Hitt's memo as explicitly crediting Bazeries for the wheel cipher. Hitt was in fact punctilious about assigning proper credit [66] . for most military communications; the operational difficulties, then as now, would have been prohibitive. But what of high-level diplomatic correspondence? Kahn tells many tales of cryptanalysis of such messages between 1882 and 1918; might things have been different? Might the German Foreign Ministry been able to protect Zimmermann's fateful telegram with a one-time pad if they had known of Miller's work? And if so, would the U.S. have entered World War I when it did? It is worth noting that Germany did adopt a one-time pad superencipherment for its diplomatic codes right after World War I [38] . 9 Finally, it takes nothing away from Vernam or Mauborgne's own creativity to acknowledge that Miller had the same idea. If nothing else, their system-an online, automated mechanism that operated on the 0 s and 1 s of Baudot code-was far more usable and was the precursor to today's computerized stream ciphers.

O O O O O B I O O O O B I O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Capacitance matrix technique for avoiding spurious eigenmodes in the solution of hydrodynamic stability problems by Chebyshev collocation method We present a simple technique for avoiding physically spurious eigenmodes that often occur in the solution of hydrodynamic stability problems by the Chebyshev collocation method. The method is demonstrated on the solution of the OrrSommerfeld equation for plane Poiseuille flow. Following the standard approach, the original fourth order differential equation is factorised into two second-order equations using a vorticity-type auxiliary variable with unknown boundary values which are then eliminated by a capacitance matrix approach. However the elimination is constrained by the conservation of the structure of matrix eigenvalue problem, it can be done in two basically different ways. A straightforward application of the method results in a couple of physically spurious eigenvalues which are either huge or close to zero depending on the way the vorticity boundary conditions are eliminated. The zero eigenvalues can be shifted to any prescribed value and thus removed by a slight modification of the second approach. Spectral methods are known to achieve exponential convergence rate [3] , which makes them particularly useful for solving numerically demanding differential eigenvalue problems which arise in hydrodynamic stability analysis [12] . Unfortunately, besides providing accurate and efficient solutions for a certain number of leading eigenvalues, spectral methods often produce physically spurious unstable modes, which cannot be removed by increasing the numerical resolution [8] . For detailed discussion of these modes we refer to Boyd [2] . Such physically spurious eigenvalues can appear in all types of spectral methods including Galerkin [16] , tau [4] and collocation approximations [3] , unless some kind of ad hoc approach is applied to avoid them. In the Galerkin method, spurious eigenvalues can be removed by using the basis functions also as the test functions instead of separate Chebyshev polynomials [17] . A number of approaches avoiding spurious eigenvalues have also been found for the tau method [7, 11, 10] . The same can be achieved also for the collocation (or pseudospectral) method by using two distinct interpolating polynomials [9] . Following the approach of McFadden et al. [11] for the tau method, Huang and Sloan [9] use a Lagrange interpolating polynomial for second-order terms which is by two orders lower than the Hermite interpolant used for other terms. The choice of the latter polynomial depends on the particular combination of the boundary conditions for the problem to be solved [15, p. 493] .. The objective of this paper is to present a simple method avoiding spurious eigenmodes in the Chebyshev collocations method which uses only the Lagrange interpolating polynomial applicable to general boundary conditions. Our approach is based on the capacitance matrix technique which is used to eliminate fictitious boundary conditions for a vorticity-type auxiliary variable. The elimination can be performed in two basically different ways which respectively produce a pair of infinite and zero spurious eigenvalues. The latter can be shifted to any prescribed value by a simple modification of the second approach. The main advantage of our method is not only its simplicity but also applicability to more general problems with complicated boundary conditions. The paper is organised as follows. In the next section we introduce the OrrSommerfeld problem for plane Poiseuille flow, which is a standard test case for this type of method. Section 3 presents the basics of the Chebyshev collocation method that we use. The elimination of the vorticity boundary conditions, which constitutes the basis of our method, is performed in Sec. 4. Section 5 contains numerical results for the Orr-Sommerfeld problem of plane Poiseuille flow. The paper is concluded by a summary of results in Sec. 6. We have developed a simple technique for avoiding physically spurious eigenmodes in the solution of hydrodynamic stability problems by the Chebyshev collocation method, which was demonstrated on the Orr-Sommerfeld equation for plane Poiseuille flow. The method is based on the factorisation of the original fourth order differential equation into two second-order equations using a vorticity-type auxiliary variable which has no explicit boundary conditions. The main element of the method is the elimination of the vorticity boundary values by using a capacitance matrix approach to obtain a standard matrix eigenvalue problem. Although the elimination is constrained by of the structure of eigenvalue problem, it can be still done in two basically different ways. Both approaches result in couple of physically spurious eigenvalues, which are either huge or close to zero depending on the way the vorticity boundary values are eliminated. We showed that these spurious eigenvalues are due to the double singularity of the transformation matrices which eliminate the vorticity boundary conditions by multiplying either the stiffness or mass matrices of the original generalised eigenvalue problem. By a slight modification of the second approach, the zero eigenvalues can be shifted to any prescribed value and thus moved to the region of numerically spurious eigenvalues at the end of spectrum.. The main advantage of our method is not only its simplicity but also its applicability to more general stability problems with complex boundary conditions involving several variables. An example of such a problem is that of 3D linear stability of MHD duct flow using a non-standard vector stream function and vorticity formulation, which results in the coupling of the stream function components through the boundary conditions [14] . In this case neither Galerkin nor collocation method with the ad hoc approach of Huang and Sloan [9] is applicable because no simple basis functions satisfying the boundary conditions can be constructed. . where c i = 1 for 0 < i < N and c i = 2 for i = 0, N.

O O O O O O O O O O O B I O O B I I O O O O O O O O O O O O O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O