Indicators

for

a

webometric

ranking

of

open

access

repositories

Abstract

The

Ranking

Web

of

World

Repositories

is

introduced.

The

objective

is

to

promote

Open

access

initiatives

(OAI)

supporting

the

use

of

repositories

for

scientific

evaluation

purposes.

A

set

of

metrics

based

on

web

presence,

impact

and

usage

is

discussed.

The

Ranking

is

built

on

indicators

obtained

from

web

search

engines

following

a

model

close

to

the

Impact

Factor

one.

The

activity

accounts

for

a

of

the

index,

including

number

of

pages,

pdf

files

and

items

in

Google

Scholar

database,

while

the

visibility

takes

into

account

the

external

inlinks

received

by

the

repository

(the

other

The

Ranking

provides

the

Top

repositories

from

a

total

of

worldwide,

with

a

strong

presence

of

US,

German

and

British

institutional

repositories

and

the

leadership

of

the

large

subject

repositories.

Results

suggest

the

need

to

take

into

consideration

other

file

formats

and

the

usage

information,

an

option

is

not

feasible

today.

Among

the

new

aims

of

the

OAI

is

not

only

to

motivate

institutions

to

develop

scientific

papers

repositories

but

also

to

provide

statistical

information

about

these

contents

(Harnad

Armbruster

Ideally

three

groups

of

indicators

should

be

deployed:

activity

related

(documents

deposited

by

author,

institution,

subject),

usage

related

(visits,

visitors,

downloads)

and

visibility

related

(citations

and

web

links).

Today

there

is

no

universal

source

for

many

of

these

indicators,

with

some

interesting

initiatives

on

the

way

(http:/

/projectcounter.org/;

The

usage

data

is

an

important

indicator

for

the

future,

representing

an

innovation

as

its

bibliometric

counterpart

(journal

circulation)

has

not

been

frequently

used.

The

diversity

of

statistics

available

is

very

large

(visits,

visitors,

downloads,

referrers,

referrals…)

but

there

is

no

open

reports

available

for

most

of

the

repositories

and

even

when

the

data

is

published

the

lack

of

standards

avoid

the

comparative

analysis..

The

Ranking

Web

has

been

able

to

combine

web

data

for

producing

a

list

of

institutional

and

thematic

repositories.

The

advantages

of

these

preliminary

efforts

are

evident

as

organizations

are

being

classified

according

to

their

commitment

to

the

OAI.

Current

success

of

discipline

focused

repositories

is

not

an

indicator

of

future

developments

as

more

and

more

institutions

are

developing

deposit

mandates

mostly

to

populate

local

repositories.

This

movement

is

especially

interesting

as

deriving

indicators

for

evaluation

purposes

for

institutions

could

be

easier

to

monitor

and

to

compare

with

other

bibliometric

sources

and

previous

analysis

enhancing

the

value

of

the

web

data..

Even

if

the

Ranking

Web

is

providing

a

solid

basis

for

such

analysis,

more

indicators,

especially

those

related

to

usage

and

citation,

are

badly

needed:.

After

the

demise

of

Live

Academic

the

web

bibliographic

databases

available

for

global

analyses

are

few

and

the

only

one

with

citation

data

is

Google

Scholar.

Scholar

could

play

an

important

role

Thelwall

Meho

and

Yang

but

a

non

beta

version

is

long

awaited

and

tags

based

searching

is

mandatory

for

obtaining

clean

and

detailed

results.

Surprisingly

the

webmasters

of

many

repositories

are

not

paying

enough

attention

to

facilitate

the

citation

of

individual

papers,

as

the

URLs

are

very

long,

uninformative

and

without

adequate

file

format

suffix.

If

this

situation

is

not

resolved,

then

the

number

of

citations

does

not

increase

adequately

for

performing

further

statistical

analysis..

Regarding

usage,

the

current

generation

of

free

tools

represented

by

Google

Analytics

have

an

important

added

value

as

they

can

represent

a

universal

standard,

but

unfortunately

it

is

not

offering

yet

academic

oriented

capabilities.

The

usage

information

provided

by

individual

repositories

is

very

scant,

not

standardised,

and

easy

to

manipulate..

From

a

practical

point

of

view,

the

Ranking

is

targeting

institutional

mandates

in

order

not

only

to

reinforce

its

use

but

suggesting

a

statistics

rich

environment.

It

is

also

important

that

new

repositories

take

into

consideration

the

indexing

and

positioning

in

commercial

search

engines

like

Google,

Yahoo

or

Bing

(Live):

Suggestions

here

include

the

use

of

independent

domains

or

subdomains,

avoid

the

barriers

to

robots

and

adding

semantic

value

to

metadada..

The

results

show

that

more

research

should

be

done

regarding

the

rich

file

formats,

as

the

use

of

solely

pdf

files

is

not

providing

a

complete

picture.

Some

personal

communication

propose

not

take

them

into

account

but

this

is

against

the

objective

of

measuring

the

contents.

Another

alternative

that

will

be

tested

in

January

edition

will

be

to

combine

all

the

main

filetypes.

Future

editions

will

consider

bibliographic

citations

and

usage

data

when

these

indicators

become

available.

The

July

Ranking

consists

of

a

Directory

of

repositories,

from

which

the

Top

are

ranked.

Table

shows

the

distribution

by

country

countries

plus

one

international

category

are

represented),

with

special

attention

to

the

leaders..

The

three

more

important

countries,

according

to

the

number

of

repositories

ranked

in

the

top

positions,

are

the

USA

of

the

Top

are

US

repositories),

Germany

and

UK

with

other

European

countries,

Japan

and

Canada

also

well

represented.

Table

shows

the

main

indicators

of

the

twenty

top

ranked

repositories

(numbers

are

ordinals,

with

the

lowest

values

representing

the

top

positions).

Most

of

the

first

ones

are

thematic,

that

usually

are

older

(Arxiv

since

larger

and

prestigious

at

least

in

some

disciplines.

However

the

institutional

ones

are

far

more

numerous

and

probably

due

to

the

copyright

restrictions

(institutions

are

the

holders

of

the

rights

of

their

authors)

will

be

those

to

persist.

Some

comments

on

the

results

shown:.

-CiteSeer

is

now

changing

to

a

new

interface

CiteSeerX

that

although

still

in

beta

is

already

ranked

worldwide

-HAL,

the

French

Open

archives

system

maintain

not

only

a

central

repository

(archives-overtes.fr)

but

also

institutional

ones

like

the

INRIA

subset

that

in

fact

receives

more

links

than

the

parent

one

vs.

position

in

visibility).

-The

Smithsonian/NASA

Astrophysics

Data

System

does

not

appear

in

the

Top

positions

because

they

have

two

subdomains

for

the

same

repository

harvard.edu

and

that

is

a

bad

naming

practice

and

penalise

its

web

visibility..

In

general

the

consistency

among

the

web

indicators

are

low,

as

leaders

according

to

one

indicator

are

not

so

strong

in

the

other

ones.

A

Spearman's

q

test

(Table

shows

that

the

correlation

between

size,

pdf

files

and

Scholar

items

is

significant

(although

it

is

very

low)..

There

are

several

possible

explanations

for

this

behaviour:

-Many

institutions

have

several

independent

repositories,

one

for

dissertations,

another

for

papers

and

even

other

one

for

reports.

In

the

extreme

Caltech

have

different

entries

in

the

Directory.

-Invisibility,

the

inability

of

search

engines

crawlers

for

collect

data

due

to

barriers

in

the

design

of

the

web

databases,

is

probably

a

serious

problem.

Since

mid

nineties

Cybermetrics/Webometrics

is

helping

to

describe

in

a

quantitative

way

the

scholarly

communication

processes,

focusing

not

on

traditional

citation

databases

(WoK,

Scopus)

but

on

the

information

available

in

the

publicly

accessible

Web.

This

includes

all

the

contents

available

through

crawlers

or

search

engines

robots

(Visible

Web),

but

also

those

published

in

the

so

called

Invisible,

Deep

or

Hidden

Web

(Wouters

et

al.

usually

databases

or

other

resources

that

need

a

search

gateway

to

recover

them

(no

longer

invisible

as

Google

and

other

engines

really

crawl

and

index

their

contents).

This

includes

web

academic

databases,

scientific

repositories

and

electronic

journals

(Fig.

Cybermetrics

Lab

has

developed

in

the

last

years

a

series

of

indicators

for

measuring

Web

activity

and

visibility

of

academic

and

research

institutions

(Aguillo

et

al.

Using

search

engines

as

intermediaries,

it

is

possible

to

extract

quantitative

information

about

the

web

contents

published

by

these

institutions,

including

both

formal

papers

but

also

informal

material

related

to

other

missions

such

teaching,

divulgation

or

technological

transfer

(Barjak

Originally

aimed

to

promote

the

Open

access

initiatives

(OAI)

the

Ranking

Web

of

World

Universities

(Aguillo

et

al.

has

been

proved

as

a

useful

tool

also

for

institutions

evaluation.

Unfortunately

it

is

not

easy

to

split

the

components

involved

in

the

rank

of

a

university,

as

it

is

an

overall

measurement

combining

different

aspects

and

activities.

Due

to

this

but

using

a

similar

methodological

approach,

the

Cybermetrics

Lab

decided

to

focus

on

the

repositories

themselves

for

describing

and

comparing

the

success

of

OAI

in

the

academic

sector..

A

first

step

was

to

classify

the

resources

in

order

to

make

a

homogeneous

database

of

repositories.

Due

to

feasibility

reasons

all

the

personal

pages

were

excluded,

although

even

considering

the

large

numbers

involved,

an

analysis

of

files

in

formats

like

pdf,

doc

or

ps

could

be

done

for

closed

lists:

departments,

institutions,

selected

topics

(Barjak

et

al.

The

analysis

of

electronic

journals

is

progressing

through

more

traditional

bibliometric

and

citation

analysis,

so

individual

journals

and

portals

were

also

excluded..

Using

OpenDOAR

(www.opendoar.org)

and

ROAR

(roar.eprints.org)

directories

of

repositories,

among

other

sources,

a

candidate

list

of

repositories

was

assembled.

After

checking

for

thematic

and

institutional

repositories,

cleaning

those

whose

primary

contents

does

not

consist

of

scientific

papers,

and

excluding

the

mentioned

portals

of

journals

and

harvesters,

the

final

list

includes

only

those

with

an

autonomous

domain

or

subdomain..

For

each

entry,

the

following

statistics

were

recovered

from

search

engines:.

-Size,

measured

from

the

number

of

pages

from

Google,

Live

Search,

Yahoo

Search

and

Exalead.

Several

alternative

measures

were

discarded

such

as

the

size

in

number

of

bytes

because

it

is

strongly

linked

to

the

size

of

graphics

(a

single

large

true

colour

graph

could

have

a

weight

in

order

of

several

hundred

bytes,

more

than

dozens

of

only

text

pages).

-Pdf

files,

number

of

documents

in

pdf

format

from

Google

and

Yahoo.

There

are

other

formats

that

can

be

used

in

the

repositories,

mainly

Word

(doc

and

rtf)

and

html

and

similar

ones,

but

only

a

few

repositories

use

them

as

unique

or

duplicate

documents

and

in

fact

they

are

already

counted

in

the

previous

indicator.

-Scholar,

total

number

of

entries

in

Google

Scholar

(still

in

beta

version,

with

a

the

largest

coverage

of

all

the

remaining

public

web

academic

databases,

but

very

noisy),

and

-Visibility,

accounting

number

of

external

inlinks

extracted

from

Yahoo

and

Exalead

(Google

only

provides

link

numbers

per

page,

not

for

full

domains

or

subdomains).

PageRank

has

been

considered

as

a

measure

of

link

visibility

but

the

figures

offered

by

the

Google

Toolbar

are

integers

from

to

in

a

logarithmic

scale.

Most

of

the

values

are

or

and

it

is

very

difficult

to

climb

to

higher

values

that

make

this

indicator

unusable

for

ranking

purposes..

The

numbers

were

log-normalised

(adding

one

unit

to

avoid

max

and

min

values

were

excluded

(median,

mean

otherwise)

and

then

converted

into

ordinals:

first

(highest

value)

is

one,

second

is

two

and

so

on..

The

four

ranks

were

aggregated

to

obtain

a

unique

value

describing

the

overall

presence

and

impact

of

the

repository

in

the

web.

To

better

reflect

the

contribution

of

each

variable

a

weight

was

assigned

according

to

a

subjective

model.

These

values

are

''a

priori''

weights

because

they

are

decided

prior

to

collection

of

data

although

minor

adjustments

could

be

done

after

examination

of

results..

The

model

adopted

is

based

in

the

Impact

Factor

indicator,

that

proposed

a

ratio

between

the

activity

and

impact

Transferring

the

model

from

bibliometrics

to

webometrics

means

that

number

of

publications

is

now

number

of

web

pages,

while

the

number

of

citations

is

now

the

number

of

external

inlinks.

In

order

to

reinforce

the

weight

of

more

formal

academic

material,

both

the

documents

in

pdf

format

and

the

items

recovered

from

Google

Scholar

database

are

taking

into

account.

To

choose

only

the

pdf

files

instead

of

a

wider

representation

of

Rich

files

(doc,

rtf,

ppt,

ps

and

others)

is

for

avoiding

counting

documents

twice

if

they

are

available

in

two

different

formats.

As

the

number

of

total

files

already

count

these

other

types,

this

indicator

gives

a

closer

approximation

to

actual

number

of

deposited

documents,

excluding

descriptive

and

navigation

pages.

In

Table

the

three

components

of

activity:

size

pdf

files

and

Scholar

preserve

the

combined

weight

of

assigned

in

the

model.

The

distribution

of

weights

derived

from

an

informed

guess,

but

perhaps

it

should

be

reconsidered

for

future

editions..

A

first

beta

version

was

published

in

February

After

receiving

feedback

regarding

repositories

missed

in

the

first

draft

list

and

the

variables

involved,

a

second

beta

version

were

published

during

May

The

final

version,

corresponding

to

July

is

available

from

the

ranking's

webportal

and

it

is

the

source

for

the

current

analysis.

Further

editions

will

be

published

two

times

(January

and

July)

per

year..

The

main

change

is

the

extra

weight

given

to

recent

records

in

Scholar

database.

The

value

for

this

indicator

is

now

the

mean

between

normalised

total

number

of

items

and

those

published

in

the

period

(Fig.

B O O O B O B I B O O B O O O B O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O B O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Combining algebraic effects with continuations We consider the natural combinations of algebraic computational effects such as side-effects, exceptions, interactive input/output, and nondeterminism with continuations. Continuations are not an algebraic effect, but previously developed combinations of algebraic effects given by sum and tensor extend, with effort, to include commonly used combinations of the various algebraic effects with continuations. Continuations also give rise to a third sort of combination, that given by applying the continuations monad transformer to an algebraic effect. We investigate the extent to which sum and tensor extend from algebraic effects to arbitrary monads, and the extent to which Felleisen et al.'s C operator extends from continuations to its combination with algebraic effects. To do all this, we use Dubuc's characterisation of strong monads in terms of enriched large Lawvere theories. It is a very great pleasure to contribute to this Festschrift for John Reynolds. Continuations have been an abiding interest of John's, starting with his 1972 paper on definitional interpreters and with his most recent contribution being a historical piece on the origins of continuations [48] [49] [50] [51] [52] . Continuations have had several uses in programming languages and their theory. They have provided translations between languages, been used to give semantics for programming languages and have been employed in compilers. They also appear explicitly in programming languages where the programmer is given access to them as 'first-class entities' via control operators such as Rees and Clinger's call/cc [46] or Felleisen et al.'s C [10, 11] . Two such languages are the untyped language Scheme [1] and the typed language ML, particularly New Jersey Standard ML [8] .. In this paper, we concern ourselves with continuations in programming languages, and in particular, the relation between continuations and other computational effects such as exceptions, side-effects, input/output and nondeterminism. This work forms part of a research programme to develop a theory of computational effects, following on seminal work of Eugenio Moggi who proposed the use of strong monads T to uniformly model computational effects [33, 34] . He further introduced the computational λ-calculus as a basic functional call-by-value language for computational effects, modelled in the Kleisli category of T , and introduced monad transformers as a way of combining effects [6] .. One needs to have operations in the language to actually make the effects happen. The research programme focuses on these operations and their interaction as primary, via equational or Lawvere theories. In [40] it was shown how various well-known such monads arise as free algebra monads for natural theories of the operations; in [18] it was shown how monads for combinations of such effects corresponded to natural combinations of the underlying equational or Lawvere theories, viz. the sum or tensor of theories. This yielded a systematic account of monad transformers as well as a corresponding theory of operation transformers. Where possible, the work was carried out at a rather general level, that of enriched category theory [22] , for example considering Lawvere V -theories [43] and their combinations. One benefit of the more general approach is the ability to include domain-theoretic considerations, taking V to be, for example, ω-Cpo, the category of ω-cpos and continuous functions. A survey of the work done so far on this programme of an algebraic theory of computational effects appears in [42] .. The operations considered are special ones, termed algebraic operations [41] . Working in Set, these are transformations of the form op X : (T X ) I −→ (T X ) O subject to a certain naturality requirement. The monads that arise from equational theories of such operations are also special: they have a rank, meaning that for some cardinal κ, they preserve κ-filtered colimits, see [2, 23] . Indeed, in all natural examples, they have finite or countably infinite rank. In our previous work, the equivalence between V -monads of, say, countable rank and countable Lawvere V -theories was central to our analysis.. There are also natural operations that are not algebraic, with a notable example being exception handlers. Following a suggestion of Filinski, one may call the algebraic operations constructors in that they create effects, whereas operations such as exception handlers are deconstructors in so far as they analyse what effects have happened. It seems that even when deconstructors are present, the monads are given by the theories on the constructors, as is, for example, the case for exceptions. However, at present no general mathematical account of deconstructors is available.. Moggi could include continuations via the monad R R − . However this monad does not have a rank. Furthermore, none of the natural operations are algebraic; in fact they do not even have the required form. For example, consider the typed version C σ : ((σ → 0) → 0) → 0 of Felleisen et al.'s control operator, introduced by Griffin [15] . This is modelled by a transformation of the form C X : ¬ T ¬ T X −→ T X where ¬ T X = def T 0 X . As well as not having the right form to be an algebraic operation, C − inherently involves contravariance owing to the use of ¬ T . It may perhaps best be thought of as a logical operation rather than an algebraic one, as is indeed suggested by the various works linking continuations and classical logic within the Curry-Howard paradigm, e.g., [15, 36] . We choose to focus on C for reasons which we find persuasive, though possibly not definitive: most other control operators are definable from it; the corresponding rule is central in the natural deduction treatments of classical logic; and the operator is mathematically both natural and elegant. An operator not definable from C is prompt, but it fits within classical subtractive logic rather than classical implicational logic [3] .. In Section 2 we discuss the structure needed to support the combination of the computational λ-calculus, continuations and algebraic effects. It consists of a monad T , a continuations transformation C, and a map of monads S → T where S is the monad for the algebraic operations at hand. We particularly consider what axioms to impose on the continuations operator. As regards combinations of continuations with other effects, one can apply exception and state transformers, and there is also the well-known continuations transformer. We examine the three constructions in Sections 3-5 respectively. As we shall see, strong axioms on the continuations transformation are not preserved by the sum construction. Also, other than in the case of the continuations transformer, we unfortunately have no universal characterisation of the combinations in terms of axioms on exceptions, states, and continuations.. A question arises as to the mathematical status of these combinations, and of the corresponding extensions of the operations. One problem is that if one wishes to think in terms of Lawvere theories, then one needs a notion of large Lawvere theory, to correspond to monads without a rank. Fortunately, an enriched notion of such theories has previously been developed by Dubuc [9] . He showed that giving an arbitrary strong monad on a complete and cocomplete Cartesian closed category V is equivalent to giving what we call a large Lawvere V -theory. That equivalence is less complex than the size-restricted one: given a strong monad T , its large Lawvere V -theory L T is given by K l(T ) op , the opposite of the Kleisli V -category. But the price for that simplicity is that sums and, we conjecture, tensors do not exist in general. We recall and explain the result in Appendix B, and we feel free to use it as convenient: it is often simpler to work in terms of theories and, indeed, that is how we discovered several of our results.. We end the paper in Section 6 with a formula for a typical combination of effects, making clear the elegance and simplicity obtained by our analysis, followed by some discussion of what is missing in our account and what remains to be done.. For the sake of simplicity of exposition, we express our results without referring to enrichment, that is we take V = Set. However, all of our general results extend to the enriched case in a standard way [21] ; see [18] for an example. In particular, they extend to the important case where V = ω-Cpo, where the strong monads are exactly the locally continuous monads often considered in domain theory, i.e., those respecting lubs of ω-chains of morphisms. Using the constructs we have developed, there is a natural formula for the combination of all of exceptions, sideeffects, interactive input/output, (binary) nondeterminism and continuations:. or, explicitly. which we note is linear, having the form M E (M S (C(M I /O (F + )))) with each M derived from + or ⊗ applied to a particular monad. To omit effects, one can omit the corresponding parts of the formula. We have no independent justification of these proposals, but they are consistent with all the cases we know. A similar formula is available if we pass to ω-Cpo and add nontermination. We have given accounts of the combinations of continuations and other effects in terms of a unary or a binary operation on monads, together with the construction of a continuation operator for the combination. The main thing missing in our account is an understanding of why these are the right choices (if indeed they are!). For the combinations of other effects with each other, the choices were justified computationally in terms of the equations involving the sets of operations inherited from each effect [18] , but there is nothing of that sort here.. It may be that we are simply taking the wrong approach, as continuations are, as mentioned above, of a logical rather than an algebraic character. For example, the computational correlate of implication is the Kleisli exponential, and that is justified by its universal property relative to the monad. However, we did not succeed in finding such characterisations other than for the continuations monad transformer.. Closely related questions concern finding the right, possibly even complete, axiomatisation of the computational λ-calculus with continuations and the various effects, cf. [17, 54, 55, 27] . Further, investigations of continuations and effects would not be complete without a treatment of their operational semantics: see [39] for work on the operational semantics of effects in the context of λ c , but without continuations, and [54, 55] for operational semantics for combinations of continuations and specific effects. As remarked above, an interesting test of the strength of axioms for continuations is to establish the soundness of an operational semantics. One would also like to understand combinations of continuations with local effects, such as the local store or exceptions, as well as with other calling mechanisms.. There are various other computational calculi one might consider when investigating the combination of algebraic effects with continuations. For example, for call-by-value, there are the call-by-value λµ-calculus [35] and [30] , a fine-grained variant of λ c ; for call-by-name there is the call-by-name λµ-calculus [36] ; and for their combination there are the λµµ-calculus [7] and the call-by-push-value calculus [28] .. In addition to callcc/throw and raise/handle, New Jersey Standard ML provides two additional constructs, catch and escape, based on the idea of implementing exceptions via a stored handler [53] . Thielecke 1 and Filinski 2 have independently observed that the monad (R S ) (R S ) − , where S = def R E , can be used to model these constructs. This monad can be viewed either as the tensor product of the read-only state monad with the continuations monad, or else as the continuations monad transformer applied to the read-only state monad.. Although d T is then invertible, we cannot use its inverse to model the C operator, as that would validate an equation that Laird [27] shows to be broken in the presence of exceptions. Moreover, since the only left inverse of d T is its inverse, C cannot even be a left inverse of d T . This also follows from an observation of Filinski that the catch/escape constructs break C-APP.. Filinski also showed, as mentioned in [29] , that catch/escape breaks the equation

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O B O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


On the success of empirical studies in the international conference on software engineering Critiques of the quantity and quality of empirical evaluations in software engineering have existed for quite some time. However such critiques are typically not empirically evaluated. This paper fills this gap by empirically analyzing papers published by ICSE, the prime research conference on Software Engineering. We present quantitative and qualitative results of a quasi-random experiment of empirical evaluations over the lifetime of the conference. Our quantitative results show the quantity of empirical evaluation has increased over 29 ICSE proceedings but we still have room to improve the soundness of empirical evaluations in ICSE proceedings. Our qualitative results point to specific areas of improvement in empirical evaluations. Empirical evaluation has received much attention in software engineering research in general, and in high profile software engineering conferences, such as the International Conference on Software Engineering (ICSE). This trend is based on critiques of the quantity of empirical evaluations in software engineering, as well as the soundness of empirical evaluations performed since the early days of software engineering research. This paper addresses both these issues. We present an empirical evaluation of peer-reviewed ICSE papers over the lifetime of ICSE.. Despite the 29 years of ICSE proceedings, empirically evaluated critiques of empirical evaluations are not seen. Keynote talks (e.g. [3] ), panels (e.g. [24] ) and workshops (e.g. [18] ) support, instruct and debate empirical evaluations, but the results of such efforts are not yet known. This paper fills this gap by producing a firsttime look at the evolution and growth of empirical evaluations within ICSE.. External to ICSE, numerous recommendations regarding how empirical evaluations should be performed can be found [6] , [8] , [13] , and a few empirically evaluated critiques have also been performed [14] [27] . These empirically evaluated critiques suggest that the percentage of papers containing empirical evaluations in software engineering journals and conferences is smaller than it should be. General opinion strongly suggests the same idea [8] . We sample ICSE proceedings, as the prime research conference in software engineering, to direct improvement efforts in software engineering empirical evaluation.. Given the context of our study we proceed as follows. Section 2 presents background wok and Section 3 presents the hypotheses of our study. Section 4 presents our experimental design via a discussion of the materials and procedure. Section 5 presents our results with some analysis. Section 6 focuses on analysis as it pertains to our hypotheses. Section 6 also discusses the highlights and lowlights of our results. Section 7 addresses the validity of our study and Section 9 concludes this paper. We have provided empirically validated results of two research hypotheses regarding the current state of empirical evaluations in ICSE papers. We accept our first research hypothesis, that the quantity of empirical evaluations performed has increased over 29 years of ICSE proceedings. We reject our second research hypothesis and accept the null hypothesis, that the soundness of empirical evaluations has not improved over 29 years of ICSE proceedings. In terms of the first hypothesis, we report p 1 =0.571 and p 2 =0.8 showing the proportion of papers with an evaluation component is larger in more recent years than in earlier years of the proceedings of ICSE. We have a clear indication of an increase in the amount of empirical evaluations in ICSE. In terms of the second hypothesis, we see little consistent definition of the type of studies being performed and the lack of improvement in self-confirmatory studies strongly suggests that the soundness of empirical work has not improved. Additionally we see extremely little hypothesis specification. We are forced to reject our second research hypothesis and accept its null hypothesis. This paper presented a quasi-random experiment with quantitative and qualitative results to indicate the current weaknesses of empirical studies in ICSE. It was our intent to focus on issues in a specific community to focus improvement efforts in software engineering empirical evaluation. Provided some qualitative, textual, opinion-oriented evaluation. E.g. compare and contrast, oral discussion of advantages and disadvantages.. The Target and Used Population is a 'yes' or 'no' answer to the question, does the target population match the used population? In other words, is the sample on which the evaluation was performed representative of the addressed population? As software engineering is targeting to help industrial software development, we assumed the target population was industry, unless otherwise stated by the author.. The Evaluation Type asks the question, who developed the subject under study and who evaluated it? Evaluation Type can be either of self-confirmatory or independent evaluation. In a selfconfirmatory evaluation the authors play a large role in the production of the object of study (e.g. developed the tool or a method) and performed the evaluation. In an independent evaluation the authors evaluated a third party object.. In addition, we examined the papers for a 'yes' or 'no' answer to the question, are the hypotheses clearly stated? We also examined the papers for blatant illegal analysis on metrics. A prime example of this is calculating the mean on data from a Likert scale. Thirdly we determined if the evaluation contained any replication or was a replicated study. Lastly, we determined if the study reported primarily positive or negative results.. Our analysis using these measures was replicated internally. The first author evaluated all papers and the other authors validated this analysis by blindly evaluating 6 papers each, which were randomly assigned. This was an independent evaluation. In light of the quantitative results, we discuss our hypotheses, and lowlights and highlights of the results. We provide results for the entire sample, then results for each cluster to show the improvement or decline of these results over the lifetime of ICSE. The Study Type, Sampling Type, Target and Used Population, and Evaluation Type, were evaluated from two perspectives: first from what the authors stated in their respective papers, and second from our perspective. Our perspective judged the four parameters based on the definitions provided in Section 3. When the paper did not provide enough information to state or infer the definition of each of the four parameters, the parameter was labeled Undefined (UD). If there was no evaluation component, the paper was listed as Not-Applicable (N/A). For all figures used in the results, the legend is as follows. AS means "Author Selected" and represents the values for the parameter from the author perspective. IS means "Investigator Selected" and represents the values for the given parameter from our perspective. No meta reviews were identified in any of the papers sampled.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Quality and effectiveness in Web-based customer support systems The quality of a Web-based customer support system involves the information it supplies, the service it provides, and characteristics of the system itself; its effectiveness is reflected by the satisfaction of its users. This paper presents the results of a study of quality and effectiveness in Web-based customer support systems. Data from a survey of 726 Internet users were used to test theoretically expected relationships. The results of this study indicate that information and system quality determine effectiveness while service quality has no impact. Practical implications for managers and designers are offered. # This study tested a model concerning how various dimensions of quality affect Web-based customer support system effectiveness. Many firms have realized, as their marketplaces have become more global and service oriented, that customer support is critical to their competitiveness. A survey of 350 information systems (IS) executives revealed that connecting to customers and suppliers is one of their top 10 priorities. Among this same group, 60% of the respondents indicated that developing applications to support their customers was the most important focus for their system development efforts [68] . Another survey of 454 information technology (IT) managers indicated that 76% of the respondents were evaluated on their ability to use IT effectively to improve customer service [21] .. Bill Gates of Microsoft said that he believes that customer service is destined to become the primary value added function in every business. He predicted that Web-based customer support was likely to have significant advantages when compared to customer support based on traditional media [26] . A 1999 study by the Yankee Group found that close to 80% of large companies currently were providing (or planed to provide) Web-based customer support access [35] . Another study by the Yankee Group found that 77% of their respondents handled 25% of their total customer support requests via the Web [79] .. Today, Microsoft's Web-based customer support gets over 100,000 unique customer visits per day. By handling this volume of customer support online, Microsoft has been able to maintain a constant level of Information & Management 40 (2003) [757] [758] [759] [760] [761] [762] [763] [764] [765] [766] [767] [768] phone support during a period of sales growth. At Novell, Web-based customer support has reduced phone support by 45%. At Network Associates and Great Plains, Web-based customer support has reduced call volume by 37 and 20%, respectively [1] . Many firms are taking advantage of the Webbased technologies to give customers direct access to their customer support knowledge base [13] . With the option of self-service, customers can choose to access support knowledge directly through the Internet. Webbased customer support systems can be used for internal and external customer support [46, 75] .. The Web environment allows consumers to recover from their mistakes [43] and to overcome some difficulties associated with traditional media; e.g. it may ameliorate problems of accessibility, bottlenecks, interaction, and identification. It also promises potential benefits for firms, including reduced transaction costs, reduced time to complete transactions, reduced clerical errors, faster responses to new market opportunities, improved monitoring of customer choices, improved market intelligence, more timely dissemination of information to stakeholders, and more highly customized advertising and promotion [6] .. It is clear that Web-based customer support systems are important. What is not so clear is: what makes Web-based customer support systems effective? This paper identifies, and empirically establishes, the quality dimensions that underlie Web-based customer support system effectiveness.. Quality is a characteristic of a product or service that reflects how well it meets the needs of its consumers; as such, it is associated with product or service satisfaction [55] . Although consumers ultimately decide whether quality exists, organizations aspire to provide the features that their customers require [22] . The judgments that consumers make about product or service quality are based on what they feel they need. Organizations attempt to determine customer requirements and then to treat these results as guidelines for their products and services.. For many firms, improved quality has become an essential ingredient for successful competition [23] , while the standards by which their quality is judged have become global. In the IS realm, measurements of several dimensions of quality (information quality, system quality, and service quality) have been used to assess IS [80] .. Quality has been described as conformance to requirements, while satisfaction has been defined as conformance to expectations. Under ideal circumstances, there would be no difference between consumer judgment of quality and experienced satisfaction, but circumstances are often less than ideal. Users may have different requirements and meeting the requirements of all users is challenging. The situation is even more difficult in a Web-based customer support environment. The Web has a vast number of users, a non-homogenous group, who are not confined by organizational context and for whom use of the Web is optional. Firms that wish to build successful Web systems must attract users to visit and revisit their sites voluntarily. And potential users can be very different. Their needs are difficult to define, much less to meet. However, providing quality, meeting users' expectations, is critical for the success of Web-based customer support systems [11] .. Studying the impacts of customer support, not just through the Web, has been a major focus in the marketing literature [7, 9, [51] [52] [53] 82] . Customer support includes a technical aspect (i.e. were things done correctly?) and a relationship aspect (i.e. was the customer treated properly?) [8] . Customer support may be used to avoid adversity and build long-term relationships [54] . The technical component of customer support is addressed by both information and system quality, the relationship aspect by service quality. Our research employed a Web-based survey; this was thought to be an appropriate way to gather data from users of Web-based customer support systems. Web-based surveys have been used in prior research [73, 74] . Tan and Teo list several characteristics that make a Web-based survey different from mail surveys, including sampling frame, quality of data, response rate, and length of time needed. The target audience for our survey included students taking Web-related courses in business, management, IS, or computer science.. The fundamental question to be asked before measuring user satisfaction is: ''whose satisfaction should be measured and with what object?'' [69] . User satisfaction measures have been applied in many IS research settings: satisfaction of chief executives on the success of the overall MIS effort [10] ; user satisfaction on implementation success [27, 28] ; sales representative satisfaction with a new computer system [47] ; executive satisfaction with an IS which aided decisions [48] ; manager satisfaction on project success [64] ; manager satisfaction as imputation of IS value [39] ; and user satisfaction for the effectiveness of group decision support system [16] .. Because our objective was to examine how satisfied users of a Web-based customer support system were with the particular application they used, only people who had used a Web-based customer support system participated. Satisfaction evaluation, unlike service quality perception assessment, requires experience with the service.. Our Web-based survey consisted of items using a 5-point Likert-type scale having values ranging from (1) strongly disagree to (5) strongly agree. Two pilot studies were conducted with Web-based customer support users from Fortune 2000 companies. Based on feedback from these, the format and readability of the survey's questions were modified. Two professors validated the appropriateness of the instrument for the student population.

O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Ordering from Satan's menu: a survey of requirements specification for formal analysis of cryptographic protocols The application of formal methods to cryptographic protocol analysis has been a growth area recently. Most of the attention has been paid to the design of languages for the speciÿcation of cryptographic protocols and algorithms for evaluating their security. However, the ability to specify their desired behavior correctly is also important; indeed many perceived protocol aws arise out of a misunderstanding of the protocol's requirements. In this paper, we give a survey of research in requirements speciÿcation for formal analysis of cryptographic protocols. We start with a brief history of the use of requirements speciÿcation for cryptographic protocols. We then outline some of the main current trends and areas of research. We conclude with a discussion of some open problems. We have given a brief survey of research in expressing cryptographic protocol requirements. We believe that at this point we have a good handle on the speciÿcation of the standard secrecy and correspondence requirements of security protocols. It appears possible to derive techniques that are compatible with just about any type of formal system, and we have a vast range of requirement speciÿcation styles, from one end of the extensional-intensional spectrum to the other.. There are of course a number of areas in which work on cryptographic protocol requirements needs to be extended. One is in making the requirements language userfriendly. Security protocols, and thus their requirements, can be complex; even more so when one must consider operation in partial failure modes such as compromise of temporary session keys. Thus, it makes sense to concentrate on ways of making requirements languages easier to use. In this paper, we discussed some of the work on graphical requirements languages that attempts to address this problem. Of course, graphical requirements languages, and other methods for representing complex requirements, can only take us so far. We expect that one of the emerging problems in this area will not only expressing complex requirements, but limiting the complexity of such requirements in the ÿrst place.. There are some other areas which could also use more exploring. For example, many electronic commerce protocols must satisfy various types of non-safety requirements. Is it possible to develop ways of characterizing and specifying these requirements in ways that are particularly relevant to security protocols, as has been done for the safety properties of secrecy and correspondence? Another area of research has to do with interoperability. Increasingly, many protocols will rely upon other protocols to supply some of their security services. What is the best way to specify services needed by one protocol in terms of requirements upon another? We hope to see research in these and other emerging areas in the near future.

O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A memory-efficient pipelined implementation of the aho-corasick string-matching algorithm With rapid advancement in Internet technology and usages, some emerging applications in data communications and network security require matching of huge volume of data against large signature sets with thousands of strings in real time. In this article, we present a memory-efficient hardware implementation of the well-known Aho-Corasick (AC) string-matching algorithm using a pipelining approach called P-AC. An attractive feature of the AC algorithm is that it can solve the string-matching problem in time linearly proportional to the length of the input stream, and the computation time is independent of the number of strings in the signature set. A major disadvantage of the AC algorithm is the high memory cost required to store the transition rules of the underlying deterministic finite automaton. By incorporating pipelined processing, the state graph is reduced to a character trie that only contains forward edges. Together with an intelligent implementation of look-up tables, the memory cost of P-AC is only about 18 bits per character for a signature set containing 6,166 strings extracted from Snort. The control structure of P-AC is simple and elegant. The cost of the control logic is very low. With the availability of dual-port memories in FPGA devices, we can double the system throughput by duplicating the control logic such that the system can process two data streams concurrently. Since our method is memory-based, incremental changes to the signature set can be accommodated by updating the look-up tables without reconfiguring the FPGA circuitry. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 ( String matching has been studied extensively in the past 30 years. A string Y of length s is a sequence of characters c 1 c 2 . . . c s . Let = {Y 1 , Y 2 , . . . Y n } be a finite set of strings called keywords or signatures, and let I be an arbitrary string (the input stream to the string-matching engine). The string-matching problem is to locate and identify all the substrings of I which are signatures in . Pioneering string-matching algorithms include the methods of Aho and Corasick [1975] , Knuth et al. [1977] , and Wu and Manber [1992] . String matching has found widespread applications in text editors, word processing, bibliographic search, and pattern recognition. With rapid advancements in Internet technology and usage, some emerging applications in data communications and network security require the matching of a huge volume of data against large signature sets with thousands of strings. For example, the Snort intrusion detection system contains more than 6,000 strings in its rule set. These new applications sparkle renewed interests in the design of high-speed string-matching engines using FPGA or ASIC.. Sophisticated hardware implementation techniques have been reported in the literature. Baker and Prasanna [2005] presented a hardware implementation of the Knuth-Morris-Pratt (KMP) algorithm [Knuth et al. 1977] . Since the KMP algorithm is designed to match the input stream against a single string, one matching unit is required per string, and the hardware system is composed of a linear array of matching units. The methods of Clark and Schimmel [2004] , Baker and Prasanna [2006] , and Sourdis and Pnevmatikatos [2004] are based on pre-decoded characters with hardwired logic circuits. The system is optimized with respect to the given set of signatures and the characteristics of the FPGA devices. The FPGA is reconfigured when there are changes to the signature set. However, the long latency required in offline generation of the optimized hardwired circuits is considered as a major disadvantage in a network intrusion detection system that demands fast responses to hostile conditions. Other implementation techniques based on hashing [Cho and Mangione-Smith 2005; Papadopoulos and Pnevmatikatos 2005; Sourdis et al. 2008] ; Bloom filters [Dharmapurikar et al. 2005; Dharmapurikar and Lockwood 2006; Lu et al. 2006] ; and ternary content addressable memory (TCAM) [Yu et al. 2004] have also been proposed.. Hardware-assisted string-matching engines based on the AC algorithm have gained much popularity in recent years. An attractive feature of the AC algorithm is that it can solve the string-matching problem in time linearly proportional to the length of the input stream. All the strings in the signature set are integrated into a single deterministic finite automaton (DFA) such that the processing time is independent of the size of the signature set. A major disadvantage of the AC algorithm is the high memory cost required to store the transition rules of the DFA. Consider a signature set with n strings and an average length L. The maximum number of states in the AC state graph is equal to n × L. There are 256 transition edges from each state. If n = 5,000 and L = 20, there are up to 25.6 million transition edges in the state graph. Implementing such a large transition rule table in embedded devices is not feasible.. Space-time trade-off in string matching using DFA and nondeterministic finite automaton (NFA) has been studied extensively. The advantage of DFA is that the processing time is deterministic. To process one input character, the automaton only needs to perform one table look-up and makes one state transition. The disadvantage of DFA is the high memory cost. On the other hand, NFA is well known for its superior space efficiency over DFA. The state graph maintained by the NFA is reduced to a tree. The disadvantage of using NFA is that the automaton can make multiple state transitions for each input character. If the NFA is implemented using a memory-based approach, multiple table look-up operations are required for each input character. To avoid the speed penalty, researchers map the finite automata to hardwired logic circuits. However, hardwired circuits lack flexibility. Whenever the signature set is updated, the circuits need to be recompiled.. Previous studies had either focus on (i) reducing the memory space in the DFA using various techniques, such as compression, encoding, and transition edge reduction, or (ii) simplification of the logic for implementing NFA with hardwired circuits. In this article, we present a memory-based pipelined processing approach that possesses the advantages of DFA and NFA such that it can achieve the computation efficiency of DFA and space efficiency of NFA. The work presented in this article is an extension of our previously proposed pipelined architecture for string matching [Pao et al. 2008] . We observe a very subtle property of the NFA for the string-matching problem. Each state in the state graph represents a distinct substring of a pattern in the signature set. If we assign a level number to each node in the state graph according to the length of the substring represented by the corresponding state, we will find that there can be, at most, one active state on each level at any given time. We show that by making use of this property, the pipelined architecture can achieve deterministic processing speed and state graph reduction at the same time.. Conceptually, each pipeline stage will make one state transition for each input character. In order to achieve a processing speed of one character per cycle, each table look-up operation should be completed in one cycle. The conventional approach to implement large look-up tables (LUTs) is based on hashing. However, collisions are not avoidable for dynamic signature set. When collisions occur, a table look-up operation may require multiple memory accesses. As a result, the throughput of the system will be degraded. In this article, we present a space-efficient method to implement the required LUTs such that a look-up operation is guaranteed to be completed in one cycle.. The performance of the pipelined architecture is evaluated using a signature set with 6,166 static strings extracted from the Snort database downloaded in April 2008. The normalized storage cost of our method is only 18 bits per character (bits/char) in the signature set. Moreover, the control logic of the proposed method is very simple. If multiport memory modules are available, we can duplicate the control logic and implement multiple pipelines that share one copy of the LUTs. Typical FPGAs are equipped with dual-port memories. Hence, the system can process two independent input streams concurrently and double the throughput with very little overhead.. The organization of this article is as follows. In Section 2, we define some terminology and give a formal characterization of the AC automaton. We identify the major issues in hardware implementation and give a detailed review of related work. The proposed pipeline system is presented in Section 3. We discuss the motivation and explain why the proposed pipelined processing approach can guarantee the elimination of backward transitions in the DFA. In Section 4, we present how to implement the LUTs using an approach called direct indexing plus bit-selection (DIBS). The success of DIBS depends on an intelligent strategy in assigning numeric identifiers (IDs) to states in the DFAs, and string segments. In Section 5, we present performance evaluation and comparison with previously published AC-based methods. Section 6 concludes the article. Hardware implementation of the AC algorithm using FPGA or ASIC has received much attention in the past few years. A major issue in the design of string-matching engine based on the AC algorithm is the prohibitive memory requirements. In this article, we show that by incorporating pipelined processing, all cross edges in the AC state graph can be removed. This offers very substantial reduction of the memory cost in the hardware implementation. We have derived an intelligent strategy to assign physical IDs to states and string segments such that the LUTs can be implemented with good efficiency, and the look-up operation is guaranteed to complete in one cycle. Our design is memory-based. Incremental changes to the signature set can be accommodated by updating the contents of the LUTs without the needs to reconfigure the FPGA.. The control structure of the pipeline system is simple and elegant. If dualport memories are available, we can duplicate the control logic to allow the pipeline system to process two data streams concurrently. By doing so, the system throughput is increased to two characters per cycle. We have detailed the design of our method using a signature set extracted from the Snort database. The signature set contains 6,166 strings with close to 100K characters. The memory cost of our method is only 18 bits/char, while the cost of the control logic is only 0.04 LUT/char. The pipelines can operate at 253MHz when implemented on the Stratix II FPGA device.. A fundamental property of AC-based string-matching method is that the whole pattern set needs to be stored in embedded memory. Hence, the memory cost cannot be lower than 8 bits/char. For antivirus systems, such as ClamAV, the number of strings in the virus database is over 82 thousands, and the average pattern length is about 102 bytes. The overall size of the ClamAV pattern set exceeds 8Mbyte, which is more than 80 times the size of the Snort pattern set. If a string-matching engine is to be built for the ClamAV pattern set using the AC-based approach, over 20Mbyte of embedded memory will be required. The required memory resource is well beyond the capacity of today's FPGA devices. In a related study by the first author [Pao et al. 2010 ], a hybrid architecture for matching the ClamAV pattern set was developed, where P-AC was one of the major building blocks. The hybrid architecture has two major processing pipelines, namely the P-AC and the QSV. QSV is a checksum-based approach that uses quick sampling of fixed-length prefix plus on-demand verification of variable-length suffix. The overall memory cost of the hybrid architecture for the ClamAV pattern set is about 1.4Mbyte (i.e., only 1.4 bits/char).. Specification of intrusion patterns and virus using regular expression is gaining popularity. In a future study, we will investigate hardware architectures to accelerate the matching of regular expressions.

O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O