Model-based

testing

of

global

properties

on

large-scale

distributed

systems

Context:

Large-scale

distributed

systems

are

becoming

commonplace

with

the

large

popularity

of

peerto-peer

and

cloud

computing.

The

increasing

importance

of

these

systems

contrasts

with

the

lack

of

integrated

solutions

to

build

trustworthy

software.

A

key

concern

of

any

large-scale

distributed

system

is

the

validation

of

global

properties,

which

cannot

be

evaluated

on

a

single

node.

Thus,

it

is

necessary

to

gather

data

from

distributed

nodes

and

to

aggregate

these

data

into

a

global

view.

This

turns

out

to

be

very

challenging

because

of

the

system's

dynamism

that

imposes

very

frequent

changes

in

local

values

that

affect

global

properties.

This

implies

that

the

global

view

has

to

be

frequently

updated

to

ensure

an

accurate

validation

of

global

properties.

Objective:

In

this

paper,

we

present

a

model-based

approach

to

define

a

dynamic

oracle

for

checking

global

properties.

Our

objective

is

to

abstract

relevant

aspects

of

such

systems

into

models.

These

models

are

updated

at

runtime,

by

monitoring

the

corresponding

distributed

system.

Method:

We

conduce

real-scale

experimental

validation

to

evaluate

the

ability

of

our

approach

to

check

global

properties.

In

this

validation,

we

apply

our

approach

to

test

two

open-source

implementations

of

distributed

hash

tables.

The

experiments

are

deployed

on

two

clusters

of

nodes.

Results:

The

experiments

reveal

an

important

defect

on

one

implementation

and

show

clear

performance

differences

between

the

two

implementations.

The

defect

would

not

be

detected

without

a

global

view

of

the

system.

Conclusion:

Testing

global

properties

on

distributed

software

consists

of

gathering

data

from

different

nodes

and

building

a

global

view

of

the

system,

where

properties

are

validated.

This

process

requires

a

distributed

test

architecture

and

tools

for

representing

and

validating

global

properties.

Model-based

techniques

are

an

expressive

mean

for

building

oracles

that

validate

global

properties

on

distributed

systems.

Large-scale

distributed

systems

are

becoming

commonplace

with

the

increasing

popularity

of

peer-to-peer

or

cloud

computing.

For

instance,

the

Gnutella

system

shares

petabytes

of

data

among

millions

of

users.

Data

intensive

applications,

based

on

Google's

MapReduce

process

several

petabytes

of

data

every

day,

on

large

clusters

of

commodity

machines,

in

a

way

that

is

also

resilient

to

machine

failures..

The

high

popularity

of

these

systems

contrasts

with

the

lack

of

integrated

test

solutions

to

ensure

their

general

quality

under

normal

and

abnormal

conditions.

A

main

reason

is

the

complexity

of

reproducing

a

real

world

environment

together

with

a

nonintrusive

test

environment.

This

is

because

the

scale

of

the

system

affects

several

test

components,

such

as:

test

controllability

fault-injection

logging

facilities,

and

oracle

calculation,

among

others.

In

the

precise

case

of

the

oracle,

the

validation

of

global

properties

becomes

a

major

problem,

as

they

depend

on

values

that

are

spread

throughout

the

system.

This

problem

is

faced

when

testing,

for

instance,

the

global

correctness

of

the

routing

algorithm

of

a

system.

In

these

Elsevier

B.V.

All

rights

reserved..

systems,

efficient

message

routing

depends

on

the

correct

state

of

local

routing

tables,

which

must

be

maintained

frequently,

according

to

the

dynamic

state

of

the

system:

arbitrary

network

latencies,

node

failures,

and

churn.

Hence,

the

actual

content

of

a

routing

table

is

nondeterministic

and

highly

dynamic,

making

it

nonobvious

to

tell

whether

it

is

correct

at

any

given

point

in

time.

The

global

correctness

of

the

routing

algorithm

depends

on

the

(volatile)

content

of

the

routing

tables

in

each

local

node

that

is

very

hard

to

aggregate

into

a

global

view

in

a

timely

and

scalable

way.

Furthermore,

the

correctness

of

the

global

view

can

only

be

verified

at

given

states:

it

may

be

invalid

right

after

churn,

but

must

be

valid

after

a

certain

delay

in

a

stable

state..

A

typical

approach

for

testing

such

a

feature

in

a

distributed

system

consists

of

a

centralized

controller

and

several

testers,

each

one

controlling

a

single

port

or

node

interface

The

tester

is

the

application

that

runs

in

the

same

logical

devices

as

system

nodes,

and

controls

their

execution

and

their

volatility,

making

them

leave

and

join

the

system

at

any

time,

according

to

the

needs

of

a

test.

The

controller

sends

the

test

inputs,

controls

the

synchronization

of

the

test

case

execution

and

receives

the

outputs

(or

local

verdicts)

from

each

tester

However,

building

a

global

verdict

from

the

information

gathered

locally

can

be

a

very

difficult

problem.

For

instance,

in

a

system

where

each

node

maintains

a

set

of

references

to

its

physically

closest

neighbors

(e.g.,

Pastry

the

only

way

to

validate

the

correct

construction

of

the

system

would

be

to

first

gather

information

from

all

nodes,

then

calculate

the

distance

between

them,

and

finally

check

if

the

contents

of

the

reference

set

are

actually

the

closest

neighbors.

A

similar

problem

arises

when

verifying

load

balance

on

MapReduce

systems,

which

distribute

their

load

burden

across

their

nodes,

including

storage,

query

processing,

and

computations.

To

verify

their

algorithm

of

load-balancing,

one

must

gather

information

from

all

nodes,

which

can

be

a

large

amount

of

data,

and

check

for

system

usage

information

(e.g.,

partitioning

of

datasets)..

In

this

paper,

we

present

an

approach

leveraging

the

idea

of

model

at

runtime

to

provide

a

dynamically

built

oracle

for

testing

properties

in

large-scale

distributed

systems.

This

approach

focuses

on

global,

liveness,

observable

and

controllable

properties..

More

precisely,

it

focuses

on

a

particular

class

of

properties

that

cannot

be

calculated

by

a

single

node

or

by

a

portion

of

the

system;

that

are

eventually

true;

that

are

observable

from

the

system

interface;

and

that

respond

to

external

events.

We

propose

to

efficiently

keep

updating

a

global

model

of

the

system

during

its

execution.

This

model

is

then

instantiated

and

evolved

at

runtime,

by

monitoring

the

corresponding

distributed

system,

and

serve

as

oracle

for

distributed

tests.

On

the

implementation

side,

we

show

that

standard

Model-Driven

Engineering

(MDE)

technology

such

as

Kermeta

can

be

used

to

easily

implement

the

oracle

part

of

such

model-based

distributed

tests.

We

use

this

approach

to

test

topology-related

properties

on

two

open-source,

structured

systems.

This

approach

extends

our

previous

work

where

we

presented

a

methodology

and

an

architecture

to

test

local

properties

of

large-scale

distributed

systems,

which

can

be

verified

locally

to

each

node.

The

addition

of

a

global

model

and

an

efficient

update

mechanism

allows

also

to

test

global

properties..

The

rest

of

the

paper

is

organized

as

follows.

The

next

section

presents

a

real

world

motivation

case.

Section

introduces

some

fundamental

concepts

in

large-scale

distributed

systems

and

some

global

topology

properties

these

systems

must

satisfy.

Section

presents

our

approach

to

represent

and

check

these

properties,

as

well

as

our

architecture

for

testing

distributed

systems.

Section

describes

our

validation

through

implementation

and

experimentation

on

two

open-source

systems.

Section

discusses

related

work.

Section

concludes.

In

this

paper,

we

presented

a

model-based

approach

for

checking

global

liveness

properties

that

must

be

ensured

by

different

large-scale

distributed

systems.

We

claim

that

global

properties

should

be

checked

at

runtime,

at

real

scale,

using

non-invasive

distributed

testers,

and

that

model-based

testing

is

an

expressive

and

adaptable

technique

to

specify

and

check

the

global

liveness

properties

of

a

system..

In

our

approach,

test

sequences

put

the

system

into

states

where

such

properties

may

be

violated

or

lead

to

a

degradation

of

system

performance

and

behavior,

while

models

provide

a

high

abstraction

level

to

represent

a

global

view

and

the

required

properties

of

the

SUT.

Models

are

used

as

live

oracles,

which

have

a

view

of

the

current

state

of

the

system

and

can

detect

property

violations..

Along

with

the

approach,

we

presented

a

software

test

architecture

for

executing

test

cases.

This

architecture

ensures

the

correct

execution

of

test

sequences,

i.e.,

that

each

step

of

the

sequence

had

executed

correctly

by

all

concerned

node,

before

executing

the

succeeding

step.

In

this

architecture,

test

sequences

and

node

adapters

are

written

in

Java,

and

the

oracle

is

written

in

Kermeta,

a

dedicated

language

for

model

transformation..

We

illustrated

this

approach

by

testing

the

reliability

of

two

routing

algorithms

under

churn.

Results

showed

common

flaws

in

both

routing

strategies

and

clear

differences.

For

instance,

OpenChord

could

not

build

a

single

system

(i.e.,

a

strongly

connected

graph)

during

real-scale

experiments,

revealing

a

defect

that

would

not

have

been

detected

without

a

global

view

of

the

system..

It

is

important

to

note

that

the

approach

is

not

limited

to

reliability

testing.

It

can

also

target

other

distributed

software

testing

techniques,

such

as

system,

load,

and

elasticity

testing.

Indeed,

system

testing

and

load

testing

were

the

subject

of

previous

experiences.

The

approach

is

not

limited

to

a

specific

class

of

distributed

system

either,

but

to

specific

classes

of

properties.

More

precisely,

to

properties

that

need

a

global

view

of

the

system

to

be

checked,

and

that

are

only

observable

at

specific

states

of

the

system..

In

our

approach,

we

are

only

interested

in

the

state

of

the

system

at

some

specific

points,

reducing

exchanged

messages

during

a

test.

This

choice

prevents

us

from

checking

safety

properties;

i.e.,

properties

that

should

always

be

true.

However,

these

properties

are

typically

local

and

could

be

checked

using

assertions.

Since

we

chose

not

to

instrument

the

code

of

the

SUT,

we

can

access

neither

the

internal

states

of

a

node

nor

some

particular

attributes,

such

as

exchanged

messages

raised

during

a

given

query..

In

future

work,

going

beyond

the

oracle

issue,

we

will

explore

model-based

test

scenario

generation.

Models

at

runtime

and

search-based

algorithms

can

help

identifying

and

controlling

the

nodes

in

a

current

topology

for

pushing

routing

algorithms

into

their

limits;

for

instance

the

isolation

and

rejoin

of

a

node

subset

from

the

remaining

of

the

system

to

check

the

connectivity

property.

When

we

started

the

development

of

our

experiments,

we

intended

to

use

OCL

to

implement

the

oracle.

Indeed,

the

declarative

nature

of

OCL

simplifies

the

specification

of

global

properties.

However,

our

first

attempts

to

evaluate

OCL

expressions

on

models

with

several

hundreds

of

nodes

showed

poor

performance,

which

lead

us

to

use

an

imperative

language

instead..

In

our

approach,

we

separate

the

oracle,

test

sequences,

and

node

adapters,

allowing

these

three

parts

to

evolve

independently..

Adapters

depend

strongly

on

system

node

interface

and

must

be

rewritten

when

testing

different

systems.

Different

test

sequences

and

oracles

that

test

the

same

system

share

the

same

adapters.

Test

sequences

and

oracles

depend

on

adapters,

and

can

be

reused

for

testing

different

systems,

if

adapters

provide

the

same

interface.

As

the

global

model

evolves,

allowing

the

representation

of

new

information

and

hence

the

verification

of

additional

properties,

testers

must

collect

more

information.

This

implies

changes

in

the

adapter,

which

performs

additional

queries

on

the

SUT,

and

also

in

the

message

aggregation

function.

The

latter

is

only

needed

if

the

information

can

be

combined

and

reduced..

The

high

level

of

abstraction

of

model-based

tools

eases

the

representation

and

the

validation

of

global

properties.

These

tools

ensure

that

the

models

representing

the

oracle

data

are

sound

(with

respect

to

their

meta-model)

during

and

after

the

execution

of

test

sequences.

A

possible

limit

of

this

kind

of

tools

concerns

the

size

of

the

models:

most

model-based

tools

are

based

on

the

Eclipse

Modeling

Framework

(EMF),

which

is

not

adapted

to

deal

with

large

models

Our

approach

focus

on

specific

classes

of

properties:

global,

liveness,

observable,

and

controllable

properties.

It

is

not

adapted

to

verify

local

properties,

which

do

not

require

a

global

view

of

the

system.

It

is

not

adapted

to

verify

safety

properties

either,

since

they

require

an

analysis

of

all

the

historical

states

of

the

system.

Since

we

do

not

instrument

the

SUT,

we

cannot

verify

properties

that

are

not

observable

from

the

public

interfaces

of

the

system

and

that

do

not

respond

to

external

events..

In

the

current

implementation

of

the

architecture,

testers

can

force

nodes

to

end

their

execution

either

normally

or

abnormally.

This

allows

test

sequence

to

inject

''macro-level''

faults

and

implement

scenarios

that

are

not

interested

in

the

origins

of

a

failure.

However,

the

architecture

cannot

inject

specific

faults,

e.g.,

disk,

network,

bugs.

We

intend

to

combine

the

architecture

with

fault-injection

tools

to

overcome

this

limitation..

Another

limitation

of

the

current

architecture

concerns

the

reproducibility

of

tests,

i.e.,

it

does

not

provide

repeatable

automated

tests

In

our

experiments,

we

relied

on

the

infrastructure

to

ensure

the

use

of

the

same

environment

for

different

executions.

The

use

of

an

automated

staging

system

with

support

to

large-scale

environments

(e.g.,

Weevil

and

Mulini

to

deploy

and

execute

tests

can

overcome

this

limitation..

While

Properties

and

may

be

verified

using

information

available

at

the

SUT

interface,

the

verification

of

Property

is

more

complex.

In

order

to

measure

the

number

of

exchanged

messages,

one

must

monitor

the

communication

on

all

nodes

of

the

SUT,

filter

the

messages

that

are

not

related

to

the

self-organization

and

verify

that

the

number

of

messages

exchanged

corresponds

to

Oððlog

NÞ

Þ.

In

the

tests

presented

above,

we

used

a

different

approach:

we

measure

the

time

needed

for

self-healing

in

different

scales.

If

the

time

increases

quadratic-logarithmically,

we

consider

that

the

property

is

respected.

The

tests

also

showed

an

error

in

the

bootstrapping

process

of

OpenChord:

the

time

needed

to

create

a

valid

system

with

more

than

nodes

is

unsatisfactory,

making

the

implementation

unusable.

A

thorough

analysis

of

the

source

code,

the

execution

log

and

of

the

resource

usage

revealed

a

design

error.

Indeed,

when

a

join

request

arrives

at

the

bootstrapper

node,

it

creates

one

thread

to

process

the

request.

Thus,

when

several

requests

arrive

at

the

same

time,

the

node

spends

more

time

creating

threads

(and

context

switching)

than

actually

processing

the

requests.

Applying

the

Proactor

design

pattern

would

fix

this

error.

In

preliminary

work

we

presented

a

centralized

test

architecture

(i.e.,

central

coordinator

managing

distributed

testers)

for

systems.

With

this

architecture,

we

can

keep

information

about

running

tests

(e.g.,

routing

table

entries)

in

global

variables

placed

in

the

central

component.

Moreover,

we

investigated

two

issues:

we

demonstrated

that

volatility

is

a

key-parameter

when

testing

a

large-scale

system;

and

we

investigated

scalability

issues

of

such

architecture

resulting

from

the

cost

of

coordination

management

of

tests.

During

these

investigations,

we

executed

several

tests

on

different

systems.

These

tests

were

limited

to

oracles

that

used

deterministic

data

and

could

be

decided

locally

at

each

tester.

The

model-based

approach

presented

here,

allows

oracles

to

use

global

information;

hence,

allowing

us

to

write

more

elaborated

tests

to

check

global

properties..

Liu

et

al.

propose

an

analogous

approach.

The

authors

implemented

a

tool,

WiDS

Checker,

which

stores

events

from

a

distributed

application

and

uses

these

events

to

replay

the

execution

of

applications

developed

with

WiDS.

The

replay

uses

only

one

simulation

process.

During

the

replay,

their

tool

checks

for

predicate

violations

and,

when

a

violation

is

caught,

it

can

present

the

trace

that

leads

to

the

violation.

Unlike

our

approach,

they

use

a

pre-compiler

to

inject

tracking

code

to

the

SUT

and

have

access

to

its

internal

properties

(white-box

testing).

In

our

approach,

we

do

not

instrument

the

code

and

only

have

access

to

public

properties

(black-box

testing).

In

WiDS,

verifications

are

done

post-mortem

and

not

at

runtime.

nodes

correspond

to

nodes

per

machine

in

the

clusters.

Classes

SocketEndpoint

and

RequestHandler..

MaceODB

is

to

some

extent,

similar

to

WiDS

Checker:

it

injects

tracking

code

to

a

domain-specific

language,

Mace.

The

injected

code

is

also

used

to

check

for

liveness

and

safety

properties,

but

at

runtime.

Unlike

our

approach,

where

properties

are

written

in

Kermeta,

MaceODB

uses

a

declarative

language

to

specify

properties.

Thanks

to

this

language

the

tool

can

decide

which

node

(or

set

of

nodes)

can

check

for

a

given

property

and

to

optimize

message

exchange

between

nodes..

WiDS

Checker

and

MaceODB

are

both

close

to

our

approach.

However,

their

goal

is

to

help

debugging

of

distributed

systems

and

not

testing.

They

observe

the

execution

of

a

SUT,

in

all

its

states,

but

do

not

use

test

sequences

to

drive

this

SUT

to

given

states.

As

they

do

not

implement

a

test

architecture

and

their

test

code

runs

in

the

same

process

as

the

SUT,

they

cannot

execute

more

complex

test

scenarios,

involving

synchronization,

churn

simulation,

or

load

and

scale

variation..

Common

test

architectures

for

distributed

systems

are

generally

based

on

centralized

test

controller.

Ulrich

et

al.

for

instance,

present

a

test

architecture

for

distributed

systems,

called

Test

and

Monitoring

Tool

(TMT),

which

coordinates

actions

by

synchronization

of

events.

This

architecture

uses

a

global

tester

and

a

distributed

tester.

The

global

tester

divides

the

test

cases

in

small

parts

called

partial

test

cases

(PTC).

Each

PTC

is

assigned

to

a

distributed

tester

and

can

be

executed

in

parallel

to

another

PTC

with

respect

to

a

function

that

controls

mutual

exclusivity.

The

behavior

of

the

distributed

testers

is

controlled

by

a

Test

Coordination

Procedure

(TCP)

which

coordinates

the

PTCs

execution

by

synchronization

events.

By

using

synchronization

events,

the

distributed

testers

do

not

need

to

control

the

execution

of

the

entire

test

case.

Each

tester

runs

independently

its

partial

test

case.

Another

test

architecture

was

presented

by

Walter

et

al.

for

conformance,

interoperability,

performance

and

real-time

testing.

This

architecture

works

as

a

toolbox

of

components

(e.g.,

communication,

test

coordination,

etc.)

which

can

be

combined

to

develop

a

specific

test

architecture.

The

components

can

be

also

used

to

simulate

failures

along

the

tests

like

network

delay

or

noise.

For

instance,

it

allows

creating

failures

in

the

network

to

simulate

volatility.

Because

these

architectures

are

tailored

for

small-scale

systems

and

rely

on

a

centralized

topology,

they

scale

up

linearly

and

are

not

well-suited

to

large-scale

systems..

Some

test

architectures

are

tailored

for

specific

large-scale

systems,

such

as:

MapReduce,

Publish-subscribe

systems

(PSS)

and

The

MRUnit

is

a

test

framework

for

MapReduce

systems.

The

framework

provides

a

JUnit-like

approach

where

developers

write

Java-based

code

to

accelerate

industry

acceptance.

MRUnit

mocks

the

MapReduce

system

to

test

individually

and

locally

the

map

and

the

reduce

functions

of

a

MapReduce

job.

In

comparison

to

our

approach,

MRUnit

exclusively

addresses

functional

unit

testing.

It

neither

tests

an

entire

MapReduce

distributed

computation,

nor

provides

test

facilities

to

overcome

volatility

issues

that

arise

with

a

large

number

of

machines..

Another

large-scale

testing

approach

is

presented

for

PSS,

that

are

similar

to

systems.

Some

model-checking

tools

are

based

on

this

approach,

such

as

Bogor

and

Cadence

SMV

While

they

consider

the

volatile

nature

of

PSS,

scalability

is

an

open

issue.

Along

testing,

nodes

are

simulated

as

threads

and

the

size

of

the

SUT

may

be

bounded

by

the

testing

machine

resources.

Therefore,

large-scale

systems

cannot

be

fully

tested

since

implementation

flaws

are

strongly

related

to

the

system

scale

The

Testing

and

Test

Control

Notation

is

a

standardized

test

scripting

language

that

applies

to

different

application

domains:

telecommunications,

protocols,

aerospace,

grid

application

workflows

etc.

Along

with

the

test

language,

also

provides

a

test

execution

architecture,

based

on

the

CTMF

specification

Similarly

to

our

approach,

separates

test

sequences

from

test

adapters,

which

are

specific

to

the

SUT.

In

comparison

to

our

approach,

adapters

are

language-independent

and

support

data

coding

and

decoding.

The

test

architecture

uses

a

centralized

test

controller

and

is

not

fully

adapted

for

large-scale

systems..

Other

approaches

also

provide

a

language

to

test

large-scale

systems

In

fact,

they

differ

from

the

presenting

a

declarative

language.

These

languages

are

analogous

to

the

Structured

Query

Language

(SQL)

widely

used

to

query

data

base

systems.

Our

approach

could

take

benefit

from

these

languages

to

enhance

testability

of

large-scale

systems..

Network

or

overlay

discrete-event

simulation

tools,

such

as

AgentJ

J-Sim

SimJava

or

PeerSim

are

an

interesting

complementary

approach

to

actual

deployment.

These

tools

can

execute

thousands

or

even

million

of

nodes

on

a

single

machine,

simulating

network

communication,

and

multithreading.

Thus,

simulation

is

more

economic

than

actual

deployment

and

can

be

used

before

actual

deployment

to

check

the

adequacy

of

test

cases

Simulation

can

also

be

used

to

check

global

properties

in

a

simpler

manner,

since

all

nodes

run

in

the

same

machine.

However,

from

a

testing

perspective,

simulation

tools

have

two

main

issues.

The

first

issue

is

that

they

often

involve

rewriting

a

version

of

an

application

for

simulation.

The

second

is

that

there

is

no

real

concurrence

and

faults

related

to

concurrence

or

load

may

not

manifest

themselves.

For

instance,

we

used

PeerSim

to

create

an

OpenChord

DHT

with

nodes

and

checked

for

bootstrap

faults,

similar

to

the

ones

described

in

Section

Contrarily

to

its

behavior

during

our

experiments,

OpenChord

could

create

a

single

system

under

simulation..

The

Pigeon

framework

is

a

network

simulator,

which

addresses

both

volatility

and

scalability

issues

to

test

massively

multiplayer

online

games

(MMGs).

The

authors

believe

that

the

performance

of

MMGs

is

the

most

crucial

problem

that

should

be

addressed

since

the

network

latency

is

affected

by

the

frequent

propagation

of

updates

during

a

game.

To

monitor

the

network,

Pigeon

uses

reflection

and

aspect-oriented

programming

to

include

additional

code

to

the

SUT.

While

this

framework

is

more

adapted

to

simulate

the

peculiarities

of

large-scale

systems,

it

shares

the

same

two

issues

of

other

simulation

tools.

Cloud

computing

offers

a

highly-scalable

environment

that

runs

any

kind

of

system.

For

instance,

large-scale

systems

with

centralized

components,

such

as:

GoogleFS

(with

master

nodes),

and

MapReduce

(with

job

trackers);

or

decentralized

large-scale

Tables

(DHT)

The

development

of

these

systems

is

raising

new

requirements

and

issues,

other

than

those

already

tackled

during

the

development

of

traditional

distributed

systems:.

Large-scale

Large-scale

distributed

systems

are

expected

to

connect

a

large

number

of

nodes

(from

thousands

up

to

several

millions).

Dynamicity

Resources

may

be

dynamically

added

to

or

removed

from

the

system.

This

concerns

physical

nodes,

as

well

as

software

services.

Heterogeneity

of

resources

The

nodes

that

compose

the

system

are

disparate:

smartphones,

laptops,

clusters,

supercomputers,

etc.

Diversity

of

purposes

These

systems

are

used

on

different

domains,

from

data

sharing

to

massive

processing

applications.

Stateless

protocols

Nodes

may

receive

events

defined

in

their

interfaces

in

any

order

and

at

any

moment.

In

essence,

communication

consist

of

independent

requests-responses

pairs.

Symmetry

Several

nodes

play

identical

roles,

ensuring

reliability

(there

is

no

single

point

of

failure)

and

load

balance

(load

is

distributed

symmetrically

across

nodes).

Elasticity

The

system

scales

out

and

in

quickly,

adapting

itself

to

load.

In

the

current

version

the

test

architecture

is

implemented

in

Java

(version

global

test

sequences

and

adapters

are

implemented

as

Java

methods.

Meta-information

about

test

steps

(e.g.,

the

subset

of

testers

that

should

execute

a

step,

timeout,

etc.)

are

described

as

Java

annotations.

Listings

and

present

examples

of

a

node

adapter

and

a

test

sequence,

respectively..

Besides

exposing

nodes'

interfaces,

adapters

also

describe

a

method

for

updating

the

global

model.

When

a

tester

receives

an

update

request,

it

queries

the

node

it

controls,

computes

the

differences

with

the

previously

sent

information

and

sends

these

differences

to

the

test

controller..

We

use

Kermeta

(version

to

implement

the

oracle

part

of

the

test,

i.e.,

the

runtime

model

and

the

verification

methods:

connectivity

and

diameter

Since

Kermeta

is

interoperable

with

Java

(it

compiles

to

Java

bytecode),

its

integration

with

the

test

controller

is

flawless.

The

former

is

implemented

using

a

depthfirst

search

algorithm

and

the

latter

is

implemented

using

the

Floyd-Warshall

algorithm

for

the

''all

pairs

shortest-path

problem''

(see

its

implementation

in

Appendix

A).

B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Performance Modeling of Hybrid MPI/OpenMP Scientific Applications on Large-scale Multicore Supercomputers In this paper, we present a performance modeling framework based on memory bandwidth contention time and a parameterized communication model to predict the performance of OpenMP, MPI and hybrid applications with weak scaling on three large-scale multicore supercomputers: IBM POWER4, POWER5+ and BlueGene/P, and analyze the performance of these MPI, OpenMP and hybrid applications. We use STREAM memory benchmarks and Intel's MPI benchmarks to provide initial performance analysis and model validation of MPI and OpenMP applications on these multicore supercomputers because the measured sustained memory bandwidth can provide insight into the memory bandwidth that a system should sustain on scientific applications with the same amount of workload per core. In addition to using these benchmarks, we also use a weak-scaling hybrid MPI/OpenMP large-scale scientific application: Gyrokinetic Toroidal Code (GTC) in magnetic fusion to validate our performance model of the hybrid application on these multicore supercomputers. The validation results for our performance modeling method show less than 7.77% error rate in predicting the perfor- mance of hybrid MPI/OpenMP GTC on up to 512 cores on these multicore supercomputers. Today, the trend in high performance computing systems has been shifting towards cluster systems with multicore. Further, Multicore processors are usually configured hierarchically (e.g., multiple multicores compose a multichip module to form a node) to form a compute node of parallel systems. While multicore presents significant new opportunities such as on-chip high inter-core bandwidth and low latency, it also presents new challenges in the form of inter-core resource conflict and contention. For many years, there has been considerable concern over the growing imbalance between memory subsystem performance and processor performance. Many of us fear that multicore continues to exacerbate memory bandwidth problems, however, the stall in processor frequency has diminished the corresponding worsening of memory latency in terms of processor-clocks [11] .. Levesque et al. [11] observed from the AMD architectural discussion that when excluding messaging performance, the primary source of contention when moving from single core to dual core is memory bandwidth, and confirmed this assumption by the testing with STREAM [12] and Membench microbenchmarks. In our previous work [20, 19, 18] , we also found that memory bandwidth contention is the primary source of performance degradation for L2/L3 shared architectures such as CrayXT4/XT5 and IBM Power4/Power5 and BlueGene/P systems using NAS Parallel benchmarks and large-scale scientific applications such as the Gyrokinetic Toroidal Code (GTC) [7] when increasing the number of cores per node. Therefore, in this paper, we focus on scientific applications with memory bandwidth contention problems, and present a performance modeling framework based on memory bandwidth contention time on small number of cores to predict the performance on larger number of cores.. Multicore supercomputers provide a natural programming paradigm for hybrid MPI/OpenMP applications. Current hybrid parallel programming paradigms such as hybrid MPI/OpenMP need to efficiently exploit the potential offered by such multicore supercomputers. Generally, MPI is consid-ered optimal for process-level coarse parallelism and OpenMP is optimal for loop-level fine grain parallelism. Combining MPI and OpenMP parallelization to construct a hybrid program is not only to achieve multiple levels of parallelism but also to reduce the communication overhead of MPI at the expense of introducing OpenMP overhead due to thread creation and increased memory bandwidth contention. In this paper, we analyze the performance of MPI, OpenMP and hybrid applications on three large-scale multicore supercomputers: IBM POWER4, POWER5 and BlueGene/P, and present a performance modeling framework based on memory bandwidth contention time and parameterized communication model to predict the performance of OpenMP, MPI and hybrid applications with weak scaling on these multicore supercomputers.. The experiments conducted for this work utilize multicore supercomputers with different number of cores per node. P655 [14] has 8 cores per node, Hydra [15] has 16 cores per node, and BlueGene/P (Intrepid) [3] at Argonne National Laboratory (ANL) has 4 cores per node. Further, each system has a different node memory hierarchy. We use the MPI and OpenMP STREAM memory benchmarks [12] and Intel's MPI benchmarks [9] to provide initial performance analysis of MPI and OpenMP applications on these multicore supercomputers. In addition to using these benchmarks, we also use a hybrid MPI/OpenMP large-scale scientific application: a 3D particle-in-cell application GTC in magnetic fusion to validate our performance models of these MPI, OpenMP and hybrid applications. Our experimental results for our performance modeling method show less than 7.77% error rate in predicting the performance of MPI, OpenMP and hybrid MPI/OpenMP GTC on up to 512 cores of these multicore supercomputers.. The remainder of this paper is organized as follows. Section 2 discusses the architecture and memory hierarchy of three large-scale multicore supercomputers used in our experiments, and compares their performance using STREAM benchmarks and Intel's MPI benchmarks. Section 3 proposes and discusses a performance modeling framework for hybrid MPI/OpenMP applications based on memory bandwidth contention time and a parameterized communication model. Section 4 validates our performance models using the hybrid GTC. Section 5 discusses some related work and Section 6 concludes this paper. In the remainder of this paper, we assume that the job scheduler for each multicore supercomputer always dispatches one process to one core or one thread to one core. We describe the system configuration as MxN whereby M denotes the number of nodes with N cores per node. All experiments were executed multiple times to ensure consistency of the performance data. Prophesy system [16] is used to collect all application performance data. In this paper, we analyzed and compared the performance of MPI, OpenMP and hybrid applications on three large-scale multicore supercomputers, IBM POWER4, POWER5+ and BlueGene/P using MPI and OpenMP STREAM benchmarks, Intel's MPI benchmarks and the hybrid MPI/OpenMP GTC, and presented a practical performance modeling framework for weak-scaling hybrid MPI/OpenMP applications based on the memory bandwidth contention time and the parameterized communication model to model and predict the performance of hybrid and OpenMP GTC on these multicore supercomputers. The experimental results for our performance modeling method showed less than 7.77% error rate in predicting the performance of the hybrid MPI/OpenMP GTC on up to 512 cores on the three multicore supercomputers. We also used the performance model to estimate and compare the memory bandwidth contention times for the MPI and OpenMP GTC.. For weak-scaling scientific applications, the measured sustained memory bandwidth can provide insight into the memory bandwidth that a system should sustain on these scientific applications with the same amount of workload per core. However, for strong scaling scientific applications, it is hard to measure the sustained memory bandwidth for the different amount of decreased workload per core. This is a limitation for our performance modeling framework. We believe that our performance modeling framework can be applied to any weak-scaling hybrid MPI/OpenMP applications with memory bandwidth contention problems. Our future work will focus on exploring performance modeling for strong scaling scientific applications, and investigating performance-power trade-off models in our MuMMI project [13] . Levesque et al [11] observed that the primary source of contention when moving from single core to dual core on AMD Opteron architecture is memory bandwidth, and proposed a simple performance model for sequential applications to extrapolate the quad-core performance by assuming the time spent in the execution component remains the same, but the time spent in memory bandwidth contention will increase proportional to the reduction in effective memory bandwidth per core. In this paper, we generalize the performance model for OpenMP and hybrid parallel applications to extrapolate the performance of the multicore node (which consists of one or several multicores).. Barker et al [5] discussed using performance modeling to design large scale systems throughout their life cycle from early design through production use, presented an application-centric performance model development framework, and addressed the importance of performance modeling. Our performance model has more focus on multicore, and takes into account memory bandwidth contention from multicore.. Adhianto and Chapman [1] proposed a cost efficient approach to model and evaluate parallel OpenMP, MPI and hybrid MPI + OpenMP with reasonable accuracy based on a combination of static analysis (using the OpenUH compiler) and feedback from runtime benchmarks (Sphinx and Perfsuite) for both communication and multithreading efficiency measurement, and predicted the performance of a parallel matrix multiply using Cannon algorithm.. Aversa et al. [4] described simulation-based techniques for performance prediction of hybrid MPI/OpenMP code in different working conditions using HeSSE (Heterogeneous System Simulation Environment), and predicted the performance of a parallel N-body code on a SMP cluster.. There are many approaches to modeling MPI communication cost, such as LogP [6] , LogGP [2] , collective communication models [21] and so on, in this paper, we just take a straightforward empirical method to manually parameterize the communication time for each MPI subroutine by the product of the time for each MPI subroutine with the given message size and number of processors, and the number of calls on the given system to generate a parameterized communication model, then based on the performance data for each MPI subroutine on a given system stored in the Communication database, we can estimate the total communication cost on the system.

B I O B I O O O O B I O O O O O O B I O O O B I I I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I I I O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O B I I I O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O B I O O O B I O O O O B I I I O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Visualizing genomic information across chromosomes with PhenoGram Background: With the abundance of information and analysis results being collected for genetic loci, user-friendly and flexible data visualization approaches can inform and improve the analysis and dissemination of these data. A chromosomal ideogram is an idealized graphic representation of chromosomes. Ideograms can be combined with overlaid points, lines, and/or shapes, to provide summary information from studies of various kinds, such as genome-wide association studies or phenomewide association studies, coupled with genomic location information. To facilitate visualizing varied data in multiple ways using ideograms, we have developed a flexible software tool called PhenoGram which exists as a web-based tool and also a command-line program. Results: With PhenoGram researchers can create chomosomal ideograms annotated with lines in color at specific base-pair locations, or colored base-pair to base-pair regions, with or without other annotation. PhenoGram allows for annotation of chromosomal locations and/or regions with shapes in different colors, gene identifiers, or other text. PhenoGram also allows for creation of plots showing expanded chromosomal locations, providing a way to show results for specific chromosomal regions in greater detail. We have now used PhenoGram to produce a variety of different plots, and provide these as examples herein. These plots include visualization of the genomic coverage of SNPs from a genotyping array, highlighting the chromosomal coverage of imputed SNPs, copy-number variation region coverage, as well as plots similar to the NHGRI GWA Catalog of genome-wide association results. Conclusions: PhenoGram is a versatile, user-friendly software tool fostering the exploration and sharing of genomic information. Through visualization of data, researchers can both explore and share complex results, facilitating a greater understanding of these data.As the types and amount of genomic data being collected continue to increase, so does the need for tools to visualize, analyze, and share these data. One useful data visualization approach for genomic results is the use of chromosomal ideograms. An ideogram is a graphical representation of chromosomes, and these plots have been used with the addition of overlaid points, lines, and shapes to provide summary BioData Mining With the ever increasing amounts of data being collected, visually summarizing data can be important for providing insight into complex results. Multiple data results can be plotted across chromosomes, providing useful summary information, and aiding in data analyses as well as sharing results. PhenoGram offers a robust feature set, allowing researchers to plot data of many kinds across a chromosomal ideogram according to preference. In the future we will be adding in additional color option choices for plots, as well as additional software features, to expand plotting options with PhenoGram. The features of PhenoGram can further facilitate the exploration and sharing of genomic information. PhenoGram was developed in Ruby, using the RMagick graphics library. The software can be downloaded for use at the command line. The software can also be used via a web-based graphical user interface without a need for downloading the software, and a screen capture of the web interface is shown in Figure 1 . Both the web-based graphical user interface as well as the stand-alone software are available at: http://visualization. ritchielab.psu.edu. An example file is available at that site for trying out PhenoGram with the web-based graphical user interface. There are multiple options that can be used to create various plots, and Table 1 shows the complete list of command line arguments, which are also available on the web interface. A single, tab-delimited input file is required to produce a PhenoGram plot. At a minimum, the input file must contain columns to identify the chromosome, and the basepair position or base-pair to base-pair region to be plotted. Other columns such phenotype, annotation, ancestry or group, and position-color provide additional PhenoGram visualization options. Table 2 summarizes the formatting parameters of the input file. The file must be tab delimited. PhenoGram will ignore any unrecognized headers.

O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Management's Role in the Approval and Administration of Decision Support Systems Decision support systems (DSS) The last fifteen years have witnessed tremendous decision support systems (DSS) advances. The worth of the DSS concept has been shown by the number of decision support systems which have been successfully developed and used by organizations. Vendors have flocked to the marketplace with hardware and software products which are useful in building and operating decision support systems. There is now an annual summer conference dedicated to decision support systems.. There have also been significant advances in DSS research. The earliest research efforts focused on creating systems which we now call DSS. A marketing and production planning system, a corporate planning system, and a system for supporting brand marketing decision making are all noteworthy examples (Scott Morton [19] , Hamilton and Moses [7] , and Little [14] ). Important conceptual work soon followed which aided in clarifying the goals and characteristics of DSS (Scott Morton [19] , Sprague and Watson [22, 23] , and Keen and Scott Morton [11] ). Other authors helped identify the unique developmental methodology required by DSS (Gerrity [4] and Ness [18] ). Currently, DSS research has progressed to the point where laboratory experiments are being conducted [13] . There are now enough decision support systems in operation that large scale field studies are possible [28] .. A relatively unexplored DSS area is management's role in their approval and administration. For example, what are the motivations for creating a DSS? How is a proposed DSS project evaluated? How do firms plan and organize for building a DSS? What techniques are used to review and control DSS projects? How is a DSS managed as an organizational entity? These topics are increasing in importance as a growing number of organizations are considering building decision support systems.. Answers to questions such as these have been suggested in part by insightful observers of the DSS scene and by individuals with first-hand experience in building decision support systems. However, to the authors' knowledge, no large scale field study research has been conducted on the approval and administration of DSS. Because of this, the authors recently completed a detailed investigation of eighteen decision support systems and report here how they were approved and are administered. It is believed that this information should be of interest and value to other DSS researchers and to practitioners who currently have or are considering the creation of decision support systems. Eighteen decision support systems were investigated to determine how they were approved and how they are administered. The following statements summarize the study's findings.. The development of a DSS was usually preceded by a realization of its potential value in decision making. This value may result from the impact of more accurate, timely, and/or new information. An organizational champion was often an important factor in the decision to create a DSS. Typically, this person is a middle or upper level manager in the department where the DSS will be used. ". The evaluation of the desirability of creating a DSS was found to be based largely upon intuition. This was due primarily to the difficulty of quantifying the potential benefits. In general, a DSS was created if the potential benefits exceed expected costs of an acceptable, low level.. In planning for DSS, attention was focused on building the current DSS, but with some thought given to the creation of future decision support systems. In nearly all cases, the ultimate users were primarily responsible for the development of the DSS. Frequently, however, the CBIS group provides hardware, systems software, and communications support.

O O O O O O O O O O O B I I I O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A robust optimization approach for dynamic traffic signal control with emission considerations We consider an adaptive signal control problem on a signalized network whose traffic flow dynamic is described by the Lighthill-Whitham-Richards (LWR) model (Lighthill and Whitham, 1955; Richards, 1956) . Such problem explicitly considers traffic-derived emission as side constraints. We seek to tackle this problem using a mixed integer mathematical programming approach. Such a problem class, which we call LWR-Emission (LWR-E), has been analyzed before to certain extent. Since mixed integer programs are practically efficient to compute in many cases (Bertsimas et al., 2011) , the mere fact of having integer variables is not the most significant challenge to computing solutions of MIPs; rather, it is the presence of the nonlinear and nonconvex emission-related constraints that render the program computationally expensive.To address this issue, we proposed an efficient and practical way of solving the LWR-E problem, by formulating it as a mixed integer linear program (MILP). This methodology relies on the existence of a strong correlation between the aggregate vehicle emissions rate and certain macroscopic traffic quantities, such as the number of vehicles on a link. This correlation is fitted by a function and deviations from this function are treated as parameter uncertainty. We then apply robust optimization techniques to reformulate the emissions-related constraints into convex and tractable forms. To further reduce the computational cost, we employ the link transmission model to represent traffic dynamics. The final proposed MILP explicitly captures vehicle spillback, avoids traffic holding and features time-varying signal cycle and splits, as well as the aforementioned emissions constraints. Traffic signals are essential elements in the management of transportation networks. Over the past decades, signal control strategies have evolved greatly: from simple fixed-time plans based on historical data and updated infrequently throughout the day to adaptive control systems that update continuously in response to real-time traffic information. The performance of a signal control system depends on two primary factors: the optimization procedure employed and the objective of the optimization. We provide a general framework for treating emission-related constraints and/or objectives in a signal optimization problem. For traffic dynamics we employ the LWR model (or more accurately, the link-based kinematic wave model) with extension to signalized networks. Such a problem, which we call LWR-E, is difficult to solve using a mathematical programming approach due to the highly nonlinear and nonconvex emission constraints. To overcome this, we propose a robust optimization approach guided by a statistical learning of certain macroscopic emission models.. Specifically, we seek to represent vehicle emission rates (of any type) on the link level using only macroscopic traffic quantities inherent in the link-based traffic network model. Through extensive numerical simulations that incorporate a variety of vehicle emission models (Penic and Upchurch, 1992; Post et al., 1984) , we are able to recover certain well-defined relationships between the aggregate emission rates (AER) on a link and the link occupancy. Given any (piecewise) polynomial approximation of the macroscopic relationships mentioned above, we can express the emission constraints using the link-based variables and the bounds of the uncertainty set, without resorting to any knowledge regarding the dynamics inside each link. Particularly helpful is the fact that the robust formulation yields linear constraints for emissions when the aforementioned macroscopic relationships are linear or piecewise linear with concavity or convexity. In addition, we are able to incorporate, besides emission bound constraints, a wider range of emission considerations such as minimizing the total emission amount, and the environmental equity. Remarkably, all these problems can be reformulated into mathematically tractable forms and, in most cases, mixed integer linear programs.. A numerical study on a hypothetical network demonstrates the effectiveness of the LWR-E problem and the proposed solution method in bounding/reducing emissions, while maintaining a satisfactory performance of the signalized network in terms of maximizing throughput. However, as pointed out by Scenario III in the study, the local bounding/minimization of emissions may lead to network-wide degradation in terms of both air quality and throughput. Thus a well-balanced signal timing plan with environmental concerns requires multiple criteria and proper weighting of different objectives.

O B I O O O O B I O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


The Birth of New Genes by RNA- and DNA-Mediated Duplication during Mammalian Evolution Gene duplication has long been recognized as a major force in genome evolution and has recently been recognized as an important source of individual variation. For many years the origin of functional gene duplicates was assumed to be whole or partial genome duplication events but recently retrotransposition has also been shown to contribute new functional protein coding genes and siRNA's. In this study we utilize pseudogenes to recreate more complete gene family histories and compare the rates of RNA and DNA-mediated duplication and new functional gene formation in five mammalian genomes. We find that RNA-mediated duplication occurs at a much higher and more variable rate than DNA-mediated duplication, and gives rise to many more duplicated sequences over time. We show that while the chance of RNA-mediated duplicates becoming functional is much lower than that of their DNA-mediated counterparts, the higher rate of retrotransposition leads to nearly equal contributions of new genes by each mechanism. We also find that functional RNA-mediated duplicates are closer to neighboring genes than non-functional RNA-mediated copies, consistent with cooption of regulatory elements at the site of insertion. Overall new genes derived from DNA and RNA-mediated duplication mechanisms are under similar levels of purifying selective pressure, but have broadly different functions. RNA-mediated duplication gives rise to a diversity of genes but is dominated by the highly expressed genes of RNA metabolic pathways. DNA-mediated duplication can copy regulatory material along with the protein coding region of the gene and often gives rise to classes of genes whose function are dependent on complex regulatory information. This mechanistic difference may in part explain why we find that mammalian protein families tend to evolve by either one mechanism or the other, but rarely by both.p3 The impact of changes in gene copy number on both evolution and human health are under increasing scrutiny. While the creation of new genes and the modulation of gene copy-number via duplication has long been recognized as an important mechanism for the evolution of lineage-specific traits (Ohno 1970 ), a number of recent studies have suggested that variation in gene family size may be even more widespread than previously appreciated (Fortna et al. 2004) and that gene copy number variation between individuals may account for differences in disease predisposition within populations (McCarroll and Altshuler 2007; Redon et al. 2006 ).. Four primary mechanisms of gene duplication have been described (for review see (Hahn in press; Li 1997): whole genome duplication or polyploidization (Hurley et al. 2005; Skrabanek and Wolfe 1998; Van de Peer et al. 2003) , unequal crossing over or tandem duplication (Shoja and Zhang 2006) , duplicative transposition (Bailey et al. 2003; Friedman and Hughes 2004; Samonte and Eichler 2002) , and retrotransposition (Harrison et al. 2005; Zhang et al. 2002) . Of these, whole genome duplication, tandem duplication, and duplicative transposition are DNAmediated duplication events (abbreviated here as DD for DNA Duplication), while retrotransposition is RNA-mediated (abbreviated here as RD for RNA Duplication). Whole genome duplication has been important to the evolution of many lineages (Jaillon et al. 2004 ; Van de Peer et al. 2003) , but it is a relatively rare event. Unlike whole genome duplication events, tandem duplication and duplicative transposition (sometimes collectively referred to as segmental duplications) occur continuously and have contributed significantly to the divergence of gene content between mammalian genomes. Duplication by retrotransposition also occurs quite frequently, but because newly retrotransposed gene copies are duplicated by an RNAmediated mechanism they lack the promoter and other flanking regulatory sequences of the parental gene. For this reason, retroduplication events have long been believed to give rise primarily to non-functional pseudogenes (Petrov and Hartl 1999; Shemesh et al. 2006) . Recent studies however, have indicated the presence of many apparently functional retrocopies in various mammalian genomes, challenging traditional perspectives on the relevance of this event to genome evolution (Sakai et al. 2007; Svensson et al. 2006; Vinckenbosch et al. 2006) . Very recently retrotransposition has also been shown to contribute siRNA genes to the genome (Tam et al. 2008; Watanabe et al. 2008) .. In this study we compare the rates of new gene formation by DNA-mediated duplication (DD) p4 and RNA-mediated duplication (RD) in five eutherian genomes. We show that, while genes arising from DD events are up to six times more likely to remain functional than those arising from RD events, the number of RD events is nearly ten times that of DD events, resulting in roughly equal quantitative contributions of new genes from each mechanism. Our analysis further shows that (1) duplicate genes generated by each mechanism are under similar levels of constraint on their protein coding regions but belong to distinctly different classes of genes, (2) gene families tend to evolve by one duplication mechanism or the other, but rarely both, (3) silent site substitution profiles of RD duplicate copies are consistent with bursts of retrotransposition during mammalian evolution, while DNA-mediated duplication appears to occur at a more stable rate, and (4) RD copies under stabilizing selective pressure have insertion sites significantly closer to the nearest genes.

O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O


Triangular bubble spline surfaces We present a new method for generating a G n -surface from a triangular network of compatible surface strips. The compatible surface strips are given by a network of polynomial curves with an associated implicitly defined surface, which fulfill certain compatibility conditions. Our construction is based on a new concept, called bubble patches, to represent the single surface patches. The compatible surface strips provide a simple G n -condition between two neighboring bubble patches, which are used to construct surface patches, connected with G n -continuity. For n ≤ 2, we describe the obtained G n -condition in detail. It can be generalized to any n ≥ 3. The construction of a single surface patch is based on Gordon-Coons interpolation for triangles.Our method is a simple local construction scheme, which works uniformly for vertices of arbitrary valency. The resulting surface is a piecewise rational surface, which interpolates the given network of polynomial curves. Several examples of G 0 , G 1 and G 2 -surfaces are presented, which have been generated by using our method. The obtained surfaces are visualized with reflection lines to demonstrate the order of smoothness. The present paper describes a new methodology for generating a G n -surface from a triangular mesh given by a network of curves with an associated implicitly defined surface, called compatible surface strips. Our method is a multipatch scheme, which is one of the main approaches for constructing a smooth surface from a triangular or quadrilateral mesh (cf. [4, 12] ). The fundamental idea of this approach is the construction of single surface patches, pieced together with G n -continuity, which define the desired G nsurface. A survey of this concept is presented in [17] . The amount of existing literature on this topic is large; some references are [6, 7, 13, 16, 18, 19, 20] .. Based on the multi-patch approach, several methods for constructing a G n -surface, which interpolates a given network of curves, have been developed (cf. [9, 14, 15, 24] ). In general, a mesh of curves has to fulfill certain compatibility conditions, especially at the vertices, to be feasible for G n -interpolation. For instance, [10, 11] describe geometric constraints on a network of curves which have to be satisfied for the case of G 1 and G 2 -smooth surfaces. A method for generating a smooth triangular (rectangular) surface from given three triangular (four rectangular) surface patches is presented in [21, 22] . The construction is based on a generalization of the Gordon-Coons interpolation.. In [8] , Hahn describes an algorithm for filling k-sided polygonal holes with quadrilateral surface patches, meeting at a common vertex. The single surface patches are generated with the help of Cordon-Coons interpolation in such a way that they are joined with G n -continuity. The main difference to our approach consists in the fact, that Hahn's construction uses an explicitly constructed diffeomorphism between neighboring patches in order to satisfy the G n -conditions. Consequently, the construction of a single patch depends on the neighboring patches. The choice of the Hermite boundary data for one edge defines already the data for all other edges. Moreover, the construction is not perfectly symmetric, since the cross boundary derivatives are always specified on one side of the edges.. The novelty of our method consists in the use of a network of compatible surface strips instead of a network of curves. Starting from a triangular network of compatible surface strips, we generate a triangular spline surface with G n -continuity, interpolating the given curves of the compatible surface strips. Moreover, we do not need to specify the diffeomorphism between neighboring patches, since the G n -conditions are specified as contact between implicit and parametric surfaces.. Our method is based on a new type of surface patches, called bubble patches, for representing the individual surfaces. The use of these surface patches provides a simple and natural way to define free-form surfaces over triangular meshes and possesses several advantages. The construction of a single patch is simple and independent of the neighboring patches. In addition, it works uniformly for triangular meshes with vertices of arbitrary valency. Consequently, modifications of the connectivity of the mesh only affect those patches that are actually modified. The resulting surfaces are piecewise rational with G n -continuity. The remainder of the paper is organized as follows. In Sections 2 and 3 we introduce some basic definitions. In particular, we introduce the idea of a triangular network of compatible surface strips, which is a network of polynomial curves with an associated implicitly defined surface, fulfilling certain compatibility conditions. A possible construction of a suitable network is presented in Appendix B.. Section 4 describes a new methodology, called bubble patches, for representing triangular surface patches. In Section 5 we use the compatible surface strips to describe a simple G n -condition between two bubble patches. This provides a simple method to generate Hermite boundary data of the patches, which guarantees G n -continuity between two neighboring patches.. Section 6 describes the construction of C n -bubble patches by using Gordon-Coons interpolation for triangles. This interpolation scheme is a well known tool for generating a smooth function, interpolating the given Hermite boundary data and is summarized in Appendix A. Section 7 presents several examples of generated G 0 , G 1 and G 2 -surfaces and verifies their smoothness with the help of reflection lines. Finally, we conclude the paper. In this paper we have presented a simple method for constructing a G n -surface from a triangular network of compatible surface strips, which is a network of curves with an associated implicitly defined surface.. Summing up, our construction scheme works as follows. For each triangle we generate a single surface patch, represented by a new methodology, called bubble patch. Thereby a single surface patch is constructed with the help of triangular Coons interpolation in such a way that the patches are connected with G n -continuity. This is achieved by generating Hermite boundary data of the bubble patch for the Gordon-Coons method, which ensures the desired G ncontinuity between the patches. For this we use the compatible surface strips to describe a simple G n -condition between the patches, which provides the suitable Hermite boundary data. Our algorithm is explained for n ≤ 2 in detail and several examples are presented.. The advantages of our method are as follows. Our construction is local and simple and works uniformly for meshes of arbitrary valency. The resulting surfaces are piecewise rational surface patches, which are pieced together with G n -continuity. By using low degree approximations (e.g. splines) for the Hermite boundary data we obtain (spline) surface patches of low degree. The method can be extended to quadrilateral and mixed triangular and quadrilateral meshes with vertices of arbitrary valency. An example from a quadrilateral mesh is shown in Fig 5. As a possible topic for future work, we are currently trying to derive a geometric subdivision scheme from our construction, by sampling points and Hermite data from the constructed surface. The use of a globally defined implicit representation could provide a new and promising approach to verify the smoothness of the obtained limit surfaces, which does not require local parametrizations and can deal uniformly with vertices of any valency.. k,j (t) t=l = δ i,j δ k,l for j ∈ {0, . . . , n} and l ∈ {0, 1}.. A general construction scheme of the Gordon-Coons method of degree n for generating a C n -function h : T → R is as follows:. ∂s j h(r, s) (r,s)=(u,v) for i, j ∈ {0, . . . , n}. Consider the given Hermite boundary data. for i ∈ {0, . . . , n}, satisfying the twist compatibility condition (9) .. (ii) For a function g : T → R, define the operators P 1 , P 2 and P 3 as. Cube Bar Tube Rotated horse shoe Rotated horse shoe and. where the indices for g indicate differentiation and. (iii) Then the function h is given by. for any choice of i, j ∈ {1, 2, 3} with i = j, where the Boolean sum operator P i ⊕ P j is given by. The fulfillment of the twist compatibility condition (9) for the vertex v k for k ∈ {1, 2, 3} with i = j = k = i is sufficient for applying the interpolant (A.1); see [2, Theorem 2.1]. An example of an interpolant (A.1) is. for which the explicit formulas for n = 0 and n = 1 are presented in Subsection A.2 and A.3, respectively (cf. [2] ). Clearly, the interpolant (A.1) is not symmetric. According to [2] , a more symmetric interpolant can be obtained by averaging interpolants (A.1).. [2, Theorem 2.4] describes similar interpolants, which do not need the twist compatibility condition (9) to be satisfied. Instead, they need some additional assumptions concerning the Hermite boundary data (3) and (4).

O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O