The

art

of

mathematics

retrieval

The

design

and

architecture

of

MIaS

(Math

Indexer

and

Searcher),

a

system

for

mathematics

retrieval

is

presented,

and

design

decisions

are

discussed.

We

argue

for

an

approach

based

on

Presentation

MathML

using

a

similarity

of

math

subformulae.

The

system

was

implemented

as

a

math-aware

search

engine

based

on

the

state-ofthe-art

system

Apache

Lucene.Scalability

issues

were

checked

against

more

than

arXiv

documents

with

million

mathematical

formulae.

Almost

three

billion

MathML

subformulae

were

indexed

using

a

Solr-compatible

Lucene.

The

solution

to

the

problem

of

mathematical

formulae

retrieval

lies

at

the

heart

of

building

digital

mathematical

libraries

(DML).

There

have

been

numerous

attempts

to

solve

this

problem,

but

have

found

widespread

adoption

within

the

wider

mathematics

community.

And

as

yet,

there

is

no

widely

accepted

agreement

on

the

math

search

format

to

be

used

for

mathematical

formulae

by

library

systems

or

by

Google

Scholar..

MathML

standard

by

has

become

the

standard

for

mathematics

exchange

between

software

tools.

Almost

no

MathML

is

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page.

To

copy

otherwise,

to

republish,

to

post

on

servers

or

to

redistribute

to

lists,

requires

prior

specific

permission

and/or

a

fee.

written

directly

by

authors-they

typically

prefer

a

compact

notation

of

some

T

E

X

flavour

such

as

L

A

T

E

X

or

AMSL

A

T

E

X.

The

designer

of

a

search

system

for

mathematics

is

thus

faced

with

the

task

of

converting

data

to

a

unifying

format,

and

allowing

DML

users

to

use

their

preferred

notation

when

posing

queries..

[AMS]L

A

T

E

X

or

similar

T

E

X

macropackages

are

the

typical

preferences;

Presentation

MathML

or

Content

MathML

are

used

only

when

available

as

outputs

of

a

software

system..

During

the

integration

of

existing

DMLs

into

larger

projects

such

as

EuDML

the

unsolved

math

search

problem

becomes

evident-DML

without

math

search

support

is

an

oxymoron.

We

have

evaluated

several

systems

for

our

goals:

MathDex

is

an

MSE

developed

in

Bremen/Saarbrücken

by

Kohlhase

et

al.

Our

evaluation

has

lead

us

to

conclude

that

there

is

no

satisfactory,

math-aware

and

scalable

solution.

For

this

reason,

we

designed

and

implemented

a

new

robust

solution

for

retrieving

mathematical

formulae..

Section

presents

our

design

of

this

scalable

and

extensible

system

for

searching

mathematics,

taking

into

account

not

only

inherent

structure

of

mathematical

formulae

but

also

formula

unification

and

subformulae

similarity

measures.

Our

evaluation

of

a

prototypical

implementation

on

the

Apache

Lucene

open

source

full-featured

search

engine

library

is

presented

in

Section

The

paper

concludes

with

a

description

of

the

WebMIaS

interface

and

listing

future

work

directions

in

Section

and

with

a

summary

in

Section

Art

is

never

chaste.

It

ought

to

be

forbidden

to

ignorant

innocents,

never

allowed

into

contact

with

those

not

sufficiently

prepared.

Yes,

art

is

dangerous.

Where

it

is

chaste,

it

is

not

art.

Pablo

Picasso

We

have

presented

an

approach

to

mathematics

searching

and

indexing-the

architecture

and

design

of

the

MIaS

system,

and

its

WebMIas

interface.

The

feasibility

of

our

approach

has

been

verified

on

large

corpora

of

real

mathematical

papers

from

arXMLiv.

Scalability

tests

have

confirmed

that

the

computing

power

needed

for

fine

math

similarity

computations

is

readily

available

and

allows

the

use

of

this

technology

for

projects

on

world-wide

scale.

Math

Indexer

and

Searcher

demonstrated

the

ability

to

index

and

search

a

relatively

vast

corpus

of

real

scientific

documents.

Its

usability

is

greatly

improved

thanks

to

its

preprocessing

functions

together

with

the

formulae

weighting

model.

The

ability

to

search

for

exact

and

similar

formulae

and

subformulae,

more

so

with

customizable

relevance

computation,

demonstrates

an

unquestionable

contribution

to

the

whole

search

experience..

It

is

very

difficult,

if

not

impossible,

to

completely

verify

the

correctness

of

the

theoretical

considerations

made

in

the

previous

sections

and

thus

correctness

of

search

results.

For

this

purpose,

a

sufficiently

large

corpus

of

documents

with

fully

controlled

content

would

be

needed.

For

any

assembled

query,

there

should

exist

beforehand

a

complete

list

of

the

documents

ordered

by

their

relevance

to

the

query

to

compare

the

actual

results

to.

On

the

other

hand,

the

real

world

relevance

of

the

results

being

returned

needs

to

be

verified

by

an

extensive

user

study

and

perhaps

several

parameters

need

to

be

adjusted

for

the

best

results.

We

have

applied

an

empirical

approach

to

the

evaluation

so

far.

For

these

purposes

we

created

a

demo

web

interface,

see

Section

The

Math

Indexer

and

Searcher

is

written

in

Java.

The

role

of

fulltext

indexing

and

searching

core

is

performed

by

Apache

Lucene

The

mathematical

part

of

document

processing

can

be

seen

as

a

standalone

pluggable

extension

to

any

full-text

library,

however

it

needs

custom

integration

for

each

one.

In

the

case

of

Lucene,

a

custom

Tokenizer

(MathTokenizer)

has

been

implemented..

When

searching

for

mathematical

formulae,

their

weights

need

to

be

considered

in

the

final

score

of

the

document.

The

scoring

function

of

our

MIaS

system

adds

one

parameter

to

the

Lucene's

standard

practical

scoring

function

(described

in

detail

at

http:

lucene/search/Similarity.html)-weight

w

of

one

matched

formula:.

If

a

document

contains

the

same

formula

more

than

once

(each

occurrence

can

have

different

weight

assigned),

the

average

value

of

all

the

weights

is

taken

into

consideration

(avg(w)).

O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


High Quality Navigation in Computer Games Navigation plays an important role in many modern computer games. Currently the motion of entities is often planned using a combination of scripting, grid-search methods, local reactive methods and flocking. In this paper we describe a novel approach, based on a technique originating from robotics, that computes a roadmap of smooth, collision-free navigation paths. Because the vast amount of computation time is spend in the pre-processing phase, navigation during the execution of an application is almost instantaneous. The created roadmap can be queried to obtain high-quality paths. Furthermore, the applications of the roadmap are not limited to navigating an entity. Therefore, besides navigation for an entity, two other applications are presented; one for planning the motion of groups of entities and one for creating smooth camera movements through an environment. All applications are based on the same underlying techniques. Many games genres, specifically Real Time Strategy (RTS) and (tactical) First/Third Person Shooters (FPS/TPS), contain large numbers of entities. These entities can for example be world inhabitants, non playing characters (NPCs), and vehicles. These games heavily depend on the correct and believable motion of entities. To create those motions, the games must plan the paths of entities between locations in the game world. Currently this is typically achieved using a combination of scripting, A * -based grid search, local reactive methods and flocking.. In scripting, the designer explicitly described the paths that can/must be followed by the entities. This is normally part of level design. Scripting is a time consuming process for the designer. In addition, it can lead to repetitive behavior that is easily observed by the user. Scripting gets increasingly complicated when many entities move in the same space and when spaces become larger (as is the case in modern gaming).. Grid based methods divide the world in a grid of cells and plan motion using an A * -based search on the free cells (see e.g. (1; 2) ). If the game world is small and grid-like, this technique is useful. However, when the world becomes complicated and large and many entities must move around, grid based methods take a large amount of computation time. Pruning the search reduces the time but might lead to wrong paths. Also, motions created by grid search tend to look unnatural. In order produce smooth paths using grid searches, the paths need to be be smoothed in the post-processing phase resulting in expensive queries.. Reactive methods adapt a previously computed motion to obstacles found near the path that were not taken into account during initial planning; for example other entities or small, movable objects (see e.g. (3; 4; 5; 6)). Even though theoretically reactive methods can be used to compute full paths, this normally leads to dead-lock situations in which the entities no longer know where to move (e.g. they get stuck in a corner of the room). Another problem with reactive methods is that it is often difficult to adapt the internal animation of the character to the motions produced.. Flocking originates from the animation community. Reynolds first introduced the approach in (7) . Based on the natural phenomena seen in birds and sheep, the entities in flocking look to their direct neighbors and, based on some simple rules, try to do the same as them. This results in very natural movement of the entities, but guarantees nothing on the paths. For example, it is not guaranteed that the entities will ever reach their goal. Flocking is often used in conjunction with other method to overcome this problem, typically by guiding one or more entities using other motion planning techniques, see e.g. (8) .. In addition to the above issues, in some applications the game world is not completely known at design time because it contains for example a door or a moving car. Also there is a tendency to allow games to automatically generate game worlds rendering existing navigation methods less efficient. Therefore there is a need for algorithms that automatically (i.e. without the need of a human designer) allow for high quality navigation using only negligible com-putation time.. During the last three decades, many path planning approaches have been developed in the field of robotics, that may be applicable to path planning in games. One popular path planning technique in robotics is the Probabilistic Roadmap Method (PRM). It has been studied by many authors, see e.g. (9; 10; 11; 12) . In a pre-processing phase this method builds a roadmap of possible motions of the robot through the environment. When a particular path planning query must be solved, a path is retrieved from this roadmap using a simple and fast graph search. The PRM approach is suited for very complicated environments. Unfortunately, the roadmap produced by the method can be rather wild, leading to low quality paths consisting of straight line segments, that require a lot of time-consuming smoothing during game play.. In this paper we will describe a novel path planning approach, building on the PRM method, that can be effectively applied to games. The approach also constructs a roadmap of possible motions but guarantees that the paths are short, have enough clearance from the obstacles, and are C 1 continuous, leading to natural looking motions. See Figure 1 for an example of a roadmap computed using our approach. After constructing the roadmap, which can be done in the pre-processing phase, paths can be retrieved almost instantaneously, and do not require any post-processing.. Besides the standard application, in which the roadmap is used for planning the paths for individual entities, we describe two additional applications. First, we consider the motion of groups of entities. This problem is often solved using a combination of grid-based planning and flocking (7; 13). Unfortunately, this can lead to unwanted behavior where the group of entities splits up (see Figure 2 for an example). We will use the smooth paths computed by the new planning approach as a backbone path and then use a social potential field approach to guide the flock through a corridor around this path (extending our earlier work in (14) ). This results in a natural motion in which the group is guaranteed to stay together.. In many games, moving the camera through an environment is one of the most fundamental operations. Currently the camera is often under direct control of the user or under indirect control of the user. Direct camera control is difficult, it easily leads to motion sickness due to redundant motions, and is often not required. Building on our earlier work in (15) we describe a method in which the user only specifies positions of interest and the camera automatically moves to such positions using a smooth, collision free motion. For computing the camera path we will use the new planning approach described. This is then combined with techniques to control the view direction and the speed of the camera to obtain a camera motion that is pleasant to watch. (a) A group of five characters should attack the site pointed to by the arrow.. (b) The group inappropriately splits up, loosing some of its troops. Fig. 2 . One of the problems with the current techniques for motion planning for multiple units is that the group splits up to reach the goal. This scene was taken from Command and Conquer: Generals from EA Games. In this paper we have described a novel technique for automatic construction of high-quality roadmaps in games that can be used for computer controlled entities to quickly navigate. We have also shown how this technique can be used to plan the motion for groups of entities and to navigate a camera. Due to the nature of our roadmap, the approach also works in large environments, where grid-based method typically fail (requiring too much storage and processing time).. We presented our method as a 2-dimensional approach in which entities move on a ground plane. It is easy to extend it to e.g. terrains and even motions in buildings in which the roadmap would automatically follow the corridors and stairs.. Roadmap construction is best seen as being part of the construction of the game world. It is easy to incorporate special requirements from the level designer. For example, the designer can add fake obstacles to force the path to e.g. stay on the sidewalks of a road. Also the designer can easily manipulate the roadmap graph by manually adding, changing, or removing nodes. Moreover, weights can be added to the graph to e.g. indicate preferred routes.. Since the creation of the roadmap on which the navigation is based is done completely automatic, our methods are also suited for applications in which the game world is not completely known at design time. The behavior of the group can be controlled by adjusting the coherence parameters, lateral dispersion and longitudinal dispersion. Figures 12(a) to 12(c) show a group of 50 entities moving through an environment. In these pictures the lateral and longitudinal dispersion is varied, resulting in a longer, more stretched group (12(a)) or more compact group (12(c)). the entities that passed the narrow passage earliest, wait for the last entity to pass the passage.. To test the usability in real-time games, we constructed a demo application written in C++. In this demo application, the units are allowed to move at a maximum speed of 15 per second (can off course be any arbitrary value). From experiments with this demo application we can conclude that, in order to produce paths reliably (that is without collisions), we are allowed to move 3 per iteration. Although the physics engine of a typical computer game runs at 50-60 Hz, the positions and velocities of the units in our demo application need to be updated at 5 iterations per second. Every iteration (pre-)calculates about 0.2 seconds of movement. The resulting processor usages for different group sizes are given in the following For example, it takes 0.02 seconds to calculate a second of movement for a group of 100 units, i.e. only 2% of processor time.. (a) The group with high longitudinal dispersion and very low lateral dispersion.. (b) The group with medium longitudinal dispersion and medium lateral dispersion.. (c) The group with low longitudinal dispersion and larger lateral dispersion.. (d) The group moves through a narrow passage, making the first wait for the last ones (movement is from the lowerleft to the upper-right). Fig. 12 . A group of 50 entities moving in a virtual world. These paths and behaviors are created with the same approach, only by varying the parameters.. We implemented our approach in a walk-through system for virtual worlds called Quest3D. This is a DirectX based 3D development package. The navigation techniques were added as a module using Visual C++. In Figure 14 a screenshot of this application is shown. Rather than letting the user steer the camera directly, we display a map in the top left corner. By clicking on the map the user indicates the position he wants to move to. An automatic smooth camera motion is then calculated such that the user can focus on the environment instead of the camera control. Again, because the vast amount of calculation is done in the pre-processing phase, the processor time required for camera control during the running of the application is minimal. Experiments indicate that paths generated by the methods described in this section are pleasant to watch.

O O B O B I B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Logic and Memory Design Based on Unequal Error Protection for Voltage-scalable, Robust and Adaptive DSP Systems Abstract In this paper, we propose a system level design approach considering voltage over-scaling (VOS) that achieves error resiliency using unequal error protection of different computation elements, while incurring minor quality degradation. Depending on user specifications and severity of process variations/channel noise, the degree of VOS in each block of the system is adaptively tuned to ensure minimum system power while providing "just-theright" amount of quality and robustness. This is achieved, by taking into consideration block level interactions and ensuring that under any change of operating conditions, only the "less-crucial" computations, that contribute less to block/system output quality, are affected. The proposed approach applies unequal error protection to various blocks of a system-logic and memory-and spans multiple layers of design hierarchy-algorithm, architecture and circuit. The design methodology when applied to a multimedia subsystem shows large power benefits (up to 69% improvement in power consumption) at reasonable image quality while tolerating errors introduced due to VOS, process variations, and channel noise. Miniaturization of devices has resulted in the integration of numerous complex and power hungry processing units on a single portable device, the operation of which is mainly constrained by limited battery lifetime. Due to the quadratic dependence of power on voltage, supply voltage over-scaling (VOS) has been investigated as an effective method to reduce power [1] . However, VOS increases the delays in all computation paths and can result in incorrect or incomplete computation of certain paths. Besides power consumption, process variations also pose major design concern with technology scaling, resulting in delay errors [2] . Conventional wisdom dictates up-scaling the supply voltage or logic gate up-sizing to prevent delay errors and to achieve higher parametric yield. However, such techniques come at a cost of increased power and/or die area. Hence, meeting the contradictory requirements of high yield (in presence of unreliable components), low power and high quality is becoming exceedingly challenging in nanometer designs.. Methodologies that jointly address the issues of low power and error resiliency in digital signal processing (DSP) blocks have been proposed recently in [3, 4] . The methodology in [3] ensures tolerance to delay errors, utilizing the concept of unequal error protection. By protecting the "crucial" computations (more contributive to output quality) through algorithm/architecture co-design, low power at minor quality degradation is achieved. On the other hand, algorithmic noise tolerance (ANT) in [4] adds a reduced precision replica of the original block that acts as error control; it estimates potential errors of the main block and tolerate any timing violations by selecting the most correct computed output. Such methodologies have proven to be efficient for the design of individual blocks, by being able to tolerate errors due to VOS/process variations, intelligently trading off quality-of-results.. However, when these blocks are integrated in a system, then the interactions between the individual blocks have to be considered. First, we observe that lower power or good quality for an individual block does not necessarily translate to lowest power or best quality for the overall system. Interestingly, while some blocks may operate at lower power and achieve "good" block level quality, subsequent blocks might still be consuming higher power without improving the system quality. This can be due to the fact that they are performing "redundant" computations based on VOS-affected incorrect outputs (albeit "less crucial one") of the preceding blocks. These computations can be thought of as "redundant" or "lesscrucial" for subsequent blocks as they unnecessarily increase system power without improving the overall quality. Second, depending on dynamically changing system/user requirements and operating conditions (process variations, channel noise), the best quality may not always be required or achievable.. In this paper, we propose a system level design approach that achieves error resiliency under variations while considering VOS, using using unequal error protection of different computation elements, while incurring minor quality degradation. By taking into account block level interactions, the proposed approach provides the "right" amount of quality at the right amount of power consumption (by configuring the degree of VOS). Specifically, depending on power, user quality requirements and severity of process variations/channel noise, the degree of VOS for each block is adaptively tuned to ensure minimum system power while providing "just-theright" amount of quality and robustness. This is achieved by guaranteeing that the delay errors due to VOS affect only the less-crucial (redundant) computations of the block (system) by efficient algorithmic/architectural modifications.. Apart from logic blocks, memory elements are also a ubiquitous part of today's systems and consume significant percentage of overall system power. From system perspective logic blocks interact with memory elements. It is possible, while logic blocks consume low power, the memory elements continue to consume high power, storing unnecessary computations that are potentially erroneous due to scaled voltage or process variations in logic blocks. Therefore, there is a need to expand the scope of unequal error protection to memory elements. It should be observed that standard memory cells (6 transistor SRAM cells or 6T cells) are vulnerable to failures under voltage scaling [5] . Hence, we apply a preferential storing policy in which the crucial computations are stored in robust memory banks composed of robust 8T-cells [6] and less crucial computations are stored in conventional 6T-cells. Such configuration ensures low probability of failure for the crucial banks even under VOS or process variations due to the robust nature of 8T cells, translating to lower quality loss. Furthermore, the less crucial banks can be dynamically turned off depending on quality/power user requirements, allowing system to adapt to various operating conditions and significantly reducing the overall system power when best quality is not required/achievable. The proposed approach departs from a recently published work [7] in which authors considered bit-significance rather than transform level-computations (such as in DCT/IDCT) as discussed in this paper.. The proposed approach is applied to system level joint design of logic (focusing on DCT/IDCT) and memory blocks of a sub-system which is ubiquitous in various applications (i.e. from lossy compression of audio (e.g. MP3) and images (e.g. JPEG) to spectral methods for the numerical solution of partial differential equations). In this paper we focus on a multimedia sub-system in order to show the implications and required modifications in each sub-block of such a system. Our contributions in this paper can be summarized as follows: 1) Considering VOS based DCT implementation we design a voltage scalable and robust IDCT by accounting for block level interactions in a multimedia sub-system. 2) We identify crucial/less-crucial computations for maintaining high output image quality in IDCT. The IDCT coefficients are modified such that only the less-crucial IDCT computations (based on potentially incorrect DCT outputs) are affected under VOS (unequal error protection for different computation elements). This avoids any unnecessary power increase, while ensuring minimal quality degradation of IDCT as well as multimedia sub-system. 3) Adaptively tune the degree of VOS for each logic block (DCT, IDCT) for achieving minimum system power in the presence of process variations/channel noise, meeting user specifications by using unequal error protection, unlike existing implementations [8] [9] [10] . 4) A hybrid memory architecture is proposed that ensures correct read/write operations of crucial computations even under VOS. 5) The proposed architecture, inherently conceals channel noise at scaled Vdd's during transmission over noisy channel.. The rest of this paper is organized as follows. In section 2 the proposed system level approach focusing on logic is explained. In section 3 the application of our technique to a multimedia sub-system is described. Section 4 presents the unequal error protection scheme applied to memory elements of a sub-system. Section 5 discusses the system level design for power awareness and error tolerance. Section 6 discusses the integration of the proposed multimedia sub-system within a wireless system. Finally, conclusions are drawn in section 7. In this paper, we propose a design approach based on system level interactions that provides "right" amount of quality and power for each block, while dissipating minimal system power under the given operating conditions. This is achieved by ensuring that, only the less-crucial computations can get affected by the operating conditions (VOS, process variations, channel noise), thereby incurring minimal quality degradation. An unequal error protection scheme was also applied to memory design that allows significant power reduction by disabling adaptively, less crucial banks of the memory. Furthermore, the utilization of robust 8T cells in crucial banks allowed voltage over-scaling resulting in further power reduction in the system memory. In addition, the VOS based design methodology can be used as a channel error concealment technique while minimizing system power consumption (by VOS). To demonstrate the efficiency of the proposed approach, it was applied to a multimedia sub-system. However, the proposed methodology can also be applied for designing other DSP systems where interaction between blocks need to be considered for providing "just-the-right" amount of power/quality and robustness. We implemented the proposed IDCT and a conventional multiplierless IDCT (using original 8-bit coefficients) architecture in Verilog. We obtained synthesized Verilog netlists from Design Compiler [15] , using ARM standard cell libraries [16] which were converted to HSPICE netlists using Calibre [17] . The average power consumption at 300 MHz in IBM 90 nm technology was obtained using NanoSim. Moreover, using a set of 24 images [13], average PSNR for the proposed and conventional IDCT were obtained. The comparison results are shown in Table 2 .. We would like to mention that the variance of the PSNR at different Vdd levels, 1.2 V, 1.1 V, 0.71 V were 2.84, 4.77 and 5,71, respectively. Note that the proposed architecture allows large power savings with graceful quality degradation while conventional design fails under VOS. In addition note that the multiplexers impose a 4% delay overhead in the proposed design. Table 3 presents the PSNR values of some representative images at various voltages. Note that quality degradation at scaled voltages depends on the image and its information (i.e. number of high frequencies). However, in all cases our architecture provides acceptable output quality (PSNR above 20 dB) since it protects intelligently the significant parts that carry-out most of the information.. As we mentioned apart from unequal error protection, other techniques for the design of voltage scalable and error resilient logic blocks exist such as ANT [4] . In ANT, an estimator block (which is actually a reduced precision replica) is utilized to identify any error in computations in presence of delay variations due to Vdd scaling; additionally, a decision block is used to select the correct output at scaled Vdds. It is important to note that such techniques have also to be modified in order to account for the system level interactions between the various blocks. In addition, we need to note that unequal error protection provide more power savings compared to ANT since it doesn't require the addition of any hardware to protect the significant computations [3] . This leads to minor area and power overhead even at nominal voltage of operation. All in all, algorithm along with architectural innovations, as suggested in the proposed approach, can yield smaller efficient designs/ systems that allow extreme power savings at both the Figure 4 Proposed voltage scalable and process tolerant 1D-IDCT architecture. nominal (due to reduced effective switched capacitance) and scaled Vdds.

O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Thread transparency in information flow middleware Abstract. Applications that process continuous information flows are challenging to write because the application programmer must deal with flow-specific concurrency and timing requirements, necessitating the explicit management of threads, synchronization, scheduling and timing. We believe that middleware can ease this burden, but middleware that supports control-flow centric interaction models such as remote method invocation does not match the structure of these applications. Indeed, it abstracts away from the very things that the information-flow centric programmer must control. We are defining Infopipes as a high-level abstraction for information flow, and we are developing a middleware framework that supports this abstraction directly. Infopipes handle the complexities associated with control flow and multi-threading, relieving the programmer of this task. Starting from a high-level description of an information flow configuration, the framework determines which parts of a pipeline require separate threads or coroutines, and handles synchronization transparently to the application programmer. The framework also gives the programmer the freedom to write or reuse components in a passive style, even though the configuration will actually require the use of a thread or coroutine. Conversely, it is possible to write a component using a thread and know that the thread will be eliminated if it is not needed in a pipeline. This allows the most appropriate programming model to be chosen for a given task, and existing code to be reused irrespective of its activity model. The benefit of middleware platforms is that they handle application-independent problems transparently to the programmer and hide underlying complexity. CORBA or RPC, for instance, provide location transparency by hiding message passing and marshalling. Hiding of complexity relieves programmers from tedious tasks and allows them to focus on the important aspects of their applications.. The way that a middleware platform can hide complexity without hiding power is to provide higher-level abstractions that are appropriate for the supported class of applications. In order to choose a suitable abstraction, it is necessary to make some assumptions about the functionality that typical applications require. For example, a common abstraction provided by current middleware is the client-server architecture and request-response interaction, where control flows to the server and back to the client. However, this model is inappropriate for an emerging class of informationflow applications that pass continuous streams of data among producers and consumers. Building these applications on existing middleware requires programmers to specify control-flow behaviors, which are not key aspects of the application. Moreover, existing middleware has inadequate abstractions for specifying data-flow behaviors, including quality of service and timing, which are key aspects of the application.. We propose a new middleware platform for information-flow applications that is based on a producer-consumer architectural model and the Infopipe abstraction. Infopipes simplify the task of building distributed streaming applications by providing basic components such as pipes, filters, buffers, and pumps [2, 28] . Each component specifies the properties of the flows that it can support, including data formats and QoS parameters. When stages of a pipeline are connected, flow properties for the composite can be derived, facilitating the composition of larger building blocks and the construction of incremental pipelines.. The need for concurrently active pipeline stages introduces significant complexity in the area of thread management that can be hidden in the middleware. Hence, our platform frees the programmer from the need to deal with thread creation, destruction, and synchronization. Moreover, the actual control flow is managed by the middleware and is decoupled from the way pipeline components are implemented, be they active or passive objects. We call this approach thread transparency. It simplifies programs and allows reuse of infopipe components. In the same way that RPC systems automatically generate code for parameter marshalling and message handling, our middleware handles thread management and generates glue code that allows Infopipe components to be reused in different activity contexts.. Section 2 describes the Infopipe middleware platform we are developing. Thread transparency is discussed in Section 3. Section 4 describes the current implementation. Related work is summarized in Section 5 before the conclusions in Section 6. Infopipes let us build information flow pipelines from pre-defined components in a way that is analogous to the way that a plumber builds a water flow system from off-the-shelf parts.. The most common components have one input port and one output port. Such pipes can transport information, filter certain information items, or transform the information. Buffers provide temporary storage and remove rate fluctuations. Pumps are used to keep the information flowing. Corresponding to these roles each port has a appropriate polaritiy. Pumps have two ports with positive polarity, one pulling items from upstream and one pushing them downstream. Buffers, in contrast, have two negative ports being pulled from or pushed into. Filters and transformers have two ports of opposite polarity [1, 2] . Sources and sinks have only one port, which can be either positive or negative.. More complex components have more ports. Examples are tees for splitting and merging information flows. Splitting includes splitting an information item into parts that are sent different ways, copying items to each output (multicast), and selecting an output for each item (routing). Merge tees can combine items from different sources into one item or pass on information to the output in the order in which it arrives at any input.. In combining components of a pipeline it is important to check the compatibility of supported flows and to evaluate the characteristics of the composite Infopipe. Each basic or composite Infopipe has a Typespec that describes the flows that it supports. Typespecs provide information about supported formats of data items, interaction properties such as the capability of operating in push or pull mode, and ranges of QoS parameters that can be handled.. Transport protocols can be integrated into the Infopipe framework by encapsulating them as netpipes. These netpipes support plain data flows and may manage low-level properties such as bandwidth and latency. Marshalling filters on either side translate the raw data flow to and from a higher-level information flow. These components also encapsulate the mapping of QoS properties, which is described in more detail in Section 2.4.. In building an Infopipe an application developer needs to combine appropriate filters, buffers, pumps, network pipes, feedback sensors and actuators as well as control components. To facilitate this task, our framework provides a set of basic components to control the timing and a feedback toolkit for adaptation control [8] . Components for processing specific types of flow need to be developed by application programmers, but can easily be reused in various applications. For instance, developers of video on demand, video conferencing, and surveillance tools can all use any available video codec components. Figure 1 shows a simple video pipeline from a source producing compressed data to a display. At the producer side frames are pumped through a filter into a netpipe encapsulating a best-effort transport protocol. The filter drops frames when the network is congested. The dropping is controlled by a feedback mechanism using a sensor on the consumer side. This lets us control which data is dropped rather than suffering arbitrary dropping in the network. After decoding the frames, they are buffered to reduce jitter. A second pump controlling the output timing finally releases the frames to the display. Infopipes provide a framework for building information flow pipelines from components. This abstraction extends uniformly from source to sink. The application controls the setup of the pipeline, configuring its behavior based on QoS parameters and other properties exposed by the components.. The Infopipe platform manages concurrent activity in the pipeline and encapsulates synchronization in high-level communication mechanisms. To specify scheduling policies the application programmer needs only to choose appropriate pumps, which interact with the underlying scheduler and control the actual timing. Neutral components such as filters can be implemenented as active objects, passive consumers, passive producers, or conversion functions, whichever is most suitable for a given task, and existing code can be reused regardless of its implementation style with respect to threading. The Infopipe platform transparently handles creation of and communication between threads and coroutines. This is very much like the way in which CORBA transparently handles marshalling and remote communication.. We have implemented most middleware functionality for local pipelines. Using this platform, we have built several video processing components and configured a simple video player application. The supported functionality is being extended by distributed setup, resource reservations, and feedback mechanisms. Some related work aims at integrating streaming services with middleware platforms based on remote method invocations such as CORBA. The CORBA telecoms specification [25] defines stream management interfaces, but not the data transmission. Only extensions to CORBA such as TAO's pluggable protocol framework [17] allow the efficient implementation of audio and video applications [24] .. One approach for adding quality of service support to CORBA has been introduced by the QuO architecture [35] . It complements the IDL descriptions with specifications of QoS parameters and adaptive behavior in domain specific languages. From these declarative descriptions so called delegates are generated and linked to the client application in a similar way to that in which stubs are generated from an IDL. QuO, however, has not been built for streaming applications and interaction is based on remote method invocations.. A model for specifying flow quality and interfaces has been proposed as part of the MULTE project [29] . Compatibility and conformance rules are used for type checking and stream binding. This model is more formal, but less flexible, than our current approach using Typespecs.. Similarly to Infopipes, the Regis environment [20] separates the configuration of distributed programs from the implementation of the program components. The Darwin language is used to describe and verify the configurations. Components, which execute as threads or processes, are implemented in C ++ with headers generated from Darwin declarations. While the Infopipe implementation described here also uses C ++ for pipeline setup, there are plans for developing an Infopipe Composition and Restructuring Microlanguage [28] .. Open middleware platforms and communications frameworks such as Open-ORB [3] and Bossa Nova [14] offer a flexible infrastructure that supports QoSaware composition and reflection. While these frameworks do not provide specific streaming support, they can serve as a basis for building information flow middleware.. Event-based middleware such as Echo [7, 11] provides a type-safe and efficient way of communicating data and control information in a distributed and heterogeneous environment. A higher-level Infopipe layer can also be built on top of these platforms.. Ensemble [34] and Da CaPo [27] are protocol frameworks that support the composition and reconfiguration of protocol stacks from modules. Both provide mechanisms to check the usability of configurations and use heuristics to build the stacks. Unlike these frameworks for local protocols, Infopipes use a uniform abstraction for handling information flow from source to sink, possibly across several network nodes.. The x-kernel protocol architecture [10] associates processes with messages rather than protocols. In this way, messages can be shepherded through the entire protocol stack without incurring any context switch overhead. We support this thread-per-packet approach for Infopipe components that are implemented in a way that allows direct method calls. Alternatively, developers may choose to program in an thread-like style if this simplifies the program structure.. The Scout operating system [23] generalizes from the x-kernel by combining linear flows of data into paths. Paths provide an abstraction to which the invariants associated with the flow can be attached. These invariants represent information that is true of the path as a whole, but which may not be apparent to any particular component acting only on local information. This idea -providing an abstraction that can be used to transmit non-local information -is applicable to many aspects of information flows, and is one of the principles that Infopipes seek to exploit. For instance, in Scout paths are the unit of scheduling, and a path, representing all of the processing steps along its length, makes information about all of those steps available to the scheduler. This is similar to the way that a section of an Infopipe between two passive components is scheduled by one pump.. Structuring data processing applications as components that run asynchronously and communicate by passing on streams of data items is a common pattern in concurrent programming [e.g. 18]. Flow-Based Programming applies this concept to the development of business applications [22] . While the flowbased structure is well-suited for building multimedia applications, it must be supplemented by support for timing requirements. Besides integrating this timing control via pumps and buffers, Infopipes facilitate component development and pipeline setup by providing a framework for communication and threading.. The VuSystem [19] is a multimedia platform that has several similarities to Infopipes: applications are structured as pipeline components processing information flows, there are interfaces for flow and control communication, and no particular real-time support from the operating system is needed. VuSystem, however, is single-threaded and timing and flow are controlled by the data processing components themselves. Infopipes, in contrast, support multiple threads, preemptive scheduling, and a choice of several programming styles for components and more elaborate consistency checks for pipeline setup.. For constructing streaming applications from components, there are also free and commercial frameworks [21, 30, 32] . GStreamer and DirectShow support setup of local pipelines without timing and QoS control. They provide services to automatically configure components for the conversion of data formats. GStreamer supports component function-style push and thread-style implementations, but does not have pumps to encapsulate timing control. RealSystem is a distributed framework that allows file source components to be used in servers as well as in local clients. The actual transmission is hardcoded into the RealServer and may be configured by adaptation rules. The development of the Infopipe middleware described in Section 2 is still in progress. We have implemented the activity-related functionality discussed in the previous section and part of the Typespec processing. A local video player has been built on top of it.. The platform is built on a message-based user-level thread package [12, 13, 15] implemented in C ++ . Each thread consists of a code function and a queue for incoming messages. Unlike conventional threads, the code function is not called at thread creation time but each time a message is received. After processing a message, the code function returns, but the thread is terminated only when the return value is -1. In this way, code functions resemble event handlers, but may be suspended waiting for other messages or may be preempted. Inter-thread communication is performed by sending messages to other threads, either synchronously if there remains nothing to do for a thread until a reply is received, or asynchronously whenever a reply is not needed immediately, or no reply is required at all. Network packets and signals from the operating system are mapped to messages by the platform, allowing all kinds of events to be handled by a uniform message interface.. The Infopipe platform creates a thread for each pump. If there is no need for coroutines in the section of a pipeline that is controlled by a particular pump, the thread calls the pull methods of all components upstream of the pump, then calls push with the returned item on the components downstream of the pump, and finally returns to the pump, which schedules the next pull. This is the situation in configurations a), b), and c) in Figure 9 . For configurations d), g), and h) there are two coroutines and for configurations e) and f) there are three coroutines associated with the pump. If such coroutines are needed, each of them is implemented by an additional thread of the underlying thread package. Their synchronous interaction is implemented on top of it.. Infopipe push and pull calls between coroutines and control events are mapped to asynchronous inter-thread messages. Although push and pull are synchronous to the Infopipe programmer, synchronous messages cannot be used, because then the thread would not be responsive to control events. Instead, the thread blocks waiting for either a control message or the data reply message. A control event is dispatched to the appropriate handler and then the thread blocks again. After receiving the reply message the code function of the thread is resumed. In this way the middleware implementation establishes synchronous communication of data items between coroutines, while control events can be handled even if the component is blocked in a pull or push.. The thread package supports scheduling by attaching priorities to threads as well as by attaching constraints to messages. In the latter case, the effective priority of a thread is derived by the scheduler from the constraint of the message that the thread is currently processing or, if the thread is waiting for the CPU, on the constraint of the first message in its queue. If no constraint is specified for the message, a static priority is used that is assigned to the thread when it is created. The package provides a priority inheritance scheme that modifies this behavior as necessary to avoid priority inversion, for instance, when a thread receives a message with a higher priority than that of the message it is currently processing.. In the Infopipe framework, message constraints are assigned by the pumps. Messages between coroutines inherit the constraint from the message received by the sending component, applying the constraint to the entire coroutine set. In this way, the pump controls the scheduling in its part of the pipeline across coroutine boundaries.. While other systems for concurrency such as µC ++ provide coroutines directly [5] , this message-based approach facilitates the processing of control events and the scheduling of concurrent activities according to different timing constraints [13] .. The component developer indicates his choice of implementation style by inheriting from the appropriate base class and by overriding a run method for a thread-style component, a push method for a consumer, a pull method for a producer, and a convert method for a function-style component. Additionally, a handler for control events needs to be provided. For pipeline components that change the Typespec of flows the inherited implementation of the type query must be overridden.. Pipelines are configured by a high-level C ++ interface. Composition and start of a simple video player could be implemented by mpeg_file source("test.mpg"); mpeg_decoder decode; clocked_pump pump (30) ; // 30 Hz video_display sink; source>>decode>>pump>>sink; main_channel.send_event(START);. If the components were not compatible, the composition operator >> would throw an exception. This simple example does not compensate for jitter caused by varying decoding times. The last line starts the pipeline by broadcasting a control event, to which the pump reacts. In a video player for complex presentations consisting of several streams, an additional control component would register for global control events such as START and in response dispatch START events to individual pipelines at the start time of their stream relative to the start time of the overall presentation.. A context switch between the user level threads takes about 1 µs; the time for a mere function call is two orders of magnitude shorter. Hence, the approach that we have presented in which threads and coroutines are introduced only when necessary is mostly important for pipelines that handle many control events or many small data items, such as a MIDI mixer. For these applications, and if kernel-level threads are used, allocating a thread for each pipeline component would introduce a significant context switching overhead.

O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O


A set of multi-touch graph interaction techniques Interactive node-link diagrams are useful for describing and exploring data relationships in many domains such as network analysis and transportation planning. We describe a multi-touch interaction technique set (IT set) that focuses on edge interactions for node-link diagrams. The set includes five techniques (TouchPlucking, TouchPinning, TouchStrumming, TouchBundling and PushLens) and provides the flexibility to combine them in either sequential or simultaneous actions in order to address edge congestion. In this research we focus on redesigning node-link graph exploration interfaces for use with multi-touch input. Nodelink graphs are critical in many domains such as the visualization of communication and social networks, and transportation planning. However, they raise specific interaction challenges; for example, congestion of nodes and links [23] , crossing of links [19] , and overview and detail [8] . The use of multi-touch input to address these challenges is promising because the increase in available contact points may support more significant edge detangling and because multi-point input has been shown to enable better collaborative analysis [13] . By applying multi-touch capabilities to graph manipulations we extend the possibilities of graph interaction through combinations of simple actions.. We designed an interaction technique set (IT set) that supports a broad range of interactive analysis styles for nodelink graphs, with a special emphasis on the manipulation of edges. We created a group of interaction techniques by redesigning some existing interactive techniques (EdgePlucking [22] , EdgeLens [23] , and pinning) and combining these with interactive variations of algorithmic edge techniques Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITS'10, November 7-10, 2010, Saarbrücken, Germany. Copyright 2010 ACM 978-1-4503-0399-6/10/11...$10.00. such as edge bundling [12] , and a new technique inspired by the human perception of motion [1] . The design of the IT set is driven by the need to provide flexible and powerful techniques to enable collaborative analysis of graphs, but also to make a system that can reliably interpret simultaneous gestures by one or several people, and enhance capabilities by enabling combinations of gestures. In this paper we present a new multi-touch interaction technique set designed for the exploration of node-link diagrams.. The set was designed to allow analysts to perform wellknown tasks for the domain, but it puts special emphasis on the relatively ignored aspect of edge manipulation. Some techniques that form the set adapt single-point techniques to multi-point interfaces, others are entirely new.. Our technique set was designed to enable the exploration of node-link diagrams through a powerful, memorable, discoverable and efficient combination of techniques; furthermore, the techniques are designed to avoid distinguishability problems that arise in multi-touch, multi-user sets when touches from separate gestures and/or people are interpreted as belonging to one gesture and vice versa.. Also it is our experience that the combination and simultaneous use of these techniques with multi-touch input broadens the interaction possibilities with respect to sequentially constrained single-point interfaces. For example, our interface allows one to separate irrelevant edges with one hand and simultaneously perform an operation on the remaining set of edges with the other hand (e.g., strumming or bundling). This operation feels natural and straightforward with this set, but would require planning and a longer sequence of operations with a single-point interface. While promising in its ability to offer combinations variations both sequentially and simultaneously, this IT set will need to be tested in real world situations with real world tasks and challenges. Two main areas of prior work relate to this research: graph visualization, and multi-touch interaction technique sets.. Interactive Node-link Graph Visualization. Graph layouts have been often applied to real-world data and tasks. Due to the scale and complexity of real world data, these layouts tend to be dense and often contain difficult to read edge configurations [11] . Much previous work on graph layouts has focused on algorithmic approaches to create readable layouts [2] , and on the significance of issues such as edge crossings and bends in layout aesthetics [18] .. Interactive approaches that try to mitigate problems such as edge congestion include zoom and local magnification techniques [4, 10, 20] . These approaches tend to work on the graph as a whole or focus on the nodes, despite the fact that edges had been identified as one of the central aesthetic issues. Recent interaction advances that deal directly with edges in connected diagrams include Wong et al.'s EdgeLens [23] and EdgePlucking [22] , Holten's edge bundling [12] and Moscovich et al.'s link-based navigation techniques [16] all of which serve as inspiration and base for this work.. Multi-touch Interaction Technique Sets. Multi-touch input is an active area of research with a history of several decades [6] ; several studies have shown that enabling multi-touch can be beneficial for interaction [17, 7] . However, most relevant for this work are the recent advances in the development of multi-touch gesture sets. Although based on a large amount of work on sketch recognition and free-hand gestures [15, 3] , it was only during the last few years that HCI research has addressed the construction of gesture sets for multi-touch applications. Wu et al. identify the creation of multi-touch gesture sets as a relevant problem, and provide guidelines to build effective sets [24] . Wobbrock et al. also address the creation of gesture sets, this time through user elicitation [21] , which has also been applied to the creation of diagrams [9] . Our work draws from previous research on multi-touch gesture sets but applies it to edge-based node-link manipulation.

O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O B B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O B B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O


Detecting Holes and Antiholes in Graphs Abstract. In this paper we study the problems of detecting holes and antiholes in general undirected graphs, and we present algorithms for these problems. For an input graph G on n vertices and m edges, our algorithms run in O(n + m 2 ) time and require O(nm) space; we thus provide a solution to the open problem posed by Hayward et al. in [17] asking for an O(n 4 )-time algorithm for finding holes in arbitrary graphs. The key element of the algorithms is the use of the depth-first-search traversal on appropriate auxiliary graphs in which moving between any two adjacent vertices is equivalent to walking along a P 4 (i.e., a chordless path on four vertices) of the input graph or on its complement, respectively. The approach can be generalized so that for a fixed constant k ≥ 5 we obtain an O(n k−1 )-time algorithm for the detection of a hole (antihole resp.) on at least k vertices. Additionally, we describe a different approach which allows us to detect antiholes in graphs that do not contain chordless cycles on five vertices in O(n + m 2 ) time requiring O(n + m) space. Again, for a fixed constant k ≥ 6, the approach can be extended to yield O(n k−2 )-time and O(n 2 )-space algorithms for detecting holes (antiholes resp.) on at least k vertices in graphs which do not contain holes (antiholes resp.) on k − 1 vertices. Our algorithms are simple and can be easily used in practice. Finally, we also show how our detection algorithms can be augmented so that they return a hole or an antihole whenever such a structure is detected in the input graph; the augmentation takes O(n + m) time and space.

O B O B O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Protecting location privacy against inference attacks GPS-enabled mobile devices are a quickly growing market and users are starting to share their location information with each other through services such as Google Latitude. Location information, however, is very privacy-sensitive, since it can be used to infer activities, preferences, relationships, and other personal information, and thus access to it must be carefully protected. We provide a formal definition of location privacy that incorporates an adversary's ability to predict location and discuss possible implementation of access control mechanisms that satisfy this definition. To support our reasoning, we analyze a preliminary data set to evaluate the accuracy of location prediction. Soon the vast majority of mobile devices will be equipped with some form of localization capability; already, most smart phones include a GPS receiver. This has led to the rise of location-based services on a number of mobile platforms, including Symbian, iPhone, and Android. Novel applications, such as Google Latitude [3] have opened up the possibilities of sharing location information with other users [2, 3, 5] .. Location sharing raises significant privacy concerns [1] , since a location, such as a bar or a hospital, can be used to infer a user's personal activities. Therefore, location-sharing services (LSSs) have introduced access controls that allow the user to specify what location data may be shared with whom. For example, Google Latitude allows a user to authorize others' access to his or her location; it also allows a user to enter a decoy location manually. Glympse [2] specifies a time duration during which location information is shared. These interfaces provide coarse-grained controls. Researchers in pervasive computing have proposed more fine-grained access control schemes [4, 6, 8] that make use of context information such as location, time of day, and so on. These rules both better represent the users' actual The solid line represents an actual path of a user visiting a hospital. We assume that the hospital is a private place and the library is a public place. A safe LSS would disclose location points denoted by black nodes.. sharing desires and at least partially automate the decisions to provide seamless integration of location sharing into people's daily lives.. One additional danger of sharing location information, however, is that it can lead to inference of previous or past locations. For example, a person traveling along a trajectory is likely to remain along that path. Things get significantly more complex as more background data is introduced. For example, walking and driving paths follow a predictable pattern, following streets and sidewalks; furthermore, each person exhibits more specific patterns in their activities. For example, Figure 1 shows two potential walking paths leading to a hospital and a library. Given background knowledge, it is possible to infer that a user traveling towards the intersection (black circles) is likely to visit one of these two places. A user who turns left at the intersection (white circles) may then be assumed to be going to the hospital. Therefore, if the user wishes to hide visits to the hospital, it is important to stop revealing his or her location earlier as well.. We, therefore, propose to develop a new access-control scheme that prevents such inference attacks. Our basic approach is to model an adversary as a location predictor that predicts future movements of a target user from his previous movements with certain probabilities. Intuitively, our access control scheme discloses a user's location information only if an unauthorized user cannot predict that the user moves to some private location with a sufficiently high probability. To model outside knowledge, we conservatively assume that the adversary has access to a complete history of the previous movements of the target user. We base our predictions on higher-order Markov models, which have shown to be very good at predicting user location movements [9] . Our preliminary results with actual GPS traces of a single user show that we can predict the user's next movement with the accuracy of 60% using a first-order Markov model, and we can improve the accuracy by 10% by considering multiple previous movements with a higher-order Markov model.. The rest of the paper is organized as follows. Section 2 introduces our system model for LSSs in this paper, and Section 3 describes a location predictor based on the Markov model. We present our preliminary results of experiments in Section 4. We cover related work in Section 5 and conclude in Section 6. Figure 2 shows our system model for LSSs. We assume that a Alice is interested in receiving Bob's location movements. Bob, carrying a GPS-enabled mobile devices periodically sends location-timestamp pairs (loc k , t k ) to the LSS for k ∈ N . In our model, the LSS is completely trusted and receives all of the pairs: We study an issue of inference attacks on GPS traces when we support mobile users' privacy policies for LBSs. We discuss that a traditional access-control mechanism that hides only the private locations of a user is not sufficient since an unauthorized user might be able to infer the user's future movements to private locations from her previous movements. We define an adversary who has access to a mobile user's previous location data as a location predictor based on the Markov model, and then gives location privacy metrics under the presence of such inference attacks. Our preliminary experimental results show that it is possible to predict a mobile user's future location with high accuracy. Location privacy has been studied heavily in the context of the anonymization and obfuscation of location data (See [7] for a comprehensive survey). The focus of research in this area is to ensure that no anonymized and/or obfuscated data is associated with an individual. This inference problem concerning location privacy is different from ours since we consider the inference problem in access-control systems for LSSs, which release identifiable location data of mobile users.

O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O