LOD

Visibility

Culling

and

Occluder

Synthesis

Level-of-detail

occlusion

culling

is

a

novel

approach

to

the

management

of

occluders

that

can

be

easily

integrated

into

most

current

visibility

culling

algorithms.

The

main

contribution

of

this

paper

is

an

algorithm

that

automatically

generates

sets

of

densely

overlapping

boxes

with

enhanced

occlusion

properties

from

non-convex

subsets.

We

call

this

method

occluder

synthesis

because

it

is

not

sensitive

to

the

way

the

objects

are

tesselated

but

to

the

space

enclosed

by

them.

The

extension

of

this

technique

by

allowing

a

bounded

amount

of

image

error

is

also

discussed.

We

show

that

visibility

computations

can

be

based

on

a

multiresolution

model

which

provides

several

representations

of

these

occluders

with

varying

visibility

accuracy.

Our

tests

show

that

occlusion

performance

in

tesselated

scenes

is

improved

severely

even

if

no

imageerror

is

allowed.

Real-time

inspection

of

very

large

models,

with

hundreds

of

thousands

of

faces,

often

surpasses

the

hardware

performance

of

current

high-end

workstations.

This

has

led

to

substantial

research

into

devising

complimentary

software-based

techniques

for

image

acceleration,

including

level-of-detail

rendering,

visibility

culling,

texturing,

compression

and

adaptive

processing..

Level-of-detail

rendering

(LOD-rendering

for

short)

refers

to

the

possibility

of

rendering

objects

that

cover

a

small

portion

of

the

screen

using

a

simpli

ed

version

of

them

instead

of

the

original

representation.

Accurate

representations

are

reserved

for

close,

large

or

important

objects

Visibility

culling

deals

with

the

identi

cation

of

those

portions

of

the

scene

potentially

visible

from

a

dynamic

viewpoint.

At

least

two

su

cient

conditions

for

invisibility

can

be

identi

ed.

View

frustum

culling

discards

the

parts

of

the

scene

that

are

outside

the

eld

of

view.

Occlusion

culling

keeps

the

graphics

hardware

from

drawing

the

parts

that

are

occluded

by

front-end

objects.

The

simplest

form

of

occlusion

culling

is

backface

culling,

which

discards

those

polygons

whose

normal

is

facing

away

from

the

viewer..

All

visibility

culling

algorithms

have

computationally

intensive

pre-processing

stages.

Preprocessing

typically

includes

the

computation

of

some

kind

of

hierarchical

data

structure

to

store

the

scene

and

an

occluder

selection

step

Only

objects

obscured

by

a

single

occluder

will

be

identi

ed

as

not

visible.

The

moment

when

this

identi

cation

takes

place

separates

purely

preprocessing

algorithms

from

those

that

perform

non-trivial

computations

while

navigating

Algorithms

in

the

former

family

subdivide

the

navigational

space

into

cells

of

constant

visibility.

As

part

of

the

pre-process,

the

algorithm

computes

the

potentially

visible

set

(PVS

for

short)

for

each

cell.

Algorithms

in

the

latter

family

compute

the

PVS

at

navigation-time

and

re-compute

it

each

time

the

viewpoint

changes.

In

either

case,

elements

in

the

PVS

are

those

objects

of

the

scene

that

were

not

obscured

by

any

of

the

selected

occluders

when

viewed

from

the

current

cell

or

viewpoint.

The

overestimation

of

the

PVS

is

sensitive

to

the

selected

occluder

set.

Almost

all

occlusion

culling

algorithms

require

the

occluders

to

be

convex

for

both

e

ciency

(faster

algorithms

are

known

for

some

geometric

problems

when

the

input

is

a

convex

polytope)

and

simplicity

(the

visibility

inside

a

polyhedral

volume

can

be

easily

determined

from

the

visibility

at

its

vertices

if

the

occluders

are

convex

In

fact,

current

algorithms

consider

an

object

as

invisible

only

if

it

is

obscured

completely

by

a

single

occluder..

An

object

with

one

half

obscured

by

occluder

C

i

and

the

other

half

occluded

by

occluder

C

j

will

not

be

classi

ed

as

invisible

because

the

involved

computation

is

comparable

with

that

of

working

with

concave

occluders

(recall

that

the

union

of

convex

objects

is

not

necessarily

convex)..

Moreover,

current

occlusion

culling

algorithms

compute

the

occluder

set

by

using

raw

occluder

selection.

They

traverse

the

scene

looking

for

convex

faces

or

polyhedra.

Each

time

a

convex

is

found,

its

potential

occlusion

degree

is

estimated

and

if

it

is

greater

than

a

given

threshold

value,

the

element

(face

or

polyhedra)

is

selected

as

an

occluder.

Since

there

is

no

scene

preprocessing

prior

to

the

occluder

selection,

each

occluder

corresponds

to

an

actual

face

of

the

scene.

Therefore

raw

occluder

selection

is

much

more

sensitive

to

the

boundary

of

the

objects

and

the

way

they

are

tesselated

(e.g.

as

a

triangle

mesh),

than

to

the

underlying

pointsets.

Even

if

a

basic

face

merge

preprocessing

is

carried

out

before

raw

occluder

selection,

the

occluders

remain

on

the

boundary

faces

of

the

scene

geometry

and

objects

with

small

details

and

bevels

on

their

surfaces

make

impossible

to

get

big

convex

polygons

from

them

just

by

merging

coplanar

faces..

For

all

the

above

reasons,

current

occlusion

culling

algorithms

work

with

occluders

much

smaller

than

they

could

actually

be

and

often

need

human

intervention

for

nding

e

ective

occluders..

An

open

problem

addressed

in

this

paper

is

occluder

synthesis.

Given

an

arbitrary

scene

involving

concave

and

highly

tesselated

objects,

our

proposal

is

to

compute

a

minor

set

of

possiblyoverlapping

new

convex

objects

with

enhanced

occlusion

properties

that

will

act

as

occluders

during

the

PVS

computation.

The

computation

of

synthesized

occluders

has

two

main

steps:

aggregation,

which

builds

a

new

discrete

representation

of

the

scene

from

which

both

exact-visibility

and

bounded-error

occluders

can

be

extracted,

and

convex

extraction,

which

computes

a

denselyoverlapping

set

of

boxes

from

the

previous

representation..

Another

consideration

to

be

taken

into

account

in

occluder

synthesis

is

that

in

a

highly

occluded

scene

(as

urban

and

ship

environments),

there

are

usually

narrow

holes

among

objects.

Classical

visibility

culling

algorithms

produce

a

potential

visible

set

that

also

includes

objects

that

are

visible

only

through

these

holes

and

which

have

not

signi

cant

contribution

on

the

nal

image.

The

second

contribution

of

this

paper

is

a

multiresolution

version

of

our

occluder

synthesis

procedure

that

produces

a

simpli

cation

of

the

scene

by

connecting

disconnected

shells.

Obviously,

this

approach

could

produce

visibility

errors

on

the

image

but

as

we

show

they

are

almost

unnoticeable.

Figure

shows

an

overview

of

our

proposed

modi

cation

of

the

classical

LOD

rendering

pipeline

including

occluder

synthesis.

The

visibility

analysis

computation

can

be

done

in

a

pre-process

step

for

each

cell

in

which

the

scene

is

subdivided,

or

at

navigation-time.

During

this

visibility

analysis

step,

and

according

to

the

viewpoint

and

the

error

tolerance,

occluders

of

the

correct

LOD

resolution

are

selected

to

compute

visibility.

Some

highlights

of

our

proposal

are:.

It

always

extracts

convex

parts

of

underlying

polyhedra,

even

if

the

input

model

does

not

provide

polyhedral

information

(as

in

the

case

of

polygon

soups)..

It

can

generate

many

overlapping

convex

occluders

from

one

single

object..

It

can

take

the

whole

scene

as

its

input

data

and

deals

with

polygon

soups

as

e

ectively

as

with

highly

structured

scenes..

The

output

of

the

algorithm

is

restricted

to

boxes,

which

have

a

very

compact

representation

and

whose

shadow

frustra

can

be

trivially

computed

in

time.

Furthermore,

convex

polygons

can

be

extracted

from

these

boxes

in

a

straightforward

manner,

if

required

for

visibility

computations..

The

rest

of

the

paper

is

structured

as

follows.

Section

discusses

previous

research

on

occluder

synthesis.

Section

details

our

contribution

on

loss-less

occluder

synthesis

based

on

the

construction

of

an

intermediate

volumetric

model

of

the

scene

and

on

the

extraction

of

overlapping

boxes.

The

novel

concept

of

LOD

occlusion

culling

is

presented

in

Section

jointly

with

the

analysis

of

image

error.

Finally,

Sections

and

show

some

results

and

summarize

some

conclusions

and

future

work.

We

have

implemented

all

the

algorithms

presented

in

this

paper

and

tested

them

on

several

publicly

available

models.

Figure

shows

the

results

of

our

lossy

version

of

the

occluder

synthesis

algorithm

on

a

model

of

the

St.

Pauls

cathedral

from

faces).

On

both

cases

the

covering

was

above

and

the

maximum

error,

measured

as

the

deviation

from

the

occluder

to

the

original

surface

was

less

than

and

respectively..

We

did

some

experimentation

to

check

the

dependence

of

our

method

on

the

number

of

occluders

and

seeds.

In

order

to

get

an

accurate

evaluation

of

the

goodness

of

the

synthesis

algorithm,

we

ran

the

examples

with

a

naive

version

of

the

sort

and

cut

lter

which

only

considers

the

di

erence

between

the

volume

of

the

box

whose

contribution

is

being

evaluated

and

the

volume

of

its

maximum

intersection

with

the

boxes

selected

previously.

Figure

shows

that

twelve

occluders

and

twice

that

number

of

seeds

su

ce

to

cover

signi

cant

portions

(more

than

of

the

St.

Pauls

cathedral.

And

Figure

reveals

that

regardless

of

the

number

of

occluders

adopted,

it

is

easy

to

achieve

coverings

above

with

a

few

seeds.

Figure

shows

graphically

the

di

erential

region

using

the

lossy

occluders

generated

from

the

octree

of

the

St.

Pauls

model

(e

t

The

red

pixels

in

Figure

(b)

correspond

to

the

screen

region

covered

by

the

occluders

but

not

covered

by

the

original

object.

Although

any

red

pixel

can

be

erroneous,

usually

only

a

little

subset

of

these

points

are

actually

wrong

because

for

a

visible

polygon

to

be

considered

as

not

visible,

its

screen

projection

must

be

completely

enclosed

in

the

red

and

blue

area..

In

order

to

measure

the

actual

improvement

on

occlusion

power,

we

tested

our

method

on

a

highly

occluded

environment.

We

chose

a

publicly

available

city

model

Figure

from

with

triangles.

We

took

samples

along

a

path

crossing

the

city

and

culled

the

scene

using

raw

and

loss-less

synthesized

occluders

separately.

For

each

occluder

set,

and

for

each

sample,

occluders

that

were

inside

a

xed-size

box

centered

at

the

sample

viewpoint

were

selected.

The

box

was

su

ciently

big

to

include

all

the

signi

cant

occluders.

As

a

matter

of

fact,

it

included

many

more:

thousands

of

triangles

in

the

case

of

raw

occluders

and

less

than

fty

in

the

case

of

synthesized

ones.

The

simulation

algorithm

then

chose

the

occluders

with

better

occlusion

behavior

and

discarded

the

rest.

The

sets

of

visible

polygons

of

the

remaining

occluders

were

then

mixed

to

obtain

a

potentially

visible

set

of

the

current

sample.

Note

that

this

occluder

selection

step

does

certainly

select

the

best

possible

triangles,

so

the

simulation

is

a

little

biased

in

favor

of

raw

occlusion.

Visibility

culling

algorithms

do

not

have

the

time

to

do

such

an

extensive

search

at

each

viewpoint.

Figure

shows

the

di

erences

between

the

number

of

culled

polygons

using

only

raw

occluders

and

only

synthesized

occluders.

It

is

clearly

favorable

to

the

use

synthesized

occluders.

The

minimum

and

maximum

values

of

the

di

erence

between

synthesized

and

raw

were

and

polygons,

respectively.

Its

mean

was

of

polygons

with

a

standard

deviation

of

Our

method

allowed

an

extra

culling

of

a

of

the

scene

and

enhanced

occlusion

signi

cantly

at

the

vast

majority

of

the

samples..

Running

times

of

the

convex

extraction

algorithm

on

the

St.

Pauls

and

the

city

model

are

shown

in

Figure

The

octree

construction

of

the

St.

Pauls

cathedral

and

the

city

model

took

and

minutes

respectively

on

a

MHz

MIPS

processor.

We

have

introduced

LOD

occlusion

culling,

a

new

concept

that

accelerates

navigation

in

complex

scenes

and

that

can

be

incorporated

into

current

navigation

frameworks

that

use

visibility

culling..

The

experimental

results

show

an

improvement

of

visibility

computations,

the

generation

of

more

accurate

PVS,

and

high

quality

images

without

visibility

error

(loss-less

visibility)

or

with

a

tted

image

error

(lossy

visibility)

by

means

of

two

intuitive

parameters

indicating

the

maximum

thickness

and

area

of

the

error

region..

We

have

also

developed

a

new

algorithm

for

the

generation

of

synthesized

occluders.

The

algorithm

has

two

main

steps:

aggregation

and

convex

extraction,

and

it

guarantees

the

visibility

validity

of

the

output

occluders.

It

generates

sequences

of

densely

overlapping

boxes

that

need

not

to

be

inside

the

original

concave

pointset.

Each

of

these

two

features

enhances

occlusion

performance.

Modifying

the

aggregation

step,

the

algorithm

can

be

used

to

obtain

multiresolution

occluder

sets.

This

algorithm

bene

ts

not

only

from

complex

and

concave

regions,

but

also

from

almost

opaque

ones,

thus

greatly

accelerating

the

visualization..

The

occluder

selection

step

also

bene

ts

from

the

use

of

synthesized

occluders.

Dynamic

occlusion

algorithms

restrict

the

number

of

selected

occluders

used

at

each

viewpoint

to

avoid

overhead.

But

as

the

number

of

elements

of

the

synthesized

set

is

many

orders

of

magnitude

smaller

than

the

number

of

polygons

in

the

input,

the

selection

of

occluders

at

each

viewpoint

is

faster.

And

more

occluders

can

be

taken

into

consideration

without

incurring

in

excessive

overhead..

Our

future

work

includes

the

integration

of

LODocclusion

into

our

visibility

algorithm

The

occluder

synthesis

algorithm

will

allow

the

generation

of

level-of-detail

hierarchies

of

synthesized

occluders

with

varying

degrees

of

visibility

accuracy.

As

in

LOD

rendering,

image

delity

can

be

traded

for

a

higher

frame

ratio.

We

plan

to

use

lossy

occluders

for

determining

both

completely

visible

and

hardly

visible

objects,

which

can

be

displayed

using

a

coarse

representation

of

them..

Storing

a

high-resolution

octree

is

very

memoryconsuming.

Fortunately,

it

is

not

necessary

to

have

simultaneously

all

the

terminal

nodes

of

the

octree

subdivided

until

the

maximum

subdivision

level.

During

the

top-down

construction

of

the

octree,

it

can

be

decomposed

into

strongly

connected

subsets

which

can

be

re

ned

separately.

Occlusion

culling

literature

has

focused

mainly

on

the

type

of

spatial

data

structures

and

on

the

method

of

nding

whether

a

volume

is

obscured

(there

is

a

survey

on

this

eld

elsewhere

Though

current

visibility

culling

algorithms

are

known

to

be

very

sensitive

to

the

boundary

representation

of

the

scene,

very

little

e

ort

has

been

devoted

to

lessen

this

shortcoming..

Recently,

Law

and

Tan

proposed

a

new

framework

that

integrated

simpli

cation

and

occlusion

culling

techniques.

In

their

proposal,

occluders

are

synthesized

in

three

steps.

First,

a

potential

object

is

selected

from

the

scene

and

its

geometry

is

decimated

using

current

simpli

cation

algorithms..

In

the

second

step,

and

in

order

to

preserve

occlusion

properties,

those

vertices

of

the

decimated

object

that

lie

outside

the

original

one

are

translated

till

bounded

by

it.

Finally,

edges

in

the

decimated

model

are

also

perturbed

so

to

ensure

convexity.

Their

tests

showed

the

algorithm

to

be

able

to

generate

synthesized

convex

occluders

for

signi

cant

frame-ratio

improvements..

However,

Law

and

Tan's

algorithm

has

some

drawbacks.

As

noted

by

the

authors,

validity

of

the

resulting

occluder

is

not

always

ensured

by

the

occlusion

synthesis

procedure

as

the

perturbation

stages

can

yield

objects

with

no

occlusion

power.

Furthermore,

the

algorithm

outputs

one

single

occluder

for

each

input

and

it

is

sensitive

to

the

tessellation

of

the

input

objects..

Klosowski

and

Silva

presented

a

di

erent

but

also

lossy

approach

that

does

not

compute

PVS

explicitly.

Instead,

it

is

based

on

computing

a

priority

order

for

the

polygons

that

maximizes

the

likelihood

of

rendering

visible

polygons

before

occluded

ones.

The

user

sets

a

bound

on

the

number

of

polygons

to

render

in

each

frame,

and

the

algorithm

renders

those

with

the

maximum

visibility

likelihood.

As

a

result

of

the

polygon

bound,

it

is

usual

to

obtain

errors

in

the

nal

image

due

to

the

absence

of

visible

objects

and

the

presence

of

obscured

ones.

A

brief

survey

on

related

problems

seems

to

point

that

nding

a

global

optimal

to

this

problem

is

by

no

means

an

easy

task.

For

instance,

the

potatopeeling

problem

(a.k.a.

the

convex

skull

problem)

consists

in

computing

the

largest

convex

contained

in

a

given

n-vertex

object.

It

has

been

studied

both

in

and

with

several

convex

shapes.

Arbitrary

convex

shapes

lead

to

prohibitive

solutions

even

in

cases

Better

solutions

are

known

for

restricted

convex

objects.

The

axis-parallel

rectangle

of

largest

area

inside

a

general

polygon

can

be

found

in

O(n

log

In

the

above

inclusion

approaches,

the

output

model

consists

always

in

a

single

convex

object.

But

in

order

to

improve

occlusion

as

much

as

possible,

we

should

extract

several

convex

objects

from

every

input

component.

The

partitioning

problem

(divide

a

given

object

into

a

disjoint

set

of

simple

components)

has

turned

out

to

be

generally

well

solvable.

In

a

general

polyhedron

can

be

partitioned

in

O(N

convex

parts

in

O(nN

time,

where

n

and

N

denote

the

numbers

of

edges

and

re

ex

vertices,

respectively

Unfortunately,

partitioning

tends

to

produce

small

convex

parts

which

are

not

useful

as

occluders..

For

the

covering

problem

(where

overlapping

is

allowed),

however,

no

satisfactory

solutions

are

known.

Even

for

the

case,

nding

the

minimal

covering

is

an

NP-complete

problem

The

only

polynomial

time

algorithms

known

are

for

covering

orthogonal

polygons.

Finding

the

minimum

covering

of

a

simply

connected

(i.e.

without

holes)

n-vertex

orthogonal

polygon

takes

O(n

log

n+nm)

time

where

m

is

the

number

of

edges

in

the

visibility

graph

that

are

either

horizontal,

vertical

or

form

the

diagonal

of

an

empty

rectangle.

However,

it

is

restricted

to

non-piercing

(and

thus

smaller)

rectangles.

Better

solutions

are

known

for

rectangles

and

squares.

Covering

a

simply-connected

orthogonal

polygon

with

a

minimum

number

of

squares

can

be

done

in

O(n

k)

time,

k

being

the

number

of

output

squares

Unfortunately,

these

techniques

have

not

been

extended

to

Moreover,

the

covering

of

potential

occluders

does

not

share

the

optimality

criteria

of

classic

covering

algorithms.

For

occlusion

culling,

we

are

interested

in

as

large

as

possible

convex

components,

while

classic

covering

literature

looks

for

low-density

(i.e.

hardly

overlapping)

coverings.

O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O


TURINstream: A Totally pUsh, Robust, and effIcieNt P2P Video Streaming Architecture Abstract-This paper presents TURINstream, a novel P2P video streaming architecture designed to jointly achieve low delay, robustness to peer churning, limited protocol overhead, and quality-of-service differentiation based on peers cooperation. Separate control and video overlays are maintained by peers organized in clusters that represent sets of collaborating peers. Clusters are created by means of a distributed algorithm and permit the exploitation of the participant nodes upload capacity. The video is conveyed with a push mechanism by exploiting the advantages of multiple description coding. TURINstream design has been optimized through an event driven overlay simulator able to scale up to tens of thousands of peers. A complete prototype of TURINstream has been developed, deployed, and tested on PlanetLab. We tested our prototype under varying degree of peer churn, flash crowd arrivals, sudden massive departures, and limited upload bandwidth resources. TURINstream fulfills our initial design goals, showing low average connection, startup, and playback delays, high continuity index, low control overhead, and effective quality-of-service differentiation in all tested scenarios. P EER-TO-PEER (P2P) streaming has proved to be a viable and efficient approach to support the broadcasting of live or prerecorded media over the Internet. Peers contribute their uplink bandwidth by forwarding content to their connected peers. Global available resources (aggregate uplink bandwidth) grow as the number of peers increases, thus making the approach potentially able to scale to a large number of users.. Nowadays P2P streaming architectures can be broadly classified in two classes: tree-based (e.g., [1] - [3] ) and mesh-based (e.g., [4] , [5] ). In tree-based approaches (also termed as pushbased), the overlay is composed of several diverse trees that are used to multicast the video packets to the peers. The tree based approach can easily exploit multiple description coding (MDC) [6] , [7] , where the video is encoded onto a set of independent descriptions to be pushed across separate trees. The mesh-based approach is based on file swarming mechanisms where participating peers form a randomly connected mesh and use gossip-like protocols for the creation and the administration of the overlay; buffer maps are frequently exchanged among the peers to signal the available video chunks. Each peer aims at retrieving the video stream by explicitly requesting its missing chunk (pull-based approach). Recently, [8] provided a comparison between the two approaches as well as the identification of similarities and differences. Push/tree P2P system generally guarantees lower startup and playback delays, which depend on the topological features of the overlay. Despite these advantages, the results in [8] indicate that the mesh-based approach generally exhibits a superior performance over the tree-based approach. The main shortcomings of the tree-based systems are reported to be the large overhead due to the organization of the tree topology, the difficulty to respond to the dynamics of the peers, and the suboptimal exploitation of the upload bandwidth. Nevertheless, the mesh-based approach suffers from other problems; although several mesh-based mechanisms that deployed MDC have been proposed [9] , [10] , the use of MDC techniques is not straightforward since one has to guarantee that the descriptions follow independent distribution trees; indeed the average decoded video quality is maximized if one assumes independent losses of the descriptions. Access control policies and service differentiation are difficult to implement. Furthermore, mesh-based approaches suffer from the tradeoff between control overhead and delay. In fact, to minimize delay, peers must notify their neighbors of available packets as frequently as possible, thus resulting in high control overhead. On the other hand, to reduce control messages, notifications must be aggregated over time, thus making the delay higher. An attempt to bring the benefits of tree-based approaches in the mesh-based scenario is presented in [11] . The authors propose a pull-push hybrid protocol where packets are pushed along the trees, dynamically created according to a pull-based protocol. In [12] , another mesh-based system that uses a push data dissemination approach is presented.. In the present paper, we take a different viewpoint. In particular, we leverage on the topological guarantees of a tree structured overlay yielding short delays and allowing to exploit MDC video, while improving on the side of robustness to peer dynamics and bandwidth utilization. These two latter objectives have been addressed with the introduction of a small set of fully connected collaborating peers forming what we termed a cluster. The contributions of this work are the design, simulation, and implementation of TURINstream, a novel multi-tree P2P streaming architecture where tree nodes are represented by clusters of peers. The concept of cluster has already been introduced in some previous research; as opposed to such related works, briefly recalled in Section V, where the clusters are organized in a hierarchical structure, the overlay topology proposed in this paper is very flexible and permits a more efficient exploitation of the resources. Separation of data and control overlays, cluster-based organization, exploitation of MDC techniques, and tree-based control overlay are all exploited in the TURINstream design to achieve:. • high degree of robustness providing playback continuity in order to minimize freezing or blackout periods despite peers departures, failures, and churning. When a peer leaves or crashes, the cluster does not disconnect from the network and, while the cluster is being repaired, the video content keeps reaching the cluster members. At the same time, the probability of a joint failure of all the peers supplying the video to a cluster becomes negligible as the number of multiple trees increases; • low connection, startup, and playback delays. These metrics all depend on the depth of the control tree whose scaling is logarithmic with respect to the total number of participating peers; • high scalability and low control overhead to allow a very large number of users to join the application; • rewards for altruistic (cooperative) users and penalties for selfish (un-cooperating) ones. In fact, the join algorithm we define allows for the adoption of access control policies in terms of bandwidth and peer fairness; in other terms, differentiated quality-of-service policies can be used to pay back more collaborative peers, i.e., peers that share a higher uplink bandwidth. The TURINstream design has been aided by an event driven overlay simulator able to scale up to tens of thousands of users. Finally, the major contribution of the paper is the development of a complete prototype of TURINstream which has allowed us to deploy and test the application on PlanetLab. The TURINstream architecture has been used to stream H.264/AVC video encoded with the MDC technique presented in [7] . Such MDC codec is compliant with H.264/AVC and has allowed us to use the standard RTP [13] protocol to transport the video packets and RTSP/SDP [14] , [15] protocols to signal the video session. We tested our prototype under varying degree of peer churn, flash crowd arrivals, sudden massive departures, and limited upload bandwidth resources. The main finding is that TURINstream fulfills our initial design goals, showing low average connection, startup, and playback delays, high continuity index, low control overhead, and effective quality-of-service differentiation in all the scenarios we addressed.. The outline of the paper is as follows: Section II presents the protocol operations, Section III describes the simulator we developed, while Section IV presents the TURINstream prototype and the tests performed on PlanetLab. Finally, Section V discusses similar approaches in the literature while Section VI concludes the paper with the outline of some of the future developments. In this paper, we presented TURINstream, an architecture for P2P-based video streaming. It exploits separation of data and control overlays, cluster-based organization, MDC video coding, and tree-based control overlay to achieve a high degree of robustness, low connection, startup and playback delays, high scalability, low control overhead, and differentiated quality-of-services for different classes of users.. TURINstream design has been optimized through an event driven overlay simulator able to scale up to tens of thousands of users. A complete prototype of TURINstream has been developed, deployed, and tested on PlanetLab under varying degree of peer churn, flash crowd arrivals, sudden massive departures, and upload bandwidth limitation. Moreover, the prototype has been equipped with state-of-the-art MDC video coding, allowing us to test the streaming protocol under realistic video traffic. The main finding is that TURINstream fulfills our initial design goals and proved to be competitive with other architectures.. Future efforts will be devoted to devise more efficient exploitation of all the available upload bandwidth of participating users as well as to cope with the delay in adaptation to bandwidth fluctuations. Moreover, network aware algorithms will be considered for optimal cluster formation. The TURINstream architecture encompasses two key elements, namely the MDC video distribution on a multi-tree topology and the organization of the peers in clusters. In the past, many multi-tree solutions have been proposed where the node of the trees are the peers as opposed to the TURINstream clusters. Here we mention CoopNet [1] and Splitstream [2] that are two well-known approaches. In CoopNet, a set of random trees is built by using a centralized approach. All the peers joining and leaving the overlay exploit a resourceful server node that coordinates the overlay construction and optimization. Because of the absence of a distributed and local algorithm for the construction of the trees, the overlay reorganization in response to peer crashes or bandwidth fluctuation can be costly. Moreover, CoopNet exploits a high number of video descriptions (up to 16) to make the offered service reliable. On the other hand, SplitStream represents a viable distributed approach to the construction of the multi-tree; in the latter case, the use of interior-node-disjoint trees is proposed, where each peer is an interior node in at most one tree. This solution limits the impact of the departure of a peer to a single tree. Nevertheless, it has the negative drawback that every peer is a leaf node in the distribution of out of the descriptions. In presence of churn, the quality-of-service in terms of both delays and continuity decreases with the height of the tree. This fact heavily limits the overall quality offered to each peer.. A few peer-to-peer architectures have introduced the concept of clustering in recent years as well. The Zigzag [22] protocol, which is an enhanced version of NICE [23] , builds a single clustered multicast tree. As opposed to TURINstream in ZigZag, all the peers belong to a cluster at level 0, whose cluster-head belongs to a cluster at level 1; then level 1 cluster-head belong also to a cluster in level 2 and so on. As a consequence, in [22] , a top level cluster-head belongs to all the clusters; on the other hand, in TURINstream, each cluster-head has only a local knowledge of the overlay making it more robust to peer churn. In particular, TURINstream repair operations are quicker since the topology is more flexible and does not require split and merging of clusters as in ZigZag. Moreover, TURINstream uses a multi-tree for content distribution whereas ZigZag employs a single multicast tree. Finally, in [22] , simulation-based results without the exploitation of MDC have been reported. Reference [24] presents another layered cluster architecture very similar to ZigZag from the point of view of the overlay topology. The focus of this work is on optimal exploitation of peer bandwidth whereas the proposed architecture, as most of the structured topologies, is not resilient to peer churn and departures. In particular, in dHCPS, the cluster-heads represent a single point of failure. Furthermore, experimental results in [24] have been worked out for small and stable systems (80 peers on PlanetLab) and considering a video server equipped with 3.2 Mbps upload bandwidth capacity; actual scalability and resilience are thus a concern for this architecture.. Another goal pursued in this work is service differentiation based on peers contribution so as to promote altruistic behaviors. In [25] , a centralized solution that organizes peer in a direct acyclic graph and adapts the video streaming rate based on the peer contributions is proposed and analyzed by simulation. The system proposed in [25] requires a parent node to transcode incoming stream to serve its children at different rates. Transcoding is more flexible in terms of the achievable rates but is highly demanding in terms of computational resources with respect to the use of MDC, where peers can simply drop some description to shape the rate.

O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Synchronisation

effects

on

the

behavioural

performance

and

information

dynamics

of

a

simulated

minimally

cognitive

robotic

agent

Abstract

Oscillatory

activity

is

ubiquitous

in

nervous

systems,

with

solid

evidence

that

synchronisation

mechanisms

underpin

cognitive

processes.

Nevertheless,

its

informational

content

and

relationship

with

behaviour

are

still

to

be

fully

understood.

In

addition,

cognitive

systems

cannot

be

properly

appreciated

without

taking

into

account

brain-bodyenvironment

interactions.

In

this

paper,

we

developed

a

model

based

on

the

Kuramoto

Model

of

coupled

phase

oscillators

to

explore

the

role

of

neural

synchronisation

in

the

performance

of

a

simulated

robotic

agent

in

two

different

minimally

cognitive

tasks.

We

show

that

there

is

a

statistically

significant

difference

in

performance

and

evolvability

depending

on

the

synchronisation

regime

of

the

network.

In

both

tasks,

a

combination

of

information

flow

and

dynamical

analyses

show

that

networks

with

a

definite,

but

not

too

strong,

propensity

for

synchronisation

are

more

able

to

reconfigure,

to

organise

themselves

functionally

and

to

adapt

to

different

behavioural

conditions.

The

results

highlight

the

asymmetry

of

information

flow

and

its

behavioural

correspondence.

Importantly,

it

also

shows

that

neural

synchronisation

dynamics,

when

suitably

flexible

and

reconfigurable,

can

generate

minimally

cognitive

embodied

behaviour.

Oscillatory

neural

activity

is

closely

related

to

cognitive

processes

and

behaviour

(Engel

et

al.

Buzsáki

More

specifically,

it

has

been

claimed

that

the

flexible

synchronisation

of

firing

neurons

is

a

fundamental

brain

mechanism

(von

der

Malsburg

Singer

Singer

Womelsdorf

et

al.

Synchronisation

may

mediate

the

interactions

between

neurons

and

be

an

active

mechanism

of

large-scale

integration

of

neuronal

assemblies,

impacting

on

motor

control

and

cognitive

performance

(Hatsopoulos

et

al.

Varela

et

al.

Jackson

et

al.

Moreover,

a

range

of

pathological

states

are

related

to

abnormal

neuronal

synchronisation

regimes

(Glass

Brown

Arthuis

et

al.

Whereas

increasingly

sophisticated

computational

and

imaging

techniques

help

us

to

observe

and

record

from

different

physiological

aspects

of

the

nervous

system,

a

great

part

of

the

challenge

lies

in

the

comprehension

of

this

avalanche

of

data

and

its

relationship

with

behaviour

(Chialvo

In

addition,

the

brain

presents

spontaneous

electrochemical

activity

that

is

constantly

shaped

by

the

body's

constraints

and

time-varying

environmental

stimuli

(Katz

The

brain,

therefore,

not

only

processes

information

but

also

produces

it,

and

cognitive

phenomena

are

a

product

of

brain-body-environment

interactions

(Stewart

et

al.

Situated

artificial

agents,

in

this

sense,

provide

an

appropriate

experimental

scenario

to

study

the

principles

of

intelligent

behaviour

(Boden

There

is

a

rich

literature

exploring

the

relationship

between

neural

synchronisation

and

complex

motor

control

in

embodied

(robotic)

rhythmical

behaviours

(Taga

Ijspeert

et

al.

Pitti

et

al.

where

synchronisation

appears

as

a

more

intuitive

underlying

mechanism;

however,

to

date

there

has

been

very

little

research

on

the

wider

issues

of

neuronal

synchronisation

in

the

generation

of

embodied

cognitive

behaviours.

Therefore,

using

concepts

from

Evolutionary

Robotics

(Harvey

et

al.

Floreano

and

Mattiussi

Floreano

and

Keller

we

explore

the

role

of

synchronisation

in

the

performance

of

a

simulated

robotic

agent

during

the

execution

of

two

different

minimally

cognitive

tasks:

the

first,

a

categorical

perception

task

(Beer

Izquierdo

Dale

and

Husbands

in

which

the

robot

has

to

discriminate

between

moving

circles

and

squares;

the

second,

an

orientation

task,

where

the

robotic

agent

has

to

approach

moving

circles

with

both

normal

and

inverted

vision,

adapting

to

both

conditions.

These

tasks

were

chosen

for

being

currently

regarded

as

benchmarks

in

the

evolutionary

robotics

and

adaptive

behaviour

communities,

with

categorical

perception

underpinning

cognitive

systems

(Harnad

The

study

reported

here

is

intended

to

shed

some

light

on

the

role

of

neural

synchronisation

in

simple

embodied

cognitive

behaviours.

More

specifically

it

aims

to

test

whether

different

degrees

of

coupling

in

the

agent's

oscillatory

neural

network,

which

will

encourage

more

or

less

synchrony

in

the

network

dynamics,

have

an

effect

on

the

performance

of

the

agent

and

determine

if

there

are

circumstances

in

which

more

(or

less)

synchrony

is

better

suited

to

the

generation

of

adaptive

behaviour

in

the

context

of

the

tasks

studied..

The

neuronal

model

employed

is

based

on

the

Kuramoto

Model

of

coupled

phase

oscillators

(Kuramoto

which

has

been

extensively

studied

in

the

Statistical

Physics

literature,

with

recent

applications

in

a

biological

context

due

to

its

relatively

simple

and

abstract

mathematical

formulation

yet

complex

activity

that

can

be

exploited

to

clarify

fundamental

mechanisms

of

neuro-oscillatory

phenomena

without

making

too

many

a

priori

assumptions

(Ermentrout

and

Kleinfeld

Cumin

and

Unsworth

Kitzbichler

et

al.

Breakspear

et

al.

The

model

explicitly

captures

the

phase

dynamics

of

units

that

alone

have

spontaneous

oscillatory

activity

and

once

connected

can

generate

emergent

rhythmic

patterns.

Its

synchronisation

regime

can

be

adjusted

by

one

parameter

(Strogatz

suiting

our

study,

whilst

also

avoiding

any

problems

in

obtaining

phase

information

[an

issue

in

other

models

which

consider

frequency

and

amplitude

dynamics

(Pikovsky

et

al.

Phase

relationships

contain

a

great

deal

of

information

on

the

temporal

structure

of

neural

signals,

are

associated

with

cognition

and

relate

to

memory

formation

and

retrieval

(Li

and

Hopfield

Izhikevich

Varela

et

al.

Kunyosi

and

Monteiro

Furthermore,

it

has

been

shown

that

firing

and

bursting

neurons

can

be

modelled

as

oscillators

(Murray

Hence,

the

Kuramoto

Model

is

highly

relevant,

at

a

certain

level

of

abstraction,

to

modelling

neural

mechanisms

underlying

adaptive

and

cognitive

behaviours..

Analysis

of

results

is

centred

on

how

the

information

dynamics

between

the

nodes

of

the

network,

the

agent's

body

and

the

environment

vary

depending

on

the

current

synchronisation

status

of

the

system

and

how

this

is

reflected

in

the

behaviour

being

displayed.

Information

Theory

provides

a

framework

for

quantifying

and

emphasizing

the

nonlinear

relationships

between

variables

of

the

system,

hence

its

suitability

in

Biology

and

Robotics

studies

(Rieke

et

al.

Rolls

and

Treves

Lungarella

and

Sporns

In

this

context,

agent-environment

systems

pose

extra

challenges

in

devising

and

interpreting

a

sensible

measurement

of

information

flow,

for

they

normally

have

noisy

and

limited

data

samples,

asymmetrical

relationships

among

elements

of

the

system

and

temporal

variance

(i.e.

sensory

and

motor

patterns

may

vary

over

time).

Transfer

Entropy

(TE),

in

this

scenario,

is

suggested

as

a

suitable

and

robust

informationtheoretic

tool

(Lungarella

et

al.

and

has

also

been

applied

to

investigate

real

neural

assemblies

and

other

neuroscience

problems

(Borst

and

Theunissen

Gourévitch

and

Eggermont

Buehlmann

and

Deco

McDonnell

et

al.

Vicente

et

al.

it

will,

thus,

be

used

in

our

analysis..

The

paper

is

organised

as

follows:

the

next

section

presents

some

theoretical

background

to

the

main

concepts

explored

in

this

study:

oscillators

and

synchronisation,

focusing

on

the

Kuramoto

Model,

evolutionary

robotics

(ER)

and

the

two

minimally

cognitive

tasks

studied,

and

Information

Theory

in

an

agent-environment

context,

describing

TE.

We

then

present

the

methods

adopted

to

develop

the

experiments

and

analysis,

covering

the

ER

framework,

the

details

of

the

active

categorical

perception

and

orientation

under

normal

and

inverted

vision

tasks,

a

description

of

the

genetic

algorithm

used

to

optimize

the

parameters

of

the

system,

and

concluding

with

details

of

the

time-series

analysis

using

TE.

Following

this,

we

present

the

results,

which

show

a

difference

in

performance

and

behaviour

depending

on

the

synchronisation

regime

of

the

network,

relating

the

observed

sensorimotor

strategies

with

the

dynamics

and

the

information

flow

of

the

system.

The

paper

closes

with

a

discussion

on

the

results

obtained

and

future

work

proposals.

Current

research

on

neurophysiological

processes

shows

that

oscillatory

neural

activity

is

closely

related

to

cognition

and

behaviour,

with

synchronisation

mechanisms

playing

a

key

role

in

the

integration

and

functional

organization

of

different

cortical

areas

(Varela

et

al.

Engel

et

al.

Womelsdorf

et

al.

Nevertheless,

its

informational

content

and

relationship

with

behaviour-and

hence

cognitionare

still

to

be

fully

understood

(Rieke

et

al.

Mazzoni

et

al.

Deco

et

al.

Flanders

In

this

context,

following

an

Evolutionary

Robotics

approach,

we

evolved

simulated

robotic

agents

controlled

by

a

spontaneously

rhythmic

network

of

coupled

phase

oscillators,

showing

that

their

performances

in

variations

of

a

categorical

perception

task

depend

on

the

synchronisation

regime

of

the

network.

This

was

also

shown

to

be

the

case

for

an

orientation

task

under

normal

and

inverted

vision,

but

the

dependence

on

the

synchronisation

regime

was

quite

different

from

in

the

first

task.

The

analysis

focused

both

on

a

behavioural

level

description,

investigating

the

agent/object

trajectories,

and

on

a

mechanism

level

description,

exploring

the

dynamics

and

the

information

transfer

properties

within

and

between

the

agent

and

the

environment..

The

design

of

the

experimental

framework

attempted

to

capture

relevant

properties

of

its

biological

counterparts

that

would

suit

the

aims

of

our

study.

The

network

model

is

inspired

by

the

Kuramoto

Model

of

coupled

phase

oscillators

whilst

the

robotics

approach

explores

the

enactive

nature

of

sensory

and

cognitive

processes.

Our

focus

is

on

three

aspects:

the

information

flow

between

sensors

and

motors,

capturing

the

body-environment

interaction;

the

average

flow

between

nodes

of

the

network,

stressing

the

internal

communication;

and

the

evolution

in

time

of

the

average

net

flow

from

a

given

node

to

all

its

counterparts,

revealing

at

various

moments

of

the

task

a

higher-lower

participation

in

the

overall

information

dynamics..

The

first

experiment

explored

the

fitness

variation

of

evolved

individuals

under

different

network

couplings

K

for

a

categorical

perception

task.

The

results

showed

that

there

is

a

statistically

significant

difference

in

performance

for

different

strengths

of

coupling,

with

the

best

fitness

evolution

performance

obtained

when

K

K

c

the

critical

value.

Although

the

term

'critical'

here

is

not

directly

related

to

its

meaning

for

self-organised

criticality

(Beggs

previous

work

(Kitzbichler

et

al.

showed

that

the

Kuramoto

Model

presents

properties

of

critical

systems

(e.g.

greater

information

transmission,

storage,

sensitivity

to

external

stimulus)

when

its

coupling

is

tuned

to

the

critical

value

(Eq.

Our

results

confirm

this

for

the

categorical

perception

task..

For

the

agent

with

best

performance

(K

K

c

the

dynamical

analysis

shows

that

there

is

a

very

rich

phase

behaviour

in

the

critical

coupled

network,

with

nodes

oscillating

at

a

variety

of

frequencies,

occasionally

synchronising

and

forming

independent

functionally

connected

clusters.

In

the

Effective

TE

(E

T

E)

analysis,

we

notice

a

higher

flow

magnitude

in

the

sensorimotor

loop

time

series

(ETE

sm

and

ETE

ms

which

has

a

behavioural

correspondence

and

relates

to

sharper

adjustments

in

the

agent's

trajectory,

suggesting

that

the

agent

is

more

able

to

convert

sensory

information

into

motor

responses

to

maximize

its

performance;

we

also

observe

a

much

broader

internal

communication

between

oscillators,

with

a

great

variety

of

functional

connections

in

comparison

with

other

coupling

configurations.

Remarkably,

even

though

the

network

is

fully

connected,

the

flow

is

not

symmetric,

and

the

higher

performance

of

the

critical

coupled

network

can

be

linked

to

its

flexible

ongoing

dynamics

which

can

be

modulated

by

the

environmental

context..

The

second

experiment

studied

orientation

to

falling

circles

under

normal

and

inverted

vision,

particularly

interesting

for

presenting

to

the

agent

an

ambiguous

and

conflicting

scenario..

We

found

that

the

performance

increases

with

the

degree

of

coupling,

but

this

time

reaches

its

peak

between

and

c

decreasing

for

the

very

strong

coupling

values

greater

than

that.

We

compared

three

evolved

individuals,

one

with

no

coupling

c

one

with

strong

coupling

c

and

the

other

with

very

strong

coupling

c

between

oscillators.

The

first

is

unable

to

succeed

on

the

task

whilst

the

and

c

agents

produce

good

performance

in

both

sensory

modes,

with

the

c

agent

outperforming

the

very

strongly

coupled

one..

Dynamical

analysis

shows

that

the

c

network

rapidly

synchronises,

with

some

nodes

escaping

the

entrained

state

only

when

the

corresponding

sensory

input

is

large.

However,

there

is

a

diverse

phase

dynamics,

with

conflicting

values

resulting

in

different

phase

dynamics

that

are

exploited

to

generate

fit

behaviour

that

can

cope

with

the

different

visual

conditions.

Therefore,

even

though

the

network

has

a

fixed,

all-to-all

pre-established

coupling

configuration,

with

no

explicit

plasticity

mechanisms,

it

can

respond

and

adapt

to

conflicting

sensory

conditions

and

obtain

good

performance

on

the

task.

This

is

further

stressed

by

the

TE

analysis

which

shows

that

the

normal

and

inverted

vision

conditions

elicit

a

difference

in

the

information

flow

at

specific

points

of

the

agent's

trajectory.

The

results

highlight

the

asymmetry

of

information

flow

and

its

behavioural

correspondence

obtained

in

fully

connected

oscillatory

networks

only

by

changing

their

level

of

synchronisation.

Hence,

the

results

suggest

that

synchronised

networks

with

an

'intermediate'

level

of

coupling

are

more

able

to

reconfigure

functional

connections

not

only

when

comparing

across

different

coupling

configurations

but

also

between

the

two

vision

conditions,

reflecting

its

superior

performance

on

the

task..

Concluding

our

analysis

for

both

experiments,

consider

Fig.

b

which

display

the

formation

of

groups

of

oscillators

during

the

execution

of

the

task

for

the

selected

coupling

conditions

for

Experiments

and

respectively.

The

network

has

initially

single-node

'clusters',

each

corresponding

to

an

oscillator.

A

group

(cluster)

is

merged

whenever

the

phase

difference

between

two

given

nodes

is

below

a

certain

threshold

(chosen

to

be

here)

and

the

frequencies

are

similar

(less

than

z

apart).

Hence,

values

vary

from

(single,

giant

cluster

containing

all

the

oscillators)

to

(single-node

clusters)..

In

the

first

experiment,

the

uncoupled

network

is

associated

with

a

large

number

of

single-node

groups,

whereas

the

strongly

coupled

network

has

only

one

big

group

through

nearly

all

of

the

task,

indicating

complete

synchronisation.

Curiously,

the

critically

coupled

network

alternates

between

these

two

extremes.

The

same

behaviour

is

observed

for

Experiment

with

the

exception

that

the

very

strong

coupled

network

desynchronises

at

the

end

of

the

task,

as

revealed

by

earlier

analysis.

These

plots

reinforce

an

important

aspect

raised

in

the

Introduction

and

mentioned

throughout

the

paper:

synchronisation

dynamics

modulate

the

assembly,

collapse

and

reconfiguration

of

functional

sub-networks.

In

our

experiments,

a

higher

diversity

of

such

assemblies

are

present

in

the

top

performing

agents,

linking

the

adaptability

of

the

agent

to

the

flexibility

of

its

functional

connections..

The

previous

analysis

is

complemented

by

Fig.

d

which

present

the

dynamics

of

a

metastability

index

λ

and

the

Largest

Lyapunov

Exponent

(L

L

E)

for

the

chosen

scenarios.

λ

(Eq.

is

calculated

as

the

estimated

variance

of

the

synchronisation

index

r

(Eq.

over

all

time

points

t

T

and

reflect

the

diversity

of

synchronisation

states

obtained

for

each

coupling

configuration

of

the

network

(Shanahan

The

Largest

Lyapunov

Exponent,

calculated

using

the

phase

variables

θ

i

i

N

(Maistrenko

et

al.

indicate

how

chaotic

the

phase

dynamics-and

hence

our

system-are.

To

obtain

the

L

L

E,

we

followed

the

method

described

in

Rosenstein

et

al.

which

is

specially

designed

to

be

robust

and

fast

in

small

data

sets..

In

both

experiments,

we

notice

that

the

metastability

decreases

(as

expected)

as

the

coupling

increases,

showing

that

there

is

less

variation

in

the

synchronisation

index.

The

dynamics

of

the

number

of

groups

relate

to

the

metastability

index,

with

the

critically

coupled

network

also

appearing

in

an

intermediate

position.

An

interesting

point

for

the

critical

coupling

condition

is

to

notice

how

the

agent/environment

interaction

in

the

categorical

perception

task

reduces

the

spontaneous

chaoticity

level

of

the

network

(calculated

for

the

autonomous

network

dynamics),

whereas

in

the

inverted

vision

scenario,

near

the

critical

coupling,

chaotic

behaviour

is

more

prominent

when

the

agent

is

engaged

in

the

task

than

when

under

the

influence

of

the

autonomous

network

dynamics.

Also,

note

that

the

Largest

Lyapunov

Exponent

is

close

to

zero

in

the

uncoupled

autonomous

scenario

for

both

tasks

c

This

is

anticipated

and

due

to

the

lack

of

external

stimulus,

which

makes

each

node

of

the

network

oscillate

independently

in

a

quasiperiodic

way.

The

sensorimotor

coupling

changes

this

behaviour

(even

without

connections

between

nodes,

coupling

can

be

seen

as

occurring

'indirectly'

through

the

environment),

and

the

network

activity

becomes

more

chaotic.

Therefore,

the

above

observations

reflect

the

adaptability

and

modulation

of

the

network

activity

previously

noticed

in

the

information

flow

analysis.

It

also

suggests

that

although

neural

synchronisation

clearly

plays

an

important

role

in

the

generation

of

behaviour

in

the

fittest

agents,

other

kinds

of

transient

dynamics

are

also

exploited

(Santos

et

al.

There

are

some

limitations

for

the

framework

presented

here.

Starting

with

the

network

model,

the

original

formulation

of

the

Kuramoto

Model

does

not

take

into

account

important

properties

of

real

nervous

systems

(e.g.

spatial

distribution

of

units,

transmission

delays,

asymmetrical

interactions),

therefore

an

important

step

towards

a

more

biologically

plausible

architectures

would

be

to

implement

extended

versions

of

the

model

which

tackle

some

of

these

constraints

(Tass

Cumin

and

Unsworth

Breakspear

et

al.

These

extensions

may

also

unfold

more

complex,

metastable

dynamics

which

are

intrinsically

connected

to

dynamic

pattern

formation

in

brain

activity

and

hence

are

fundamental

to

adaptive

behaviour

(Omelchenko

et

al.

Tognoli

and

Kelso

Chialvo

Shanahan

As

for

the

task

scenarios,

the

experiments

were

conducted

following

an

evolutionary

robotics

approach

which,

despite

all

the

advantages

mentioned

throughout

the

paper,

suffers

from

high

computational

cost

for

fitness

assessments

which

can

be

a

limitation

in

scaling

up

to

more

complex

tasks

(see

Bongard

for

an

encouraging

technique

to

tackle

this

problem).

Also,

simulated

experiments

have

to

be

carefully

set-up

to

avoid

undesirable

gaps

when

transferring

to

real

world

scenarios.

In

this

sense,

the

addition

of

noise

during

simulations

is

essential

to

obtain

good

cross-platform

performance

(Jakobi

et

al.

Another

useful

approach

would

be

to

incorporate

ontogenetic

plasticity

mechanisms

to

the

robot's

controller,

which

are

shown

to

greatly

increase

adaptability

to

sensory

disruptions

and

dynamic

environments

(Urzelai

and

Floreano

It

is,

nevertheless,

a

promising

approach

to

testing

hypotheses

which

have

not

been

extensively

explored

yet

in

the

context

of

the

relationship

between

neural

synchronisation

and

behaviour.

Finally,

the

TE

analysis,

in

common

with

other

information

theoretic

tools,

relies

on

the

proper

choice

of

parameters,

on

accurate

probability

density

estimations

and

require

large

number

of

data

points

for

robust

estimations.

Using

the

extensions

to

the

original

TE

measure

suggested

in

Marschinski

and

Kantz

we

reduced

some

of

the

undesirable

effects

caused

by

small

data

sets.

Likewise,

further

work

should

consider

varying

the

orders

of

the

Markov

processes

(parameters

m

and

n

in

Eq.

for

they

influence

the

estimated

value

of

TE

and

may

be

used

to

reveal

time

dependences

since

motor,

sensors

and

nodes

may

operate

at

different

time

scales

(Lungarella

and

Sporns

Gourévitch

and

Eggermont

The

Matlab

source

code

for

our

numerical

simulations

is

available

at

or

from

the

authors

on

request.

In

the

first

experiment,

individuals

are

evolved

to

perform

categorical

perception

under

different

coupling

configurations,

calculated

by

re-scaling

the

current

value

of

K

c

(see

the

Sect.

for

details).

Figure

presents

the

fitness

statistics

for

the

best

evolved

agents

obtained

in

evolutionary

runs

at

different

coupling

configurations.

Notice

that

the

agent's

overall

performance

is

poor

in

the

no

coupling

case

c

smoothly

increases

as

the

coupling

strengthens

(peaking

at

K

K

c

and

then

sharply

falls

for

strong

coupling

c

onwards)..

These

results

indicate

that

tuning

the

oscillatory

networks

in

a

certain

region

of

their

synchronisation

regime

may

have

a

direct

impact

on

the

performance

and

evolvability

of

the

agent,

but

would

this

result

hold

had

we

had

a

different

task?

We

therefore

performed

a

second

set

of

experiments

by

changing

from

categorical

perception

to

the

circle

Figure

presents

the

fitness

statistics

for

the

best

evolved

agents

obtained

in

evolutionary

runs

at

different

coupling

configurations.

Notice

that

the

performance

increases

with

the

coupling

and

reaches

its

peak

with

values

of

K

around

c

whilst

in

the

previous

experiments

we

observed

a

decrease

in

fitness

with

coupling

values

larger

than

K

c

It

is

clear

from

these

two

figures

that

the

synchronisation

regime

the

network

is

tuned

to

influences

the

behavioural

performance

of

the

agent.

In

order

to

clarifying

why

this

happens

and

to

explore

the

underlying

mechanisms,

we

investigate

the

performance,

system

dynamics

and

information

flow

properties

of

three

evolved

agents

with

different

couplings

strengths,

which

encourage

different

degrees

of

synchronisation,

within

each

of

the

above

experiments

(agents

are

no

longer

under

the

influence

of

the

evolutionary

process

during

these

analyses).

These

were

chosen

to

be

c

K

c

and

c

for

analysis

(categorical

perception)

and

c

c

and

c

for

analysis

(orientation),

to

represent

weak,

'optimal'

and

strong

coupling

for

those

behaviours.

All

experiments

are

carried

out

with

agents

chosen

to

have

similar

t

u

(network

update

time,

see

Sect.

and

with

final

fitness

values

as

close

as

possible

to

the

median

of

the

fitnesses

obtained

in

the

evolutionary

process

for

the

given

coupling

scenario.

B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Hierarchical dynamic slicing Dynamic slicing is a widely used technique for program analysis, debugging, and comprehension. However, the reported slice is often too large to be inspected by the programmer. In this work, we address this deficiency by hierarchically applying dynamic slicing at various levels of granularity. The basic observation is to divide a program execution trace into "phases", with data/control dependencies inside each phase being suppressed. Only the inter-phase dependencies are presented to the programmer. The programmer then zooms into one of these phases which is further divided into subphases and analyzed. We also discuss how our ideas can be used to augment debugging methods other then slicing (such as "fault localization", a recently proposed trace comparison method for software debugging). Dynamic slicing [2, 13] is a well-known technique for program analysis and comprehension. Given a program P , an input I and an "observable error", dynamic slicing can be used to find out statements of P executed under input I which can be responsible for the error (via control or data flow). Typically, the "observable error" is specified as a slicing criterion (l, v) -a variable v and a line number l. Thus, if the value of variable v in line number l is "unexpected" we Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. perform slicing w.r.t. the criterion (l, v). The resultant slice can be inspected to explain the reason for the unexpected value. Dynamic slicing naturally corresponds to the notion of debugging -it tries to find out the reason for an unexpected variable value (like an unexpected output) for a given test input. However, for most real programs, the dynamic slices are too large for humans to inspect and comprehend. So, we either need to prune dynamic slices or we need tools to help a programmer understand a large dynamic slice.. In this paper, we take the second route. However, our method can be combined with techniques for pruning a dynamic slice (such as the recent work [29] ). We build a dynamic slicing method where the human programmer is gradually exposed to a slice in a hierarchical fashion, rather than having to inspect a very large slice after it is computed. The key idea is simple -we systematically interleave the slice computation and comprehension steps. Conventional works on slicing have only concentrated on the computation of the slice, comprehension of the slice being left as a post-mortem activity. In our work, we integrate the two activities in a synergistic fashion as follows.. • Computation of the slice is guided (to a limited extent) by the human programmer so that very few control/data dependencies in a large slice need to be explored and inspected.. • The programmer's comprehension of the slice is greatly enhanced by the nature of our slice computation which proceeds hierarchically. Thus, for programs with long dependence chains, this allows the programmer to gradually zoom in to selected dynamic dependencies.. To understand the potential benefits one can gain from our method, let us examine the reasons which make the comprehension of dynamic slices difficult.. • Many programs have long dependence chains spanning across loops and function boundaries. These dependence chains are captured in the slice. However, the slice being a (flat) set of statements, much of the program structure (loops/functions) is lost. This makes the slice hard to comprehend.. • Programs often also have lot of inherent parallelism. So, a slice may capture many different dependence chains.. We now discuss how hierarchical computation/exploration of slices can help programmers to comprehend large slices containing these two features -(a) long dependence chains, and (b) many different dependence chains. Figure 1(a) shows an example program with a long dependence chain. Consider an execution trace of the program ...3, 4, 5, 6 -where lines 3,4,5,6 of Figure 1 (a) are executed. Slicing this execution trace w.r.t. the criterion (line6, y) (i.e., the value of y at line 6) yields a slice which contains lines 3, 4, 5, 6 as well as lines inside the body of the functions f 1, f 2, f 3. In other words, since the slice is a (flat) set of statements, the program structure is lost in the slice. This structure is explicitly manifested in Figure 1(b) , where we show the dependence chain in a hierarchical fashion. In other words, the dependencies inside the functions f 1, f 2, f 3 are not shown. Here, a hierarchical exploration of the dependence chains will clearly be less burdensome to the programmer. Thus, in Figure 1(b) , by inspecting the dependencies hierarchically, the programmer may find it necessary to inspect the dependencies inside a specific function (say f 2). As a result, we can avoid inspecting the dependence chain(s) inside the other functions (in this case f 1, f 3). Now, let us consider programs with many different dependence chains. Figure 2 (a) shows a schematic program with several dependence chains, and hence substantial inherent parallelism. If the slicing criterion involves the value of y in line 6 -we need to consider the dependencies between y and x3, y and x2, as well as, y and x1. These three dependencies are shown via broken arrows in Figure 2 (b). Again, with the programmer's intervention, we can rule out some of these dependencies for exploration and inspection. In summary, our method works as follows. Given an execution trace (corresponding to a program input) containing an observable behavior which is deemed as an "error" by the programmer, we divide the trace into phases. This division is typically done along loop/procedure/loop-iteration boundaries so that each phase corresponds to a logical unit of program behavior. Only the inter-phase data and control dependencies are presented to the programmer; the intraphase dependencies are completely suppressed. The programmer then identifies a likely suspicious phase which is then subjected to further investigation in a similar manner (dividing the phase into sub-phases, computing depen- dencies across these sub-phases and so on). This process continues until the error is identified. Of course, an underlying assumption here is that the programmer will be able to identify the erroneous statement once this statement is pinpointed to him/her. One may comment that such a hierarchical exploration of dynamic dependencies involves programmer's intervention, whereas conventional dynamic slicing is fully automatic. Here we should note that, the process of error detection by using/exploring a dynamic slice involves a huge manual effort; the manual effort in exploring/comprehending the slice simply happens after the computation of the slice. In our hierarchical method, we are interleaving the computation and comprehension of dynamic dependencies. As in dynamic slicing, the computation of the dynamic dependencies is automatic in our method; only the comprehension involves the programmer. Moreover, we are gradually exposing the programmer to the complex chain(s) of program dependencies, rather than all at once -thereby allowing better program comprehension.. Technical Contributions. The technical contributions of the paper can be summarized as follows.. • In this paper, we propose hierarchical dynamic slicing which systematically interleaves computation and comprehension of dynamic dependencies for program debugging. During this process, a program execution trace is divided into phases at various levels of granularity. For each phase, we collect and report dynamic data/control dependencies across phases. Detailed data and control dependence computation inside each phase is not exposed to the programmer, thereby reducing program understanding effort (as compared to inspecting the dynamic slice).. • During hierarchical dynamic slicing, the generation of "program phases" is critical. In this paper, we propose an algorithm for dividing an execution trace into phases in a hierarchical fashion; this hierarchy corresponds to the levels of hierarchy we will explore for uncovering the dynamic dependencies gradually. Our phase division algorithm tries to divide a trace along control structure boundaries such as procedure calls and loop boundaries. We compare our notion of program phases with previous works on phase detection (e.g., see [7] ). These works have defined phases based on aggregate performance metrics (e.g., which basic blocks are executed may define a phase).. • We have evaluated our approach on several subject programs, including programs from the Software artifact Infrastructure Repository (SIR) [8] . Our experimental results show that our approach could significantly reduce the number of statements/variables which the programmer needs to examine (as compared to conventional dynamic slicing). We have also evaluated the number of manual interventions that may be required for hierarchical dynamic slicing.. • Finally, we have applied our ideas of hierarchical dependence exploration to fault localization, a dynamic analysis method which compares a run showing an error with one that does not (thereby locating the cause of the error) [6, 10, 18] .. Clearly, the notion of phase is central to our approach. In the next section, we present our notion of phases, which is then used in Section 3 to develop our slicing algorithm. In this paper, we have proposed hierarchical dynamic slicing to aid the comprehension of dynamic slices. The proposed application is in program debugging, where the programmer is gradually guided through complex program dependence chains. This is as opposed to the arduous task of understanding a full dynamic slice, where all of the comprehension is left to the programmer.. We have conducted detailed experiments on well-known subject programs written in Java drawn from the SIR repository [8] to evaluate the effectiveness of this approach. Our experiments show a substantial reduction in program understanding effort for our subject programs. In future, we plan to conduct more detailed experiments for hierarchical slicing as well as hierarchical fault localization. The idea of slicing came from the work of Weiser [27] , who proposed it as a program understanding aid. Subsequently, it was observed that the slice corresponding to a specific input (called the dynamic slice) is much smaller than the static slice. Dynamic slicing was first proposed by Korel and Laski [13] and subsequently investigated in a rich body of work (e.g., [2, 22, 29] ).. It has been shown that dynamic slicing can be used in program debugging [3, 14] . In fact, it can significantly reduce the number of statements to be inspected for locating the cause of an error [23, 30] . However, the resultant dynamic slice is typically reported to a programmer without any post-processing. For real-life programs, the dynamic slice is often too large to be inspected/examined by a human programmer. Recently, Sridharan et al. proposed "thin slicing" -a mechanism to hierarchically explore the static slice according to data flow [21] . In this paper, we have proposed an algorithm to hierarchically construct and explore the dynamic slice according to the control constructs in the program. We belive our work can be fruitfully combined with a dynamic adaptation of thin slices.. The idea of using the phases of an execution trace for debugging has appeared in earlier works. Miller and Choi [15] propose to do so by presenting the dynamic dependence graph of each phase to the user. This is effectively exposing the dynamic dependence chains inside the phase completely to the programmer, thereby burdening him/her with lot of redundant information! Our approach is exactly the reverse -we seek to hide the dynamic dependence chains inside a phase. Instead we summarize a phase via its "input" and "output" variables, which is gleaned from the inputs/outputs of the program as well as those of the preceding and succeeding phases.. Balmas [4] proposed hierarchical exploration of static program dependence graphs. This approach was later extended for hierarchical visualization of dynamic data dependencies [5] . We note that [5] only discusses better visualization of dynamic dependencies, whereas we interleave the dependence computation and comprehension steps. Indeed this is the main thesis of our approach -we feel that program comprehension cannot be left as a post-mortem activity, and should be used to guide dependence computation. Additionally, we have proposed a phase division method which helps identify and structure the exact feedback needed from the programmer in general -the programmer needs to select one from among a given set of inter-phase dependencies. Last, but certainly not the least, we have conducted detailed experiments to show that our approach has the potential to make dynamic slicing more useful for software debugging.. As far as slicing tools are concerned, existing dynamic slicing tools [1, 17, 24, 14] highlight dynamic slices in the source code browser. Typically, it is the programmer's responsibility to analyze the statements / statement instances, and identify the suspicious statement instances for debugging.. The works of [9, 12] extend algorithmic debugging (where a user queries about behavior of procedures) by pruning some of the queries via dynamic slicing. These works bear some similarity to our work, since they also rely on summarizing the behaviors of execution trace fragments (or phases). However to summarize a phase (say corresponding to a procedure call), they rely on a static summary of the procedure itself. In particular, they summarize the variable definitions of a procedure, which in the context of Java programs will require static points-to analysis. In contrast, we only seek to identify the inter-phase dynamic dependencies which can proceed efficiently without any points-to analysis.. Recently, various software fault localization approaches have been proposed. They proceed by comparing two runs of a buggy program [6, 10, 11, 18, 26] , where the two runs differ in whether any error is observed. Unlike dynamic slicing, these methods require that there exist some execution runs without the observable error, so that the comparison can proceed. We have employed our hierarchical exploration on these techniques, thereby augmenting them. Our initial experiments (see Section 4.2) show good potential in using such hierarchical fault localization to locate program errors.. Finally, we note that our approach is very different from the recently proposed Hierarchical Delta Debugging method [16] . This work seeks to simplify the program input that causes a program to fail. In this endeavor, it exploits the hierarchy present in the program input (e.g., if the program input is an XML or HTML file). Our approach, on the other hand, seeks to hierarchically detect and explore the control/data dependence chains in a program. In this section, we discuss the implementation and experiments for our hierarchical dependence exploration method. Table 1 : Descriptions of subject programs used to evaluate the effectiveness of our hierarchical dynamic slicing approach for debugging.

O B I B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B