Locating

moving

objects

in

car-driving

sequences

This

paper

presents

a

system

for

the

search

and

detection

of

moving

objects

in

a

sequence

of

images

previously

captured

by

a

camera

installed

in

a

conventional

vehicle.

The

objective

is

the

design

and

implementation

of

a

software

system

based

on

optical

flow

analysis

to

detect

and

identify

moving

objects

as

perceived

by

a

driver,

taking

into

account

that

these

objects

could

interfere

with

the

driver's

behavior,

either

through

distraction

or

by

posing

an

actual

danger

that

may

require

an

active

response.

The

problem

presents

significant

difficulties

because

the

vehicle

travels

on

conventional

roads,

open

to

normal

traffic.

Consequently,

the

scenes

are

recorded

with

natural

lighting,

i.e.,

under

highly

variable

conditions

(intensity,

shadows,

etc.).

Furthermore,

the

use

of

a

moving

camera

makes

it

difficult

to

properly

identify

static

objects

such

as

the

road

itself,

signals,

buildings,

landscapes,

and

moving

objects

of

the

same

speed,

such

as

pedestrians

or

other

vehicles.

The

proposed

method

consists

of

three

stages.

First,

the

optical

flow

is

calculated

for

each

image

of

the

sequence,

as

a

first

estimate

of

the

apparent

motion.

In

a

second

step,

two

segmentation

processes

are

addressed:

the

optical

flow

itself

and

the

images

of

the

sequence.

Finally,

in

the

last

stage,

the

results

of

these

two

segmentation

processes

are

combined

to

obtain

the

movement

of

the

objects

present

in

the

sequence,

identifying

both

their

direction

and

magnitude.

The

quality

of

the

results

obtained

with

different

sequences

of

real

images

makes

this

software

suitable

for

systems

to

study

driver

behavior

and

to

help

detect

danger

situations,

as

various

international

traffic

agencies

consider

in

their

research

projects.

This

paper

addresses

the

detection

of

moving

objects

that

could

interfere

with

driver

behavior,

either

through

distraction

or

by

posing

an

actual

danger.

The

scene

is

recorded

as

seen

by

the

driver,

i.e.,

with

a

moving

camera.

The

optical

flow

obtained

from

these

video

sequences

is

computed

to

obtain

an

estimate

of

the

apparent

motion

present

in

the

scene.

The

camera

is

installed

inside

a

conventional

vehicle

driving

on

public

roads.

The

moving

camera

and

the

variable

natural

lighting

pose

a

serious

challenge

for

calculating

optical

flow.

Taking

the

sequences

while

driving

on

actual

roads

also

implies

a

large

number

of

objects

captured

by

the

images

(vehicles,

vegetation,

signs,

buildings,

etc.)

This

makes

it

difficult

to

correctly

determine

optical

flow.

The

main

novelty

present

in

our

solution

is

the

combined

use

of

two

independent

and

complementary

segmentation

processes,

optical

flow

segmentation

and

raw

image

segmentation.

Once

both

processes

are

finished,

an

additional

computation

stage

is

dedicated

to

find

matches

between

both

segmented

image

results.

This

stage

also

identifies

objects

with

relatively

high

apparent

motion

to

detect

and

point

out

any

danger

situation.

All

these

processes

require

highly

computing-intensive

operations.

Due

to

this

fact,

a

parallel

real-time

version

of

this

system

has

been

developed

and

implemented

as

described

in

In

this

parallel

version,

all

processing

stages

are

carried

out

at

a

per

second

(fps)

rate

when

applied

to

small

images

or

at

a

rate

when

highresolution

images

are

used.

These

data

come

from

a

rather

basic

prototype,

but

we

envisage

running

these

processes

on

a

powerful

up-to-date

multi-core

quad

processor

PC

to

achieve

fps

for

full

HD

images.

Apparent

motion

identified

in

this

way

is

similar

to

apparent

motion

as

estimated

by

a

human

driver.

Therefore,

potential

dangers

that

could

be

identified

by

a

driver

who

is

aware

of

his

activity

can

be

identified

by

a

system

based

on

the

techniques

described

in

this

paper..

Once

OF

has

been

computed,

all

similar

optical

flow

vectors

are

grouped

together,

because

they

probably

belong

to

be

the

same

object.

The

complexity

of

the

process

is

increased

by

the

fact

that

the

camera

is

also

moving,

because

all

objects

in

the

image

sequence

seem

to

move

away

from

a

given

point,

the

focus

of

expansion

(FOE),

and

there

is

no

static

object

that

can

be

used

as

a

reference.

In

fact,

depending

on

the

movement

of

the

camera

relative

to

the

rest

of

the

objects,

three

situations

can

be

distinguished:.

Static

camera

with

moving

objects:

this

is

the

simplest

case

since

the

images

have

a

fixed

background,

which

helps

differentiate

the

moving

objects

more

clearly.

An

example

of

this

case

could

be

the

sequences

captured

by

roadside

traffic

cameras.

Moving

camera

with

static

objects:

this

scenario

presents

more

difficulties,

since

there

is

no

static

background

to

easily

determine

the

moving

objects.

However,

the

knowledge

that

the

objects

in

the

scene

are

static

makes

it

easier

to

establish

reference

points

for

movement.

Examples

of

this

case

could

be

the

landscape

sequences

in

commercial

movies.

Moving

camera

with

moving

objects:

this

is

the

special

case

addressed

in

this

paper,

in

which

both

the

camera

and

the

objects

are

moving

in

the

scene.

It

is

the

most

complex

case

since

there

is

no

available

reference.

The

camera

is

installed

on

board

a

vehicle

traveling

at

a

regular

speed,

ranging

from

a

few

kilometers

per

hour

to

km/h..

In

the

sequences

taken

with

the

moving

camera,

all

of

the

pixels

seem

to

recede

from

a

given

point

which

is

known

as

the

FOE.

The

FOE

is

the

point

on

the

horizon

at

which

the

camera

is

aimed.

It

seems

to

be

static

and,

therefore,

does

not

generate

optical

flow.

As

the

distance

of

the

pixels

from

the

FOE

increases,

they

create

a

higher

optical

flow

vector,

i.e.,

if

an

object

is

near

the

edge

of

the

image,

it

appears

to

move

faster

than

if

it

is

close

to

the

FOE.

Even

the

direction

of

the

vector

is

affected,

since

all

objects

appear

to

radially

move

away

from

the

FOE.

This

means

that

vectors

from

the

same

object

could

present

different

magnitudes

or

directions,

making

the

task

of

grouping

them

highly

complex..

Moreover,

due

to

the

camera

movement,

optical

flow

can

be

rather

high

even

when

all

the

objects

in

the

scene

are

static.

This

optical

flow

can

be

determined

and

the

pattern

of

movement

established,

but

it

is

not

uniform

or

constant

because

the

vehicle

speed

and

direction

change

considerably.

Depending

on

the

speed

of

the

camera

and

on

the

direction

of

its

motion,

the

apparent

pixel

movement

or

movement

pattern

is

obtained,

determining

the

velocity

of

a

static

object

relative

to

the

camera.

Furthermore,

the

images

are

taken

outdoors

with

variable

light

conditions,

so

shadows

and

reflections

constantly

appear

and

disappear

as

the

car

moves,

interfering

with

the

optical

flow

calculation..

In

addition

to

calculating

and

segmenting

the

optical

flow,

the

image

is

also

segmented

in

search

of

the

objects

composing

it.

Although

a

specific

kind

of

image

sequences

is

used,

restrictions

have

not

been

applied,

as

the

aim

is

to

obtain

general

solutions

applicable

to

any

kind

of

sequences.

The

system

tries

to

locate

moving

objects

and

to

assign

them

vectors

with

its

velocity

with

respect

to

the

camera,

based

solely

on

the

video

sequences

taken

from

the

inside

of

a

moving

car

traveling

along

a

road..

Two

approaches

were

used

in

analyzing

the

images:

static,

i.e.,

the

size

and

shape

of

the

objects

are

identified;

and

dynamic,

i.e.,

analyzing

the

motion

in

the

sequence

to

obtain

velocity

vectors.

In

a

subsequent

phase,

these

partial

results

are

combined

so

that

the

objects

are

mapped

to

velocity

vectors

as

shown

in

Figure

By

combining

the

best

of

each

approach,

satisfactory

results

can

be

obtained,

since

only

the

edges

of

uniform

objects

generate

optical

flow.

Consequently,

correct

shapes

of

uniform

objects

can

not

be

obtained

using

only

velocity

vectors.

On

the

other

hand,

static

images

alone

do

not

allow

obtaining

velocity

vectors

of

moving

objects..

The

general

approach

followed

is

shown

in

Figure

in

which

the

stages

of

the

process

are

represented

by

rectangular

shapes,

and

the

results

obtained

by

elliptical

shapes.

The

images

used

for

this

work

have

been

taken

with

a

camera

located

inside

the

vehicle

simulating

the

driver's

point

of

view

as

shown

in

Figure

This

paper

has

addressed

the

search

and

detection

of

moving

objects

in

an

image

sequence

using

a

moving

camera.

Starting

from

the

optical

flow

and

image

segmentation

processes,

the

objects

in

the

scene

and

their

motion

(both

direction

and

magnitude)

are

obtained..

To

obtain

an

estimate

of

the

apparent

movement,

the

optical

flow

of

each

image

in

the

sequence

is

calculated.

This

optical

flow

is

then

segmented,

and

similar

vectors

are

grouped

since

they

are

expected

to

belong

to

the

same

object.

In

a

third

step,

the

segmentation

of

images

is

addressed

to

distinguish

the

objects

present

in

the

scene.

Finally,

the

results

of

the

two

kinds

of

segmentation

are

grouped

to

associate

each

object

to

its

apparent

motion..

The

method

described

has

been

applied

to

detect

moving

objects

as

perceived

by

a

driver

which

could

interfere

with

the

driver's

behavior,

either

by

causing

a

distraction

or

by

posing

real

dangers

that

require

immediate

attention.

The

input

data

consists

of

video

sequences

recorded

by

a

camera

mounted

on

a

conventional

vehicle

driving

on

public

roads,

implying

several

challenges

when

calculating

the

optical

flow:

moving

camera

(all

objects

appear

to

move)

and

variable

natural

lighting

(constantly

changing

shadows

and

reflections)..

The

results

show

that

most

of

the

relevant

objects

from

the

scene

are

properly

detected

and

associated

to

their

corresponding

movement,

while

unrelated

objects

are

only

grouped

when

they

are

quite

far

and

very

difficult

to

distinguish,

even

to

the

human

eye.

This

section

looks

at

the

segmentation

of

multiple

image

sequences,

in

which

all

process

steps

are

considered:

image

segmentation,

blob

computation,

optical

flow

computation

followed

by

segmentation,

cluster

recognition,

and

finally,

cluster-blob

association.

The

process

followed

is

described

below:.

Starting

from

the

original

image,

the

segmentation

process

described

in

Section

is

applied,

obtaining

a

list

of

blobs

in

an

XML

file.

Optical

flow

is

generated

for

the

image

sequence

using

the

Lucas-Kanade

algorithm.

Optical

flow

segmentation

is

applied

to

obtain

a

list

of

clusters

in

an

XML

file.

Blobs

and

clusters

created

in

the

previous

steps

are

associated..

The

only

input

data

is

the

image

sequences

to

be

analyzed.

Several

test

sequences,

around

images

each,

have

been

recorded

with

a

interleaved

frames

per

second

camera

at

pixels.

Therefore,

every

ms,

a

complete

image

frame

is

generated,

after

combining

two

image

fields

provided

by

the

interleaved

camera

at

intervals.

The

main

problem

of

using

an

interleaving

camera

derives

from

the

delay

in

capturing

both

image

fields,

ms,

causing

a

mismatch

in

the

final

image.

The

camera

is

placed

inside

the

experiment

vehicle,

and

the

images

acquired

include

the

road

but

also

part

of

the

experimental

vehicle

such

as

the

dashboard,

steering

wheel,

and

roof

areas.

To

simplify

the

localization

process,

the

search

will

only

consider

the

area

of

the

image

showing

the

road.

This

section

briefly

describes

the

basis

on

which

the

system

analyzed

in

this

paper

is

supported.

As

indicated

earlier,

the

main

novelty

of

our

solution

is

the

combined

use

of

optical

flow

segmentation

and

raw

image

segmentation.

Moreover,

these

processes

should

ideally

be

carried

out

in

real

time.

Consequently,

several

of

the

application

areas

involved

deserve

to

be

placed

in

context,

specifically,

algorithms

for

optical

flow

computation

and

their

parallelization,

and

segmentation

of

the

optical

flow

results.

In

this

section,

the

implementation

of

the

Lucas-Kanade

algorithm

proposed

by

Correia

is

described

because

this

implementation

has

been

used

to

compute

the

optical

flow

prior

to

being

segmented.

All

the

parameters

used

in

this

section

are

obtained

from

the

original

sequential

implementation

by

Correia

presented

in

This

implementation

starts

by

smoothing

the

image

sequence

with

a

spatio-temporal

Gaussian

filter

to

attenuate

temporal

and

spatial

aliasing,

as

shown

in

It

applies

a

smoothing

Gaussian

filter:.

In

this

implementation

σ

is

and

therefore,

pixels

are

required:

the

central

one

and

4σ

pixels

at

each

side.

This

one-dimensional

symmetrical

Gaussian

filter

is

applied

three

times,

first

on

the

temporal

't'

dimension,

then

on

the

spatial

'X'

dimension,

and

finally

on

the

spatial

'Y'

dimension..

The

result

of

applying

the

Gaussian

smoothing

filter

to

an

image

can

be

seen

in

Figure

which

shows

the

original

image

Figure

and

three

steps:

the

result

for

the

temporal

filter

(Figure

for

the

spatial

filters

(Figure

and

the

final

result

(Figure

After

smoothing,

the

next

step

of

the

Lucas-Kanade

algorithm

is

to

compute

the

spatio-temporal

derivatives

for

the

three

dimensions:

t,

x,

and

y

(I

t

I

x

I

y

Using

the

previously

computed

image,

smoothed

on

t,

X

and

Y,

and

applying

a

numerical

approximation,

the

derivatives

(I

t

I

x

I

y

are

separately

computed

making

use

of

the

five-point

central

finite

differences

method,

used

to

compute

the

first

order

of

derivative

with

fourth

order

of

accuracy

on

one-dimensional

grid,

based

on

central

finite

differences

Taking

h

because

the

distance

between

two

consecutive

pixels

is

one,

the

one-dimensional

array

to

be

used

as

the

convolution

coefficient

mask

in

the

computation

of

the

partial

derivatives

is

obtained

as

follows:

Noise

parameters

are

σ

σ

and

σ

p

The

estimated

velocity

vectors

whose

highest

eigenvalue

of

A

T

W

A

is

less

than

are

considered

unreliable

(noise)

and

are

discarded

Figure

shows

the

optical

flow

computed

for

the

image

of

Figure

The

processing

steps

have

been

analyzed

The

vehicle

carrying

the

camera

is

overtaking

the

vehicle

on

the

right

while

it

is

being

overtaken

(quite

fast)

by

the

vehicle

on

the

left.

This

introduces

some

noise

in

the

results,

since

it

would

require

a

higher

temporal

resolution

to

correctly

handle

the

movement

of

objects

at

such

speed.

Figure

shows

two

images

of

a

video

sequence

that

has

been

processed

with

this

algorithm

and

also

the

corresponding

optical

flow.

In

this

sequence,

a

vehicle

can

be

observed

on

the

right

going

slower

than

the

vehicle

carrying

the

camera,

and

a

second

vehicle

on

the

left

is

changing

lanes.

Also

visible

is

a

traffic

information

panel

on

the

upper-right

corner

of

the

image.

The

optical

flow

generated

by

these

three

objects,

road

markings,

and

other

elements

in

the

image

is

also

shown

in

Figure

The

algorithm

used

for

building

the

vector

clusters

is

as

follows:.

The

starting

point

is

an

optical

flow

vector

matrix

mOpticalFlow(i,

j)

and

an

empty

cluster

list

lCluster

vector

is

associated

to

the

maximum

similarity

cluster,

i.e.,

the

cluster

with

the

smallest

return

value

in

similVect.

Once

associated,

the

average

vector

and

other

characteristic

properties

of

this

cluster

are

recalculated..

To

minimize

the

time

spent

on

calculating

the

similarity

function,

the

algorithm

avoids

calculating

the

distance

of

each

vector

v

from

all

the

other

vectors

within

each

cluster.

Instead,

a

cluster

bounding

box

containing

all

the

vectors

in

the

same

cluster

is

defined

and

an

initial

filtering

step

performed.

If

the

vector

distance

from

any

of

the

bounding

box

polygons

is

greater

than

a

predefined

value

peakDist,

then

MaxGAP

is

returned..

Not

only

peakDist

but

also

the

maximum

thresholds

peakAng

and

peakMag

must

be

defined.

If

the

comparison

returns

values

above

these

thresholds,

the

vectors

are

not

considered

similar.

It

is

also

necessary

to

determine

the

values

of

certain

constant

parameters

to

weight

the

angle

(kAng),

magnitude

difference

(kMag),

and

distance

(kDist)

between

vectors..

Determining

all

these

values

formed

part

of

an

experimental

process

considering

different

kinds

of

video

sequences.

From

these

experiments,

the

values

obtained

were

as

follows:

the

maximum

distance

or

peakDist

was

fixed

as

pixels.

This

means

that

optical

flow

vectors

belonging

to

the

same

cluster

are

separated

less

than

pixels.

A

higher

value

for

this

parameter

would

lead

to

interpret

several

clusters

as

being

the

same.

A

lower

value

for

the

parameter

would

artificially

lead

to

a

cluster

being

interpreted

as

several

independent

clusters..

The

maximum

value

for

the

angle

differences

peakAng

was

set

to

Consequently,

no

optical

flow

vector

can

be

associated

to

a

cluster

if

its

angle

difference

from

the

average

cluster

vector

is

greater

than

This

peak

value

may

seem

to

be

excessive,

but

it

is

derived

from

the

actual

use

of

a

moving

camera.

The

camera

movement

can

generate

optical

flow

vectors

with

fast

changing

directions,

especially

if

the

observed

objects

are

close

to

the

optical

axis

of

the

camera..

The

peak

difference

in

terms

of

magnitude,

peakMag,

is

also

experimentally

set

at

a

value

of

This

is

justified

by

the

small

differences

observed

when

comparing

the

optical

flow

vectors

generated

from

different

pixels

of

the

same

object..

Finally,

the

weights

used

for

the

expression

in

Section

have

been

assigned

the

following

values:

kAng

kMag

and

kDist

This

means

that

a

much

greater

importance

is

assigned

to

the

difference

between

angles,

while

the

magnitudes

are

considered

significantly

less

important

and

the

distances

within

the

image

almost

negligible.

The

overall

similarity

measure

is

thus

established

in

the

system

as

α

m

d

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Vehicle classification framework: a comparative study Video surveillance has significant application prospects such as security, law enforcement, and traffic monitoring. Visual traffic surveillance using computer vision techniques can be non-invasive, cost effective, and automated. Detecting and recognizing the objects in a video is an important part of many video surveillance systems which can help in tracking of the detected objects and gathering important information. In case of traffic video surveillance, vehicle detection and classification is important as it can help in traffic control and gathering of traffic statistics that can be used in intelligent transportation systems. Vehicle classification poses a difficult problem as vehicles have high intra-class variation and relatively low inter-class variation. In this work, we investigate five different object recognition techniques: PCA + DFVS, PCA + DIVS, PCA + SVM, LDA, and constellation-based modeling applied to the problem of vehicle classification. We also compare them with the state-of-the-art techniques in vehicle classification. In case of the PCA-based approaches, we extend face detection using a PCA approach for the problem of vehicle classification to carry out multi-class classification. We also implement constellation model-based approach that uses the dense representation of scale-invariant feature transform (SIFT) features as presented in the work of Ma and Grimson (Edge-based rich representation for vehicle classification. Paper presented at the international conference on computer vision, 2006, pp. 1185-1192) with slight modification. We consider three classes: sedans, vans, and taxis, and record classification accuracy as high as 99.25% in case of cars vs vans and 97.57% in case of sedans vs taxis. We also present a fusion approach that uses both PCA + DFVS and PCA + DIVS and achieves a classification accuracy of 96.42% in case of sedans vs vans vs taxis. Visual traffic surveillance has attracted significant interest in computer vision, because of its significant application prospects. Efficient and robust localization of vehicles from an image sequence (video) can lead to semantic results, such as 'Vehicle No. 3 stopped,' 'Vehicle No. 4 is moving faster than Vehicle No. 6.' However, such information can be more relevant if we not only can detect vehicles but also can classify them. Information such as gap, headway, stopped-vehicle detection, speeding vehicle, and class of a vehicle can be useful for intelligent transportation systems [1] . Monitoring vital assets using video surveillance has increased in recent years. The class of a detected vehicle can supply important information that can be used to make sure that certain types of vehicles do not appear in certain areas under surveillance. Multi-camera systems such as the one used in [2] can benefit immensely if the information regarding the classes of vehicles is available, as vehicle classification can be used in matching objects detected in non-overlapping field of views from different cameras.. Object detection and tracking has achieved good accuracy in recent years. However, the same cannot be said about object classification. Object recognition in case of still images has the problem of dealing with the clutter in the scene and a large number of classes. Object recognition in video sequences has the benefit of using background segmentation to remove clutter [3] . However, images obtained from video surveillance cameras are generally of low resolution, and in case of traffic video surveillance, the vehicles cover very small areas of these images, making the classification problem challenging. Vehicle classes such as cars and vans are difficult to differentiate as they have similar sizes. Therefore, classification techniques that use global features such as size and shape of the detected blob do not yield satisfactory results.. For this work, we consider three vehicle classes: cars, vans, and taxis. Classes like bus, semi, and motorcycle were not included because they are relatively easy to classify based on their size. We considered three different scenarios: cars vs vans, sedans vs taxis, and sedans vs vans vs taxis. Taxis and sedans are disjoint subsets of class cars. Therefore, results of sedans vs taxis will demonstrate the relevance of our approach when interclass variability is low. For the purpose of this paper, we used a dataset provided in [4] .. In [3] , Ambardekar et al. presented a comprehensive traffic surveillance system that can detect, track, and classify the vehicles using a 3D model-based approach. The classification using the 3D model-based approach requires camera parameters and orientation of a vehicle which can be calculated using tracking results. When vehicle orientation information is available, the methods presented in this paper can be used in a traffic surveillance system such as [3] to improve the vehicle classification accuracy.. In this work, we present five different vehicle classification techniques that can be used in combination with a consideration to the requirements of the scenario and do not require camera calibration. The two main contributions of our work are the following: (1) We present several approaches (PCA + DFVS, PCA + DIVS, PCA + SVM, LDA, and constellation model) and improvements over the published results that used state-of-the-art techniques. (2) We perform a comparative study of these and other approaches in the literature for the purpose of vehicle classification.. There are similarities between the problem of face detection and vehicle recognition especially in the typical size of an image sample under consideration. In face detection, the problem is finding a face from non-face image samples. However, in case of vehicles, we have multiple classes, and we want to differentiate between them. Turk and Pentland used PCA to form eigenfaces that can reliably recognize faces [5] . We extend the face detection based on PCA and implement three different techniques: PCA + DFVS, PCA + DIVS, and PCA + SVM. In these approaches, we create a principal component space (PCS) using PCA which we call vehicle space. In case of PCA + DFVS, the decision is made by finding the distance from a separate vehicle space for each class, and therefore, it is named distance from vehicle space (DFVS). On the other hand, PCA + DIVS predicts the class of a test image after projecting a test image onto a combined vehicle space, and distance from each class is calculated in vehicle space and hence named distance in vehicle space (DIVS). We achieved an overall accuracy as high as 95.85% in case of sedans vs vans vs taxis using PCA + DFVS. In the difficult case of sedans vs taxis, we achieved a 97.57% accuracy using PCA + DFVS which is higher than any published results using this dataset [6, 7] . PCA + DIVS yielded 99.25% accuracy in case of cars vs vans. Our results match or surpass the results in all the cases considered in [4, 8] . PCA depends upon most expressive features (MEFs) that can be different from most discriminant features (MDFs) [9] ; therefore, we also implement LDA that relies on MDFs. We observed that PCA + DIVS approach works better when the classes have more inter-class variation, e.g., cars vs vans, and PCA + DFVS seems to work better even in the difficult case of sedans vs taxis when inter-class variation is low. Therefore, we devised a new fusion approach that combines the benefits of both the approaches to classify sedans vs vans vs taxis, and were able to achieve classification accuracy of 96.42%.. Constellation models [10, 11] have been shown to be able to learn to recognize multiple objects using a training set of just a few examples. In [4] , Ma and Grimson used a constellation model with mean-shift clustering of scaleinvariant feature transform (SIFT) features to classify vehicles. However, the mean-shift clustering is considerably slow. In our implementation, we used K-means clustering. We also use an expectation maximization algorithm that considers up to 6 Gaussians and choose the number of Gaussians that maximizes the maximum likelihood for training data. We achieved similar accuracy with considerably less computation complexity compared to results achieved in [4] . In [4] , Ma and Grimson dealt with only a two-class classification problem. We extend the approach by performing classification in the three class case of sedans vs vans vs taxis.. The rest of the paper is organized as follows: Section 2 discusses previous work. Section 3 gives details about the techniques compared in this paper, and Section 4 describes and compares the results obtained. Section 5 discusses the conclusions and future work. In this work, we investigated and compared five different approaches for vehicle classification. Using the PCA + DFVS (eigenvehicle) approach, we were able to achieve an accuracy of 97.57% in the challenging case of sedans vs taxis which is higher than any published results using this dataset. PCA + DIVS outperformed all other approaches investigated in this paper in the case of cars vs vans. We also extended the constellation model approach [4] for classifying all three vehicle classes at the same time. LDA performed reliably but did not produce the best results in any of the cases we experimented on. PCA + SVM did not perform satisfactorily, but more experimentation with the choice of kernel and parameters might improve the results. Overall, PCA + DFVS approach achieves good results. However, the constellation model-based approach can be configured to work better in the presence of partial occlusion and minor rotations. We also presented an approach that combines two approaches and achieves improvements over using just one approach. We report accuracy of 96.42% in case of sedans vs vans vs taxis using a fusion of approaches. We can use the SIFT-PCA features to train the constellation models. Also, features other than SIFT, such as LoG affine regions can be used for modeling. The performance of the constellation model deteriorates as we extend it to multiple classes. A boosting algorithm can be used to choose the appropriate features for training.. In this paper, we used the images extracted from surveillance video captured using a fixed-angle camera. In the real traffic surveillance videos, vehicles can have different orientations and different view angles and sizes. The problem of orientation can be solved using camera self-calibration and the result of a tracking algorithm. Appearance-based algorithms have limited ability to model different view angles. A 3D model-based approach with strong thresholds (low false positives) can be used to train an appearance-based approach for better accuracy. In this work, we have considered five different approaches. This section provides details about the experimental setup used during testing, the effect of different parameter choices on the results, and the comparison between the different approaches. In our dataset, we have three types of vehicles: cars, vans, and taxis. Sedans and taxis are the disjoint subsets of class cars. The dataset provided in [4] has 50 images of each class for training and 200 images of cars, vans, and sedans each and 130 images of taxis. For the case of cars vs vans, we use 50 images from each class for training and 200 images of each class for testing. For the case of sedans vs taxis, we use 50 images from each class for training, while 200 images of sedans and 130 images of taxis are used for testing. We use the same experimental setup as that used in [6, 7] , so that a fair comparison is performed. In the case of sedans vs vans vs taxis, we use 50 images of each class for training and 200 images of sedans, and 200 images of vans and 130 images of taxis for testing. Previously published results do not consider such as a three-class case.

B I O O O O B I O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Comparison of Tree-Child Phylogenetic Networks Phylogenetic networks are a generalization of phylogenetic trees that allow for the representation of nontreelike evolutionary events, like recombination, hybridization, or lateral gene transfer. While much progress has been made to find practical algorithms for reconstructing a phylogenetic network from a set of sequences, all attempts to endorse a class of phylogenetic networks (strictly extending the class of phylogenetic trees) with a well-founded distance measure have, to the best of our knowledge and with the only exception of the bipartition distance on regular networks, failed so far. In this paper, we present and study a new meaningful class of phylogenetic networks, called tree-child phylogenetic networks, and we provide an injective representation of these networks as multisets of vectors of natural numbers, their path multiplicity vectors. We then use this representation to define a distance on this class that extends the well-known Robinson-Foulds distance for phylogenetic trees and to give an alignment method for pairs of networks in this class. Simple polynomial algorithms for reconstructing a tree-child phylogenetic network from its path multiplicity vectors, for computing the distance between two tree-child phylogenetic networks and for aligning a pair of tree-child phylogenetic networks, are provided. They have been implemented as a Perl package and a Java applet, which can be found at In this paper, we have presented two methods for comparing pairs of tree-child phylogenetic networks: a metric and an alignment algorithm. While the former quantifies the similarity between two networks in a way that allows to soundly establish whether a network is more similar to a second one than to a third one, the latter allows the interactive visualization of the differences between two networks. They are respectively the first true distance and the first alignment method defined on a meaningful class of phylogenetic networks strictly extending the class of phylogenetic trees, let alone the natural extension of the bipartition metric to regular networks. Since the class of tree-child phylogenetic networks includes the galled trees, this distance and this alignment can be used to compare the latter.. Tree-child phylogenetic networks with no redundant arcs (for which there is already a path from the tail to the head of the arc), known as normal networks, have been recently proposed by S.J. Willson as the class of networks where to look for meaningful phylogenies [47] , and polynomial time algorithms are known for reconstructing a normal network from distances [45] , [46] , [48] , but for the moment no reconstruction algorithm for arbitrary tree-child phylogenetic networks has been developed. Therefore, it remains an interesting open question to characterize the sets of sequences whose evolution can be explained by means of a tree-child network and to provide an algorithm to reconstruct this network, as well as to characterize the computational complexity of these problems.. On the other hand, several reconstruction methods for time-consistent tree-sibling phylogenetic networks have been proposed by Nakhleh and collaborators. Since no true distance for these networks is known so far, it is an interesting open question whether our distance and alignment method can be extended to these networks or not.. The Supplementary Material referenced in the paper is available at http://bioinfo.uib.es/~recerca/phylonet works/mudistance/. Fig. 12

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Towards

a

solid

solution

of

real-time

fire

and

flame

detection

Abstract

Although

the

object

detection

and

recognition

has

received

growing

attention

for

decades,

a

robust

fire

and

flame

detection

method

is

rarely

explored.

This

paper

presents

an

empirical

study,

towards

a

general

and

solid

approach

to

fast

detect

fire

and

flame

in

videos,

with

the

applications

in

video

surveillance

and

event

retrieval.

Our

system

consists

of

three

cascaded

candidate

regions

proposing

by

a

background

model,

fire

region

classifying

with

color-texture

features

and

a

dictionary

of

visual

words,

and

temporal

verifying.

The

experimental

evaluation

and

analysis

are

done

for

each

step.

We

believe

that

it

is

a

useful

service

to

both

academic

research

and

realworld

application.

In

addition,

we

release

the

software

of

the

proposed

system

with

the

source

code,

as

well

as

a

public

benchmark

and

data

set,

including

video

clips

covered

both

indoor

and

outdoor

scenes

under

different

conditions.

We

achieve

an

Recall

with

Precision

on

the

data

set,

and

greatly

improve

the

performance

by

state-of-the-arts

methods.

Fire

catastrophes

always

cause

big

lost

to

human

kind.

For

example,

Dhaka

fire

on

June

caused

over

people

died

and

over

injured.

Forest

fires

in

Russia

in

summer

caused

over

billion

US

dollars

lost.

Therefore,

robust

early-warning

fire

alarm

systems

are

always

in

need

to

protect

people's

safety

and

properties.

An

fast

and

accurate

fire

detection

method

is

the

key

technology

needed

in

such

systems.

As

a

result,

fire

or

flame

detection

has

received

great

attention

by

research

community

for

decades

In

recent

years,

fire

detection

also

find

applications

in

event

retrieval

on

fire

in

digital

image

or

video

archives

Existing

fire

detection

methods

can

be

classified

into

two

categories:

sensorbased

methods

and

vision-based

methods.

Sensor-based

methods

employed

information

captured

by

special

instruments

such

as

infra-red

sensors

and

smoke

sensors,

which

are

expensive

and

not

easy

to

get

Moreover,

those

sensors

typically

detect

the

presence

of

certain

particles

generated

by

fire

ionisation

or

photometry

rather

than

the

combustion

itself.

Therefore,

fire-alarm

systems

based

on

those

sensors

usually

have

time

delay

and

result

in

high

false

rates

In

contrast,

vision-based

methods

used

the

normal

R-G-B

images

or

videos

have

several

advantages

Firstly,

cameras

are

becoming

more

functional

with

steady

dropping

price.

Thus

images

and

videos

can

provide

more

detailed

visual

information

for

fire

detection

cheaply.

Moreover,

surveillance

cameras

already

installed

in

public

places

can

also

be

used

for

fire

detection.

Secondly,

the

response

time

can

be

faster

than

traditional

sensors,

as

cameras

do

not

need

to

wait

for

the

smoke

or

heat

to

diffuse.

Finally,

compared

to

traditional

point

sensors,

cameras

can

monitor

a

broader

area,

creating

a

higher

possibility

of

fire

detection

at

early

stage

Our

system

presented

in

this

paper

belongs

to

the

latter

category,

and

aims

to

detect

fire

in

video

clips..

Despite

of

the

growing

needs

and

interests

in

fire

detection,

there

is

still

not

a

large

number

of

work

on

fire

detection

in

the

computer

vision

literature

Building

a

robust

fire

detection

system

is

challenging

in

following

two

aspects:

fire

or

flame

is

flexible

in

shape

and

intensity,

and

it

has

no

fixed

structure

or

appearance.

Though

color

is

a

relative

distinct

feature

widely

used

for

fire

detection,

the

fire

color

appeared

in

images

and

videos

is

affected

by

the

camera

quality

(e.g.

resolution,

sharpness)

and

settings

(e.g.

white

balance).

The

fire

also

appears

widely

various

in

scale.

Fig

shows

six

fire

images

with

different

shape,

intensity,

color,

and

scale.

Thus,

it

is

hard

to

build

a

solid

and

generic

model

for

fire.

And

real

application

requires

a

fast

fire

detection

method

that

can

work

in

real-time.

Complicated

methods

or

models

can

not

find

a

position

in

real

applications.

Some

fire

detection

methods

used

filter

banks

frequency

transforms

and

motion

tracking

techniques,

thus

need

more

computational

processing

time

and

not

suitable

for

real-time

application

There

are

mainly

four

kinds

of

limitations

in

existing

fire

detection

systems:

merely

based

on

color,

or

color

plus

motion

information.

Such

as

and

only

used

color

to

detect

fire.

Limited

clues

may

lead

to

high

false

alarm.

merely

worked

for

some

special

situations,

such

as

tunnel

fire

used

many

heuristic

fixed

thresholds,

such

as

and

which

restricted

their

applications.

And

tested

on

limited

data

set,

such

as

Cho

et

al.

tested

their

method

on

six

fire

video

clips,

and

Ko

et

al.

performed

their

experiments

on

twelve

video

clips.

Though

they

showed

good

results

on

the

small

data

set,

their

systems

may

be

impractical

for

other

untested

situation.

Furthermore,

as

far

as

we

know,

there

are

no

standard

fire

data

sets..

Above

mention

four

limitations

lead

to

the

big

gap

to

real-world

fire

detection

applications.

The

method

we

proposed

in

this

paper

is

expected

to

narrow

the

gap.

We

have

randomly

downloaded

fire

video

clips

from

public

video

sharing

web

sites

This

data

set

is

much

larger

than

those

used

by

Project

page.

The

data

set

and

supplementary

materials

can

be

downloaded

from

the

anonymous

web

page:

other

researchers.

We

carried

out

a

comprehensive

empirical

study

and

experiments

on

this

data

set.

Based

on

the

experiment

results,

we

adopt

three

most

effective

fire

features,

including

a

local

and

a

global

color

descriptor

in

CIE

Lab

space,

and

a

SURF

(Speeded

Up

Robust

Features

texture

descriptor.

Then

a

Support

Vector

Machine

(SVM)

is

employed

to

find

fire

regions.

Based

on

our

experiment,

the

RBF(Radial

Basis

Function)

SVM

kernel

works

the

best

for

fire

detection.

Different

with

most

existing

fire

detection

methods

which

used

only

local

features

on

pixel

level,

we

find

that

our

proposed

global

color

feature

and

local

SURF

texture

feature

on

regions

made

a

good

contribution

for

fire

detection.

Finally,

a

temporal

verification

is

applied

to

reduce

the

false

alarm.

The

flowchart

of

our

system

is

shown

in

Our

system

did

not

use

any

complicated

features

or

tools,

thus

it

is

fast

and

can

detect

fire

in

real-time.

Compared

to

Toreyin

and

Cetin's

method

proposed

in

ours

has

higher

recall

and

precision

in

fire

detection

on

our

large

testing

data

set.

The

remainder

of

this

paper

is

organized

as

follows..

Section

discusses

the

detailed

implementation

of

our

system

based

on

empirical

study.

Section

introduces

the

framework

of

our

fire

detection

system.

Section

presents

our

experiment

results.

Comparisons

with

Toreyin

and

Cetin's

method

proposed

in

are

also

given

in

this

section.

Finally

Section

summarizes

this

paper

and

draws

the

conclusion.

In

this

paper,

we

present

a

robust

real-time

fire

detection

method

based

on

empirical

study.

To

carry

out

the

empirical

experiments,

we

collected

so

far

the

largest

open

data

set

for

fire

detection

in

video.

Experiments

show

that,

overall,

LAB

histogram

plus

SURF

texture

descriptor,

plus

the

RBF

SVM

kernel,

lead

to

the

best

performance

of

an

recall

with

precision

on

the

data

set,

which

greatly

improved

the

performance

by

state-of-the-arts

methods..

For

future

works,

we

can

study

more

detailed

fire

characteristics

to

effectively

detect

the

fire

on

initial

combustion

and

exclude

the

fire-like

objects

such

as

car

light

in

night

video.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Comparing error minimized extreme learning machines and support vector sequential feed-forward neural networks Error minimized extreme learning machines Support vector sequential feed-forward neural networks Sequential approximations a b s t r a c t Recently, error minimized extreme learning machines (EM-ELMs) have been proposed as a simple and efficient approach to build single-hidden-layer feed-forward networks (SLFNs) sequentially. They add random hidden nodes one by one (or group by group) and update the output weights incrementally to minimize the sum-of-squares error in the training set. Other very similar methods that also construct SLFNs sequentially had been reported earlier with the main difference that their hidden-layer weights are a subset of the data instead of being random. These approaches are referred to as support vector sequential feed-forward neural networks (SV-SFNNs), and they are a particular case of the sequential approximation with optimal coefficients and interacting frequencies (SAOCIF) method. In this paper, it is firstly shown that EM-ELMs can also be cast as a particular case of SAOCIF. In particular, EM-ELMs can easily be extended to test some number of random candidates at each step and select the best of them, as SAOCIF does. Moreover, it is demonstrated that the cost of the computation of the optimal outputlayer weights in the originally proposed EM-ELMs can be improved if it is replaced by the one included in SAOCIF. Secondly, we present the results of an experimental study on 10 benchmark classification and 10 benchmark regression data sets, comparing EM-ELMs and SV-SFNNs, that was carried out under the same conditions for the two models. Although both models have the same (efficient) computational cost, a statistically significant improvement in generalization performance of SV-SFNNs vs. EM-ELMs was found in 12 out of the 20 benchmark problems. Feed-forward neural networks (FNNs) are a popular machine learning approach for classification and regression problems with very interesting properties (see, for example, Bishop, 1995) . As a specific type of FNNs, the single-hidden-layer feed-forward networks (SLFNs) play an important role in practical applications. Since the optimal number of hidden nodes is problem dependent and unknown in advance, users often choose the number of hidden nodes by trial-and-error. Once the architecture is fixed, an iterative learning algorithm such as back-propagation gradient descent is normally applied to adjust the weights in the output and hidden layers simultaneously.. There exist, however, FNN models that construct the network sequentially, so that the number of hidden units is a result ✩ This work was supported in part by the Ministerio de Ciencia e Innovación (MICINN), under project TIN2009-13895-C02-01. of the learning process rather than being fixed a priori. For a review of constructive FNNs see, for example, (Kwok & Yeung, 1997) . Recently, error minimized extreme learning machines (EMELMs) have been proposed as a simple and efficient approach to build SLFNs sequentially (Feng, Huang, Lin, & Gay, 2009 ). EMELMs are an incremental extension of the previously presented extreme learning machines (ELMs) (Huang, Zhu, & Siew, 2006) . Both methods use random hidden nodes and find the output weights to minimize the sum-of-squares error in the training set by solving a linear system of equations. The specific features of EM-ELMs with respect to ELMs are that they add random hidden nodes one by one (or group by group) and update the output weights incrementally in an efficient way by taking advantage of the incremental construction of the hidden-layer output matrix involved in the linear system. Other recent extensions of ELMs can be found in Liang, Huang, Saratchandran, and Sundararajan (2006) and Miche et al. (2010) .. Very similar methods that also construct SLFNs sequentially had been reported earlier (Chen, Cowan, & Grant, 1991; Romero & Alquézar, 2006; Vincent & Bengio, 2002) . They all find the optimal linear weights of the output layer by solving the same linear system. In fact, the idea of adding random hidden units was already stated in the sequential approximation with optimal coefficients and interacting frequencies (SAOCIF) algorithm (Romero & Alquézar, 2002 as a possible strategy to be used and, as shown in Romero and Alquézar (2006) , the solution of the linear system can be computed efficiently thanks to the incremental construction of the hidden-layer output matrix. The EMELMs and SAOCIF with random selection strategy can easily be shown to be essentially equivalent (see Section 2), although it will be demonstrated in Section 3.1 that the cost of the output-layer weight computation described in Feng et al. (2009) is greater than the cost of the corresponding one in SAOCIF described in Romero and Alquézar (2006) .. Another strategy proposed in Alquézar (2002, 2006) to be used within SAOCIF was to take hidden-layer weights always as a subset of the data (input strategy). In this case, the resulting method is equivalent to the orthogonal least squares learning (OLSL) algorithm (Chen et al., 1991) and to kernel matching pursuit with pre-fitting (KMP-prefit) (Vincent & Bengio, 2002) . All of them select the hidden-layer weights among the input vectors, a property shared with support vector machines (SVMs) (Vapnik, 1995) . By this reason, these approaches are referred to as support vector sequential feed-forward neural networks (SVSFNNs) (Romero & Toppo, 2007) .. SV-SFNNs and SVMs were compared experimentally in Romero and Toppo (2007) . Very similar accuracies were found, although computational times were lower for SVMs. Regarding the number of support vectors, SV-SFNNs constructed models with less hidden units than standard SVMs and in the same range as ''sparse'' SVMs (Keerthi, Chapelle, & DeCoste, 2006) . On the other hand, EM-ELMs were compared in Feng et al. (2009) with other sequential algorithms, namely resource allocation networks (RAN) (Platt, 1991) and minimum resource allocation networks (MRAN) (Yingwei, Sundararajan, & Saratchandran, 1997) , as well as with the original ELMs (Huang, Zhu et al., 2006) . EM-ELMs obtained better performance and less training time than RAN and MRAN, and a similar performance but less training time than ELMs.. This work shows a comparison between EM-ELMs (i.e. SAOCIF with random strategy) and SV-SFNNs (i.e. SAOCIF with input strategy). An experimental study on 10 benchmark data sets for classification problems and 10 benchmark data sets for regression problems is presented in which the two methods are compared in the same conditions and using the same software. Since both approaches can be adjusted to have the same computational cost (each candidate weight vector for a hidden unit is either generated randomly or selected randomly among the input vectors), the goal is finding out whether there is any difference in generalization performance between EM-ELMs and SV-SFNNs. In other words, does the use of inputs (support vectors) as hidden unit weights provide any advantage over pure random selection?. Intuitively, the input strategy should perform better than the random one, since the input strategy samples the weights from the underlying distribution of the data. This information is lost in the random strategy, that ignores the statistical properties of the data. By contrast, the random strategy could take profit of its high flexibility to explore areas of the space of weights that the input strategy cannot. The growing interest in models that randomly select the weights of the hidden units (see Feng et al., 2009; Liang et al., 2006; Liu & Wang, 2010; Miche et al., 2010 , and references therein) makes the comparison of the input and random strategies relevant. Moreover, it should be noted that, on one hand, EM-ELMs have been proposed to build SLFNs as a superior alternative (due to their simplicity, efficiency and effectiveness) to common neural net approaches like back-propagation gradient descent (Feng et al., 2009 ) and, on the other hand, input vectors are rarely used as hidden-layer weights in SLFNs, especially in the case of additive units (i.e. two-layer perceptrons). First, it has been shown that EM-ELMs can be cast as a particular case of the SAOCIF method (with Random strategy) for constructing SLFNs. The only two real differences between the original EMELMs proposed in Feng et al. (2009) and the SAOCIF method with Random strategy proposed in Romero and Alquézar (2006) are: (i) the incremental method in which the same optimal (in a least squares sense) output-layer weights are recalculated each time a random hidden unit is added (or tested), and (ii) the number of random candidates tested for each hidden unit. Regarding the first difference, it has been demonstrated in Section 3.1 that the cost of the computation described in Feng et al. (2009) is greater than the corresponding one in the (generalized) method described in Romero and Alquézar (2006) . Since both methods compute the same optimal output-layer weights, there is no problem in using the more efficient one also for EM-ELMs, thus removing the difference. The second difference can also be eliminated either by restricting SAOCIF to test a single candidate at each step (as the original EM-ELMs do) or extending EM-ELMs to test some number of random candidates at each step and select the best of them (as SAOCIF does). Both possibilities have been explored in the experimental study carried out in this work.. Seemingly, a third difference between EM-ELM algorithm and SAOCIF with Random strategy is that the former allows group by group addition of hidden units whereas the latter does not. However, this is not a real difference, because when a single random candidate is used per hidden unit, adding a group of N hidden units is completely equivalent to N additions of one hidden unit, and SAOCIF may also perform group addition in the same way that EM-ELMs. Note, however, that if several candidates per hidden unit are allowed (something that EM-ELMs do not foresee) then one-by-one and group-by-group additions are not equivalent. We affirm that it is much more significant to allow several candidates per unit in one-by-one addition than adding single-candidate units group by group. Second, we have claimed that an alternative sequential method to construct SLFNs can be based on selecting the hidden-layer weights among the input vectors in the training set. This method, which has been referred to as SV-SFNNs (Romero & Toppo, 2007) or SAOCIF with Input strategy (Romero & Alquézar, 2006) , is essentially equivalent to the OLSL algorithm and to KMP-prefit. In order to assess the relative performance of both approaches (EMELMs vs. SV-SFNNs) in a fair manner, an empirical study has been realized on twenty benchmark data sets, 10 for classification and 10 for regression, under the same conditions and using the same software.. The experimental comparison between EM-ELMs and SVSFNNs presented in the paper draws two interesting conclusions that can be further investigated in future research. The first one is that selecting the hidden-layer weights as a subset of the input data, even if this selection is done randomly, yields better generalization results than selecting the hidden-layer weights in a purely random manner from scratch (like EM-ELMs do). As discussed at the end of Section 1, this is not an obvious result. Indeed, no statistically significant difference between the average performances obtained by the two strategies was found in eight of the benchmark problems, but SV-SFNNs showed a statistically significant improvement in generalization performance in the other twelve. One might ask whether there is any noticeable difference between these two groups of problems. For classification problems, data sets with a higher number of variables (see Table 1 and imagine for instance an arbitrary threshold of 20 inputs) were the ones in which SV-SFNNs outperformed EM-ELMs (with the exceptions of the German credit and the Segmentation problems). Although this can be considered as a reasonable result, which may be justified by the difficulty in finding adequate decision boundaries in highdimensional input spaces from randomly distributed hidden-layer weights, the underlying hypothesis needs further validation in future studies. For regression problems this trend is not so clear.. The second conclusion of the experimental study is that, independently of the strategy used (input or random), the number of candidates for the hidden-layer weights is a parameter that controls the trade-off between the generalization performance, the computational cost and the number of hidden units of the final models. In general terms, by increasing the number of candidates at each step of the sequential algorithm (recall that in the originally proposed EM-ELMs this number is 1), the generalization is improved and the final number of hidden units is reduced, at the expense of a higher training time. However, as pointed in Section 3.1, the computational cost of trying C candidates is lower than C times the cost of trying a single candidate, due to the incremental way in which the optimal output-layer weights are calculated.. Both strategies can be further improved. The random strategy could be modified so as to obtain the weights taking into account the underlying distribution of the data. The input one could take profit of sample selection methods (see Plutowski, Cottrell, & White, 1996 , for example) to select the input vectors in a better way than randomly. In addition, regularization techniques can also be used for both strategies. There are several reasons that can explain why the input strategy performs better than the random one, as explained next. On the one hand, the structural risk minimization principle defines a trade-off between the quality of the approximation to the data and the complexity of the model (Vapnik, 1995) . According to this principle, given two models with the same empirical risk, the model with lower complexity (approximation capability) should be preferred. In our experiments we did not observe differences between the empirical risks of the input and random strategies, although it is clear that the random strategy has greater approximation capability than the input one. On the other hand, choosing the hidden-layer weights among the input vectors is more likely to obtain better candidates (in the sense of informative candidates, such as prototypes, border points, etc.) than sampling it randomly. In some sense, choosing the hidden-layer weights among the input vectors is related to sampling the underlying distribution. The output function of an SLFN (i.e. a fully connected FNN with a single hidden layer of N h units and m linear output units) can be expressed as a linear combination of simple (basis) functions:. where ω i ∈ R n and b i ∈ R are the learning parameters of the hidden units, λ i ∈ R m are the output-layer weights connecting the i-th hidden unit to the m output units, ϕ is the activation function of the hidden units, ϕ(ω i , b i , x) is the output of the i-th hidden unit with respect to the input x, and λ 0 ∈ R m denotes the bias terms (if any) of the linear output units.. Although a lot of activation functions ϕ (even not neuron alike) can be used that allow universal approximation capability, the more usual choices are the Gaussian RBF applied to a distance between an input vector and a center. and the sigmoid (e.g. hyperbolic tangent) applied to a scalar product of the input and weight vectors (this will be referred to as a sigmoid additive unit (Feng et al., 2009) . For Gaussian RBF units, ω i ∈ R n and b i ∈ R + are the center and the impact factor of the i-th RBF unit. For sigmoid additive units,. n is the weight vector connecting the input layer to the i-th hidden unit and b i ∈ R is the bias of the i-th hidden unit. In our experiments presented in Section 3, a third activation function has also been tested, the sine applied to the scalar product (i.e. a sine additive unit). For a given set of training examples {(. the outputs of the network are equal to the targets, we have. Eq. (5) can be written compactly as. where H is an L×N h matrix called the hidden-layer output matrix of the network. , λ is an N h ×m matrix containing the output-layer weights, and T is an L × m matrix containing the target values in the training set. The output layer biases can be added by including in H a first column with a fixed value of 1 (and increasing N h by 1).. Normally, the number of training examples L will be much greater than the number of hidden units N h and an exact solution of (6) cannot be expected. Then, the usual cost function in SLFNs (and in general in FNNs) is the sum-of-squares error. It is well known (e.g. Bishop, 1995) that, to minimize E, the optimal output-layer weights can be computed aŝ. is the pseudo-inverse (or Moore-Penrose generalized inverse) of the hidden-layer output matrix H. The sum-of-squares error can be expressed as • Preprocessing. Categorical attributes were converted to dummy variables (the p different categories were represented with p input variables, so that only the input variable associated to its category is one, and all others are zero). The rest of the attributes (including the target variable for regression data sets) were scaled to mean zero and variance one.. • Random weights. In the random strategy, hidden-layer weights were uniformly chosen within the same range of values that the input values (after scaling). In this way, the ranges of the hidden-layer weights were the same for both strategies.. • Activation functions. Three types were used: Gaussian RBF (2), sigmoid additive (3) and sine additive (4) units, but with a further multiplicative positive parameter γ introduced for a wider search. Specifically, γ multiplies the distance ∥x − ω i ∥ in the RBF units and the scalar product ω i · x in the additive units. • Parameters and model selection. A hidden-unit candidate weight vector was not considered valid if the associated linear equations system could not be solved or if the 1-norm of the solution (the output-layer weights) was greater than a certain value M. This can be seen as a form of regularization. M was set to 1024. We fixed N max = 99 and ϵ = 0, so that N max hidden units were always added. These values were selected according to our previous experience with these models (recall that the aim of the experiments is to compare EM-ELMs and SV-SFNNs under the same conditions for the two models). In order to get an adequate value for the γ parameter, much more problemdependent, a search was performed ranging γ from 2 −10 to 2 5 . The same search was performed for all the models, and repeated for every activation function.. • Model training and testing. The methods were trained and tested over 30 training-validation-test different random partitions (80% training, 10% validation, 10% test) of the whole data set. For every configuration (defined by a given strategy, C max , activation function and γ ), the networks with the lowest errors in the validation subsets were selected as the final models. For classification data sets, the accuracies of the final models were given by the average accuracies measured in the test subsets. For regression data sets, the performance of the final models was measured by the following normalized squared error (NSE) (Bishop, 1995) :. where K is the number of examples in the test subset, f (x) is the final model and t is the mean target value in the test subset. The sizes of the final models are defined by their average number of hidden units. Tables 3 and 4 show the average accuracies of the best final models (among all γ ) for the two strategies (input and random) and the three activation functions tried (Gaussian RBF, sine additive, sigmoid additive) using the methodology previously described for the 10 classification data sets studied. Table 3 displays the results of the methods for C max = 1, where the input strategy is fully comparable to EM-ELMs, and Table 4 displays the results for C max = 59. For the 10 regression data sets studied, Tables 5 and 6 show the average NSE of the best final models for C max = 1 and C max = 59 respectively. Tables 7-10 show the average number of hidden units in the final models selected for each combination of strategy and activation function for C max = 1 and C max = 59.

O B I I I I O B I I I I I B I I I I B I I I I I B I O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Evaluating cost taxonomies for information systems management The consideration of costs, benefits and risks underpin many Information System (IS) evaluation decisions. Yet, vendors and project-champions alike tend to identify and focus much of their effort on the benefits achievable from the adoption of new technology, as it is often not in the interest of key stakeholders to spend too much time considering the wider cost and risk implications of enterprise-wide technology adoptions. In identifying a void in the literature, the authors of the paper present a critical analysis of IS-cost taxonomies. In doing so, the authors establish that such cost taxonomies tend to be esoteric and difficult to operationalize, as they lack specifics in detail. Therefore, in developing a deeper understanding of IS-related costs, the authors position the need to identify, control and reduce IS-related costs within the information systems evaluation domain, through culminating and then synthesizing the literature into a frame of reference that supports the evaluation of information systems through a deeper understanding of IS-cost taxonomies. The paper then concludes by emphasizing that the total costs associated with IS-adoption can only be determined after having considered the multi-faceted dimensions of information system investments. Organizations increasingly rely on Information Systems (IS) to acquire a competitive edge, with this often translating to an increased budget allocation to support their investments in an IS infrastructure. The purchase of Information Technology (IT) and development of an IS are seen as a source of increasing cost and concern to management [7, 17] . In-line with the rapid advancements in IT, and with the cost of technology continuing to fall, it is no longer efficient for management to try and reduce direct infrastructure costs alone, as competitors are able to secure similar if not the same discount on hardware and software.. There is the widespread recognition of the need to consider the social and organizational implications associated with the adoption of IT/IS from a cost perspective, if management are to seek competitiveness through reducing their IS-costs. Adding to the complexity of socio-organizational cost identification and evaluation is then the difficulty of their accommodation within the portfolio of investment appraisal techniques available to management [12] . As IS increasingly evolve with the needs of the business environment through the use of improved integration technologies, any robust evaluation process will require a deeper understanding of the cost implications throughout the information systems projectsÕ lifecycle [14] . In doing so, allowing managers to develop a better understanding of the implications associated with their investment as it matures and evolves. It is here where this paper attempts to address a void in the literature, through developing a frame of reference that will allow decision-makers to the navigation through the portfolio of cost taxonomies for information systems evaluation. Organizations are increasingly seeking a competitive edge through cost-managing their technology. However, this management should not be restricted to reducing the cost of IT alone but rather through identifying where IS-related costs lie. This paper has sought to provide an insight for decision-makers into the complexity of identifying the costs associated with information system investments. It has demonstrated that although traditional approaches to investment appraisal are inadequate in accommodating the portfolio of investment related costs, emerging integrated approaches are making considerable progress to support tier inclusion. The authors of this paper have identified eight IS-cost taxonomies, which seek to contribute towards developing a deeper understanding of IS-related costs through their identification and classification. Emerging approached to evaluation such as the balanced scorecard and information economics attempt to integrate many of the limitations inherent in traditional approaches. In light of this, integrated approaches have the potential of accommodate indirect/hidden costs associated with information systems adoption. Therefore, to carry out a robust comprehensive evaluation process, it is necessary for decisionmakers to not only integrate hidden/indirect costs into the decision-making process but also categorize such costs and ensure that this forms part of the organizational learning for future information systems adoption. The rational here being Ôif such costs can be identified; they can then be managedÕ. Such management supports reduction and control, which lie at the genesis of the rational that underpins the efficiency and effectiveness of cost managing information systems.

O O B O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A Multi-Orientation Analysis Approach to Retinal Vessel Tracking Abstract This paper presents a method for retinal vasculature extraction based on biologically inspired multi-orientation analysis. We apply multi-orientation analysis via so-called invertible orientation scores, modeling the cortical columns in the visual system of higher mammals. This allows us to generically deal with many hitherto complex problems inherent to vessel tracking, such as crossings, bifurcations, parallel vessels, vessels of varying widths and vessels with high curvature. Our approach applies tracking in invertible orientation scores via a novel geometrical principle for curve optimization in the Euclidean motion group SE(2). The method runs fully automatically and provides a detailed model of the retinal vasculature, which is crucial as a sound basis for further quantitative analysis of the retina, especially in screening applications. In this paper we demonstrated that by representing image information in an invertible orientation score, one can exploit the disentanglement of crossing structures to track blood vessels through crossing points. We introduced a new algorithm that tracks vessel edges through the orientation score of an image (ETOS). The ETOS algorithm can generally be used with both invertible and non-invertible orientation scores, which were in this paper constructed with cake wavelets and Gabor wavelets respectively. We demonstrated that best results were obtained using invertible orientation scores. We also introduced a fast alternative method based on vessel centerline tracking through a multi-scale set of noninvertible orientation scores (CTOS). While the CTOS algorithm is very fast, the multi-scale approach makes the algorithm less stable at critical vessel points (crossings and parallel vessels) compared to ETOS.. The ETOS algorithm was used as a basis for our vasculature tracking algorithm, which we used to construct detailed hierarchical models of the retinal vasculature. Within this paper we validated the reliability of the width measurements provided by the models using ground truth data, and showed that our method performs excellent in comparison to other state of the art algorithms. Validation of the topology of the models showed that our method constructs clean topological models of the vasculare tree, i.e. they contain very few false positive vessels.. Tracking within orientation scores relies on a novel and basic geometrical principle (V-plane optimization) within a sub-Riemanian manifold within SE(2). In Appendix A we relate probabilistic approaches to optimal curves in SE(2) to this geometric principle. Here we include supporting examples of analytic and numeric computations on completion fields on SE(2). Such completion fields are obtained from collision probabilities on SE(2), while the orientation score provide a whole distribution of these particles. The next step in future work is to apply tracking by geometric control [19] in enhanced orientation scores, obtained by applying stochastic processes for [24, 18] contour completion and enhancement directly on the score.. A A mathematical underpinning of optimization in the e η − e θ tangent planes V Before we provide the key-geometrical principle behind our ETOS algorithm, we make some comments on the moving frame of reference that lives in the domain of an orientation score.. The domain of an orientation requires a curved (nonEuclidean) geometry as can be seen in Fig. 1 . To this end we identify the coupled space of positions and orientations R 2 S 1 with SE(2). This needs a bit of explanation. To see whether a local line element (x , θ ) ∈ R 2 × S 1 with position x ∈ R 2 and orientation θ is aligned with its neighbors one applies a rigid body motion g = (x, R θ ) ∈ SE(2), with counter-clockwise rotation and translation via Eq. (10). This puts a 1-to-1 correspondence between rigid body motions (x, y, R θ ) and elements from R 2 × S 1 as every local line element (x, y, θ) can be obtained by a local line element (0, 0, 0) positioned at the origin 0 = (0, 0) pointing in say x-direction:. Via this identification we see that the curved domain of an orientation score is due to the non-commutative group structure of the rigid body motion (also known as roto-translation group or Euclidean motion group) SE(2) in the plane. Indeed, a concatenation of two rigid body motions is again a rigid body motion and this induces the group-product given by Eq. (11) It is a semi-direct product of rotations SO(2) and translations R 2 , as the rotation part affects the position part. As a result this group product is non-commutative, i.e. gg can be very different from g g. Therefore it is conceptually wrong to consider the domain of an orientation scores as the flat Cartesian space R 2 × S 1 with Euclidean metric and it is wrong to just take derivatives of orientation scores only in ∂ x direction or only in ∂ y direction. For formal underpinning of this statement see [17, Thm.21] , for a short illustration see [31, Fig.2.6 ]. Instead one must differentiate along the moving frame of reference {∂ θ , ∂ ξ , ∂ η } with ∂ θ , ∂ ξ = cos θ∂ x + sin θ∂ y , ∂ η = − sin θ∂ x + cos θ∂ y .. This a priori moving frame of reference coincides with the socalled Lie algebra of left-invariant vector fields on the group SE(2). The curved geometry in the orientation score requires us to apply parallel transport along this moving frame of reference. For instance in Fig. 1(d) we see examples of autoparallel curves whose tangent vector is always constant to the moving frame of reference. The non-commutative structure of SE(2) (due to the fact that rotations and translations do not commute) is reflected by the commutators in the Lie-algebra. where [A, B] = AB − BA. As a result first moving in ξ direction and then moving in θ direction brings us to a different point in SE(2) ≡ R 2 ×S 1 than first moving in θ direction and then moving in ξ direction (as can be deduced in Fig. 1(b) ). .

O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O