On

Using

Feedforward

Neural

Networks

for

Clinical

Diagnostic

Tasks

Georg

Dorffner

Austrian

Research

Institute

for

Artificial

Intelligence

Schottengasse

Vienna

Austria

Tel

Fax

email

georg@ai.univie.ac.at

and

Dept.

of

Medical

Cybernetics

and

Artificial

Intelligence

University

of

Vienna

Gerold

Porenta

Dept.

of

Cardiology

Clinic

of

Internal

Medicine

University

of

Vienna

Medical

School

Waehriger

Guertel

Vienna

Tel

Fax

email

porenta@vm.akh?wien.ac.at

On

Using

Feedforward

Neural

Networks

for

Clinical

Diagnostic

Tasks

Georg

Dorffner

Austrian

Research

Institute

for

Artificial

Intelligence

and

Dept.

of

Medical

Cybernetics

and

Artificial

Intelligence

University

of

Vienna

georg@ai.univie.ac.at

Gerold

Porenta

Dept.

of

Cardiology

Clinic

of

Internal

Medicine

University

of

Vienna

Medical

School

porenta@vm.akh?wien.ac.at

Abstract

In

this

paper

we

present

an

extensive

comparison

between

several

feedforward

neural

network

types

in

the

context

of

a

clinical

diagnostic

task

namely

the

detection

of

coronary

artery

disease

-LRB-

CAD

-RRB-

using

planar

dipyridamole

stress-redistribution

scintigrams

We

introduce

results

from

well-known

-LRB-

e.g.

multilayer

perceptrons

or

MLPs

and

radial

basis

function

networks

or

RBFNs

-RRB-

as

well

as

novel

neural

network

techniques

-LRB-

e.g.

conic

section

function

networks

-RRB-

which

demonstrate

promising

new

routes

for

future

applications

of

neural

networks

in

medicine

and

elsewhere

In

particular

we

show

that

initializations

of

MLPs

and

conic

section

function

networks?which

can

learn

to

behave

more

like

an

MLP

or

more

like

an

RBFN?can

lead

to

much

improved

results

in

rather

difficult

diagnostic

tasks

Introduction

In

this

paper

we

present

an

extensive

comparison

between

several

feedforward

neural

network

types

in

the

context

of

a

clinical

diagnostic

task

namely

the

detection

of

coronary

artery

disease

-LRB-

CAD

-RRB-

using

planar

dipyridamole

stress-redistribution

scintigrams

We

demonstrate

that

given

the

peculiar

aspects

of

many

clinical

applications?such

as

small

and

unbalanced

training

sets?the

widely

used

multilayer

perceptrons

-LRB-

MLP

-RRB-

and

the

backpropagation

learning

rule

need

not

be

the

method

of

choice

to

arrive

at

optimum

re

sults

We

introduce

results

from

well-known

-LRB-

e.g.

radial

basis

function

networks

or

RBFN

-RRB-

as

well

as

novel

neural

network

techniques

-LRB-

e.g.

conic

section

function

networks

or

CSFN

-RRB-

which

demonstrate

promising

new

routes

for

future

applications

of

neural

networks

in

medicine

and

elsewhere

In

particular

we

show

that

initializations

of

MLPs

and

conic

section

function

networks?which

can

learn

to

behave

more

like

an

MLP

or

more

like

an

RBFN

lead

to

much

improved

results

in

the

rather

difficult

task

of

detecting

significant

CAD

from

planar

thallium

images

using

coronary

angiography

as

the

reference

method

We

also

discuss

the

viability

of

MLP

initialization

for

screening

a

multitude

of

different

possible

training

sets

and

thus

reducing

the

training

time

during

the

design

of

an

optimal

network

for

generalization

Detecting

Coronary

Artery

Disease

with

Neural

Networks

Since

in

a

cooperation

between

the

Austrian

Research

Institute

for

Artificial

Intelligence

and

the

Dept.

of

Cardiology

at

the

University

of

Vienna

we

have

been

investigating

the

possible

use

of

neural

networks

in

the

assessment

of

coronary

artery

disease

-LRB-

CAD

-RRB-

-LSB-

-RSB-

The

reference

method

to

confirm

or

exclude

CAD

is

coronary

angiography

which

is

an

evasive

procedure

with

an

associated

rate

of

complication

including

rare

cases

of

death

Among

a

variety

of

non-invasive

tests

myocardial

thallium

scintigraphy

has

proven

useful

in

the

detection

of

CAD

-LSB-

-RSB-

Diagnostic

statements

of

clinical

relevance

in

CAD

concern

the

presence

of

disease

-LRB-

yes/no

-RRB-

the

localization

-LRB-

vascular

territory

-RRB-

and

the

extent

of

disease

-LRB-

single

or

multivessel

-RRB-

The

very

nature

of

the

interpretation

of

scintigrams

lends

itself

nicely

to

an

automated

solution

employing

neural

networks

In

particular

data

from

the

segmental

analysis

of

scintigraphy

was

chosen

as

input

to

a

three-layer

perceptron

while

the

output

was

designed

to

correspond

to

one

of

the

three

diagnostic

classifications

as

mentioned

above

The

process

of

diagnosis

in

this

setting

is

a

one-shot

mapping

from

the

input

data

to

the

output

Thus

the

involved

knowledge

-LRB-

in

the

sense

of

artificial

intelligence

-RRB-

or

structures

have

to

be

considered

as

low-level

information

in

contrast

to

knowledge

sources

that

many

medical

expert

systems

have

to

deal

with

At

the

same

time

this

technique

appears

easily

generalizable

to

other

similar

problems

of

medical

image

classification

and

interpretation

Planar

thallium

scintigraphy

generates

images

of

the

myocardial

tracer

uptake

after

pharmacological

stress

with

dipyridamole

and

at

rest

in

standard

views

of

the

heart

-LRB-

i.e.

the

projections

anterior

LAO

degrees

LAO

degrees

-RRB-

The

original

scintigrams

are

stored

on

computer

file

as

x

pixel

matrices

The

complete

encoding

of

such

an

image

would

require

a

very

large

number

of

input

units

which

is

associated

with

several

limitations

First

large

input

patterns

require

extreme

digital

storage

space

Secondly

the

number

of

input

units

increases

the

number

of

connections

and

thus

the

training

time

Thirdly

training

networks

with

large

numbers

of

connections

would

require

a

similarly

large

number

of

training

cases

as

the

network

has

to

achieve

optimization

of

a

process

involving

as

many

degrees

of

freedom

as

connections

In

our

case

a

maximum

of

patient

records

were

available

far

too

few

to

match

the

number

of

connections

in

a

full

fletched

network

taking

entire

images

as

input

For

these

reasons

a

method

of

reducing

the

size

of

a

single

input

pattern

was

desired

from

the

beginning

The

method

we

employed

is

based

on

segmental

analysis

of

the

circumferential

profile

and

generates

normalized

count

activities

in

five

anatomical

segments

per

planar

view

In

addition

segmental

washout

rates

were

also

obtained

All

results

were

achieved

using

this

reduced

input

representation

of

values

This

shows

that

while

analysis

of

images

without

preprocessing

may

exceed

available

resources

in

many

cases

efficient

solutions

can

be

achieved

by

data

reduction

algorithms

preserving

the

relevant

information

For

all

the

data

sets

an

expert

diagnosis

was

available

while

for

a

subgroup

for

patients

also

results

from

angiography

were

obtained

Two

series

of

experiments

were

conducted

using

either

the

expert

diagnosis

of

the

results

from

angiography

as

target

output

cases

of

the

former

population

and

cases

of

the

latter

one

were

taken

for

training

-LRB-

leaving

the

rest

as

a

test

set

for

evaluating

the

generalization

performance

-RRB-

Alternatively

the

diagnosis

from

expert

readings

of

the

scintgrams

or

from

angiography

were

used

as

target

output

The

prototype

network

consisted

of

input

units

-LRB-

corresponding

to

segmental

thallium

uptake

and

washout

values

-RRB-

to

hidden

units

and

to

output

units

-LRB-

depending

on

the

type

of

diagnosis

-RRB-

Different

variations

of

the

backpropagation

learning

rule

-LSB-

-RSB-

were

applied

The

goal

for

the

initial

studies

was

to

find

the

network

with

the

best

performance

on

the

test

set

The

test

set

was

varied

in

order

to

achieve

this

goal

while

keeping

it

as

small

as

possible

to

increase

the

confidence

level

of

the

results

Results

were

to

for

the

expert

output

and

to

for

the

angiography

results

In

a

subsequent

study

-LRB-

described

in

-LSB-

-RSB-

-RRB-

the

goal

was

to

evaluate

the

robustness

of

the

method

over

different

conditions

Five

different

training

sets

were

selected

from

the

original

populations

and

the

average

results

over

all

runs

were

plotted

Three

different

outputs

were

used

either

one

output

unit

indicating

the

presence

of

CAD

or

two

units

representing

the

location

of

the

disease

or

two

units

representing

the

extent

of

disease

The

results

showed

that

the

network

could

predict

the

expert

's

readings

far

better

than

the

angiography

results

While

for

the

former

the

results

indicated

that

the

diagnostic

performance

of

the

neural

network

appeared

potentially

promising

the

network

trained

on

the

angiography

data

did

not

achieve

a

level

that

could

be

usefully

applied

in

clinical

practice

Thus

the

goal

of

the

present

work

was

to

improve

the

results

on

predicting

the

angiography

readings

Although

this

is

a

very

specific

goal

this

application

can

serve

as

a

testbed

for

similarly

structured

medical

applications

Therefore

in

the

next

section

we

briefly

review

some

aspects

common

to

many

diagnostic

tasks

in

clinical

medicine

Clinical

Diagnosis

and

Supervised

Learning

Clinical

decision

making

and

diagnosis

frequently

involves

the

identification

of

normal

and

abnormal

patterns

obtained

from

diagnostic

tests

In

a

task

of

classifying

patterns

representing

clues

about

a

disease

based

on

supervised

neural

networks

the

goal

is

to

find

an

optimum

network

which

is

likely

to

yield

reproducible

and

accurate

results

in

future

routine

use

Given

a

set

of

pathological

and

normal

cases

the

task

is

therefore

to

use

a

small

fraction

of

that

data

for

training

and

to

keep

a

large

fraction

for

evaluation

-LRB-

test

set

-RRB-

It

is

generally

hypothesized

that

the

network

with

the

best

prediction

rate

on

the

test

set

will

also

lead

to

high

prediction

rates

when

presented

with

novel

data

-LRB-

not

available

during

the

network

design

-RRB-

Thus

as

it

was

done

in

the

above-cited

work

several

subsets

of

the

sample

set

have

to

be

selected

for

several

training

runs

to

identify

an

optimum

network

In

general

the

larger

the

test

set

is

in

relation

to

the

training

set

the

more

reliable

the

results

get

The

application

described

above

exemplifies

several

aspects

of

clinical

decision

making

that

may

impede

this

kind

of

application

of

supervised

neural

networks

diagnostic

tests

may

be

associated

with

risks

to

the

patient

such

as

morbidity

and

mortality

for

invasive

tests

-LRB-

e.g.

angiography

-RRB-

or

radiation

burden

-LRB-

scintigraphy

-RRB-

Consequently

very

often

only

a

rather

small

number

of

training

samples

are

available

for

the

same

reasons

plus

ethical

reasons

prohibiting

the

application

of

invasive

techniques

to

healthy

subjects

training

sets

are

often

unbalanced

For

instance

in

the

case

of

angiography

the

number

of

normal

cases

is

considerably

smaller

than

the

number

of

CAD

cases

-LRB-

vs.

-RRB-

Furthermore

normal

reference

cases

may

include

a

systematic

bias

since

angiography

is

only

conducted

if

there

is

sufficient

evidence

suggesting

CAD

for

the

routine

clinical

use

of

diagnostic

systems

the

indication

of

confidence

values

i.e.

values

expressing

the

probability

of

a

diagnosis

being

correct

are

of

critical

importance

For

neural

networks

this

can

be

achieved

by

an

indication

of

how

many

percent

of

cases

were

classified

correctly

at

a

certain

threshold

value

of

activation

at

the

output

units

To

arrive

at

reliable

confidence

values

many

experiments

with

large

test

sets

are

advisable

This

again

puts

restrictions

on

the

size

of

the

training

sets

Similar

problems

have

also

been

encountered

in

other

clinical

applications

conducted

by

the

Austrian

Research

Institute

for

Artificial

Intelligence

such

as

the

classifcation

of

EEG

data

-LRB-

-LSB-

-RSB-

-RRB-

or

the

detection

of

alarms

in

CTG

signals

in

laboring

women

The

research

reported

here

aimed

at

exploring

several

methods

and

alternative

learning

rules

to

deal

with

such

problems

all

in

the

context

of

the

application

of

detecting

CAD

from

myocardial

scintigrams

Methodology

of

the

Conducted

Experiments

To

study

and

compare

the

different

neural

network

types

several

series

of

experiments

have

been

conducted

For

all

these

experiments

the

focus

of

the

study

was

on

the

prediction

of

the

presence

of

CAD

with

angiography

as

standard

As

mentioned

above

from

a

total

data

set

of

cases

always

-LRB-

normal

-RRB-

were

included

in

the

training

set

and

-LRB-

-RRB-

in

the

test

set

To

cross-evaluate

the

consistency

of

each

method

and

to

conduct

a

wide-spread

search

for

the

optimum

network

five

different

training

sets

of

the

same

size

and

distribution

and

with

minimal

overlap

were

chosen

-LRB-

henceforth

labeled

through

-RRB-

Each

rule

with

each

parameter

setting

was

trained

on

all

five

sets

and

tested

with

each

of

the

five

corresponding

test

sets

respectively

The

diagnostic

performance

of

clinical

tests

is

often

specified

as

a

pair

of

sensitivity

-LRB-

percentage

of

correctly

classified

pathological

cases

-RRB-

and

specificity

-LRB-

percentage

of

correctly

classified

normal

cases

-RRB-

values

Thes

two

values

are

independent

on

each

other

in

that

increasing

one

of

them

will

decrease

the

other

Therefore

for

a

more

accurate

descriptio

socalled

ROC

curves

that

plot

sensitivity

over

specificty

are

given

to

assess

the

interdependence

of

the

two

values

Since

there

is

only

one

output

unit

sensitivity

and

specificity

can

be

altered

for

a

given

network

by

shifting

the

threshold

that

discriminates

between

normal

and

abnormal

-LRB-

decides

whether

the

output

is

interpreted

as

or

-RRB-

Theoretically

any

threshold

in

the

interval

-RSB-

-LSB-

-LRB-

excluding

the

values

and

-RRB-

could

be

meaningful

since

the

criterion

is

optimal

separation

given

certain

constraints

-LRB-

such

as

a

certain

specificty

-RRB-

To

evaluate

each

network

the

threshold

was

varied

between

and

at

steps

of

For

any

given

specificity

value

-LRB-

corresponding

to

every

number

between

and

of

normal

cases

being

misclassified

-RRB-

the

maximum

achievable

sensitivity

value

-LRB-

the

minimum

number

of

pathological

cases

misclassified

-RRB-

was

recorded

For

the

purpose

of

the

present

study

we

plotted

sensitivity

-LRB-

on

the

y-axis

-RRB-

over

specificity

values

-LRB-

on

the

x-axis

-RRB-

instead

of

ROC

curves

either

as

polygone

curves

or

as

bar

graphs

Only

specificity

values

above

-LRB-

above

correctly

classified

normals

-RRB-

were

considered

since

results

below

that

are

of

limited

utility

If

single

values

could

not

be

computed

-LRB-

meaning

that

shifting

the

threshold

by

resulted

in

a

jump

of

false

positives

by

two

-RRB-

then

they

were

linearly

interpolated

If

values

at

specificities

above

-LRB-

or

false

positives

-RRB-

could

not

be

computed

then

was

assumed

-LRB-

extrapolation

would

have

been

too

error-prone

-RRB-

Thus

results

in

this

area

are

not

always

reliable

Fig.

shows

such

a

curve

for

the

best

multilayer

perceptron

It

demonstrates

that

sensitivity

Figure

A

sensitivity

over

specificity

curve

for

the

best

network

trained

on

the

angiography

standard

using

multilayer

perceptrons

and

backpropagation

critically

drops

below

when

specificity

is

raised

to

just

or

more

Thus

the

results

are

of

little

use

in

practical

application

Alternatively

bar

graphs

are

used

to

show

the

same

results

Wherever

possible

hidden

units

were

used

-LRB-

since

this

was

the

value

chosen

to

be

optimum

in

the

previous

studies

-LSB-

-RSB-

-RRB-

Multilayer

Perceptrons

and

Radial

Basis

Function

Networks

In

the

original

long-term

study

-LRB-

-LSB-

-RSB-

-RRB-

only

multilayer

perceptrons

using

the

backpropagation

learning

rule

were

used

It

is

probably

fair

to

say

that

this

type

of

network

belongs

to

the

most

widely

applied

ones

in

medicine

-LRB-

e.g.

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-RRB-

and

elsewhere

As

is

widely

known

backpropagation

is

extremely

slow

and

requires

a

large

number

of

training

steps

In

the

case

of

the

angiography

data

the

number

of

training

steps

-LRB-

each

step

comprising

one

cycle

through

backpropagation

based

on

a

randomly

chosen

training

pair

-RRB-

ranged

from

to

Compared

to

other

applications

this

is

rather

low

but

still

required

long

training

times

given

the

large

number

of

runs

that

had

to

be

performed

during

the

design

of

the

optimum

network

Another

fact

about

backpropagation

is

that

it

works

best

if

training

sets

are

balanced

i.e.

if

the

numbers

of

positive

and

negative

cases

are

about

equal

This

was

not

possible

to

achieve

with

the

angiography

data

since

only

out

of

cases

were

normal

but

any

training

set

smaller

than

was

not

sufficient

for

acceptable

results

Thus

the

training

sets

were

chosen

so

as

to

keep

the

ratio

of

pathological

and

normal

cases

about

equal

for

training

and

test

sets

No

duplication

of

cases

-LRB-

a

possible

way

of

achieving

balance

-RRB-

was

conducted

so

as

not

to

bias

the

network

toward

individual

cases

Consequently

the

results

achieved

are

probably

influenced

by

the

negative

impact

of

the

unbalanced

training

sets

An

indication

of

this

are

the

rather

low

specificity

values

for

high

sensitivity

-LRB-

as

in

fig.

-RRB-

suggesting

a

bias

toward

the

more

numerous

pathological

cases

An

alternative

network

which

is

becoming

increasingly

popular

for

practical

applications

is

the

radial

basis

function

network

-LRB-

-LSB-

-RSB-

-LSB-

-RSB-

-RRB-

Such

networks

have

even

been

applied

to

a

similar

case

of

detection

of

CAD

in

thallium

scintigrams

-LRB-

-LSB-

-RSB-

-RRB-

The

difference

to

multilayer

perceptrons

is

best

described

when

considering

the

so-called

decision

regions

each

unit

in

either

network

can

apply

to

the

space

of

input

patterns

MLP

units

divide

the

space

into

two

half-spaces

by

hyperplanes

-LRB-

reflected

in

the

linear

term

of

the

weighted

sum

-RRB-

while

RBFN

units

cut

out

hyperspherical

regions

around

training

points

-LRB-

reflected

by

the

quadratical

term

of

the

Euclidean

distance

and

by

the

Gaussian

activation

function

for

an

overview

see

-LSB-

-RSB-

and

-LSB-

-RSB-

-RRB-

Theoretically

an

RBFN

can

be

built

with

only

positive

examples

-LRB-

or

at

least

with

fewer

negative

than

positive

examples

-RRB-

since

the

decision

regions

are

closed

and

can

be

set

on

the

basis

of

single

training

points

while

MLPs

need

pairs

of

positive

and

negative

samples

to

set

the

hyperplanes

What

training

time

is

concerned

RBFNs

have

a

decisive

advantage

over

MLPs

Since

the

first

layer

-LRB-

the

hidden

layer

-RRB-

is

initialized

by

setting

the

centers

of

the

hyperspheres

according

to

the

training

samples

only

the

connections

from

the

hidden

units

to

the

output

units

have

to

be

trained

Since

this

part

is

a

linear

associator

the

simple

delta

rule

-LRB-

-LSB-

-RSB-

-RRB-

is

sufficient

which

on

average

needs

fewer

cycles

and

less

CPU

time

to

achieve

good

results

The

biggest

disadvantage

of

RBFNs

is

that

they

can

be

too

restrictive

on

the

basis

of

their

closed

decision

regions

Given

a

pattern

of

input

values

one

unit

becomes

most

sensitive

only

to

a

certain

small

range

of

values

everything

above

and

below

counts

as

negative

-LRB-

see

also

-LSB-

-RSB-

-RRB-

In

other

words

patterns

are

classified

according

to

a

nearest-neighbor

criterion

given

several

prototypes

-LRB-

represented

by

several

hidden

units

-RRB-

It

depends

on

the

application

whether

this

is

appropriate

In

the

case

of

scinitgrams

at

the

input

it

might

not

be

at

least

not

totally

There

the

input

data

is

derived

from

concentrations

of

over

several

regions

of

the

heart

Over

large

parts

of

the

input

the

criterion

for

CAD

is

certainly

whether

this

concentration

is

very

high

or

very

low

not

whether

it

is

at

a

certain

value

-LRB-

and

neither

above

nor

below

-RRB-

In

this

sense

aspects

of

the

distruibution

of

the

input

data

overlooked

in

most

applications

critically

decide

upon

the

appropriateness

of

either

MLP

or

RBFN

In

many

cases

however

it

is

very

difficult

or

impossible

to

determine

the

data

distribution

and

thus

decide

upon

which

type

of

network

-LRB-

MLP

or

RBFN

-RRB-

is

most

appropriate

Fig.

shows

the

results

of

applying

an

RBFN

as

described

in

-LSB-

-RSB-

to

the

angiography

data

-LRB-

black

bars

-RRB-

compared

to

the

previous

results

from

backpropagation

-LRB-

white

bars

-RRB-

Since

78 89 100 10203

78 89 100 10203

Figure

The

results

from

radial

basis

function

networks

-LRB-

black

bars

-RRB-

compared

to

multilayer

perceptrons

with

backpropagation

-LRB-

white

bars

-RRB-

depicted

as

sensitivity

over

specificity

curves

for

all

five

training

sets

-LRB-

evaluated

over

the

corresponding

test

sets

-RRB-

there

are

training

vectors

a

full-fletched

RBFN

consists

of

hidden

units

-LRB-

black

bars

-RRB-

Backpropagation

was

done

with

a

learning

rate

of

and

a

momentum

factor

of

-LRB-

see

-LSB-

-RSB-

-RRB-

until

all

training

cases

were

learned

with

a

tolerance

of

The

delta

rule

to

train

the

hidden-to-output

units

of

the

RBFN

was

used

with

a

learning

rate

of

for

steps

For

both

learning

rules

training

pairs

were

presented

randomly

The

results

show

that

the

RBFN

with

hidden

units

was

consistently

better

in

only

one

of

the

five

cases

while

in

the

other

four

it

tended

to

be

inferior

This

indicates

that

indeed

this

method

might

be

too

restrictive

unless

some

favorable

training

set

is

chosen

Initializing

Multilayer

Perceptrons

As

described

in

the

previous

section

a

decisive

advantage

of

RBFNs

is

the

ease

with

which

they

can

be

trained

Weights

between

the

input

and

hidden

layer

are

directly

set

through

an

initialization

procedure

namely

identical

to

the

input

values

of

one

training

sample

In

the

subsequent

training

of

the

hidden-to-output

weights

with

the

delta

rule

convergence

is

guaranteed

and

the

network

can

not

get

stuck

in

local

minima

As

a

result

a

single

training

run

is

sufficient

to

guarantee

an

optimum

result

for

a

current

parameter

setting

-LRB-

which

too

is

usually

smaller

than

for

the

MLP

-RRB-

while

an

MLP

usually

has

to

be

trained

several

times

with

different

inital

-LRB-

random

-RRB-

weight

configurations

in

order

to

rule

out

local

minima

To

make

an

MLP

profit

from

similar

effects

we

developed

a

simple

method

of

initializing

the

input-to-hidden

layer

weights

-LRB-

independently

also

reported

in

-LSB-

-RSB-

-RRB-

The

basic

idea

is

a

direct

analogy

to

the

initialization

of

RBFNs

namely

setting

the

decision

borders

-LRB-

the

surface

dividing

the

decision

regions

-RRB-

according

to

training

samples

For

an

MLP

unit

given

a

pair

of

a

positive

and

a

negative

sample

this

easily

translates

to

setting

a

hyperplane

which

optimally

divides

the

two

points

This

is

the

case

if

the

hyperplane

dissects

the

line

connecting

the

two

points

halfway

between

the

points

while

being

perpendicular

to

that

line

-LRB-

see

fig.

for

the

three-dimensional

case

-RRB-

If

the

two

points

are

cluster

centers

this

kind

of

separation

is

called

a

Voronoi

tesselation

Through

solving

the

corresponding

set

of

equations

-LRB-

defining

the

hyperplane

-RRB-

the

following

formulas

for

calculating

the

weights

and

the

thresholds

are

easily

derived

-LRB-

see

-LSB-

-RSB-

and

-LSB-

-RSB-

wij

x?ji

x?ji

j

n

i

x?ji

x?ji

yj

n

i

wijxi

j

-LRB-

-RRB-

where

x?ji

and

x?ji

are

the

i-th

input

values

of

the

j-th

positive

and

negative

sample

respective

ly

-LRB-

assuming

that

there

be

one

hidden

unit

j

for

each

pair

-RRB-

yj

is

the

net

input

of

the

j-th

unit

before

applying

a

sigmoid

transfer

function

This

methodology

nicely

reflects

the

abovementioned

need

for

negative

samples

in

an

MLP

It

is

clear

that

among

the

large

number

of

X?j

X?j

yj

j

Figure

Setting

a

hyperplane

corresponding

to

the

weighted

sum

in

the

propagation

rule

such

that

it

optimally

separates

a

positive

-LRB-

X

-RRB-

and

a

negative

training

sample

-LRB-

X

-RRB-

possible

pairs

in

practical

applications

it

is

advisable

to

choose

only

a

subset

It

is

also

clear

that

an

optimum

outcome

of

such

an

initialization

procedure

can

only

be

expected

if

not

arbitrary

data

points

are

chosen

but

cluster

centers

derived

from

cluster

analysis

-LRB-

e.g.

from

LVQ

-LSB-

-RSB-

-RRB-

But

as

the

results

below

demonstrate

cluster

analysis

need

not

be

necessary

To

improve

results

among

the

hidden

units

there

should

be

detectors

of

both

positive

and

negative

cases

Negative

detectors

are

easily

derived

by

switching

x?ji

and

x?ji

in

the

above

formulas

Setting

the

hidden-to-output

weights

can

be

done

the

same

way

as

in

the

RBFN

i.e.

by

applying

the

delta

rule

Fig.

shows

the

rather

surprising

results

of

applying

the

initialization

procedure

based

on

randomly

chosen

pairs

of

positive

and

negative

samples

of

the

angiography

data

plus

the

delta

rule

for

steps

with

a

learning

rate

of

-LRB-

grey

bars

backpropagation

is

depicted

as

white

bars

as

before

-RRB-

To

make

immediate

comparsion

with

the

backpropagation

results

possible

a

constant

number

of

hidden

units

was

chosen

for

all

runs

In

many

cases

-LRB-

and

-RRB-

initialization

plus

delta

rule

-LRB-

the

direct

analogy

to

RBFNs

-RRB-

lead

to

improved

results

over

backpropagation

Especially

in

areas

of

specificity

between

and

the

curves

are

considerably

flatter

thus

permitting

high

sensitivity

values

These

improvements

raise

the

neural

network

method

applied

to

scintigrams

plus

output

from

angiography

to

a

level

that

might

be

clinically

useful

Another

important

application

of

the

initialization

method

can

be

derived

from

these

experiments

The

results

show

that

the

goodness

of

the

classification

after

simple

initialization

-LRB-

i.e.

all

weights

are

initialized

no

delta

rule

-RRB-

strongly

correlates

with

the

goodness

of

results

of

the

final

trained

networks

-LRB-

delta

rule

and

original

backpropagation

-RRB-

In

other

words

the

in

78 89 100 10203

78 89 100 10203

Figure

The

results

from

MLP

initialization

plus

delta

rule

-LRB-

dark

grey

bars

-RRB-

compared

to

multilayer

perceptrons

with

backpropagation

-LRB-

white

bars

-RRB-

depicted

as

sensitivity

over

specificity

curves

for

all

five

training

sets

-LRB-

evaluated

over

the

corresponding

test

sets

-RRB-

itializtion

can

serve

as

a

good

indicator

for

which

training

set

might

be

optimal

for

maximizing

the

final

prediction

rate

-LRB-

as

defined

above

-RRB-

Thus

it

can

be

used

for

screening

a

large

number

of

training

sets

so

as

to

choose

a

small

number

of

promising

candidates

for

the

actual

gradient

descent

training

Valuable

time

in

the

design

of

the

network

solution

can

be

gained

this

way

Conic

Section

Function

Networks

-LRB-

CSFN

-RRB-

To

combine

the

advantages

of

MLPs

and

RBFNs

a

unification

of

the

two

has

been

developed

called

Conic

Section

Function

Networks

-LRB-

CSFN

see

-LSB-

-RSB-

-RRB-

It

is

based

on

the

observa

tion

that

both

hyperplane

and

hypersphere

are

special

cases

of

generalized

conic

section

functions

In

the

n-dimensional

space

of

input

patterns

each

decision

border

is

seen

as

the

intersection

of

an

n

dimensional

hypercone

with

that

space

-LRB-

fig.

shows

an

example

for

n

in

a

frontal

view

-RRB-

According

to

this

the

net

input

of

a

CSFN

unit

is

given

through

the

X

X

S

a

plane

of

data

points

plane

of

degenerated

cone

cones

at

different

opening

angles

Figure

A

three-dimensional

cone

in

a

frontal

view

intersecting

a

two-dimensional

input

space

forming

a

circle

ellipse

or

-LRB-

being

degenerated

as

a

plane

-RRB-

a

straight

line

The

tip

of

the

cone

and

the

angle

are

kept

constant

while

the

opening

angle

is

altered

following

equation

-LRB-

for

a

derivation

see

-LSB-

-RSB-

-RRB-

yj

n

i

-LRB-

xi

si

-RRB-

aij

n

i

xi

sij

cos

j

xn

-LRB-

-RRB-

where

j

is

the

opening

angle

of

the

cone

j

-LRB-

hidden

unit

j

-RRB-

One

can

easily

see

that

this

formula

conains

two

major

parts

roughly

corresponding

to

an

MLP

-LRB-

weighted

sum

-RRB-

and

an

RBF

term

-LRB-

Euclidean

distance

-RRB-

For

this

we

have

to

equate

the

a

factors

with

the

weights

in

an

MLP

and

the

s

factors

with

the

weights

or

better

still

offsets

or

center

coordinates

of

an

RBFN

Contrary

to

work

in

-LSB-

-RSB-

the

two

sets

of

degrees-of-freedom

are

not

identical

but

separate

This

results

in

a

doubling

of

resources

by

introducing

two

degrees-of-freedom

per

connection

certainly

one

major

drawback

of

the

CSFN

More

precisely

all

aij

are

given

through

aij

wij

cos

-LRB-

j

j

-RRB-

i

n

an

j

sin

j

j

-LRB-

-RRB-

The

parameters

sij

are

set

identical

to

the

coordinates

of

a

positive

class

sample

like

in

the

RBFN

-LRB-

or

in

the

above-described

initializiation

of

MLPs

-RRB-

si

n

determines

and

can

be

chosen

such

that

-LRB-

degrees

-RRB-

The

most

attractive

feature

of

CSFNs

is

that

there

exists

one

major

parameter

-LRB-

-RRB-

the

varia

tion

of

which

can

gradually

shift

a

CSFN

unit

from

being

an

MLP

unit

-LRB-

separating

the

input

space

by

a

hyperplane

Here

the

cone

degenrates

to

a

being

a

plane

-RRB-

to

being

an

RBFN

unit

-LRB-

-RRB-

separating

the

space

by

a

hypersphere

-RRB-

where

remains

constant

This

one

parameter

can

rather

easily

be

adapted

by

one

of

several

rules

-LRB-

also

called

conefolding

-RRB-

for

instance

a

gradient

descent

rule

maximizing

the

separability

of

each

unit

This

rule

is

applied

as

follows

First

the

network

is

initialized

for

instance

by

the

method

of

setting

hyperplanes

according

to

eq

-LRB-

-RRB-

Then

an

input

pattern

is

presented

and

activations

are

propagated

to

the

hidden

layer

Finally

is

adapted

according

to

the

following

equation

j

yj

sin

-LRB-

-RRB-

si

n

cos

-LRB-

-RRB-

yj

sin

j

j

ffi

ffi

if

y?j

and

wjk

if

yj

and

wjk

otherwise

-LRB-

-RRB-

where

y?j

and

y?j

are

the

two

major

parts

of

the

net

input

-LRB-

equation

-LRB-

-RRB-

-RRB-

y?j

n

i

-LRB-

xi

sij

-RRB-

wij

y?j

n

i

xi

sij

-LRB-

-RRB-

such

that

yj

y

-LRB-

cos

-LRB-

-RRB-

-RRB-

si

n

sin

-LRB-

-RRB-

y

cos

j

-LRB-

-RRB-

andwjk

is

the

weight

from

hidden

unit

j

to

the

output

unit

-LRB-

either

also

initialized

or

acquired

through

learning

indicating

whether

the

unit

is

a

positive

or

negative

detector

-RRB-

A

derivation

can

again

be

found

in

-LSB-

-RSB-

If

many

closed

regions

exist

in

the

input

data

then

some

of

the

hidden

units

starting

off

as

MLP

units

will

learn

to

set

close

to

and

thus

form

RBFs

Thus

the

above

problem

of

the

unknown

appropriateness

of

the

type

of

decision

region

is

largely

solved

automatically

The

reverse

approach

starting

as

RBFN

and

letting

some

units

open

their

regions

is

possible

too

but

was

not

tested

in

this

project

A

possible

negative

effect

can

be

overfitting

of

the

training

data

which

can

be

controled

by

a

parameter

limiting

the

maximum

change

of

The

effect

of

adapting

can

be

expected

to

be

larger

if

not

single

data

but

cluster

centers

are

used

for

initialization

Therefore

for

a

second

round

of

experiments

a

cluster

analysis

done

by

the

LVQ

method

-LRB-

-LSB-

-RSB-

-LSB-

-RSB-

p.

-RRB-

was

applied

to

all

five

training

sets

resulting

in

to

positive

and

to

negative

clusters

-LRB-

positive

and

negative

samples

were

clustered

separately

-RRB-

Fig.

shows

the

effect

of

applying

learning

of

both

with

single

data

-LRB-

left

column

-RRB-

and

cluster

center

initialization

-LRB-

right

column

-RRB-

Dark

grey

bars

depict

the

results

of

initialization

light

grey

bars

depict

the

results

after

cone-folding

Eq

-LRB-

-RRB-

was

applied

twice

with

each

training

pair

-LRB-

which

empirically

turned

out

to

be

an

optimum

number

in

this

application

-RRB-

leading

to

an

additional

training

run

of

only

cycles

Subsequently

the

delta

rule

was

applied

for

cycles

during

which

training

pairs

were

presented

randomly

The

graphs

show

that

this

type

of

gradient

descent

learning

on

lead

to

consistent

albeit

sometimes

small

improvements

especially

in

the

interesting

areas

of

specificity

of

to

As

expected

improvements

were

larger

after

the

cluster

initialization

Notable

exceptions

are

training

set

and

to

some

extent

Discussion

In

a

comparative

summary

figs.

and

show

the

results

of

multilayer

perceptrons

-LRB-

MLPs

-RRB-

with

backpropagation

-LRB-

white

diamonds

-RRB-

radial

basis

function

networks

-LRB-

RBFNs

black

circles

-RRB-

conic

section

function

networks

-LRB-

CSFNs

-RRB-

with

initialization

-LRB-

dark

grey

crosses

-RRB-

and

with

cone

folding

-LRB-

gradient

descent

learning

over

light

grey

triangles

-RRB-

Comparisons

for

both

initialization

by

single

data

points

and

by

cluster

centers

are

shown

Initialization

of

RBFNs

by

cluster

centers

has

also

been

tried

but

lead

to

inferior

results

-LRB-

and

was

thus

omitted

-RRB-

The

graphs

show

that

MLPs

are

consistently

outperformed

with

the

best

results

due

to

CSFNs

Interestingly

RBFNs

CSFNs

with

single

data

initialization

and

CSFNs

with

cluster

center

initialization

seem

to

complement

each

other

in

that

in

all

five

training

sets

one

of

them

produces

the

best

results

This

appears

to

indicate

a

high

dependency

on

the

chosen

training

set

which

is

not

surprising

given

the

small

number

of

training

samples

It

also

indicates

that

by

using

all

of

the

described

neural

network

methods

satisfying

results

can

still

be

achieved

in

most

cases

single

single

single

single

single

single

single

single

single

single

Figure

The

results

from

gradient

descent

learning

over

the

major

CSFN

parameter?or

cone-folding

-LRB-

light

grey

bars

-RRB-

compared

to

the

results

after

MLP

initialization

-LRB-

dark

grey

bars

-RRB-

depicted

as

sensitivity

over

specificity

curves

for

all

five

training

sets

-LRB-

evaluated

over

the

corresponding

test

sets

-RRB-

Cluster

Cluster

Cluster

Cluster

Cluster

Figure

The

comparative

results

from

backpropagation

-LRB-

white

area

diamonds

-RRB-

MLP

initialization

plus

delta

rule

-LRB-

dark

grey

area

crosses

-RRB-

and

cone-folding

-LRB-

light

grey

triangles

-RRB-

for

initialization

with

cluster

centers

depicted

as

sensitivity

over

specificity

curves

for

all

five

training

sets

-LRB-

evaluated

over

the

corresponding

test

sets

-RRB-

Since

for

training

set

backpropagation

leads

to

better

results

the

areas

are

depicted

in

reverse

Single

Single

Single

Single

Single

Figure

The

comparative

results

from

backpropagation

-LRB-

white

area

diamonds

-RRB-

MLP

initialization

plus

delta

rule

-LRB-

dark

grey

area

crosses

-RRB-

and

cone-folding

-LRB-

light

grey

triangles

-RRB-

for

initialization

with

single

-LRB-

randomly

chosen

-RRB-

data

point

pairs

depicted

as

sensitivity

over

specificity

curves

for

all

five

training

sets

-LRB-

evaluated

over

the

corresponding

test

sets

-RRB-

The

results

confirm

the

initial

observation

that

clean

RBFNs

overall

might

be

inappropriate

for

this

given

task

The

MLP

initializations

show

consistently

better

results

than

RBFNs

-LRB-

with

the

exception

of

one

training

set

-RRB-

The

further

improvements

of

CSFNs

show

that

the

optimal

solutions

lie

in

between

the

two

extremes

MLP

or

RBFN

-LRB-

open

or

closed

decision

regions

-RRB-

but

closer

to

the

MLP

type

Therefore

this

application

demonstrates

the

ability

of

CSFNs

to

help

solve

the

problem

of

appropriate

decision

regions

automatically

From

the

comparative

results

the

following

strategy

for

future

similar

applications

can

be

derived

Use

initialization

to

quickly

scan

over

several

training

sets

and

identify

the

most

promising

candidates

Then

use

cone-folding

to

improve

or

fine-tune

the

results

Where

neither

of

them

leads

to

acceptable

results

RBFNs

or

MLPs

-LRB-

in

that

order

-RRB-

can

still

be

tried

Coming

back

to

the

aspects

of

clinical

decision

making

with

supervised

neural

networks

-LRB-

section

-RRB-

the

results

suggest

improved

solutions

for

these

kinds

of

application

In

particular

MLP

initialization

and

cone-folding

complementing

conventional

techinques

like

RBFNs

have

proven

capable

of

achieving

useful

results

even

in

the

case

of

extremely

small

training

sets

and

a

large

variation

between

several

randomly

chosen

ones

RBFNs

and

CSFNs

have

furthermore

proven

to

be

less

sensitive

to

unbalanced

training

sets

As

figs.

and

depict

large

sensitivity

can

be

paired

with

relatively

large

specificty

suggesting

less

bias

toward

the

more

numerous

pathological

cases

When

looking

at

the

best

results

-LRB-

e.g.

training

sets

and

for

the

cluster

method

fig.

-RRB-

one

can

clearly

see

the

essential

improvements

over

the

previous

results

from

backpropagation

-LRB-

fig.

-RRB-

Sensitivity

considerably

drops

below

only

at

specificity

values

of

or

The

identification

of

patients

suffering

from

significant

CAD

is

clinically

important

to

guide

in

proper

patient

management

In

particular

the

identification

of

a

subgroup

of

patients

that

may

benefit

from

interventional

procedures

such

as

percutaneous

transluminal

angioplasty

or

coronary

bypass

surgery

to

restore

adequate

nutritional

blood

flow

to

the

myocardium

is

a

primary

diagnostic

concern

in

clinical

cardiology

Currently

coronary

angiography

is

the

method

of

choice

to

define

precisely

coronary

anatomy

prior

to

interventional

procedures

However

the

quest

for

noninvasive

techniques

that

may

permit

to

select

accurately

patients

at

risk

is

still

a

major

research

topic

While

the

prototype

of

the

neural

network

as

utilized

in

a

previous

study

did

not

meet

the

strict

requirements

of

diagnostic

accuracy

to

be

of

clinical

utility

these

new

techniques

presented

in

the

present

study

may

overcome

some

limitations

so

that

applicability

in

a

clinical

setting

may

ultimately

be

achieved

However

further

research

especially

with

respect

to

different

scintigraphic

techniques

such

as

SPECT

imaging

and

an

extension

of

the

case

data

base

is

needed

before

clinical

utility

can

be

reassessed

Conclusion

In

this

paper

we

have

presented

results

from

experiments

with

several

different

types

of

feedforward

neural

network

applied

to

the

problem

of

detecting

coronary

artery

disease

-LRB-

CAD

-RRB-

in

planar

thallium

scintigrams

while

using

readings

from

angiography

as

the

reference

By

introducing

the

novel

technique

of

initializing

multilayer

perceptrons

-LRB-

MLP

-RRB-

and

the

novel

conic

section

function

network

-LRB-

CSFN

-RRB-

we

have

succeeded

in

improving

the

results

of

this

particular

task

so

as

to

make

it

more

clinically

useful

The

experiments

have

shown

that

MLPs

with

backpropagation

can

be

among

the

poorest

methods

in

the

presented

collection

of

learning

rules

However

we

do

not

want

to

simply

bash

that

kind

of

popular

neural

network

model?many

discussions

with

similar

criticism

have

sprung

up

in

neural

network

literature

since

backpropagation

has

become

popular?but

wanted

to

contribute

to

a

more

general

discussion

around

practical

applications

of

neural

networks

in

medicine

-LRB-

and

in

other

domains

-RRB-

We

have

discussed

two

extremes

of

a

spectrum

of

neural

network

classifiers

with

respect

to

their

propagation

rules

-LRB-

and

decision

borders

-RRB-

and

have

introduced

and

applied

a

novel

neural

network

type

that

can

place

itself

on

any

point

between

the

two

extremes

We

have

also

pointed

out

the

use

of

some

quick

methods

of

setting

weights

in

a

network

before

any

time-consuming

gradient-descent

learning

is

necessary

We

have

done

all

this

in

the

light

of

some

peculiar

aspects

of

clinical

diagnostic

tasks?such

as

limited

and

unbalanced

training

sets

or

the

need

for

large

evaluation

sets

to

extract

confidence

values

In

this

sense

this

work

can

be

viewed

as

a

contribution

to

improved

medical

applications

of

neural

networks

in

the

future

Acknowledgments

This

work

has

been

done

as

part

of

the

EC-funded

ESPRIT-II

project

NEUFODI

Neural

Networks

for

Forecasting

and

Diagnosis

Applications

The

Austrian

contribution

to

this

project

has

been

funded

by

the

Austrian

Industrial

Research

Promotion

Fund

under

grant

no.

I

wish

to

express

my

gratefulness

to

Prof.

Trappl

for

encouragment

and

inspiration

I

further

thank

my

colleagues

namely

Herbert

Wiklicky

Erich

Prem

Claudia

Ulbricht

and

Guenter

Linhart

for

fruitful

discussions

and

many

valuable

hints

for

this

research

Dr.

Heinz

Sochor

head

of

the

Working

Group

of

Cardiovascular

Nuclear

Medicine

at

the

Dept.

of

Cardiology

provided

ample

support

for

this

research

project

The

clinical

data

processing

for

this

work

was

funded

by

Informatica

Ges.m.b.H

References

OE?ae

ffl

AE

ss

AE?o

oe

-LRB-

-LRB-

OE

ae

fi?fi

fi

ffif

fi

OE

-RRB-

oe

ffff

ffiflfl

ffl

O

-LRB-

ss

ae

OE

ffiflffi

fi

B

A

C

A

ff

fl

ae

A

A

o

ffi

ae

ae

A

o

-LRB-

ff

ffl

ae

ss

A

O

ss

oe?ss

OE

A

C

C

C

AE

ffff

ffl

fflffi

O

oe

-LRB-

AE

B

C

O

o

C

OE

D

C

E

-LRB-

-RRB-

fi

C

A

C?AE

oe

AE

C

fi

OE

<?C,8,??ss??ae4<:=4??&??#:224:??<=??:990.>

ss

O

C

o

ff

ae

C

A

ae

-RRB-

fflfl

fi

AE

C

ae

ff

OE

ffl?fi

ffl

fi

ss

ss

o

ae

B

fflffl

-LRB-

oe

fl

ss

ss

AE

o

ae

B

F#

ffi

ss

ss

B

-LRB-

O

-LRB-

F

ffl

o

ae

fi

ss

o

-LRB-

fflfi?fl

O.

oe

Oo

e

fflfl

-LRB-

O

ss

ae

B

OE

AE

O

oe

AE

B

ss

O

F

B

oe

-LRB-

oe

-LRB-

-RRB-

oe

fl

ff

fl

angiography
conic section function networks
feedforward neural networks
multilayer perceptrons
radial basis function networks
thallium scintigraphy

clinical diagnosis and decision making
neural network initialization