A

Unified

Information

Extraction

and

Coding

Network

Colin

Fyfe

and

Douglas

R

McGregor

colin@cs.strath.ac.uk

douglas@cs.strath.ac.uk

Department

of

Computer

Science

University

of

Strathclyde

Glasgow

Ext.

June

Abstract

A

new

form

of

neural

network

is

introduced

which

is

shown

to

be

capable

of

performing

a

topology

preserving

feature

mapping

The

mapping

is

performed

in

steps

firstly

the

network

performs

a

Principal

Component

Analysis

and

then

secondly

makes

a

coding

of

each

Principal

Component

The

network

uses

only

simple

Hebbian

learning

in

each

section

and

is

characterised

by

simplicity

homogeneity

locality

of

information

use

and

parallelism

A

biologically

more

plausible

variation

of

the

original

model

is

shown

to

be

capable

of

the

same

mapping

but

at

a

loss

of

some

simplicity

Introduction

This

paper

introduces

a

novel

artificial

neural

network

architecture

which

performs

a

vector

quantization

Vector

quantization

is

used

in

signal

processing

applications

to

encode

a

high

dimensional

signal

in

order

to

minimise

processing/transmission

costs

The

basic

aim

is

to

associate

with

each

group

of

vectors

of

the

raw

data

a

code

which

uniquely

identifies

that

group

If

the

vectors

of

the

group

are

sufficiently

alike

and

the

code

is

sufficiently

representative

of

the

group

then

the

error

when

the

code

is

used

to

represent

a

vector

in

the

group

can

be

made

acceptably

small

The

most

common

type

of

artificial

neural

networks

used

to

perform

a

vector

quantization

is

that

developed

by

Kohonen

-LRB-

Figure

-RRB-

In

the

Kohonen

network

each

neuron

in

the

receiving

layer

sums

the

weighted

activations

it

receives

from

the

neurons

in

its

receptive

field

A

competition

takes

place

between

these

neurons

and

the

neuron

which

has

the

greatest

activation

is

declared

the

winner

its

weights

and

those

of

its

neighbours

are

moved

towards

the

input

pattern

's

weights

while

those

of

the

other

neurons

are

moved

further

away

The

net

result

is

that

if

the

same

input

pattern

or

one

similar

to

it

is

presented

again

the

same

neuron

is

most

likely

to

win

again

It

has

been

shown

that

after

a

suitable

training

period

the

neurons

of

the

second

layer

form

a

map

of

the

inputs

which

preserves

some

aspect

of

the

topology

of

the

input

data

Such

nets

are

known

as

self-orgainising

nets

as

there

is

no

teacher

input

to

the

net

such

as

is

provided

for

back-propagation

nets

or

perceptrons

We

will

introduce

a

new

Receptive

Field

Layer

of

Competing

Neurons

Layer

of

Input

Neurons

Competing

neuron

Figure

A

Kohonen

network

the

input

data

is

fed

forward

to

neurons

in

the

competing

layer

each

of

which

has

a

receptive

field

which

overlaps

with

its

neighbours

receptive

fields

The

network

organises

into

a

topology

preserving

mapping

type

of

artificial

neural

network

which

also

performs

a

topology

preserving

mapping

from

inputs

to

outputs

The

new

network

is

also

a

self-organising

net

The

technique

used

here

will

be

to

project

each

vector

of

the

input

data

onto

the

principal

components

-LRB-

see

below

-RRB-

of

the

input

data

's

covariance

matrix

and

then

code

each

direction

of

the

principal

components

in

a

topology

preserving

manner

Both

parts

of

this

process

will

be

performed

by

an

exceptionally

simple

neural

network

which

uses

a

simple

feedback

mechanism

The

design

of

the

network

is

based

on

biologically

plausible

principles

and

the

overiding

design

criterion

is

the

retention

of

simplicity

in

the

network

In

sections

and

we

describe

each

part

separately

in

section

we

describe

the

full

network

and

give

an

example

of

its

use

Information

Extraction

Principal

Component

Analysis

is

a

technique

for

finding

those

direction

-LRB-

s

-RRB-

of

multidimensional

data

which

contain

as

much

of

the

information

in

the

data

as

possible

By

projecting

the

data

onto

these

directions

we

retain

as

much

of

the

information

in

as

few

values

as

possible

It

is

well

known

that

the

optimal

linear

solution

to

this

projection

is

to

be

found

in

terms

of

the

eigenvectors

corresponding

to

the

largest

eigenvalues

of

the

input

data

covariance

matrix

Neural

networks

provide

a

novel

way

to

calculate

Principal

Components

from

on-line

data

It

has

been

shown

-LSB-

Fyfe

-RSB-

that

a

layer

of

interneurons

-LRB-

Figure

-RRB-

whose

weights

are

modified

using

simple

Hebbian

learning

can

be

used

to

calculate

Principal

Components

The

basic

network

was

shown

to

be

equivalent

to

Oja

's

Subspace

Algorithm

-LSB-

Oja

-RSB-

which

finds

the

Principal

of

the

input

data

The

learning

rule

for

the

interneuron

network

is

simple

Hebbian

Learning

Hebb

Principal

Subspace

is

the

space

spanned

by

the

Principal

Components

i.e.

the

weights

are

a

linear

combination

of

the

Principal

Components

-LSB-

Hebb

-RSB-

postulated

that

if

a

neuron

was

to

repeatedly

assist

in

activating

another

neuron

the

synaptic

efficiency

of

the

connection

between

the

neurons

would

be

enhanced

In

artificial

neural

network

terms

this

is

usually

interpreted

as

the

rule

If

wij

is

the

weight

of

the

connection

between

an

input

neuron

with

activation

xj

and

an

output

neuron

with

activation

yi

then

the

weight

of

the

connection

between

them

should

increase

by

the

amount

wij

xjyi

where

is

some

small

positive

learning

rate

There

is

one

major

defect

with

this

simple

implementation

of

the

Hebb

rule

if

no

corrective

action

is

taken

the

weights

tend

to

increase

without

limit

In

the

interneuron

network

the

raw

data

-LRB-

x

values

-RRB-

are

fed

forward

to

the

input

neurons

which

simply

sum

any

activation

which

reaches

them

and

output

the

sum

as

y

values

These

values

are

fedforward

via

weights

to

the

interneurons

which

again

simply

sum

their

inputs

to

give

the

z

values

these

are

then

fed

back

as

an

inhibitory

activation

via

the

same

weights

to

the

input

neurons

and

subtracted

from

the

x

inputs

There

is

no

explicit

weight

decay

normalisation

or

clipping

of

weights

in

the

interneuron

model

If

we

consider

the

network

as

a

transformation

from

inputs

x

to

interneuron

outputs

z

and

consider

the

effects

of

these

rules

on

individual

neurons

we

have

yi

xi

X

k

wkizk

-LRB-

-RRB-

zi

X

j

wijyj

initially

X

j

wijxj

-LRB-

-RRB-

Therefore

substituting

equation

in

the

simple

Hebbian

learning

rule

wij

yizj

-LRB-

xi

X

k

wkizk

-RRB-

zj

-LRB-

xizj

zj

X

k

wkizk

-RRB-

This

last

formulation

of

the

learning

rule

is

exactly

the

learning

rule

for

Oja

's

-LRB-

-RRB-

Subspace

Algorithm

Experimental

results

have

previously

been

given

to

substantiate

this

analysis

-LSB-

Fyfe

-RSB-

i.e.

the

weights

wi

into

each

interneuron

do

in

fact

converge

to

span

the

Principal

Subspace

However

as

a

model

of

biological

information

processing

the

model

suffers

from

one

major

defect

the

weights

of

the

connections

from

the

interneuron

z

to

the

input

neuron

y

are

assumed

to

be

identical

to

those

from

the

input

neuron

y

to

the

interneuron

z

An

extension

to

the

model

designed

to

rectify

this

defect

was

proposed

in

-LSB-

Fyfe

-RSB-

and

a

fuller

analysis

given

in

-LSB-

Fyfe

-RSB-

The

new

network

has

different

feedforward

and

feedback

weights

In

matrix

terms

we

write

y

x

Vz

-LRB-

-RRB-

z

Wy

initially

Wx

-LRB-

-RRB-

W

wzyT

-LRB-

-RRB-

VT

vzyT

-LRB-

-RRB-

where

x

is

the

input

vector

z

is

the

vector

of

interneuron

activations

y

is

the

vector

of

input

neuron

activations

and

the

initial

values

of

both

VT

and

W

are

small

random

numbers

not

correlated

in

any

way

with

each

other

Note

that

the

learning

rules

for

W

and

VT

are

identical

up

to

the

learning

rate

and

use

only

simple

Hebbian

learning

Since

we

are

interested

in

the

weights

to

and

from

the

interneurons

we

will

use

the

convention

that

wi

is

the

vector

of

weights

into

the

ith

xn

yn

interneuron

interneuron

w

v

inputs

outputs

inputneurons

Figure

The

interneurons

sum

the

weighted

y

values

which

are

themselves

calculated

by

subtracting

from

the

corresponding

x

values

the

weighted

sum

of

the

z

values

interneuron

and

vi

is

the

vector

of

weights

from

the

ith

interneuron

Then

wi

is

a

row

of

W

while

vi

is

a

column

of

V

The

basic

network

will

only

converge

to

the

Principal

Subspace

an

algorithm

involving

the

phased

creation

of

interneurons

was

shown

to

ensure

convergence

of

both

vi

and

wi

to

the

actual

Principal

Components

The

interneuron

network

is

characterised

by

simplicity

there

are

no

logistic

or

hyperbolic

functions

to

be

calculated

there

is

no

additional

computation

within

the

learning

rule

there

is

no

sequential

passing

back

of

errors

or

decay

terms

homogeneity

every

interneuron

is

performing

exactly

the

same

calculation

as

its

neighbours

every

input

neuron

is

performing

exactly

the

same

calculation

as

its

neighbours

locality

of

information

each

interneuron

uses

only

the

information

which

it

receives

from

its

own

connections

similarly

with

the

input

neurons

which

calculate

the

y

values

These

properties

of

the

network

appear

to

encourage

a

potential

parallel

implementation

however

it

is

clear

that

a

phased

creation

of

interneurons

does

not

utilise

this

inherent

parallelism

Using

Distance

Differences

Some

form

of

asymmetry

in

relation

to

the

interneuron

activity

is

necessary

to

force

convergence

to

the

actual

Principal

Components

All

previously

described

models

have

introduced

asymmetry

in

terms

of

the

phasing-in

of

interneuron

creation

However

in

real

biological

neural

networks

there

is

an

innate

asymmetry

in

terms

of

the

distances

between

neurons

This

will

manifest

itself

as

different

times

to

respond

to

a

signal

depending

on

the

distance

which

the

signal

must

travel

-LRB-

assuming

that

there

is

some

uniformity

in

the

speed

of

information

transfer

-RRB-

This

differential

is

used

in

a

new

model

where

different

interneurons

take

different

lengths

of

times

to

respond

to

the

input

signal

x.

Therefore

while

the

activation

from

the

input

neurons

is

transmitted

to

all

interneurons

at

the

same

time

each

interneuron

's

response

takes

a

different

length

of

time

to

feedback

to

the

input

neurons

Thus

this

negative

feedback

is

felt

and

used

in

a

phased

manner

Also

we

embed

the

learning

process

in

the

feedback

loop

so

that

we

now

postulate

a

learning

and

activation-transmission

V

W

Table

Results

of

the

Differential

Distance

Model

each

column

shows

the

converged

weights

between

one

interneuron

and

the

input

neurons

after

learning

on

data

from

independent

zero

mean

Gaussians

with

descending

variances

process

which

takes

place

in

the

order

in

which

the

following

equations

are

given

initial

value

of

y

x

-LRB-

-RRB-

z

Wy

-LRB-

-RRB-

y

y

vizi

-LRB-

-RRB-

wi

ziyT

-LRB-

-RRB-

vi

ziyT

-LRB-

-RRB-

Other

than

the

first

two

steps

-LRB-

the

acceptance

of

the

initial

activation

x

and

its

forward

transmission

to

the

interneurons

-RRB-

the

process

-LRB-

defined

by

Equations

and

-RRB-

is

repeated

for

each

interneuron

in

turn

This

corresponds

to

the

feedback

from

the

interneurons

being

received

at

different

times

-LRB-

perhaps

depending

on

the

physical

distance

which

the

activation

must

traverse

perhaps

depending

on

the

efficiency

of

transmission

of

the

interneuron

-RRB-

This

process

results

in

the

weights

of

the

first

-LRB-

fastest

-RRB-

interneuron

learning

the

first

Principal

Component

the

second

fastest

interneuron

learns

the

second

Principal

Component

etc.

Experimental

results

from

a

network

with

inputs

and

interneurons

are

given

in

Table

In

order

to

demonstrate

the

effect

of

the

network

we

have

here

followed

-LSB-

Oja

et

al.

-RSB-

in

using

specialised

input

data

for

demonstration

purposes

each

input

xi

is

chosen

from

independent

zero

mean

Gaussian

distributions

with

the

variance

of

the

variance

of

The

advantage

of

such

data

is

that

the

first

Principal

Component

is

-LRB-

-RRB-

i.e.

totally

in

the

direction

of

the

second

is

-LRB-

-RRB-

i.e.

totally

in

the

direction

of

etc.

Clearly

the

first

principal

components

have

been

found

by

the

interneurons

Note

that

the

crucial

difference

between

this

model

and

previous

models

is

the

embedding

of

the

learning

process

in

the

activation

reception

process

When

this

is

done

the

resulting

network

is

more

similar

to

a

Sanger-type

-LSB-

Sanger

-RSB-

network

rather

than

an

Oja-type

network

The

kth

interneuron

is

learning

to

extract

the

maximum

amount

of

information

which

is

left

after

the

previous

-LRB-

-RRB-

interneurons

have

extracted

their

information

Equivalence

to

Sanger

's

Algorithm

Sanger

's

algorithm

has

as

a

learning

rule

wij

yi

-LRB-

xj

iX

k

ykwkj

-RRB-

in

a

totally

feedforward

architecture

where

the

outputs

at

y

are

given

by

yi

X

j

wijxj

We

can

show

that

the

interneuron

network

using

these

rules

is

equivalent

to

Sanger

's

algorithm

Let

the

y

values

be

indexed

with

the

time

of

feedback

from

the

interneurons

Then

yj

-LRB-

-RRB-

is

the

initial

value

of

yj

at

time

i.e.

yj

-LRB-

-RRB-

xj

yj

-LRB-

-RRB-

is

the

value

of

yj

after

receiving

the

feedback

activation

from

the

first

-LRB-

and

hence

closest

-RRB-

interneuron

i.e.

yj

-LRB-

-RRB-

yj

-LRB-

-RRB-

Note

that

the

time

values

are

only

ordinal

indices

they

do

not

imply

equal

intervals

between

feedback

activations

Similarly

if

yj

-LRB-

-RRB-

is

the

value

of

yj

after

receiving

feedback

from

the

first

interneurons

then

yj

-LRB-

-RRB-

yj

-LRB-

-RRB-

yj

-LRB-

-RRB-

k

vkjzk

-LRB-

-RRB-

In

general

if

yj

-LRB-

i

-RRB-

is

the

value

of

yj

after

receiving

feedback

from

the

first

i

interneurons

yj

-LRB-

i

-RRB-

yj

-LRB-

-RRB-

iX

k

vkjzk

xj

iX

k

vkjzk

-LRB-

-RRB-

Therefore

vij

wij

yj

-LRB-

i

-RRB-

zi

-LRB-

yj

-LRB-

-RRB-

iX

k

vkjzk

-RRB-

zi

zi

-LRB-

xj

iX

k

vkjzk

-RRB-

which

is

exactly

Sanger

's

formulation

The

Interneuron

Coding

Network

The

various

algorithms

-LRB-

such

as

that

above

-RRB-

describing

learning

within

the

interneuron

network

have

been

shown

to

extract

the

maximum

information

from

sets

of

stochastic

data

The

next

obvious

question

is

to

decide

what

a

network

should

do

with

such

information

when

it

has

been

extracted

Some

form

of

coding

would

be

helpful

in

classifying

such

data

The

interneuron

coding

network

was

developed

in

appreciation

of

the

way

in

which

Carlson

-LSB-

Carlson

-RSB-

amended

the

basic

network

of

Rubner

and

Schulten

-LSB-

Rubner

and

Schulten

-RSB-

a

PCA

network

in

order

to

create

a

coding

network

While

we

have

wished

to

emulate

his

success

we

have

the

continuing

design

ethos

based

on

the

retention

of

as

many

of

the

attractive

features

of

the

basic

interneuron

network

as

possible

those

of

simplicity

homogeneity

locality

of

information

use

and

parallelism

Our

aim

is

to

create

a

network

which

will

take

a

set

of

raw

data

and

code

it

so

that

different

sections

of

the

data

are

coded

differently

and

such

that

data

which

have

most

similarity

are

most

alike

in

codes

i.e.

a

topology-preserving

network

A

binary

code

is

easiest

to

implement

with

a

simple

threshold

Since

we

require

several

bits

for

each

codeword

we

suggest

a

network

such

as

shown

in

Figure

This

network

is

shown

with

only

one

input

and

coding

interneurons

Raw

data

at

the

x

input

is

converted

to

a

binary

coded

vector

z

at

the

coding

interneurons

There

is

only

one

major

difference

between

this

network

and

those

investigated

previously

each

interneuron

has

a

threshold

above

which

its

weighted

inputs

must

sum

in

order

to

force

a

positive

firing

if

the

threshold

is

not

reached

the

interneuron

will

have

a

negative

activation

In

detail

ffl

Each

interneuron

in

turn

receives

a

weighted

sum

-LRB-

in

this

case

of

only

-RRB-

of

the

y

values

however

each

interneuron

has

a

threshold

above

which

its

activation

will

INPUT

DATA

CODING

INTERNEURONS

THE

CODING

NETWORK

CODES

Figure

The

Coding

Network

Raw

data

is

fed

in

from

the

left

binary

codes

emerge

from

the

right

have

a

positive

value

and

below

which

the

activation

will

be

negative

We

choose

the

threshold

for

all

interneurons

to

be

Then

if

zi

is

the

activation

of

this

interneuron

zi

if

wix

if

wix

ffl

This

activation

is

then

returned

-LRB-

weighted

-RRB-

and

subtracted

from

the

y

values

y

-LRB-

t

-RRB-

y

-LRB-

t

-RRB-

vizi

where

y

-LRB-

t

-RRB-

is

the

value

of

y

at

time

t

-LRB-

and

y

-LRB-

-RRB-

x

-RRB-

ffl

The

weights

wi

and

vi

are

updated

according

to

the

same

rule

as

previously

i.e.

a

simple

Hebbian

learning

rule

wi

wi

yzi

vi

vi

yzi

Note

that

the

zi

values

are

either

or

changing

the

weights

is

the

sole

method

of

learning

in

the

system

to

ensure

that

appropriate

codes

are

found

We

wish

to

emphasise

that

we

have

not

programmed

a

threshold

nor

any

specific

function

which

monitors

the

variance

of

the

data

and

adjusts

the

network

's

response

appropriately

We

have

retained

an

extremely

simple

network

with

only

a

single

modification

to

the

previous

network

Results

A

typical

set

of

results

is

shown

in

Table

These

results

are

for

an

interneuron

network

which

is

learning

from

a

set

of

x-values

generated

from

a

uniform

between

and

The

network

used

a

learning

rate

of

and

ran

for

iterations

Several

points

are

worth

noting

use

a

uniform

distribution

here

to

make

it

clear

why

each

weight

has

converged

to

the

value

it

has

converged

to

Decimal

Decimal

Table

A

section

of

coding

for

vectors

produced

by

a

network

for

input

data

from

a

uniform

distribution

between

and

Learning

rate

number

of

trials

We

have

replaced

with

to

highlight

the

binary

nature

of

the

code

Interneuron

Weight

-LRB-

w

-RRB-

Table

The

weights

which

the

above

network

learned

using

only

simple

Hebbian

learning

ffl

First

the

coding

seems

fairly

inefficient

in

that

the

first

figure

is

always

This

is

due

to

our

insistence

that

all

means

are

zero

Thus

the

first

code

element

is

always

for

inputs

-LRB-

see

Section

-RRB-

ffl

If

we

wish

a

code

where

the

first

interneuron

performs

maximum

discrimination

-LRB-

i.e.

in

the

above

example

all

inputs

less

than

would

be

coded

as

all

inputs

would

be

coded

as

-RRB-

we

would

use

a

threshold

which

will

also

learn

a

rule

such

as

j

j

ffzj

where

j

is

the

threshold

for

the

jth

interneuron

is

an

entirely

local

rule

and

easy

to

implement

however

in

keeping

with

our

design

philosophy

of

maintaining

simplicity

we

have

not

implemented

that

here

ffl

The

code

is

topology

preserving

similar

inputs

have

similar

outputs

-LRB-

see

Section

-RRB-

ffl

Experiments

have

shown

that

larger

networks

have

no

difficulty

in

providing

more

detailed

codes

and

require

only

a

slight

increase

in

time

as

each

element

of

the

coding

is

done

on

the

error

remaining

after

the

previous

interneurons

have

performed

their

coding

ffl

A

topological

feature

map

using

the

interneuron

network

has

one

major

advantage

over

e.g.

a

Kohonen

feature

map

it

can

easily

be

re-implemented

to

show

a

hierarchy

of

subfeatures

-LRB-

see

Section

-RRB-

by

adding

a

new

level

of

coding

interneuron

ffl

Lower

valued

digits

are

automatically

coded

more

slowly

and

hence

are

less

prone

to

an

unusual

input

creating

large

changes

Statistics

and

Weights

We

will

investigate

what

the

weights

are

actually

learning

by

considering

their

values

at

convergence

In

general

we

are

using

simple

Hebbian

learning

so

wi

y

-LRB-

i

-RRB-

zi

where

the

subscript

denote

the

ith

interneuron

and

y

-LRB-

i

-RRB-

denotes

the

value

of

y

at

time

i

The

proof

that

vi

wi

is

similar

to

that

shown

in

-LSB-

Fyfe

-RSB-

and

will

not

be

repeated

y

-LRB-

-RRB-

x

x

Figure

The

original

zero

mean

symmetric

distribution

-LRB-

as

seen

by

the

first

interneuron

-RRB-

is

transformed

into

a

bipolar

distribution

-LRB-

which

is

seen

by

the

second

interneuron

-RRB-

by

the

activity

of

the

first

interneuron

here

-LRB-

It

may

be

thought

of

as

obvious

since

v

and

w

are

updated

according

to

the

same

learning

rule

using

the

same

local

information

-RRB-

We

investigate

interesting

cases

before

looking

at

the

general

case

A

zero

mean

symmetric

distribution

ffl

Consider

the

weight

to

the

first

interneuron

Then

y

-LRB-

-RRB-

-LRB-

x

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

since

So

using

angled

brackets

to

denote

the

ensemble

average

at

convergence

-LRB-

-RRB-

-LRB-

-RRB-

With

a

zero

mean

symmetric

distribution

when

x

is

negative

and

when

x

is

positive

therefore

jxj

and

so

jxj

at

convergence

i.e.

wi

converges

to

the

expected

value

of

the

absolute

value

of

the

input

data

i.e.

the

mean

absolute

value

This

has

the

effect

of

mapping

the

two

halves

of

the

distribution

to

a

tighter

-LRB-

bipolar

-RRB-

distribution

which

is

mostly

contained

within

the

interval

-LSB-

jxj

jxj

-RSB-

See

Figure

A

positive

compact

distribution

By

compact

we

mean

a

distribution

which

contains

no

holes

ffl

Consider

as

before

we

can

show

that

at

convergence

-LRB-

-RRB-

-LRB-

-RRB-

With

this

distribution

for

all

input

values

of

x

So

x

i.e.

the

mean

value

of

the

input

data

ffl

Now

consider

the

weight

to

the

second

interneuron

Then

y

-LRB-

-RRB-

-LRB-

y

-LRB-

-RRB-

-RRB-

-LRB-

y

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

since

But

y

-LRB-

-RRB-

x

x

x

Now

if

x

x

i.e.

x

then

while

if

x

x

i.e.

x

then

Thus

at

convergence

equals

jx

xj

Therefore

at

convergence

is

bisecting

the

residual

area

of

the

distribution

after

has

subtracted

out

the

mean

So

for

this

distribution

is

performing

the

task

which

performed

for

the

symmetric

distribution

In

general

for

values

of

x

drawn

from

any

distribution

wi

y

-LRB-

i

-RRB-

zi

-LRB-

y

-LRB-

i

-RRB-

wizi

-RRB-

zi

-LRB-

y

-LRB-

i

-RRB-

zi

wi

-RRB-

Therefore

wi

y

-LRB-

i

-RRB-

zi

at

convergence

and

if

y

-LRB-

i

-RRB-

then

zi

while

if

y

-LRB-

i

-RRB-

then

zi

Therefore

at

convergence

wi

jy

-LRB-

i

-RRB-

j

In

general

the

z

values

correspond

to

the

coding

taking

place

while

the

y

values

represent

the

error

after

the

coding

has

taken

place

The

coding

of

a

uniform

distribution

is

shown

in

Figure

We

use

a

uniform

distribution

this

time

to

make

it

easy

to

corroborate

that

the

network

is

performing

an

efficient

coding

note

from

Tabel

that

jxj

x

and

jx

xj

etc.

Reconstruction

Error

We

show

that

if

we

use

the

vectors

produced

by

this

method

to

reconstruct

the

original

vectors

we

can

make

the

expected

absolute

reconstruction

error

from

a

final

code

-LRB-

sometimes

called

the

mean

quantization

error

-RRB-

arbitrarily

small

by

simply

adding

new

coding

interneurons

Let

us

assume

that

we

can

not

i.e

that

there

exists

an

ffl

such

that

all

mean

absolute

reconstruction

errors

are

greater

than

ffl

We

note

from

the

above

that

this

is

equivalent

to

showing

that

the

value

of

wi

ffl

for

all

i.

Note

that

for

a

finite

distribution

the

maximum

possible

error

after

the

first

coding

is

EMax

max

-LRB-

jxMax

jxjj

jxMin

jxjj

jxj

-RRB-

where

xMax

xMin

is

the

largest

-LRB-

resp

smallest

-RRB-

possible

member

of

the

distribution

Thus

EMax

is

finite

Consider

a

particular

input

x.

From

the

results

in

the

last

section

because

the

system

is

creating

the

coding

at

z

and

subtracting

the

weighted

coding

at

y

the

value

y

-LRB-

i

-RRB-

is

simply

the

error

between

the

code

found

by

the

first

i

coding

interneurons

and

the

input

x

after

i

codings

have

taken

place

Therefore

jy

-LRB-

i

-RRB-

j

is

the

mean

absolute

error

after

i

codings

of

the

input

Now

y

-LRB-

i

-RRB-

y

-LRB-

i

-RRB-

wizi

y

-LRB-

i

-RRB-

jy

-LRB-

i

-RRB-

jzi

If

y

-LRB-

i

-RRB-

zi

and

y

-LRB-

i

-RRB-

y

-LRB-

i

-RRB-

jy

-LRB-

i

-RRB-

j

jy

-LRB-

i

-RRB-

j

jy

-LRB-

i

-RRB-

j

while

if

y

-LRB-

i

-RRB-

zi

and

y

-LRB-

i

-RRB-

y

-LRB-

i

-RRB-

jy

-LRB-

i

-RRB-

j

jy

-LRB-

i

-RRB-

j

jy

-LRB-

i

-RRB-

j

Therefore

the

amplitude

of

y

-LRB-

i

-RRB-

is

the

difference

between

the

absolute

value

of

y

-LRB-

-RRB-

and

the

mean

absolute

value

of

y

-LRB-

-RRB-

Thus

jy

-LRB-

i

-RRB-

j

jfjy

-LRB-

i

-RRB-

j

jy

-LRB-

i

-RRB-

jgj

jfjy

-LRB-

i

-RRB-

j

wigj

jjy

-LRB-

i

-RRB-

j

fflj

since

all

terms

are

positive

Therefore

the

absolute

error

at

each

stage

is

decreasing

by

more

than

ffl

Therefore

the

mean

absolute

error

is

also

decreasing

by

at

least

ffl

at

each

stage

Now

the

initial

maximum

error

is

EMax

which

is

finite

and

the

absolute

error

is

decreasing

by

a

finite

amount

each

time

and

so

can

not

remain

above

ffl

for

all

time

Therefore

we

can

make

the

quantization

error

arbitrarily

small

by

continuing

the

coding

for

a

sufficient

number

of

interneurons

Topology

Preservation

In

stating

that

we

have

a

topology

preserving

coding

we

mean

that

similar

inputs

should

be

projected

onto

similar

outputs

and

similar

outputs

should

be

the

representations

of

similar

inputs

This

is

only

approximately

true

in

general

of

feature

maps

using

neural

nets

e.g.

a

Kohonen

-LSB-

Kohonen

-RSB-

map

attempts

to

project

the

input

space

onto

a

network

in

such

a

way

that

the

most

essential

neighbourhood

relationships

between

data

in

the

input

space

are

preserved

Yet

input

data

can

be

constructed

which

do

not

permit

a

-LRB-

or

-RRB-

mapping

to

adequately

represent

all

topological

equivalences

in

the

data

We

give

an

intuitive

notion

of

topology

preservation

here

a

more

formal

proof

is

given

in

the

appendix

Consider

an

n-unit

coding

of

a

set

of

input

values

Let

a

particular

point

x

be

represented

by

zi

where

zi

zing

The

subscript

i

denotes

an

ordering

of

the

z

values

-LRB-

i.e.

of

the

coding

-RRB-

such

that

zi

zj

for

all

i

j

Then

any

input

x

x

is

represented

by

the

vector

zi

or

by

zi

or

zi

for

all

x

such

that

j?xj

wn

the

nth

weight

i.e.

similar

inputs

are

represented

by

similar

outputs

Now

consider

two

distinct

input

values

x

and

u

which

are

represented

by

the

same

zj

Then

x

nX

i

wizji

x

u

nX

i

wizji

u

where

x

and

u

are

the

errors

in

the

representations

after

the

first

n

codings

have

taken

place

Therefore

x

u

x

u

twodist.results

Figure

The

vertical

axis

shows

the

coding

produced

by

the

interneuron

network

for

inputs

in

the

range

to

Note

the

lack

of

differentiation

taking

place

in

the

range

x

for

which

there

was

no

input

data

From

the

previous

section

we

know

that

the

value

of

wn

can

be

made

arbitrarily

small

and

so

a

single

code

can

be

made

to

represent

only

similar

values

Clearly

a

similar

argument

will

show

that

if

the

values

x

and

u

are

represented

by

contiguous

codes

the

values

x

and

u

can

only

be

at

most

apart

Thus

similar

codes

represent

similar

values

A

Data

Driven

Coding

We

will

in

Section

go

on

to

use

the

coding

network

in

such

a

way

that

we

feel

entitled

to

describe

it

as

a

feature

map

As

a

prelude

to

that

we

give

an

example

here

to

illustrate

the

behaviour

of

the

coding

with

a

particular

distribution

We

generated

a

distribution

composed

of

disjoint

sets

A

f

-LRB-

x

y

-RRB-

x

y

b

f

-LRB-

x

y

-RRB-

x

y

The

coding

network

was

given

inputs

from

this

distribution

and

asked

to

code

them

The

results

are

illustrated

in

Figure

Similar

inputs

have

been

given

similar

codes

and

those

areas

which

are

not

represented

in

the

distribution

have

received

little

or

no

coding

Note

that

there

exists

a

code

for

potential

inputs

in

the

range

x

however

there

is

no

discrimination

taking

place

within

this

range

the

network

has

not

had

to

perform

any

coding

within

this

range

as

there

was

no

data

in

its

training

set

in

the

range

A

Hierarchical

Feature

Map

We

now

propose

the

complete

network

shown

in

Figure

The

left

half

of

the

network

is

the

basic

interneuron

network

described

in

Section

the

right

half

of

the

network

is

the

interneuron

coding

network

described

in

Section

The

first

section

will

extract

the

INPUT

DATA

CODE

FOR

DIRECTION

CODE

FOR

DIRECTION

CODE

FOR

DIRECTION

INFORMATION

EXTRACTION

NETWORK

INFORMATION

CODING

NETWORK

Figure

The

complete

information

extraction

and

coding

network

The

double

headed

arrow

represents

both

the

feedforward

weights

and

the

feedback

weights

maximum

information

from

the

raw

input

data

i.e.

data

will

be

projected

onto

those

directions

which

contain

maximum

information

the

second

section

will

code

the

data

along

each

dimension

independently

All

parts

of

the

system

use

unsupervised

learning

Our

learning

rule

continues

to

be

simple

Hebbian

learning

with

no

weight

decay

or

clipping

of

weights

Since

the

network

is

topology

preserving

in

each

direction

it

can

be

shown

to

be

topology

preserving

in

the

space

spanned

by

these

directions

Therefore

the

construction

can

be

viewed

as

forming

a

feature

map

which

is

topology

preserving

in

the

major

information

directions

of

the

data

While

it

is

possible

to

create

continuous

multi-dimensional

maps

which

are

topologically

different

from

their

projection

onto

this

subspace

each

such

anomaly

will

tend

to

swing

the

principal

components

in

the

direction

of

the

anomaly

suggesting

that

such

anomalies

can

at

most

be

a

minor

part

of

the

input

data

Further

as

will

be

shown

augmenting

this

type

of

feature

map

to

take

account

of

such

features

is

a

simple

process

The

inherently

modular

nature

of

the

network

-LRB-

see

Figure

-RRB-

allows

us

to

consider

the

effects

of

augmenting

the

network

as

a

purely

local

process

This

modular

nature

is

a

direct

consequence

of

the

Principal

Component

Analysis

performed

by

the

first

section

which

leads

to

orthogonal

input

vectors

for

the

second

section

Augmenting

a

Map

The

desire

to

augment

a

map

may

be

brought

about

by

circumstances

The

map

is

too

crude

as

too

little

information

has

been

extracted

from

the

original

input

data

To

extract

more

information

from

the

raw

data

we

must

find

a

new

Principal

component

along

which

to

project

the

data

Therefore

we

must

create

a

new

data

extraction

interneuron

i.e.

in

the

central

layer

of

the

network

Now

by

adding

our

new

interneuron

at

the

end

of

the

learning

process

described

in

Equations

and

we

are

not

disturbing

the

learning

in

any

of

the

other

directions

which

have

already

been

found

Therefore

the

new

direction

can

be

found

without

disturbing

the

principal

components

already

found

therefore

the

existing

codes

are

not

disturbed

and

the

coding

of

the

new

dimension

can

be

done

independently

of

the

existing

codes

The

map

is

too

crude

as

too

little

discrimination

has

taken

place

in

a

particular

direction

Note

again

that

the

modular

nature

of

the

map

allows

the

discrimination

in

each

direction

to

be

modified

independently

of

that

in

the

other

directions

Further

in

the

coding

within

a

particular

direction

we

may

simply

add

a

new

coding

interneuron

into

the

network

and

provided

it

learns

after

all

the

others

have

learned

it

will

simply

learn

to

bisect

the

remaining

information

after

the

others

have

subtracted

their

activations

In

other

words

a

new

coding

interneuron

will

simply

provide

increased

discrimination

within

that

direction

and

will

affect

neither

the

coding

in

other

directions

nor

the

existing

coding

in

its

direction

Experimental

results

have

confirmed

this

analysis

Note

that

the

potential

for

improving

a

map

after

it

has

been

constructed

is

an

improvement

on

the

Kohonen

map

whose

parameters

must

be

specified

in

advance

An

Example

Kohonen

and

Ritter

-LSB-

Ritter

and

Kohonen

-RSB-

used

the

data

shown

in

Table

to

illustrate

the

emergence

of

a

semantic

map

using

a

Kohonen

feature

map

They

trained

a

Kohonen

network

to

learn

to

associate

the

name

of

each

animal

with

the

attributes

with

which

it

might

be

associated

They

then

showed

that

by

entering

the

name

-LRB-

and

only

the

name

-RRB-

of

the

animal

into

the

network

ffl

different

nodes

responded

maximally

to

different

animals

ffl

the

spatial

organisation

-LRB-

on

a

grid

-RRB-

grouped

similar

animals

together

We

have

chosen

to

repeat

this

experiment

with

the

interneuron

network

as

it

might

be

thought

that

organising

such

a

set

of

data

might

be

difficult

for

a

statistics-based

network

such

as

the

interneuron

network

Figure

shows

that

the

first

principal

components

are

sufficient

to

make

the

major

differentiation

into

animals

and

birds

and

within

each

group

there

is

some

differentiation

into

subgroups

However

the

advantage

of

using

the

interneuron

network

as

an

encoding

network

is

that

further

differentiation

is

possible

by

looking

at

other

directions

and

the

third

principal

component

shows

clearly

the

differentiation

into

subgroups

taking

place

in

this

direction

This

is

equivalent

to

a

semantic

interpretation

of

data

which

allows

us

to

be

aware

of

similarities

in

attributes

within

a

group

while

being

aware

of

differences

in

other

attributes

between

members

of

the

same

group

This

is

the

feature

which

allows

us

to

call

the

map

a

hierarchical

feature

map

Following

Kohonen

and

Ritter

each

animal

was

represented

by

a

vector

the

first

bits

of

which

were

except

for

the

bit

which

identified

the

particular

animal

The

other

bits

were

the

animal

's

attributes

as

shown

in

Table

The

output

data

were

plotted

in

the

obvious

manner

each

output

vector

was

identified

as

a

binary

number

-LRB-

-RRB-

and

converted

to

decimal

to

give

the

coordinates

in

each

direction

While

we

do

not

believe

that

this

is

a

good

-LRB-

or

particularly

useful

-RRB-

application

of

a

data-driven

network

we

believe

that

this

example

highlights

the

hierarchical

nature

of

the

coding

possible

with

an

interneuron

network

dove

hen

duck

goose

owl

hawk

eagle

fox

dog

wolf

cat

tiger

lion

horse

zebra

cow

small

medium

big

legs

legs

hair

hooves

mane

feathers

hunt

run

fly

swim

Table

Animal

names

and

their

attributes

Second

Component

FirstComponent

120 130 140 150

Dove

HenDuck

Goose/Owl/Hawk

Eagle

FoxDog

Wolf

Cat

Tiger

Lion

Horse/Zebra

Cow

50 100 150 200

Component

Fox

Wolf

Dog

Cat

Eagle

Owl

Hawk

Tiger

Lion

Dove

Hen

Goose

Duck

Cow

Horse

Zebra

Figure

The

results

of

the

experiment

displayed

in

graphical

form

Attribute/class

First

PC

Second

PC

Third

PC

dove

hen

duck

goose

owl

hawk

eagle

fox

dog

wolf

cat

tiger

lion

horse

zebra

cow

small

medium

big

legs

legs

hair

hooves

mane

feathers

hunt

run

fly

swim

Table

The

first

Principal

Components

of

the

input

data

's

covariance

matrix

It

is

easily

verified

that

they

form

an

orthonormal

basis

of

the

dimensional

subspace

The

Principal

Component

Analysis

Another

advantage

of

the

proposed

method

is

that

to

some

extent

it

allows

us

to

look

inside

the

results

in

order

to

investigate

how

the

results

came

about

This

is

particularly

interesting

in

examples

such

as

the

current

one

where

the

data

have

strong

everyday

connotations

for

us

Table

shows

the

first

Principal

Components

of

the

covariance

matrix

of

the

data

of

Table

as

identified

by

the

interneuron

weights

of

the

middle

layer

It

is

easily

verified

that

the

vectors

form

an

orthonormal

basis

of

a

Dimensional

subspace

of

the

data

ffl

The

first

Principal

Component

is

most

strongly

identifying

animal

type

features

animals

tend

to

be

big

have

legs

and

hair

some

have

hooves

or

a

mane

somewhat

more

hunt

and

run

ffl

The

second

Principal

Component

completes

the

job

the

birds

all

are

represented

by

a

negative

component

as

are

the

small

medium

legs

feathers

hunt

fly

and

swim

attributes

Also

it

can

be

seen

that

the

more

prototypical

bird-like

features

have

larger

absolute

values

e.g.

jflyj

jswimj

since

the

prototypical

bird

is

more

likely

to

fly

than

swim

Note

also

that

the

cat

has

a

small

negative

value

brought

about

by

its

prototypical

bird-like

attribute

of

smallness

Thus

in

the

map

-LRB-

Figure

-RRB-

the

cat

appears

closest

to

the

birds

in

the

direction

of

the

second

PC

ffl

The

Third

Principal

Component

seems

to

be

mainly

differentiating

the

hunters

from

the

non-hunters

though

differentiation

in

size

and

between

fliers

and

swimmers

is

also

taking

place

Note

that

the

components

defining

e.g.

horse

and

zebra

are

identical

in

all

directions

as

there

is

nothing

in

the

input

data

provided

which

will

allow

us

to

discriminate

between

these

groups

Similarly

the

attributes

legs

and

hair

are

identically

represented

as

they

are

identically

distributed

in

the

information

we

have

given

Reparing

a

Damaged

Map

The

situation

described

in

this

section

is

more

complex

than

that

in

the

last

Again

possibilities

must

be

considered

Where

damage

has

occurred

in

the

coding

layer

Here

we

consider

what

happens

when

a

neuron

is

damaged

loses

its

learning

-LRB-

its

weight

assumes

a

small

random

number

-RRB-

yet

still

remains

in

the

same

physical

position

with

the

same

physical

potential

for

learning

Experimental

results

have

shown

that

even

in

the

worst

case

that

of

damaging

the

coding

interneuron

which

extracts

the

greatest

information

-LRB-

i.e.

the

first

coding

interneuron

-RRB-

relearning

is

very

fast

The

network

has

the

advantage

that

when

it

loses

an

interneuron

all

of

the

weights

which

learn

after

the

damaged

interneuron

's

activities

increase

in

the

order

in

which

they

are

learning

there

is

no

complex

dog-fight

which

is

sometimes

seen

in

artificial

neural

networks

All

of

the

other

interneurons

whose

weights

had

increased

on

a

pro-rata

basis

gradually

lose

their

gains

to

the

recovering

interneuron

till

the

previously

existing

weight

values

are

regained

If

the

new

interneuron

is

deemed

to

be

irrepairable

i.e.

does

not

recover

its

ability

to

learn

the

other

interneurons

simply

step

up

their

information

coding

capacities

in

order

i.e.

each

weight

increases

until

it

is

performing

exactly

the

same

discrimination

that

the

weight

above

it

was

previously

performing

Where

damage

has

occurred

in

the

information

extraction

layer

This

is

potentially

the

most

disruptive

damage

in

the

network

when

interneuron

in

the

central

layer

is

damaged

the

interneurons

after

it

in

the

learning

sequence

are

able

to

take

advantage

of

its

incapacity

to

capture

its

previously

held

information

This

will

obviously

have

an

effect

on

those

coding

interneurons

which

will

have

to

readjust

their

weights

in

order

to

accommodate

the

new

informationbearing

capacity

of

the

first

layer

interneurons

Each

weight

will

tend

to

increase

as

it

is

now

working

for

an

interneuron

which

has

captured

a

little

more

of

the

available

information

in

the

network

However

this

is

a

minimal

rearrangement

each

coding

neuron

is

most

similar

in

weight

to

its

equivalent

coding

neuron

for

the

previous

direction

All

coding

neurons

will

augment

their

weights

in

unison

and

no

realignment

of

coding

is

necessary

All

increases

and

decreases

in

weight

values

are

done

as

though

synchronised

there

is

never

an

example

of

the

complex

untwisting

which

we

see

in

developing

Kohonen

nets

Again

experimental

results

have

confirmed

this

analysis

Note

again

that

the

interneuron

map

is

an

improvement

on

the

Kohonen

map

in

which

the

destruction

of

a

single

competitive

neuron

reduces

that

network

's

ability

to

respond

appropriately

to

the

region

which

was

optimally

represented

by

that

neuron

Summary

We

have

described

a

novel

robust

topology-preserving

neural

network

which

seems

to

have

significant

advantages

over

current

networks

with

equivalent

properties

We

must

Tanh

-LRB-

x

-RRB-

x

tanh

-LRB-

x

-RRB-

Figure

The

Tanh

-LRB-

-RRB-

function

emphasise

that

both

parts

of

the

network

are

necessary

ffl

Without

the

first

section

the

second

section

would

merely

divide

up

the

information

as

a

whole

In

other

words

the

amplitude

of

the

message

would

determine

the

coding

and

no

account

would

be

taken

of

its

content

ffl

Without

the

second

section

the

first

section

would

find

those

directions

of

maximum

information

but

an

observer

would

not

know

e.g.

if

an

output

of

came

from

a

direction

with

zero

mean

and

variance

or

mean

and

variance

The

second

section

of

the

network

is

essential

to

calibrate

the

mapping

A

Biological

Implementation

It

is

well

known

that

biological

neurons

are

not

accurately

modelled

by

either

simple

linear

summation

neurons

nor

by

neurons

which

have

step

functions

as

activation

functions

Instead

a

degree

of

non-linearity

of

response

has

been

found

which

is

usually

modelled

by

a

sigmoidal

function

Using

tanh

-LRB-

-RRB-

-LRB-

See

Figure

-RRB-

as

an

activation

function

for

the

interneurons

allows

a

unified

model

of

the

above

two

types

of

interneurons

to

be

created

ffl

tanh

-LRB-

-RRB-

is

approximately

linear

in

its

middle

section

We

may

adjust

the

range

of

middle

section

over

which

it

is

linear

by

using

the

parameter

in

zi

tanh

-LRB-

P

j

wijyj

-RRB-

To

get

a

large

linear

section

requires

a

small

value

of

Experimental

results

have

confirmed

that

is

sufficiently

small

to

approximate

a

linear

function

with

which

Principal

Components

can

be

found

as

before

ffl

tanh

-LRB-

-RRB-

may

be

made

more

dichotomous

by

adjusting

the

parameter

upwards

A

value

of

O

-LRB-

-RRB-

is

sufficient

to

create

an

interneuron

which

performs

a

smoothed

coding

of

the

input

data

-LRB-

but

see

below

-RRB-

The

larger

the

value

of

the

more

step-like

the

function

becomes

Therefore

a

single

type

of

interneuron

with

different

parameters

can

perform

both

the

information

extraction

and

the

information

coding

described

above

Both

sets

of

interneurons

will

have

an

activation

function

tanh

-LRB-

P

j

wijyj

-RRB-

The

sole

difference

is

Figure

The

coding

produced

by

the

sixth

coding

interneuron

on

the

output

of

the

first

principal

component

when

both

are

using

tanh

-LRB-

-RRB-

that

interneurons

in

the

first

layer

have

the

parameter

while

those

in

the

second

have

the

parameter

The

first

value

is

of

course

dependent

on

the

distribution

of

the

input

data

and

is

based

on

distributions

with

single

figure

standard

deviation

and

mean

the

second

depends

on

the

amount

of

discrimination

-LRB-

length

of

the

code

-RRB-

required

Using

such

an

activation

function

with

the

information

extracting

interneurons

has

several

implications

The

output

values

z

at

these

interneurons

are

all

in

the

range

-LRB-

-RRB-

Thus

the

weights

in

the

first

part

of

the

network

will

grow

much

larger

than

previously

The

learning

rate

in

this

network

can

be

made

much

larger

than

before

as

the

z

values

are

constrained

to

this

small

range

Previous

networks

used

a

learning

rate

of

O

-LRB-

-RRB-

with

a

tanh

-LRB-

-RRB-

activation

function

a

learning

rate

of

O

-LRB-

-RRB-

is

possible

However

there

is

a

drawback

to

the

use

of

this

activation

function

in

the

coding

layer

with

the

values

stated

codes

for

the

first

coding

interneurons

in

each

direction

agree

with

those

found

with

the

step

function

network

However

for

the

later

coding

interneurons

a

lack

of

discrimination

develops

leading

to

very

imprecise

codes

By

the

coding

interneuron

we

have

lost

all

claim

to

have

a

topology-preserving

network

To

show

that

this

can

not

be

wholly

explained

by

the

foreshortening

of

the

outputs

from

the

first

section

of

the

network

to

the

range

-LRB-

-RRB-

we

have

used

the

same

data

which

produced

the

coding

shown

in

Table

and

multiplied

the

outputs

of

the

first

section

by

to

put

them

in

the

same

approximate

range

as

before

The

activation

values

of

the

sixth

digit

-LRB-

i.e

sixth

coding

interneuron

-RRB-

are

shown

in

Figure

Clearly

ffl

the

code

can

not

be

called

topology

preserving

in

the

neighbourhood

of

the

changes

from

to

ffl

the

number

of

differentiated

codes

is

much

smaller

than

it

should

be

This

effect

can

be

ameliorated

by

increasing

the

value

of

from

to

The

results

of

this

can

be

seen

in

Figure

Clearly

however

using

this

activation

function

means

that

we

have

lost

the

ability

to

subclass

at

will

almost

infinitely

often

Whether

this

is

important

in

a

biological

context

must

still

be

investigated

Figure

The

coding

produced

by

the

sixth

coding

interneuron

on

the

output

of

the

first

principal

component

when

both

are

using

tanh

-LRB-

-RRB-

Conclusion

A

new

method

of

creating

a

topology-preserving

feature

map

has

been

presented

it

is

a

stage

process

involving

projection

of

the

input

data

onto

the

directions

of

maximum

information

followed

by

a

discrimination

process

within

each

direction

The

simple

interneuron

network

performs

the

projection

while

the

interneuron

coding

network

performs

the

discrimination

Both

use

only

simple

Hebbian

learning

with

no

normalisation

weight

decay

or

clipping

of

weights

The

sole

difference

between

the

sets

of

interneurons

is

that

the

coding

interneurons

have

a

threshold

which

must

be

achieved

before

their

activation

becomes

positive

An

attempt

to

produce

a

biologically

feasible

unification

of

the

two

types

of

interneurons

was

partially

successful

in

that

a

single

type

of

interneuron

with

different

values

of

a

parameter

in

its

activation

function

has

been

shown

to

be

capable

of

performing

both

tasks

necessary

to

produce

the

feature

map

the

single

addition

of

an

activation

function

tanh

-LRB-

x

-RRB-

allowed

us

to

dispense

with

the

threshold

in

the

coding

interneurons

This

network

requires

parameter

tuning

if

it

is

to

perform

more

than

a

crude

discrimination

in

each

direction

A

Approximate

Topological

Equivalence

We

will

use

the

results

from

Section

in

this

proof

recall

that

for

every

ffl

there

exists

an

n

such

that

wn

ffl

Let

M

be

the

metric

space

defined

by

that

-LRB-

sub

-RRB-

set

of

the

real

numbers

defined

by

the

probability

distribution

of

the

raw

data

and

the

metric

d

defined

by

the

usual

Euclidean

distance

metric

Let

be

the

metric

space

defined

by

the

set

of

codes

-LRB-

of

the

real

numbers

in

M

-RRB-

and

the

metric

defined

by

the

usual

Euclidean

distance

metric

-LRB-

now

calculating

in

binary

-RRB-

It

is

not

possible

to

prove

that

the

there

is

a

topology-preserving

function

which

maps

M

to

a

particular

value

of

however

it

is

possible

to

prove

that

there

exists

a

mapping

from

the

set

M

to

a

member

of

the

family

g

where

is

the

coding

which

has

length

n

-LRB-

i.e.

formed

by

using

n

coding

interneurons

-RRB-

In

other

words

we

can

make

our

mapping

as

close

to

a

topology

preserving

mapping

as

possible

by

chosing

n

appropriately

We

shall

create

an

ordering

fCig

of

the

codes

in

based

on

the

size

of

their

binary

values

Thus

Ci

Cj

for

i

j.

Note

that

for

this

coding

on

if

f

-LRB-

x

-RRB-

is

the

function

which

codes

the

inputs

i.e.

f

M

then

f

-LRB-

Ci

-RRB-

f

-LRB-

Ci

-RRB-

wn

the

weight

of

the

nth

level

of

the

coding

Note

also

that

the

greatest

distance

between

values

coded

by

the

same

code

is

also

wn

ffl

First

consider

the

mapping

f

M

Then

take

any

point

c

M

we

require

to

prove

that

a

ffi

such

that

-LRB-

f

-LRB-

c

-RRB-

f

-LRB-

x

-RRB-

-RRB-

ffl

M

d

-LRB-

c

x

-RRB-

ffi

Choose

the

coding

such

that

wn

ffl

Let

ffi

wn

Let

f

map

c

to

class

Ci

the

ith

class

of

Then

for

all

x

such

that

d

-LRB-

c

x

-RRB-

ffi

wn

f

-LRB-

x

-RRB-

is

either

in

class

Ci

or

in

one

of

its

neighbours

Ci

or

Ci

So

f

-LRB-

x

-RRB-

is

within

of

f

-LRB-

c

-RRB-

i.e.

f

-LRB-

x

-RRB-

-LRB-

f

-LRB-

c

-RRB-

ffl

f

-LRB-

c

-RRB-

ffl

-RRB-

when

d

-LRB-

x

c

-RRB-

i.e.

-LRB-

f

-LRB-

c

-RRB-

f

-LRB-

x

-RRB-

-RRB-

ffl

when

d

-LRB-

c

x

-RRB-

ffi

ffl

Now

consider

the

mapping

f

M

Then

take

any

point

Ci

Given

any

ffl

we

must

prove

that

d

-LRB-

f

-LRB-

Ci

-RRB-

f

-LRB-

x

-RRB-

-RRB-

ffl

M

-LRB-

Ci

x

-RRB-

ffi

Choose

n

such

that

wn

ffl

this

defines

the

actual

representative

of

as

We

chose

ffi

to

be

equal

to

Then

-LRB-

Ci

ffi

Ci

ffi

-RRB-

is

equivalent

to

x

Ci

Ci

or

Ci

Now

the

maximum

distance

between

the

values

which

code

to

Ci

or

Ci

and

those

which

map

to

Ci

is

i.e.

times

the

remaining

error

after

the

nth

coding

i.e.

d

-LRB-

f

-LRB-

x

-RRB-

f

-LRB-

-RRB-

-RRB-

ffl

for

all

x

in

the

declared

interval

Now

any

particular

interneuron

coding

network

is

not

truly

topology

preserving

however

it

can

be

made

arbitrarily

close

to

such

a

network

by

increasing

the

length

of

the

coding

Therefore

any

particular

coding

is

performing

an

approximate

topologypreserving

mapping

We

define

an

preserving

network

as

an

interneuron

network

in

which

for

all

points

c

M

-LRB-

f

-LRB-

c

-RRB-

f

-LRB-

x

-RRB-

-RRB-

ffl

M

d

-LRB-

c

x

-RRB-

ffi

where

f

-LRB-

x

-RRB-

f

-LRB-

c

-RRB-

are

the

binary

codes

Note

that

this

is

equivalent

to

defining

wn

Then

each

network

in

the

sequence

of

interneuron

networks

of

increasing

discrimination

is

an

preserving

network

with

the

value

of

defined

as

for

the

specific

mapping

References

-LSB-

Carlson

-RSB-

Carlson

A.

-LRB-

-RRB-

Anti-hebbian

learning

in

a

non-linear

neural

network

Biological

Cybernetics

-LCB-

-LSB-

Fyfe

-RSB-

Fyfe

C.

-LRB-

-RRB-

Asymmetric

learning

in

interneurons

In

World

Conference

on

Neural

Nets

-LSB-

Fyfe

-RSB-

Fyfe

C.

-LRB-

-RRB-

Interneurons

which

identify

principal

components

In

Recent

Advances

in

Neural

Networks

-LSB-

Fyfe

-RSB-

Fyfe

C.

-LRB-

-RRB-

Pca

properties

of

interneurons

In

ICANN

-LSB-

Hebb

-RSB-

Hebb

D.

-LRB-

-RRB-

The

Organisation

of

Behaviour

Wiley

-LSB-

Kohonen

-RSB-

Kohonen

T.

-LRB-

-RRB-

Self-Organization

and

Associative

Memory

Springer-Verlag

-LSB-

Oja

-RSB-

Oja

E.

-LRB-

-RRB-

Neural

networks

principal

components

and

subspaces

International

Journal

of

Neural

Systems

-LSB-

Oja

et

al.

-RSB-

Oja

E.

Ogawa

H.

and

Wangviwattana

J.

-LRB-

-RRB-

Pca

in

fully

parallel

neural

networks

In

Taylor

A.

editor

Artificial

Neural

Networks

-LSB-

Ritter

and

Kohonen

-RSB-

Ritter

H.

and

Kohonen

T.

-LRB-

-RRB-

Self-organising

semantic

maps

Biological

Cybernetics

-LCB-

-LSB-

Rubner

and

Schulten

-RSB-

Rubner

J.

and

Schulten

K.

-LRB-

-RRB-

Development

of

feature

detectors

and

self-organisation

Biological

Cybernetics

-LSB-

Sanger

-RSB-

Sanger

T.

-LRB-

-RRB-

Analysis

of

the

two-dimensional

receptive

fields

learned

by

the

generalized

hebbian

algorithm

in

response

to

random

input

Biological

Cybernetics