Node

Growth

Mechanisms

In

Self-Configuring

Networks

PETER

FLETCHER

DEPARTMENT

OF

COMPUTER

SCIENCE

KEELE

UNIVERSITY

KEELE

STAFFORDSHIRE

U.K.

-LRB-

0782 621111

ext.

-RRB-

E-mail

-LRB-

JANET

-RRB-

TECHNICAL

REPORT

Running

headline

Node

growth

ABSTRACT

A

self-configuring

network

starts

with

only

input

nodes

and

without

supervision

grows

into

a

configuration

in

which

it

can

simulate

the

input

environment

Each

new

node

grown

is

responsible

for

simulating

a

new

aspect

of

the

environment

However

the

networks

grown

in

this

way

although

successful

are

unnecessarily

large

and

too

tangled

to

understand

In

this

paper

I

investigate

how

small

changes

in

the

criterion

for

establishing

new

nodes

lead

to

drastic

changes

in

the

resulting

network

Criteria

inspired

by

syntactic

pattern

recognition

lead

to

much

smaller

faster

and

more

understandable

networks

If

the

network

establishes

a

wrong

node

this

may

distort

its

future

learning

preventing

the

right

nodes

from

ever

being

established

I

add

mechanisms

which

allow

it

to

remove

mistakenly

established

nodes

enabling

it

to

unravel

its

own

mistakes

In

the

simple

geometric

learning

problem

considered

this

leads

to

nearoptimal

networks

Improving

performance

is

not

an

end

in

itself

its

purpose

is

reveal

the

fundamental

capabilities

and

limitations

of

this

learning

formalism

Table

of

Contents

Introduction

The

strength

of

a

new

node

Example

problem

Node

growth

in

the

example

problem

The

syntactic

viewpoint

Disestablishing

bad

nodes

The

need

for

disestablishment

and

two

mechanisms

Correctness

proof

for

the

suspicion

procedure

Example

of

suspicion

and

disestablishment

Conclusions

References

Node

Growth

Mechanisms

In

Self-Configuring

Networks

PETER

FLETCHER

Introduction

In

Fletcher

-LRB-

-RRB-

I

described

an

unsupervised

algorithm

for

growing

a

feedforward

Boolean

network

for

an

arbitrary

pattern

learning

task

Here

is

a

quick

recapitulation

ffl

The

network

runs

in

two

phases

waking

-LRB-

observing

the

environment

-RRB-

and

dreaming

-LRB-

simulating

the

environment

-RRB-

Its

task

is

to

simulate

the

environment

correctly

that

is

its

dream

states

should

have

the

same

probability

distribution

as

its

waking

states

ffl

To

simulate

a

particular

environment

it

needs

to

choose

an

appropriate

set

of

nodes

and

connections

Initially

it

has

only

input

nodes

it

grows

a

node

at

a

time

each

higher-level

node

being

connected

to

two

lower-level

nodes

-LRB-

in

the

simplest

version

of

the

algorithm

-RRB-

ffl

The

next

node

to

be

established

is

chosen

from

a

large

and

ever-changing

supply

of

candidate

new

nodes

which

compete

with

each

according

to

their

strengths

ffl

In

dreaming

the

network

is

disconnected

from

the

environment

and

states

are

chosen

by

negotiation

between

the

nodes

with

each

node

responsible

for

one

aspect

of

the

state

ffl

This

division

of

responsibility

between

the

nodes

in

dreaming

is

as

follows

Each

node

can

only

see

its

local

state

the

activations

of

itself

and

its

lower-level

neighbours

Its

job

is

to

ensure

that

the

probability

distribution

of

the

local

state

in

dreaming

matches

that

observed

in

waking

Given

the

right

connectivity

if

each

node

does

its

job

correctly

then

the

global

probability

distribution

in

dreaming

will

match

that

in

waking

as

required

This

may

be

compared

with

other

self-organising

and

recruitment

learning

methods

-LRB-

Feldman

Reilly

et

al.

Pearlmutter

Hinton

Carpenter

Grossberg

Fanty

Kohonen

Ash

Diederich

b

Geman

Honavar

Uhr

Knerr

et

al.

M?ezard

Nadal

Miller

et

al.

Alpaydin

Fahlman

Lebiere

Frean

Hirose

et

al.

-RRB-

Note

that

this

algorithm

genuinely

learns

the

connectivity

instead

of

merely

changing

weights

and

it

lacks

the

built-in

modules

and

layering

of

structured

networks

The

algorithm

has

strengths

and

weaknesses

The

strengths

are

the

clear

division

of

labour

between

the

nodes

the

sound

logical

semantics

of

dreaming

-LRB-

see

the

correctness

proof

in

Fletcher

-LRB-

-RRB-

-RRB-

and

its

ability

to

use

learned

concepts

as

a

basis

for

further

learning

by

growing

nodes

on

top

of

previously

grown

nodes

without

restriction

starting

with

no

built-in

domain

knowledge

-LRB-

By

domain

knowledge

I

mean

such

things

as

preprocessing

a

similarity

measure

to

support

clustering

and

in

the

case

of

visual

input

knowledge

of

which

pixels

are

neighbours

-RRB-

The

main

weakness

is

the

mechanism

for

establishing

new

nodes

which

uses

an

ad

hoc

numerical

strength

function

to

select

the

next

node

to

be

established

and

assigns

it

a

Boolean

activation

function

in

an

ad

hoc

way

In

this

paper

I

argue

that

this

weakness

leads

to

over-large

tangled

networks

Tinkering

with

this

mechanism

in

ways

suggested

by

the

theory

of

syntactic

pattern

recognition

can

lead

to

much

smaller

and

more

understandable

networks

Networks

based

on

syntactic

pattern

recognition

have

been

considered

by

Lucas

Damper

-LRB-

-RRB-

and

includes

a

comparison

between

their

approach

and

mine

Further

improvements

follow

if

the

network

is

allowed

to

remove

nodes

which

it

has

mistakenly

established

in

I

describe

some

rudimentary

mechanisms

for

this

which

lead

to

nearly

optimal

networks

in

an

example

problem

In

I

wonder

what

significance

this

has

for

the

theory

of

learning

focusing

on

the

inadequate

feature

language

The

strength

of

a

new

node

Let

us

take

a

closer

look

at

the

concept

of

strength

used

in

selecting

the

next

node

to

be

established

There

are

three

types

of

node

input

established

and

new

nodes

The

new

nodes

compete

with

each

other

to

become

established

that

is

to

be

the

next

node

in

the

network

's

growth

Each

new

node

i

is

connected

to

two

lower-level

input

or

established

nodes

j

and

k

The

local

state

of

i

consists

of

aj

and

ak

the

Boolean

activations

of

j

and

k

-LRB-

note

that

i

itself

has

no

activation

while

it

is

still

new

-RRB-

It

records

the

relative

frequency

of

each

possible

local

state

separately

in

waking

and

dreaming

to

estimate

the

two

local

probability

distributions

This

gives

it

eight

probabilities

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

From

this

information

the

node

has

to

calculate

a

strength

that

is

a

measure

of

how

much

the

network

's

performance

would

improve

if

it

were

established

next

-LRB-

The

strength

need

not

be

a

real

number

it

may

be

of

any

totally

ordered

data

type

-RRB-

The

new

node

with

the

consistently

highest

strength

is

chosen

for

establishment

the

weakest

new

nodes

die

and

are

recycled

with

new

connections

thus

furnishing

a

continual

supply

of

new

nodes

for

consideration

This

process

is

self-evidently

reasonable

except

for

one

doubt

can

a

new

node

determine

from

the

above

eight

numbers

what

effect

it

would

have

on

performance

and

if

so

how

As

a

first

step

in

answering

this

note

that

the

eight

numbers

satisfy

several

constraints

The

four

pw

probabilities

add

up

to

and

so

do

the

four

pd

probabilities

Also

pw

-LRB-

aj

ak

-RRB-

pw

-LRB-

aj

ak

-RRB-

is

the

waking

probability

that

aj

true

and

this

must

equal

the

dreaming

probability

that

aj

true

since

it

is

the

job

of

node

j

to

ensure

this

-LRB-

They

will

not

be

exactly

equal

but

i

should

ignore

the

discrepancy

since

it

is

not

its

job

to

correct

j

's

errors

-RRB-

This

gives

the

constraint

pw

-LRB-

aj

ak

-RRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

and

by

a

similar

argument

applied

to

k

pw

-LRB-

aj

ak

-RRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

It

follows

that

there

is

a

real

number

d

such

that

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

d

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

d

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

d

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

d

There

are

thus

four

degrees

of

freedom

any

three

of

the

pw

probabilities

and

d

d

is

particularly

interesting

as

it

measures

the

difference

between

the

two

distributions

If

node

i

is

established

then

its

job

will

be

to

influence

the

choice

of

dream

states

to

make

pd

pw

in

other

words

to

reduce

d

to

The

larger

d

is

therefore

the

more

effect

the

node

will

have

in

bringing

the

global

dreaming

distribution

closer

to

the

global

waking

distribution

This

suggests

we

define

strength

as

jd

j

This

is

elegant

but

not

very

convincing

It

relies

on

a

vague

notion

of

the

size

of

effect

a

node

has

on

the

global

distribution

it

is

not

clear

how

this

could

be

made

precise

nor

whether

a

node

could

calculate

it

given

only

local

information

Also

the

choice

of

d

to

represent

this

degree

of

freedom

is

arbitrary

any

one-to-one

function

of

d

and

either

pw

or

pd

would

have

equal

claim

to

represent

the

difference

between

pw

and

pd

If

one

looks

at

the

deviation

between

pw

-LRB-

aj

ak

-RRB-

and

pd

-LRB-

aj

ak

-RRB-

for

example

it

is

arguable

that

it

is

not

the

size

of

the

deviation

that

matters

but

the

size

relative

to

pw

-LRB-

aj

ak

-RRB-

and

pd

-LRB-

aj

ak

-RRB-

This

should

be

seen

in

conjunction

with

the

problem

of

choosing

activation

functions

If

node

i

is

to

be

established

it

needs

to

choose

a

Boolean

function

to

calculate

its

activation

ai

from

aj

and

ak

this

should

be

chosen

so

that

ai

detects

a

useful

feature

discovered

in

the

input

There

are

sixteen

possible

functions

of

which

eight

are

merely

negations

of

the

other

eight

Of

the

remaining

eight

one

is

trivial

-LRB-

identically

true

or

identically

false

-RRB-

and

aj

and

ak

are

not

serious

candidates

since

these

features

are

already

detected

by

nodes

j

and

k

That

leaves

five

to

choose

from

aj

ak

aj

ak

aj

ak

aj

ak

aj

ak

where

is

exclusive-or

Suppose

one

waking

probability

say

pw

-LRB-

aj

ak

-RRB-

is

much

higher

than

the

corresponding

dreaming

probability

pd

-LRB-

aj

ak

-RRB-

then

aj

ak

occurs

much

more

often

than

expected

in

the

environment

and

so

represents

a

significant

conjunction

of

features

not

previously

known

about

by

the

network

Hence

we

should

choose

the

activation

function

ai

aj

ak

so

that

i

becomes

a

detector

for

this

significant

conjunction

The

other

case

of

interest

is

when

say

pw

-LRB-

aj

ak

-RRB-

is

very

small

compared

with

pd

-LRB-

aj

ak

-RRB-

Then

this

is

also

a

sign

of

a

significant

relationship

between

aj

and

ak

not

previously

known

about

Treating

aj

ak

as

an

impossible

conjunction

since

its

probability

is

so

low

there

is

now

only

one

Boolean

function

not

trivial

or

equivalent

to

aj

or

ak

we

choose

ai

aj

ak

This

throws

light

on

the

question

of

the

strength

function

it

suggests

that

we

take

the

ratios

of

the

four

corresponding

pairs

of

probabilities

and

combine

the

ratios

The

result

will

be

large

when

one

of

the

cases

of

interest

occurs

A

strength

function

of

this

form

is

strength

ratio

-LRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

-RRB-

ratio

-LRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

-RRB-

ratio

-LRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

-RRB-

ratio

-LRB-

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

-RRB-

where

ratio

is

a

function

which

divides

the

larger

of

its

two

arguments

by

the

smaller

the

division

being

regularised

so

as

not

to

produce

too

large

a

ratio

when

one

argument

is

very

small

One

may

then

choose

the

activation

function

according

to

which

ratio

is

largest

This

form

of

function

produces

better

results

than

simply

using

jd

j

although

it

expresses

roughly

the

same

degree

of

freedom

It

is

the

one

used

in

Fletcher

-LRB-

-RRB-

and

in

of

this

paper

I

hope

I

have

not

succeeded

in

convincing

the

reader

that

this

is

the

answer

to

the

strength

problem

It

is

a

hacker

's

solution

which

works

on

the

example

problem

largely

because

I

tinkered

with

it

until

it

did

However

it

does

at

least

suggest

that

the

local

probability

distributions

contain

enough

information

to

select

good

nodes

to

establish

Example

problem

I

shall

illustrate

the

problem

of

the

strength

function

by

examining

in

detail

the

node

growth

for

the

example

problem

of

Fletcher

-LRB-

-RRB-

My

attitude

to

computer

simulations

is

the

same

as

in

that

paper

they

are

not

experiments

but

heuristic

explorations

intended

to

suggest

conclusions

which

then

have

to

stand

up

on

their

logical

merits

Simulations

are

described

in

this

paper

for

their

expository

value

not

as

experimental

evidence

false

false

true

false

false

false

false

true

false

false

true

true

true

true

true

false

false

true

false

false

false

false

true

false

false

false

false

false

true

false

false

false

false

true

false

false

false

false

true

false

false

false

false

true

false

true

true

true

true

true

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

The

grid

of

input

nodes

-LRB-

b

-RRB-

Two

example

input

patterns

The

example

problem

consists

of

a

square

input

grid

with

one

input

node

for

each

pixel

Figure

-LRB-

a

-RRB-

shows

a

grid

with

the

pixels

numbered

The

rows

are

labelled

and

the

columns

are

labelled

and

Figure

-LRB-

b

-RRB-

shows

two

input

patterns

generated

by

the

environment

each

consisting

of

a

vertical

line

and

a

horizontal

line

Since

there

are

five

possible

positions

for

each

line

there

are

possible

input

patterns

each

equally

probable

The

network

begins

with

input

nodes

no

established

nodes

and

a

supply

of

new

nodes

and

must

grow

into

a

configuration

in

which

it

can

dream

pictures

as

in

Figure

-LRB-

b

-RRB-

Note

that

it

is

not

told

that

the

pixels

form

a

square

grid

it

has

to

work

this

out

for

itself

If

one

were

designing

such

a

network

by

hand

one

would

probably

begin

by

constructing

a

set

of

row

detectors

and

column

detectors

Figure

shows

the

row

detector

for

the

activation

function

is

for

each

established

node

The

others

are

similar

i

j

Figure

The

row

detector

subnetwork

for

The

established

nodes

all

have

activation

function

The

subnetwork

in

Figure

works

as

a

row

enforcer

as

well

as

a

row

detector

In

waking

it

detects

whether

the

whole

row

is

true

in

dreaming

it

determines

whether

the

whole

row

is

to

be

true

Consider

the

activations

of

nodes

i

and

j

the

activation

function

of

i

is

ai

aj

Now

the

conjunction

aj

never

occurs

in

waking

since

whenever

aj

is

true

so

are

both

and

and

hence

the

whole

row

so

node

i

prevents

aj

from

occurring

in

dreaming

either

It

follows

from

this

and

the

activation

function

that

ai

aj

in

every

waking

and

dream

state

Applying

the

same

argument

everywhere

in

Figure

all

the

established

nodes

have

the

same

activation

Any

one

is

true

if

and

only

if

all

the

input

nodes

in

the

row

are

true

The

same

applies

to

the

other

row

and

column

detectors

The

network

consisting

of

all

row

and

column

detectors

will

dream

pictures

containing

varying

numbers

of

horizontal

and

vertical

lines

-LRB-

depending

on

which

row

and

column

detectors

are

true

-RRB-

plus

a

few

stray

pixels

To

obtain

one

horizontal

and

one

vertical

line

we

need

to

connect

the

row

detectors

and

connect

the

column

detectors

The

row

detectors

are

connected

by

building

the

row-choosing

subnetwork

shown

in

Figure

on

top

of

them

the

nodes

labelled

may

be

any

established

node

from

the

corresponding

row

detector

and

the

other

nodes

have

activation

function

y

z

x

w

Figure

The

row-choosing

subnetwork

The

established

nodes

have

activation

functions

or

as

shown

This

subnetwork

works

as

follows

in

dreaming

Node

w

observes

that

ax

always

in

waking

so

it

enforces

the

same

condition

in

dreaming

Further

if

ax

is

true

then

by

x

's

activation

function

one

of

ay

must

be

true

but

they

are

never

both

true

in

waking

so

one

must

be

true

and

the

other

false

Whereas

if

ax

is

false

then

by

x

's

activation

function

both

ay

and

must

be

false

y

and

z

behave

similarly

to

x

The

result

of

all

this

is

that

exactly

one

of

is

true

-LRB-

each

with

probability

-RRB-

this

forces

one

row

of

input

nodes

to

be

true

There

is

a

similar

column-choosing

subnetwork

which

requires

one

of

to

be

true

-LRB-

each

with

probability

-RRB-

and

this

forces

one

column

of

input

nodes

to

be

true

Assuming

the

two

subnetworks

work

independently

each

input

node

is

forced

to

be

true

with

probability

since

it

is

attached

to

one

row

detector

and

one

column

detector

Now

each

input

node

wants

to

be

true

with

probability

because

this

is

the

observed

probability

in

waking

it

achieves

this

by

being

true

when

and

only

when

it

is

forced

to

be

from

above

-LRB-

when

its

row

or

column

is

true

-RRB-

The

consequence

is

that

a

dream

picture

consists

of

one

horizontal

line

one

vertical

line

and

nothing

else

Each

row

and

column

is

true

with

probability

so

each

picture

occurs

with

probability

as

required

This

argument

shows

that

if

each

node

makes

its

local

dreaming

probability

distribution

match

its

local

waking

probability

distribution

then

the

global

distributions

must

also

match

It

assumes

statistical

independence

between

the

rowchoosing

subnetwork

and

the

column-choosing

subnetwork

this

assumption

may

be

verified

by

supposing

it

false

and

deducing

a

violation

of

the

local

probability

distributions

as

follows

If

it

is

false

then

there

are

a

row

and

a

column

-LRB-

say

and

-RRB-

such

that

the

joint

probability

of

choosing

both

is

less

than

-LRB-

the

figure

predicted

from

independence

-RRB-

Therefore

the

probability

of

or

is

greater

than

which

implies

that

the

probability

that

the

common

input

node

is

true

is

also

greater

than

but

this

deviates

from

node

's

waking

probability

of

Hence

statistical

independence

is

a

consequence

of

enforcing

the

local

probability

distributions

not

a

supplementary

assumption

Thus

this

network

of

input

nodes

and

established

nodes

is

successful

in

simulating

the

environment

and

it

appears

to

be

the

smallest

which

would

do

so

The

idea

generalises

in

an

obvious

way

to

input

grids

of

any

size

I

shall

call

this

the

ideal

network

and

take

it

as

the

standard

against

which

to

compare

the

networks

grown

by

the

learning

algorithm

Node

growth

in

the

example

problem

The

purpose

of

this

section

is

to

trace

and

account

for

the

sequence

of

nodes

established

by

the

learning

algorithm

in

the

example

problem

using

a

grid

This

purpose

will

be

only

partly

achieved

since

only

the

first

few

levels

of

the

network

are

understandable

Initially

there

are

only

input

nodes

and

new

nodes

so

each

new

node

is

connected

to

two

input

nodes

There

are

two

cases

either

the

two

input

nodes

are

in

the

same

row

or

same

column

-LRB-

call

this

the

collinear

case

-RRB-

or

the

input

nodes

are

in

different

rows

and

different

columns

-LRB-

the

non-collinear

case

-RRB-

Let

i

be

a

collinear

new

node

connected

to

nodes

and

say

its

local

probability

distributions

are

shown

in

Figure

Let

j

be

a

non-collinear

new

node

connected

to

nodes

and

say

its

local

probability

distributions

are

shown

in

Figure

false

true

false

true

waking

false

true

false

true

dreaming

Figure

The

local

waking

and

dreaming

probability

distributions

for

a

node

i

connected

to

collinear

nodes

and

There

are

four

possible

local

states

-LRB-

false

or

true

false

or

true

-RRB-

and

the

probability

for

each

is

shown

together

with

marginal

totals

false

true

false

true

waking

false

true

false

true

dreaming

Figure

The

local

probability

distributions

for

node

j

connected

to

non-collinear

nodes

and

Note

that

the

dreaming

distributions

are

the

same

for

i

and

j

in

both

cases

the

two

input

nodes

are

choosing

their

activations

independently

in

dreaming

For

i

we

have

d

indicating

that

collinear

input

nodes

are

positively

correlated

for

j

we

have

d

indicating

that

non-collinear

input

nodes

are

negatively

correlated

The

magnitude

of

d

is

greater

in

the

collinear

case

suggesting

that

the

positive

correlations

are

the

most

significant

features

in

the

environment

visible

to

the

network

at

this

stage

Therefore

one

of

the

collinear

nodes

-LRB-

say

i

-RRB-

will

be

the

first

to

be

established

it

chooses

the

activation

function

ai

New

nodes

now

have

the

option

of

connecting

to

i

as

well

as

to

input

nodes

The

interesting

case

is

a

node

k

connecting

to

i

and

another

node

in

the

same

row

say

Its

local

probability

distributions

are

shown

in

Figure

-LRB-

Again

the

dreaming

distribution

is

obtained

from

the

independence

of

i

and

during

dreaming

-RRB-

false

true

false

true

i

waking

false

true

false

true

i

dreaming

Figure

The

local

probability

distributions

for

node

k

connected

to

nodes

i

and

false

true

false

true

k

waking

false

true

false

true

k

dreaming

Figure

The

local

probability

distributions

for

node

l

connected

to

nodes

k

and

Node

k

is

very

strong

due

to

the

in

the

waking

distribution

-LRB-

see

the

formula

for

strength

at

the

end

of

-RRB-

reflecting

the

fact

that

when

ai

is

true

the

whole

of

row

must

be

true

Hence

nodes

like

k

are

quickly

established

as

soon

as

the

i-type

nodes

on

which

they

depend

have

been

established

The

activation

function

chosen

is

ak

ai

due

to

pw

-LRB-

ai

-RRB-

as

explained

in

Subsequent

nodes

to

be

established

are

l

where

al

ak

and

m

where

am

ai

-LRB-

being

another

i-type

node

in

the

same

row

as

i

say

-RRB-

The

local

probability

distributions

for

l

and

m

before

establishment

are

shown

in

Figures

and

The

network

so

far

is

shown

in

Figure

false

true

false

true

i

waking

i

false

true

false

true

i

dreaming

i

Figure

The

local

probability

distributions

for

node

m

connected

to

nodes

i

and

i

j

i

m

l

k

Figure

The

different

types

of

nodes

which

may

be

established

and

the

connections

between

them

Most

of

the

first

established

nodes

are

of

one

of

these

types

Referring

back

to

the

ideal

network

in

it

is

clear

that

nodes

of

type

i

and

m

are

what

we

wanted

they

are

part

of

a

developing

row

detector

However

nodes

of

type

l

are

unwanted

-LRB-

j-type

nodes

are

not

established

-RRB-

The

significant

point

however

is

that

k-type

nodes

have

the

wrong

activation

function

k

had

a

choice

between

defining

ak

ai

-LRB-

on

the

grounds

that

pw

-LRB-

ai

-RRB-

is

higher

than

pd

-LRB-

ai

-RRB-

-RRB-

and

defining

ak

ai

-LRB-

on

the

grounds

that

pw

-LRB-

ai

-RRB-

-RRB-

and

it

wrongly

chose

the

latter

One

may

fix

this

by

altering

the

procedure

which

chooses

the

activation

function

so

that

it

is

more

ready

to

choose

according

to

pw

being

higher

than

pd

than

the

other

way

round

Then

appropriate

row-detector

nodes

are

established

such

as

-LRB-

-RRB-

-LRB-

-LRB-

-RRB-

-RRB-

However

other

undesirable

nodes

appear

particularly

of

the

form

The

type

of

node

that

is

favoured

depends

on

the

arbitrary

details

of

the

strength

function

and

the

choice

of

activation

functions

However

in

each

case

the

nodes

are

not

what

one

expects

and

soon

the

role

of

each

node

becomes

impossible

to

understand

due

to

the

tangled

network

structure

The

resulting

networks

are

successful

in

that

they

can

indeed

simulate

the

environment

but

they

tend

to

be

several

times

the

size

of

the

ideal

network

and

the

dreaming

algorithm

takes

longer

to

settle

down

and

is

less

successful

in

making

the

local

probability

distributions

match

No

doubt

it

is

possible

by

trial

and

error

to

contrive

a

strength

function

and

choice

of

activation

functions

which

produces

the

ideal

network

However

to

do

so

would

be

to

miss

the

point

of

the

exercise

The

moral

is

that

our

lack

of

a

principled

approach

to

strength

and

choice

of

activation

functions

means

that

the

node

growth

has

no

clear

sense

of

direction

This

conclusion

follows

not

from

performance

statistics

-LRB-

which

I

have

deliberately

omitted

-RRB-

but

from

the

example

nodes

above

where

it

is

not

evident

from

the

local

probability

distributions

which

are

the

best

nodes

to

establish

The

syntactic

viewpoint

In

the

absence

of

a

general

probabilistic

theory

of

node

growth

in

arbitrary

environments

to

make

further

progress

I

think

we

need

a

world

model

that

is

a

set

of

assumptions

about

the

structure

of

the

environment

to

guide

learning

This

limits

the

universality

of

the

learning

algorithm

of

course

The

goal

is

to

obtain

the

greatest

benefit

from

the

weakest

assumptions

My

informal

world

model

is

that

the

input

picture

is

blank

-LRB-

all

pixel

activations

false

-RRB-

except

where

features

-LRB-

sets

of

true

activations

-RRB-

are

present

the

picture

is

then

the

pixel-by-pixel

disjunction

of

all

features

present

Features

may

be

classified

according

to

the

role

they

play

in

the

total

picture

The

goal

of

the

network

is

to

discover

the

features

and

classify

them

In

the

example

problem

the

features

are

the

lines

and

are

classified

together

as

rows

since

there

is

always

exactly

one

of

them

in

the

picture

-LRB-

they

are

rival

fillers

for

one

slot

of

the

picture

specification

to

use

terminology

from

a

different

branch

of

artificial

intelligence

-RRB-

and

and

are

classified

together

as

columns

similarly

A

natural

formalism

for

expressing

this

a

phrase-structure

grammar

Picture

Row

Column

Row

j

j

j

j

Column

j

j

j

j

where

A

B

C

means

that

a

feature

A

consists

of

subfeatures

B

C

-LRB-

in

general

one

would

need

to

specify

how

they

are

linked

together

-RRB-

and

A

B

j

j

C

means

that

features

B

C

are

classified

together

as

varieties

of

feature

A

This

grammatical

technique

for

specifying

patterns

and

the

recognition

and

learning

procedures

suggested

by

it

is

called

syntactic

pattern

recognition

-LRB-

Gonzalez

Thomason

Fu

-RRB-

and

has

been

applied

to

neural

networks

by

Lucas

Damper

-LRB-

-RRB-

This

approach

is

particularly

appealing

since

there

is

a

well-developed

theory

of

formal

languages

and

a

grammar

like

the

one

above

immediately

suggests

a

recognition

network

Probability

distributions

may

be

modelled

by

stochastic

grammars

learning

-LRB-

grammatical

inference

-RRB-

is

possible

and

correction

of

distorted

patterns

may

be

specified

-LRB-

if

not

implemented

-RRB-

as

a

process

of

syntax-directed

translation

This

may

be

a

route

towards

a

sound

theory

of

neural

networks

at

the

very

least

it

is

a

welcome

shift

of

perspective

For

a

grammar

like

the

one

above

the

corresponding

network

has

an

input

node

for

every

terminal

and

an

established

node

for

every

non-terminal

For

every

production

rule

A

B

C

there

are

connections

from

B

C

into

A

and

A

's

activation

is

the

conjunction

of

the

activations

of

B

C.

For

a

production

rule

A

B

j

j

C

the

connections

are

the

same

but

the

activation

function

is

disjunction

The

resulting

networks

are

more

general

than

mine

in

two

respects

Recursive

production

rules

lead

to

feedback

connections

this

powerful

and

important

aspect

I

lead

aside

in

this

paper

Also

there

may

be

more

than

two

incoming

connections

to

an

established

node

this

is

inessential

as

one

may

always

implement

a

finite

conjunction

or

disjunction

by

repeated

binary

conjunctions

or

disjunctions

If

one

restricts

the

grammars

to

conform

to

my

networks

the

class

of

grammars

is

close

to

that

used

by

Lucas

Damper

-LRB-

-RRB-

They

use

strictly

hierarchical

context-free

grammars

where

each

non-terminal

is

assigned

a

level

and

each

production

rule

is

of

the

form

A

BC

with

B

and

C

one

level

below

A

If

one

relaxes

this

by

allowing

B

and

C

to

be

at

any

level

below

A

then

one

obtains

essentially

my

class

of

grammars

The

reason

for

their

tight

restrictions

is

that

known

grammatical

inference

algorithms

are

very

computationally

expensive

for

more

general

grammars

They

consider

a

different

type

of

problem

from

mine

recognising

sequences

of

short

line

segments

called

direction

vectors

produced

by

preprocessing

in

a

character

recognition

system

Learning

is

a

process

of

cluster

formation

each

input

sequence

is

assigned

to

the

nearest

cluster

unless

it

is

too

far

from

any

existing

cluster

in

which

case

it

is

taken

as

the

prototype

of

a

new

cluster

The

main

difference

from

my

example

problem

is

that

they

provide

the

network

with

a

built-in

knowledge

of

the

geometry

of

the

input

grid

via

the

preprocessing

-LRB-

which

turns

the

pixel

image

into

a

sequence

of

direction

vectors

-RRB-

and

the

distance

measure

used

for

clustering

My

interest

is

in

how

a

network

might

learn

this

geometry

for

itself

from

the

statistical

properties

of

the

input

this

is

one

reason

why

I

avoid

clustering

techniques

-LRB-

the

primary

reason

is

the

poor

semantics

of

distance

functions

-RRB-

Having

introduced

the

syntactic

perspective

I

will

not

in

this

paper

make

full

use

of

it

-LRB-

See

for

critical

discussion

of

this

-RRB-

I

simply

use

it

to

justify

or

rationalise

the

following

decision

The

activation

function

of

an

established

node

i

connected

to

j

and

k

must

be

either

conjunction

or

disjunction

the

choice

being

made

according

to

whether

pw

-LRB-

aj

ak

-RRB-

is

greater

or

less

than

pd

-LRB-

aj

ak

-RRB-

respectively

Moreover

strength

must

be

a

function

only

of

pw

-LRB-

aj

ak

-RRB-

pd

-LRB-

aj

ak

-RRB-

and

p

-LRB-

aj

-RRB-

p

-LRB-

ak

-RRB-

the

last

is

the

probability

that

aj

ak

would

have

if

aj

and

ak

were

statistically

independent

and

is

the

same

in

waking

and

dreaming

-LRB-

which

is

why

I

have

dropped

the

subscript

-RRB-

Call

these

three

probabilities

simply

pw

pd

and

p

This

decision

amounts

to

ignoring

the

deviations

for

aj

ak

aj

ak

and

aj

ak

even

as

in

some

of

the

examples

in

when

the

waking

probability

is

zero

Strength

is

calculated

as

follows

If

nodes

j

and

k

belonged

to

unrelated

parts

of

the

network

then

the

events

aj

and

ak

would

be

statistically

independent

hence

the

probability

of

aj

ak

would

be

p

The

correct

probability

is

pw

so

the

difference

pw

p

may

be

called

the

correlation

between

aj

and

ak

which

it

is

i

's

task

to

enforce

The

difference

pd

p

is

the

correlation

already

enforced

by

the

network

prior

to

i

's

establishment

The

node

should

be

given

a

high

strength

if

pw

p

is

large

enough

to

be

worth

enforcing

and

significantly

different

from

pd

p

-LRB-

hence

not

already

enforced

-RRB-

Among

the

multitude

of

strength

functions

with

these

properties

the

following

works

well

Strength

is

set

to

zero

if

pw

p

and

pw

pd

differ

in

sign

otherwise

it

is

either

ratio

-LRB-

pw

p

pd

-RRB-

or

ratio

-LRB-

pw

pd

pd

-RRB-

whichever

is

smaller

in

modulus

Here

ratio

is

a

regularised

division

operation

as

in

the

purpose

of

dividing

by

pd

is

to

give

a

larger

weight

to

differences

in

small

probabilities

as

in

The

results

of

this

choice

of

strength

and

activation

functions

are

shown

in

the

next

two

figures

Figure

shows

some

typical

row

detectors

They

are

not

identical

to

the

ideal

row

detectors

in

however

they

are

just

as

good

since

by

the

same

argument

as

in

all

the

activations

in

a

row

detector

are

constrained

to

be

the

same

and

to

be

true

when

and

only

when

all

the

input

nodes

in

the

row

are

true

Note

however

that

is

sub-optimal

in

that

it

uses

five

established

nodes

instead

of

four

Its

nodes

were

established

in

the

order

a

b

c

d

e

c

was

a

mistake

-LRB-

its

activation

is

not

forced

to

be

equal

to

those

of

a

and

b

-RRB-

but

e

corrected

the

mistake

at

the

cost

of

an

extra

node

The

mistake

was

not

the

fault

of

the

strength

function

if

there

had

been

a

new

node

connected

to

and

a

it

would

have

been

stronger

than

c

-LRB-

see

the

local

probability

distributions

Figures

and

-RRB-

and

so

would

have

been

established

instead

The

explanation

is

that

the

network

can

not

try

every

possible

new

node

and

hence

occasionally

overlooks

the

strongest

possible

node

a

b

d

c

e

Figure

Examples

of

typical

row

detectors

grown

using

only

and

activation

functions

and

with

no

disestablishment

of

nodes

a

b

c

d

d

Figure

A

typical

row-choosing

subnetwork

grown

using

only

and

activation

functions

and

with

no

disestablishment

of

nodes

is

not

present

it

should

have

been

established

instead

of

d.

Figure

shows

a

typical

row-choosing

subnetwork

The

nodes

are

established

in

the

order

a

b

c

d

The

final

node

d

is

a

mistake

should

have

been

chosen

instead

As

new

nodes

would

be

stronger

than

d

so

again

the

mistake

is

the

fault

of

the

new

node

mechanism

for

not

trying

rather

than

the

fault

of

the

strength

function

Unfortunately

in

this

case

the

mistake

can

not

be

corrected

If

the

new

node

is

subsequently

formed

it

will

find

that

the

correlation

between

c

and

is

already

roughly

enforced

by

the

rest

of

the

network

via

b

and

d

The

correlation

is

not

well

enough

enforced

to

choose

a

single

row

correctly

in

each

dream

state

-LRB-

sometimes

both

and

will

be

chosen

and

sometimes

no

row

will

be

-RRB-

but

it

is

too

well

enforced

for

to

justify

establishing

itself

The

result

is

a

network

of

correct

row

and

column

detectors

-LRB-

occasionally

with

a

sub-optimal

arrangement

-RRB-

and

imperfect

row

and

column-choosing

subnetworks

a

good

approximation

to

the

ideal

network

in

For

simplicity

I

have

been

considering

the

problem

In

fact

the

imperfections

only

become

noticeable

in

the

case

the

network

often

solves

the

problem

perfectly

I

conclude

that

the

new

strength

function

and

activation

functions

work

well

on

this

problem

but

to

be

completely

successful

one

needs

a

sufficiently

large

supply

of

new

nodes

to

try

every

possible

way

of

connecting

to

two

existing

nodes

before

choosing

the

strongest

Mistakes

in

choosing

the

strongest

node

can

sometimes

be

corrected

but

not

always

This

problem

will

obviously

get

worse

as

the

problem

size

is

increased

Disestablishing

bad

nodes

The

need

for

disestablishment

and

two

mechanisms

The

detailed

discussion

of

the

previous

section

has

revealed

the

inadequacy

of

the

strategy

underlying

node

growth

This

was

that

the

algorithm

seeks

the

best

configuration

by

establishing

one

node

at

a

time

where

each

node

established

is

the

best

addition

to

the

existing

network

-LRB-

using

the

Incremental

Growth

Assumption

-LRB-

Fletcher

-RRB-

-RRB-

This

strategy

may

be

criticised

on

several

grounds

one

weakness

is

that

the

algorithm

chooses

the

next

node

by

competition

between

candidates

and

since

it

can

not

try

every

possible

candidate

it

may

miss

the

best

node

This

was

obvious

from

the

start

but

I

hoped

that

if

the

wrong

node

was

established

then

subsequent

nodes

would

correct

the

error

resulting

in

a

network

that

was

not

far

from

optimal

The

case

of

the

row-choosing

subnetwork

at

the

end

of

the

previous

section

illustrates

the

fallacy

of

this

The

mistakenly

established

node

partly

does

the

job

of

the

node

which

should

have

been

established

and

this

inhibits

the

latter

from

being

established

later

There

is

a

strong

case

therefore

for

allowing

the

network

to

remove

bad

established

nodes

The

main

objection

to

this

is

that

it

threatens

the

incremental

growth

idea

The

intention

was

that

at

each

stage

of

growth

the

network

should

be

the

best

possible

network

of

its

size

due

to

the

optimal

way

in

which

it

was

built

up

if

one

removes

nodes

without

restriction

then

no

such

claim

can

be

formulated

for

the

resulting

network

A

more

pragmatic

objection

is

that

if

one

removes

a

hidden

node

-LRB-

one

with

connections

to

higher-level

nodes

-RRB-

then

these

connections

must

be

reconnected

somewhere

else

or

the

higher

nodes

must

be

removed

with

repercussions

that

might

spread

throughout

a

large

part

of

the

network

The

point

here

is

that

a

hidden

node

has

two

roles

to

enforce

its

local

probability

distributions

and

to

supply

input

to

higher

nodes

Lacking

any

general

answer

to

these

I

shall

simply

describe

two

disestablishment

mechanisms

both

working

on

non-hidden

nodes

only

to

illustrate

what

is

possible

The

first

mechanism

is

a

limited

form

of

backtracking

When

a

node

i

is

established

the

network

compares

its

strength

with

the

strength

-LRB-

at

the

time

of

establishment

-RRB-

of

the

last

node

to

be

established

j

If

i

is

stronger

and

if

i

is

not

connected

to

j

then

the

network

decides

that

j

was

a

mistake

and

that

i

should

have

been

established

instead

hence

it

removes

j

while

establishing

i.

-LRB-

Otherwise

it

simply

establishes

i

as

usual

-RRB-

This

answers

both

of

the

objections

The

algorithm

may

still

be

thought

of

as

seeking

the

best

network

of

each

size

though

it

now

allows

itself

to

change

its

mind

by

replacing

j

by

i

moreover

it

only

removes

j

when

it

is

not

hidden

Note

that

this

backtracking

mechanism

never

backtracks

more

than

one

step

This

is

effective

in

correcting

most

of

the

mistakes

in

building

row

and

column

detectors

In

Figure

the

bad

node

c

is

removed

when

d

is

established

since

d

is

stronger

Unfortunately

this

fails

to

address

the

main

problem

that

when

a

bad

node

is

established

it

may

inhibit

the

right

node

from

being

established

by

partly

doing

its

job

for

it

To

remedy

this

requires

two

steps

the

right

node

must

be

able

to

judge

that

although

its

job

is

being

partly

done

it

is

not

being

done

well

enough

and

the

node

should

be

still

be

established

once

the

right

node

has

been

established

the

bad

node

must

see

that

it

is

redundant

and

remove

itself

The

first

step

needs

a

change

in

the

strength

function

The

case

of

interest

is

when

jpw

pj

is

large

indicating

a

high

waking

correlation

worth

establishing

a

node

for

and

jpd

pj

is

not

as

large

indicating

that

the

job

of

enforcing

the

correlation

is

already

being

partly

done

by

the

existing

network

jpw

pdj

could

be

quite

small

Strength

therefore

should

be

governed

by

jpw

pj

rather

than

jpw

pdj

However

we

can

not

ignore

jpw

pdj

altogether

as

this

would

lead

to

the

establishment

of

redundant

nodes

where

jpw

pdj

is

zero

-LRB-

or

very

small

-RRB-

indicating

that

the

local

probability

distributions

already

match

and

so

the

node

's

job

is

already

being

done

by

the

network

So

how

small

does

jpw

pdj

have

to

be

before

we

dismiss

the

node

as

redundant

One

could

set

an

arbitrary

threshold

for

jpw

pdj

However

a

more

interesting

method

is

to

use

more

of

the

information

which

is

available

to

the

new

node

A

new

node

connected

to

nodes

j

and

k

can

do

more

than

collect

statistics

on

the

local

probability

distributions

in

waking

and

dreaming

it

can

observe

the

dreaming

process

and

record

not

just

the

final

activations

chosen

for

j

and

k

but

also

the

strengths

with

which

those

activations

are

implied

-LRB-

Here

I

am

referring

for

the

first

time

in

this

paper

to

the

logical

negotiation

algorithm

by

which

dream

states

are

chosen

-LRB-

Fletcher

-RRB-

this

algorithm

chooses

activations

for

each

node

with

differing

logical

strengths

These

logical

strengths

are

not

to

be

confused

with

the

strength

of

a

new

node

in

the

competition

for

establishment

-RRB-

The

new

node

can

also

use

its

local

probability

distributions

to

determine

the

logical

strength

with

which

it

would

contribute

to

the

negotiation

process

if

it

were

established

Thus

for

each

dream

state

it

can

judge

whether

it

would

influence

the

state

chosen

This

gives

us

a

more

powerful

criterion

for

the

strength

of

a

new

node

A

new

node

is

strong

if

and

only

if

jpw

pj

is

large

and

the

deviation

between

pw

and

pd

is

sufficient

that

it

would

always

influence

the

choice

of

dream

state

if

it

were

established

By

this

means

the

right

node

may

be

established

even

after

a

bad

node

which

partly

does

its

job

The

second

step

referred

to

above

is

that

the

bad

node

must

see

that

it

is

now

superfluous

-LRB-

assuming

it

serves

no

other

function

-RRB-

We

need

a

way

of

determining

how

much

a

given

suspect

node

is

contributing

to

the

choice

of

states

during

dreaming

so

that

if

it

is

not

contributing

it

may

be

removed

This

approach

is

in

the

spirit

of

Mozer

and

Smolensky

-LRB-

-RRB-

Alpaydin

-LRB-

-RRB-

Karnin

-LRB-

-RRB-

Le

Cun

et

al.

-LRB-

-RRB-

and

Siestma

Dow

-LRB-

-RRB-

who

try

in

various

ways

to

assess

the

relevance

or

saliency

of

a

node

to

a

network

's

performance

-LRB-

Alpaydin

's

method

uses

a

sleep

phase

but

it

is

not

analogous

in

role

to

my

dreaming

phase

-RRB-

In

my

case

the

contribution

of

a

node

may

be

assessed

by

delving

into

the

semantics

of

the

dreaming

process

Periodically

every

non-hidden

established

node

places

itself

under

suspicion

Assume

that

only

one

node

does

this

at

a

time

-LRB-

perhaps

due

to

global

coordination

or

message-passing

between

the

nodes

or

because

these

events

are

so

rare

that

the

probability

of

two

happening

at

once

is

negligible

-RRB-

Then

every

time

a

dream

state

is

chosen

the

suspect

node

has

to

determine

whether

it

is

influencing

the

choice

The

straightforward

way

to

do

this

is

to

choose

a

state

with

and

without

the

suspect

node

participating

and

compare

the

results

However

the

negotiation

process

takes

a

long

time

to

settle

down

this

takes

up

almost

all

the

computer

time

in

a

simulation

-LRB-

and

would

also

in

a

parallel

implementation

-RRB-

So

I

take

a

short-cut

first

the

network

chooses

a

state

without

the

suspect

node

's

participation

then

it

brings

in

the

suspect

node

settles

down

again

-LRB-

this

usually

happens

almost

immediately

-RRB-

and

sees

if

the

state

has

changed

If

over

a

long

run

of

dream

states

the

suspect

never

makes

a

difference

to

the

choice

then

it

is

redundant

and

it

removes

itself

This

procedure

should

be

carried

out

by

the

suspect

node

itself

without

the

need

for

global

coordination

The

final

comparison

of

the

two

states

appears

non-local

but

in

fact

to

determine

whether

the

state

has

changed

the

suspect

need

only

look

at

its

own

and

its

neighbours

activations

-LRB-

this

is

proved

in

the

next

subsection

-RRB-

The

only

non-local

aspect

is

that

the

suspect

needs

to

be

told

when

the

whole

network

has

settled

down

this

would

perhaps

be

done

by

a

special

node

with

direct

access

to

all

nodes

which

determines

when

the

network

has

stopped

changing

and

informs

all

nodes

Some

mechanism

of

this

sort

is

necessary

for

the

dreaming

algorithm

anyway

This

is

a

very

mild

form

of

non-locality

hardly

worse

than

having

a

global

clock

sending

time

signals

instantaneously

to

all

nodes

The

procedure

is

not

self-evidently

correct

The

dream

state

arrived

at

by

first

leaving

out

the

suspect

node

and

then

including

it

might

not

be

the

same

as

the

state

chosen

if

the

suspect

had

participated

from

the

start

In

other

constraint

satisfaction

methods

minimising

an

energy

function

for

example

the

two

procedures

would

in

general

converge

to

different

local

minima

The

underlying

reason

for

this

is

the

poor

semantics

of

energy

functions

one

can

not

even

formulate

let

alone

prove

a

correctness

property

Fortunately

my

constraint

satisfaction

mechanism

is

based

on

logical

inference

and

has

a

sound

semantics

this

allows

me

to

state

and

prove

the

correctness

of

the

suspicion

procedure

The

proof

is

in

the

next

subsection

It

builds

on

the

correctness

proof

in

Fletcher

-LRB-

-RRB-

however

even

without

reference

to

this

the

strategy

of

the

proof

should

be

understandable

In

I

will

show

examples

of

the

suspicion

mechanism

in

action

Correctness

proof

for

the

suspicion

procedure

Let

i

be

the

suspect

node

Let

N

be

the

network

and

the

network

with

i

and

its

connections

removed

From

Fletcher

-LRB-

-RRB-

we

have

the

following

terminology

L

the

language

of

the

network

Lj

the

local

languages

one

for

each

node

j

T

the

theory

of

the

network

with

consequence

relation

Tj

the

local

theories

with

consequence

relations

j

one

for

each

node

j

ss

the

indistinguishability

relation

on

L

A

the

set

of

axioms

p

the

inference

state

OE

the

ordering

on

inference

states

a

the

state

of

the

network

OEOE

the

ordering

on

states

FPC

the

full

propagation

cycle

FPI

the

full

propagation

invariant

which

holds

at

the

start

of

each

FPC

These

are

all

defined

with

respect

to

N.

Let

FPC

and

FPI

be

corresponding

things

for

The

following

observations

are

straightforward

they

are

simply

variations

on

the

theme

of

everything

in

is

in

N

ffl

L.

ffl

A.

ffl

States

of

correspond

one-to-one

with

states

of

N

that

is

every

state

of

may

be

extended

uniquely

to

a

state

of

N

by

assigning

a

value

to

ai

consistent

with

i

's

neighbours

Call

the

correspondence

m

N.

ffl

Any

inference

state

of

is

an

inference

state

of

N.

ffl

is

the

restriction

of

OE

to

inference

states

of

that

is

for

inference

states

and

of

-LRB-

-RRB-

OE

ffl

T

is

a

conservative

extension

of

that

is

for

any

G

and

X

G

X

-LRB-

-RRB-

G

X

The

reason

is

that

the

only

difference

between

T

and

is

that

T

has

an

extra

constant

ai

and

an

axiom

that

ai

equals

a

Boolean

combination

of

other

constants

clearly

this

does

not

increase

the

power

of

the

theory

ffl

For

each

node

j

i

the

local

language

Lj

and

the

local

theory

Tj

ffl

For

X

Y

if

X

ss

Y

then

X

Y

-LRB-

by

the

previous

observation

-RRB-

ffl

FPI

implies

FPI

for

inference

states

of

-LRB-

by

most

of

the

above

-RRB-

Let

Alg

be

the

usual

full

propagation

dreaming

algorithm

namely

set

the

inference

state

p

to

repeat

FPC

until

p

is

unchanged

choose

the

unique

state

a

such

that

a

j

S

a

pa

Let

Alg

s

be

the

suspicion

algorithm

namely

set

the

inference

state

p

to

repeat

FPC

until

p

is

unchanged

choose

the

unique

state

such

that

j

S

a

pa

repeat

FPC

until

p

is

unchanged

choose

the

unique

state

a

such

that

a

j

S

a

pa

compare

m

-LRB-

-RRB-

with

a

Then

the

task

is

to

show

that

the

two

algorithms

lead

to

the

same

state

a

while

Alg

s

also

finds

the

state

which

Alg

would

have

found

if

i

had

not

been

present

THEOREM

Alg

and

Alg

s

both

lead

to

the

OEOE-maximum

state

a

The

state

chosen

in

Alg

s

is

the

PROOF

Invoke

the

following

lemmas

from

Fletcher

-LRB-

-RRB-

the

numbering

in

that

paper

is

shown

here

in

square

brackets

Each

lemma

may

be

applied

to

both

N

and

LEMMA

A

There

are

only

finitely

many

inference

states

satisfying

FPI

-LSB-

Essentially

Lemma

-RSB-

LEMMA

B

If

FPI

holds

and

a

FPC

is

executed

then

the

FPC

-LRB-

i

-RRB-

halts

-LSB-

Lemmas

and

-LRB-

d

-RRB-

-RSB-

-LRB-

ii

-RRB-

preserves

FPI

-LSB-

Lemma

-LRB-

e

-RRB-

-RSB-

-LRB-

iii

-RRB-

increases

p

in

OE

or

leaves

it

unchanged

-LSB-

Lemma

-RSB-

LEMMA

C

If

p

satisfies

FPI

and

is

unchanged

by

FPC

then

there

is

a

unique

a

such

that

a

j

S

a

pa

and

this

a

is

the

OEOE-maximum

-LSB-

The

conclusion

of

the

lemma

is

the

conclusions

of

Lemma

-LRB-

c

-RRB-

and

the

Theorem

which

follow

from

Lemmas

and

via

the

conclusion

of

-LRB-

b

-RRB-

-RSB-

Repeating

the

argument

from

the

earlier

paper

in

Alg

FPI

holds

after

step

and

is

preserved

by

each

FPC

while

p

increases

in

OE

-LRB-

Lemma

B

-RRB-

there

are

only

finitely

many

possible

p

-LRB-

Lemma

A

-RRB-

so

step

must

halt

The

inference

state

after

step

satisfies

FPI

and

is

unchanged

by

FPC

and

hence

-LRB-

Lemma

C

-RRB-

the

state

a

chosen

in

step

is

the

OEOE-maximum

as

required

Similarly

for

Alg

s

FPI

holds

after

step

and

is

preserved

by

each

FPC

while

p

increases

in

there

are

only

finitely

many

possible

p

so

step

must

halt

The

inference

state

after

step

satisfies

FPI

and

is

unchanged

by

FPC

and

hence

the

state

chosen

in

step

is

the

as

required

FPI

implies

FPI

so

FPI

also

holds

after

step

Each

FPC

in

step

preserves

FPI

while

p

increases

in

OE

there

are

only

finitely

many

possible

p

so

step

must

halt

The

inference

state

after

step

satisfies

FPI

and

is

unchanged

by

FPC

and

hence

the

state

a

chosen

in

step

is

the

OEOE-maximum

as

required

In

fact

there

is

a

stronger

result

not

only

the

final

states

but

also

the

final

inference

states

coincide

in

the

two

algorithms

However

our

lemmas

are

not

strong

enough

to

establish

this

and

it

is

not

needed

to

justify

the

suspicion

procedure

Finally

I

will

justify

the

claim

made

in

the

previous

subsection

that

step

of

Alg

s

the

comparison

of

a

with

m

-LRB-

-RRB-

may

be

done

locally

by

the

suspect

node

Let

j

and

k

be

the

lower-level

nodes

connected

to

i

Then

the

claim

is

embodied

in

the

following

theorem

THEOREM

At

the

end

of

Alg

s

a

m

-LRB-

-RRB-

if

and

only

if

a

and

m

-LRB-

-RRB-

assign

the

same

values

to

ai

aj

and

ak

PROOF

As

already

shown

a

is

the

OEOE-maximum

state

of

N

while

is

the

maximum

state

of

If

a

m

-LRB-

-RRB-

then

we

have

m

-LRB-

-RRB-

OEOE

a

and

m

-LRB-

a

-RRB-

Unravelling

the

meaning

of

these

ffl

there

is

an

axiom

-LRB-

X

a

-RRB-

of

A

such

that

a

j

X

and

m

-LRB-

-RRB-

j

X

while

a

and

m

-LRB-

-RRB-

agree

on

which

axioms

of

A

stronger

than

-LRB-

X

a

-RRB-

they

satisfy

ffl

there

is

an

axiom

-LRB-

Y

b

-RRB-

of

such

that

j

Y

and

m

-LRB-

a

-RRB-

j

Y

while

m

-LRB-

a

-RRB-

and

agree

on

which

axioms

of

stronger

than

-LRB-

Y

b

-RRB-

they

satisfy

The

only

way

to

reconcile

these

apparently

conflicting

propositions

is

to

say

that

b

a

and

-LRB-

X

a

-RRB-

A

n

that

is

-LRB-

X

a

-RRB-

is

an

axiom

generated

at

i.

Therefore

X

Li

that

is

X

is

expressed

in

terms

of

ai

aj

and

ak

Since

a

and

m

-LRB-

-RRB-

differ

in

satisfying

X

they

must

differ

in

the

values

they

assign

to

ai

aj

and

ak

as

required

The

converse

is

trivial

Example

of

suspicion

and

disestablishment

The

purpose

of

this

section

is

to

complement

the

formal

treatment

above

with

a

illustration

of

the

role

the

suspicion

mechanism

is

intended

to

play

In

the

case

it

is

not

needed

so

I

will

describe

a

run

Both

disestablishment

methods

are

in

force

but

I

shall

concentrate

on

the

suspicion

mechanism

except

where

the

two

mechanisms

interact

Most

row

and

column

detectors

are

established

perfectly

the

rest

have

one

or

two

superfluous

nodes

The

only

exception

is

the

column

the

network

makes

several

overlapping

attempts

to

build

a

column

detector

shown

in

Figure

One

of

the

attempts

is

completely

removed

as

the

others

grow

to

make

it

redundant

g

is

removed

after

i

and

j

make

it

redundant

the

removal

of

n

is

unfortunate

since

it

served

to

link

two

fragments

but

it

is

later

replaced

with

q

The

result

is

two

partial

column

detectors

which

between

them

work

almost

as

well

as

a

complete

column

detector

-LRB-

they

agree

more

than

of

the

time

-RRB-

Figure

The

column

detector

The

nodes

are

established

in

alphabetical

order

a

node

which

is

later

disestablished

is

shown

as

an

unfilled

circle

with

its

connections

drawn

as

dotted

lines

Once

all

the

row

and

column

detectors

have

formed

the

row-choosing

and

column-choosing

subnetworks

grow

on

top

of

them

Figures

and

In

Figure

the

algorithm

makes

one

mistake

in

connecting

node

e

to

c

instead

of

d

It

recovers

not

by

removing

e

but

by

establishing

f

and

g

so

that

d

is

now

redundant

and

is

promptly

removed

The

resulting

subnetwork

is

perfect

Figure

The

row-choosing

subnetwork

The

nodes

are

established

in

alphabetical

order

a

node

which

is

later

disestablished

is

shown

as

an

unfilled

circle

with

its

connections

drawn

as

dotted

lines

The

column-choosing

subnetwork

in

Figure

gets

itself

into

a

mess

in

repeated

attempts

to

connect

to

Node

e

is

the

first

mistake

it

should

have

connected

to

d

instead

of

b.

Growth

continues

correctly

with

f

and

g

then

there

is

another

mistake

with

h.

e

is

now

redundant

since

the

combination

c

d

f

h

does

its

job

for

it

so

it

is

disestablished

i

is

correctly

established

next

and

since

it

is

stronger

at

the

time

of

establishment

than

the

previous

node

h

was

h

is

removed

by

the

backtracking

mechanism

Thus

both

errors

have

been

corrected

j

and

k

follow

this

is

another

mistake

as

they

should

not

both

be

connected

to

i

It

is

corrected

just

as

in

the

row-choosing

subnetwork

by

establishing

l

so

that

j

is

then

removed

as

redundant

The

subnetwork

is

now

complete

and

correct

Unfortunately

the

next

node

to

be

established

is

an

unwanted

conjunction

node

-LRB-

unrelated

to

this

subnetwork

and

not

shown

in

Figure

-RRB-

which

is

stronger

than

l

hence

l

is

removed

by

the

backtracking

mechanism

Undaunted

the

network

immediately

establishes

another

node

m

to

replace

l

The

bad

conjunction

node

is

later

disestablished

as

redundant

The

column-choosing

subnetwork

is

now

finished

and

is

perfect

Figure

The

column-choosing

subnetwork

The

nodes

are

established

in

alphabetical

order

a

node

which

is

later

disestablished

is

shown

as

an

unfilled

circle

with

its

connections

drawn

as

dotted

lines

If

one

continues

to

run

the

algorithm

there

are

no

good

nodes

left

to

establish

so

the

occasional

spurious

conjunction

node

is

established

due

to

statistical

fluctuations

and

unwanted

nodes

are

occasionally

removed

I

have

included

no

mechanism

to

stop

the

learning

when

there

are

no

good

nodes

left

as

in

any

realistic

environment

learning

would

never

be

complete

In

summary

the

node

growth

algorithm

shows

a

remarkable

capacity

for

tying

itself

up

in

knots

and

then

unravelling

the

knots

often

in

unexpected

ways

I

have

described

one

of

the

more

dramatic

runs

others

go

more

smoothly

but

with

similar

results

Perhaps

I

should

repeat

the

caution

at

the

beginning

of

I

am

not

presenting

experimental

evidence

here

but

simply

illustrating

the

possibilities

of

node

growth

and

disestablishment

My

purpose

will

be

achieved

if

I

have

convinced

the

reader

that

in

a

spontaneously

growing

network

disestablishment

is

necessary

and

that

it

is

feasible

to

disestablish

during

growth

using

a

concept

of

node

relevance

which

is

logically

well-founded

and

efficiently

and

locally

measurable

Conclusions

I

have

shown

how

the

performance

of

the

network

on

the

example

problem

can

be

improved

drastically

by

three

modifications

ffl

changing

the

strength

function

for

new

nodes

ffl

changing

the

choice

of

activation

function

in

a

newly

established

node

ffl

disestablishing

nodes

occasionally

This

in

itself

is

not

remarkable

most

programs

can

be

fine-tuned

to

work

well

on

a

particular

problem

It

is

only

important

in

so

far

as

it

suggests

general

insights

about

the

ability

of

a

network

to

configure

itself

given

only

local

information

and

thus

contributes

to

the

goal

of

a

theory

of

parallel

learning

I

draw

the

following

conclusions

ffl

Successful

node

growth

depends

critically

on

a

theoretical

foundation

for

the

new-node

strength

function

and

choice

of

activation

functions

Even

the

flimsiest

theoretical

justification

-LRB-

the

sketchy

remarks

about

syntactic

pattern

recognition

in

-RRB-

by

limiting

the

activation

functions

to

and

vastly

improves

the

network

The

strength

function

is

still

a

weak

point

ffl

New

nodes

need

not

rely

solely

on

their

local

probability

distributions

to

calculate

their

strength

they

can

observe

the

dreaming

process

and

determine

how

much

influence

they

would

have

on

the

result

if

they

were

established

ffl

If

the

wrong

node

is

established

this

sometimes

distorts

the

future

course

of

learning

the

network

may

never

recover

if

the

bad

node

prevents

good

nodes

from

being

established

Thus

it

needs

to

disestablish

bad

nodes

this

can

be

done

in

a

rough-and-ready

way

Credit

assignment

is

easy

because

the

clear

semantics

of

dreaming

defines

the

role

of

each

established

node

The

performance

improvements

described

in

this

paper

leave

us

freer

to

see

the

fundamental

limitations

of

this

network

and

the

learning

formalism

that

underlies

it

Consider

applying

the

network

to

a

harder

environment

for

example

spaghetti

pictures

-LRB-

containing

a

curve

which

twists

horizontally

and

vertically

around

the

picture

-RRB-

or

pictures

of

rectangles

How

will

it

learn

and

what

things

will

it

never

learn

no

matter

how

long

it

runs

First

it

is

better

at

conjunction

than

disjunction

It

will

learn

to

recognise

longer

and

longer

line

segments

as

it

discovers

the

strong

positive

correlations

between

neighbouring

collinear

pixels

it

will

also

learn

some

corners

Then

in

dreaming

it

will

generate

noisy

spaghetti-like

pictures

with

many

line

segments

some

joined

into

longer

curves

What

it

will

never

learn

is

that

all

horizontal

line

segments

should

be

classified

together

as

horizontal

bars

-LRB-

of

which

there

must

be

two

in

a

rectangle

picture

-RRB-

each

line

segment

is

learned

as

an

individual

feature

although

built

up

out

of

shorter

segments

Thus

the

network

never

learns

position

or

scale

invariance

It

can

not

do

so

since

the

only

criterion

it

has

for

learning

a

disjunction

of

aj

and

ak

is

that

pw

-LRB-

aj

ak

-RRB-

be

very

small

compared

with

pd

-LRB-

aj

ak

-RRB-

this

is

too

weak

a

criterion

as

for

most

pairs

of

features

the

probability

of

cooccurrence

is

very

small

anyway

particularly

when

the

picture

grid

is

large

It

works

well

for

it

is

laborious

for

and

and

would

be

impracticable

for

A

much

better

criterion

is

that

features

aj

and

ak

should

be

classified

together

if

they

play

a

similar

role

in

the

total

picture

for

example

being

the

upper

horizontal

bar

in

a

rectangle

picture

This

is

completely

beyond

the

ability

of

the

network

The

second

deeper

limitation

is

the

impoverished

language

for

defining

features

restricted

to

fixed

conjunctions

and

disjunctions

The

only

way

to

define

horizontal

bar

is

to

define

every

possible

horizontal

line

segment

as

a

conjunction

of

pixels

and

then

make

a

grand

disjunction

of

them

all

Position

scale

and

orientation

invariances

must

be

implemented

by

representing

explicitly

every

possible

transformation

This

is

absurd

and

does

not

deserve

to

work

The

missing

ingredient

is

a

notion

of

iteration

a

horizontal

bar

is

any

number

of

pixels

linked

horizontally

This

corresponds

to

admitting

recursion

in

the

associated

grammar

which

in

turn

implies

feedback

connections

in

the

network

It

also

probably

implies

breaking

the

fixed

one-to-one

correspondence

between

nodes

and

features

It

follows

that

the

architectural

constraints

of

my

network

are

now

an

obstacle

to

progress

and

need

rethinking

Node

growth

is

a

very

crude

form

of

concept

formation

The

challenge

for

the

future

is

to

make

it

less

crude

while

retaining

the

local

nature

of

the

algorithm

A

new

generation

of

self-configuring

networks

which

meets

this

challenge

is

now

under

design

References

Alpaydin

E.

-LRB-

-RRB-

Neural

Models

of

Incremental

Supervised

and

Unsupervised

Learning

Ph.D.

thesis

Ecole

Polytechnique

F?ed?erale

de

Lausanne

Switzerland

Ash

T.

-LRB-

-RRB-

Dynamic

node

creation

in

backpropagation

networks

Connection

Science

Carpenter

G.A.

Grossberg

S.

-LRB-

-RRB-

The

ART

of

adaptive

pattern

recognition

by

a

self-organizing

neural

network

Computer

Diederich

J.

-LRB-

-RRB-

Steps

towards

knowledge-intensive

connectionist

learning

In

J.

Pollack

J.

Barnden

-LRB-

Eds

-RRB-

Advances

in

Connectionist

and

Neural

Computation

Theory

Norwood

New

Jersey

Ablex

Diederich

J.

-LRB-

-RRB-

Instruction

and

high-level

learning

in

connectionist

networks

Connection

Science

Fahlman

S.E.

Lebiere

C.

-LRB-

-RRB-

The

cascade-correlation

architecture

In

D.

S.

Touretzky

-LRB-

Ed

-RRB-

Advances

in

Neural

Information

Processing

Systems

San

Mateo

California

Morgan-Kaufmann

Fanty

M.

-LRB-

-RRB-

Learning

in

Structured

Connectionist

Networks

Ph.D.

thesis

Computer

Science

Department

University

of

Rochester

Feldman

J.A.

-LRB-

-RRB-

Dynamic

connections

in

neural

networks

Biological

Cybernetics

Fletcher

P.

-LRB-

-RRB-

A

self-configuring

network

Connection

Science

Frean

M.

-LRB-

-RRB-

The

Upstart

Algorithm

a

method

for

constructing

and

training

feedforward

neural

networks

Neural

Computation

Fu

K.S.

-LRB-

-RRB-

Syntactic

Pattern

Recognition

and

Applications

Englewood

Cliffs

New

Jersey

Prentice-Hall

Geman

S.

-LRB-

-RRB-

Notes

on

a

self-organising

machine

In

G.

E.

Hinton

J.

A.

Anderson

-LRB-

Eds

-RRB-

Parallel

Models

of

Associative

Memory

updated

edition

Hillsdale

New

Jersey

Lawrence

Erlbaum

Associates

Gonzalez

R.C.

Thomason

M.G.

-LRB-

-RRB-

Syntactic

Pattern

Recognition

an

Introduction

Reading

Massachusetts

Addison-Wesley

Hirose

Y.

Yamashita

K.

Hijiya

S.

-LRB-

-RRB-

Back-propagation

algorithm

which

varies

the

number

of

hidden

units

Neural

Networks

Honavar

V.

Uhr

L.

-LRB-

-RRB-

A

network

of

neuron-like

units

that

learns

to

perceive

by

generation

as

well

as

reweighting

of

its

links

In

D.

S.

Touretzky

G.

E.

Hinton

T.

J.

Sejnowski

-LRB-

Eds

-RRB-

Proceedings

of

the

Connectionist

Models

Summer

School

San

Mateo

California

Morgan-Kaufmann

Karnin

E.D.

-LRB-

-RRB-

A

simple

procedure

for

pruning

back-propagation

trained

neural

networks

IEEE

Transactions

on

Neural

Networks

Knerr

S.

Personnaz

L.

Dreyfus

G.

-LRB-

-RRB-

Single

layer

learning

revisited

a

stepwise

procedure

for

building

and

training

a

neural

network

In

F.

Fogelman-Souli

e

J.

H?erault

-LRB-

Eds

-RRB-

Neurocomputing

Algorithms

Architectures

and

Applications

NATO

ASI

Series

Berlin

Springer

Kohonen

T.

-LRB-

-RRB-

The

neural

phonetic

typewriter

Computer

Le

Cun

Y.

Denker

J.S.

Solla

S.A.

-LRB-

-RRB-

Optimal

brain

damage

In

D.

S.

Touretzty

-LRB-

Ed

-RRB-

Advances

in

Neural

Information

Processing

Systems

San

Mateo

California

Morgan-Kaufmann

Lucas

S.M.

Damper

R.I.

-LRB-

-RRB-

Syntactic

neural

networks

Connection

Science

M?ezard

M.

Nadal

J.-P

-LRB-

-RRB-

Learning

in

feedforward

layered

networks

the

tiling

algorithm

Journal

of

Physics

A

Miller

G.F.

Todd

P.M.

Hegde

S.U.

-LRB-

-RRB-

Designing

neural

networks

using

genetic

algorithms

In

J.

D.

Schaffer

-LRB-

Ed

-RRB-

Proceedings

of

the

Third

International

Conference

on

Genetic

Algorithms

San

Mateo

California

Morgan-Kaufmann

Mozer

M.C.

Smolensky

P.

-LRB-

-RRB-

Using

relevance

to

reduce

network

size

automatically

Connection

Science

Pearlmutter

B.A.

Hinton

G.E.

-LRB-

-RRB-

G-maximization

an

unsupervised

learning

procedure

for

discovering

regularities

In

J.

S.

Denker

-LRB-

Ed

-RRB-

Neural

Networks

for

Computing

The

American

Institute

of

Physics

Snowbird

Utah

April

Reilly

D.L.

Cooper

L.N.

Elbaum

C.

-LRB-

-RRB-

A

neural

model

for

category

learning

Biological

Cybernetics

Siestma

J.

Dow

R.J.F.

-LRB-

-RRB-

Creating

artificial

neural

networks

that

generalize

Neural

Networks

neural networks
recruitment learning
unsupervised learning

self-configuring networks
self-organising networks